OdinNext is a 138M-parameter causal language model that replaces softmax self-attention with an HGRN2-style gated linear recurrence. This repository is the base pretrained model — trained from scratch on ~101.6B tokens of curated data (the Dolmino mix) on two AMD Strix Halo (gfx1151) machines. This is a base model: it completes and continues text. It is not an instruction-tuned or chat model — no SFT, DPO, RLHF, or chat template. An instruction-tuned variant is available at joelhenwang/OdinNext-138M-Instruct. — Repo: joelhenwang/OdinNext-138M-Base — main: EMA-shadowed weights (decay 0.999), recommended. — live: raw training weights at the same step. — Context window: 2,048 tokens in the released inference code. — License: Apache-2.0. > Uses custom Transformers code. Loading with trustremotecode=True executes > Python from this repo. Review the files or pin a commit before trusting it. with a per-layer state shaped [B, nheads, headfdim, headidim] = [B, 6, 128, 128]. This state is constant in size with respect to context length**, giving O(1)-per-token decoding rather than a growing KV cache. Hybrid RoPE: even layers (0, 2, …, 14) apply RoPE to q/k (θ = 100,000); odd layers are…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: joelhenwang
Теги: odinnext, hgrn2, linear-attention, recurrent, causal-lm, custom_code, base-model, fp16
Лайков: 3 | Загрузок: 100
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
