MONOSTEP is a small (~16.6M parameter) experimental research model trained on GSM8K. Instead of predicting one token at a time, it predicts a fixed block of SLOTS = 4 tokens per forward pass through a set of sequential «slot» heads — a lightweight take on multi-token prediction. It is built on the GPT-2 byte-level tokenizer with a few added chat/control special tokens. > ⚠️ This is a tiny, rough research artifact. Expect imperfect, playful answers — > it is not suitable for production use. — Trunk — token + learned positional embeddings, a normfirst Transformer encoder (GELU, dimfeedforward = 4·dmodel), followed by masked mean-pooling over the sequence and a final LayerNorm. Produces a single shared vector hshared summarizing the prefix. — Slots — SLOTS independent heads applied sequentially. Each slot takes [hshared, hprev], runs a small residual MLP + LayerNorm, and projects to the vocabulary. The hidden state is threaded slot-to-slot so the block is generated left-to-right within one forward pass. — Decoding — autoregressive in blocks of SLOTS tokens: emit up to 4 tokens, append the non- ones to the context, and repeat until . Special tokens added to the GPT-2 tokenizer (order…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: wop
Теги: gsm8k, multi-token-prediction, experimental, en
Лайков: 4 | Загрузок: 15
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
