A 100.6M-parameter decoder-only language model by Manprot Author: Dominik Andrzejewski > Note: this is not a pure base model. Its lineage includes a chat fine-tune, so it can show chat-style behaviour. It continues text but does not follow instructions reliably. — Training lineage — Datasets — Architecture — Evaluation — Compute — Loading — Limitations — Repository contents — Author and citation 1. Pretraining (75M): a 75M-parameter model was pretrained from scratch on English text, single pass. 2. Growth to 100M: grown from 12 to 18 layers (identity-initialised) and pretrained further on new documents. 3. Chat fine-tune: fine-tuned for chat. These weights are the starting point of the expansion. 4. Expansion: continued pretraining on new documents from the same sources, to step 19,000 with a 262k-token batch. 5. Cooldown: the learning rate was decayed to its minimum over 8,000 steps (2.10B tokens) on a second, unseen corpus, starting from step 19,000. This is the checkpoint released here. Two other cooldowns are not in this release’s lineage: a local 170-step experiment from step 13,000, and a 4,000-step cooldown from step 15,000 that was the previous version of this repository…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: Manprot
Теги: vbq, pretraining, decoder-only, small-language-model, en
Лайков: 3 | Загрузок: 648
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
