Cosmos-T-80M is the first model in the Cosmos-T series — small, from-scratch, decoder-only Transformers pretrained on chain-of-thought data for research and demos. It is an instruct-style model trained with explicit … reasoning blocks. > ⚠️ Research / demo model. 80M parameters trained on only ~215k tokens. It is intentionally small so you can run it on a free Kaggle T4 or in a HF Space demo. It is not a useful general assistant and will produce incoherent or hallucinated output on most prompts. The point of this release is the architecture + training recipe, not state-of-the-art quality. — Tied embeddings — without tying, the 152k Qwen vocab alone would cost ~117M params (embed + head) and blow the The training loss descends cleanly to ~0.45, but the validation loss bottoms out around step 300 (val ≈ 5.6) and then climbs to 7.09 by step 1650. This is heavy overfitting, and is the expected behavior for an 80M-parameter model trained on only ~215k tokens (roughly 0.005 tokens per parameter, ~4000× below Chinchilla-optimal). Interpretation: the model has memorized the reasoning style and most of the surface patterns of the chain-of-thought corpus (train perplexity ~1.57 is…
Модальности:
Генерация текста
Области применения:
Логика и рассуждение Следование инструкциям
Задача: Генерация текста
Автор: wop
Теги: chain-of-thought, reasoning, instruct, pretrained-from-scratch, small-model, decoder-only, transformer, qwen-tokenizer
Лайков: 4 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
