A decoder-only language model with 296,960 parameters, pretrained from scratch on 200M tokens of HuggingFaceFW/fineweb-edu (sample-10BT). It is an extreme variant of DedeProGames/DynamicMind-Mini (8.9M params), scaled down roughly 30x. The name is the point: this model is not built to be good. It is built to find out what a third of a million parameters actually buys you. A 3.3x scale-up trained under identical conditions — same tokenizer, same data, same token budget, same aspect ratio — is published as DedeProGames/LowOnMind-1M. The two models form a controlled pair; see Scaling comparison below. Parameter split: 65,536 in the tied embedding, 231,360 across the six blocks, 64 in the final norm. — QK-Norm — per-head RMSNorm on queries and keys before RoPE. Costs 32 params per block and is what makes an LR of 2e-3 survivable at this scale. — Precomputed RoPE — cos/sin are built once and reused instead of being recomputed every forward, and the cache re-expands automatically for sequences longer than maxpositionembeddings. — Scaled residual init — oproj and downproj initialize at std / sqrt(2 numlayers) instead of a flat 0.02. — 1024-token vocabulary — required to fit the…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: DedeProGames
Теги: lowonmind, tiny-lm, pretrained-from-scratch, scaling-limits, custom_code, en
Лайков: 4 | Загрузок: 866
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
