Architecture details, inherited from DynamicMind-Mini with modifications: GQA, SwiGLU, RMSNorm, tied embeddings, QK-Norm per head, precomputed RoPE with automatic re-expansion, and residual projections initialized at std / sqrt(2 numlayers). At 203 tokens per parameter this run still sits far past the Chinchilla-optimal ratio. Train and validation loss tracked each other for the entire run — this model is parameter-limited, not data-limited. Perplexity is not comparable across tokenizers, but it is comparable between these two models because they share one. Bits per character (loss / ln 2 / 2.35 chars-per-token) is the portable figure. With a 1024-token byte-level vocabulary, no long word exists as a single token — the model has to assemble every one of them from fragments. The fraction of emitted words that are real English words measures this directly, and neither validation loss nor a multiple-choice benchmark captures it. Measured over 64 unconditional samples (5,647 words). The reference lexicon is every lowercase word appearing at least 5 times in a 20k-document sample of the training corpus, so the corpus row is the practical ceiling rather than 100%. At 98.0% against a…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: SLM-Archive
Теги: lowonmind, tiny-lm, pretrained-from-scratch, scaling-limits, custom_code, en
Лайков: 3 | Загрузок: 148
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
