Chinchilla-1 is a ~73.5M parameter instruction-tuned language model built on the nanochat d6 architecture. The name references Chinchilla scaling laws — this is a research-scale model, not a production LLM. This model was created as a learning exercise in SFT infrastructure (Modal, torchrun, MuonAdamW) and to explore nanochat’s custom architecture (value embeddings, smear mechanism, backout residuals). It should be treated as a baseline for understanding how small transformers behave after SFT — not as a usable chatbot. — Tool-augmented reasoning — correctly orchestrates / tokens to invoke Python for tasks like letter counting. Both checkpoints correctly emit ‘strawberry’.count(‘r’) and interpret the result. See evaluation Q7. — Basic Python generation — produces correct, idiomatic code for simple algorithmic tasks (e.g. palindrome: s == s[::-1] with alphanumeric filtering). See evaluation Q6. — Chat format fluency — consistently generates proper / conversation markers with no format breakages. Fluent, grammatically correct English prose. See evaluation §5.3–5.4. — Lightweight — 73.5M params, runs on CPU at ~2 tok/s, fits in ~280 MB (bf16) or ~140 MB (fp16 via NANOCHATDTYPE`).…
Модальности:
Генерация текста
Области применения:
Вызов функций (Tool use)
Задача: Генерация текста
Автор: rajofearth
Теги: nanochat, tool-use, small-llm, research, custom-architecture, en
Лайков: 3 | Загрузок: 9
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
