Kiyo-65M is a decoder-only language model pretrained from scratch on 80B tokens drawn from FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-v3-train. It is the smallest member of the Kiyo family and uses a Llama-style decoder architecture with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings, following the SmolLM2 architecture family and reusing its 49k-token vocabulary. Kiyo-65M scales that recipe down to 15 layers and a 512-dimensional residual stream, roughly halving the parameter count while keeping the same data mixture and training code. It is trained independently from a random initialization, not distilled or pruned from Kiyo-135M. Zero-shot accuracy (%) on the same four benchmarks used for Kiyo-230M-Preview, from the Open SLM Leaderboard. This is a base model, not instruction-tuned — it continues text rather than following instructions. At 65M parameters and 80B training tokens it still produces fluent, grammatical continuations and remains strong on language completion, but accuracy falls off sharply on quantitative reasoning, multi-step logic, and context tracking, where it sits close to chance on the harder items. The 2,048-token context…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: DedeProGames
Теги: llama, causal-lm, language-model, base-model, smollm, kiyo, conversational, en
Лайков: 3 | Загрузок: 1,227
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.