A 122.7M parameter language model trained from scratch. TextModel-v1 is a Llama-style decoder-only transformer, built to be a strong, honestly-evaluated entry on small language model benchmarks, in particular GCI-Bench. TextModel-v1 was trained from random initialization, not fine-tuned from any existing checkpoint. Training data was 1.64 billion tokens: 85% FineWeb-Edu, mixed with 15% synthetically generated documents covering causal reasoning across 20 topic domains and six causal structure templates. The synthetic data was generated fresh for this project and never overlaps with any published benchmark’s actual test items. Sequences were packed to 1024 tokens, trained for 50,000 steps at batch size 32 on a single A100. This checkpoint is step 50,000 of that run, chosen after sweeping all saved checkpoints against the benchmarks below and picking the one with the strongest overall result. All numbers below are from the reference evaluation scripts run directly against this checkpoint, not self-reported estimates. GCI-Bench measures whether a model’s attention and gradient dynamics prioritize causally relevant context over distractor text, using the reference harness published at…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: TobiasLogic
Теги: llama, from-scratch, tiny-llm, en, eval-results
Лайков: 3 | Загрузок: 22
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
