A 3-million-parameter language model trained from scratch. It is an arithmetic specialist: it handles integer arithmetic well for its size, and stays near chance on general tasks, at a footprint that runs almost anywhere. The model reads and writes numbers least-significant-digit first, with a digit-atomic tokenizer. Aligning carry propagation with reading order is what lets a model this small do arithmetic. The digit reversal happens inside the tokenizer, so you pass and receive normal text. This requires trustremotecode=True. Full test sets, 0-shot. accnorm` for multiple choice, accuracy for ArithMark-2. The model is built for integer arithmetic. Non-arithmetic tasks sit near chance, which is expected at this size. Pretrained from scratch with a digit-atomic 4k tokenizer, least-significant-digit order, and a large stream of freshly generated synthetic integer arithmetic (addition, subtraction, multiplication, exact division, mixed and parenthesised multi-operator expressions), mixed with TinyStories for language. Then a short fine-tune on public benchmark train splits plus more arithmetic. No evaluation items were used at any stage. Fresh generation, never reusing a fixed pool,…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: MaliosDark
Теги: llama, small-language-model, slm, from-scratch, tiny, nexus-erebus, arithmetic, en
Лайков: 3 | Загрузок: 33
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
