A 1,002,720-parameter decoder-only base language model, pretrained from scratch on 5B tokens (40% FineWeb-Edu / 60% DCLM-baseline). No finetuning. Architecturally a stock Llama, so it loads with plain transformers — no trustremotecode, no custom modelling file: lm-eval 0.4.12, zero-shot, accnorm, float32, CUDA. ArithMark-3 via AxiomicLabs’ bencharithmark-3.py (float32, default accnorm). ArithMark-3 sits at the 25% chance floor, which is the honest number for a 1M-parameter base model that was never trained on arithmetic. A finetuned variant of this model reaches 40.40 on the public ArithMark-3 set and 35.5% on AxiomicLabs’ private memorization test, but drops to 21.5% on their private variety set after the problems are rephrased — so its arithmetic does not generalize across wording, and it was declined for leaderboard ranking on that basis. Those weights and the full write-up are at opencerebral/littlerock-1M-arithmax. This repository is the base model, with no arithmetic finetuning at all. The 4,096-token vocabulary is a deliberate constraint: at d=96 a stock 50k tokenizer’s embedding table alone would be several times the entire parameter budget. The tokenizer is a byte-level…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: opencerebral
Теги: llama, small-language-model, pretrained-from-scratch, en, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 307
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
