From-scratch ~25M-parameter Qwen2-style decoder LM trained in under 4h on a single RTX 5060 Ti (16GB). 0.8B-token weighted mix: fineweb-edu 70% / infiwebmath 10% / DCLM-baseline 20%, block-shuffled. ~3000 steps at 262,144 tok/step, cosine LR, 8-bit AdamW. Notes: — ArithMark-3.0 (AxiomicLabs/Arithmark-3.0) is the strongest relative result (+7.9 pts over random), consistent with the math share of the pretraining mix. — ARC-challenge raw accuracy sits below chance due to a length bias in unnormalized scores; acc_norm is the meaningful metric there. Expect shallow world knowledge and weak performance on knowledge-heavy benchmarks due to the model’s small parameter count and limited training budget.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: CodeSoft
Теги: qwen2, 25M, en, text-generation-inference, endpoints_compatible
Лайков: 4 | Загрузок: 1,366
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
