A sub-150M language model pretrained from scratch by AltSlate Labs, for the Tiny-ML Leaderboard. The flagship of the Jugnu family: the kept R2 recipe (Qwen3 arch + value residuals + Muon) scaled to 25.2B tokens under a WSD schedule with modest decay-phase educational upweighting. transformers>=4.51 is required (the model builds on the Qwen3 architecture). It’s a standard AutoModelForCausalLM otherwise — no extra packages. This model uses value residuals (a custom attention pathway: vi = vproji(x) + λi·v0). Stock frompretrained would silently drop that pathway and degrade the model (~6 pts ARC-Easy, ~0.18 byte-ppl). It ships custom modeling code with automap, so load it VR-aware (trustremotecode=True): Sanity check that the value-residual pathway loaded (22 vrlambda` params, mean ≈ 0.48): Beats the JugnuLM-110M (R0) baseline on all three leaderboard metrics (BLiMP +1.3, ARC-Easy +2.65, byte-ppl 1.8735 vs 1.95), and posts the family’s best BLiMP and byte-ppl. INVALID LANGUAGE PAIR SPECIFIED. EXAMPLE: LANGPAIR=EN|IT USING 2 LETTER ISO OR RFC3066 LIKE ZH-CN. ALMOST ALL LANGUAGES SUPPORTED BUT SOME MAY HAVE NO CONTENT
Модальности:
Генерация текста
Задача: Генерация текста
Автор: altslate
Теги: jugnu_vr, jugnu, tiny-lm, value-residual, muon, pretrained-from-scratch, custom_code, en
Лайков: 3 | Загрузок: 566
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
