min-spark 1.5 is a 9.36M-parameter language model with native effort levels. It reaches an Intelligence Index of 9.79, higher than any sub-10M model currently on the Open SLM leaderboard, after training on just 2.0B tokens. Much better results for far less training. min-spark 1.5 beats min-spark 1.1 while training on 8× fewer tokens with 5.7× less compute. It also beats Hummingbird-V2, the strongest other sub-10M model on the leaderboard, which trained on 10B tokens, five times as many. min-spark runs a few shared transformer blocks several times in a row. Each pass has its own small adapter, so it can do something different from the pass before. In earlier versions those adapters were held back. Version 1.5 fixes that: — Adapters on every layer. Every pass now adapts all four projections of each shared block instead of only the attention input, at rank 24 instead of 16. — Stable adapters. Adapters read normalized inputs and train at a steadier learning rate, so none of them can grow out of control and shut off part of a block. — No wasted passes. Earlier versions trained a fourth pass that the model learned to ignore. Version 1.5 trains three, and all three do real work. — A…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: MinimaLabs
Теги: minspark, custom_code, tiny, looped, sub-10m
Лайков: 3 | Загрузок: 1,136
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.