A Matryoshka language-model suite: three sub-models of increasing size (500M ⊂ 1.5B ⊂ 3B) stacked into a single nested architecture and trained end-to-end in one run. Running the full model produces all three sub-models’ outputs in a single forward pass, while activating fewer parameters and writing less KV cache than three independent models. This reduces the total parameter count of the suite (3.20B vs. 5.20B for an equivalent set of independent models, −38%), enables free online distillation from the largest sub-model to the smaller ones at every step, and is well-suited to speculative decoding — the draft model is contained within the verifier and shares its early layers and KV cache. All sub-models share the SmolLM2 tokenizer (V = 49,152) and use RoPE (θ = 1e5). Sub-model m+1 consumes sub-model m’s output through a parameter-free norm-rescaled junction: o^m is rescaled to match the norm of a fresh embedding covering the new channels, then concatenated. Depth triplet (n₁,n₂,n₃) = (24,10,5) (39 layers total) was chosen to match the KV-cache and per-token FLOPs of a Vanilla 3B baseline. The 500M sub-model uses the same width/depth as the Vanilla 500M for a strictly comparable…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: nthngdy
Теги: matriochka, matryoshka, nested-models, speculative-decoding, model-suite, distillation, custom_code, en
Лайков: 4 | Загрузок: 389
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
