A single 33M-parameter multi-token prediction head bolted onto a frozen MiniCPM5-1B, trained with self-distillation (FastMTP recipe) + a direct acceptance-rate loss (LK losses) — with honestly reported, per-benchmark acceptance rates instead of a single cherry-picked number. If you searched for multi-token prediction, MTP head, speculative decoding for small LLMs, or how to speed up LLM inference without a separate draft model — this is a complete, reproducible worked example, including the failure modes. — What: one extra Llama decoder layer (+ input projection, 33M params ≈ 2% of the backbone) that recursively drafts k=1..3 future tokens from the frozen base model’s own hidden states, embeddings, and LM head. No separate draft model. Backbone weights untouched. — Recipe: FastMTP (single shared MTP head, self-distillation on the model’s own outputs) + LK losses (hybrid KL/Total-Variation objective that directly optimizes acceptance rate instead of plain cross-entropy). — Cost: trained entirely on free Kaggle T4x2 GPUs, ~8,400 self-distilled sequences total. — Result (held-out, teacher-forced argmax acceptance vs. the backbone’s own predictions): — Not: a claim of production…
Модальности:
Генерация текста
Области применения:
Диалог / чат Финансы
Задача: Генерация текста
Автор: ewin-reg
Теги: llama, multi-token-prediction, mtp, speculative-decoding, self-speculative-decoding, inference-acceleration, draft-model, mtp-head
Лайков: 3 | Загрузок: 38
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
