This is a reproduction of «Multi-Token Prediction via Self-Distillation» (arXiv 2602.06019) adapted for a single NVIDIA RTX 5090 (32GB). The original paper used 4x NVIDIA GH200 (384GB total) with Llama-3.1-8B. We scaled it down to Llama-3.2-1B on consumer hardware. Standard language models predict one token at a time (autoregressive decoding). MTP trains the model to predict multiple future tokens simultaneously using online self-distillation: 1. A frozen teacher (the original model) generates soft probability distributions 2. A trainable student (same architecture) learns to predict k future tokens at each position 3. At inference, ConfAdapt decoding emits multiple tokens when the model is confident, falling back to single-token when uncertain — ConfAdapt works: k=8 with ConfAdapt matches k=1 quality while being 1.8x faster (avg 2.82 tokens emitted per step) — Quality drop is expected: The ~2% accuracy drop from baseline is consistent with our smaller setup (1B model, 500M training tokens vs paper’s 8B model, 2B tokens) — The core claim holds: Multi-token decoding via ConfAdapt preserves g
Модальности:
Генерация текста
Задача: Генерация текста
Автор: celestialcreator
Теги: llama, multi-token-prediction, speculative-decoding, self-distillation, mtp, consumer-gpu, rtx-5090, paper-reproduction
Лайков: 3 | Загрузок: 39
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
