First public extraction of Google Gemma 4’s Multi-Token Prediction (MTP) drafter weights from LiteRT format into standard PyTorch safetensors. Google trained all Gemma 4 variants with MTP heads for built-in speculative decoding, then deliberately stripped them from the public HuggingFace release, retaining them exclusively in LiteRT on-device format. This repository provides the extracted drafter as a standard PyTorch module with a working speculative decoding implementation. The 35% top-1 acceptance from dequantized INT4/INT8 mobile weights is the expected ceiling — quantization noise is irreversible. The 80% top-5 overlap makes these weights immediately useful for: — Tree-based speculative decoding (top-k draft candidates verified in parallel) — vLLM/SGLang native spec decode (zero-overhead verification eliminates the Python loop bottleneck) — Architecture reference for training EAGLE3 draft heads from full BF16 activations The MTP drafter is a lightweight 4-layer transformer (78M parameters) designed as a one-step recurrent cell: Critical architectural details: — Q-only It attends over the base model’s shared KV banks from layers…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: SeatownSin
Теги: gemma4, mtp, multi-token-prediction, speculative-decoding, litert, extraction, draft-model, en
Лайков: 4 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
