LithosAI · draft model for lossless speculative decoding with Kimi-K3.1-Pro A DSpark draft head. In each decode step it reads hidden states that the target has already computed, proposes a block of tokens in one parallel pass, and the target verifies the block in one forward pass. Every token that is kept is drawn from the target’s own distribution, so outputs are exactly those of Kimi-K3.1-Pro; only the speed changes. At block size 8 the head reaches a mean acceptance length of 3.448 over 10 public benchmarks (math, science, code, software engineering, chat, writing, multilingual, RAG, QA, roleplay, summarization and 10K-token low-entropy inputs) and 3.281 on 95K-token AA-LCR prompts. The inference engine reads dflashconfig.targetlayerids from config.json` and exports those five target layers to the draft. Setup. Kimi-K3.1-Pro on one 8x B300 node, tensor parallel 8, FP8 KV cache. Block size 8 (verify 8). Temperature 1.0, top-p 0.95. Up to 16 requests in flight (8 for AA-LCR). Every benchmark is run on its full test set; all 5,399 requests succeeded. Prompts. Each sample is one user turn rendered with the target’s native chat template at thinkingeffort` max, with no system prompt…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: LithosAI
Теги: qwen3, speculative-decoding, dspark, draft-model, sglang, vllm, kimi-k3.1
Лайков: 3 | Загрузок: 198
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
