This repository contains a DFlash draft model for moonshotai/Kimi-K3 trained only on a generic data mix (no tool calls, agentic traces, etc). It is not a standalone language model. It is intended to be paired with the target model in a speculative decoding server. DFlash uses a lightweight block diffusion draft model to propose multiple tokens in parallel. The target model verifies those proposals, improving serving throughput while preserving the target model’s output distribution. This model should be used with an inference server that supports DFlash speculative decoding. An example SGLang deployment is: Block size 8 is the recommended default. Block size 16 gives longer accept lengths. On certain workloads we have measured peak per-request decode throughput approaching 900 output tok/s at concurrency 1 with an experimental block size 16 nvcutedsl` linear attention verify backend (877 tok/s on GSM8K and 862 tok/s on MATH500). — Runtime: SGLang on 8x NVIDIA B300 GPUs, tensor parallel size 8, bfloat16 — Backends: trtllmmla target attention, trtllmmha DFlash draft attention, ptxkda linear attention prefill, triton linear attention decode and verify, flashinfermxfp4 MoE runner -…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: modal-labs
Теги: qwen3, feature-extraction, dflash, speculative-decoding, speculative-decoding-draft, block-diffusion, draft-model, diffusion-language-model
Лайков: 4 | Загрузок: 6,444
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
