An EAGLE3 draft model that accelerates inference for zai-org/GLM-4.7-Flash (30B MoE, ~3B active) through speculative decoding. 1.66x mean speedup at B=1 across 4 benchmarks on a single H200. ——|—————|————-|———|————-|—————| Protocol: B=1: 5 warmup + 20 measured (sequential). B=32: 15 warmup + 60 measured (32 concurrent). Metrics from server-side Prometheus. Benchmarked with our SGLang fork (tails-mpt/sglang, commit 63291f7f51). Upstream SGLang may produce different speedups due to scheduling overhead differences. Pinned dependencies: sgl-kernel 0.3.18.post2, flashinfer 0.6.6, torch 2.9.1+cu126. Verify accept rate > 0% after startup to confirm the draft model loaded correctly.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: thoughtworks
Теги: llama, speculative-decoding, eagle3, glm, draft-model, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 469
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
