A W4A16-quantized DSpark speculator for zai-org/GLM-5.2-FP8, tuned for memory-bandwidth-bound hardware — specifically a two-node NVIDIA DGX Spark (GB10) cluster running GLM-5.2 at TP2. It is a quantization of RedHatAI/GLM-5.2-speculator.dspark (epoch-3). The weights were not retrained — only their precision changed. Measured: +8.7% decode throughput at no measurable cost in draft acceptance. On a DGX Spark the binding constraint is memory bandwidth, and a speculative drafter is essentially a weight read: it streams its own parameters once per speculative cycle. Direct profiling put the drafter at 15.88 ms of an ~89 ms cycle — around 18% of decode time spent reading 7 GiB of BF16 weights. Quantizing the drafter’s hot weights to 4 bits cuts that read ~4×. Because speculative decoding is lossless, a worse drafter cannot corrupt output — it can only propose tokens that get rejected. So the only thing quantization can cost is acceptance, and that is the number to watch. Measured on 2× DGX Spark (GB10, 121 GiB unified LPDDR each), TP2 over 200 Gb ConnectX/RoCE, GLM-5.2 753B MoE, p2048/o256/c1, 15 repeats per arm measured back-to-back in one session. — Throughput: +8.7%, 2.8σ, 95% CI…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: sapidlabs
Теги: speculators, speculative-decoding, dspark, compressed-tensors, w4a16, dgx-spark, custom_code
Лайков: 4 | Загрузок: 77
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
