A sub-4-bit routed-expert quantization of Qwen/Qwen3.8-Flash-Next, built to fit the full 262,144-token context of a 125B-A6B multimodal MoE into a single 128 GB unified-memory GB10 (DGX Spark) with room left over — and to stay fast while doing it. 98.74 tok/s single-stream at 30 tokens of code context, 93.90 tok/s at 255,297 tokens, 67.48 GiB steady VRAM, full 256K context, MTP k=3 speculative decoding intact. > Read the Quality section before you use this. The 3-bit cut > costs a measured, statistically significant −2.66 pp pooled accuracy, and the damage is > concentrated in code (−4.88 pp HumanEval). This is not a free win, and this card will not > pretend it is. Re-quantized from the original BF16 weights, not from an existing NVFP4 build — quantizing a quantized tensor compounds the error. The 96 expert shards were streamed through a 12-shard ring with consumed shards deleted, so peak disk stayed bounded well under the full BF16 footprint. The obvious choice is float8e4m3 group scales. It fails here: at group 64 on these experts, 99.6 % of the scales land in the e4m3 subnormal range**, where the representation error of the scale itself swamps the 3-bit weight error. Measured…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: HamboneLabs-AI
Теги: vllm, qwen4_exp, moe, uint3, sub-4-bit, quantization, modelopt, mtp
Лайков: 3 | Загрузок: 197
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
