AWQ-calibrated mixed-precision quantization of DeepSeek-V4-Flash-0731 (304B MoE, 43 layers, 256 routed experts/layer, Hyper-Connections, HISA attention) for Apple Silicon via oMLX. Routed experts carry the aggressive quantization; everything else stays at 8-bit. Backbone: 278.6B quantized params · 101.4 GB · BPW 2.91 oMLX build with quantized MTP drafter: 108.2 GB total (drafter 7.04 GB, down from 10.86 GB unquantized) The MTP/DSpark speculative drafter is quantized to the same 2/3-bit recipe, not left at native precision. BPW above is quoted for the backbone weights; the drafter is additional and is not part of the 278.6B count. Generation speed is essentially flat with context — TPOT stays at ~20–22 ms from 1k to 16k, i.e. ~47–51 tok/s throughout. The MTP/DSpark drafter is doing its job. Build the custom kernels for long context. If oMLX reports dsaindexerscores/dsatopkindices unavailable (glmmoedsa extension not built), the sparse-attention indexer falls back to a slow MLX path and long-context prefill is several times slower than it needs to be. Rebuild with OMLXWITHCUSTOMKERNEL=1`. Generation speed is unaffected — this is purely a TTFT issue, and it is the largest single…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: True2456
Теги: mlx, deepseek_v4, omlx, awq, moe, quantized, deepseek, 8-bit
Лайков: 3 | Загрузок: 352
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
