REAP-pruned DeepSeek V4 Flash at K180 (180 routed experts per MoE layer, 30% pruning), with a hybrid mixed-precision quantization targeting NVIDIA DGX Spark (GB10 Blackwell, sm121). Supports full 1M token context** on a single 128 GB DGX Spark via managed memory + FP8-packed KV cache. This model uses a hybrid quant strategy optimized for DGX Spark’s bandwidth profile (273 GB/s LPDDR5X, ~97 GB/s managed-memory serving path): The bottleneck in MoE decode is gate+up expert projection (~2/3 of expert bytes). Standard GGUF quants use IQ2XXS here. NVFP4 reads more bytes (4.50 vs 2.06 bpw) but offers substantially better precision at the same bit budget, with dequant bandwidth at ~140 GB/s on GB10. In practice, NVFP4 decode is slightly slower than IQ2XXS due to the higher byte volume, but the precision gain is well worth the small throughput cost. Down experts stay Q2K (already saturating at ~160 GB/s). Attention and shared experts stay Q80 for quality. The HF source checkpoint stores experts in MXFP4 (e2m1 nibbles + e8m0 per-32 block scale). NVFP4 is e2m1 + e4m3 per-16 + per-expert fp32 scale2. The e2m1 nibbles are identical** — the conversion is a scale-only transform. No weight…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: sleepyeldrazi
Теги: ds4, gguf, deepseek, deepseek-v4, deepseek-v4-flash, mixture-of-experts, reap, expert-pruning
Лайков: 3 | Загрузок: 29
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
