DeepSeek-V4-Flash-0731 compressed from 155.42 GiB to 91.93 GiB (1.69×) so it fits and serves on ONE NVIDIA DGX Spark (GB10, 128 GB unified memory) — where the source checkpoint requires two. Only the routed experts were re-quantized, using product-codebook (CB) quantization at the C16 rung in the NVFP4CBK16 format. The attention and dense/shared-FFN hot path ships unchanged from the DeepSeek source. > ⚠️ This will not load with stock transformers. config.json declares > quantmethod: «prismaquant» and automap is null. It needs vLLM plus the gridbook > out-of-tree plugin. See Serving. Model is optimized for serving on Spark sm121a . Serving recipe will be updated once optimization is completed and validated. — Expert weight cost: 2.28125 bits per parameter. — 86 CB stacks = 43 body layers × 2 (downproj, gateupproj). Every routed-expert stack in the body is covered; 0 layers left dense. — The MTP layer is BF16 passthrough** — its experts were not re-quantized. Headline: on agentic tool use this artifact is statistically indistinguishable from the unquantized original, on half the hardware. Response time depends on configuration — see the gate warning below before reading the…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: JasonW2025
Теги: vllm, deepseek, deepseek-v4, mixture-of-experts, quantized, nvfp4, codebook-quantization, prismaquant
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
