A 2-bit quantization of deepseek-ai/DeepSeek-V4-Flash (299 B params, MoE) that serves on a single NVIDIA DGX Spark (GB10, 128 GB, sm121) — where the ~159 GB FP8/MXFP4 source needs ~2 — at ~22 tok/s single-stream decode on realistic chat (up to ~40 on long-form), coherent. > ⚠️ This checkpoint is NOT loadable by stock vLLM/transformers. It uses a custom vq2 > quantization method served by a small open-source plugin. You must install the serving stack: > https://github.com/iggerask/deepseek-v4-flash-2bit-gb10 (Apache-2.0). The plugin auto-registers > in vLLM (no fork) and adds the sm_121 kernel replacements DS4 needs on consumer Blackwell. The model weights are a derivative of DeepSeek-V4-Flash and are governed by that model’s license (linked above). The serving code (plugin + kernels) is Apache-2.0. It is an instruct model — use the chat template (llm.chat / a chat endpoint), not raw completion. Plus the model’s built-in MTP draft head (K=2 spec-decode, the measured optimum). A second lever, FR-Spec (frequency-shortlisted draft lmhead, frspecnvfp4ds4.pt), is shipped but currently inactive under the plugin load path (falls back to the full draft lmhead; measured marginal…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: iggerask
Теги: vllm, deepseek_v4, deepseek, moe, 2-bit, vq, nvfp4, quantized
Лайков: 3 | Загрузок: 35
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
