4-bit GPTQ (W4A16) quantization of deepseek-ai/DeepSeek-V4-Flash-0731, produced with GPTQModel. This is the V1 release (first calibration run). A V2 release, requantized with a different calibration dataset, is planned as a separate repository (DeepSeek-V4-Flash-0731-GPTQ-Int4-V2) once available. — allenai/c4 — zh, en, ja splits (mangled/mis-decoded documents filtered out) — codeparrot/codeparrot-clean-valid — Python source zh/en/ja/code were weighted evenly. seqlen=2048 was chosen above this model’s sparse-indexer indextopk (512) so the top-k pruning path is actually exercised during calibration, while staying within the indexer’s dense-attention memory budget on the quantization rig. This model requires a vLLM build with DeepSeek-V4 GPTQ support on the Ampere (SM86, e.g. RTX 3090/4090-class, non-Hopper) backend, which is not yet present in either upstream vLLM or the unmodified vllm-dsv4-ampere community fork. The required patch is available at: https://github.com/okdshin/vllm-dsv4-ampere/tree/gptq-w4a16-support Apply the main-port/vllm overlay from that branch on top of a vanilla vLLM checkout, then serve as usual, e.g.: (—cpu-offload-gb may be needed depending on available…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: sokada4
Теги: deepseek_v4, gptq, gptqmodel, 4bit, w4a16, int4, quantized, vllm
Лайков: 4 | Загрузок: 970
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
