sfxnz/DeepSeek-V4.1-Flash-EXL3 - Каталог нейросетей
Генерация текста

sfxnz/DeepSeek-V4.1-Flash-EXL3

Добавлено:
sfxnz/DeepSeek-V4.1-Flash-EXL3

EXL3 pack of deepseek-ai/DeepSeek-V4.1-Flash. Routed experts are 2.0 bpw MCG. In the default revision they are re-encoded with tail-biting Viterbi plus a scale refit, and the lm_head is MXFP8. Other tensors keep the source dtypes. Engram tables stay on NVMe at serve time. This pack exists so 2× NVIDIA DGX Spark (GB10) can serve the model at tensor-parallel 2. Native MXFP4/MXFP8 is about 511 GB and does not fit 2× Spark UMA. The two Engram shards are about 95 GiB each. Keep Hugging Face xet enabled. Do not set HFHUBDISABLEXET. The recipe measured the Viterbi revision against 2.0bpw-mcg-lmhead-mxfp8 in an interleaved 2+2 boot comparison (round 35 P4). Teacher-forced NLL on 40 fixed passages was 0.139 vs 0.243 nats/token (−42%, 39 of 40 passages lower). GSM8K-100 read 97/97 vs 95/94, MMLU 4×57 206/204 vs 196/198, and tool-call exact args 88/88 vs 84/88. Decode ms per step is unchanged; speed moves only through speculative acceptance (proselong +7.0%, L.A.I.L prose −2.4% in that comparison). The 2× DGX Spark cookbook is sfxnz/DeepSeek-V4.1-Flash-EXL3-vLLM-2x-DGX-Spark. Clone that repo, download this revision, build the image on both nodes, and run ./run.sh. Model weights are MIT from…

Модальности:
Генерация текста


Задача: Генерация текста
Автор: sfxnz
Теги: exllamav3, exl3, quantized, vllm
Лайков: 3  |  Загрузок: 171

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.