A GGUF quant of 0xSero/DeepSeek-V4-Flash-0731-REAP, the K160 expert-pruned checkpoint of deepseek-ai/DeepSeek-V4-Flash-0731. The pruning is 0xSero’s work: 160 of 256 routed experts kept per MoE scope, top-6 routing unchanged. I only did the quantization and the testing below. I built this to fit a 4x RTX 3090 rig (96 GB) with a 256K context window. The file is 89.9 GB, which leaves enough room on four 24 GB cards for a quarter million tokens of KV cache. Routed experts carry the Q3K and Q4K bulk. Attention, shared experts, the indexer and the output head stay at Q8_0. No imatrix was used. 180.4B parameters after pruning, 89.92 GB of tensor data, so about 3.99 bits per weight. I ran this against DeepSeek-V4-Flash-0731-UD-IQ2M`, which keeps all 284B parameters at 2.56 bpw and weighs almost exactly the same, 90.9 GB. Both served from llama.cpp at temperature 0, 150 scenarios total. One scenario out of 150 is noise, so I read this as a tie. The two builds do not fail on the same items though: K160 won CLI-16/17/28/30/40 and HA-17/19, the 2-bit build won CLI-01/03/11/35 and HA-08/13. Same level, slightly different error profile. Which is the interesting part. Same 90 GB budget, and it…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: alesha-pro
Теги: gguf, deepseek-v4, mixture-of-experts, reap, expert-pruning, llama.cpp, 4×3090, en
Лайков: 3 | Загрузок: 102
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
