Ablation-guided ‘heterogeneous’ GGUF quantization of DeepSeek-V4-Flash-0731. 95.8 GB single file. Fits 128 GB memory, leaving ~32 GB for KV cache (~150K tokens) to enable multi-turn sessions. Quantization schema derived from the Deepseek V4 Pro model’s quantization recommendations (hence, heterogeneous), refined with ablation signal analysis. I was curious if the models have been trained on and thus have ‘internalized’ their own architecture such that they could advise an effective distribution. Three-tier expert quantization, ranked by confabulation-routing ablation signal from a differential heatmap over the model’s own expert activation patterns: Approach is verified by DSL architecture documents regarding MoE routing and attention architecture. Layers 32 and 40 showed the highest confabulation-routing signal (routing decisions most likely to produce hallucinated output when degraded), so they get the most bits. Layers 13, 26, 36, 42 are the next tier. Everything else is sparse enough that Q2_K preserves quality. Requires llama.cpp with DeepSeek-V4 support. Mainline llama.cpp gained V4 support in mid-2026; earlier builds may need the antirez fork. Minimum context size is 4096…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: salen-00
Теги: gguf, deepseek, deepseek-v4, moe, quantized, ablation-guided, endpoints_compatible, conversational
Лайков: 3 | Загрузок: 42
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
