Straight talk first: if you have a 512 GiB machine, you don’t need this — the unpruned UD-IQ1S (508 GB) runs fine there by streaming experts from SSD (llama.cpp MoE streaming, 5.3–6.3 tok/s decode measured on an M3 Ultra). This build exists for the machines below that: it cuts Qwen3.8 down to what an English + code** deployment actually uses, along two axes at once — 1. expert pruning: keep the 304 of 512 experts per MoE layer that a measured en+code workload actually routes to; 2. width pruning: inside every surviving expert, keep the 6 of 8 256-channel superblocks that carry the most activation energy (intermediate width 2048 → 1536). Result: 246 GB (229 GiB), quantization untouched — every surviving byte is byte-identical to the UD-IQ1S it came from (1.56 bpw experts, no requantization anywhere). And no fork, no PR branch, no custom runtime: it loads and generates on stock mainline llama.cpp, verified.** Expert selection is count-based (how often the router picked each expert), not gate-weighted REAP saliency: on Kimi-K3 the same shortcut recovered 90.9% of saliency mass vs 93.5% for full saliency selection. Disclosed here because it is a methodological difference from our K3…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: hellohazime
Теги: gguf, qwen3.8, expert-pruning, width-pruning, reap, llama.cpp, endpoints_compatible, imatrix
Лайков: 3 | Загрузок: 95
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
