This GGUF requires the linked patched llama.cpp build; stock llama.cpp cannot load it. The model of Schiltmans/Qwen3.8-Flash-Next-Coder-cut240, packed as GGUF. It is ISTA-DASLab’s IQ1_M Coder GGUF of Qwen3.8-Flash-Next (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF at 5348543e) with 768 of its 12,288 routed experts removed. Each layer keeps 185 to 255 of 256 routed experts, 240 on average. The kept experts’ bytes are unchanged, and each router keeps only the rows of its layer’s kept experts. Tested with Metal on one 48 GB M4 Pro, all layers on the GPU and -fa on; other backends were not checked. Stock llama.cpp reads qwen4exp.expertcount as one number and stops with key qwen4exp.expertcount has wrong type arr but expected type u32. The fork alexschiltmans/llama.cpp-per-layer-experts, branch per-layer-experts, holds the change as commit 1f12597d768503aac85479a708e8cc02fd5f3d2a on llama.cpp release b11750: expertcount` can hold one count per layer, and qwen4exp builds each layer’s router and expert tensors with that layer’s count. What the commit changes and how to check it: the fork’s README. Prerequisites: an Apple Silicon Mac, git, CMake, Xcode’s command-line tools, uv for…
Модальности:
Генерация текста
Области применения:
Генерация кода Диалог / чат
Задача: Генерация текста
Автор: Schiltmans
Теги: gguf, qwen4_exp, moe, expert-pruning, apple-silicon, code, endpoints_compatible, imatrix
Лайков: 3 | Загрузок: 1,469
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
