GGUF builds of Qwen3.6-11B-A3B-Niwaki-4bit-mlx — Qwen3.6-35B-A3B pruned to 19B total / 3.3B active parameters — for llama.cpp and everything built on it. Niwaki (庭木): every routed expert individually width-pruned to the neurons its own routed tokens actually use, reconstructed to compensate, then briefly distilled from the full model, and stored at low precision. A paper with the full method is coming soon. Reference Qwen3.6-35B-A3B at Q80 measures 6.95** under the identical protocol (llama-perplexity, WikiText-2 test, 512-token windows). These llama.cpp numbers are not directly comparable to the MLX repo’s 2048-window benchmarks; the relative standings match across both. Generation battery (measured on the canonical MLX weights; reference scores 0.89 / 0.78): bigram-diversity avg/min = 0.82 / 0.62 across an 8-prompt code/reasoning/chat/creative battery — see the limitation note below. The recommended UD-Q3K build is quantized structure-aware (importance matrices calibrated on the same code-weighted corpus as the model itself), mirroring the artifact’s native allocation: the always-active backbone (attention, shared experts, embeddings) is kept at high precision (Q6K) while the…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: neopolita
Теги: gguf, llama.cpp, moe, pruning, mixture-of-experts, endpoints_compatible, imatrix, conversational
Лайков: 3 | Загрузок: 346
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
