Two GGUF builds of a mechanically converted sparse-MoE version of Qwen3.8-27B. The dense SwiGLU feed-forward networks were re-partitioned into a MoE layout by checkpoint surgery. These are exact Top-12 conversions — all 12 routed experts are active, so the model is functionally the source dense model, reorganized and quantized. It is not a new or improved model, and it is not a quality-preserving sparse model. This weights release accompanies the source release and the experiment write-up: the conversion works, the sparsity does not. Read the repository before using this for anything serious. Qwen3.8-27B has numhiddenlayers = 64 (48 linear-attention + 16 full-attention), hiddensize = 5120, FFN intermediatesize = 17408, vocab 248320, native context 262144. Each dense FFN intermediate dimension F = 17408 was split into a disjoint partition: F/4 = 4352 shared channels plus 12 × F/16 = 1088 routed channels (4352 + 12 × 1088 = 17408). Nothing was added; channels were reassigned. This is the root cause of the whole experiment’s failure: Top-K routing deletes network width instead of selecting redundant, independently trained expert capacity. These GGUFs need a llama.cpp build that (a)…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: bwn2000
Теги: llama.cpp, gguf, moe, mixture-of-experts, qwen, quantization, en, endpoints_compatible
Лайков: 3 | Загрузок: 292
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.