A GPTQ 4-bit (group size 128, symmetric) quantization of IFM/K2-Horizon-MoVA-36B-A4B, MBZUAI IFM’s sparse frontier model with MoVA (Mixture-of-Values) attention — 36B total / ~4B active parameters, 512K context, Apache-2.0. This is a community quantization, not affiliated with or endorsed by MBZUAI IFM. K2-Horizon uses a custom architecture (modeltype: k2horizon, trustremotecode) with two independent Mixture-of-Experts systems: — a normal FFN MoE — mlp.experts.{0..99} (100 experts, top-8) + a shared expert, and — a novel MoVA attention where the value projection is replaced by selfattn.vexperts.{0..63} (64 routed value experts, top-4) plus a value router. Off-the-shelf quantizers do not know this architecture. GPTQModel’s model DSL also assumes a single expert count, but here the two MoE systems have different counts (100 vs 64). This quant was produced with a custom GPTQModel model definition that writes the 64 value-experts out explicitly while expanding the 100 FFN experts dynamically. The two routers (mlp.gate, selfattn.vrouter), lmhead, embeddings and all norms are kept in BF16** (standard MoE practice). 372 linear modules per sparse layer are quantized. — Tool: GPTQModel…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Siladrim
Теги: k2_horizon, k2-horizon, mova, moe, gptq, int4, 4-bit, quantized
Лайков: 3 | Загрузок: 163
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
