BF16 weights for CoderX — a code-targeted expert prune of Ornith-1.5-35B-A3B: 256 experts per layer reduced to 184, ~35.9B → ~26.7B, still A3B active. Router width, attention, norms and the MTP speculative head are untouched. Same expert budget and the same competence map as its sibling Ornith-1.5-27B-A3B-Coder — the entire difference is which 184 experts survive. Our per-category competence map drives selection, with a REAP-style stability floor (—protect 6) layered on top: the 6 experts per layer that our ranking would drop but REAP ranks highest are rescued, paid for by evicting the 6 lowest-ranked experts among our keeps. Verified against the Coder map: exactly 6 experts per layer differ, on all 40 layers (0/40 layers identical). The 72 evicted experts per layer are discarded, not merged. The arm is built with —merging none, so every retained expert is bit-identical to its counterpart in the base, and the build asserts it rather than assuming it (verifyarmidentity.py → ARMIDENTITYOK). No fine-tuning, no distillation, no expert folding. Everything that produced this checkpoint lives in omnimergekit, under recipes/ornith1535ba3bprune/`: The three rungs are a genuine nested…
Модальности:
Генерация текста
Области применения:
Генерация кода Диалог / чат
Задача: Генерация текста
Автор: ManniX-ITA
Теги: qwen3_5_moe_text, code, expert-pruning, moe, mtp, omnimergekit, ornith, ream
Лайков: 3 | Загрузок: 332
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.