A 33%-expert-pruned version of Tencent Hunyuan Hy3 (295B / A21B), compressed with REAP (Router-weighted Expert Activation Pruning) — pruning only, no expert merging. This checkpoint reduces the number of routed experts in tencent/Hy3 from 192 → 128 per MoE layer (−33%) using the REAP saliency criterion, as implemented in the SamsungSAILMontreal/ream codebase. The result is a smaller, drop-in Hunyuan V3 model (~200B total parameters) that keeps the same 21B active-parameter path and the full 256K context window. Because the router is left untouched and no experts are averaged together, this is a pure REAP checkpoint — the pruned model retains the original router’s independent, input-dependent control over the surviving experts. The active-parameter path is unchanged because top-8 routing and the single shared expert are preserved; pruning removes only rarely / weakly used routed experts, so per-token FLOPs stay the same while the total memory footprint drops by roughly one third. Everything except the routed-expert count is preserved, including the router, attention, shared expert, dense layer, and RoPE configuration. At 33% expert pruning, MMLU drops by only a few points while…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: OpenMOSE
Теги: hy_v3, moe, mixture-of-experts, expert-pruning, reap, hunyuan, hy3, compression
Лайков: 4 | Загрузок: 115
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
