> [!TIP] > Support this work → · X · GitHub · REAP paper · Cerebras REAP 20% expert-pruned version of Qwen/Qwen3-Coder-Next using Cerebras REAP (Router-weighted Expert Activation Pruning). REAP removes 20% of MoE experts (102 of 512) while preserving the model’s routing behavior and output quality. The active parameter count per token is unchanged since the router still selects 10 experts per token from the remaining pool. This yields a ~14% reduction in total disk/memory footprint with minimal quality loss. REAP (ICLR 2026) prunes Mixture-of-Experts models by scoring expert importance using: 1. Router gate values — how often and how strongly the router selects each expert 2. Expert activation norms — magnitude of each expert’s output contribution 3. Frequency-weighted saliency — combining routing frequency with activation importance 4. Router logit renormalization — maintains output distribution after expert removal 5. Layerwise application — independent per-layer pruning decisions for stability 22,000 samples (no-refusal subset: 21,000), packed to 16,384 token sequences: Total tokens observed: ~90.5M across 6,391 packed sequences. 10-task lm-eval suite, 200 samples per…
Модальности:
Генерация текста
Области применения:
Диалог / чат Генерация кода
Задача: Генерация текста
Автор: 0xSero
Теги: qwen3_next, cerebras, expert-pruning, moe, pruning, qwen3, qwen3-coder, reap
Лайков: 3 | Загрузок: 147
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
