GLM-4.7-Flash with half of its routed experts removed by RAZOR, a training-free expert pruning method. Every MoE layer keeps 32 of its original 64 routed experts. No gradient updates or recovery training were applied: the retained weights are the base model’s own weights. Pruning touches only the routed expert pool. Attention, the shared expert, the embedding, the LM head and the router’s remaining rows are untouched, so the compute per token is essentially unchanged while expert storage halves. A 25% variant is available as GLM-4.7-Flash-RAZOR-24B-A3B-E48of64. Requires a Transformers build that contains the native glm4moelite implementation. The checkpoint stores bfloat16 weights in five safetensors shards. RAZOR asks whether the surviving computation can replace an expert’s function, rather than how often or how strongly the expert fires. For a token routed to the set $S$ with normalized weights $wj$, let $c=sum{jin S} wj fj$ be the routed mixture and $rj=fj-c$ the consensus residual of expert $j$. Deleting a selected expert $i$ makes the router promote its highest-ranked unselected expert $r$, whose pseudo-weight $w_r$ is its score divided by the original selected score sum.…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Nickyang
Теги: glm4_moe_lite, moe, mixture-of-experts, pruning, expert-pruning, razor, glm, conversational
Лайков: 3 | Загрузок: 201
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.