This repository contains a REAM-compressed version of Tencent Hy3, produced with Akicou/ream — a REAM/REAP-style Mixture-of-Experts compression framework. REAM (Router Expert Activation Merging) — originally proposed by SamsungSAILMontreal/ream (Jha et al., 2026) — is a compression technique for MoE models that combines expert merging with pseudo-pruning: 1. Calibration — Run a small set of hardcoded prompts through the model to collect router logits and expert output activations. 2. REAP Saliency — Compute per-expert importance as S[i] = mean(||hi(x)|| × pi(x)) over tokens routed to expert i. 3. Pseudo-Grouping — Select the top-K most salient experts as centroids, then assign nearby experts using gated similarity (50% hidden state + 50% router logit distribution). 4. Merge — Within each group, align neurons via Hungarian permutation and merge with saliency-weighted averaging. 5. Prune — Non-centroid experts are removed. The router is shrunk to match the new expert count, keeping only centroid rows. This implementation follows the original REAM paper closely, including: — Centroid-only router update (REAP-style) — Zero saliency edge-case handling (unused experts get a small…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Akicou
Теги: hy_v3, hy3, tencent, ream, reap, merge, merging, compression
Лайков: 4 | Загрузок: 74
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
