Qwen3.5-REAP-20B-A3B is a Mixture of Experts (MoE) model created by applying Router-weighted Expert Activation Pruning (REAP) to Qwen/Qwen3.5-35B-A3B. Key Achievement: 35B → 20B total parameters | ~3B active per token | 50% expert pruning | Shared expert preserved — transformers >= 5.x (from main branch) — the qwen35moe model type is not in any released version yet — torch >= 2.4 with CUDA support — ~35 GB disk + ~40 GB VRAM (or use devicemap=»auto»` for CPU offloading) Tested with vLLM v0.17.0rc1 (vllm/vllm-openai:cu130-nightly). Requires —enforce-eager due to hybrid attention. The pruning was done using this Docker image which has all dependencies pre-installed: > Note: Both transformers and peft must be installed from main — the released versions don’t support qwen35moe and have a HybridCache import error respectively. Qwen3.5-35B-A3B uses a hybrid attention architecture distinct from previous Qwen models: — 30 linear attention layers using GDN (Gated Delta Network) with conv1d kernels — 10 full attention layers (standard multi-head attention with RoPE) at every 4th layer — Fused expert format: Experts are stored as Qwen35MoeExperts with stacked tensors (gateupproj:…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: atbender
Теги: qwen3_5_moe_text, reap, pruned, moe, qwen3.5, mixture-of-experts, linear-attention, conversational
Лайков: 4 | Загрузок: 19
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
