A surgically expert-pruned openai/gpt-oss-120b: 48 of 128 experts kept per layer (top-48 by REAP saliency), reducing the checkpoint from 65.2 GB to 26.4 GB while matching the full model on the target task. Native MXFP4 expert weights are preserved unmodified; serves in stock vLLM (CUDA and Intel XPU) and converts to GGUF. — 36 layers x 48 experts, top-4 routing, ~45B total params, ~5.1B active — Fits a single 32 GB GPU with 128k context (fp8 KV cache) — Router softmax renormalizes natively over surviving experts; no retraining Experts ranked per layer by REAP saliency (Cerebras, arXiv:2510.13999): the mean over routed tokens of (renormalized router gate x L2 norm of the expert output), computed layerwise on a single RTX 4080 (16 GB) with MXFP4 dequantized on the fly. Calibration: 60 production prompts from a resume/job scoring workload plus the full model’s own generation traces (harmony analysis channel + final JSON). Calibrating on prompt text alone destroys output-format compliance; including generation traces is what makes deep pruning survivable. Saliency reproduces across independent hosts/GPUs at rank correlation 0.997 (top-48 set overlap 99.2%). Kappa vs production…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: bryanvine
Теги: gpt_oss, moe, expert-pruning, reap, mxfp4, conversational, 8-bit
Лайков: 4 | Загрузок: 522
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
