> 12% FFN compression with live teacher distillation. Drop-in replacement — no custom kernels, no runtime changes. Dystrio Sculpt structurally compresses transformer FFN layers, producing dense models that load with standard transformers. Use case: Local/throughput — speed sweet spot (1.25x prefill) > KV-cache footprint is unchanged — Sculpt only compresses FFN layers, not attention. — Method: Structural FFN pruning with importance-aware block selection + live teacher distillation (alpha=0.5) — Keep fraction: 0.88 (12% of FFN neurons removed) — Repair: 8-stage cosine-LR fine-tuning with best-checkpoint restore — Training data: generalv2 mixture (WikiText, OpenHermes 2.5, MMLU, HellaSwag, GSM8K, OpenOrca) — Hardware: 1x NVIDIA H200 141GB — Output:** Standard dense transformer — loads with any HuggingFace-compatible framework — HuggingFace Transformers — vL
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: dystrio
Теги: qwen3_5_text, dystrio, sculpt, pruned, compressed, efficient, dense, drop-in-replacement
Лайков: 3 | Загрузок: 27
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
