The first INT4 AutoRound quantization of the Heretic (uncensored) Qwen3.5-397B — the largest open-source abliterated model. A 4-bit symmetric quantization of trohrbaugh/Qwen3.5-397B-A17B-heretic, generated using Intel AutoRound v0.12.0. The original multimodal architecture (Qwen35MoeForConditionalGeneration`) is fully preserved — text, vision, video, and reasoning all work. Keeping critical layers at full precision is essential for MoE model quality: — Vision encoder weights (model.visual.blocks.) — BF16 — Shared expert projections (layers × gateproj, upproj, downproj) — FP16 — Shared expert gates (sharedexpertgate) — FP16 — MoE routing gates — FP16 — lmhead — original precision The shared expert is activated for every token (unlike the 511 routed experts where only 16 fire per token). Preserving its precision is critical for maintaining output quality. > The 13 GB difference from Intel’s canonical INT4 is because the Heretic fork does not include MTP (Multi-Token Prediction) weights. MTP is a speculative decoding feature — its absence does not affect output quality. The Heretic is significantly faster than the canonical quantization on identical hardware. The speed gain likely…
Модальности:
Генерация текста Компьютерное зрение
Области применения:
Генерация кода Логика и рассуждение Диалог / чат
Задача: Генерация текста
Автор: happypatrick
Теги: qwen3_5_moe, autoround, gptq, int4, quantization, uncensored, heretic, abliterated
Лайков: 4 | Загрузок: 407
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
