INT4 weight-only (W4A16) quantization of Qwen/Qwen3.6-35B-A3B — a hybrid Mixture-of-Experts model (256 experts, top-8, ~3B active) with GatedDeltaNet linear-attention, a vision tower and an MTP head. — AWQ activation-aware scale search + symmetric INT4 weights, groupsize 32, MSE observer (llm-compressor). — Quantized: the MoE expert FFNs only (the bulk of the weights). — Kept bf16 (quality-sensitive): selfattn, the router (mlp.gate), sharedexpert, GatedDeltaNet (linearattn), lmhead, embeddings, vision tower, MTP head. — Format: compressed-tensors (pack-quantized). Full recipe: recipe.yaml`. On Ampere the same checkpoint can be served as W4A8 (int4 weights + int8 dynamic activations) with the vllm-ampere-optimized fork, which adds the INT8-activation MoE Marlin path:
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Avesed
Теги: qwen3_5_moe, qwen3.6, moe, int4, w4a16, awq, compressed-tensors, vllm
Лайков: 3 | Загрузок: 703
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
