Q4KXL quantization of Qwen/Qwen3.8-27B for MLX on Apple silicon, with AWQ imatrix pre-scaling and the MTP head kept inline. AWQ here means activation-aware pre-scaling applied before MLX affine quantization — the same sense as mlx-community/-4bit-AWQ. The stored format is standard MLX affine, not AutoAWQ/GPTQ packing, so this will not load in vLLM or autoawq. MTP` means the multi-token-prediction head ships inside the main shards rather than as a separate drafter. This model is 6.02 bpw, not 4 bpw. The Q4KXL label follows the Unsloth naming convention, where —q-bits sets the base class of a mixed-precision recipe rather than the average. Higher-precision classes then sit above that base and dominate the file. The real distribution: Sidecars are the per-group scales and biases required by affine quantization at group size 64; they carry no parameters but do occupy space. Every quantized tensor, by module and resolved bit width (370 explicit per-layer overrides; anything not overridden inherits the 4-bit top-level default): Attention oproj and GDN outproj are pinned high because AWQ cannot correct them (no preceding norm to fold into), so they carry uncompensated quantization error…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: igorvibes
Теги: mlx, qwen3_5, awq, imatrix, mtp, conversational, 4-bit
Лайков: 3 | Загрузок: 593
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
