3-bit base mixed-precision quantization of Qwen/Qwen3.6-27B for Apple Silicon, using the Unsloth Dynamic quantization strategy via mlx-node. Benchmarked on Apple M3 Max 128GB via examples/lm.ts (best decode tok/s across turns 2–4, steady-state). Steady-state decode: 16.2 tok/s on Apple M3 Max 128GB (best of turns 2–4, examples/lm.ts capitals chat with reasoningEffort: ‘low’). Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. Based on Unsloth Dynamic 2.0 per-tensor KLD analysis. Sensitive layers get higher bits with AWQ correction, while the bulk of FFN weights are aggressively quantized. imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead). AWQ-correctable projections (q/k/v, inprojqkv/z) are quantized at 5-bit via inputlayernorm. Non-AWQ-correctable projections (oproj, out_proj) are kept at bf16 — their inputs come from attention/GDN computation, not from a norm layer. — Unsloth — Quantization strategy based on their per-layer KLD benchmarks and Dynamic 2.0 methodology — Qwen Team — For the Qwen3.6 model family — Apple MLX — For…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Brooooooklyn
Теги: mlx-node, qwen3_5, mlx, quantized, awq, 3-bit, qwen3.6, hybrid-attention
Лайков: 3 | Загрузок: 161
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
