8 quantizations. One CPU setup. Quality, file size and throughput in the same record. Reading the table: Q80/Q6K are essentially lossless (+0.01 PPL). Q4KM is the knee of the curve; 2.5 GB (31% of BF16) for +0.30 PPL, and it’s the fastest generator of the non-Q3 set. Q3KM is a real quality cliff (+2.23 PPL), which is exactly where imatrix earns its keep. imatrix («importance matrix») quantization uses a calibration set (wikitext-2 wiki.train.raw, 11 MB; standard practice) to learn which weight directions matter most, so the quantizer spends its bits where they count. Measured here for base Qwen3-4B (the delta vs the same non-imatrix quant): The pattern is the well-documented one, now reproduced at this scale: imatrix pays off disproportionately at low bit-counts, where every bit matters. At Q4KM the gain is real but small; at Q3KM; where the plain quant is already 2.23 PPL worse than F16 – imatrix reduces the perplexity gap (back to +1.39). The cost is a one-time calibration pass (~3 h on this CPU; trivial on GPU) and a ~2.5 MB .dat file. Rule of thumb: run imatrix when you drop to Q3 or below. The two imatrix files let you A/B this yourself. Eight quantizations, CPU-only, 16…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: b4ph
Теги: llama.cpp, gguf, quantization, qwen3, benchmark, low-ram-inference, cpu-inference, perplexity
Лайков: 3 | Загрузок: 957
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.