Quantization test results for K2-Horizon-7B (dense) and LFM2.5-8B-A1B (MoE), comparing standard llama.cpp quantization against a fork that adds five new non-linear super-block types: JQ2K to JQ6K. On both models, four of the five new types perform materially better than their counterpart on every measured metric (mean and 99% KLD, Δp RMS, top-1 agreement). JQ2K is the exception. The size of the gain depends on the model: on the dense K2-Horizon-7B, JQ3K and JQ6K cut the mean KLD by more than half, while on the MoE LFM2.5-8B-A1B` the improvements range from 13.8% to 23.2%. The five new types are non-linear (NL) super-block (K) types. Each new type spends exactly the same index payload as its K-quant counterpart: JQ3K, JQ4K and JQ5K` files are therefore exactly as large as their K-quant counterparts. These three are direct replacements of their K counterparts: same super-block, same bytes per block, same file. With the quant recipe, imatrix and quantization parameters all fixed and equal, the quality difference comes solely from the quant format and the new types are superior on every metric at all three levels on both models: On K2-Horizon-7B, JQ3K is the standout. At the same 4.81…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: eaddario
Теги: gguf, experimental, quantization, en, endpoints_compatible, conversational
Лайков: 3 | Загрузок: 6,660
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
