GGUF quantizations of IFM/K2-Horizon-7B, made directly from the publisher’s BF16 GGUF rather than from the FP8 checkpoint. Q6K is the llama.cpp quant name corresponding to “Q6 K”; there is no Q6KM` format. Q6K and Q4KM were produced with a 252-tensor importance matrix collected over 200 × 512-token chunks (~100k tokens) from the WikiText-2 training split. The imatrix was collected with the Q80 model because the 16.76 GiB BF16 graph exceeded the local Metal working-set limit. Q80 is a close proxy to BF16 here (measured mean KLD 0.000467). Output and token embedding tensors intentionally use llama.cpp’s normal mixed-quant fallback because imatrix collection does not process output.weight` by default. Small held-out WikiText-2 guardrail, measured against BF16 logits over two 256-token chunks with llama.cpp commit 35999d101cf2233fc54f09c3c8d599da7303ce02: This is a deliberately small local regression test, not a general capability benchmark. Task-specific evaluation is still recommended. K2 Horizon support is not yet in a released upstream llama.cpp build at the time of publication. Use the publisher’s model/K2Horizon fork or a future upstream build that includes K2 Horizon support.…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: 0xzknw
Теги: gguf, llama.cpp, k2-horizon, imatrix, quantized, en, endpoints_compatible, conversational
Лайков: 3 | Загрузок: 120
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
