High-quality imatrix GGUF quantizations of tencent/Hy3, Tencent’s 295B-parameter Mixture-of-Experts model with ~21B active parameters per token. Produced with a Hy3-enabled llama.cpp branch (build 9909, commit 56142c5f8). Every quant in this repo was measured for quality (KL divergence, perplexity, and top-token agreement against BF16 reference logits) and throughput (llama-bench on 8× RTX PRO 6000 Blackwell) — full raw data is included under benchmarks/. The GGUFs are converted from the BF16 base model with the MTP/NextN block excluded (—no-mtp) — intentional for normal serving. Q8_0 was intentionally skipped for this release. Sizes below are weights only — leave headroom for KV cache and compute buffers. Use —split-mode layer for multi-GPU. (—split-mode tensor crashed in CUDA decode during testing on this Hy3 GGUF path — see Known Limitations.) The chat template is embedded in the GGUF, so llama-server / llama-cli chat modes work out of the box. Quality was measured with llama-perplexity —kl-divergence against BF16 no-MTP reference logits generated from tencent/Hy3. — Corpus: WikiText-2 raw test (llama.cpp scripts/get-wikitext-2.sh) — Slice: 128 chunks × context 512…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: LordNeel
Теги: llama.cpp, gguf, hy3, hunyuan, quantized, imatrix, mixture-of-experts, endpoints_compatible
Лайков: 4 | Загрузок: 167
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
