Derived from wtdcode/GLM-5.3-Flash-AWQ-W4A16 @ abd7b07 (INT4 AWQ experts, compressed-tensors pack-quantized), itself a quantization of zai-org/GLM-5.3-Flash (MIT). This repo changes only the tensors listed under «What changed»; requantinfo.json` lists every quantized tensor. Status: JKeys LLC’s production checkpoint on 4x CMP 170HX since 2026-09-15. Requantized on the box from -aggr-w8-mtpq8c in four minutes with the mixA recipe. INT4, symmetric, group size 32 (group2) on the shared experts of every MoE layer (129 tensors), the dense MLPs of layers 0-2 (9) and lmhead (1), converted from the INT8 per-channel tensors. Everything else, including the INT8 per-channel MTP head, is -mtpq8c. Public Spec-Bench suite (vllm bench serve —dataset-name specbench`, 80 prompts, 256 output tokens, temperature 0), full stack: single-stream 91.2 tok/s (TPOT p50 9.6 ms, TTFT p50 188 ms); concurrency 4: 142.7; 8: 164.7; 16: 163.9 with micro-batch cap 2 and 198.0 with cap 4. Prefill ~2.4k tok/s serial; ~8 concurrent 23k-token requests fit the KV pool. Acceptance rate of the drafter and the degeneracy probe unchanged vs -mtpq8c; the mixA recipe’s perplexity cost was measured on the -aggr-w8 base at…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: JJ48-24
Теги: glm5_next, compressed-tensors, sm80, cmp-170hx, vllm, conversational
Лайков: 3 | Загрузок: 348
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
