This repository provides an AWQ quantized build of GLM-4.7-Flash repackaged for vLLM using the compressed-tensors runtime layout. > Why this quant is different (MoE-aware calibration) > > — During calibration we activate all experts inside each MoE block (not just top-k chosen by the router). > — This captures worst-case activations across the entire mixture, producing more robust scales with lower drift when rare experts fire at inference time. > — The quant script explicitly does not ignore shared experts (fixes typical smoothing issues in MoE with AWQ). :contentReference[oaicite:0]{index=0} > > Runtime requirements: > • vLLM nightly build (MoE + GLM-Flash path) and Transformers 5.0. > • trustremotecode must be enabled. — Sharded quantized weights (.safetensors) + index (model.safetensors.index.json) — config.json with compressed-tensors metadata (quantizationconfig, weightformat, etc.) — Tokenizer artifacts (tokenizer.json, tokenizer.model, merges/vocab as applicable) > This package targets vLLM (compressed-tensors). Loading directly with vanilla 🤗 frompretrained` is not supported. — Group size controls how many consecutive weights share one set of quantization scales. — GS32…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: TheHouseOfTheDude
Теги: vllm, conversational, moe, awq, w8a16, group-size-32, compressed-tensors, quantized
Лайков: 3 | Загрузок: 35
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
