This repository contains a 4-bit RTN-quantized version of the Tencent Hy3 Mixture-of-Experts model. Hy3 is a 295B-parameter MoE model with 21B active parameters, 192 experts (top-8 routed), and a 256K context window. Quantizing it to 4-bit brings the weight footprint down to ~150 GB, making it possible to serve the full 256K context on a single 8×24 GB node with KVarN KV cache compression. This model requires a patched vLLM build. Stock vLLM cannot load it due to several bugs in weight-name remapping, dynamic GPTQ MoE group-size handling, and TurboQuant KV cache shaping: + fixes to allow run AWQ/GPTQ quants of Mimo-v2.5 + chery picked commit: [Attention] Add KVarN: calibration-free variance-normalized KV-cache quantization backend from https://github.com/vllm-project/vllm/pull/46812 > Set TORCHCUDAARCHLIST to your GPU’s compute capability to skip unused architectures and speed up the build (e.g. 8.6 for RTX 3090, 8.9 for RTX 4090, 9.0` for H100). Quantized with GPTQModel (7.2.0) using the RTN (Round-To-Nearest) fallback strategy. This model is verified to run with Tensor Parallel on 8x NVIDIA RTX 3090 (24 GB each) GPUs with the full 256K (262,144) context window, using…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: avtc
Теги: vllm, hy_v3, gptq, 4-bit, quantization, hunyuan, hy3, moe
Лайков: 3 | Загрузок: 22
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
