NVFP4 post-training quantization of Olafangensan/GLM-4.7-Flash-heretic for long-context multi-GPU inference with vLLM. > [!IMPORTANT] > The Hugging Face UI «Model size» badge is auto-inferred from packed NVFP4 safetensors and may show an incorrect parameter count for this repo. > Use the architecture statement below as source of truth: GLM-4.7-Flash (30B-A3B MoE). This release uses NVFP4 (4-bit) quantization, not 8-bit quantization. — Base architecture: GLM-4.7-Flash (30B-A3B MoE) — Parameter count for this release: unchanged from the base model architecture — Note: the ~17.8GB model.safetensors file size is a quantized checkpoint size and does not mean the model is 18B parameters. Known issue on some stock vLLM 0.16.x + vllm-node setups: — assistant content may be null — output may be dumped into reasoning fields with broken formatting This model requires a custom vLLM image. Do not use stock vllm/vllm-openai as-is. These measurements were taken on a 4x RTX 3090 host with vLLM and this NVFP4 export. — Use —optimization-level 3 (O3), not default O2. — Use TP2 on an NVLink-connected pair (CUDAVISIBLEDEVICES=0,2 + —tensor-parallel-size 2). — Keep —max-model-len 131072 (128k…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: alphakek
Теги: glm4_moe_lite, quantized, modelopt, nvfp4, vllm, moe, conversational, en
Лайков: 3 | Загрузок: 74
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
