An NVFP4-quantized, text-only build of Qwen/Qwen3.8-27B: vision tower removed, MTP head grafted back in bf16 so vLLM speculative decoding works, exported in modelopt format for the native NVFP4 path on NVIDIA Blackwell / SM120. This build was an experiment on a 32 GB RTX 5090, and its conclusions still apply there. Two things have changed since. There are now vision-kept successors, quantized from the bf16 base with the vision tower and the MTP head intact: — dbrasdasilva/Qwen3.8-27B-NVFP4-MTP — mixed FP8/NVFP4, 24 GB — dbrasdasilva/Qwen3.8-27B-FP8-MTP — FP8 throughout, 30 GB Both also carry calibrated FP8 KV-cache scales (kscale/vscale), which this build does not. Without them —kv-cache-dtype fp8 runs at scale 1.0 and loses accuracy silently — vLLM says so in the log, and it is easy to miss. On 96 GB the central trade-off disappears. The «pick two» conclusion below is a property of a 32 GB KV budget, not of the model. Measured on an RTX PRO 6000 Blackwell 96 GB with vLLM 0.30.0, vision on, MTP on, —max-model-len 262144, —kv-cache-dtype fp8, —gpu-memory-utilization 0.95: Vision, MTP and the model’s native 262 k context coexist with room to spare, so the 32 GB question — does…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: dbrasdasilva
Теги: qwen3_5, image-text-to-text, qwen3.8, nvfp4, quantized, modelopt, mtp, speculative-decoding
Лайков: 3 | Загрузок: 692
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
