NVFP4 Quantized Devstral-2-123B for TensorRT-LLM on NVIDIA Blackwell GPUs NVFP4 ONLY works on NVIDIA Blackwell GPUs: — NVIDIA B200, B300, GB200 — NVIDIA RTX 5000 series (Blackwell architecture) — Any GPU with SM100+ compute capability Will NOT work on: A100, H100, RTX 4000 series, or older GPUs. The quantization was performed using NVIDIA ModelOpt with specific patches required for the ministral3 architecture. The checkpoint format from ModelOpt exporttensorrtllmcheckpoint is currently incompatible with TensorRT-LLM expected NVFP4 tensor layout. TensorRT-LLM expects specific scaling factor tensors (weightsglobalscalingfactor, weightsblockscalingfactorinterleaved, alpha) that ModelOpt export does not provide. This is a known format mismatch between ModelOpt 0.27.1 and TensorRT-LLM 1.0.0rc6/1.2.0rc5. The base model uses ministral3 architecture which requires latest transformers from git: Stable versions (4.53-4.56) will fail with KeyError: ministral3. 1. Wait for ModelOpt/TRT-LLM compatibility fix — Format mismatch is a known issue 2. Use vLLM — May support NVFP4 on Blackwell with proper configuration 3. Direct PyTorch inference — Load quantized model in PyTorch with modelopt…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: LorenzoNava
Теги: mistral3, conversational
Лайков: 3 | Загрузок: 25
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
