Tested with vLLM 0.32.0rc1 (a development build). Every result below was measured with thinking off; BFCL-v3 ran at a 65,536-token context and the other checks at 4,096. On a 16 GB GPU, add —gpu-memory-utilization 0.92 —max-num-batched-tokens 2048, and lower the first value if the GPU also drives a display; this is expected to work, not tested on a 16 GB card. For images and audio, drop —language-model-only; audio also needs pip install «vllm«. The reply is in choices[0].message.content. Tool calls use the same endpoint with a tools list and come back in choices[0].message.toolcalls`. !Generation speed one request at a time for this FP8 checkpoint and the BF16 original, with weight sizes. Bold marks the best in each column. One NVIDIA RTX 5090, vLLM 0.32.0rc1, thinking off. Full results, memory and method. The checksums section lists every file. vllm serve turintech/Nemotron-3.5-Bolt-E4B-Artemis-FP8 downloads the 11 weight and configuration files, 12.73 GB in all. — Good for serving with vLLM in 12.69 GB of weights, with 37% more KV cache than the original and generation 32 to 43% faster, at the same BFCL and GSM8K scores within one question. — Pick another file when the…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: turintech
Теги: gemma4, image-text-to-text, turintech, artemis, nemotron, vllm, quantized, fp8
Лайков: 3 | Загрузок: 61
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
