Community GGUF quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. These files contain the standard 52-layer inference model. The trailing MTP prediction head was removed during BF16-to-GGUF conversion with —no-mtp; it was not deleted from an already-quantized file. Every published quant was produced from that metadata-consistent no-MTP BF16 GGUF and load-and-generation tested before upload. Sizes are decimal GB and binary GiB. GPU examples are practical full-offload targets, not guarantees: runtime memory also depends on context length, KV-cache type, backend, compute buffers, and driver overhead. The TPS figures were measured on the DGX Spark GB10 described below and should not be treated as expected speed on the listed consumer GPUs. For the most headroom, choose a file at least 1-2 GiB below available VRAM before allocating the KV cache. Very long contexts can require substantially more memory than the model file alone. — 8 GB: partially offload the 2.47 BPW file to the GPU and keep the remaining layers in system RAM. — 12 GB: 2.47 BPW with a modest context; 2.97 BPW is possible only with very little headroom and carefully chosen KV-cache settings. — 16 GB:…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: vcruz305
Теги: gguf, nemotron, moe, imatrix, mamba, endpoints_compatible, conversational
Лайков: 4 | Загрузок: 5,640
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
