— Used mmangkad/Qwen3.6-27B-NVFP4 as base model (Thank you for doing ModelOpt quant!) — Another mixed precision quant: ssmout is bf16, other ssm norms and weights are in F32. — Wv and attention output is in bf16. — FFN tensors are in nvfp4, with modelopt’s input scale / weight scale kept. Currently llama.cpp doesn’t have the best implementation for correctly derived nvfp4, but it’s only a small loss in perplexity. — Resulting BPW is 6.98, for 5090 you can do 130k KV cache at 16 bit. — Recommend FP16 KV cache. Do NOT follow previous recommendation for BF16 KV cache!!!* — converthftogguf with mmangkad’s nvfp4 checkpoint — generate «keep» list to preserve tensors — llama-quantize to q80 for the rest of the tensors NVFP4 related tensors are untouched: weight scales for proper fp4 scaling are kept unchanged.
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: stevelikesrhino
Теги: gguf, GGUF, llama.cpp, NVFP4, Quantized, endpoints_compatible, imatrix, conversational
Лайков: 4 | Загрузок: 65
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.