NVFP4 (NVIDIA FP4, weight-only NVFP4A16) quantization of empero-ai/Qwythos-9B-Claude-Mythos-5-1M — a Claude-Mythos/Fable-trace reasoning fine-tune of Qwen3.5-9B (qwen35: a dense hybrid GatedDeltaNet (linear-attention) + full-attention** model with a vision tower and a 1M-token YaRN context). Variant: NVFP4A16 — 4-bit NVFP4 (FP4, E2M1 with FP8 per-block micro-scales + FP32 global scale, group size 16) weights; activations BF16. Built for NVIDIA Blackwell (e.g. Thor / RTX 50xx) FP4 tensor cores. Disk size: ~11.2 GB (vs ~18.8 GB BF16). The reduction is modest because the GatedDeltaNet linearattn layers (24 of 32), the 248K-vocab embeddings + lmhead, and the vision tower are all kept in BF16; only the attention and MLP linears are FP4. Quantized by: sahilchachra Tooling: llm-compressor 0.12 modelfreeptq (data-free) → compressed-tensors Quantized to NVFP4 (128 modules): selfattn.{q,k,v,o}proj on the 8 full-attention layers + mlp.{gate,up,down}proj on all 32 layers. Kept BF16: the GatedDeltaNet linearattn layers, the vision tower, the MTP head, token embeddings, lmhead`, norms. Data-free weight-only PTQ via llm-compressor’s modelfreeptq — streams the safetensors from disk and applies…
Модальности:
Генерация текста
Области применения:
Логика и рассуждение Диалог / чат Вызов функций (Tool use)
Задача: Генерация текста
Автор: sahilchachra
Теги: qwen3_5, nvfp4, fp4, w4a16, reasoning, uncensored, long-context, function-calling
Лайков: 4 | Загрузок: 123
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
