NVFP4 (4-bit) quantization of inclusionAI/Ling-3.0-flash (BailingMoeV3, 42-layer hybrid MoE + MTP) produced with NVIDIA ModelOpt using the DeepSeek-V4 experts-only recipe, served natively by the inclusionAI vllm-ling-v3 fork (FlashInfer-CUTLASS NVFP4 MoE). — Size: 81.3 GB (24 shards) — 3.14x compression over the 255 GB BF16 source — Quantization: weight-only NVFP4 (E2M1 packed 2-per-byte, E4M3 block scales, FP32 per-tensor scale), activationscheme: dynamic — Scope: the 61,440 routed-expert projections (layers 2–41) are NVFP4; attention (KDA + MLA), shared experts, routers, norms, embeddings, lmhead and the full MTP layer remain BF16 — Hardware: designed for NVIDIA GB10 / SM120/SM121-class GPUs Two project-local fork patches are > Note on chattemplate.jinja: this checkpoint ships with enablethinking defaulting to off (harness-compatible; matches the published benchmark rows). Pass chattemplatekwargs: {«enablethinking»: true} per request for reasoning traces, or restore the upstream template from inclusionAI/Ling-3.0-flash` for the…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: r0b0tlab
Теги: vllm, bailing_hybrid, nvfp4, modelopt, moe, quantized, conversational, custom_code
Лайков: 3 | Загрузок: 66
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
