Quality-first mixed-precision ModelOpt quantization of poolside/Laguna-S-2.1, exported directly from Poolside’s BF16 checkpoint. This is a serving-oriented checkpoint, not a smallest-possible or single-GPU quant. The guiding rule was to leave the model’s signal path intact and quantize only the routed experts. All attention projections, shared experts, MoE routers, embeddings, dense layer 0, and the LM head remain BF16. In particular, no shared expert in layer 47 was quantized. Terminal-Bench 2.1 with Poolside’s PoolCLI harness is running now. Results will be added when the matched BF16, Poolside NVFP4, and this checkpoint campaigns finish. The KLD results below are complete five-run gates; they are not a substitute for downstream benchmark results. — 39 routed-expert layers use W4A16 NVFP4. — 3 routed-expert layers use FP8. — 5 routed-expert layers remain BF16. — Effective routed-expert weight precision: 5.9468 bits. — Audited export: 92,469,054,276 bytes, 10 shards, 101,377 tensors. — ModelOpt producer: 0.46.0.dev184+g8813b7001. Placement was selected with KL-guided AutoQuantize over 512 samples from nemotron-post-training-v3, 512 tokens per sample. The chosen placement was then…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: jackzampolin
Теги: laguna, vllm, modelopt, nvfp4, w4a16, fp8, quantized, conversational
Лайков: 3 | Загрузок: 22
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
