A 4-bit MXFP4 quantization of poolside/Laguna-S-2.1, produced with qstream. The routed MoE experts (~96% of the weights) are quantized to MXFP4; everything quality-sensitive stays BF16. We lead with deterministic, reproducible faithfulness metrics. (Task evals served over vLLM with continuous batching are not bitwise-deterministic — the same prompt at temperature=0 can yield different reductions depending on batch composition — so task accuracies carry a little serving noise; we quote one full-test-set score, where that noise is small.) — Only the routed experts changed. ~96% of the weights are re-quantized to MXFP4; everything else is bit-identical BF16 to the source. So the model is the base model except for 4-bit rounding on the expert GEMMs. — The error is uniform and diffuse. All 36,096 expert tensors sit in a tight band (median relative error 0.1117, p95 0.1124, max 0.1356) with no pathological layers — pure 4-bit rounding, no outliers to hide. Laguna is natively supported in vLLM ≥ 0.23, and this checkpoint uses the stock compressed-tensors MXFP4 path — no patches needed: Keep —max-model-len bounded: only the 12 full-attention layers grow KV with context (~48 KiB/token in…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: olka-fi
Теги: laguna, laguna-s-2.1, moe, mxfp4, compressed-tensors, quantized, vllm, conversational
Лайков: 3 | Загрузок: 126
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
