This is a PrismaQuant 4.75-bit mixed-native quantization of mistralai/Mistral-Medium-3.5-128B, exported for vLLM using the compressed-tensors format. This artifact is intended for vLLM/FlashInfer serving. It is not a vanilla Transformers checkpoint. PrismaQuant is a Fisher-weighted, mixed-precision quantization toolkit. Rather than forcing the whole model into one dtype, it predicts the loss penalty of quantizing each Linear independently — Δloss ≈ 0.5 · Htrace · MSEW, where Htrace is the Linear’s Fisher diagonal trace from a calibration probe and MSEW is the format-specific weight reconstruction error. A small ILP then picks per-Linear formats from a menu of {NVFP4, MXFP8E4M3, FP8SOURCE, BF16} to minimize total predicted Δloss under a target bits-per-weight budget. Because the cost model is principled and additive across Linears, it lets the toolkit stack several layers of optimization without each one stepping on the others. The shipped checkpoint was produced with the full stack below. — Fisher-weighted ILP picks one of NVFP4 / MXFP8 / FP8SOURCE / BF16 per Linear under the global bpp budget. — Pareto sweep across bpp targets surfaces the knee — the smallest size at which…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: rdtand
Теги: vllm, mistral3, mistral, compressed-tensors, prismaquant, nvfp4, fp8, quantized
Лайков: 3 | Загрузок: 36
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
