A 295B-A21B MoE served on one 128 GB DGX Spark at 2.9 bits per parameter, through stock vLLM with an out-of-tree plugin — no forked runtime, no core patches. Weights are stored in the gridbook codebook formats (NVFP4-CB / FP8-CB: product-VQ over 8-weight vectors whose codebook values live on hardware grids, two-tier scale coding) and decoded inside custom CUDA/Triton kernels at matmul time, so prefill runs on native tensor cores instead of paying a dequantization tax. Formats + kernels: github.com/RobTand/gridbook. Produced by the PrismaQuant measured-allocation pipeline. — Base model: tencent/Hy3 (production release, Apache 2.0), 80 layers, 192 experts top-8 + 1 shared, GQA 8 KV heads. — Artifact: 5 safetensors shards, 105.7 GB total, 2.9 bpp over quantizable parameters. Joint-menu allocation, measured not hand-picked: codebook rungs compete with vanilla NVFP4/FP8 per Linear, and the measurement chose — 36 Linears landed on vanilla FP8; vanilla NVFP4 was offered and never won a unit. Full breakdown in Format allocation. — MTP included, CB-quantized at FP8-CB K44 (2.6 GB vs 7.5 GB BF16), serving as a k=1 speculative-decode drafter out of the box. The rung was selected by a…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: rdtand
Теги: gridbook, hy_v3, quantization, nvfp4, codebook, vllm, single-gpu, prismaquant
Лайков: 3 | Загрузок: 20
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
