A W4A4 + paired-4:8 sparse compressed checkpoint of moonshotai/Kimi-K2.5, produced with MoESQ. The MoE expert weights are NVFP4 with paired-4:8 structured sparsity, and they are stored sparse: only the kept values plus a small mask are on disk, not a dense NVFP4 tensor with zeros in place. The target is NVIDIA Blackwell (SM100 and SM120) sparse tensor cores. — Base model: moonshotai/Kimi-K2.5 (MoE, 384 routed experts, 61 layers) — Compression: NVFP4 W4A4 plus paired-4:8 sparsity on the routed MoE experts — Effective weight precision: about 2 bits/weight on routed-expert linears — Checkpoint size: 347.6 GiB, vs 524.8 GiB for the same weights stored as dense NVFP4 (0.66×) — Kernel: paired-4:8 sparse NVFP4 grouped GEMM (CUTLASS, SM100 and SM120) through vLLM’s paired48nvfp4` MoE backend Links — Paper: Hardware-Native Joint Sparse-Qu It contains the compression code, the vLLM integration and the kernels. — Other MoESQ checkpoints: ISTA-DASLab/moesq collection This checkpoint does not load in upstream vLLM. The MoESQ repository installs a patched vLLM v0.30.0 (the paired48nvfp4` MoE backend and…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: ISTA-DASLab
Теги: kimi_k25, feature-extraction, kimi-k2.5, moe, nvfp4, w4a4, semi-structured-sparsity, sparse-storage
Лайков: 3 | Загрузок: 271
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.