NOSA is a trainable sparse attention mechanism designed for KV-cache offloading with an explicit locality constraint, paired with an inference system (NOSI) to realize its efficiency. It improves long-context/long-generation quality over prior offloading baselines while boosting decoding throughput by up to 5.04× vs FullAttn, 1.92× vs InfLLMv2, and 1.83× vs ShadowKV on 1B/3B/8B LLMs. The model was presented in the paper NOSA: Native and Offloadable Sparse Attention. We train 1B, 3B, and 8B models using FullAttn, InfLLMv2, DMA, and NOSA. The following NOSA models have been released on Hugging Face: Please reach out to the authors if additional baseline models (FullAttn, InfLLMv2, or DMA) are needed. You may open an issue on the GitHub repository or contact the authors directly via email.
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: openbmb
Теги: llama, conversational, custom_code, en, zh, text-generation-inference, endpoints_compatible
Лайков: 4 | Загрузок: 144
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
