This is a reproduction of the SeerAttention paper. The model contains additional learnable AttnGate modules on top of meta-llama/Meta-Llama-3-8B-Instruct. The AttnGate modules accelerate long-context attention inference by enabling block-level sparsity. During training, the AttnGates are optimized via self-distillation while keeping the original model weights frozen. Specifically, the AttnGates learn to mimic the 2D-maxpooled outputs of the attention maps. At inference time, the soft scores produced by the gates are converted into binary masks, thereby reducing both the I/O overhead and computational cost of the attention mechanism.
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: SeerAttention
Теги: llama, conversational, text-generation-inference, endpoints_compatible
Лайков: 4 | Загрузок: 13
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
