This repo only contains the AttnGates’ weights for Llama-3.1-8B-Instruct Model. SeerAttention introduces learnable AttnGate modules to accelerate the computationally intensive prefill stage of long-context large language models (LLMs) via dynamic block-level sparsity. The AttnGates are trained in a parameter-efficient self-distillation framework, where they learn to mimic the 2D max-pooled attention patterns of the original frozen model, preserving its integrity while avoiding costly retraining. During inference, these gates generate block-sparse binary masks by applying threshold/TopK to their learned soft scores, enabling efficient computation through a custom block-sparse FlashAttention kernel. For R1-Distilled models, we remove the two passes generation setup (think + summary), we directly ask the models to output anwser after thinking. The generation max length is set to 10240.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: SeerAttention
Теги: llama, endpoints_compatible
Лайков: 4 | Загрузок: 21
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
