HyLo converts a pretrained Transformer into a hybrid instead of pretraining one from scratch. Each layer is converted to one of two types: attention layers become Multi-head Latent Attention (MLA), which caches a low-rank latent rather than full keys and values, while the remaining layers become linear blocks (Gated DeltaNet or Mamba-2) that hold a fixed-size recurrent state and no KV cache. The converted model is then trained for long context, retaining short-context accuracy while running on a small fraction of the original KV cache. This checkpoint upcycles meta-llama/Llama-3.2-1B-Instruct into 8 MLA layers and 8 Gated DeltaNet layers (16 layers total, 1.61B parameters), and was supervised-fine-tuned at 65,536 tokens (64K) of context with meta-llama/Llama-3.1-8B-Instruct as the distillation teacher. Introduced in Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling (arXiv:2604.24715), where it appears as HyLo-Llama-8MLA8GDN in Table 2 (Llama-3.2-1B backbone). Layer types are placed by index, not in a repeating pattern: the MLA layers sit where the base model is most sensitive to losing full attention. — MLA layers (8): [0, 2, 4, 6, 8, 10, 12, 14] — Gated DeltaNet…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: amd
Теги: llama, hybrid, mla, gated-deltanet, long-context, kv-cache-compression, distillation, amd
Лайков: 3 | Загрузок: 101
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
