RWKV-GLM-4.7-Flash-exp is an alpha-stage experimental model that converts GLM-4.7-Flash into a fully linear-attention-dominant architecture using the RADLADS distillation methodology. Every single layer runs RWKV-7 — there are no standalone self-attention layers. This model introduces PrimeRWKV, a new architectural paradigm consisting of two layer types: Unlike conventional hybrid architectures that alternate between linear and full attention layers (e.g., Jamba, Griffin, Zamba), TICA (Tiny Infused Causal Attention) takes a fundamentally different approach: attention is infused directly within an RWKV-7 block as a lightweight gated auxiliary path, rather than occupying its own layer. — NoPE (No Positional Encoding): The RWKV-7 decay mechanism handles positional information; TICA relies solely on the causal mask. — Few-Head GQA: 4 query heads / 2 KV heads with head dimension 128 — extremely compact. — QK-Norm: RMSNorm on Q and K for training stability without positional encodings. — LoRA-Gated Output: A learned sigmoid gate controls how much TICA contributes, allowing the model to modulate attention strength per-token. — Zero-Initialized Output Projection: TICA starts with zero…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: OpenMOSE
Теги: rwkv07i_moe, rwkv, rwkv7, hybrid, linear-attention, distillation, tica, moe
Лайков: 4 | Загрузок: 97
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.