A 15B-parameter token-mixer supernet derived from Apriel-1.6 via stochastic distillation. Every decoder layer exposes four trained mixer options—Full Attention, Sliding Window Attention, Gated DeltaNet, and Kimi Delta Attention—enabling flexible architecture selection from a single checkpoint. — Model Size: 15B parameters — Layers: 48 decoder layers, each with 4 mixer variants — Context Length: 262K positions (runtime dependent) — Languages: English (best) — Supernet architecture: Single checkpoint containing 4 mixer types at every layer, yielding 4⁴⁸ ≈ 7.9 × 10²⁸ possible architectures — Four mixer types: Full Attention (FA), Sliding Window Attention (SWA, window=4096), Gated DeltaNet (GDN), Kimi Delta Attention (KDA) — Stage 1 distillation checkpoint: Trained via During training, all four mixer types at each layer were trained simultaneously using stochastic local sampling—each layer’s mixer was drawn…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: ServiceNow-AI
Теги: apriel2, image-text-to-text, conversational, custom_code
Лайков: 4 | Загрузок: 93
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.