The i3-200M Model (aka Redherring) is a novel hybrid architecture combining convolutional/recurrent layers with full attention layers for efficient language modeling. This architecture uniquely blends RWKV-style time-mixing with Mamba state-space dynamics in the early layers, followed by standard multi-head attention in deeper layers. — Total Parameters: ~169.85M — Architecture: 10 Hybrid (RWKV-Mamba) + 6 Full Attention Layers = 16 Total Layers — Vocabulary Size: 32,000 tokens — Hidden Dimension (dmodel): 512 — Attention Heads: 16 — State Dimension (dstate): 32 — Max Sequence Length: 256 — Tokenization: BPE 1. Hybrid Architecture: Combines the efficiency of recurrent/convolutional processing with the power of attention — Early layers use RWKV-Mamba hybrid for efficient sequence processing — Later layers use full multi-head attention for complex pattern recognition 2. Memory-Optimized Training: — Streaming vocabulary building (no full text storage) — Vocabulary caching (build once, reuse) — Efficient chunk frequency counting — Automatic memory cleanup 3. Multi-Dataset Pre-training: Trained on diverse text sources for robust language understanding — TinyStories: Narrative and…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: i3-lab
Теги: rwkv-hybrid, i3-architecture, hybrid-model, rwkv-mamba, custom_code, en, endpoints_compatible
Лайков: 3 | Загрузок: 37
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
