RWKV-Red-Team/ARWKV-7B-Preview-0.1-NoG

This version doesn’t have the parameter «g». It freezes the MLP and uses 7B logits for distillation alignment. This is an early preview of our 7B parameter RNN-based model, trained on 2k context length (only stage-2 applied, without SFT or DPO) through 3-stage knowledge distillation from Qwen2.5-7B-Instruct. While being a foundational version, it demonstrates: — ✅ RWKV-7’s efficient recurrence mechanism — ✅ No self-attention, fully O(n) — ✅ Constant VRAM usage — ✅ Single-GPU trainability Roadmap Notice: We will soon open-source different enhanced versions with: — 🚀 16k+ context capability — 🧮 Math-specific improvements — 📚 RL enhanced reasoning model

Модальности:
Генерация текста

Области применения:
Диалог / чат

Задача: Генерация текста
Автор: RWKV-Red-Team
Теги: rwkv_hybrid, conversational, custom_code, en, zh
Лайков: 4  |  Загрузок: 18

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.