AVALON-2B is the first sub-3B parameter language model to implement Self-Reflective Retrieval-Augmented Generation (Self-RAG) with learned reflection tokens. Built upon Qwen 3.5 2B, AVALON introduces a novel training pipeline that teaches the model to generate retrieval decision tokens without external retrieval infrastructure. — First Sub-3B Self-RAG: Breaks the 7B parameter barrier for self-reflective capabilities — On-Device Ready: 1.5GB quantized (Q4KM) runs at 40+ tok/s on Apple M3 — 82.5% Token Accuracy: Reliable generation of [Retrieval], [No Retrieval], and [Utility:X] tokens — No Catastrophic Forgetting: +0.41% MMLU improvement over base model AVALON generates special reflection tokens to enable adaptive retrieval: — Base Model: Qwen 3.5 2B (18 GDN + 6 Softmax attention layers) — Parameters
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: nuroai
Теги: qwen3_5_text, self-rag, retrieval-augmented-generation, qwen, on-device, edge-ai, reflection-tokens, conversational
Лайков: 3 | Загрузок: 24
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
