A 56M‑parameter, from‑scratch recurrent language model that combines a per‑channel leaky integrator (FWKV) with the RWKV‑8 ROSA copy‑signal mechanism. It is a research experiment designed to explore how far purely recurrent architectures can go on small‑scale, curated conversational data. The model was chat‑tuned on HuggingFaceH4/ultrachat200k` and uses a simple two‑role template: — Architecture: 14 stacked FWKV blocks. Each block replaces the standard attention with a fixed, data‑independent decayed accumulator: stateₜ = W·stateₜ₋₁ + kₜ·vₜ, where W = clamp(sigmoid(w), min=0.1). The recurrence is computed exactly via a vectorised parallel scan (no approximations). — ROSA (Rapid Online Suffix Automaton): A parameter‑free, causal predictor that injects the token that historically followed the longest matching suffix of the current context. The ROSA signal is embedded and added to the input representation of each token. — Factorised embedding/head: 128‑dimensional embedding space, projected to a 512‑dimensional model space, with tied weights. — Context length: 1024 tokens. No positional embeddings are used, making the context window “free” in terms of parameters. — Tokenizer: GPT‑2…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: FWKV
Теги: fwkv, rosa, rwkv, linear-transformer, recurrent-neural-network, chat, ultrachat, custom-architecture
Лайков: 3 | Загрузок: 29
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
