This all-middle-looped AdamW experiment is published separately at Banaxi-Tech/BananaMind-2.1-Lite-All-Looped-AdamW. It does not overwrite the original Lite or the partially looped AdamW experiment. BananaMind 2.1 Lite is a 24,949,999-parameter causal language model built for the under-25M class. — 19,950,029 parameters in the tied-token Transformer core — 4,999,970 parameters in a two-hash causal trigram module — BananaMind 2 Nano’s 8,192-token tokenizer — 13 physical layers and 24 effective layer passes — execution: L1 → (L2 → … → L12) × 2 → L13 — L1 is the prelude, L2-L12 are one recurrent middle stack, and L13 is the coda — every middle-layer visit reuses the same physical layer weights on pass two — 4,096-token context The trigram module uses two independent 51,699-entry tables with 48 features per hash. Their 96 concatenated features are projected to the 384-wide residual stream. The trigram representation is computed once and reinjected before every visit to L5 and L9, once in each recurrent-stack pass. L5 and L9 have separate learned injection scales, both initialized to 0.5. The NPset allocation is 75% normalized code and 25% raw originalcode`. Curriculum weights move…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: Banaxi-Tech
Теги: bananamind21_lite_25m, causal-lm, custom-code, recurrent-stack, all-middle-looped, n-gram, custom_code, en
Лайков: 3 | Загрузок: 212
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.