Spark-2 is a high-efficiency, sub-10M parameter language model developed by SurjoLabs. With an Intelligence Index of 9.14, it proves that combining 3:1 Grouped-Query Attention (GQA), value-subtraction projection (XSA), and deep recurrent weight-sharing yields exceptional reasoning capabilities at micro scale. Spark-2 demonstrates that specialized recurrent architectures can match or exceed standard transformers multiple times their size when trained on dense, high-quality data. Custom Architecture: Built on a Llama-style backbone featuring XSA Attention (orthogonal value-subtraction) and Recurrent Weight-Sharing. By passing activations twice through its 6 recurrent layers, the model achieves the computational depth of a 15-layer model (1 prelude + 6×2 recurrent + 2 coda) while storing only 9 physical layers of weights (~9.15M total parameters). Optimized Tokenizer: Uses a custom 4,096-vocabulary English tokenizer. Restricting vocabulary size prevents embedding bloat, keeping ~87% of all parameters dedicated strictly to transformer logic. Extreme Overtraining: Trained on ~20.97 Billion tokens (20,000 steps 1,048,576 tokens/step) to drive dense knowledge compression and syntax…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: SurjoLabs
Теги: spark2a, causal-lm, custom-architecture, slm, small-language-model, conversational, custom_code, en
Лайков: 4 | Загрузок: 484
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
