Speck2-140M-Instruct is a 140.7M parameter English instruction-tuned language model. It was initialized from Speck2-140M, a hybrid model pretrained from scratch on 20B tokens with a three-phase curriculum, then fully fine-tuned for one epoch on the 500,000-conversation SpeckChat2 mixture. The Speck2 release updates the pretrained base while retaining the SpeckChat2 instruction-tuning data and one-epoch post-training recipe. The model uses a native chat template with optional system messages and was trained with assistant-only loss. The architecture is unchanged from Speck2-140M apart from 3 added role-token embeddings. It contains 18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward. Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream through learned projections. The system message is optional and may appear only first. Remaining roles must alternate between user and assistant. No default system prompt is injected. During generation, the prompt ends after n, and terminates the response. Speck2-140M-Instruct works with the Transformers Auto classes through its bundled custom model and…
Модальности:
Генерация текста
Области применения:
Диалог / чат Следование инструкциям
Задача: Генерация текста
Автор: specklabs
Теги: speck, causal-lm, instruction-tuned, chat, hybrid, grouped-query-attention, causal-convolution, custom_code
Лайков: 3 | Загрузок: 172
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
