Speck1-140M-Instruct is a 140.7M parameter English instruction-tuned language model. It was initialized from Speck1-140M, a hybrid model that interleaves global grouped-query attention with gated causal convolution, then fully fine-tuned for one epoch on SpeckChat1. The model uses a native chat template with optional system messages and was trained with assistant-only loss. The architecture is unchanged from Speck1-140M apart from 3 added role-token embeddings. It contains 18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward. Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream via learned projections. The system message is optional and may appear only first. Remaining roles must alternate between user and assistant. No default system prompt is injected. During generation, the prompt ends after n, and terminates the response. Speck1-140M-Instruct works with the Transformers Auto classes through its bundled custom model and tokenizer code. Set trustremotecode=True when loading it. The bundled generation path is validated for single-prompt greedy decoding. Direct forward passes support…
Модальности:
Генерация текста
Области применения:
Диалог / чат Следование инструкциям
Задача: Генерация текста
Автор: specklabs
Теги: speck, causal-lm, instruction-tuned, chat, hybrid, grouped-query-attention, causal-convolution, custom_code
Лайков: 4 | Загрузок: 1,421
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
