Veyra-30M-Instruct is the first public instruction-tuned checkpoint in the Veyra 30M line. It is built on top of veyra-ai/Veyra-30M-Base and is intended to test how far a 30M-parameter English base model can be pushed with masked ChatML supervised fine-tuning. — Base model: veyra-ai/Veyra-30M-Base — Parameters: approximately 34.6M — Language: English — Context length: 1024 tokens — Architecture: decoder-only causal language model — Chat format: ChatML — License: Apache 2.0 — Status: experimental instruct release Veyra-30M is a compact decoder-only transformer-style model. — Vocabulary size: 8,192 — Hidden size: 512 — Layers: 8 — Layer pattern: alternating attention and MLP-only blocks — Query heads: 8 — Key/value heads: 2 — Attention type: grouped-query attention — MLP: SwiGLU — Normalization: RMSNorm — Positional encoding: RoPE — Context length: 1024 tokens — Tied input/output embeddings — KV cache support for generation This checkpoint was trained with masked ChatML supervised fine-tuning. — Base checkpoint: veyra-ai/Veyra-30M-Base — SFT tokens: 100M non-padding SFT tokens — Supervised assistant tokens: approximately 35M supervised assistant tokens — Loss masking: system and…
Модальности:
Генерация текста
Области применения:
Следование инструкциям Диалог / чат
Задача: Генерация текста
Автор: veyra-ai
Теги: onnx, llama, instruct, chatml, tiny-language-model, local-ai, experimental, veyra
Лайков: 3 | Загрузок: 256
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
