Veyra2-Mango-15M-Base is a 15.7M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation. dclmbaseline: 50% finephrase: 20% cosmopediav2: 10% finemath4plus: 10% ultrafinewebmultistyle: 5% ultrafineweb_qa: 5% This stage linearly transitions from the Stage 1 mixture to the Stage 2 mixture. finephrase: 30% dclmbaseline: 30% cosmopediav2: 18% finemath4plus: 10% ultrafinewebmultistyle: 5% ultrafineweb_qa: 5% ultrachat: 2% — Final step: 14,306 — Tokens seen: 30,000,000,000 — Tokens per step: 2,097,022 — Sequence length: 2048 — Last train loss: 2.8336 import torch from transformers import AutoTokenizer, AutoModelForCausalLM model = AutoModelForCausalLM.frompretrained( modelid, torchdtype=torch.float16, devicemap=»auto», ) prompt = «In the 19th century» inputs = toke
Модальности:
Генерация текста
Задача: Генерация текста
Автор: veyra-ai
Теги: llama, causal-lm, base-model, veyra, small-language-model, en, model-index, text-generation-inference
Лайков: 4 | Загрузок: 1,128
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
