A 49.8M-parameter base language model, pretrained from scratch (random initialisation, own tokenizer, no teacher, no distillation) on 20B tokens of openly licensed English text in 8.58 H100-hours. It was built for the Global Innovation Build Challenge V2, Track 01 (foundational LLM development, 50M-parameter cap). Weights: https://huggingface.co/Unluckyexe/gibc-v2-lm. Code, training logs, ablations and every eval JSON: https://github.com/Unluckyathecking/gibc-v2-lm. This is a base model: it continues text and has no instruction or chat tuning. The model has no KV cache, so generate() is not wired up; the loop above (or scripts/demo.py in the GitHub repo) is the intended way to sample. Context is 1,024 tokens. Every training document starts with (id 0), and the tokenizer prepends it by default; scores are best with it. — From scratch. Random initialisation; tokenizer trained by us on a 3 GB sample of the training mix. No pretrained weights, teacher model or synthetic/LLM-generated dataset was used. — Tokens. 20B (38,146 steps x 524,288 tokens). — Optimiser. Muon (lr 0.02, wd 0.01) on block matrices, AdamW (lr 3e-3) on embeddings, norms and scalars; warmup-stable-decay schedule (1%…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: Unluckyexe
Теги: gibc, small-language-model, from-scratch, pretrained, base-model, custom_code, en, model-index
Лайков: 3 | Загрузок: 159
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
