A mixture-of-experts language model trained from scratch: 998,007,552 parameters in total, 148,660,992 active per token (998.0M total / 148.7M active). Own tokenizer, own data pipeline, no fine-tuned base. Code and ten languages: English, Indonesian, Malay, Simplified and Traditional Chinese, Japanese, Korean, Tamil, Hindi (Devanagari and romanised) and romanised Urdu. This is the base model: it continues text and code; it does not follow instructions (the chat model is AneekC/quipu-moe-1B-A149M-chat). At this size it is often wrong. It is published as an honest record of what a small sparse model learns under a $20 compute cap, with every A/B result, including the ones that lost. Fields marked not yet measured had no result file when this card was generated. Mix achieved (share of training tokens): code 60.0%, engLatn 28.0%, otherlanguages 12.0%. Training tokens in the shard set: 8,300,000,000 (the train split; validation and evaluation splits are separate). Code licences kept: mit, apache-2.0, bsd-2-clause, bsd-3-clause, isc, cc0-1.0, unlicense. Language identification: every text document was checked with the fastText specialist from AneekC/lid-specialists-9plus1 (MIT, revision…
Модальности:
Генерация текста
Области применения:
Генерация кода Мультиязычность
Задача: Генерация текста
Автор: AneekC
Теги: from-scratch, mixture-of-experts, small-language-model, multilingual, code, en, id, ms
Лайков: 3 | Загрузок: 196
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
