A 109.5M-parameter Swahili (Kiswahili) base language model, pretrained from scratch on 1.41B tokens on a single NVIDIA T4. This is the first KW5 model. Its successor is kw5-149M. This is a base model. It does next-token prediction only. The instruction tuned version is kw5-109M-instruct. Output with torch.manualseed(0)` on CPU (sampling is on, so yours will differ): Fluent, and wrong: Mauritius, Seychelles, Zimbabwe and Swaziland do not border Tanzania. That is typical of this model. No is needed. Pretraining packed documents without a leading BOS. An earlier revision of this card said every training sequence began with ; it did not. Adding one anyway is harmless: on held-out text it moves the loss by 0.009 nats. The shipped generationconfig.json` samples at temperature 0.2, top-p 0.9, repetition penalty 1.3, which is what the output above used. The config allows 2048 positions, but the model was only ever trained at 1024. The planned extension to 2048 was gated behind a decay phase that never ran (see Training), so ropetheta` is still 10,000. Measured loss by position on 12 long Swahili documents: Past 1024 tokens the model is reading positions it never saw. Keep prompt plus…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: regnant-io
Теги: llama, swahili, kiswahili, causal-lm, base-model, pretrained, african-languages, low-resource
Лайков: 3 | Загрузок: 933
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
