 NorOLMo 1.0 is a base (not instruction-tuned) large language model, continually pre-trained on Norwegian and Scandinavian data starting from the English OLMo2-13B model. It is the only modern fully open large language model specifically adapted for Norwegian and Sámi. At the same time, its performance is comparable to other multilingual open models (see the Evaluation section below). «Fully open» here means that the entirety of the trainining data and training pipeline is transparent and reproducible, not only that the model weights are available. The model was trained for 33 000 steps on around 275 billion tokens. Maximum sequence length was set to 4 096 in the beginning of training, and then was extended to 16 384 starting from the checkpoint step31000. Intermediate checkpoints are published here as branches. The main branch` contains the model’s weights after step 33 000 (stage 3). In a few months, we hope to follow with post-trained versions of NorOLMo. Below is a comparison of fully-open models supporting Norwegian. The figure shows the aggregate score across all 35 NorEval 1.1 tasks (5 categories,…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: HPLT
Теги: olmo2, norwegian, norsk, HPLT, nb, nn, no, se
Лайков: 3 | Загрузок: 2,439
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
