Poro 2 8B Base is an 8B parameter decoder-only transformer created through continued pretraining of Llama 3.1 8B to add Finnish language capabilities. It was trained on 165B tokens using a carefully balanced mix of Finnish, English, code, and math data. Poro 2 is a fully open source model and is made available under the Llama 3.1 Community License. Poro 2 was created in a collaboration between AMD Silo AI, the TurkuNLP group of the University of Turku, and High Performance Language Technologies (HPLT). Training was conducted on the LUMI supercomputer, using compute resources generously provided by CSC — IT Center for Science, Finland. This model demonstrates how continued pretraining can efficiently add new language capabilities to existing models while maintaining performance in the original domains. Through the combination of English and Finnish training data, we achieve a model that substantially outperforms the base Llama 3.1 8B model in Finnish while maintaining solid English proficiency. For more details on our training and data curation process, check out our Continued Pretraining Playbook. The Poro 2 model family includes both 8B and 70B models, and there are three…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: LumiOpen
Теги: llama, fi, en, text-generation-inference, endpoints_compatible
Лайков: 4 | Загрузок: 1,724
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
