MKLLM-7B is an open-source Large Language Model for the Macedonian language. The model is built on top of the amazing Mistral-7B-v0.1 model by continued pretraining on a mix of Macedonian and English text. A corpus of around 300M tokens, repeated in 2 epochs, was used for the training and even though this might be considered small compared to other similar projects, the resulting model is very capable in understanding and processing the Macedonian language. This is the translate-tuned version of MKLLM-7B. It was trained by taking MKLLM-7B and then performing QLoRA instruction training with axolotl by using the alpaca format. We tested the model on the Flores dataset against some translation specific models, the much larger Llama3-70B and Google Translate as currently the best solution for EN-MK translation. We did not include any models from the same size in the testing as their results were suboptimal. In order to leverage the instruction training your prompt should follow the alpaca format: Here’s a full example of translating «Hi there, how’s it going?»
Модальности:
Генерация текста
Задача: Генерация текста
Автор: trajkovnikola
Теги: mistral, axolotl, mk, en, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 21
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
