In our latest endeavour, we performed continued pre-training of a large language model (Mistral-7b-v0.1) to understand and generate text in new languages, including Serbian, Bosnian and Croatian using an innovative approach. Rather than depending only on extensive datasets in the target language, our method utilizes a more compact set of both synthetic and human-curated data along with some mixture of CC Web data, which is implemented in two strategic phases: 1. Establishing a comprehensive demonstration of all grammatical and orthographic rules pertinent to the language. 2. Supplying a diverse array of examples that not only reinforce these rules but also integrate a wide range of linguistic nuances. While our approach is uniquely tailored to our objectives, we have drawn some inspiration from recent advancements in language model training. Specifically, the conceptual strategies discussed in the paper ADAPTING LARGE LANGUAGE MODELS VIA READING COMPREHENSION provided valuable insights, though our methods diverge significantly in practice. By adopting this inspired approach, we aim to efficiently teach the model new languages with a balanced blend of accuracy and linguistic…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: iskonai
Теги: mistral, en, sr, hr, bs, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 31
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
