We present ELMOD — Efficient Language Model for On-Device Deployment — a compact (2.7B) German language model designed for efficient inference on resource-constrained hardware. This work was performed as research project in the scope of the compute time project ELMOD at NHR@FAU. ELMOD was trained on a limited computational budget (55k H100 GPU hours) using exclusively publicly available data. We developed a suite of German-specific data preprocessing, which differ from English-oriented counterparts in their handling of morphological variation, compounding, and orthographic conventions, to ensure high model performance, we developed a suite of German-specific data preprocessing. Furthermore, we introduced an quality filtering and rephrasing step, which increased the instructional quality of the data, improved performance during the annealing phase, and reduced overall compute requirements. Thanks to our architectural model choices as well as our data choices, including prefiltering, our educational-quality filtering and rephrasal to raise the educational-quality, ELMOD is the strongest performer in its size class ( !Evaluation results of our instruction-tuned models Evaluation…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: fraunhofer-iis
Теги: gpt_neox, conversational, de, en, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 510
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
