MiniPLM-Qwen-1.2B is a 1.2B model with Qwen achitecture pre-trained from scratch on the Pile using the MiniPLM knowledge distillation framework with the offcial Qwen1.5-1.8B as the teacher model. We also open-source the pre-training corpus refined by Difference Sampling in MiniPLM for reproducibility. MiniPLM models achieves better performance given the same computation and scales well across model sizes:
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: MiniLLM
Теги: qwen2, conversational, en, text-generation-inference, endpoints_compatible
Лайков: 4 | Загрузок: 48
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.