This is quantized version of instruction-pretrain/InstructLM-500M created using llama.cpp This repo contains the general models pre-trained from scratch in our paper Instruction Pre-Training: Language Models are Supervised Multitask Learners. We explore supervised multitask pre-training by proposing Instruction Pre-Training, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of Instruction Pre-Training. Instruction Pre-Training outperforms Vanilla Pre-training in both general pre-training from scratch and domain-adaptive continual pre-training. In pre-training from scratch, Instruction Pre-Training not only improves pre-trained base models but also benefits more from further instruction tuning. In continual pre-training, Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B. 🤗 We share our data and models with example usages, feel free to open…
Модальности:
Генерация текста
Области применения:
Следование инструкциям
Задача: Генерация текста
Автор: QuantFactory
Теги: gguf, en, endpoints_compatible
Лайков: 3 | Загрузок: 310
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
