This model is a model that performed continued pre-training and fine-tuning (instruction tuning) using the depth up-scaling (DUS) technique disclosed by Upstage. Similar to the methodology disclosed in the paper, we expanded from 32 transformer blocks to 48 blocks and then continued pre-training with the public dataset. Pre-training was performed for 3 days using 4 ml.g5.48xlarge instances from AWS (NVIDIA A10G GPU x 32ea). For pre-training, we used a sample set from Wikipedia. Note that performance is not guaranteed since only a small number of datasets were used for the experiment. The number of samples for training set is just around 1.5 million after tokenization. For distributed training, all weights were trained without adapter techniques, and sharding parallelization was performed with ZeRO-2. The presets are as follows. After performing pre-training, instruction tuning and alignment tuning were performed sequentially. This process only took about 10 hours using AWS ml.g5.24xlarge (NVIDIA A10G GPU x 4ea). The dataset used for instruction tuning is a sample set of the OpenOrca dataset, and the dataset used for alignment tuning is Intel’s orcadpopairs dataset. All fine-tuning…
Модальности:
Генерация текста
Области применения:
Диалог / чат Следование инструкциям
Задача: Генерация текста
Автор: daekeun-ml
Теги: phi, conversational, en, text-generation-inference
Лайков: 3 | Загрузок: 214
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
