Notus is a collection of fine-tuned models using Direct Preference Optimization (DPO) and related RLHF techniques. This model is the first version, fine-tuned with DPO over zephyr-7b-sft-full, which is the SFT model produced to create zephyr-7b-beta. Following a data-first approach, the only difference between Notus-7B-v1 and Zephyr-7B-beta is the preference dataset used for dDPO. In particular, when we started building distilabel, we invested time understanding and deep-diving into the UltraFeedback dataset. Using Argilla, we’ve found data issues in the original UltraFeedback dataset, leading to high-scores for bad responses (more details in the training data section). After curating several hundreds of data points, we decided to binarize the dataset using the preference ratings, instead of the original critique overallscore`, and verified the new dataset with Argilla. Using preference ratings, instead of critiques scores, led to a new dataset where the chosen response is different in ~50% of the cases. Using this new dataset with DPO we fine-tuned Notus, a 7B model, that surpasses Zephyr-7B-beta and Claude 2 on AlpacaEval. > Important note: While we opted for the average of…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: LoneStriker
Теги: mistral, dpo, rlaif, preference, ultrafeedback, conversational, en, model-index
Лайков: 2 | Загрузок: 10
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.