Mistralclassificationheadqlora has a new transformer head attached to it for sequence classification task and then resulting model has been finetuned on dair-ai/emotion dataset using QloRA. The model has been trained for 1 epoch on 1x A40 GPU. The evaluation loss for the emotion-head-3 attached to it was 1.313**. The base model used was This experiment was performed using Transformer-heads library The training script for attaching a new transformer head for classification task using QLoRA is following: For evaluating the transformer head that has been attached to the base model, you can refer to the following colab notebook Colab Notebook for Evaluation trainepochs = 1 evalepochs = 1 loggingsteps = 1 trainbatchsize = 4 evalbatch_size = 4 outputdir=»emotionlinearprobe», learningrate=0
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Syed-Hasan-8503
Теги: generated_from_trainer, classification, Transformer-heads, finetune, chatml, gpt4, synthetic data, distillation
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
