This model is a fine-tuned version of mistralai/Mistral-7B-v0.3-Instruct on the ReBatch/ultrachat400knl, the BramVanroy/stackoverflow-chat-dutch and the BramVanroy/norobotsdutch datasets. This model is a Dutch chat model, originally developed from Mistral 7B v0.3 Instruct and further finetuned first with SFT on multiple datasets. The model could generate wrong, misleading, and potentially even offensive content. Use at your own risk. Use with mistrals chat template. It achieves the following results on the evaluation set: — Loss: 0.8596 This model was trained with QLoRa in bfloat16 with Flash Attention 2 on one A100 PCIe, using the sft script from the alignment handbook on RunPod. The following hyperparameters were used during training: — learningrate: 0.0002 — trainbatchsize: 3 — evalbatchsize: 6 — seed: 42 — distributedtype: multi-GPU — gradientaccumulationsteps: 2 — totaltrainbatchsize: 6 — optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 — lrschedulertype: cosine — lrschedulerwarmupratio: 0.1 — num_epochs: 1 — PEFT 0.11.1 — Transformers 4.41.2 — Pytorch 2.2.0+cu121 — Datasets 2.19.1 — Tokenizers 0.19.1 The Mistral-7B-v0.3-Instruct model, on
Модальности:
Генерация текста
Области применения:
Диалог / чат Следование инструкциям
Задача: Генерация текста
Автор: ReBatch
Теги: mistral, alignment-handbook, trl, sft, generated_from_trainer, conversational, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 27
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
