This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the HuggingFaceH4/ultrachat200k and the rohansolo/BBHindiHinglish datasets. The goal was to test fine-tuning using biilingual datasets and techniques, to see if bilingual capabilities can be added while improving performance. Model performs really well in Hindi, but can be mediocre at best in Hinglish, which we’re attributing to translation quality. Note — Scores for BB-L-01-7B were created using EluetherAI’s lm eval tool, currently awaiting results from open-llm to ensure independent results. All other model results are from Openllm leaderboard The following hyperparameters were used during training: — learningrate: 2e-05 — trainbatchsize: 32 — evalbatchsize: 16 — seed: 42 — distributedtype: multi-GPU — numdevices: 8 — gradientaccumulationsteps: 2 — totaltrainbatchsize: 512 — totalevalbatchsize: 128 — optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 — lrschedulertype: cosine — numepochs: 1 — Transformers 4.35.0 — Pytorch 2.1.0+cu118 — Datasets 2.14.6 — Tok Rush and…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: rohansolo
Теги: mistral, alignment-handbook, generated_from_trainer, conversational, hi, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
