— Suppression of unknown word generation by using byte fallback in SentencePiece tokenizer and conversion to huggingface Tokenizers format — Pretrained by wikipedia dataset and cc100 dataset — Use of Mistral 300M — Fine-tuning ce-lery/japanese-mistral-300m-base with kunishou/databricks-dolly-15k-ja If you want to restruct this model, you can refer this Github repository. — Preprocess with sentencepiece — Pretraining with flash attention2 and torch.compile and DeepSpeed — Fine-tuning with databricks-dolly-15k-ja If you find my mistake,error,…etc, please create issue. If you create pulreqest, I’m very happy! The following hyperparameters were used during training: — learningrate: 5e-06 — trainbatchsize: 4 — evalbatchsize: 4 — seed: 42 — distributedtype: multi-GPU — gradientaccumulationsteps: 64 — totaltrainbatchsize: 256 — optimizer: Adam with betas=(0.9,0.95) and epsilon=0.0001 — lrschedulertype: cosine — lrschedulerwarmupsteps: 1000 — numepochs: 200 — mixedprecision_training: Native AMP — Transformers 4.35.2 — Pytorch 2.1.1+cu121 — Datasets 2.14.5 — Tokenizers 0.14.1
Модальности:
Генерация текста
Области применения:
Следование инструкциям
Задача: Генерация текста
Автор: ce-lery
Теги: tensorboard, mistral, generated_from_trainer, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 37
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
