lambda/pythia-12b-deduped-synthetic-instruct - Каталог нейросетей
Генерация текста

lambda/pythia-12b-deduped-synthetic-instruct

Добавлено:
lambda/pythia-12b-deduped-synthetic-instruct

This model is created by finetuning EleutherAI/pythia-12b-deduped on the Dahoas/synthetic-instruct-gptj-pairwise. — Finetuned by: Lambda — Model type: Transformer-based Language Model — Language: English — Pre-trained model: EleutherAI/pythia-12b-deduped — Dataset: Dahoas/synthetic-instruct-gptj-pairwise — Library: transformers — License: Apache 2.0 The model was trained on the Dahoas/synthetic-instruct-gptj-pairwise. We split the original dataset into the train (first 32000 examples) and validation (the remaining 1144 examples) subsets. We finetune the model for 4 epoches with the help of deepspeed. This took 8xA100 80GB 17 hours, where we set batchsizepergpu to 4 (so global batch size is 32), and learning rate to 0.0000025` (with linear decay to zero at the last trainig step). You can find a Weights and Biases record here.

Модальности:
Генерация текста

Области применения:
Следование инструкциям


Задача: Генерация текста
Автор: lambda
Теги: gpt_neox, causal-lm, pythia, en, text-generation-inference, endpoints_compatible
Лайков: 3  |  Загрузок: 18

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.