This is the MPT-30B-Instruct but with added support to finetune using peft (tested with qlora). It is not finetuned further, the weights are the same as the original MPT-30B-Instruct. I have not traced through the whole huggingface stack to see if this is working correctly but it does finetune with qlora and outputs are reasonable. Inspired by implementations here https://huggingface.co/cekal/mpt-7b-peft-compatible/commits/main https://huggingface.co/mosaicml/mpt-7b/discussions/42. MPT-30B-Instruct is a model for short-form instruction following. It is built by finetuning MPT-30B on Dolly HHRLHF derived from the Databricks Dolly-15k and the Anthropic Helpful and Harmless (HH-RLHF) datasets. It is also trained on Competition Math, Duorc, CoT GSM8k, Qasper, Quality, Summ Screen FD and Spider. License: CC-By-SA-3.0_ This model was trained by MosaicML and follows a modified decoder-only transformer architecture. Blog post: Raising the bar for open-source foundation models Codebase (mosaicml/llm-foundry repo) * Questions: Feel free to contact us via the MosaicML Community Slack! Bespokenizer46 > I need to convince my friend, Phyllis, that she should train a custom LLM for her Fortune…
Модальности:
Генерация текста
Области применения:
Следование инструкциям
Задача: Генерация текста
Автор: eluzhnica
Теги: mpt, Composer, MosaicML, llm-foundry, custom_code, text-generation-inference
Лайков: 2 | Загрузок: 29
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
