This is a fine-tuned variant of MPT-7B, trained on the SAIL dataset (https://arxiv.org/abs/2305.15225). The preprocessed version can be found here: https://huggingface.co/datasets/lukasmoeller/sail_preprocessed I may have forgotten to add EOD tokens at the end of the target, might retrain if anyone is interested. MPT-7B is a decoder-style transformer pretrained from scratch on 1T tokens of English text and code. This model was trained by MosaicML. MPT-7B is part of the family of MosaicPretrainedTransformer (MPT) models, which use a modified transformer architecture optimized for efficient training and inference. These architectural changes include performance-optimized layer implementations and the elimination of context length limits by replacing positional embeddings with Attention with Linear Biases (ALiBi). Thanks to these modifications, MPT models can be trained with high throughput efficiency and stable convergence. MPT models can also be served efficiently with both standard HuggingFace pipelines and NVIDIA’s FasterTransformer. This model uses the MosaicML LLM codebase, which can be found in the llm-foundry repository. It was trained by MosaicML’s NLP team on the MosaicML…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: lukasmoeller
Теги: mpt, Composer, MosaicML, llm-foundry, StreamingDatasets, custom_code, text-generation-inference
Лайков: 3 | Загрузок: 23
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
