MPT-1b-RedPajama-200b is a 1.3 billion parameter decoder-only transformer trained on the RedPajama dataset. The model was trained for 200B tokens by sampling from the subsets of the RedPajama dataset in the same proportions as were used by the Llama series of models. This model was trained by MosaicML and follows a modified decoder-only transformer architecture. To use the optimized triton implementation of FlashAttention, you can load with attnimpl=’triton’ and move the model to bfloat16` like so: This model uses the MosaicML LLM codebase, which can be found in the MosaicML Examples Repository. The architecture is a modification of a standard decoder-only transformer. The transformer has 24 layers, 16 attention heads, and width 2048. The model has been modified from a standard transformer in the following ways: It uses ALiBi and does not use positional embeddings. It uses QK LayerNorm. * It does not use biases. The model was trained for 200B tokens (batch size 2200, sequence length 2048). It was trained on the following data mix: 67% RedPajama Common Crawl 15% C4 4.5% RedPajama GitHub 4.5% RedPajama Wikipedia 4.5% RedPajama Books 2.5% RedPajama Arxiv * 2% RedPajama…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: anas-awadalla
Теги: mosaic_gpt, custom_code
Лайков: 3 | Загрузок: 372
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
