Important Note: I have not created this MIT licensed model, I discovered it and downloaded it. It was taken down by its creators, so I reupload it. More info: GPT-JX is a 3 billion paramter autoregressive Foundational Large Language Model pre-trained on High Quality, Cleaned and Deduplicated 1.1 trillion tokens of english text and code. GPT-JX uses the base architecture of traditional Transformers Decoder with slight changes which is discussed later. GPT-JX was pre-trained on tokens for English text and 20 Programming Languages. GPT-JX shows impressing performance when compared to Large Language Models with 7 billion parameters such as LLaMa-7B-v2, Falcon-7B & MPT-7B. We made slight changes to the traditional Transformers Decoder to create the Base Architecture for our GPT-JX model, the changes are listed: — We used the SwiGLU activation function in the architecture of GPT-JX instead of ReLU. — Attention with Linear Biases(AliBi) was used as positional embeddings for GPT-JX instead of absolute positional embeddings(as used in traditional Transformers Decoder) and Rotatary Positional Embeddings(as used in case of GPT-J & GPT-NeoX) — Trainable Parameters: 2646255776 — Number of…
Модальности:
Генерация текста
Области применения:
Генерация кода
Задача: Генерация текста
Автор: KnutJaegersberg
Теги: gpt_jx, text-generation-inference, custom_code, en, code
Лайков: 3 | Загрузок: 25
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
