Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari We introduce OpenELM, a family of Open-source Efficient Language Models. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. We pretrained OpenELM models using the CoreNet library. We release both pretrained and instruction tuned models with 270M, 450M, 1.1B and 3B parameters. Our pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens. Please check license agreements and terms of these datasets before using them. We have provided an example function to generate output from OpenELM models loaded via HuggingFace Hub in generateopenelm.py`. Please refer to this link to obtain your hugging face access token. Additional arguments to the hugging face generate function can be passed via generatekwargs. As an example, to speedup the inference, you can try lookup token speculative generation by passing the…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: TommyZQ
Теги: openelm, custom_code
Лайков: 3 | Загрузок: 109
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
