— Introduction — Diff of TransNormerLLM2 — Released Weights — Benchmark Results — Inference and Deployment — Dependency Installation — Inference — Fine-tuning the Model — Community and Ecosystem — Disclaimer, License and Citation This official repo introduces the TransNormerLLM model, featuring its open-source weights. Additionally, it provides codes for Supervised Fine-tuning (SFT) and inference. TransNormerLLM evolving from TransNormer, standing out as the first LLM within the linear transformer architecture. Additionally, it distinguishes itself by being the first non-Transformer LLM to exceed both traditional Transformer and other efficient Transformer models (such as, RetNet and Mamba) in terms of speed and performance. — TransNormerLLM1 is released in Nov 2023, featuring three versions with 385M, 1B, and 7B parameters, trained on 1.4 trillion tokens. — The latest update transitions from TransNormerLLM1 to TransNormerLLM2, offering three updated versions with 1B, 3B, and 7B parameters, trained on 0.3 trillion tokens. — All versions are available as open-source under the Apache-2.0 license. — TransNormerLLM1 incorporates Simple GLU in its channel mixer, GLA in the token mixer,…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: OpenNLPLab
Теги: TransNormerLLM, custom_code, en, zh
Лайков: 3 | Загрузок: 17
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
