PLaMo 3 NICT 2604 31B Base is a 31B model distilled from a 120B model, developed by Preferred Networks, Inc. in collaboration with National Institute of Information and Communications Technology, NICT. PLaMo 3 NICT models adopt a hybrid architecture with Sliding Window Attention (SWA) and Traditional Attention layers. PLaMo 3 NICT 2604 31B Base is released under PLaMo community license. Please check the following license and agree to this before downloading. — (EN) https://plamo.preferredai.jp/info/plamo-community-license-en — (JA) https://plamo.preferredai.jp/info/plamo-community-license-ja Please check the PLaMo community license and contact us via the following form for commercial purpose. — Model size: 31B — Trained tokens: 1.5T — Developed by: — Preferred Networks, Inc. — in collaboration with National Institute of Information and Communications Technology, NICT — Model type: Causal decoder-only — Language(s): English, Japanese — License: PLaMo community license PLaMo 3 NICT 2604 31B was distilled from a 120B model which was trained with two phases, phase 1 with 2.625T tokens and phase 2 with 0.375T tokens. The percentage of datasets in each phase is shown in the following…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: pfnet
Теги: plamo3, custom_code, en, ja
Лайков: 4 | Загрузок: 268
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
