LLM-jp-3 is the series of large language models developed by the Research and Development Center for Large Language Models at the National Institute of Informatics. This repository provides llm-jp-3-3.7b-instruct3 model. For an overview of the LLM-jp-3 models across different parameter sizes, please refer to: — LLM-jp-3 Pre-trained Models — LLM-jp-3 Fine-tuned Models. — torch>=2.3.0 — transformers>=4.40.1 — tokenizers>=0.19.1 — accelerate>=0.29.3 — flash-attn>=2.5.8 — Model type: Transformer-based Language Model — Total seen tokens: 2.1T tokens The tokenizer of this model is based on huggingface/tokenizers Unigram byte-fallback model. The vocabulary entries were converted from llm-jp-tokenizer v3.0. Please refer to README.md of llm-jp-tokenizer for details on the vocabulary construction procedure (the pure SentencePiece training does not reproduce our vocabulary). The models have been pre-trained using a blend of the following datasets. We have fine-tuned the pre-trained checkpoint with supervised fine-tuning and further aligned it with Direct Preference Optimization. The datasets used for supervised fine-tuning are as follows: The datasets used for supervised fine-tuning are as…
Модальности:
Генерация текста
Области применения:
Диалог / чат Следование инструкциям
Задача: Генерация текста
Автор: llm-jp
Теги: llama, conversational, en, ja, text-generation-inference
Лайков: 3 | Загрузок: 745
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
