This model is a Kazakh language version of the LLaMA model with 1.9 billion parameters, trained for causal language modeling. The train dataset includes some mixed Russian texts, which occasionally cause the model to generate Russian text. Despite this, the model shows promising results. Future steps may include retraining with a purer dataset, fine-tuning, or using the model for various NLP tasks with additional fine-tuning. — Training Examples: Over 5.3 million examples — Training Hardware: Two NVIDIA A100 GPUs (80GB each) — Training Status: Ongoing, currently partway through the first epoch — Optimizer: Cosine with restarts scheduler — Parallelism: Distributed Data Parallel (DDP) — Number of Warmup Steps: 8000 Name: Kadyrbek Nurgali — Email: nurgaliqadyrbek@gmail.com — LinkedIn: Kadyrbek Nurgali @misc{ nurgalikadyrbek2024, author = {NURGALI, Kadyrbek}, title = {llama-1.9B-kaz (Revi The…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: nur-dev
Теги: llama, kk, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
