A 1.3B-parameter GPT-2-style language model for Russian, converted from the original ai-forever/rugpt3xl Megatron-LM checkpoint into a native HuggingFace transformers format. This is a base (pretrained) model, not instruction-tuned. It performs text completion and can be fine-tuned for downstream tasks. Details in «A family of pretrained transformer language models for Russian» paper. The tokenizer includes a simple chat template for question-answering: > Note: This is a base model, not an instruction-tuned chatbot. The chat template provides > a basic structure, but the model may not always follow instructions precisely. For reliable > conversational behavior, fine-tune the model on instruction/chat data. The model is fully compatible with standard HuggingFace training workflows. LoRA target modules: qproj, kproj, vproj, oproj, upproj, downproj This model was originally trained with DeepSpeed’s SparseSelfAttention using the alternating pattern: even layers (0, 2, 4, …) use block-sparse attention, odd layers (1, 3, 5, …) use standard dense causal attention. The sparse layers use a FixedSparsityConfig derived from the «Generating Long Sequences with Sparse Transformers» paper…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: evilfreelancer
Теги: rugpt3xl, gpt3, russian, causal-lm, conversational, custom_code, ru
Лайков: 4 | Загрузок: 100
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
