Athena-R3-7B is a 7-billion-parameter causal language model fine-tuned from DeepSeek-R1-Distill-Qwen-7B. This model is specifically tailored to enhance reasoning capabilities, making it adept at handling complex problem-solving tasks and providing coherent, contextually relevant responses. Model Developer: Aayan Mishra Model Type: Causal Language Model Architecture: Transformer with Rotary Position Embeddings (RoPE), SwiGLU activation, RMSNorm, and Attention QKV bias Parameters: 7 billion total Layers: 32 Attention Heads: 24 for query and 4 for key-value (Grouped Query Attention) Vocabulary Size: Approximately 151,646 tokens Context Length: Supports up to 128,000 tokens Languages Supported: Primarily English, with capabilities in other languages Licens The fine-tuning process involved 60 epochs over approximately 90 minutes, utilizing a curated dataset focused on reasoning tasks, including mathematical problem-solving and logical inference. This approach aimed to bolster the model’s proficiency in complex reasoning and analytical tasks. Athena-R3-7B is designed for a variety of…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: aayanmishra-ml
Теги: qwen2, text-generation-inference, unsloth, trl, conversational, en, zh, fr
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
