Custom Architecture · Custom BPE Tokenizer · Reasoning Fine-Tuned on Dual RTX 4090 Every layer of AstraGPT-7B was implemented from first principles in PyTorch. No AutoModel, no copy-paste — pure custom code. Fine-tuning was performed on a private Linux VPS with 2× NVIDIA RTX 4090 GPUs (total 48GB VRAM). The SFT (Supervised Fine-Tuning) training loop was implemented from scratch with production-grade features: Training loop features: — ✅ Gradient accumulation — effective large batch training — ✅ Mixed precision (BF16) — full RTX 4090 tensor core utilization — ✅ Cosine LR schedule with warmup — smooth convergence — ✅ Gradient clipping — stable training — ✅ W&B logging — real-time loss/LR tracking — ✅ Checkpoint saving — best model tracking by loss After fine-tuning, the LoRA a — Hallucination: Can produce confident but incorrect answers — always verify — Math limits: Complex multi-step math may fail — 7B is a small model — English-primary: Best performance in English — Reasoning trigger: tags work most reliably with explicit n prefix in prompt Built from…
Модальности:
Генерация текста
Области применения:
Генерация кода Логика и рассуждение Диалог / чат
Задача: Генерация текста
Автор: tantra-ai-labs
Теги: qwen2, from-, custom-architecture, custom-tokenizer, reasoning, chain-of-thought, think-tags, coding
Лайков: 3 | Загрузок: 29
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
