A ~120M parameter, decoder-only GPT-style base language model, trained from scratch on a single RTX Pro 6000, on roughly 4 billion tokens streamed from a mix of open web/educational/math corpora. This is a base (pretrained) model, not an instruction-tuned or chat model. It completes/continues text; it does not reliably follow instructions or answer questions directly. It also has no fine-tuning for factual accuracy — it can and does confidently generate fluent, plausible-sounding but factually incorrect content. Treat it as a small, from-scratch research/hobby model, not a production system. A separate instruction-tuned variant (SFT on UltraChat) is planned as a follow-up release; this repo is the base-only checkpoint. — Architecture: nanoGPT-style decoder-only transformer — 14 layers, 8 attention heads, 704 embedding dim — ~119.4M parameters (weight-tied embeddings/output head) — Context length: 1024 tokens — Uses PyTorch’s fused scaleddotproductattention (flash-attention kernel) — Tokenizer: tiktoken GPT-2 BPE encoding (tiktoken.getencoding(«gpt2»)), vocab size 50,257. No custom tokenizer was trained. — Optimizer: Muon (for 2D weight matrices) + AdamW (for embeddings,…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: OPENGCM
Теги: gtm-gpt, gpt, nanogpt, from-scratch, small-language-model, slm, pretraining, muon-optimizer
Лайков: 3 | Загрузок: 191
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
