A ~100M parameter, decoder-only GPT-style base language model, trained from scratch on a single RTX Pro 6000, on roughly 3 billion tokens streamed from a mix of open web/educational/math corpora. This is a base (pretrained) model, not an instruction-tuned or chat model. It completes/continues text; it does not reliably follow instructions or answer questions directly. It also has no fine-tuning for factual accuracy — it can and does confidently generate fluent, plausible-sounding but factually incorrect content. Treat it as a small, from-scratch research/hobby model, not a production system. — Architecture: nanoGPT-style decoder-only transformer — 14 layers, 10 attention heads, 640 embedding dim — ~101.6M parameters (weight-tied embeddings/output head) — Context length: 1024 tokens — Uses PyTorch’s fused scaleddotproductattention (flash-attention kernel) — Tokenizer: tiktoken GPT-2 BPE encoding (tiktoken.getencoding(«gpt2»)), vocab size 50,257. No custom tokenizer was trained. — Optimizer: Muon (for 2D weight matrices) + AdamW (for embeddings, layernorms, biases) — a hybrid setup, following the approach popularized in recent efficient-pretraining recipes. — Precision: trained…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: OPENGCM
Теги: gtm-gpt, gpt, nanogpt, from-scratch, small-language-model, slm, pretraining, muon-optimizer
Лайков: 3 | Загрузок: 98
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
