A 15.19M-parameter Llama-style language model trained from scratch on TinyStories where all 42 attention and FFN linear layers are ternary {−1, 0, +1} with per-output-channel absmean scales, in the style of BitNet b1.58. Embeddings (tied input/output) and RMSNorm gains remain FP32. Mirrors karpathy’s stories15M: dim 288 · 6 layers · 6 heads · 6 KV heads · SwiGLU hidden 768 · vocab 32,000 · context 256 · RMSNorm · RoPE · tied embeddings. 15,191,712 parameters. — ternary.pt — deployable hard-ternary checkpoint (43 MB). No latent weights: the 42 linear layers store only int8 values in {−1, 0, +1} plus one FP32 scale per output channel. Verified by reloading from disk and generating text. — ckpt.pt — full latent training checkpoint (182 MB, FP32 latent weights + optimizer + RNG, resumable, step 10,000). — tokenizer.model — the Llama 2 SentencePiece tokenizer from karpathy/llama2.c (vocab 32,000). — Data: TinyStories (~470M tokens), packed to 256-token sequences with the Llama 2 tokenizer — 10,000 steps × 65,536 tokens/step = 655M tokens, bf16 autocast, AdamW lr 6e-3 cosine to 10%, warmup 2%, weight decay 0.1, single L40S GPU (~50 min) — Straight-through estimator: FP32 latent weights…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: brianbellx
Теги: ternary, bitnet, 1.58bit, tinystories, llama
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
