> Training complete (August 2, 2026). Final state: outer step 195, val loss 2.872, > ~423M community tokens — pinned at revision > 5c39e2cb. > This model stays fully usable (snippet below). The co-op’s current run is > fineweb-150m — a 145M-param > model on FineWeb-Edu; npx coop-ai start now contributes there. This 14.8M-parameter model was pretrained from scratch with no cluster, no server, and no funding: volunteers ran training rounds on their own computers and submitted compressed pseudo-gradients as pull requests on a public Hugging Face dataset repo. A stateless GitHub Actions cron job aggregated them into outer steps — DiLoCo-style low-communication data parallelism, coordinated by nothing but free-tier infrastructure. — Training run: July 27 – August 2, 2026 · 195 outer steps · ~423M tokens (≈121% of the ~300M Chinchilla-optimal budget for this size) — Final validation loss: 2.872 cross-entropy on held-out TinyStories (≈17 perplexity over an 8k vocab; random-init baseline is ln 8192 ≈ 9.01) — Contributors: 4 identities on consumer hardware — Apple Silicon laptops, plain CPUs, and one GPU (public leaderboard) — Everything is auditable: every gradient arrived as a public PR…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: commonsense-ai
Теги: tinystories, distributed-training, diloco, community, en
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
