A reference implementation of a routed-ternary tiny language model with a bit-exact Python ↔ C99 inference engine, sized for microcontroller-class RAM budgets. The contribution is integration, not a new architecture: a complete train → ternary export → base-3 packing → C99 inference path, with bit-exact Python ↔ C parity enforced by tests. It combines three known ideas — ternary weights (BitNet b1.58), a per-token-routed 3-pathway block (Hymba, MossNet), and a byte tokenizer at super-tiny scale (Guertler 2024). — Code: https://github.com/TilelliLab/atome-lm — Project home / live in-browser demo: https://atomelm.com — License: Apache-2.0 (code, weights, everything) > ⚠️ This is a research artifact, not a product or a general chatbot. Read the > «Honest results» section below before citing any number. The honesty dossier lives in > HONESTRESULTS.md` > in the source repo. All numbers are single-seed, from the training logs shipped alongside. The 944K result reverses. At 944K parameters the FP32 vanilla baseline beats Atome by ~11% in val loss and perplexity, same recipe / same val slice / same seed. Atome’s bet is the sub-1M, MCU-class regime: the 3-pathway inductive bias substitutes…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: TilelliLab
Теги: atome-lm, ternary, bitnet, microcontroller, edge-ai, tinyml, byte-level, language-model
Лайков: 3 | Загрузок: 19
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
