A 4.3M parameter language model with ternary weights (-1, 0, +1) trained from scratch on a free-tier 2-thread CPU in 2 hours. No GPU used at any point. The model generates coherent children’s stories with dialogue and narrative structure. Bits-per-character (BPC) is a tokenizer-independent metric that fairly compares models with different vocabularies. It measures how well a model compresses raw text: BPC = totalcrossentropynats / (totalcharacters × ln(2)). Lower is better. > FlashLM v4 has only seen 2.3% of the training data that TinyStories-1M used. The validation loss curve was still declining when the 2-hour time limit was reached — this model is undertrained, not underdesigned. Extended training is planned. Evaluated on 500 TinyStories validation stories (405,081 characters). FlashLM v4 is a complete redesign from v3. The two versions are not directly comparable by validation loss because they were trained on different datasets with different vocabularies. v3’s biggest bottleneck was the output projection layer. With a 50,257-token vocabulary and dmodel=256, the softmax head consumed 86% of all training compute**, leaving the ternary model core starved. Training on…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: changcheng967
Теги: ternary, cpu-training, matmul-free, tiny-language-model, efficient-ai, bitlinear, tinystories, en
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
