A 1.27B parameter LLaMA-style decoder-only language model, trained entirely from scratch on 2x NVIDIA L40S GPUs using JAX + Flax NNX, then converted to HuggingFace Transformers format. > TL;DR — KodaLite reaches ~37% average accuracy on standard LLM benchmarks. It is severely undertrained (only 1.64B tokens vs 40B–3T for comparable models), which places it just below GPT-2-124M despite having 10× more parameters. A nice illustration of the Chinchilla scaling law: tokens matter more than parameters at this budget. Evaluated against 8 comparable ~1B-parameter models on the same benchmarks (HellaSwag, ARC-E/C, WinoGrande, PIQA, BoolQ, OpenBookQA, LAMBADA-OpenAI). The Chinchilla scaling law (DeepMind, 2022) states that a model with N parameters needs approximately 20×N training tokens to be well-trained: KodaLite has seen only 6.5% of what it would need to be competitive. A bigger but undertrained model scores lower than a smaller but well-trained one. The LAMBADA perplexity (94 vs 17 for GPT-2-124M) is the clearest signal: the base language modeling is not converged. On PIQA (physical commonsense) the gap is smallest — that kind of knowledge appears to be learned faster than factual…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: YoAbriel
Теги: llama, from-scratch, jax, conversational, en, model-index, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 57
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
