SmallCoder is a 303M parameter LLaMA-style language model trained from scratch for code generation and algorithmic reasoning. This checkpoint represents a 6B-token Supervised Fine-Tuning (SFT) run that fixed a critical End-of-Sequence (EOS) token bug from earlier versions. Despite its compact size, SmallCoder achieves state-of-the-art (SOTA) coding performance for > Trained with support from Google’s TPU Research Cloud (TRC) program. > ⚖️ SmallCoder nearly matches Mistral 7B on HumanEval while being 23× smaller. ———————- | :———: | :————————————————— | :——————————- | :———-: | ———— | :——————- | :———— | :————: | > humaneval/mbpp were computed with manual evaluation (maxnewtokens=512, temp=0.2) due to SFT format truncation issues in lm-eval. This model was trained with support from the Google TPU Research Cloud (TRC) program. Special thanks to the open datasets that enabled this work: FineWeb, StarCoderData, Nemotron, and OpenWebMath. > 🔬 SmallCoder (303M) demonstrates that a carefully designed
Модальности:
Генерация текста
Области применения:
Генерация кода
Задача: Генерация текста
Автор: Beebey
Теги: llama, smallcoder, code-llm, code-generation, sft, pretraining, tpu, 303m
Лайков: 4 | Загрузок: 18
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
