The TinyCodeLM family of tiny language models (LMs) is a collection of fully open-source pretrained and instruction tuned generative code models in 150M and 400M sizes. These models are pretrained on a mixture of open-source web text and Python code. The instruction tuned TinyCodeLM models are optimized for Python code synthesis, and are trained on synthetic edit sequence data generated with the LintSeq algorithm. Despite being trained on only 72 billion tokens of text, the models outperform many of the available tiny & open source Python code synthesis LMs on HumanEval and MBPP. The TinyCodeLM-LintSeqInstruct models are state-of-the-art on Python synthesis for their size. Model Developers Ulyana Piterbarg, Lerrel Pinto, Rob Fergus (NYU) Variations TinyCodeLM comes in two sizes (150M and 400M parameters) in pretrained and edit sequence instruction tuned variants. Output Models generate text and code. Instruction tuned models generate code via sequences of «diffs». Model Architecture TinyCodeLMs are autoregressive language models with architectures that mimic the two smallest versions of GPT-2 (Radford et al., 2019), while integrating the transformer architecture changes of the…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: upiter
Теги: olmo, endpoints_compatible
Лайков: 3 | Загрузок: 1,306
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
