IvmeLabs/Ivme-Conversate-v2-Base - Каталог нейросетей
Генерация текста

IvmeLabs/Ivme-Conversate-v2-Base

Добавлено:
IvmeLabs/Ivme-Conversate-v2-Base

İvme (Turkish: acceleration) is a series of stupidly small language models built to punch above their weight. This is the second release: a 24M parameter decoder-only base model trained from scratch, this time with a much heavier training diet. v1 produced grammatically correct sentences that did not really say anything when strung together, fluent without being about anything. v2 keeps almost the exact same architecture as v1 on purpose, and instead fixes the two things that actually mattered: far more training data, and a data mix weighted toward material that teaches a model to stay on topic across sentences. Nearly every setting above matches v1 on purpose. The point of v2 was to isolate the improvement to data and training, not to a bigger model. Every metric improved. The ARC-Easy acc_norm gain (30.85% → 39.98%) is more modest than the raw acc numbers alone suggest, but still a real improvement over v1. Weaker on long-distance dependencies and island constraints, a known hard case for small models: This pattern, strong local agreement paired with weaker long-distance syntax, is typical for models at this scale. v1 was trained Chinchilla-optimal. v2 deliberately overtrains…

Модальности:
Генерация текста


Задача: Генерация текста
Автор: IvmeLabs
Теги: ivme, language-model, transformer, rope, swiglu, muon, from-scratch, tiny
Лайков: 4  |  Загрузок: 613

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.