İvme (Turkish: acceleration) is normally a series of small autoregressive language models. This is not one of those. This is the same architecture family run through a different objective entirely: a masked/absorbing-state discrete diffusion language model instead of next-token prediction. «Exp» is doing real work in the name — this is an experiment to see what a small dSLM (diffusion small language model) can do at a scale we can train on one GPU in an afternoon, not a production model. Short version: the pipeline works, the sampler had a real bug that we found and fixed, and the model is too small and too lightly trained to have learned much beyond local fluency. All three of those are reported honestly below. Architecture lineage: IvmeLabs/Ivme-Conversate-v2-Base, same RMSNorm/RoPE/SwiGLU/tied-embedding building blocks, scaled from 24M to 130M params, with the one structural change masked diffusion actually requires: bidirectional attention instead of causal, since the model needs to see both sides of a masked token to denoise it. For reference, Ivme-Conversate-v2-Base (the AR sibling this architecture is descended from) trained on 12.85B tokens, roughly 7x more than this run.…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: IvmeLabs
Теги: expivme_diffusion, feature-extraction, language-model, transformer, rope, swiglu, diffusion, masked-diffusion
Лайков: 4 | Загрузок: 263
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
