A masked (absorbing-state) diffusion language model trained with Aether. Unlike an autoregressive model, generation does not proceed left to right. The model starts from an all-[MASK] sequence and unmasks progressively, so the number of forward passes (NFE) is a knob rather than a function of sequence length. > Perplexity here is an upper bound, not an exact likelihood. A masked > diffusion model has no exact factorization of log p(x); what is reported is a > Monte Carlo estimate of a variational bound. It is comparable to other diffusion > models evaluated the same way, and not directly comparable to an > autoregressive model’s exact perplexity, which would flatter the AR model. Research and demonstration of masked diffusion language modelling: studying the NFE-quality tradeoff, comparing sampling strategies, and as a fixture for inference-serving work. — Small and undertrained. 55,543,634 parameters and roughly 3,932,160,000 tokens. It captures vocabulary and local phrasing, not long-range coherence or factual grounding. — Unconditional. There is no prompt input; it generates from an all-[MASK] sequence. Prompt-conditioned infilling is a natural extension the architecture…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: ameyg910
Теги: aether, diffusion, masked-diffusion, language-model, mdlm
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
