By Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov We introduce BD3-LMs, a family of Block Discrete Denoising Diffusion Language Models that achieve SOTA likelihoods among diffusion models and enable generation of arbitrary-length sequences. BD3-LMs combine the strengths of autoregressive and diffusion language models by decomposing a token sequence into blocks and performing discrete diffusion within each block. By tuning the block size, we interpolate between autoregressive and diffusion models which introduces a trade-off between quality and sample efficiency. We propose a recipe of building effective BD3-LMs that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. BD3-LMs are Block Discrete Denoising Diffusion Language Models. They combine the strengths of autoregressive and diffusion language models by decomposing a token sequence into blocks and performing discrete diffusion within each block.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: kuleshov-group
Теги: bd3lm, fill-mask, diffusion, autoregressive, language-modeling, custom_code
Лайков: 4 | Загрузок: 4,517
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
