Asilarkness/DiffuRefill-1B - Каталог нейросетей
Генерация текста

Asilarkness/DiffuRefill-1B

Добавлено:
Asilarkness/DiffuRefill-1B

Status: pretraining, about half way. Step ~100k of 200k, ~46B effective training tokens, merge round 290. The weights here are a live pretraining checkpoint: fluent English, facts still unreliable, not instruction-tuned. Use globalema.pt`; it is the best checkpoint at every round. A 1.08B masked-diffusion language model decoded with confidence-targeted re-infilling: draft the whole sequence in a handful of parallel denoising passes, then spend a few more passes rewriting only the tokens the model is least sure about. An autoregressive model needs one pass per token; this one needs ~20 passes for a whole sequence, whatever its length. Tracked every round on fixed, deterministic batches, so the change between two rounds carries no sampling noise: The guard-probe curve at the top is measured on an Ultra-FineWeb slice. When the fact-dense sources were added at step ~96.8k, Ultra-FineWeb fell from 100% to 56% of the documents, and that probe rose by design while the fact probe improved quickly. From here the fact probe and the held-out CE are the numbers to watch. No public benchmark scores are reported yet: at this stage they would sit near chance and say little. Fleet. Ephemeral…

Модальности:
Генерация текста


Задача: Генерация текста
Автор: Asilarkness
Теги: diffurefill, diffusion-language-model, masked-diffusion, parallel-decoding, diloco, en
Лайков: 3  |  Загрузок: 9,272

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.