A 250M-parameter language model that decodes in three modes from one set of weights, following NVIDIA’s Nemotron-Labs-Diffusion: Tri-Mode recipe (joint AR + block-diffusion training). Built from codelion/dhara-250m-ar-base and trained to ~60B cumulative tokens (~50B added for this model). Architecture: LLaMA-style with Canon depthwise-conv layers, QK-norm, logit soft-cap, GQA, RoPE θ=8M. Demo: dhara-chat Space — chat with it and compare all three decoding modes. Chat template is ChatML + Hermes-style tools (shipped in the tokenizer); the model supports an OpenAI-style tools=[…] argument. 10 tasks (9 zero-shot + MMLU 5-shot); metric = accnorm where defined, else acc. Columns: dhara-base (the AR base, codelion/dhara-250m-ar-base), this model in AR mode and in diffusion mode (dhara-diff), and — as an external reference run through the same harness — SmolLM-135M**. Tri-mode training improves the AR base by +1.4 points (AR mode) and by +7.2 points in diffusion mode. dhara-diff (42.7) is the headline configuration — bidirectional answer scoring drives large gains over the base on sciq (+40), boolq (+13) and arc_easy (+11). Data efficiency. SmolLM-135M (48.3) was trained on ~600B…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: codelion
Теги: dhara_ar, diffusion-llm, block-diffusion, autoregressive, self-speculation, tri-mode, conversational, custom_code
Лайков: 4 | Загрузок: 3,499
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
