FSCCS/dMoE-16B - Каталог нейросетей
Генерация текста

FSCCS/dMoE-16B

Добавлено:
FSCCS/dMoE-16B

dMoE is a block-level Mixture-of-Experts (MoE) framework designed for Diffusion Large Language Models (dLLMs). By aggregating token-level expert distributions within each block into a unified block-level distribution, dMoE substantially reduces the number of uniquely activated experts during inference, mitigating memory-bound bottlenecks without sacrificing performance. — Paper: dMoE: dLLMs with Learnable Block Experts — Project Page: https://fscdc.github.io/dMoE/ — Repository: https://github.com/fscdc/dMoE — Learnable Block Experts: Introduces block-level MoE routing into dLLMs, drastically compressing the number of activated unique experts across diffusion steps. — Reduced MoE Bandwidth: Significantly reduces memory bandwidth consumed by expert weight loading during the block diffusion process. — Improved Efficiency-Accuracy Trade-off: Achieves 1.14x to 1.66x end-to-end latency speedup while maintaining competitive performance on benchmarks. — Plug-and-play on LLaDA-2.0: Built directly on top of LLaDA-2.0-mini without architectural changes.

Модальности:
Генерация текста

Области применения:
Диалог / чат


Задача: Генерация текста
Автор: FSCCS
Теги: llada2_moe, conversational, custom_code
Лайков: 4  |  Загрузок: 27

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.