PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation PARD is a high-performance speculative decoding method that also enables low-cost adaptation of autoregressive draft models into parallel draft models. It offers the following advantages: — Low-Cost Training: PARD adapts AR (autoregressive) draft models into parallel draft models with minimal overhead. Compared to pure AR draft models, PARD achieves an average inference speedup of 1.78×. By introducing a conditional drop-token strategy, PARD improves training efficiency by up to 3× while maintaining the same level of accuracy. — Generalizability: Thanks to its target-independent design, a single PARD draft model can accelerate an entire family of target models. This contrasts with target-dependent approaches such as Medusa and EAGLE, which require retraining or tuning for each new target. As a result, PARD significantly reduces both deployment complexity and adaptation cost. — High Performance: When integrated into an optimized inference framework called Transformers+ PARD delivers up to a 4.08× speedup, with LLaMA3.1 8B reaches a state-of-the-art 311.5 tokens per second. When integrated into vLLM, PARD…
Модальности:
Генерация текста
Области применения:
Диалог / чат Логика и рассуждение
Задача: Генерация текста
Автор: amd
Теги: qwen2, conversational, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 140
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
