EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a new baseline for fast decoding of Large Language Models (LLMs) with provable performance maintenance. This approach involves extrapolating the second-top-layer contextual feature vectors of LLMs, enabling a significant boost in generation efficiency. — EAGLE is: — certified by the third-party evaluation as the fastest speculative method so far. — achieving 2x speedup on gpt-fast. — 3x faster than vanilla decoding (13B). — 2x faster than Lookahead (13B). — 1.6x faster than Medusa (13B). — provably maintaining the consistency with vanilla decoding in the distribution of generated texts. — trainable (within 1-2 days) and testable on 8x RTX 3090 GPUs. So even the GPU poor can afford it. — combinable with other parallelled techniques such as vLLM, DeepSpeed, Mamba, FlashAttention, quantization, and hardware optimization. EAGLE-2 uses the confidence scores from the draft model to approximate acceptance rates, dynamically adjusting the draft tree structure, which further enhances performance. — EAGLE-2 is: — 4x faster than vanilla decoding (13B). — 1.4x faster than EAGLE-1 (13B). EAGLE-3 removes the feature…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: yuhuili
Теги: llama, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 656
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
