Gradient incorporates your data to deploy autonomous assistants that power critical operations across your business. To learn more or collaborate on a custom model, drop us a message at contact@gradient.ai. This model extends LLama-3 8B’s context length from 8k to > 160K, developed by Gradient, sponsored by compute from Crusoe Energy. It demonstrates that SOTA LLMs can learn to operate on long context with minimal training (< 200M tokens) by appropriately adjusting RoPE theta. !image/png — meta-llama/Meta-Llama-3-8B-Instruct as the base — NTK-aware interpolation [1] to initialize an optimal schedule for RoPE theta, followed by a new data-driven RoPE theta optimization technique — Progressive training on increasing context lengths similar to the Large World Model [2] (See details below) We build on top of the EasyContext Blockwise RingAttention library [3] to scalably and efficiently train on contexts up to 262144 tokens on Crusoe Energy high performance L40S cluster. GGUF is available on on Crusoe's huggingface account. Check it out here: crusoeai/Llama-3-8B-Instruct-262k-GGUF For training data, we generate long contexts by augmenting SlimPajama. Gradient is accelerating AI…
Модальности:
Генерация текста
Области применения:
Диалог / чат Следование инструкциям
Задача: Генерация текста
Автор: bullerwins
Теги: llama, meta, llama-3, conversational, en, text-generation-inference, endpoints_compatible, 8-bit
Лайков: 3 | Загрузок: 21
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
