🤗 [LongAlign Dataset] • 💻 [Github Repo] • 📃 [LongAlign Paper] LongAlign is the first full recipe for LLM alignment on long context. We propose the LongAlign-10k dataset, containing 10,000 long instruction data of 8k-64k in length. We investigate on trianing strategies, namely packing (with loss weighting) and sorted batching, which are all implemented in our code. For real-world long context evaluation, we introduce LongBench-Chat that evaluate the instruction-following capability on queries of 10k-100k length. Chat prompt template for LongAlign-7B-64k and LongAlign-13B-64k: ChatGLM3-6B-128k uses the same prompt template as ChatGLM3-6B. If you find our work useful, please consider citing LongAlign:
Модальности:
Генерация текста
Задача: Генерация текста
Автор: zai-org
Теги: llama, Long Context, en, zh, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 236
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
