Just a fiction oriented 6bpw exl2 quantization of https://huggingface.co/jondurbin/bagel-dpo-34b-v0.2 Quantized on 300K tokens of two Vicuna format chats, a sci fi story and a fiction story at a long context. This should yield better storywriting performance than the default exl2 quantization. Being a Yi model, try running a lower temperature with ~0.05 MinP, a little repitition penalty, maybe mirostat with a low tau, and no other samplers. Yi tends to run «hot» by default. 24GB GPUs can run Yi-34B-200K models at 45K-75K context with exllamav2, and performant UIs like exui. I go into more detail in this post
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: brucethemoose
Теги: llama, conversational, text-generation-inference, endpoints_compatible
Лайков: 3 | Загрузок: 15
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.