Chat & support: TheBloke’s Discord server Want to contribute? TheBloke’s Patreon page TheBloke’s LLM work is generously supported by a grant from andreessen horowitz (a16z) Multiple GPTQ parameter permutations are provided; see Provided Files below for details of the options provided, their parameters, and the software used to create them. GPTQ models for GPU inference, with multiple quantisation parameter options. 2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference * Bohan Du’s original unquantised fp16 model in pytorch format, for GPU inference and for further conversions Multiple quantisation parameters are provided, to allow you to choose the best one for your hardware and requirements. Each separate quant is in a different branch. See below for instructions on fetching from different branches. — Bits: The bit size of the quantised model. — GS: GPTQ group size. Higher numbers use less VRAM, but have lower quantisation accuracy. «None» is the lowest possible value. — Act Order: True or False. Also known as descact`. True results in better quantisation accuracy. Some GPTQ clients have issues with models that use Act Order plus Group Size. — Damp %: A GPTQ parameter…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: TheBloke
Теги: llama, en, text-generation-inference, 4-bit, gptq
Лайков: 3 | Загрузок: 16
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
