This repository contains 8-bit quantized models in GPTQ format of TheBlokes’s wizard-vicuna 13B in FP16 HF format. These models are the result of quantization to 8-bit using GPTQ-for-LLaMa. While most metrics suggest that 8-bit is only marginally better than 4-bit, I have found that the 8-bit model follows instructions significantly better. The responses from the 8-bit model feel very close to the quality of GPT-3, whereas the 4-bit model lacks some «intelligence». With this quantized model, I can replace GPT-3 for most of my work. However, a drawback is that it requires approximately 15GB of VRAM, so you need a GPU with at least 16GB of VRAM. The content below is straight copy and paste from TheBloke’s README with the 4 bit content changed to 8 bit and referencing this model. 1. Click the Model tab. 2. Under Download custom model or LoRA, enter deetungsten/wizard-vicuna-13B-GPTQ-8bit-128g. 3. Click Download. 4. Wait until it says it’s finished downloading. 5. Click the Refresh icon next to Model in the top left. 6. In the Model drop-down: choose the model you just downloaded, wizard-vicuna-13B-GPTQ-8bit-128g. 7. If you see an error in the bottom right, ignore it — it’s temporary.…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: deetungsten
Теги: llama, causal-lm, en
Лайков: 2 | Загрузок: 15
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
