Llama 2 (13B) fine-tuned on Clibrain’s Spanish instructions dataset and optimized using GPTQ. Llama 2 is a collection of pre-trained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. This is the repository for the 7B pre-trained model. Quantization methods usually belong to one of two categories: 1. Post-Training Quantization (PTQ): We quantize a pre-trained model using moderate resources, such as a calibration dataset and a few hours of computation. 2. Quantization-Aware Training (QAT): Quantization is performed before training or further fine-tuning. GPTQ falls into the PTQ category, and this is particularly interesting for massive models, for which full model training or even fine-tuning can be very expensive. Specifically, GPTQ adopts a mixed int4/fp16 quantization scheme where weights are quantized as int4 while activations remain in float16. During inference, weights are dequantized on the fly and the actual compute is performed in float16. — Memory savings close to x4 for int4 quantization, as the dequantization happens close to the compute unit in a fused kernel, and not in the GPU global memory. — Potential speedups thanks to…
Модальности:
Генерация текста
Области применения:
Следование инструкциям
Задача: Генерация текста
Автор: clibrain
Теги: llama, quantization, gptq, es, text-generation-inference, 4-bit
Лайков: 3 | Загрузок: 23
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
