LoftQ (LoRA-fine-tuning-aware Quantization) provides a quantized backbone Q and LoRA adapters A and B, given a full-precision pre-trained weight W. This model, Llama-2-13b-hf-4bit-64rank, is obtained from LLAMA-2-13b. The backbone is under LoftQ/Llama-2-13b-hf-4bit-64rank and LoRA adapters are under the subfolder=’loftqinit’`. — Stored format: torch.bfloat16 — Size: ~ 26 GiB — Loaded format: bitsandbytes nf4 — Size loaded on GPU: ~6.5 GiB — rank: 64 — loraalpha: 64 — targetmodules: [«downproj», «upproj», «qproj», «kproj», «vproj», «oproj», «gate_proj»] Training Here’s an example of loading this model and preparing for the LoRA fine-tuning. We have conducted experiments on supervised fine-tuning of GSM8K and WikiText-2. Inference Here is an example code for inference after the model has been fine-tuned on GSM8K.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: LoftQ
Теги: llama, quantization , lora, en, text-generation-inference, endpoints_compatible
Лайков: 2 | Загрузок: 10
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.