This Model is 8bit Version of EleutherAI/gpt-j-6B. It is converted by Facebook’s bitsandbytes library. The original GPT-J takes 22+ GB memory for float32 parameters alone, and that’s before you account for gradients & optimizer. So for finetuning on single GPU This model is converted into 8bit. Here’s how to run it:  The original GPT-J takes 22+ GB memory for float32 parameters alone, and that’s before you account for gradients & optimizer. Even if you cast everything to 16-bit, it will still not fit onto most single-GPU setups short of A6000 and A100. You can inference it on TPU or CPUs, but fine-tuning is way more expensive. Here, we apply several techniques to make GPT-J usable and fine-tunable on a single GPU with ~11 GB memory: — large weight tensors are quantized using dynamic 8-bit quantization and de-quantized just-in-time for multiplication — using gradient checkpoints to store one only activation per layer: using dramatically less memory at the cost of 30% slower training — scalable fine-tuning with LoRA and 8-bit Adam In other words, all of the large weight-matrices are frozen in…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: addy88
Теги: gptj, endpoints_compatible
Лайков: 2 | Загрузок: 25
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
