This repository contains google/gemma-2-9b-it quantized from 16-bit floats to 4-bit integers, using xMAD.ai proprietary technology. 1. Accuracy: This xMADified model is the best quantized version of the google/gemma-2-9b-it model (8 GB only). See Table 1 below for model quality benchmarks. 2. Memory-efficiency: The full-precision model is around 18.5 GB, while this xMADified model is only around 8 GB, making it feasible to run on a 12 GB GPU. 3. Fine-tuning: These models are fine-tunable over the same reduced (12 GB GPU) hardware in mere 3-clicks. Watch our product demo here Loading the model checkpoint of this xMADified model requires around 8 GB of VRAM. Hence it can be efficiently run on a 12 GB GPU. 1. Run the following *commands to install the required packages. If you found this model useful, please cite our research paper. For additional xMADified models, access to fine-tuning, and general questions, please contact us at support@xmad.ai and join our waiting list.
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: xmadai
Теги: gemma2, conversational, endpoints_compatible, 4-bit, gptq
Лайков: 4 | Загрузок: 12
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
