This repository contains a 4-bit GPTQ quantized version of google/medgemma-27b-text-it, optimized for high-throughput inference using vLLM and the Marlin kernel. MedGemma 27B is a text-only, instruction-tuned medical Large Language Model developed by Google and the Health AI Developer Foundation. It is trained on extensive medical datasets to excel in clinical reasoning, medical Q&A, and healthcare text comprehension. This model was quantized using GPTQModel with the following parameters to ensure maximum compatibility and speed on modern NVIDIA GPUs: Precision: 4-bit (INT4) weights, bfloat16 activations Group Size: 128 Desc Act (Activation Reordering): False (Specifically optimized for vLLM’s Marlin kernels to prevent latency penalties and maximize tokens-per-second) Calibration Dataset: Medical domain text and general knowledge prompts to preserve clinical reasoning capabilities. Thanks to 4-bit quantization, the VRAM footprint is drastically reduced compared to the original 16-bit model: Model Weights: ~15 GB VRAM Recommended GPU: Any NVIDIA GPU with 16GB+ VRAM (e.g., RTX 4080, RTX 4090, L4, L40S, A100) for comfortable context windows up to 8K tokens. Compute…
Модальности:
Генерация текста
Области применения:
Диалог / чат Медицина
Задача: Генерация текста
Автор: HarshBhanushali7705
Теги: gemma3_text, gptq, 4-bit, medical, gemma-3, medgemma, vllm, marlin
Лайков: 4 | Загрузок: 436
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
