TurboQuant TQ3-compressed version of google/gemma-4-26B-A4B-it. Compression: 3-bit TurboQuant with group size 128. Zero calibration data. Weights are decompressed back to FP16 for maximum compatibility — any framework that loads HuggingFace models can use this checkpoint directly. For smaller GPUs, use the native TQ3 checkpoint instead (12 GB on disk, loads on L40S 48GB). Quality validated on 20 multi-turn conversation scenarios scored by Llama-3.3-70B in an LLM-as-a-judge setup. This checkpoint stores standard FP16 weights. It loads like any HuggingFace model but requires transformers >= 5.5.0 for Gemma 4 support. The weights already carry TQ3 quantization noise, so re-compressing them at runtime introduces near-zero additional error while reducing GPU memory from ~52 GB to ~12 GB. Requires turboquant-plus-vllm and an A100 80GB or larger GPU (the full checkpoint must fit in GPU memory during loading before compression). Use the native TQ3 checkpoint instead. It stores packed 3-bit indices directly (12 GB on disk). Tested on L40S 48GB and H100 80GB using a custom loader that keeps weights compressed in GPU memory. 1. Loaded original BF16 checkpoint to CPU 2. Applied TurboQuant…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: varjosoft
Теги: gemma4, image-text-to-text, turboquant, tq3, compressed, quantized, moe, conversational
Лайков: 3 | Загрузок: 22
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
