Lythri/Lythri-7B-A4B-Q4_0_8-GGUF - Каталог нейросетей
Генерация текста

Lythri/Lythri-7B-A4B-Q4_0_8-GGUF

Добавлено:
Lythri/Lythri-7B-A4B-Q4_0_8-GGUF

This repository contains the Q48 quantization of Lythri-7B-A4B. Q48 is a mixed-precision quantization proposed in the Lythri technical report: critical tensors (attention projections, ffndown, PLE gates) are kept at Q8 while the remaining tensors use Q4. Both formats support native SIMD dot-product acceleration, delivering Q6K-level quality with 1.16–1.76× faster prefill on models ≥4B. For model details, evaluation results and limitations, see the original model card. Lythri models are highly sensitive to quantization. Standard 4-bit formats (Q4, Q4KM) cause noticeable quality loss, while Q6K preserves quality but is slower due to its multi-level scale structure. Q48 provides a middle ground: Search for Lythri/Lythri-7B-A4B-Q408-GGUF in the model browser and download. Q48 uses Q4 as the base type and overrides critical tensors to Q8, achieving Q6_K-level quality with 1.16–1.76× faster prefill via native SIMD dot-product acceleration. Lythri is built on Gemma 4 and is released under the Apache License 2.0.

Модальности:
Генерация текста

Области применения:
Диалог / чат


Задача: Генерация текста
Автор: Lythri
Теги: gguf, Q4_0_8, emotional-support, companion, on-device, conversational, gemma4, emotion
Лайков: 3  |  Загрузок: 480

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.