alexcovo/qwen35-9b-mlx-turboquant-tq3

This repo packages the current best TurboQuant runtime recipe we have measured for mlx-community/Qwen3.5-9B-MLX-4bit: TurboQuant 3-bit KV-cache compression (TQ3) on Apple Silicon. Short version: — this is not a new checkpoint — it is a reproducible inference overlay on top of the base MLX model — it ships the usage path, benchmark evidence, and artifacts in one place This repo does: — point to the base model weights: mlx-community/Qwen3.5-9B-MLX-4bit — replace compatible runtime KVCache slots with TurboQuantKVCache(bits=3) — document the measured performance/memory tradeoff — provide a copy-paste runnable example This repo does not: — publish modified TurboQuant changes the inference-time cache representation, not the underlying model parameters. For the measured Qwen3.5-9B MLX workload, TQ3 is the current best operating point on the speed/compression frontier: — consistently memory-positive — near FP16 speed at short context — increasingly compelling as context length grows If you want the cleanest current recommendation rather than a…

Модальности:
Генерация текста

Области применения:
Диалог / чат

Задача: Генерация текста
Автор: alexcovo
Теги: mlx, qwen3_5, turboquant, kv-cache, qwen3, apple-silicon, conversational, 4-bit
Лайков: 4  |  Загрузок: 134

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.