> Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs A 4-bit mixed-precision MLX quant of mlx-community/NVIDIA-Nemotron-3-Nano-4B-BF16 produced by mlx-optiq, the sensitivity-aware quantization toolkit for Apple Silicon. +1.6 pp on GSM8K over stock uniform 4-bit at the same per-weight precision target. Nemotron 3 Nano is a hybrid Mamba2 + attention model. Of the 42 backbone blocks, 4 are full attention and the rest are Mamba2 SSM or MLP. OptiQ measures each linear’s KL-divergence sensitivity against a bf16 reference forward pass and assigns 4-bit or 8-bit per-layer based on that signal. Sensitive layers go to 8-bit; robust ones stay at 4-bit. We follow the same naming convention llama.cpp uses for Q4KM-style mixed-precision quants: the «4-bit» label is for the predominant precision, not the weighted average. Load it with mlx-lm (the custom NemotronH modeling files ship in the repo and are picked up automatically): For mixed-precision KV-cache serving and sensitivity-aware LoRA fine-tuning, install mlx-optiq: Six-metric Capability Score (mean of MMLU + GSM8K +…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: mlx-community
Теги: mlx, nemotron_h, quantized, mixed-precision, 4bit, 8bit, optiq, apple-silicon
Лайков: 3 | Загрузок: 359
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
