mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit - Каталог нейросетей
Генерация текста

mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit

Добавлено:
mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit

> Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. All OptiQ quants · Docs NVIDIA’s Mamba-attention hybrid MoE, running locally on a Mac. This is an OptiQ mixed-precision quant of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. 22.8 GB on disk, down from 65.8 GB. Lightning is a nemotronh` hybrid: Mamba2 state-space blocks with a handful of attention layers, and a 128-expert MoE with 6 experts active per token. The routed experts carry most of the weight, so that is where the bits come from. All 164 quantizable projections were measured. Every 4-bit assignment in the model is an expert tensor: the 128-expert stack is where the size is, and the sweep found it is also where the precision is least needed. Following llama.cpp’s naming for mixed quants, the «4bit» label denotes the family, not the weighted average. Six-metric mean (the standard OptiQ text eval). Strong code generation; long-context multi-hop retrieval is the weak spot, which is what a hybrid with few attention layers would predict. It is a reasoning model and emits a block before its answer, so give it room and read the text after…

Модальности:
Генерация текста

Области применения:
Диалог / чат


Задача: Генерация текста
Автор: mlx-community
Теги: mlx, nemotron_h, optiq, quantized, 4bit, mixed-precision, nemotron, nemotron-h
Лайков: 4  |  Загрузок: 2,052

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.