Qwen3.6-35B-A3B by Alibaba Qwen Team — a 35B-parameter Mixture-of-Experts (MoE) model with only 3B active parameters per token. Apache 2.0 licensed. Supports 131K+ context, tool use, and multilingual tasks across 100+ languages. Qwen3.6-35B-A3B ships with Multi-Token Prediction (MTP) heads for speculative decoding. However, on MoE models like this one, MTP heads waste ~1-2 GB of VRAM with no measurable speed benefit — the bottleneck is expert routing through system RAM (when using —cpu-moe), not draft token verification. This build strips the MTP heads with —no-mtp to reclaim that VRAM for the KV cache and expert weights, resulting in better performance on 16GB cards. — Format: NVIDIA FP4 (NVFP4) — 4-bit E2M1 values with E4M3 scaling per 16 values + FP32 tensor scale — BPW: 4.55 bits per weight — Size: 19.7 GB — Target hardware: NVIDIA Blackwell GPUs (RTX 50 series) — requires CUDA kernels at inference — llama.cpp build 700+ with CUDA support — NVIDIA GPU with 16GB+ VRAM (tested on RTX 5060 Ti 16GB) — —cpu-moe flag required — routes MoE expert weights to system RAM while keeping attention on GPU — —flash-attn on recommended for memory ef
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: FreedomAISVR
Теги: gguf, nvfp4, moe, qwen3.6, 35b, no-mtp, quantized, en
Лайков: 3 | Загрузок: 186
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.