2-bit Hunyuan 3.0 (295B-A21B MoE) for Apple Silicon, packaged for SSD-streamed serving with MTPLX. Weights stream from SSD under a configurable memory envelope, so the model runs on Macs that cannot hold it resident. Measured envelopes: 64, 80, and 88 GiB of weight budget on a 128 GB M5 Max. Quantization: omlx oQe level-2 imatrix experts (2.44 bpw, 2.50 bpw effective with q8 residents), from mlx-community/Hy3-oQ2e. This is the MTPLX streaming layout: routed experts ship only in experts.bin, so nothing is downloaded twice. Download everything: Plain-MLX resident serving (128 GB Mac): use the upstream safetensors checkpoint at mlx-community/Hy3-oQ2e; this repo packages the same bytes for streaming. llama.cpp: not served. MLX affine quantization does not convert to GGML blocks in either direction. Real long-code prefill (exact-token gated), greedy, decode 256, bf16 KV, 3 reps. Cell = decode tok/s, mode of record = faster of AR / K1 (MTP speculative depth 1). Short-prompt arm (320-token code prompt): K1 47.7 / 22.3 / 10.8 on 88/80/64 — K1 wins every envelope there. Two regimes govern the mode choice. K1 wins at short context on every envelope, and at every context under full…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: OpensourceWTF
Теги: mlx, hy_v3, mtplx, moe, quantized, imatrix, 2-bit, streaming
Лайков: 3 | Загрузок: 94
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
