8-bit base mixed-precision quantization of deepreinforce-ai/Ornith-1.0-35B for Apple Silicon, using the Unsloth Dynamic per-tensor bit allocation (without imatrix AWQ) via mlx-node. Ornith-1.0 is a self-improving family of open-source agentic coding models. The 35B member is a Qwen3.5-VL-MoE (hybrid Gated-DeltaNet + full attention, 256 experts, vision-language) post-train. Benchmarked on a cool Apple M5 Max: median decode throughput over three 512-token generations, with a 60-second idle GPU cooldown after every generation. (Sustained decode on Apple Silicon is thermally sensitive — back-to-back benchmarking on a hot chip can understate throughput by 20–30%, so every model here was measured from a comparable cool start.) Steady-state decode: 91.5 tok/s (1.5x vs BF16) on Apple M5 Max. Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only 8 of 256 experts per token (~3B active out of 35.9B total), so the active-weight footprint streamed per token is what matters. Decoded-text quality was verified against the BF16 reference with a multi-judge review of the actual generated output (not a…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Brooooooklyn
Теги: mlx-node, qwen3_5_moe, mlx, quantized, qwen3.5, moe, hybrid-attention, gated-delta-net
Лайков: 4 | Загрузок: 204
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
