A mixed-precision MLX build of Ornith-1.5-122B, converted on Apple Silicon (M3 Max, 64 GB) and runnable with stock mlxlm or served through oMLX. These are quantization variants, not fine-tunes** — no training of any kind was performed on these weights. Routed experts carry the compression: ffngateexps and ffnupexps at 2-bit, ffndownexps at 3-bit (all affine, group size 64). Attention, the GatedDeltaNet blocks, the shared expert, embeddings and head stay at 8-bit. Norms, conv1d, Alog and dtbias remain BF16. The MTP (multi-token prediction) head is included as a grafted 4-bit shard, so the model can run native speculative decoding. Produced by a streaming GGUF reader that dequantizes one layer at a time and re-quantizes with mx.quantize (affine, group size 64), so peak RAM stays near one layer and no 244 GB BF16 intermediate is ever materialized. Measured requantization error (dequantized MLX vs. the Q8_0 source, relative L2): Dequantization is lossless; requantization costs the full price of the target width on top of the source’s own error, so a port can never beat its source. The Q8_0 source sits at KL 0.0096 vs BF16 (96.28% top-1) — roughly two orders of magnitude below the…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: hermitdave
Теги: mlx, qwen3_5_moe, apple-silicon, quantized, mixed-precision, 2-bit, 3-bit, moe
Лайков: 3 | Загрузок: 735
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
