Summary. This is a 4-bit (Q4) MLX quantization of gpt-oss-20B (sparse Mixture-of-Experts, MPx4). Group size is 32. Built for Apple Silicon with Metal acceleration. — Base model: openai/gpt-oss-20b (Apache-2.0) — Quantization: MLX Q4, qgroupsize=32 (some tensors remain FP16 for stability) — Files: MLX weight shards + config.json; tokenizer files included for drop-in use — Footprint: ~13.11 GB on disk — Intended use: local inference / research on M-series Macs — Not intended for: safety-critical decisions; outputs may be inaccurate or biased Runs on: Apple Silicon (M1 or newer) with macOS ≥ 13.5 via MLX (Metal). Not supported: Intel macOS / Linux / Windows (use a GGUF build + llama.cpp instead). RAM guidance: 24 GB+ required for Q4 gs=32 (does not work on Mac with only 16 GB). LM Studio / CLI (MLX, Q4 gs=32) ≈2k-token responses: — M1 Max (32 GB): ~62–72 tok/s, 0.30–0.40 s TTFB — M4 Pro (24 GB): ~80-90 tok/s, 0.40–0.60 TTFB — M3 Ultra (256 GB): ~130–140 tok/s, 0.20–0.30 s TTFB Throughput varies with Mac model, context, and sampler settings. Perplexity (PPL) streaming evaluation on WikiText-2; window=stride=4096, ~100k tokens, EOS inserted between docs. VariantPPL (ctx=4096) MLX…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: halley-ai
Теги: mlx, gpt_oss, apple-silicon, metal, arm64, 4-bit, group-size-32, moe
Лайков: 3 | Загрузок: 261
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
