8-bit MLX conversion of meta-models/Muse-Glimmer-30B, built for mlx-serve — the native Zig MLX server for Apple Silicon (mlxserve.com). — Affine 8-bit, group size 64, on every matmul-read weight: text attention (incl. the sigmoid output gate projection), MLPs, lmhead, and the vision tower. — Kept bf16: embedtokens and the vision position embedding table (gather-read), the patch embedder (input dim not divisible by 64), all norms and biases. — Weights, config and tokenizer are otherwise verbatim from the base repo. Trunk 32.9 GB, plus the 2.7 GB drafter below — 35.7 GB total. drafter/ holds a DFlash block-drafter for this trunk, so speculative decoding works with no flags and no second download. mlx-serve probes that subdirectory when the model loads, which also means a hot model switch brings the drafter with it. —no-drafter opts out; an explicit —drafter still wins. It is a 5-layer assistant (blocksize 16, masktokenid` 201818, reading the trunk at layers 1/13/25/37/49), shipped pre-packed at 8-bit so the server serves it as-is. One assistant pass drafts a whole block and the trunk verifies it in a single forward. All numbers below are mlx-serve (mlxserve.com) on an M4 Max (128…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: ddalcu
Теги: mlx, muse_glimmer, mlx-serve, conversational
Лайков: 3 | Загрузок: 350
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
