SSD-backed Flash-MoE package for Qwen3.8-2.4T-A95B, exported from the Unsloth UD-Q1_0 GGUF. This is not a conventional standalone GGUF. The non-routed and shared tensors are stored in model-dense.gguf; routed expert tensors are stored separately in a layer-major sidecar/ and loaded into a per-layer resident slot bank on demand. Use the Flash-MoE-enabled Qwen3.8 branch of anemll-flash-llama.cpp. Stock llama.cpp builds that do not understand this package layout or its IQ1XXXS` routed experts cannot run it. — Layers 0-91: gate, up, and down routed tensors are IQ1XXXS (276 tensors). — Layer 92: gate, up, and down routed tensors are Q2K (3 tensors). — Each layer file contains the three whole routed tensors for that layer, with 512 experts packed within each tensor. The repository is about 397 GB. Put it on a fast local SSD and preserve the directory layout: Keep the sidecar on fast directly attached storage. This runtime uses SSD reads for cache misses; network filesystems and slow external links will sharply reduce decode speed. Build the Flash-MoE branch of anemll-flash-llama.cpp, then run from its source directory: — Keep —moe-topk 10: this is the model’s native routed width. — Do…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: anemll
Теги: llama.cpp, gguf, qwen3.8, moe, flash-moe, ssd, apple-silicon, endpoints_compatible
Лайков: 4 | Загрузок: 1,766
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
