GGUF files of Qwen2.5-1.5B-MoE-A0.9B, a Mixture-of-Experts conversion of Qwen2.5-1.5B with ~0.85B active parameters per token instead of 1.54B. They run in stock llama.cpp and anything built on it (LM Studio, Ollama, Jan, …). No patches are needed. In llama.cpp it generates text 32% faster than the dense Qwen2.5-1.5B on GPU and 41% faster on CPU, at the cost of about 40% higher perplexity. See the main model card for the architecture and full benchmark results. This is a base (completion) model. It is not instruction-tuned. Use BF16, not F16, as the full-precision format: some activations in this model exceed the F16 range. LM Studio: search for Qwen2.5-1.5B-MoE-A0.9B-GGUF and download the Q4KM file. llama.cpp llama-perplexity against the original transformers model on the same 40 × 512 tokens of WikiText-2: llama-bench, Q4KM for both models, tokens per second (RTX 4060 8 GB; Ryzen 5 7500F for CPU): The Q4KM file is larger than the dense model’s (1125 vs 935 MiB) because llama.cpp’s qwen2moe needs a separate output head rather than one tied to the embeddings. This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there’s a…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: bayliner1980
Теги: gguf, llama.cpp, moe, mixture-of-experts, qwen2_moe, en, endpoints_compatible, conversational
Лайков: 3 | Загрузок: 793
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
