This is a Mixture-of-Experts (MoE) decoder built directly from the dense Qwen 3.5 base model (empero-ai/Qwythos-9B-v2) and 5 dense Qwen 3.5 fine-tuned experts, without relying on CobrIXKit. The model features 33B total parameters, 13B active parameters (A13B) per token, and a native 1,048,576-token context window. Model Class: Qwen35MoEForCausalLM (decoder-only), modeltype=qwen35moe`. Context Window: Native support for up to 1,048,576 tokens (maxpositionembeddings=1048576`). Layer Design: Every transformer layer replaces the dense mlp` with a sparse MoE block: Routing: numlocalexperts=5, numexpertspertok=2 (top-2 routing with softmax over the 5 experts). Shared Expert: An always-active shared expert based on the base MLP, gated by sigmoid(x @ w), with numsharedexperts=1`. Initialization: Router initialization is random`. Shared-expert gates are initialized to zero. embedtokens linearattn. Layer normalization parameters norm rotaryemb lmhead No weight is modified, averaged, or interpolated during assembly. The MoE model is constructed by preserving the original base-model components while routing the expert MLP projections through the sparse expert system. These modules are…
Модальности:
Генерация текста
Области применения:
Диалог / чат Генерация кода
Задача: Генерация текста
Автор: CobrIX
Теги: llama.cpp, qwen35_moe, MoE, Qwen, CobrIX-1.0-Coder-Flash, custom-architecture, causal-lm, GGUF
Лайков: 3 | Загрузок: 46
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
