Quantized GGUF of inclusionAI/Ling-2.6-flash — a 104B parameter MoE model (7.4B active) with hybrid MLA/GLA architecture. This model requires a custom llama.cpp branch with Bailing Hybrid architecture support: https://github.com/ljubomirj/llama.cpp/tree/LJ-Ling-2.6-flash-r2 While the mtp works (llama-server accepts ‘—spec-type mtp’) atm it actually slows down the decode. So the speed test below are without mtp in. IDK why mtp does not help. INVALID LANGUAGE PAIR SPECIFIED. EXAMPLE: LANGPAIR=EN|IT USING 2 LETTER ISO OR RFC3066 LIKE ZH-CN. ALMOST ALL LANGUAGES SUPPORTED BUT SOME MAY HAVE NO CONTENT It was the primary reference for porting the Bailing Hybrid architecture to llama.cpp — covering MLA attention, GLA (Gated Linear Attention) with the recurrent state kernel, MoE expert routing, and the MTP speculative decoding head. The upstream model had an off-by-one bug in the GLA decay slope: (self.layeridx — 1) was used instead of self.layeridx in the layer-dependent decay scaling. This caused incorrect decay rates for GLA layers, with the most…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: ljupco
Теги: gguf, llama.cpp, moe, quantized, bailing, en, zh, endpoints_compatible
Лайков: 3 | Загрузок: 47
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
