Q4KM quantization of badtheorylabs/BTL-4, a 35B-A3B MoE fine-tuned from deepreinforce-ai/Ornith-1.0-35B. Upstream ships only IQ2XXS` (2.30 bpw). This is the 4-bit build, quantized and measured on a 12 GB consumer GPU. tokenembd and output are held at q80; everything else follows the stock Q4KM mixture. 733 tensors, converted from the bf16 source. Decode was sampled 12 times across an 1,837-token generation: every sample fell between 52.0 and 52.4 tok/s. Throughput at 8K context was 52.4 tok/s, so the 170K window costs about 0.7%. Only 10 of 40 layers hold a growing KV cache and those use 2 KV heads, which is why 170K fits at all — roughly 10 KB/token at q80`. — Loads and generates correct code — Tool calling works — finishreason: toolcalls, correct function and arguments, through the model’s XML template with —jinja — Reasoning separation works with —reasoning-format deepseek — It is extremely verbose. An LRU cache class took 1,837 tokens for ~400 tokens of code. Reversing a linked list took 3,744 characters of reasoning for 315 characters of answer. At maxtokens: 400` it never finished thinking and returned empty content. Budget 2–4K minimum, more for hard problems. No MTP. The…
Модальности:
Генерация текста
Области применения:
Генерация кода Диалог / чат Вызов функций (Tool use)
Задача: Генерация текста
Автор: DogukanUrker
Теги: gguf, llama.cpp, quantized, qwen3_5_moe, moe, agentic, tool-use, code
Лайков: 3 | Загрузок: 26
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
