A Q80 GGUF of the Gemma 4 26B A4B (QAT) Multi-Token-Prediction «assistant» head, for use as a self-speculative draft model in llama.cpp. Pairing this head with the full model lets the target model draft several tokens per step and verify them in a single forward pass — accelerating token generation with no change to output** (speculative decoding is lossless). This is not a standalone chat model. It only contains the MTP/nextn head and is meaningless on its own — load it as the —spec-draft-model alongside the full Gemma 4 26B A4B QAT model. Measured against the full target model unsloth/gemma-4-26B-A4B-it-qat-GGUF → UD-Q4KXL, with the exact llama-server configuration shown under Usage below, same build and same prompt, only the MTP head toggled. Setup: single RTX 5090; llama-server v129; the full config under Usage; native /completion, fixed 3,000-token prompt, npredict 512, ignoreeos, cacheprompt:false`, 3 reps + warmup. Speculative-decoding speedup is content-dependent (it scales with draft acceptance), so real-world gains vary — roughly +40–55% decode on varied prose here, higher on predictable/boilerplate output. Prefill is unaffected; MTP only accelerates generation. Output…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: superbonyx
Теги: gguf, llama.cpp, speculative-decoding, mtp, gemma4-assistant, draft-model, gemma, endpoints_compatible
Лайков: 3 | Загрузок: 272
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
