This repository contains no weights. It documents running prism-ml/Ternary-Bonsai-2-27B-gguf on ROCm, which its model card does not cover — the backends listed there are CUDA, Metal, CPU. It does run on ROCm — well — but which file you pick decides whether it is fast or three times slower, and enabling MTP needs an 18-line source fix. Both are documented here with the numbers. llama-bench, quiet box, 3 reps. Both packings hold the same ternary weights — 1.75 vs 2.13 bpw is storage layout only, so quality is identical and the choice is purely speed. — ⛔ Vulkan is not viable for these kernels: 3.83 t/s on AMD RADV and 1.03 t/s on an Intel Arc 140T, against 23.78 on ROCm. Only ptq10.glsl / dequantptq10.comp exist. — ⛔ The upstream card states PQ20 is «faster at prompt processing everywhere». On gfx1151 it is inverted: −22% prefill, +18% decode. Using the grafted head from decent-jawfish/bonsai-2-27b-mtp (PQ20 + the Qwen3.8-27B MTP head, 866 tensors / blocks 0–64 / nextnpredictlayers = 1`). The MTP draft graph builds its own token-embedding lookup and skips the inverse that llmgraphcontext::buildinpembd() applies on the main path. Add, immediately after tokembd = ggmlgetrows(ctx0,…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: kingjones777
Теги: llama.cpp, gguf, ternary, rocm, amd, gfx1151, strix-halo, mtp
Лайков: 3 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
