A 13.5 GB mixed 4-bit GGUF of UkisAI’s Swift-Qwen3.8-27B, made for 16 GB GPUs. It keeps the MTP head and leaves room for a 64K q4_0 KV cache with MTP speculative decoding. It uses the same per-tensor recipe as jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller, so the two files compare directly: same size and tensor types, different weights. As of 2026-09-15, the published 4-bit Swift GGUFs are 15.3 GB or larger, which leaves too little of a 16 GB card for long context. Everything published between 12.5 and 14.8 GB is 3-bit (Q3KM, IQ3_XS and similar). This file keeps 4-bit attention, SSM and embedding tensors and takes the savings from the FFN. — Sampling follows UkisAI’s recommendation. The chat template accepts reasoningeffort low, medium or xhigh (default xhigh) and enablethinking. — MTP needs —parallel 1. — For more context, drop MTP: about 104K fits on the same card without —spec-type, at lower decode speed (measured on the jrell base quant, which has the identical layout). AMD RX 6900 XT (16 GB, ROCm), llama.cpp build 10712 (daef7b687), with a desktop session running. Every run compares this file with the jrell Qwen3.8 quant under identical settings on the card above. These are…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Языки программирования:
Swift
Задача: Генерация текста
Автор: ahmedelkilani01
Теги: gguf, llama.cpp, imatrix, mtp, qwen3.8, 16gb-vram, endpoints_compatible, conversational
Лайков: 3 | Загрузок: 203
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.