Hermes-style agentic fine-tune of Qwen3.6-27B, quantized to INT4 with a BF16 MTP overlay for speculative decoding. This model takes kai-os/Carnice-V2-27b (a Hermes-style agentic fine-tune of Qwen3.6-27B) and applies: 1. INT4 quantization via AutoRound (W4A16, groupsize=128, symmetric) — the quant grid comes from Lorbus’s Qwen3.6-27B-int4-AutoRound, delta-merged onto Carnice’s BF16 weights. This avoids re-running the full AutoRound calibration loop. 2. BF16 MTP overlay — all 29 MTP head tensors are kept in BF16 (unquantized) for clean speculative decoding acceptance. This recovers MTP AL from ~2.0 → ~3.0. 3. Patched chat template — the tool-call format is changed from Qwen3 XML to Hermes JSON (inside tags), compatible with vLLM’s —tool-call-parser hermes. Benchmarked on 2× RTX 3090 (PCIe, no NVLink) with vLLM dev205, TP=2: For comparison, the base Qwen3.6-27B INT4 (same hardware, same config) runs at ~69 narr / ~89 code TPS. Carnice is +4% on narrative, -9% on code — practically equivalent for everyday agentic use. Note for single RTX 3090: reduce —max-model-len to ~65K, set —tensor-parallel-size 1, —max-num-seqs 1. No NVLink required. PCIe-only works fine. Custom all-reduce…
Модальности:
Генерация текста
Области применения:
Диалог / чат Вызов функций (Tool use)
Задача: Генерация текста
Автор: wasifb
Теги: hermes, qwen3_5, qwen, qwen3-next, agentic, tool-use, MTP, spec-decode
Лайков: 4 | Загрузок: 0
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
