This repository contains GGUF conversions and quantizations of Tencent’s tencent/Hy-MT2-30B-A3B translation model for llama.cpp-compatible inference. SHA256 sidecar files are provided for the GGUF files when available. The Q2K, Q3KM, and Q5K_M files were quantized directly from the BF16 GGUF source, not requantized from another low-bit GGUF. Because Hy-MT2-30B-A3B uses the hyv3 architecture, conversion and quantization require llama.cpp tooling that supports hyv3. A generic llama.cpp quantizer may fail with unknown model architecture: ‘hyv3’`. Hy-MT2-30B-A3B uses the hyv3` architecture. It requires a llama.cpp build that supports this architecture. If your llama.cpp build does not support it, loading the GGUF may fail with: Use a compatible llama.cpp build/branch for Hy-V3/Hy-MT2 models. — —jinja is recommended so llama.cpp uses the chat template correctly. — -r » is recommended as a reverse prompt / stop marker because this model may emit the textual EOS marker if the runtime does not treat it as a native EOS token. — Adjust —n-gpu-layers according to your GPU memory. — For long context, make sure you have enough VRAM/RAM. KV-cache quantization may be useful on smaller GPUs if…
Модальности:
Генерация текста
Области применения:
Диалог / чат Мультиязычность Перевод
Задача: Генерация текста
Автор: litigerking
Теги: llama.cpp, gguf, hy_v3, hy-mt2, hy-v3, translation, moe, quantized
Лайков: 3 | Загрузок: 3,416
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
