🚨 CRITICAL COMPATIBILITY WARNING 🚨 These are iqk format quantizations and are EXCLUSIVE to the ikllama.cpp fork. They will NOT work on mainline llama.cpp, standard LM Studio, standard Text Generation WebUI, or KoboldCPP. You must compile and run this using ikawrakow’s llama.cpp fork (or a UI where you have manually swapped the backend to an ikllama build). All variants share the same structure: high precision on attention/gating layers and shared experts, with the routed expert layers (the bulk of model size) quantized to varying levels. Balances upgraded routed experts with compressed embeddings to save VRAM. Spends a bit more VRAM for full Q80` precision on the vocabulary, with slightly heavier experts. Uses split quant levels on routed experts (down vs gate/up) with compressed embeddings. Steps up to 5-bit routed experts with full-precision SSM alpha/beta weights. Same structure as IQ5KS but using IQ5K for the routed experts. Highest quality routed expert quantization with full-precision SSM alpha/beta. Maximum compression variant. Drops attention layers to IQ6K` and uses aggressive 2-3 bit routed expert quantization. 1. Clone and build the ikllama.cpp fork from…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: KeinNiemand
Теги: gguf, quantization, iq4_ks, iq4_k, iq4_kss, iq5_k, iq5_ks, iq6_k
Лайков: 3 | Загрузок: 143
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
