This quant collection REQUIRES ikllama.cpp fork to support advanced non-linear SotA quants and Multi-Head Latent Attention (MLA). Do not download these big files and expect them to run on mainline vanilla llama.cpp, ollama, LM Studio, KoboldCpp, etc! Though might* work in Nexesenex’s croco.cpp kobold fork (untested). NOTE ikllama.cpp` can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants. — type f32: 401 tensors — type q4K: 1 tensors tokenembd — type q6K: 1 tensors output («head») — type iq4nl: 80 tensors down — type iq3k: 320 tensors (q|o) (gate|up) — type iq4k: 160 tensors (k|v) High-end Gaming Rig Hardware AMD 9950X Overclocked infinity fabric «gear 1» clocks 2x 48GB DDR5@6400 RAM (~87GB/s benchmarked) 3090 TI FE 24GB VRAM @ 450 Watts (uncapped) PP ~500 tok/sec with 2k batches * TG ~5 tok/sec limited by RAM i/o bandwidth I tested 1. Why is it smol? I ran out of names making a bunch of similar sized quants for the Perplexity graph above lol. 2. Will you make…
Модальности:
Генерация текста
Области применения:
Генерация кода Диалог / чат
Задача: Генерация текста
Автор: ubergarm
Теги: gguf, code, imatrix, ik_llama.cpp, endpoints_compatible, conversational
Лайков: 3 | Загрузок: 107
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
