Original model: https://huggingface.co/Isotonic/OrcaAgent-llama3.2-8b Some of these quants (Q3KXL, Q4KL etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to. If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run: You can either specify a new local-dir (OrcaAgent-llama3.2-8b-Q8_0) or download them all in place (./) Click to view Q40XX information These are NOT* for Metal (Apple) or GPU (nvidia/AMD/intel) offloading, only ARM chips (and certain AVX2/AVX512 CPUs). If you’re using an ARM chip, the Q40XX quants will have a substantial speedup. Check out Q4044 speed comparisons on the original pull request To check which one would work best for your ARM chip, you can check AArch64 SoC features (thanks EloyOn!). If you’re using a CPU that supports AVX2 or AVX512 (typically server CPUs and AMD’s latest Zen5 CPUs) and are not offloading to a GPU, the Q408_8 may offer a nice speed as well: Q408_8 offers a nice bump to prompt processing and a small bump to text generation A great write up with charts showing various…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: bartowski
Теги: gguf, text-generation-inference, llama, trl, en, endpoints_compatible, conversational
Лайков: 4 | Загрузок: 825
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
