Original model: https://huggingface.co/KingNish/Reasoning-0.5b Some of these quants (Q3KXL, Q4KL etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to. Some say that this improves the quality, others don’t notice any difference. If you use these models PLEASE COMMENT with your findings. I would like feedback that these are actually used and useful so I don’t keep uploading quants no one is using. If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run: You can either specify a new local-dir (Reasoning-0.5b-Q8_0) or download them all in place (./) If you’re using an ARM chip, the Q40XX quants will have a substantial speedup. Check out Q4044 speed comparisons on the original pull request To check which one would work best for your ARM chip, you can check AArch64 SoC features (thanks EloyOn!). A great write up with charts showing various performances is provided by Artefact2 here The first thing to figure out is how big a model you can run. To do this, you’ll need to figure out how much RAM and/or VRAM you have. If you…
Модальности:
Генерация текста
Области применения:
Логика и рассуждение Диалог / чат
Задача: Генерация текста
Автор: bartowski
Теги: gguf, text-generation-inference, unsloth, qwen2, trl, sft, reasoning, en
Лайков: 4 | Загрузок: 1,509
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
