Model details: — Parameter count: 9B (source checkpoint) — Input support: text — Speculative decoding: no — imatrix: yes — details Don’t know which to choose? Grab Q4KM (5.78GB) — usually a good mix of size and performance. Download instructions available here These quants run with llama.cpp — installable in one line via llama.app: llama-server includes a built-in chat web UI, served at http://localhost:8080 by default. These quants were made with llama.cpp release b11462 — if this model’s architecture is newly supported, you’ll need that release or newer to run them. They also work in: LM Studio · koboldcpp · ramalama · Jan AI · Text Generation Web UI · LoLLMs · Atomic Chat Some of these files were built with a layout computed for this model instead of llama.cpp’s standard one-size-fits-all rules. A Q4KM is still mostly Q4K; the extra precision goes to the weights this particular model is most sensitive to. The S, M or L in a name says how much of the model stays at the base precision: about 90 % for S, 70 % for M and 50 % for L. An L name is simply the large size of its family. Q4KL is to Q4KM what Q4KM is to Q4KS; Q6KS, Q6K and Q6KL are the small, medium and large sizes of Q6K,…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: bartowski
Теги: gguf, k2-horizon, 7b, dense, open-weights, ifm, en, endpoints_compatible
Лайков: 3 | Загрузок: 6,889
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
