This model mlx-community/Kimi-K2.5-mlx-DQ3KM-q8 was converted to MLX format from moonshotai/Kimi-K2.5 using mlx-lm version 0.30.7. After the success of the first Kimi «DQ3KM» model, this is a new update for Kimi-K2.5! This is created for people using a single Apple Mac Studio M3 Ultra with 512 GB. The 4-bit version of Kimi K2 does not fit. Using research results, we aim to get 4-bit performance from a slightly smaller and smarter quantization. It should also not be so large that it leaves no memory for a useful context window. You can find more similar MLX model quants for Apple Mac Studio with 512 GB at https://huggingface.co/bibproj This time the recipe is a bit different from that of the first Kimi «DQ3KM» model, which was described there. To make to the quant perform better under stress, only the expert tensors are quantized to a mix of 3-bit and 4-bit. All the other tensors are kept at 8-bit. You could say that this quant has an 8-bit «brain» and 3-bit/4-bit experts. It is similar in size to the first quant. In the convert.py file of mlx-lm on your system ( you can see the original code here ), replace the code inside def mixedquantpredicate() with something like
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: mlx-community
Теги: mlx, kimi_k25, feature-extraction, conversational, custom_code, 4-bit
Лайков: 3 | Загрузок: 193
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
