It’s a 4-bit AWQ quantization of DeepSeek-R1-0528 671B model, it’s suitable for use with GPU nodes like 8xA100/8xH20/8xH100 with vLLM and SGLang vllm serve adamo1139/DeepSeek-R1-0528-AWQ —tensor-parallel 8 If this doesn’t work for you, you may need to manually specify quantization and datatype with —quantization awqmarlin and —dtype float16` respectively. I used AutoAWQ 0.2.8, transformers 4.48.0 and torch 2.6.0. modelingdeepseek.py was slightly modified to get around an issue mentioned here. Quantization was done on 8x H100 80GB node with 960GB of RAM and 800GB of swap. I used Unsloth’s BF16 version as a starting point but I removed quantizationconfig section from the config.json before running AWQ quantization script. Third attempt was successful, the other two failed due to memory overflow after 15+ hours of runtime each. Final attempt took about 18 hours to complete. I think I’ll make some evals to measure quantization’s impact on downstream performance, I’m not set on it fully yet. It’s the full-fat 671B model, if you don’t have access to the extreme hardware needed to run it, look into running Qwen3 8B based distilled version instead. The DeepSeek R1 model has undergone a…
Модальности:
Генерация текста
Области применения:
Диалог / чат Логика и рассуждение
Задача: Генерация текста
Автор: adamo1139
Теги: deepseek_v3, conversational, custom_code, text-generation-inference, endpoints_compatible, 4-bit, awq
Лайков: 4 | Загрузок: 274
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
