This model was obtained by quantizing the weights of deepseek-ai/DeepSeek-R1 to INT4 data type. This optimization reduces the number of bits per parameter from 8 to 4, reducing the disk size and GPU memory requirements by approximately 50%. Only non-shared experts within transformer blocks are compressed. Weights are quantized using a symmetric per-group scheme, with group size 128. The GPTQ algorithm is applied for quantization. This model was evaluated on the OpenLLM v1 benchmarks and reasoning tasks (AIME-24, GPQA-Diamond, MATH-500). For reasoning tasks we estimate pass@1 based on 10 runs with different seeds and temperature=0.6, topp=0.95 and maxnewtokens=32768`. For reasoning evals we adopted the protocol from the open-r1 repository. Please use this version of vLLM: https://github.com/vllm-project/vllm/pull/16038 We follow the standard vLLM performance benchmarking with ShareGPT dataset and observe the following metrics (lower is better): GPTQ models are faster across all metrics than AWQ models because GPTQ uses less bits-per-parameter than AWQ. More specifically, AWQ has to use smaller group-size of 64 (vs 128 in GPTQ) to preserve accuracy, and zero-points due to asymmetric…
Модальности:
Генерация текста
Области применения:
Диалог / чат Логика и рассуждение
Задача: Генерация текста
Автор: ISTA-DASLab
Теги: deepseek_v3, conversational, custom_code, text-generation-inference, endpoints_compatible, compressed-tensors
Лайков: 4 | Загрузок: 19
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
