RedHatAI/Qwen2.5-1.5B-quantized.w8a8 - Каталог нейросетей
Генерация текста

RedHatAI/Qwen2.5-1.5B-quantized.w8a8

Добавлено:
RedHatAI/Qwen2.5-1.5B-quantized.w8a8

— Model Architecture: Qwen2 — Input: Text — Output: Text — Model Optimizations: — Activation quantization: INT8 — Weight quantization: INT8 — Intended Use Cases: Intended for commercial and research use multiple languages. Similarly to Qwen2.5-1.5B, this models is intended for assistant-like chat. — Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). — Release Date: 10/09/2024 — Version: 1.0 — License(s): apache-2.0 — Model Developers: Neural Magic Quantized version of Qwen2.5-1.5B. It achieves an average score of 58.34 on the OpenLLM benchmark (version 1), whereas the unquantized model achieves 58.48. This model was obtained by quantizing the weights of Qwen2.5-1.5B to INT8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric…

Модальности:
Генерация текста

Области применения:
Диалог / чат


Задача: Генерация текста
Автор: RedHatAI
Теги: qwen2, chat, neuralmagic, llmcompressor, conversational, en, 8-bit, compressed-tensors
Лайков: 4  |  Загрузок: 355,299

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.