casperhansen/mpt-7b-8k-chat-awq

This model is originally released under CC-BY-NC-SA-4.0, and the AWQ framework is MIT licensed. Original model can be found at https://huggingface.co/mosaicml/mpt-7b-8k-chat. Machines rented from RunPod — speed may vary dependent on both GPU/CPU. H100: — CUDA 12.0, Driver 525.105.17: 92 tokens/s (10.82 ms/token) RTX 4090 + Intel i9 13900K (2 different VMs): — CUDA 12.0, Driver 525.125.06: 134 tokens/s (7.46 ms/token) — CUDA 12.0, Driver 525.125.06: 117 tokens/s (8.52 ms/token) RTX 4090 + AMD EPYC 7-Series (3 different VMs): — CUDA 12.2, Driver 535.54.03: 53 tokens/s (18.6 ms/token) — CUDA 12.2, Driver 535.54.03: 56 tokens/s (17.71 ms/token) — CUDA 12.0, Driver 525.125.06: 55 tokens/ (18.15 ms/token) A6000 (2 different

Модальности:
Генерация текста

Области применения:
Диалог / чат

Задача: Генерация текста
Автор: casperhansen
Теги: mpt, custom_code, text-generation-inference, endpoints_compatible, 4-bit, awq
Лайков: 3  |  Загрузок: 43

Открыть на HuggingFace →

Описание основано на материалах HuggingFace. Перевод выполнен автоматически.