Paper: Sparse Finetuning for Inference Acceleration of Large Language Models Code: https://github.com/neuralmagic/deepsparse/tree/main/research/mpt This model was produced from a MPT-7B base model finetuned on the GSM8k dataset with pruning applied using SparseGPT and retrain for 4 epochs with L2 distillation. Then it was exported for optimized inference with DeepSparse. GSM8k zero-shot accuracy with lm-evaluation-harness : 21.08% (FP32 baseline is 28.2%) All MPT model weights are available on SparseZoo and CPU speedup for generative inference can be reproduced by following the instructions at DeepSparse For general questions on these models and sparsification methods, reach out to the engineering team on our community Slack.
Модальности:
Генерация текста
Задача: Генерация текста
Автор: RedHatAI
Теги: onnx, mpt, deepsparse, custom_code
Лайков: 2 | Загрузок: 23
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.