This is quantized version of neuralmagic/Sparse-Llama-3.1-8B-2of4 created using llama.cpp — Model Architecture: Llama-3.1-8B — Input: Text — Output: Text — Model Optimizations: — Sparsity: 2:4 — Release Date: 11/20/2024 — Version: 1.0 — License(s): llama3.1 — Model Developers: Neural Magic This is the 2:4 sparse version of Llama-3.1-8B. On the OpenLLM benchmark (version 1), it achieves an average score of 62.16, compared to 63.19 for the dense model—demonstrating a 98.37% accuracy recovery. On the Mosaic Eval Gauntlet benchmark (version v0.3), it achieves an average score of 53.85, versus 55.34 for the dense model—representing a 97.3% accuracy recovery. This model was obtained by pruning all linear operators within transformer blocks to the 2:4 sparsity pattern: in each group of four weights, two are retained while two are pruned. In addition to pruning, the sparse model was trained with knowledge distillation for 13B tokens to recover the accuracy loss incurred by pruning. For pruning, we utilize optimized version of SparseGPT through LLM-Compressor, and for sparse training with knowledge distillation we utilize SquareHead approach. This model can be deployed efficiently using…
Модальности:
Генерация текста
Задача: Генерация текста
Автор: QuantFactory
Теги: gguf, vllm, sparsity, endpoints_compatible
Лайков: 4 | Загрузок: 391
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
