ct2fast-CodeLlama-34b-hf
Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU. Checkpoint compatible...
Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU. Checkpoint compatible...
Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU. Checkpoint compatible...
Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU. Checkpoint compatible...
This is an experimental hybrid GLM-5.2 checkpoint for vLLM/B12X serving. — the dense, attention, shared-expert, special-head, and MTP...
This repo quantizes the model using a data-free quantization tool. This release is prepared for vLLM with compressed-tensors...
Weight-only int8 (W8A16) quantization of Qwen/Qwen3.6-27B, prepared and verified to run on Huawei Ascend 310P3 (Atlas 300I Duo)...
Эта версия Qwen3-4B была преобразована для запуска на Axera NPU с использованием квантования w8a16. Для тех, кто заинтересован...
NONET is a family of offline, quantized large language models fine-tuned for question answering with direct, concise answers....
Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU. Checkpoint compatible...
Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU. Checkpoint compatible...