DeepSeek-R1-Distill-Qwen-14B-FP8-dynamic
— Model Architecture: Qwen2ForCausalLM — Input: Text — Output: Text — Model Optimizations: — Weight quantization: FP8 —...
— Model Architecture: Qwen2ForCausalLM — Input: Text — Output: Text — Model Optimizations: — Weight quantization: FP8 —...
— Model Architecture: DeepSeek-R1-Distill-Llama-70B — Input: Text — Output: Text — Model Optimizations: — Weight quantization: FP8 —...
FP8-quantized version of DeepSeek-R1-Distill-Qwen-14B, optimized for inference with vLLM. The quantization reduces the model’s memory footprint by approximately...
Modified version of gemma-2-9b-it and update chattemplate for support system role to handle cases: — Conversation roles must...
Original model: https://huggingface.co/neuralmagic/Sparse-Llama-3.1-8B-2of4 Some of these quants (Q3KXL, Q4KL etc) are the standard quantization method with the embeddings...
— Model Architecture: Llama-3.1-8B — Input: Text — Output: Text — Model Optimizations: — Sparsity: 2:4 — Weight...
— Model Architecture: Llama-3.1-8B — Input: Text — Output: Text — Model Optimizations: — Sparsity: 2:4 — Weight...
— Model Architecture: Llama-3.1-Nemotron — Input: Text — Output: Text — Model Optimizations: — Weight quantization: FP8 —...
— Model Architecture: Meta-Llama-3.2 — Input: Text — Output: Text — Model Optimizations: — Weight quantization: FP8 —...
— Model Architecture: Meta-Llama-3.1 — Input: Text — Output: Text — Model Optimizations: — Weight quantization: FP8 —...