W8A8 (8-bit weights and activations) quantized version of Qwen/Qwen3-30B-A3B-Thinking-2507 using LLM-Compressor. — 🗜️ Memory: ~50% reduction vs FP16 — 🚀 Speed: Faster inference on supported hardware — 🔗 Original model: Qwen/Qwen3-30B-A3B-Thinking-2507 — 🏗️ Recommended architectures: Ampere and older $ For reproducibility, we report the win rates evaluated by GPT-4.1. & For highly challenging tasks (including PolyMATH and all reasoning and coding tasks), we use an output length of 81,920 tokens. For all other tasks, we set the output length to 32,768. The code of Qwen3-MoE has been in the latest Hugging Face transformers and we advise you to use the latest version of transformers. The following contains a code snippet illustrating how to use the model generate content based on given inputs. For deployment, you can use sglang>=0.4.6.post1 or vllm>=0.8.5 or to create an OpenAI-compatible API endpoint: — SGLang: For local use, applications such as Ollama, LMStudio, MLX-LM, llama.cpp, and KTransformers have also supported Qwen3. Qwen3 excels in tool calling capabilities. We recommend using Qwen-Agent to make the best use of agentic ability of Qwen3. Qwen-Agent encapsulates…
Модальности:
Генерация текста
Области применения:
Диалог / чат Математика
Задача: Генерация текста
Автор: ramblingpolymath
Теги: qwen3_moe, quantized, w8a8, llm-compressor, conversational, endpoints_compatible, 8-bit, compressed-tensors
Лайков: 3 | Загрузок: 15
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
