This is an FP8 quantized version of miromind-ai/MiroThinker-v1.0-30B, created using llmcompressor (Neural Magic). Key Benefits: — ~50% smaller model size (30GB vs 60GB) — ~2x faster inference on FP8-capable GPUs (Ada Lovelace, Hopper) — Native vLLM and SGLang support — Minimal quality loss with FP8 dynamic quantization The easiest way to run this model. No setup required — just Docker with NVIDIA runtime. For optimal performance in agentic tasks (from the original MiroThinker documentation): This is a Mixture of Experts (MoE) model based on Qwen3MoE architecture: KV cache calculated for GQA with 4 KV heads, 128 headdim, 48 layers, FP16 KV* The model weights alone require 29.2 GiB, leaving minimal headroom for KV cache on a 32GB GPU. Single RTX 5090 operation is technically possible but not recommended for production: — Requires —enforce-eager (disables CUDA graphs, significant performance penalty) — Maximum context: ~2048 tokens — Throughput: ~1-2 tokens/second (severely memory-bound) — No headroom for batched requests If you only have a single 32GB GPU, this configuration will work but with poor performance: Strongly recommended: Use TP=2 with two 24GB+ GPUs for usable…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: Doradus-AI
Теги: qwen3_moe, agent, open-source, miromind, deep-research, fp8, quantized, vllm
Лайков: 4 | Загрузок: 25
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
