W4A16 (INT4 weights, BF16 activations) quantized version of OpenMOSE/Qwen3.5-REAP-262B-A17B, created using AutoRound. — Calibration dataset: NeelNanda/pile-10k (64 samples, seqlen=512) — Calibration mode: Text-only (vision encoder excluded from calibration) — Vision encoder: Preserved as clean BF16 (27-layer ViT, 1152 hidden, ~6.1GB) — sharedexpertgate layers: Kept at FP16 (60 router gate tensors) — MoE gate (router): Kept at BF16 (60 router weight tensors) — Runtime: ~4h12m on 8x NVIDIA H200 141GB — Peak VRAM: ~85GB across 8 GPUs — Peak RAM: ~804GB This model uses hybrid attention with GDN These layers produce intermediate values that exceed float16’s dynamic range (max 65504), causing NaN outputs. You must use —dtype bfloat16 (or torchdtype=torch.bfloat16). The model will produce garbage output (!!!!!!!) with float16. Tested with vLLM v0.17.0rc1 (vllm/vllm-openai:cu130-nightly). — GPU with at least 24 GB VRAM (with CPU offloading for the rest) — pip install conch-triton-kernels (required for vision encoder on Blackwell/Hopper GPUs) — A runtime patch for vLLM to handle per-expert weight format (see below) — If you don’t have…
Модальности:
Генерация текста Мультимодальность
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: atbender
Теги: qwen3_5_moe, image-text-to-text, quantized, autoround, w4a16, int4, moe, qwen3
Лайков: 4 | Загрузок: 42
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
