INVALID LANGUAGE PAIR SPECIFIED. EXAMPLE: LANGPAIR=EN|IT USING 2 LETTER ISO OR RFC3066 LIKE ZH-CN. ALMOST ALL LANGUAGES SUPPORTED BUT SOME MAY HAVE NO CONTENT The INT4 expert GEMM runs through vLLM’s built-in GPTQ-MoE path (MoeWNA16/Marlin) — no custom kernels; dense layers run as plain BF16. — Serving today requires the dsv41-quant-plugin (not yet upstreamed to vLLM) and DSV41ENGRAMDTYPE=int4 for the engram lookup. — Not compatible with the official reference implementation’s FP8 kernels (dense is BF16 here). — auto-round 0.15.0 (editable copy with DeepSeek-V4.1 source-format patch), vLLM main nightly (0.1.1.dev39+g46d2b23ac), transformers-side loading not used (model_free). — Full production log & design docs: see producer’s journal (aqa pipeline).
Модальности:
Генерация текста
Задача: Генерация текста
Автор: lvkaokao
Теги: deepseek_v41, auto-round, w4a16, int4, gptq, moe, deepseek, endpoints_compatible
Лайков: 3 | Загрузок: 241
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
