Weight-only int8 (W8A16) quantization of Qwen/Qwen3.6-27B, prepared and verified to run on Huawei Ascend 310P3 (Atlas 300I Duo) via vllm-ascend. > Weights are int8, activations stay FP16. This is not a speed optimization on 310P — > it is a memory optimization: halving the weights frees NPU memory for a much larger > KV cache, enabling up to 256K context on 4× 310P3 chips (TP=4). On Ascend 310P the stock paths have two problems for long context: — FP16 27B weights (~54 GB) leave little room for KV → context is memory-starved. — Activation-quant schemes (W8A8) crash on 310P (immature QuantBatchMatmulV3 kernel, L0C conflict). W8A16 (weight-only) sidesteps both: weights ~27 GB (×2 smaller → KV budget ~832K tokens → 256K context), and it does not touch the broken activation-quant kernel path. msmodelslim data-free RTN int8 (ascend modelslim ascendV1 format), via AutoModelForImageTextToText (keeps the multimodal wrapper; visual. left unquantized). See recipe/ for the conversion notes. On 310P, npuweightquantbatchmatmul expects weights in ND [K,N] layout, not FRACTALNZ (NZ → task not supported). Stock w8a16.py casts to NZ → fails. This repo ships a 310P scheme override…
Модальности:
Генерация текста
Области применения:
Диалог / чат
Задача: Генерация текста
Автор: adeepv
Теги: qwen3_5, ascend, 310p, vllm-ascend, w8a16, int8, quantized, conversational
Лайков: 3 | Загрузок: 73
Описание основано на материалах HuggingFace. Перевод выполнен автоматически.
