vllm-omni/docs/user_guide/diffusion/quantization/msmodelslim.md

msModelSlim Quantization

Overview

msModelSlim is an Ascend-friendly compression tool focused on acceleration, using compression techniques, and built for Ascend hardware. It includes a series of inference optimization technologies such as quantization and compression, aiming to accelerate large language dense models, MoE models, multimodal understanding models, multimodal generation models, etc.

Once you have a quantized model which is generated by msModelSlim, you can use vLLM Omni for inference by specifying the --quantization ascend parameter to enable quantization features.

Supported Schemes

Scheme	Bits	Status
W8A8	8	✅ Supported
W4A4	4	Planned

W8A8 is the first supported scheme. Additional schemes will be added in future releases.

Model Quantization

The following example shows how to generate W8A8 quantized weights for the Wan2_2 model.

Quantization Script:

msmodelslim quant \
  --model_path /path/to/wan2_2_t2v_float_weights \
  --save_path /path/to/wan2_2_t2v_quantized_weights \
  --device npu \
  --model_type Wan2_2 \
  --config_path /lab_practice/wan2_2/wan2_2_w8a8f8_mxfp_t2v.yaml \
  --trust_remote_code True

After quantization completes, the output directory will contain the quantized model files.

For more examples, refer to the official examples.

Configuration

1. CLI: pass --quantization ascend.

# Offline inference
python text_to_image.py --model <your-model> --quantization ascend

# Online serving
vllm serve <your-model> --omni --quantization ascend

Supported Models

Model	HF Models	Recommendation	ignored_layers
HunyuanImage-3.0	-	All layers	None

Currently, quantized HunyuanImage-3.0 weights have not been uploaded to public model platforms such as Hugging Face. You can use a HunyuanImage-3.0-adapted msModelSlim version to generate the quantized weights manually. We will upload the quantized weights as soon as possible.