# msModelSlim Quantization

## Overview

[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend-friendly compression tool focused on acceleration, using compression techniques, and built for Ascend hardware. It includes a series of inference optimization technologies such as quantization and compression, aiming to accelerate large language dense models, MoE models, multimodal understanding models, multimodal generation models, etc.

Once you have a quantized model which is generated by **msModelSlim**, you can use vLLM Omni for inference by specifying the --quantization ascend parameter to enable quantization features.

### Supported Schemes

| Scheme | Bits | Status |
|--------|------|--------|
| W8A8 | 8 | ✅ Supported |
| W4A4 | 4 | Planned |

W8A8 is the first supported scheme. Additional schemes will be added in future releases.

## Model Quantization

The following example shows how to generate W8A8 quantized weights for the [Wan2_2 model](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/Wan2_2/README.md).

**Quantization Script:**

```bash
msmodelslim quant \
  --model_path /path/to/wan2_2_t2v_float_weights \
  --save_path /path/to/wan2_2_t2v_quantized_weights \
  --device npu \
  --model_type Wan2_2 \
  --config_path /lab_practice/wan2_2/wan2_2_w8a8f8_mxfp_t2v.yaml \
  --trust_remote_code True
```

After quantization completes, the output directory will contain the quantized model files.

For more examples, refer to the [official examples](https://gitcode.com/Ascend/msit/tree/master/msmodelslim/example).

## Configuration

1. **CLI**: pass `--quantization ascend`.

```bash
# Offline inference
python text_to_image.py --model <your-model> --quantization ascend

# Online serving
vllm serve <your-model> --omni --quantization ascend
```

## Supported Models

| Model | HF Models | Recommendation | `ignored_layers` |
|-------|-----------|---------------|------------------|
| HunyuanImage-3.0 | - | All layers | None |

Currently, quantized HunyuanImage-3.0 weights have not been uploaded to public model platforms such as Hugging Face. You can use a [HunyuanImage-3.0-adapted msModelSlim version](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8) to generate the quantized weights manually. We will upload the quantized weights as soon as possible.