56 lines
2.3 KiB
Markdown
56 lines
2.3 KiB
Markdown
|
# msModelSlim Quantization
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend-friendly compression tool focused on acceleration, using compression techniques, and built for Ascend hardware. It includes a series of inference optimization technologies such as quantization and compression, aiming to accelerate large language dense models, MoE models, multimodal understanding models, multimodal generation models, etc.
|
||
|
|
|
||
|
|
Once you have a quantized model which is generated by **msModelSlim**, you can use vLLM Omni for inference by specifying the --quantization ascend parameter to enable quantization features.
|
||
|
|
|
||
|
|
### Supported Schemes
|
||
|
|
|
||
|
|
| Scheme | Bits | Status |
|
||
|
|
|--------|------|--------|
|
||
|
|
| W8A8 | 8 | ✅ Supported |
|
||
|
|
| W4A4 | 4 | Planned |
|
||
|
|
|
||
|
|
W8A8 is the first supported scheme. Additional schemes will be added in future releases.
|
||
|
|
|
||
|
|
## Model Quantization
|
||
|
|
|
||
|
|
The following example shows how to generate W8A8 quantized weights for the [Wan2_2 model](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/Wan2_2/README.md).
|
||
|
|
|
||
|
|
**Quantization Script:**
|
||
|
|
|
||
|
|
```bash
|
||
|
|
msmodelslim quant \
|
||
|
|
--model_path /path/to/wan2_2_t2v_float_weights \
|
||
|
|
--save_path /path/to/wan2_2_t2v_quantized_weights \
|
||
|
|
--device npu \
|
||
|
|
--model_type Wan2_2 \
|
||
|
|
--config_path /lab_practice/wan2_2/wan2_2_w8a8f8_mxfp_t2v.yaml \
|
||
|
|
--trust_remote_code True
|
||
|
|
```
|
||
|
|
|
||
|
|
After quantization completes, the output directory will contain the quantized model files.
|
||
|
|
|
||
|
|
For more examples, refer to the [official examples](https://gitcode.com/Ascend/msit/tree/master/msmodelslim/example).
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
1. **CLI**: pass `--quantization ascend`.
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Offline inference
|
||
|
|
python text_to_image.py --model <your-model> --quantization ascend
|
||
|
|
|
||
|
|
# Online serving
|
||
|
|
vllm serve <your-model> --omni --quantization ascend
|
||
|
|
```
|
||
|
|
|
||
|
|
## Supported Models
|
||
|
|
|
||
|
|
| Model | HF Models | Recommendation | `ignored_layers` |
|
||
|
|
|-------|-----------|---------------|------------------|
|
||
|
|
| HunyuanImage-3.0 | - | All layers | None |
|
||
|
|
|
||
|
|
Currently, quantized HunyuanImage-3.0 weights have not been uploaded to public model platforms such as Hugging Face. You can use a [HunyuanImage-3.0-adapted msModelSlim version](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8) to generate the quantized weights manually. We will upload the quantized weights as soon as possible.
|