[NPU] [Quant] Support HunyuanImage3 offline quantization with vLLM-Ascend on diffusion path (#2979)
Signed-off-by: betta18 <jiangmengyu1@huawei.com> Signed-off-by: jiangmengyu18 <56633611+jiangmengyu18@users.noreply.github.com> Co-authored-by: betta18 <jiangmengyu1@huawei.com>
This commit is contained in:
1 parent
e375b12680
commit
c174c95ab0
11 files changed
+147
-18
No files matched your search
@@ -0,0 +1,56 @@
|
||||
# msModelSlim Quantization
|
||||
|
||||
## Overview
|
||||
|
||||
[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend-friendly compression tool focused on acceleration, using compression techniques, and built for Ascend hardware. It includes a series of inference optimization technologies such as quantization and compression, aiming to accelerate large language dense models, MoE models, multimodal understanding models, multimodal generation models, etc.
|
||||
|
||||
Once you have a quantized model which is generated by **msModelSlim**, you can use vLLM Omni for inference by specifying the --quantization ascend parameter to enable quantization features.
|
||||
|
||||
### Supported Schemes
|
||||
|
||||
| Scheme | Bits | Status |
|
||||
|--------|------|--------|
|
||||
| W8A8 | 8 | ✅ Supported |
|
||||
| W4A4 | 4 | Planned |
|
||||
|
||||
W8A8 is the first supported scheme. Additional schemes will be added in future releases.
|
||||
|
||||
## Model Quantization
|
||||
|
||||
The following example shows how to generate W8A8 quantized weights for the [Wan2_2 model](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/Wan2_2/README.md).
|
||||
|
||||
**Quantization Script:**
|
||||
|
||||
```bash
|
||||
msmodelslim quant \
|
||||
--model_path /path/to/wan2_2_t2v_float_weights \
|
||||
--save_path /path/to/wan2_2_t2v_quantized_weights \
|
||||
--device npu \
|
||||
--model_type Wan2_2 \
|
||||
--config_path /lab_practice/wan2_2/wan2_2_w8a8f8_mxfp_t2v.yaml \
|
||||
--trust_remote_code True
|
||||
```
|
||||
|
||||
After quantization completes, the output directory will contain the quantized model files.
|
||||
|
||||
For more examples, refer to the [official examples](https://gitcode.com/Ascend/msit/tree/master/msmodelslim/example).
|
||||
|
||||
## Configuration
|
||||
|
||||
1. **CLI**: pass `--quantization ascend`.
|
||||
|
||||
```bash
|
||||
# Offline inference
|
||||
python text_to_image.py --model <your-model> --quantization ascend
|
||||
|
||||
# Online serving
|
||||
vllm serve <your-model> --omni --quantization ascend
|
||||
```
|
||||
|
||||
## Supported Models
|
||||
|
||||
| Model | HF Models | Recommendation | `ignored_layers` |
|
||||
|-------|-----------|---------------|------------------|
|
||||
| HunyuanImage-3.0 | - | All layers | None |
|
||||
|
||||
Currently, quantized HunyuanImage-3.0 weights have not been uploaded to public model platforms such as Hugging Face. You can use a [HunyuanImage-3.0-adapted msModelSlim version](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8) to generate the quantized weights manually. We will upload the quantized weights as soon as possible.
|
||||
Reference in new issue
Block a user