116 lines
4.6 KiB
Markdown
116 lines
4.6 KiB
Markdown
# msModelSlim Quantization
|
|
|
|
## Overview
|
|
|
|
[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend compression
|
|
toolkit for producing pre-quantized model checkpoints. vLLM-Omni consumes its
|
|
output either through a compatible generic Ascend checkpoint path or through
|
|
the native MXFP8 and MXFP4 merge workflows documented in this guide.
|
|
|
|
msModelSlim is static quantization: quantized weights are generated offline
|
|
before vLLM-Omni inference starts.
|
|
|
|
The validation terms on this page follow the
|
|
[support levels](overview.md#support-levels). A producer recipe or hardware
|
|
backend alone is an integration point, not model-level validation.
|
|
|
|
## Hardware Support
|
|
|
|
| Device | Support |
|
|
|--------|---------|
|
|
| NVIDIA Blackwell GPU (SM 100+) | ❌ |
|
|
| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ |
|
|
| NVIDIA Ampere GPU (SM 80+) | ❌ |
|
|
| AMD ROCm | ❌ |
|
|
| Intel XPU | ❌ |
|
|
| Ascend NPU | ✅ |
|
|
|
|
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
|
|
guide.
|
|
|
|
## Model Type Support
|
|
|
|
### Diffusion Model (Qwen-Image, Wan2.2)
|
|
|
|
| Model | Base model | Scope | Level | Notes |
|
|
|-------|------------|-------|-------|-------|
|
|
| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Integrated | Upstream msModelSlim provides a producer recipe; use the [MXFP8](mxfp8.md) or [MXFP4](mxfp4.md) workflow for vLLM-Omni's native merged formats |
|
|
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Not validated | No supported checkpoint workflow is documented |
|
|
| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Integrated | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation and validate against BF16 |
|
|
|
|
Public Hugging Face quantized weights are not available yet. Use the
|
|
[HunyuanImage-3.0 msModelSlim adaptation](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8)
|
|
to generate the checkpoint manually.
|
|
|
|
msModelSlim is also the producer used by the native offline MXFP8 and MXFP4
|
|
DualScale workflows. Those workflows preprocess and annotate each transformer
|
|
checkpoint and are auto-detected at inference; follow their method pages rather
|
|
than passing `--quantization ascend`.
|
|
|
|
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
|
|
|
| Model | Scope | Status | Notes |
|
|
|-------|-------|--------|-------|
|
|
| Qwen3-Omni | Thinker or language-model stage | Not validated | No msModelSlim omni checkpoint path is documented |
|
|
| Qwen3-TTS | TTS language-model stage | Not validated | No msModelSlim TTS checkpoint path is documented |
|
|
|
|
### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
|
|
|
|
| Model | Scope | Status | Notes |
|
|
|-------|-------|--------|-------|
|
|
| BAGEL | Stage-specific diffusion or transformer weights | Not validated | Requires a model-specific Ascend adaptation |
|
|
| GLM-Image | Stage-specific diffusion or transformer weights | Not validated | Requires a model-specific Ascend adaptation |
|
|
|
|
## Configuration
|
|
|
|
For a compatible generic Ascend checkpoint, select the Ascend quantization
|
|
method explicitly:
|
|
|
|
Offline inference:
|
|
|
|
```bash
|
|
python text_to_image.py --model <compatible-ascend-checkpoint> --quantization ascend
|
|
```
|
|
|
|
Online serving:
|
|
|
|
```bash
|
|
vllm serve <compatible-ascend-checkpoint> --omni --quantization ascend
|
|
```
|
|
|
|
## Parameters
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `--quantization` | str | - | Use `ascend` only for a compatible generic Ascend-format checkpoint; omit it for native merged MXFP8/MXFP4 checkpoints |
|
|
| `model` | str | - | Path to the quantized checkpoint generated by Ascend tooling |
|
|
|
|
Do not pass `--quantization ascend` for a Wan checkpoint processed by the native
|
|
MXFP8 or MXFP4 merge tools. Their transformer metadata selects `mxfp8` or
|
|
`mxfp4_dualscale` automatically.
|
|
|
|
Example msModelSlim command for a Wan2.2 W8A8 checkpoint:
|
|
|
|
```bash
|
|
msmodelslim quant \
|
|
--model_path /path/to/wan2_2_t2v_float_weights \
|
|
--save_path /path/to/wan2_2_t2v_quantized_weights \
|
|
--device npu \
|
|
--model_type Wan2_2 \
|
|
--config_path /path/to/wan2_2_w8a8f8_mxfp_t2v.yaml \
|
|
--trust_remote_code True
|
|
```
|
|
|
|
For HunyuanImage-3.0, use the Hunyuan-specific adaptation linked above.
|
|
|
|
## Validation and Notes
|
|
|
|
1. Run with the Ascend/NPU installation and environment.
|
|
2. The `ascend` quantization method expects weights produced by the Ascend
|
|
tooling; it is not a load-time CUDA quantizer.
|
|
3. Keep the quantized checkpoint aligned with the same model architecture and
|
|
stage config used for BF16 inference.
|
|
4. No named full-model msModelSlim checkpoint is currently scheduled in NPU
|
|
CI. Treat new producer outputs as **Integrated** until an end-to-end BF16
|
|
comparison establishes **Validated** support.
|