anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs: clarify quantization support matrix

Browse Source 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu committed 2026-08-13 15:50:56 +08:00
1 parent c98517ce7f
commit 00dab2ac95
12 files changed
+358 -160
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/quantization/msmodelslim.md
+31 -11
View File
@@ -3,12 +3,17 @@
## Overview
[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend compression
toolkit for producing pre-quantized model checkpoints. In vLLM-Omni, these
checkpoints run through the Ascend/NPU path with `--quantization ascend`.
toolkit for producing pre-quantized model checkpoints. vLLM-Omni consumes its
output either through a compatible generic Ascend checkpoint path or through
the native MXFP8 and MXFP4 merge workflows documented in this guide.
msModelSlim is static quantization: quantized weights are generated offline
before vLLM-Omni inference starts.
The validation terms on this page follow the
[support levels](overview.md#support-levels). A producer recipe or hardware
backend alone is an integration point, not model-level validation.
## Hardware Support
| Device | Support |
@@ -20,23 +25,28 @@ before vLLM-Omni inference starts.
| Intel XPU | ❌ |
| Ascend NPU | ✅ |
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
guide.
## Model Type Support
### Diffusion Model (Qwen-Image, Wan2.2)
| Model | Base model | Scope | Hardware | Notes |
|-------|------------|-------|----------|-------|
| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Ascend NPU | Upstream msModelSlim provides a Wan2.2 quantization recipe; vLLM-Omni inference validation is not listed |
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Ascend NPU | Not validated in this guide |
| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Ascend A2/A3 NPU | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation |
| Model | Base model | Scope | Level | Notes |
|-------|------------|-------|-------|-------|
| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Integrated | Upstream msModelSlim provides a producer recipe; use the [MXFP8](mxfp8.md) or [MXFP4](mxfp4.md) workflow for vLLM-Omni's native merged formats |
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Not validated | No supported checkpoint workflow is documented |
| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Integrated | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation and validate against BF16 |
Public Hugging Face quantized weights are not available yet. Use the
[HunyuanImage-3.0 msModelSlim adaptation](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8)
to generate the checkpoint manually.
msModelSlim is also the producer used by the native offline MXFP8 and MXFP4
DualScale workflows. Those workflows preprocess and annotate each transformer
checkpoint and are auto-detected at inference; follow their method pages rather
than passing `--quantization ascend`.
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
| Model | Scope | Status | Notes |
@@ -53,25 +63,32 @@ to generate the checkpoint manually.
## Configuration
For a compatible generic Ascend checkpoint, select the Ascend quantization
method explicitly:
Offline inference:
```bash
python text_to_image.py --model <quantized-model-path> --quantization ascend
python text_to_image.py --model <compatible-ascend-checkpoint> --quantization ascend
```
Online serving:
```bash
vllm serve <quantized-model-path> --omni --quantization ascend
vllm serve <compatible-ascend-checkpoint> --omni --quantization ascend
```
## Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--quantization` | str | - | Use `ascend` for msModelSlim-produced checkpoints |
| `--quantization` | str | - | Use `ascend` only for a compatible generic Ascend-format checkpoint; omit it for native merged MXFP8/MXFP4 checkpoints |
| `model` | str | - | Path to the quantized checkpoint generated by Ascend tooling |
Do not pass `--quantization ascend` for a Wan checkpoint processed by the native
MXFP8 or MXFP4 merge tools. Their transformer metadata selects `mxfp8` or
`mxfp4_dualscale` automatically.
Example msModelSlim command for a Wan2.2 W8A8 checkpoint:
```bash
@@ -93,3 +110,6 @@ For HunyuanImage-3.0, use the Hunyuan-specific adaptation linked above.
tooling; it is not a load-time CUDA quantizer.
3. Keep the quantized checkpoint aligned with the same model architecture and
stage config used for BF16 inference.
4. No named full-model msModelSlim checkpoint is currently scheduled in NPU
CI. Treat new producer outputs as **Integrated** until an end-to-end BF16
comparison establishes **Validated** support.