[Docs] Complete quantization nav and online guide (#3764)

Signed-off-by: David Chen <530634352@qq.com>
This commit is contained in:
WeiQing Chen authored and GitHub committed 2026-05-20 15:52:40 +08:00
1 parent 2aafe0bf94
commit f74cba84ac
3 files changed
+30 -13

No files matched your search

+2 -2
View File
@@ -134,7 +134,7 @@ can be pinned to BF16 via the Python API:
```python
omni = Omni(
model="<your-model>",
quantization={
quantization_config={
"method": "mxfp4_dualscale",
"ignored_layers": ["blocks.10.attn1.to_q"], # explicit per-layer override
},
@@ -397,7 +397,7 @@ names** discovered in Step 1. No code changes to the model are required.
```python
omni = Omni(
model="/path/to/your-model",
quantization={
quantization_config={
"method": "mxfp4_dualscale",
"ignored_layers": [
"blocks.0.attn1.to_qkv", # runtime name, not diffusers name
+26 -11
View File
@@ -9,20 +9,24 @@ separate quantized checkpoint.
This mode is different from pre-quantized checkpoint formats such as GGUF,
AutoRound, msModelSlim, or serialized Int8 checkpoints. Those formats are
prepared before serving and are documented in their method-specific guides.
For MXFP8 and MXFP4, use this page for load-time quantization from BF16
checkpoints, and use the method-specific pages for offline checkpoints produced
by msModelSlim and the merge tools.
## Hardware Support
| Device | FP8 W8A8 | Int8 W8A8 |
|--------|----------|-----------|
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ |
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ |
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ |
| AMD ROCm | ⭕ | ⭕ |
| Intel XPU | ⭕ | ⭕ |
| Ascend NPU | ❌ | ✅ |
| Device | FP8 W8A8 | Int8 W8A8 | MXFP8 W8A8 | MXFP4 W4A4 |
|--------|----------|-----------|------------|------------|
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ⭕ | ⭕ |
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ⭕ | ⭕ |
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ⭕ | ⭕ |
| AMD ROCm | ⭕ | ⭕ | ⭕ | ⭕ |
| Intel XPU | ⭕ | ⭕ | ⭕ | ⭕ |
| Ascend NPU | ❌ | ✅ | ✅ | ✅ |
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
guide. FP8 on Ampere may use a weight-only path where available.
guide. FP8 on Ampere may use a weight-only path where available. MXFP8 and
MXFP4 are documented for the Ascend NPU path.
## Model Type Support
@@ -32,6 +36,8 @@ guide. FP8 on Ampere may use a weight-only path where available.
|--------|-------|----------------|--------|
| FP8 W8A8 | [FP8](fp8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
| Int8 W8A8 | [Int8](int8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
| MXFP8 W8A8 | [MXFP8](mxfp8.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, Wan2.2-TI2V-5B | Ascend NPU only |
| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Ascend NPU only; TI2V-5B is not supported |
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -53,6 +59,9 @@ from vllm_omni import Omni
omni_fp8 = Omni(model="<your-model>", quantization="fp8")
omni_int8 = Omni(model="<your-model>", quantization="int8")
omni_mxfp8 = Omni(model="<your-model>", quantization="mxfp8")
omni_mxfp4 = Omni(model="<your-model>", quantization="mxfp4")
omni_mxfp4_dualscale = Omni(model="<your-model>", quantization="mxfp4_dualscale")
```
CLI:
@@ -60,6 +69,9 @@ CLI:
```bash
vllm serve <your-model> --omni --quantization fp8
vllm serve <your-model> --omni --quantization int8
vllm serve <your-model> --omni --quantization mxfp8
vllm serve <your-model> --omni --quantization mxfp4
vllm serve <your-model> --omni --quantization mxfp4_dualscale
```
Per-component routing:
@@ -77,10 +89,11 @@ config = build_quant_config({
| Parameter | Methods | Description |
|-----------|---------|-------------|
| `method` | FP8, Int8 | Quantization method, usually `"fp8"` or `"int8"` |
| `ignored_layers` | FP8, Int8 | Layer name patterns to keep in BF16/FP16 |
| `method` | FP8, Int8, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` |
| `ignored_layers` | FP8, Int8, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 |
| `activation_scheme` | FP8, Int8 | The runtime value `"dynamic"` selects online activation scaling |
| `weight_block_size` | FP8 | Optional block-wise FP8 weight quantization size |
| `num_bf16_fallback_layers` | MXFP4 DualScale | Leading transformer blocks to keep in BF16 for online `mxfp4_dualscale`; defaults to `5` |
## Validation and Notes
@@ -91,3 +104,5 @@ config = build_quant_config({
model as supported.
4. If a model already ships quantized weights, use the matching pre-quantized
method guide instead of online quantization.
5. For Ascend MXFP4 deployments, prefer offline `mxfp4_dualscale` checkpoints
when production quality is more important than avoiding preprocessing.