From f74cba84ac31657751130d7f1c6f2d4d07e7d241 Mon Sep 17 00:00:00 2001 From: WeiQing Chen <40507679+david6666666@users.noreply.github.com> Date: Wed, 20 May 2026 15:52:40 +0800 Subject: [PATCH] [Docs] Complete quantization nav and online guide (#3764) Signed-off-by: David Chen <530634352@qq.com> --- docs/.nav.yml | 2 ++ docs/user_guide/quantization/mxfp4.md | 4 +-- docs/user_guide/quantization/online.md | 37 ++++++++++++++++++-------- 3 files changed, 30 insertions(+), 13 deletions(-) diff --git a/docs/.nav.yml b/docs/.nav.yml index a60a4368d..562a49e84 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -80,6 +80,8 @@ nav: - Quantized KV Cache: user_guide/quantization/quantized_kvcache.md - FP8 W8A8: user_guide/quantization/fp8.md - Int8 W8A8: user_guide/quantization/int8.md + - MXFP8 W8A8: user_guide/quantization/mxfp8.md + - MXFP4 W4A4: user_guide/quantization/mxfp4.md - ModelOpt: user_guide/quantization/modelopt.md - GGUF: user_guide/quantization/gguf.md - AutoRound: user_guide/quantization/autoround.md diff --git a/docs/user_guide/quantization/mxfp4.md b/docs/user_guide/quantization/mxfp4.md index 2fd4a8395..7463ada23 100644 --- a/docs/user_guide/quantization/mxfp4.md +++ b/docs/user_guide/quantization/mxfp4.md @@ -134,7 +134,7 @@ can be pinned to BF16 via the Python API: ```python omni = Omni( model="", - quantization={ + quantization_config={ "method": "mxfp4_dualscale", "ignored_layers": ["blocks.10.attn1.to_q"], # explicit per-layer override }, @@ -397,7 +397,7 @@ names** discovered in Step 1. No code changes to the model are required. ```python omni = Omni( model="/path/to/your-model", - quantization={ + quantization_config={ "method": "mxfp4_dualscale", "ignored_layers": [ "blocks.0.attn1.to_qkv", # runtime name, not diffusers name diff --git a/docs/user_guide/quantization/online.md b/docs/user_guide/quantization/online.md index 3dcb8297e..b9a6861d1 100644 --- a/docs/user_guide/quantization/online.md +++ b/docs/user_guide/quantization/online.md @@ -9,20 +9,24 @@ separate quantized checkpoint. This mode is different from pre-quantized checkpoint formats such as GGUF, AutoRound, msModelSlim, or serialized Int8 checkpoints. Those formats are prepared before serving and are documented in their method-specific guides. +For MXFP8 and MXFP4, use this page for load-time quantization from BF16 +checkpoints, and use the method-specific pages for offline checkpoints produced +by msModelSlim and the merge tools. ## Hardware Support -| Device | FP8 W8A8 | Int8 W8A8 | -|--------|----------|-----------| -| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | -| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | -| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | -| AMD ROCm | ⭕ | ⭕ | -| Intel XPU | ⭕ | ⭕ | -| Ascend NPU | ❌ | ✅ | +| Device | FP8 W8A8 | Int8 W8A8 | MXFP8 W8A8 | MXFP4 W4A4 | +|--------|----------|-----------|------------|------------| +| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ⭕ | ⭕ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ⭕ | ⭕ | +| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ⭕ | ⭕ | +| AMD ROCm | ⭕ | ⭕ | ⭕ | ⭕ | +| Intel XPU | ⭕ | ⭕ | ⭕ | ⭕ | +| Ascend NPU | ❌ | ✅ | ✅ | ✅ | Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this -guide. FP8 on Ampere may use a weight-only path where available. +guide. FP8 on Ampere may use a weight-only path where available. MXFP8 and +MXFP4 are documented for the Ascend NPU path. ## Model Type Support @@ -32,6 +36,8 @@ guide. FP8 on Ampere may use a weight-only path where available. |--------|-------|----------------|--------| | FP8 W8A8 | [FP8](fp8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models | | Int8 W8A8 | [Int8](int8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image | +| MXFP8 W8A8 | [MXFP8](mxfp8.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, Wan2.2-TI2V-5B | Ascend NPU only | +| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Ascend NPU only; TI2V-5B is not supported | ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) @@ -53,6 +59,9 @@ from vllm_omni import Omni omni_fp8 = Omni(model="", quantization="fp8") omni_int8 = Omni(model="", quantization="int8") +omni_mxfp8 = Omni(model="", quantization="mxfp8") +omni_mxfp4 = Omni(model="", quantization="mxfp4") +omni_mxfp4_dualscale = Omni(model="", quantization="mxfp4_dualscale") ``` CLI: @@ -60,6 +69,9 @@ CLI: ```bash vllm serve --omni --quantization fp8 vllm serve --omni --quantization int8 +vllm serve --omni --quantization mxfp8 +vllm serve --omni --quantization mxfp4 +vllm serve --omni --quantization mxfp4_dualscale ``` Per-component routing: @@ -77,10 +89,11 @@ config = build_quant_config({ | Parameter | Methods | Description | |-----------|---------|-------------| -| `method` | FP8, Int8 | Quantization method, usually `"fp8"` or `"int8"` | -| `ignored_layers` | FP8, Int8 | Layer name patterns to keep in BF16/FP16 | +| `method` | FP8, Int8, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` | +| `ignored_layers` | FP8, Int8, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 | | `activation_scheme` | FP8, Int8 | The runtime value `"dynamic"` selects online activation scaling | | `weight_block_size` | FP8 | Optional block-wise FP8 weight quantization size | +| `num_bf16_fallback_layers` | MXFP4 DualScale | Leading transformer blocks to keep in BF16 for online `mxfp4_dualscale`; defaults to `5` | ## Validation and Notes @@ -91,3 +104,5 @@ config = build_quant_config({ model as supported. 4. If a model already ships quantized weights, use the matching pre-quantized method guide instead of online quantization. +5. For Ascend MXFP4 deployments, prefer offline `mxfp4_dualscale` checkpoints + when production quality is more important than avoiding preprocessing.