docs: clarify quantization support matrix

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-13 15:50:56 +08:00
12 changed files with 358 additions and 160 deletions
@@ -14,7 +14,10 @@ A: At first, you can check current [issues](https://github.com/vllm-project/vllm
 > Q: Does vLLM-Omni support AWQ or any other quantization?
-A: We plan to introduce GGUF FP8 prequantized models and online FP8 quantization in version 0.16.0. Support for other quantization types will follow in future releases. For details, please see our [Q1 quantization roadmap](https://github.com/vllm-project/vllm-omni/issues/1057).
+A: AWQ is not currently listed as a validated vLLM-Omni path. Quantization
 support depends on the model, pipeline stage, checkpoint format, and hardware
 backend. See the current [quantization support matrix](../user_guide/quantization/overview.md)
 for online methods, pre-quantized checkpoints, and model-level validation.
 > Q: Does vLLM-Omni support multimodal streaming input and output?
@@ -92,7 +92,7 @@ current support.
 |--------|--------------|-------------|----------|
 | **[FP8](quantization/fp8.md)** | `quantization="fp8"` | FP8 W8A8 on validated transformer stages | Memory reduction, inference speedup |
 | **[INT8](quantization/int8.md)** | `quantization="int8"` | INT8 W8A8 on validated transformer stages | Memory reduction, broad GPU compatibility |
-| **[GGUF](quantization/gguf.md)** | `quantization="gguf"` | Native GGUF transformer-only weights (Q4, Q8, etc.) | Memory reduction on consumer GPUs |
+| **[BitsAndBytes W4](quantization/bitsandbytes.md)** | `quantization="bitsandbytes"` | Online NF4/FP4 weight-only quantization on validated CUDA transformer stages | Reducing transformer weight memory without preparing a separate checkpoint |
 ## Supported Models
@@ -143,7 +143,8 @@ python text_to_video.py \
 - `--enable-layerwise-offload`: enable layerwise offloading on DiT modules.
 - `--frame-rate`: generation FPS for pipelines that require it (e.g., LTX2).
 - `--audio-sample-rate`: fallback audio sample rate when the pipeline returns audio.
- `--quantization`: quantization method (`fp8` for FP8, `gguf` for GGUF).
+- `--quantization`: online quantization method, for example `fp8`; supported
  pre-quantized checkpoints are normally auto-detected from checkpoint metadata.
 - `--flow-shift`: scheduler flow_shift parameter.
 - `--lora-path`: path to PEFT LoRA adapter folder or checkpoint file.
 - `--lora-scale`: scale factor for LoRA weights.
@@ -90,8 +90,8 @@ vllm serve "Qwen/Qwen-Image" \
    --quantization-config '{"method": "fp8"}'
 ```
-Other vLLM-Omni quantization methods, such as `gguf`, `modelopt`, `mxfp4`,
+Other vLLM-Omni quantization methods, such as `modelopt`, `mxfp4`, `mxfp8`,
-`mxfp8`, serialized checkpoints, static FP8 configs, and layer-name skip lists
+serialized checkpoints, static FP8 configs, and layer-name skip lists
 such as `ignored_layers`, are intentionally not translated by this compatibility
 shortcut. Use Diffusers-native configuration through `diffusers_load_kwargs` or
 a native vLLM-Omni pipeline for those cases.
@@ -10,6 +10,10 @@ checkpoint's `config.json` and auto-detects
 AutoRound is static quantization: no `--quantization` flag is needed at
 inference time when the checkpoint already contains the quantization config.
 The validation terms on this page follow the
 [support levels](overview.md#support-levels) defined in the overview. A listed
 checkpoint is not enough by itself to establish end-to-end model support.
 ## Hardware Support
 | Device | Support |
@@ -21,46 +25,57 @@ inference time when the checkpoint already contains the quantization config.
 | Intel XPU | ✅ |
 | Ascend NPU | ❌ |
-Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
+Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
-guide. AutoRound is Intel-supported.
+guide. This table is method-level backend availability. AutoRound is an Intel
 project; vLLM selects a compatible CUDA or XPU compute backend at load time.
 ## Model Type Support
 ### Diffusion Model (Qwen-Image, Wan2.2)
-| Model | Checkpoint | Scope | Scheme | Backend |
+| Model | Checkpoint | Scope | Scheme | Validation |
-|-------|------------|-------|--------|---------|
+|-------|------------|-------|--------|------------|
-| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
+| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | Validated |
-| Qwen-Image | Not listed | Diffusion transformer | W4A16 | Not validated |
+| Qwen-Image | `INC4AI/Qwen-Image-AutoRound-W4A16` | Diffusion transformer | W4A16 | Validated with end-to-end generation and quality comparison |
-| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
+| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
-| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
+| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
-| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
+| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | Validated; not in the scheduled Wan AutoRound job |
 CUDA execution uses GPTQ-Marlin or another compatible vLLM AutoRound backend;
 Intel XPU uses the Intel-supported backend. The text encoder and VAE stay in
 BF16 for all entries above.
 ### World Model (Cosmos3)
-| Model | Checkpoint | Scope | Scheme | Backend |
+| Model | Checkpoint | Scope | Scheme | Validation |
-|-------|------------|-------|--------|---------|
+|-------|------------|-------|--------|------------|
-| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
+| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | Integrated; no model-specific end-to-end test is maintained in-tree |
-| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
+| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | Validated with a manual serving run |
 The Cosmos VAE and guardrail components stay in BF16.
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
-| Model | Checkpoint | Scope | Scheme | Backend |
+| Model | Checkpoint | Scope | Scheme | Validation |
-|-------|------------|-------|--------|---------|
+|-------|------------|-------|--------|------------|
-| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Language-model stage | W4A16 | AutoRound |
+| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
-| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | AutoRound |
+| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
 | Qwen3-TTS | Not listed | TTS language-model stage | W4A16 | Not validated |
 AutoRound support is checkpoint-driven. A model is supported when its
 checkpoint uses a compatible INC/AutoRound config and the target stage maps to
 vLLM-Omni's runtime module names.
 For Qwen-Omni, only the thinker language model uses the AutoRound checkpoint.
 Audio and vision encoders, the talker, and waveform-decoder stages stay in
 BF16.
 ### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
-| Model | Scope | Status | Notes |
+| Model | Checkpoint | Scope | Validation |
-|-------|-------|--------|-------|
+|-------|------------|-------|------------|
-| GLM-Image | Diffusion transformer  | ✅ | `Intel/GLM-Image-int4-AutoRound` |
+| GLM-Image | `Intel/GLM-Image-int4-AutoRound` | Diffusion transformer | CI-backed |
-| BAGEL | Checkpoint-defined diffusion or transformer stage | Not validated | Requires a compatible AutoRound checkpoint |
+| BAGEL | Not listed | Checkpoint-defined diffusion or transformer stage | Not validated |
 ## Configuration
@@ -119,6 +134,14 @@ The checkpoint should contain a config like:
 At load time, vLLM-Omni builds an `OmniINCConfig`, remaps checkpoint block names
 to runtime module names, and selects the matching vLLM compute backend.
 Checkpoint auto-detection and successful layer construction establish the
 **Integrated** level. Promote a model to **Validated** only after the named
 checkpoint completes an end-to-end generation or multimodal request and its
 output passes a model-appropriate sanity or quality check. Compare against the
 BF16 baseline for quality-sensitive image and video paths. Promote it to
 **CI-backed** only when that named checkpoint is selected by scheduled hardware
 CI.
 Example checkpoint creation:
 ```bash
@@ -9,6 +9,10 @@ stay in BF16/FP16. No pre-quantized checkpoint is required.
 This is an online (dynamic) path: load the normal HuggingFace checkpoint and
 quantize during model loading via the `bitsandbytes` CUDA kernels.
 The validation terms on this page follow the
 [support levels](overview.md#support-levels). BitsAndBytes kernel support does
 not imply that every diffusion transformer has been quality-validated.
 ## Hardware Support
 | Device | Support |
@@ -21,7 +25,7 @@ quantize during model loading via the `bitsandbytes` CUDA kernels.
 | Intel XPU | ❌ |
 | Ascend NPU | ❌ |
-Legend: `✅` supported, `❌` unsupported. Non-CUDA platforms raise in
+Legend: `✅` backend available, `❌` unsupported. Non-CUDA platforms raise in
 `get_quant_method()`; CUDA requires compute capability 7.5+ (`SM 75+`).
 Requires the optional `bitsandbytes` package (`pip install bitsandbytes`).
@@ -30,11 +34,11 @@ Requires the optional `bitsandbytes` package (`pip install bitsandbytes`).
 ### Diffusion Model (Qwen-Image, Wan2.2)
-| Model | HF models | CUDA | Mode | Recommendation |
+| Model | HF models | Scope | Level | Recommendation |
-|-------|-----------|:----:|------|----------------|
+|-------|-----------|-------|-------|----------------|
-| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Yes | Online W4 weight-only | All heavy linear layers; sensitive embedders stay BF16 |
+| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Diffusion transformer | Validated | All heavy linear layers; sensitive embedders stay BF16 |
-| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Not validated | Online W4 weight-only | Compare vs BF16 before enabling |
+| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Diffusion transformer | Not validated | Compare with BF16 before enabling |
-| Wan2.2 | Wan2.2 diffusion pipelines | Not validated | Online W4 weight-only | Validate before enabling in docs |
+| Wan2.2 | Wan2.2 diffusion pipelines | One or both diffusion transformers | Not validated | Validate quality and cascade behavior before enabling |
 Other diffusion models may work if their transformer uses supported linear
 layers, but they are not validated in this guide.
@@ -104,6 +108,10 @@ Multi-GPU tensor parallelism (`tensor_parallel_size` > 1) is not validated for
 BitsAndBytes in diffusion models; each rank quantizes its own weight shard
 independently.
 Z-Image-Turbo is validated outside scheduled full-model BitsAndBytes CI. The
 real CUDA kernel smoke tests cover quantization and matrix multiplication, but
 do not establish model-level quality for unlisted architectures.
 If quality regresses, use `ignored_layers` to keep sensitive projections in BF16
 (for example `to_out` or `w2`).
@@ -13,6 +13,11 @@ components, such as the tokenizer, scheduler, text encoder, vision/audio
 encoder, and VAE, on the base checkpoint unless a model-specific guide says
 otherwise.
 The validation terms on this page follow the
 [support levels](overview.md#support-levels) defined in the overview. ModelOpt
 format detection and checkpoint adaptation are shared runtime integrations;
 each model and stage still requires its own end-to-end validation.
 !!! note
    ModelOpt checkpoints are pre-quantized checkpoints. Do not pass
    `--quantization fp8` for these checkpoints. The checkpoint
@@ -58,11 +63,11 @@ stay unquantized unless the model-specific recipe validates otherwise.
 | Intel XPU | ⭕ |
 | Ascend NPU | ❌ |
-Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
+Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
-The optional CUTLASS FP8 runtime override requires CUDA SM89+. ModelOpt NVFP4
+guide. The optional CUTLASS FP8 runtime override requires CUDA SM89+. ModelOpt
-and mixed FP8/NVFP4 diffusion checkpoints are currently validated on Blackwell
+NVFP4 and mixed FP8/NVFP4 diffusion checkpoints are currently validated on
-CUDA systems in the recipes below; other CUDA generations require separate
+Blackwell CUDA systems in the recipes below; other CUDA generations require
-backend and quality validation.
+separate backend and quality validation.
 ## Model Type Support
@@ -79,6 +84,9 @@ backend and quality validation.
 | HunyuanImage-3.0 | `feizhai123/hunyuan-image3-modelopt-mixed-experts-nvfp4-dense-fp8` | MoE diffusion transformer | Validated for ModelOpt mixed FP8/NVFP4 checkpoints |
 | Wan2.2 | Not available | Diffusion transformer | Not validated |
 The validated diffusion entries above have named checkpoint and recipe
 evidence. They are not all exercised by scheduled full-model CI.
 For full serving commands and benchmark context, see
 [`recipes/Qwen/Qwen-Image.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen-Image.md)
 and
@@ -88,8 +96,8 @@ and
 | Model | Scope | Status |
 |-------|-------|--------|
-| Qwen3-Omni | Thinker language-model stage | ModelOpt FP8 checkpoint path |
+| Qwen3-Omni | Thinker language-model stage (FP8) | Validated outside scheduled CI |
-| Qwen3-Omni | Thinker language-model stage (W4A4 NVFP4) | Validated; see [Qwen3-Omni NVFP4 W4A4](#qwen3-omni-nvfp4-w4a4-thinker) below |
+| Qwen3-Omni | Thinker language-model stage (W4A4 NVFP4) | CI-backed for the H100 Marlin fallback; native Blackwell path is recipe-validated |
 | Qwen3-TTS | TTS language-model stage | Not validated |
 Audio encoder, vision encoder, talker, and code2wav stages stay in BF16 unless
@@ -102,9 +110,9 @@ Qwen3-Omni-30B-A3B-Instruct thinker language model. The thinker text body
 (attention + MoE experts) is quantized to NVFP4 with FP8 per-tensor input
 scales; the audio encoder, vision encoder, talker, and code2wav stay in BF16.
-| Variant | HF checkpoint | Hardware |
+| Variant | HF checkpoint | Hardware and validation |
-|---------|---------------|----------|
+|---------|---------------|-------------------------|
-| W4A4 NVFP4 (full thinker) | `YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip` | sm_100+ (Blackwell, FlashInfer FP4 GEMM) |
+| W4A4 NVFP4 (full thinker) | `YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip` | sm_100+ native FlashInfer FP4 GEMM; H100 Marlin fallback is CI-backed |
 Calibration uses ModelOpt `mtq.NVFP4_DEFAULT_CFG` with the `awq_clip` algorithm
 on 1024 ultrachat samples chat-templated through the Qwen3-Omni tokenizer.
@@ -232,3 +240,6 @@ omni = Omni(
 5. Keep `--quantization fp8` for online FP8 from BF16 checkpoints; use this
   ModelOpt path only when the checkpoint already contains ModelOpt quantized
   weights and scales.
 6. The Qwen3-Omni full-model CI smoke exercises the H100 Marlin fallback. A
   scheduled Blackwell test is still required before treating the native
   FlashInfer FP4 kernel path as CI-backed.
@@ -3,12 +3,17 @@
 ## Overview
 [msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend compression
-toolkit for producing pre-quantized model checkpoints. In vLLM-Omni, these
+toolkit for producing pre-quantized model checkpoints. vLLM-Omni consumes its
-checkpoints run through the Ascend/NPU path with `--quantization ascend`.
+output either through a compatible generic Ascend checkpoint path or through
 the native MXFP8 and MXFP4 merge workflows documented in this guide.
 msModelSlim is static quantization: quantized weights are generated offline
 before vLLM-Omni inference starts.
 The validation terms on this page follow the
 [support levels](overview.md#support-levels). A producer recipe or hardware
 backend alone is an integration point, not model-level validation.
 ## Hardware Support
 | Device | Support |
@@ -20,23 +25,28 @@ before vLLM-Omni inference starts.
 | Intel XPU | ❌ |
 | Ascend NPU | ✅ |
-Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
+Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
 guide.
 ## Model Type Support
 ### Diffusion Model (Qwen-Image, Wan2.2)
-| Model | Base model | Scope | Hardware | Notes |
+| Model | Base model | Scope | Level | Notes |
-|-------|------------|-------|----------|-------|
+|-------|------------|-------|-------|-------|
-| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Ascend NPU | Upstream msModelSlim provides a Wan2.2 quantization recipe; vLLM-Omni inference validation is not listed |
+| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Integrated | Upstream msModelSlim provides a producer recipe; use the [MXFP8](mxfp8.md) or [MXFP4](mxfp4.md) workflow for vLLM-Omni's native merged formats |
-| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Ascend NPU | Not validated in this guide |
+| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Not validated | No supported checkpoint workflow is documented |
-| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Ascend A2/A3 NPU | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation |
+| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Integrated | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation and validate against BF16 |
 Public Hugging Face quantized weights are not available yet. Use the
 [HunyuanImage-3.0 msModelSlim adaptation](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8)
 to generate the checkpoint manually.
 msModelSlim is also the producer used by the native offline MXFP8 and MXFP4
 DualScale workflows. Those workflows preprocess and annotate each transformer
 checkpoint and are auto-detected at inference; follow their method pages rather
 than passing `--quantization ascend`.
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
 | Model | Scope | Status | Notes |
@@ -53,25 +63,32 @@ to generate the checkpoint manually.
 ## Configuration
 For a compatible generic Ascend checkpoint, select the Ascend quantization
 method explicitly:
 Offline inference:
 ```bash
-python text_to_image.py --model <quantized-model-path> --quantization ascend
+python text_to_image.py --model <compatible-ascend-checkpoint> --quantization ascend
 ```
 Online serving:
 ```bash
-vllm serve <quantized-model-path> --omni --quantization ascend
+vllm serve <compatible-ascend-checkpoint> --omni --quantization ascend
 ```
 ## Parameters
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
-| `--quantization` | str | - | Use `ascend` for msModelSlim-produced checkpoints |
+| `--quantization` | str | - | Use `ascend` only for a compatible generic Ascend-format checkpoint; omit it for native merged MXFP8/MXFP4 checkpoints |
 | `model` | str | - | Path to the quantized checkpoint generated by Ascend tooling |
 Do not pass `--quantization ascend` for a Wan checkpoint processed by the native
 MXFP8 or MXFP4 merge tools. Their transformer metadata selects `mxfp8` or
 `mxfp4_dualscale` automatically.
 Example msModelSlim command for a Wan2.2 W8A8 checkpoint:
 ```bash
@@ -93,3 +110,6 @@ For HunyuanImage-3.0, use the Hunyuan-specific adaptation linked above.
   tooling; it is not a load-time CUDA quantizer.
 3. Keep the quantized checkpoint aligned with the same model architecture and
   stage config used for BF16 inference.
 4. No named full-model msModelSlim checkpoint is currently scheduled in NPU
   CI. Treat new producer outputs as **Integrated** until an end-to-end BF16
   comparison establishes **Validated** support.
@@ -13,6 +13,10 @@ vLLM-Omni provides two quantization methods with different scale structures:
 | `mxfp4` | Single-scale (per-32 fine only) | Online only | Quick accuracy baseline; no checkpoint prep needed |
 | `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | Production; better accuracy; offline recommended |
 The validation terms on this page follow the
 [support levels](overview.md#support-levels). Current model guidance covers
 Wan diffusion transformers only; the text encoder and VAE stay in BF16.
 !!! tip "Recommended: `mxfp4_dualscale` offline"
    For production deployments, use the `mxfp4_dualscale` offline mode with a
    pre-quantized checkpoint produced by msModelSlim. Offline checkpoints load
@@ -34,26 +38,45 @@ vLLM-Omni provides two quantization methods with different scale structures:
 ## Hardware Support
-| Device | Support |
+| Device | Online `mxfp4` | Online `mxfp4_dualscale` | Offline `mxfp4_dualscale` |
-|--------|---------|
+|--------|-----------------|---------------------------|----------------------------|
-| NVIDIA Blackwell GPU (SM 100+) | ⭕ |
+| NVIDIA Blackwell GPU (SM 100+) | ❌ | ❌ | ❌ |
-| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ |
+| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ | ❌ | ❌ |
-| NVIDIA Ampere GPU (SM 80+) | ⭕ |
+| NVIDIA Ampere GPU (SM 80+) | ❌ | ❌ | ❌ |
-| AMD ROCm (gfx950 / MI355X) | ✅ |
+| AMD ROCm (gfx950 / MI355X) | ✅ | ❌ | ❌ |
-| Intel XPU | ⭕ |
+| Intel XPU | ❌ | ❌ | ❌ |
-| Ascend NPU (Atlas 950 A5) | ✅ |
+| Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ✅ |
-Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
+Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
 guide. Backend availability does not imply full-model checkpoint validation.
 !!! note "Why NVIDIA is marked unsupported"
    This table describes the methods currently wired into vLLM-Omni, not every
    low-precision format NVIDIA hardware could execute. `DiffusionMXFP4Config`
    has runtime implementations only for Ascend and ROCm gfx950; on CUDA it
    raises `NotImplementedError` because no CUDA MXFP4 linear method or
    checkpoint loader is registered.
    On NVIDIA, use [ModelOpt](modelopt.md) NVFP4 for the documented 4-bit
    checkpoint path. NVFP4 and OCP MXFP4 have different checkpoint metadata,
    scale layouts, and kernel contracts, so they are not interchangeable. A
    native CUDA MXFP4 path would require a CUDA linear implementation, loader
    contract, and model-level output validation.
 The ROCm path dispatches online single-scale `mxfp4` to AITER on gfx950. It has
 kernel and dispatch coverage, but this guide does not yet claim end-to-end
 quality validation for a named Wan checkpoint on ROCm. DualScale and
 pre-quantized MXFP4 checkpoints are not supported on ROCm.
 ## Model Type Support
 ### Diffusion Model (Wan2.2)
-| Model | Online | Offline | Notes |
+| Model | Online (Ascend) | Offline (Ascend) | Validation and notes |
-|-------|--------|---------|-------|
+|-------|-----------------|------------------|----------------------|
-| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade (`transformer` + `transformer_2`); both transformers quantized with the same config |
+| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | Validated outside scheduled full-model offline NPU CI; both cascade transformers are quantized |
-| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade; same scheme as T2V-A14B |
+| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | Validated outside scheduled full-model offline NPU CI; same cascade scheme as T2V-A14B |
-| Wan2.2-TI2V-5B | ❌ | ❌ | Parameter count too small; W4A4 causes unacceptable accuracy loss |
+| Wan2.2-TI2V-5B | ❌ | ❌ | Unsupported because W4A4 causes unacceptable accuracy loss |
 The choice between `mxfp4` and `mxfp4_dualscale` in **online mode** is about
 quantization quality, not model compatibility — both work on cascade (A14B) and
@@ -83,12 +106,13 @@ by the merge script); there is no offline `mxfp4` single-scale format.
 ### `mxfp4` — Single-Scale Online Mode
-Online mode requires no pre-processing. vLLM-Omni quantizes BF16 weights to
+Online mode requires no pre-processing. On Ascend, vLLM-Omni quantizes BF16
-MXFP4 at load time using `npu_dynamic_mx_quant`. A single block scale
+weights at load time using `npu_dynamic_mx_quant`. On ROCm gfx950, the same
 `mxfp4` config dispatches to the AITER online kernel. A single block scale
 (`float8_e8m0fnu`, one per 32 K elements) is computed on the fly; no
-calibration `mul_scale` is available. Applies equally to single-transformer
+calibration `mul_scale` is available. The config applies equally to
-and cascade (A14B) models — both transformers in a cascade receive the same
+single-transformer and cascade (A14B) models, but named Wan model validation is
-quantization config automatically.
+currently limited to the documented Ascend path.
 ```python
 from vllm_omni import Omni
@@ -111,10 +135,10 @@ vllm serve <your-model> --omni --quantization mxfp4
 ### `mxfp4_dualscale` — DualScale Online Mode
-Online DualScale mode computes both fine and coarse scales on the fly from BF16
+Ascend-only online DualScale mode computes both fine and coarse scales on the
-weights using `npu_dynamic_dual_level_mx_quant`. Applies equally to
+fly from BF16 weights using `npu_dynamic_dual_level_mx_quant`. It applies
-single-transformer and cascade (A14B) models. Compared to `mxfp4` online,
+equally to single-transformer and cascade (A14B) models. Compared to `mxfp4`
-DualScale provides better quantization accuracy at higher compute cost.
+online, DualScale provides better quantization accuracy at higher compute cost.
 The default configuration keeps the leading 5 transformer blocks in BF16
 (`num_bf16_fallback_layers=5`). Accuracy evaluation on Wan2.2-A14B shows this
@@ -341,6 +365,11 @@ unless they appear in `ignored_layers`.
   quantization. Use `ignored_layers` and `num_bf16_fallback_layers` to trade
   off compression vs. accuracy for precision-sensitive layers.
 8. The native methods, merge mappings, and per-transformer cascade metadata
   have unit coverage. The current Buildkite configuration does not schedule a
   named full-model offline MXFP4 checkpoint on NPU, so this path is
   **Validated**, not **CI-backed**.
 ## Adapting MXFP4 for a New Model
 This section is aimed at developers who want to add MXFP4 support to a model
@@ -15,30 +15,45 @@ This method supports three modes:
 | **Offline (Native)** | msModelSlim-exported MXFP8 weights converted to diffusers format via `merge_mxfp8_checkpoint.py` — weights and scales are loaded directly from the preprocessed checkpoint |
 | **Offline (AutoRound)** | AutoRound MXFP8 checkpoints with `data_type="mx_fp"` — auto-detected from `config.json` |
 The validation terms on this page follow the
 [support levels](overview.md#support-levels). Current model guidance covers
 Wan diffusion transformers only; the text encoder and VAE stay in BF16.
 ## Hardware Support
 | Device | Online | Offline (Native) | Offline (AutoRound) |
 |--------|--------|------------------|---------------------|
-| NVIDIA Blackwell GPU (SM 100+) | ⭕ | ⭕ | ⭕ |
+| NVIDIA Blackwell GPU (SM 100+) | ❌ | ❌ | ⭕ |
-| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ | ⭕ | ⭕ |
+| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ | ❌ | ⭕ |
-| NVIDIA Ampere GPU (SM 80+) | ⭕ | ⭕ | ⭕ |
+| NVIDIA Ampere GPU (SM 80+) | ❌ | ❌ | ⭕ |
-| AMD ROCm | ⭕ | ⭕ | ⭕ |
+| AMD ROCm | ❌ | ❌ | ⭕ |
 | Intel XPU | ✅ | ❌ | ✅ |
 | Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ⭕ |
-Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
+Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
 guide. Backend availability does not imply full-model checkpoint validation.
-**Note**: Intel XPU only supports AutoRound MXFP8 for offline mode. Use AutoRound quantized checkpoints or online mode for XPU.
+The native `DiffusionMXFP8Config` currently dispatches only to Intel XPU and
 Ascend NPU implementations. CUDA reaches `NotImplementedError`, so the NVIDIA
 rows are marked unsupported even though this is an implementation gap rather
 than a general claim about NVIDIA hardware capabilities. On NVIDIA, use
 [FP8](fp8.md) for load-time quantization or [ModelOpt](modelopt.md) FP8 for
 pre-quantized checkpoints. Adding native CUDA MXFP8 requires a registered CUDA
 linear method, a compatible checkpoint contract, and model-level validation.
 Native `mxfp8` is available online on Intel XPU and online or offline on
 Ascend. Intel XPU only supports AutoRound MXFP8 for offline mode. The AutoRound
 column is a separate checkpoint format and compute path.
 ## Model Type Support
 ### Diffusion Model (Wan2.2)
-| Model | Mode | Notes |
+| Model | Mode | Scope | Validation |
-|-------|------|-------|
+|-------|------|-------|------------|
-| Wan2.2-T2V-A14B | Online + Offline | MoE cascade; quantizes two transformers (`transformer` + `transformer_2`) |
+| Wan2.2-T2V-A14B | Online + Offline | MoE cascade: `transformer` + `transformer_2` | Validated outside scheduled full-model offline NPU CI |
-| Wan2.2-I2V-A14B | Online + Offline | MoE cascade; quantizes two transformers (`transformer` + `transformer_2`) |
+| Wan2.2-I2V-A14B | Online + Offline | MoE cascade: `transformer` + `transformer_2` | Validated outside scheduled full-model offline NPU CI |
-| Wan2.2-TI2V-5B | Online + Offline | Single transformer |
+| Wan2.2-TI2V-5B | Online + Offline | Single diffusion transformer | Validated outside scheduled full-model offline NPU CI |
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -218,3 +233,7 @@ omni = Omni(model="<autoround-mxfp8-model>")
   `--output-dir`), regenerate it with the current script. The old script
   wrote a separate `quantization_config.json` that is not read by vLLM-Omni;
   the current script injects the config directly into `transformer/config.json`.
 4. The native loader, merge mappings, and Wan cascade config propagation have
   unit coverage. The current Buildkite configuration does not schedule a
   named full-model offline MXFP8 checkpoint on NPU, so this path is
   **Validated**, not **CI-backed**.
@@ -6,7 +6,7 @@ Online quantization means vLLM-Omni computes quantized weights and scales while
 loading the model. Use it when you want memory savings without preparing a
 separate quantized checkpoint.
-This mode is different from pre-quantized checkpoint formats such as GGUF,
+This mode is different from pre-quantized checkpoint formats such as ModelOpt,
 AutoRound, msModelSlim, or serialized Int8 checkpoints. Those formats are
 prepared before serving and are documented in their method-specific guides.
 For MXFP8 and MXFP4, use this page for load-time quantization from BF16
@@ -15,18 +15,22 @@ by msModelSlim and the merge tools.
 ## Hardware Support
-| Device | FP8 W8A8 | Int8 W8A8 | MXFP8 W8A8 | MXFP4 W4A4 |
+| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | MXFP8 W8A8 | MXFP4 W4A4 |
-|--------|----------|-----------|------------|------------|
+|--------|----------|-----------|------------------|------------|------------|
-| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ⭕ | ⭕ |
+| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ❌ | ❌ |
-| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ⭕ | ⭕ |
+| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ❌ | ❌ |
-| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ⭕ | ⭕ |
+| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ❌ | ❌ |
-| AMD ROCm | ⭕ | ⭕ | ⭕ | ⭕ |
+| AMD ROCm (gfx950) | ⭕ | ⭕ | ❌ | ❌ | ✅ |
-| Intel XPU | ⭕ | ⭕ | ✅ | ⭕ |
+| Intel XPU | ⭕ | ⭕ | ❌ | ✅ | ❌ |
-| Ascend NPU | ❌ | ✅ | ✅ | ✅ |
+| Ascend NPU | ❌ | ✅ | ❌ | ✅ | ✅ |
-Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
+Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
-guide. FP8 on Ampere may use a weight-only path where available. MXFP8 and
+guide. These entries describe method-level backend availability, not validation
-MXFP4 are documented for the Ascend NPU path.
+for every model. FP8 on Ampere may use a weight-only path where available.
 On ROCm gfx950, MXFP4 means online single-scale `mxfp4`; online and offline
 `mxfp4_dualscale` are Ascend-only. The native MXFP8/MXFP4 configs do not
 currently register CUDA implementations; use FP8 or ModelOpt FP8/NVFP4 on
 NVIDIA instead.
 ## Model Type Support
@@ -36,8 +40,9 @@ MXFP4 are documented for the Ascend NPU path.
 |--------|-------|----------------|--------|
 | FP8 W8A8 | [FP8](fp8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
 | Int8 W8A8 | [Int8](int8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
 | BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Z-Image-Turbo; Qwen-Image and Wan2.2 are not validated | Validated for Z-Image-Turbo |
 | MXFP8 W8A8 | [MXFP8](mxfp8.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, Wan2.2-TI2V-5B | Validated on Ascend NPU and Intel XPU |
-| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Ascend NPU only; TI2V-5B is not supported |
+| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Validated on Ascend NPU; the ROCm gfx950 backend is integrated but not model-validated; TI2V-5B is unsupported |
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -59,6 +64,7 @@ from vllm_omni import Omni
 omni_fp8 = Omni(model="<your-model>", quantization="fp8")
 omni_int8 = Omni(model="<your-model>", quantization="int8")
 omni_bnb = Omni(model="Tongyi-MAI/Z-Image-Turbo", quantization="bitsandbytes")
 omni_mxfp8 = Omni(model="<your-model>", quantization="mxfp8")
 omni_mxfp4 = Omni(model="<your-model>", quantization="mxfp4")
 omni_mxfp4_dualscale = Omni(model="<your-model>", quantization="mxfp4_dualscale")
@@ -69,6 +75,7 @@ CLI:
 ```bash
 vllm serve <your-model> --omni --quantization fp8
 vllm serve <your-model> --omni --quantization int8
 vllm serve Tongyi-MAI/Z-Image-Turbo --omni --quantization bitsandbytes
 vllm serve <your-model> --omni --quantization mxfp8
 vllm serve <your-model> --omni --quantization mxfp4
 vllm serve <your-model> --omni --quantization mxfp4_dualscale
@@ -85,14 +92,19 @@ config = build_quant_config({
 })
 ```
 Component routing is an extension mechanism, not automatic support for every
 pipeline component. See [Quantization Scope](overview.md#quantization-scope)
 before applying it to an encoder, VAE, or decoder.
 ## Parameters
 | Parameter | Methods | Description |
 |-----------|---------|-------------|
-| `method` | FP8, Int8, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` |
+| `method` | FP8, Int8, BitsAndBytes, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"bitsandbytes"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` |
-| `ignored_layers` | FP8, Int8, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 |
+| `ignored_layers` | FP8, Int8, BitsAndBytes, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 |
 | `activation_scheme` | FP8, Int8 | The runtime value `"dynamic"` selects online activation scaling |
 | `weight_block_size` | FP8 | Optional block-wise FP8 weight quantization size |
 | `quant_type` | BitsAndBytes | Weight format: `"nf4"` (default) or `"fp4"` |
 | `num_bf16_fallback_layers` | MXFP4 DualScale | Leading transformer blocks to keep in BF16 for online `mxfp4_dualscale`; defaults to `5` |
 ## Validation and Notes
@@ -16,72 +16,123 @@ For the internal architecture and backend extension points, see the
 | Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion Flash Attention tensors during inference. | FP8 FA |
 | Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
-## Hardware Support
+Online quantization starts from a normal BF16/FP16 checkpoint and repeats the
 conversion on each fresh model load. It is the simplest path for experiments.
 Pre-quantized checkpoints store the packed weights and scales produced by a
 separate tool; they are easier to reuse across deployments and can carry
 calibrated or mixed-precision policies that load-time conversion cannot
 reproduce.
-| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | ModelOpt | MXFP8 W8A8 | MXFP4 W4A4 | AutoRound | msModelSlim |
+## Support Levels
 |--------|----------|-----------|------------------|----------|------------|------------|-----------|-------------|
 | NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
 | NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
 | NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
 | AMD ROCm | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ⭕ | ❌ |
 | Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
 | Ascend NPU | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
-Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
+Quantization support is specific to a model, component, checkpoint format, and
-guide. FP8 on Ampere may use a weight-only path where available.
+hardware backend. The tables in this guide use the following levels:
-## Model Type Support
+| Level | Meaning |
 |-------|---------|
 | **CI-backed** | A named model and quantized checkpoint are exercised by scheduled full-model hardware CI. |
 | **Validated** | A named model and checkpoint have end-to-end or quality-validation evidence, but the check is not necessarily scheduled in CI. |
 | **Integrated** | The config, loader, quantized layers, or model wiring exists, but this guide has no completed end-to-end validation for the named model and checkpoint. |
 | **Not validated** | There is no current support recommendation for this model/component combination. |
 | **Unsupported** | The combination is intentionally rejected or known not to work. |
-### Diffusion Model (Qwen-Image, Wan2.2)
+A method being available on a device does not make every model on that device
 supported. Likewise, successfully detecting a checkpoint's
 `quantization_config` establishes loader compatibility, not output quality.
-These models run a diffusion transformer as the primary inference module. The
+## Hardware × Quantization Method
 default quantization target is the transformer; tokenizer, scheduler, text
 encoder, and VAE stay on the base checkpoint unless a method guide says
 otherwise.
-| Method | Guide | Mode | Example models | Status |
+This matrix uses concrete inference paths as columns. Producer toolkits are not
-|--------|-------|------|----------------|--------|
+separate methods: NVIDIA ModelOpt produces the ModelOpt FP8/NVFP4 formats,
-| FP8 W8A8 | [FP8](fp8.md) | Online W8A8 or checkpoint FP8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
+Intel AutoRound produces AutoRound checkpoints, and Ascend msModelSlim produces
-| Int8 W8A8 | [Int8](int8.md) | Online or serialized W8A8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
+checkpoints consumed by the native MXFP or compatible Ascend paths.
 | BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Online W4 weight-only | Z-Image-Turbo; Qwen-Image/Wan2.2 not validated | Validated for Z-Image-Turbo |
 | ModelOpt | [ModelOpt](modelopt.md) | Pre-quantized FP8 checkpoints | Qwen-Image, Z-Image, FLUX.2, HunyuanImage-3.0 | Validated for ModelOpt FP8 diffusion checkpoints |
 | MXFP8 W8A8 | [MXFP8](mxfp8.md) | Online W8A8 or offline pre-quantized | Wan2.2-T2V-A14B, I2V-A14B, TI2V-5B | Ascend NPU only; validated for Wan2.2 |
 | MXFP4 W4A4 | [MXFP4](mxfp4.md) | `mxfp4`: online single-scale only; `mxfp4_dualscale`: online or offline dual-scale (offline recommended) | Wan2.2-T2V-A14B, I2V-A14B | Ascend NPU only; validated for Wan2.2 A14B cascade models; TI2V-5B not supported; offline `mxfp4_dualscale` uses calibrated `mul_scale` for best accuracy |
 | AutoRound | [AutoRound](autoround.md) | Pre-quantized W4A16 checkpoints | FLUX.1-dev; Qwen-Image/Wan2.2 not validated | Checkpoint-driven |
 | msModelSlim | [msModelSlim](msmodelslim.md) | Pre-quantized Ascend checkpoints | Wan2.2 recipe; HunyuanImage-3.0 inference target | Ascend/NPU path |
-### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
+| Hardware | [Online FP8](fp8.md) | [Int8](int8.md) | [BitsAndBytes W4](bitsandbytes.md) | [ModelOpt FP8](modelopt.md) | [ModelOpt NVFP4](modelopt.md#supported-modelopt-checkpoint-formats) | [AutoRound W4A16](autoround.md) | [OCP MXFP8](mxfp8.md) | [OCP MXFP4](mxfp4.md) |
 |----------|-----------------------|------------------|------------------------------------------|--------------------------------|-------------------------------------------------------------------------|--------------------------------------|--------------------------|--------------------------|
 | NVIDIA Blackwell (SM 100+) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
 | NVIDIA Hopper (SM 90) | ✅ | ✅ | ✅ | ✅ | ✅ (Marlin fallback) | ✅ | ❌ | ❌ |
 | NVIDIA Ada (SM 89) | ✅ | ✅ | ✅ | ✅ | ⭕ | ✅ | ❌ | ❌ |
 | NVIDIA Ampere (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ | ❌ |
 | AMD ROCm (gfx950 for MXFP4) | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ❌ | ✅ (online only) |
 | Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ✅ | ❌ |
 | Ascend NPU | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
-These models combine an AR language model with audio, vision, talker, or TTS
+Legend: `✅` backend available, `❌` rejected or no implementation, `⭕` not
-stages. Quantization is scoped to the AR language-model stage when the
+verified in this guide. This matrix does not establish model-level support; use
-checkpoint contains a supported `quantization_config`; the non-AR stages stay
+the model-and-stage tables below for that.
 in BF16 unless the model guide explicitly adds support.
-| Method | Guide | Scope | Example models | Status |
+The Int8 column covers online and serialized Int8 paths. OCP MXFP8 is online on
-|--------|-------|-------|----------------|--------|
+Intel XPU and online/offline on Ascend; Intel's separate AutoRound MXFP8
-| ModelOpt | [ModelOpt](modelopt.md) | Thinker or language-model checkpoint config | Qwen3-Omni thinker | ModelOpt checkpoint path |
+checkpoint path is described in the [MXFP8 guide](mxfp8.md). OCP MXFP4 on AMD
-| Int8 | [Int8](int8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
+means online single-scale `mxfp4` on gfx950, while MXFP4 DualScale is
-| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
+Ascend-only. NVIDIA's NVFP4 and OCP MXFP4 have different checkpoint metadata,
-| MXFP8 | [MXFP8](mxfp8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
+scale layouts, and kernel contracts. Mixed ModelOpt FP8/NVFP4 checkpoints also
-| MXFP4 | [MXFP4](mxfp4.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
+require a model-specific mixed-precision policy and validation. FP8 on Ampere
-| AutoRound | [AutoRound](autoround.md) | Thinker or language-model checkpoint config | Qwen2.5-Omni, Qwen3-Omni | Supported through AutoRound checkpoints |
+may use a weight-only path where available.
 | msModelSlim | [msModelSlim](msmodelslim.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
-### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
+## Current Model and Stage Support
-These models split generation across multiple stages. Quantization must be
+The current offline checkpoint guidance is transformer-first. A listed
-attached to the intended stage rather than applied globally.
+checkpoint quantizes the stage named in the **Scope** column; it does not imply
 that the model's encoders, VAE, decoder, or other stages are also quantized.
-| Method | Guide | Scope | Example models | Status |
+### Vendor and Runtime Ecosystems
-|--------|-------|-------|----------------|--------|
+
-| FP8 | [FP8](fp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
+The producer and runtime backend are related, but they are not the same support
-| Int8 | [Int8](int8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
+claim. A producer can emit several checkpoint formats, and each format still
-| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Stage-specific transformer or DiT module | BAGEL, GLM-Image | Not validated |
+needs a compatible vLLM-Omni loader, model-stage wiring, and validation. A
-| ModelOpt | [ModelOpt](modelopt.md) | Checkpoint-defined diffusion stage | BAGEL, GLM-Image | Requires model-specific validation |
+runtime may also provide online quantization without an offline producer path.
-| MXFP8 | [MXFP8](mxfp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
+
-| MXFP4 | [MXFP4](mxfp4.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
+| Ecosystem | Current paths | Current vLLM-Omni scope | Model evidence |
-| AutoRound | [AutoRound](autoround.md) | Checkpoint-defined stage | BAGEL, GLM-Image | No validated checkpoint listed |
+|-----------|---------------|-------------------------|----------------|
-| msModelSlim | [msModelSlim](msmodelslim.md) | Ascend-generated stage weights | GLM-Image | Requires model-specific adaptation |
+| NVIDIA | ModelOpt FP8, NVFP4, and mixed FP8/NVFP4 | Diffusion transformer or Qwen-Omni thinker | Qwen-Image and other named diffusion checkpoints are validated; Qwen3-Omni NVFP4 has H100 CI for the Marlin fallback |
 | Intel | AutoRound W4A16 and AutoRound MXFP8 | Diffusion/world-model transformer or Qwen-Omni thinker | Wan A14B and GLM-Image W4A16 are CI-backed; Qwen-Image, Qwen-Omni, and Cosmos3-Super have validation outside scheduled CI |
 | AMD ROCm | Online OCP MXFP4 through AITER on gfx950; no pre-quantized MXFP4 or AMD Quark workflow is integrated | Diffusion transformer linear layers | Integrated backend with dispatch and weight-allocation unit coverage; no named-model quality validation or scheduled gfx950 full-model CI |
 | Ascend | msModelSlim output through native MXFP8/MXFP4 merge workflows or a compatible generic Ascend format | Wan diffusion transformers | Native Wan MXFP8/MXFP4 paths are validated outside scheduled full-model NPU CI; other model families remain integrated or unvalidated |
 ### Pre-Quantized Model Matrix
 | Model family | Method | Scope | Level | Notes |
 |--------------|--------|-------|-------|-------|
 | Qwen-Image | [ModelOpt](modelopt.md) FP8 or mixed FP8/NVFP4 | Diffusion transformer | Validated | Named Qwen-Image 2512 checkpoints and recipes are available. Text encoder and VAE stay BF16. |
 | Qwen-Image | [AutoRound](autoround.md) W4A16 | Diffusion transformer | Validated | Uses `INC4AI/Qwen-Image-AutoRound-W4A16`; text encoder and VAE stay BF16. |
 | Wan2.2 I2V/T2V A14B | [AutoRound](autoround.md) W4A16 | Both diffusion transformers | CI-backed | The high-noise and low-noise DiTs are quantized; text encoder and VAE stay BF16. |
 | Wan2.2 TI2V 5B | [AutoRound](autoround.md) W4A16 | Diffusion transformer | Validated | Named checkpoint available; not covered by the scheduled Wan AutoRound job. |
 | Wan2.2 | [MXFP8](mxfp8.md) | One or both diffusion transformers | Validated | Native Ascend and AutoRound checkpoint formats are integrated; no scheduled full-model offline NPU job. |
 | Wan2.2 I2V/T2V A14B | [MXFP4 DualScale](mxfp4.md) | Both diffusion transformers | Validated | Offline calibrated checkpoints are recommended; no scheduled full-model offline NPU job. |
 | Qwen2.5-Omni | [AutoRound](autoround.md) W4A16 | Thinker language model | Validated | Encoders and output/audio stages stay BF16. |
 | Qwen3-Omni | [ModelOpt](modelopt.md) FP8 | Thinker language model | Validated | Tested outside scheduled CI. |
 | Qwen3-Omni | [ModelOpt](modelopt.md) NVFP4 | Thinker language model | CI-backed | H100 CI exercises the Marlin fallback; the native Blackwell FP4 path has separate recipe validation. |
 | Qwen3-Omni | [AutoRound](autoround.md) W4A16 | Thinker language model | Validated | Audio encoder, vision encoder, talker, and Code2Wav stay BF16. |
 | GLM-Image | [AutoRound](autoround.md) W4A16 | Diffusion transformer stage | CI-backed | Text-to-image and image-to-image full-model smoke coverage. |
 | Cosmos3-Super | [AutoRound](autoround.md) W4A16 | World-model transformer | Validated | Manually validated serving path; VAE and other components stay BF16. |
 | Cosmos3-Nano | [AutoRound](autoround.md) W4A16 | World-model transformer | Integrated | Named checkpoint is listed, but no model-specific end-to-end test is maintained in-tree. |
 | MiniMax H3 | None | None | Not validated | No offline vendor checkpoint path is documented; online FP8 targets the DiT only. |
 The method pages contain the complete checkpoint lists, including FLUX,
 Z-Image, and HunyuanImage-3.0 variants.
 ### Online Quantization
 | Model family | Methods | Scope | Level |
 |--------------|---------|-------|-------|
 | Qwen-Image | FP8, Int8 | Diffusion transformer | Validated |
 | Z-Image | Int8, BitsAndBytes W4 | Diffusion transformer | Validated |
 | Wan2.2 | MXFP8 | One or both diffusion transformers | Validated on the documented XPU/NPU paths |
 | Wan2.2 A14B | MXFP4 | Both diffusion transformers | Validated on Ascend; the ROCm gfx950 online backend is integrated but not model-validated |
 | MiniMax H3 | FP8 | DiT and optional reference DiT | Validated |
 | Qwen3-Omni | Generic online methods, such as FP8 | Thinker language model only | Integrated; not model-validated as an online recommendation |
 See [Online Quantization](online.md) for configuration and hardware details.
 ### Unlisted and Long-Tail Models
 For models not listed above, generic quantized linear layers or compatible
 checkpoint metadata are only an integration starting point. The default support
 target is the model's primary transformer, DiT, or AR language-model stage.
 Text/audio/vision encoders, VAEs, vocoders, decoders, and output stages remain
 BF16 until a model-specific integration supplies quant-aware layers, checkpoint
 mapping, end-to-end output validation, and backend coverage.
 !!! note
    "Online quantization" means vLLM-Omni computes the quantization data while
@@ -92,8 +143,8 @@ attached to the intended stage rather than applied globally.
 ### Diffusion Model (Qwen-Image, Wan2.2)
-The default target is the diffusion transformer. Component routing is available
+The default target is the diffusion transformer. `build_quant_config()` can
-through `build_quant_config()`:
+construct a component router:
 ```python
 from vllm_omni.quantization import build_quant_config
@@ -104,11 +155,19 @@ config = build_quant_config({
 })
 ```
 !!! warning "Routing syntax is not a support claim"
    A component dictionary only selects a quantization config by module prefix.
    It does not make an encoder or VAE quantizable. The model must construct
    that component from quantization-aware layers, and an offline checkpoint
    must contain matching packed weights and scales. Current user guidance
    keeps diffusion text encoders and VAEs in BF16.
 | Component | Default quantized? | Notes |
 |-----------|--------------------|-------|
-| Diffusion transformer | Yes | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim |
+| Diffusion transformer | Yes, for listed model/method pairs | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim |
 | Second diffusion transformer | Yes, for listed Wan2.2 A14B methods | `transformer` and `transformer_2` read their own offline checkpoint metadata; arbitrary mixed methods are not currently validated |
 | Text encoder | No | Keep BF16 unless a method-specific guide documents support |
-| VAE | No | Keep BF16; storage-only paths are method-specific |
+| VAE | No | Keep BF16 |
 | Scheduler/tokenizer | No | Loaded from the base model repository |
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -121,6 +180,18 @@ config = build_quant_config({
 | Talker or TTS stage | No | BF16 unless model-specific support is documented |
 | Code2Wav | No | BF16 |
 An explicit `ComponentQuantizationConfig` can be used by model developers to
 route different configs to `audio_tower`, `visual`, and `language_model`, but
 the supported Qwen-Omni checkpoints currently quantize only the thinker
 language model.
 ### World Model (Cosmos3)
 | Component | Default quantized? | Notes |
 |-----------|--------------------|-------|
 | World-model transformer | AutoRound checkpoints only | Cosmos3-Super is validated; Cosmos3-Nano is integrated but not end-to-end tested in-tree |
 | VAE and guardrail components | No | BF16 |
 ### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
 | Component | Default quantized? | Notes |
@@ -141,6 +212,7 @@ build_quant_config("fp8")
 build_quant_config({"method": "fp8", "activation_scheme": "static"})
 build_quant_config("bitsandbytes")
 build_quant_config("auto-round", bits=4, group_size=128)
 # Component routing syntax; model-specific support is still required.
 build_quant_config({"transformer": {"method": "fp8"}, "vae": None})
 build_quant_config(None)
 ```