diff --git a/docs/usage/faq.md b/docs/usage/faq.md index 0539e158b..87617b88e 100644 --- a/docs/usage/faq.md +++ b/docs/usage/faq.md @@ -14,7 +14,10 @@ A: At first, you can check current [issues](https://github.com/vllm-project/vllm > Q: Does vLLM-Omni support AWQ or any other quantization? -A: We plan to introduce GGUF FP8 prequantized models and online FP8 quantization in version 0.16.0. Support for other quantization types will follow in future releases. For details, please see our [Q1 quantization roadmap](https://github.com/vllm-project/vllm-omni/issues/1057). +A: AWQ is not currently listed as a validated vLLM-Omni path. Quantization +support depends on the model, pipeline stage, checkpoint format, and hardware +backend. See the current [quantization support matrix](../user_guide/quantization/overview.md) +for online methods, pre-quantized checkpoints, and model-level validation. > Q: Does vLLM-Omni support multimodal streaming input and output? diff --git a/docs/user_guide/diffusion_features.md b/docs/user_guide/diffusion_features.md index 7c664e88b..3dfce3b95 100644 --- a/docs/user_guide/diffusion_features.md +++ b/docs/user_guide/diffusion_features.md @@ -92,7 +92,7 @@ current support. |--------|--------------|-------------|----------| | **[FP8](quantization/fp8.md)** | `quantization="fp8"` | FP8 W8A8 on validated transformer stages | Memory reduction, inference speedup | | **[INT8](quantization/int8.md)** | `quantization="int8"` | INT8 W8A8 on validated transformer stages | Memory reduction, broad GPU compatibility | -| **[GGUF](quantization/gguf.md)** | `quantization="gguf"` | Native GGUF transformer-only weights (Q4, Q8, etc.) | Memory reduction on consumer GPUs | +| **[BitsAndBytes W4](quantization/bitsandbytes.md)** | `quantization="bitsandbytes"` | Online NF4/FP4 weight-only quantization on validated CUDA transformer stages | Reducing transformer weight memory without preparing a separate checkpoint | ## Supported Models diff --git a/docs/user_guide/examples/offline_inference/text_to_video.md b/docs/user_guide/examples/offline_inference/text_to_video.md index 1a2082f82..5dc1053e4 100644 --- a/docs/user_guide/examples/offline_inference/text_to_video.md +++ b/docs/user_guide/examples/offline_inference/text_to_video.md @@ -143,7 +143,8 @@ python text_to_video.py \ - `--enable-layerwise-offload`: enable layerwise offloading on DiT modules. - `--frame-rate`: generation FPS for pipelines that require it (e.g., LTX2). - `--audio-sample-rate`: fallback audio sample rate when the pipeline returns audio. -- `--quantization`: quantization method (`fp8` for FP8, `gguf` for GGUF). +- `--quantization`: online quantization method, for example `fp8`; supported + pre-quantized checkpoints are normally auto-detected from checkpoint metadata. - `--flow-shift`: scheduler flow_shift parameter. - `--lora-path`: path to PEFT LoRA adapter folder or checkpoint file. - `--lora-scale`: scale factor for LoRA weights. diff --git a/docs/user_guide/examples/online_serving/diffusers_pipeline_adapter.md b/docs/user_guide/examples/online_serving/diffusers_pipeline_adapter.md index 7edf7c61d..2a7a44994 100644 --- a/docs/user_guide/examples/online_serving/diffusers_pipeline_adapter.md +++ b/docs/user_guide/examples/online_serving/diffusers_pipeline_adapter.md @@ -90,8 +90,8 @@ vllm serve "Qwen/Qwen-Image" \ --quantization-config '{"method": "fp8"}' ``` -Other vLLM-Omni quantization methods, such as `gguf`, `modelopt`, `mxfp4`, -`mxfp8`, serialized checkpoints, static FP8 configs, and layer-name skip lists +Other vLLM-Omni quantization methods, such as `modelopt`, `mxfp4`, `mxfp8`, +serialized checkpoints, static FP8 configs, and layer-name skip lists such as `ignored_layers`, are intentionally not translated by this compatibility shortcut. Use Diffusers-native configuration through `diffusers_load_kwargs` or a native vLLM-Omni pipeline for those cases. diff --git a/docs/user_guide/quantization/autoround.md b/docs/user_guide/quantization/autoround.md index bafadec9c..99a4c0bcb 100644 --- a/docs/user_guide/quantization/autoround.md +++ b/docs/user_guide/quantization/autoround.md @@ -10,6 +10,10 @@ checkpoint's `config.json` and auto-detects AutoRound is static quantization: no `--quantization` flag is needed at inference time when the checkpoint already contains the quantization config. +The validation terms on this page follow the +[support levels](overview.md#support-levels) defined in the overview. A listed +checkpoint is not enough by itself to establish end-to-end model support. + ## Hardware Support | Device | Support | @@ -21,46 +25,57 @@ inference time when the checkpoint already contains the quantization config. | Intel XPU | ✅ | | Ascend NPU | ❌ | -Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this -guide. AutoRound is Intel-supported. +Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this +guide. This table is method-level backend availability. AutoRound is an Intel +project; vLLM selects a compatible CUDA or XPU compute backend at load time. ## Model Type Support ### Diffusion Model (Qwen-Image, Wan2.2) -| Model | Checkpoint | Scope | Scheme | Backend | -|-------|------------|-------|--------|---------| -| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend | -| Qwen-Image | Not listed | Diffusion transformer | W4A16 | Not validated | -| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend | -| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend | -| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend | +| Model | Checkpoint | Scope | Scheme | Validation | +|-------|------------|-------|--------|------------| +| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | Validated | +| Qwen-Image | `INC4AI/Qwen-Image-AutoRound-W4A16` | Diffusion transformer | W4A16 | Validated with end-to-end generation and quality comparison | +| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed | +| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed | +| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | Validated; not in the scheduled Wan AutoRound job | + +CUDA execution uses GPTQ-Marlin or another compatible vLLM AutoRound backend; +Intel XPU uses the Intel-supported backend. The text encoder and VAE stay in +BF16 for all entries above. ### World Model (Cosmos3) -| Model | Checkpoint | Scope | Scheme | Backend | -|-------|------------|-------|--------|---------| -| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend | -| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend | +| Model | Checkpoint | Scope | Scheme | Validation | +|-------|------------|-------|--------|------------| +| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | Integrated; no model-specific end-to-end test is maintained in-tree | +| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | Validated with a manual serving run | + +The Cosmos VAE and guardrail components stay in BF16. ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) -| Model | Checkpoint | Scope | Scheme | Backend | -|-------|------------|-------|--------|---------| -| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Language-model stage | W4A16 | AutoRound | -| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | AutoRound | +| Model | Checkpoint | Scope | Scheme | Validation | +|-------|------------|-------|--------|------------| +| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI | +| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI | | Qwen3-TTS | Not listed | TTS language-model stage | W4A16 | Not validated | AutoRound support is checkpoint-driven. A model is supported when its checkpoint uses a compatible INC/AutoRound config and the target stage maps to vLLM-Omni's runtime module names. +For Qwen-Omni, only the thinker language model uses the AutoRound checkpoint. +Audio and vision encoders, the talker, and waveform-decoder stages stay in +BF16. + ### Multi-Stage Diffusion Model (BAGEL, GLM-Image) -| Model | Scope | Status | Notes | -|-------|-------|--------|-------| -| GLM-Image | Diffusion transformer | ✅ | `Intel/GLM-Image-int4-AutoRound` | -| BAGEL | Checkpoint-defined diffusion or transformer stage | Not validated | Requires a compatible AutoRound checkpoint | +| Model | Checkpoint | Scope | Validation | +|-------|------------|-------|------------| +| GLM-Image | `Intel/GLM-Image-int4-AutoRound` | Diffusion transformer | CI-backed | +| BAGEL | Not listed | Checkpoint-defined diffusion or transformer stage | Not validated | ## Configuration @@ -119,6 +134,14 @@ The checkpoint should contain a config like: At load time, vLLM-Omni builds an `OmniINCConfig`, remaps checkpoint block names to runtime module names, and selects the matching vLLM compute backend. +Checkpoint auto-detection and successful layer construction establish the +**Integrated** level. Promote a model to **Validated** only after the named +checkpoint completes an end-to-end generation or multimodal request and its +output passes a model-appropriate sanity or quality check. Compare against the +BF16 baseline for quality-sensitive image and video paths. Promote it to +**CI-backed** only when that named checkpoint is selected by scheduled hardware +CI. + Example checkpoint creation: ```bash diff --git a/docs/user_guide/quantization/bitsandbytes.md b/docs/user_guide/quantization/bitsandbytes.md index 37ccd2471..ea98b4a72 100644 --- a/docs/user_guide/quantization/bitsandbytes.md +++ b/docs/user_guide/quantization/bitsandbytes.md @@ -9,6 +9,10 @@ stay in BF16/FP16. No pre-quantized checkpoint is required. This is an online (dynamic) path: load the normal HuggingFace checkpoint and quantize during model loading via the `bitsandbytes` CUDA kernels. +The validation terms on this page follow the +[support levels](overview.md#support-levels). BitsAndBytes kernel support does +not imply that every diffusion transformer has been quality-validated. + ## Hardware Support | Device | Support | @@ -21,7 +25,7 @@ quantize during model loading via the `bitsandbytes` CUDA kernels. | Intel XPU | ❌ | | Ascend NPU | ❌ | -Legend: `✅` supported, `❌` unsupported. Non-CUDA platforms raise in +Legend: `✅` backend available, `❌` unsupported. Non-CUDA platforms raise in `get_quant_method()`; CUDA requires compute capability 7.5+ (`SM 75+`). Requires the optional `bitsandbytes` package (`pip install bitsandbytes`). @@ -30,11 +34,11 @@ Requires the optional `bitsandbytes` package (`pip install bitsandbytes`). ### Diffusion Model (Qwen-Image, Wan2.2) -| Model | HF models | CUDA | Mode | Recommendation | -|-------|-----------|:----:|------|----------------| -| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Yes | Online W4 weight-only | All heavy linear layers; sensitive embedders stay BF16 | -| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Not validated | Online W4 weight-only | Compare vs BF16 before enabling | -| Wan2.2 | Wan2.2 diffusion pipelines | Not validated | Online W4 weight-only | Validate before enabling in docs | +| Model | HF models | Scope | Level | Recommendation | +|-------|-----------|-------|-------|----------------| +| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Diffusion transformer | Validated | All heavy linear layers; sensitive embedders stay BF16 | +| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Diffusion transformer | Not validated | Compare with BF16 before enabling | +| Wan2.2 | Wan2.2 diffusion pipelines | One or both diffusion transformers | Not validated | Validate quality and cascade behavior before enabling | Other diffusion models may work if their transformer uses supported linear layers, but they are not validated in this guide. @@ -104,6 +108,10 @@ Multi-GPU tensor parallelism (`tensor_parallel_size` > 1) is not validated for BitsAndBytes in diffusion models; each rank quantizes its own weight shard independently. +Z-Image-Turbo is validated outside scheduled full-model BitsAndBytes CI. The +real CUDA kernel smoke tests cover quantization and matrix multiplication, but +do not establish model-level quality for unlisted architectures. + If quality regresses, use `ignored_layers` to keep sensitive projections in BF16 (for example `to_out` or `w2`). diff --git a/docs/user_guide/quantization/modelopt.md b/docs/user_guide/quantization/modelopt.md index d8da4e1a9..3262089b2 100644 --- a/docs/user_guide/quantization/modelopt.md +++ b/docs/user_guide/quantization/modelopt.md @@ -13,6 +13,11 @@ components, such as the tokenizer, scheduler, text encoder, vision/audio encoder, and VAE, on the base checkpoint unless a model-specific guide says otherwise. +The validation terms on this page follow the +[support levels](overview.md#support-levels) defined in the overview. ModelOpt +format detection and checkpoint adaptation are shared runtime integrations; +each model and stage still requires its own end-to-end validation. + !!! note ModelOpt checkpoints are pre-quantized checkpoints. Do not pass `--quantization fp8` for these checkpoints. The checkpoint @@ -58,11 +63,11 @@ stay unquantized unless the model-specific recipe validates otherwise. | Intel XPU | ⭕ | | Ascend NPU | ❌ | -Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide. -The optional CUTLASS FP8 runtime override requires CUDA SM89+. ModelOpt NVFP4 -and mixed FP8/NVFP4 diffusion checkpoints are currently validated on Blackwell -CUDA systems in the recipes below; other CUDA generations require separate -backend and quality validation. +Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this +guide. The optional CUTLASS FP8 runtime override requires CUDA SM89+. ModelOpt +NVFP4 and mixed FP8/NVFP4 diffusion checkpoints are currently validated on +Blackwell CUDA systems in the recipes below; other CUDA generations require +separate backend and quality validation. ## Model Type Support @@ -79,6 +84,9 @@ backend and quality validation. | HunyuanImage-3.0 | `feizhai123/hunyuan-image3-modelopt-mixed-experts-nvfp4-dense-fp8` | MoE diffusion transformer | Validated for ModelOpt mixed FP8/NVFP4 checkpoints | | Wan2.2 | Not available | Diffusion transformer | Not validated | +The validated diffusion entries above have named checkpoint and recipe +evidence. They are not all exercised by scheduled full-model CI. + For full serving commands and benchmark context, see [`recipes/Qwen/Qwen-Image.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen-Image.md) and @@ -88,8 +96,8 @@ and | Model | Scope | Status | |-------|-------|--------| -| Qwen3-Omni | Thinker language-model stage | ModelOpt FP8 checkpoint path | -| Qwen3-Omni | Thinker language-model stage (W4A4 NVFP4) | Validated; see [Qwen3-Omni NVFP4 W4A4](#qwen3-omni-nvfp4-w4a4-thinker) below | +| Qwen3-Omni | Thinker language-model stage (FP8) | Validated outside scheduled CI | +| Qwen3-Omni | Thinker language-model stage (W4A4 NVFP4) | CI-backed for the H100 Marlin fallback; native Blackwell path is recipe-validated | | Qwen3-TTS | TTS language-model stage | Not validated | Audio encoder, vision encoder, talker, and code2wav stages stay in BF16 unless @@ -102,9 +110,9 @@ Qwen3-Omni-30B-A3B-Instruct thinker language model. The thinker text body (attention + MoE experts) is quantized to NVFP4 with FP8 per-tensor input scales; the audio encoder, vision encoder, talker, and code2wav stay in BF16. -| Variant | HF checkpoint | Hardware | -|---------|---------------|----------| -| W4A4 NVFP4 (full thinker) | `YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip` | sm_100+ (Blackwell, FlashInfer FP4 GEMM) | +| Variant | HF checkpoint | Hardware and validation | +|---------|---------------|-------------------------| +| W4A4 NVFP4 (full thinker) | `YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip` | sm_100+ native FlashInfer FP4 GEMM; H100 Marlin fallback is CI-backed | Calibration uses ModelOpt `mtq.NVFP4_DEFAULT_CFG` with the `awq_clip` algorithm on 1024 ultrachat samples chat-templated through the Qwen3-Omni tokenizer. @@ -232,3 +240,6 @@ omni = Omni( 5. Keep `--quantization fp8` for online FP8 from BF16 checkpoints; use this ModelOpt path only when the checkpoint already contains ModelOpt quantized weights and scales. +6. The Qwen3-Omni full-model CI smoke exercises the H100 Marlin fallback. A + scheduled Blackwell test is still required before treating the native + FlashInfer FP4 kernel path as CI-backed. diff --git a/docs/user_guide/quantization/msmodelslim.md b/docs/user_guide/quantization/msmodelslim.md index c03628b9d..2c2787ea9 100644 --- a/docs/user_guide/quantization/msmodelslim.md +++ b/docs/user_guide/quantization/msmodelslim.md @@ -3,12 +3,17 @@ ## Overview [msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend compression -toolkit for producing pre-quantized model checkpoints. In vLLM-Omni, these -checkpoints run through the Ascend/NPU path with `--quantization ascend`. +toolkit for producing pre-quantized model checkpoints. vLLM-Omni consumes its +output either through a compatible generic Ascend checkpoint path or through +the native MXFP8 and MXFP4 merge workflows documented in this guide. msModelSlim is static quantization: quantized weights are generated offline before vLLM-Omni inference starts. +The validation terms on this page follow the +[support levels](overview.md#support-levels). A producer recipe or hardware +backend alone is an integration point, not model-level validation. + ## Hardware Support | Device | Support | @@ -20,23 +25,28 @@ before vLLM-Omni inference starts. | Intel XPU | ❌ | | Ascend NPU | ✅ | -Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this +Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this guide. ## Model Type Support ### Diffusion Model (Qwen-Image, Wan2.2) -| Model | Base model | Scope | Hardware | Notes | -|-------|------------|-------|----------|-------| -| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Ascend NPU | Upstream msModelSlim provides a Wan2.2 quantization recipe; vLLM-Omni inference validation is not listed | -| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Ascend NPU | Not validated in this guide | -| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Ascend A2/A3 NPU | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation | +| Model | Base model | Scope | Level | Notes | +|-------|------------|-------|-------|-------| +| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Integrated | Upstream msModelSlim provides a producer recipe; use the [MXFP8](mxfp8.md) or [MXFP4](mxfp4.md) workflow for vLLM-Omni's native merged formats | +| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Not validated | No supported checkpoint workflow is documented | +| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Integrated | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation and validate against BF16 | Public Hugging Face quantized weights are not available yet. Use the [HunyuanImage-3.0 msModelSlim adaptation](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8) to generate the checkpoint manually. +msModelSlim is also the producer used by the native offline MXFP8 and MXFP4 +DualScale workflows. Those workflows preprocess and annotate each transformer +checkpoint and are auto-detected at inference; follow their method pages rather +than passing `--quantization ascend`. + ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) | Model | Scope | Status | Notes | @@ -53,25 +63,32 @@ to generate the checkpoint manually. ## Configuration +For a compatible generic Ascend checkpoint, select the Ascend quantization +method explicitly: + Offline inference: ```bash -python text_to_image.py --model --quantization ascend +python text_to_image.py --model --quantization ascend ``` Online serving: ```bash -vllm serve --omni --quantization ascend +vllm serve --omni --quantization ascend ``` ## Parameters | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `--quantization` | str | - | Use `ascend` for msModelSlim-produced checkpoints | +| `--quantization` | str | - | Use `ascend` only for a compatible generic Ascend-format checkpoint; omit it for native merged MXFP8/MXFP4 checkpoints | | `model` | str | - | Path to the quantized checkpoint generated by Ascend tooling | +Do not pass `--quantization ascend` for a Wan checkpoint processed by the native +MXFP8 or MXFP4 merge tools. Their transformer metadata selects `mxfp8` or +`mxfp4_dualscale` automatically. + Example msModelSlim command for a Wan2.2 W8A8 checkpoint: ```bash @@ -93,3 +110,6 @@ For HunyuanImage-3.0, use the Hunyuan-specific adaptation linked above. tooling; it is not a load-time CUDA quantizer. 3. Keep the quantized checkpoint aligned with the same model architecture and stage config used for BF16 inference. +4. No named full-model msModelSlim checkpoint is currently scheduled in NPU + CI. Treat new producer outputs as **Integrated** until an end-to-end BF16 + comparison establishes **Validated** support. diff --git a/docs/user_guide/quantization/mxfp4.md b/docs/user_guide/quantization/mxfp4.md index b18b3e7b3..ab377b7e4 100644 --- a/docs/user_guide/quantization/mxfp4.md +++ b/docs/user_guide/quantization/mxfp4.md @@ -13,6 +13,10 @@ vLLM-Omni provides two quantization methods with different scale structures: | `mxfp4` | Single-scale (per-32 fine only) | Online only | Quick accuracy baseline; no checkpoint prep needed | | `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | Production; better accuracy; offline recommended | +The validation terms on this page follow the +[support levels](overview.md#support-levels). Current model guidance covers +Wan diffusion transformers only; the text encoder and VAE stay in BF16. + !!! tip "Recommended: `mxfp4_dualscale` offline" For production deployments, use the `mxfp4_dualscale` offline mode with a pre-quantized checkpoint produced by msModelSlim. Offline checkpoints load @@ -34,26 +38,45 @@ vLLM-Omni provides two quantization methods with different scale structures: ## Hardware Support -| Device | Support | -|--------|---------| -| NVIDIA Blackwell GPU (SM 100+) | ⭕ | -| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ | -| NVIDIA Ampere GPU (SM 80+) | ⭕ | -| AMD ROCm (gfx950 / MI355X) | ✅ | -| Intel XPU | ⭕ | -| Ascend NPU (Atlas 950 A5) | ✅ | +| Device | Online `mxfp4` | Online `mxfp4_dualscale` | Offline `mxfp4_dualscale` | +|--------|-----------------|---------------------------|----------------------------| +| NVIDIA Blackwell GPU (SM 100+) | ❌ | ❌ | ❌ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ | ❌ | ❌ | +| NVIDIA Ampere GPU (SM 80+) | ❌ | ❌ | ❌ | +| AMD ROCm (gfx950 / MI355X) | ✅ | ❌ | ❌ | +| Intel XPU | ❌ | ❌ | ❌ | +| Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ✅ | -Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide. +Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this +guide. Backend availability does not imply full-model checkpoint validation. + +!!! note "Why NVIDIA is marked unsupported" + This table describes the methods currently wired into vLLM-Omni, not every + low-precision format NVIDIA hardware could execute. `DiffusionMXFP4Config` + has runtime implementations only for Ascend and ROCm gfx950; on CUDA it + raises `NotImplementedError` because no CUDA MXFP4 linear method or + checkpoint loader is registered. + + On NVIDIA, use [ModelOpt](modelopt.md) NVFP4 for the documented 4-bit + checkpoint path. NVFP4 and OCP MXFP4 have different checkpoint metadata, + scale layouts, and kernel contracts, so they are not interchangeable. A + native CUDA MXFP4 path would require a CUDA linear implementation, loader + contract, and model-level output validation. + +The ROCm path dispatches online single-scale `mxfp4` to AITER on gfx950. It has +kernel and dispatch coverage, but this guide does not yet claim end-to-end +quality validation for a named Wan checkpoint on ROCm. DualScale and +pre-quantized MXFP4 checkpoints are not supported on ROCm. ## Model Type Support ### Diffusion Model (Wan2.2) -| Model | Online | Offline | Notes | -|-------|--------|---------|-------| -| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade (`transformer` + `transformer_2`); both transformers quantized with the same config | -| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade; same scheme as T2V-A14B | -| Wan2.2-TI2V-5B | ❌ | ❌ | Parameter count too small; W4A4 causes unacceptable accuracy loss | +| Model | Online (Ascend) | Offline (Ascend) | Validation and notes | +|-------|-----------------|------------------|----------------------| +| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | Validated outside scheduled full-model offline NPU CI; both cascade transformers are quantized | +| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | Validated outside scheduled full-model offline NPU CI; same cascade scheme as T2V-A14B | +| Wan2.2-TI2V-5B | ❌ | ❌ | Unsupported because W4A4 causes unacceptable accuracy loss | The choice between `mxfp4` and `mxfp4_dualscale` in **online mode** is about quantization quality, not model compatibility — both work on cascade (A14B) and @@ -83,12 +106,13 @@ by the merge script); there is no offline `mxfp4` single-scale format. ### `mxfp4` — Single-Scale Online Mode -Online mode requires no pre-processing. vLLM-Omni quantizes BF16 weights to -MXFP4 at load time using `npu_dynamic_mx_quant`. A single block scale +Online mode requires no pre-processing. On Ascend, vLLM-Omni quantizes BF16 +weights at load time using `npu_dynamic_mx_quant`. On ROCm gfx950, the same +`mxfp4` config dispatches to the AITER online kernel. A single block scale (`float8_e8m0fnu`, one per 32 K elements) is computed on the fly; no -calibration `mul_scale` is available. Applies equally to single-transformer -and cascade (A14B) models — both transformers in a cascade receive the same -quantization config automatically. +calibration `mul_scale` is available. The config applies equally to +single-transformer and cascade (A14B) models, but named Wan model validation is +currently limited to the documented Ascend path. ```python from vllm_omni import Omni @@ -111,10 +135,10 @@ vllm serve --omni --quantization mxfp4 ### `mxfp4_dualscale` — DualScale Online Mode -Online DualScale mode computes both fine and coarse scales on the fly from BF16 -weights using `npu_dynamic_dual_level_mx_quant`. Applies equally to -single-transformer and cascade (A14B) models. Compared to `mxfp4` online, -DualScale provides better quantization accuracy at higher compute cost. +Ascend-only online DualScale mode computes both fine and coarse scales on the +fly from BF16 weights using `npu_dynamic_dual_level_mx_quant`. It applies +equally to single-transformer and cascade (A14B) models. Compared to `mxfp4` +online, DualScale provides better quantization accuracy at higher compute cost. The default configuration keeps the leading 5 transformer blocks in BF16 (`num_bf16_fallback_layers=5`). Accuracy evaluation on Wan2.2-A14B shows this @@ -341,6 +365,11 @@ unless they appear in `ignored_layers`. quantization. Use `ignored_layers` and `num_bf16_fallback_layers` to trade off compression vs. accuracy for precision-sensitive layers. +8. The native methods, merge mappings, and per-transformer cascade metadata + have unit coverage. The current Buildkite configuration does not schedule a + named full-model offline MXFP4 checkpoint on NPU, so this path is + **Validated**, not **CI-backed**. + ## Adapting MXFP4 for a New Model This section is aimed at developers who want to add MXFP4 support to a model diff --git a/docs/user_guide/quantization/mxfp8.md b/docs/user_guide/quantization/mxfp8.md index 823e898fd..cce64bbdd 100644 --- a/docs/user_guide/quantization/mxfp8.md +++ b/docs/user_guide/quantization/mxfp8.md @@ -15,30 +15,45 @@ This method supports three modes: | **Offline (Native)** | msModelSlim-exported MXFP8 weights converted to diffusers format via `merge_mxfp8_checkpoint.py` — weights and scales are loaded directly from the preprocessed checkpoint | | **Offline (AutoRound)** | AutoRound MXFP8 checkpoints with `data_type="mx_fp"` — auto-detected from `config.json` | +The validation terms on this page follow the +[support levels](overview.md#support-levels). Current model guidance covers +Wan diffusion transformers only; the text encoder and VAE stay in BF16. + ## Hardware Support | Device | Online | Offline (Native) | Offline (AutoRound) | |--------|--------|------------------|---------------------| -| NVIDIA Blackwell GPU (SM 100+) | ⭕ | ⭕ | ⭕ | -| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ | ⭕ | ⭕ | -| NVIDIA Ampere GPU (SM 80+) | ⭕ | ⭕ | ⭕ | -| AMD ROCm | ⭕ | ⭕ | ⭕ | +| NVIDIA Blackwell GPU (SM 100+) | ❌ | ❌ | ⭕ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ | ❌ | ⭕ | +| NVIDIA Ampere GPU (SM 80+) | ❌ | ❌ | ⭕ | +| AMD ROCm | ❌ | ❌ | ⭕ | | Intel XPU | ✅ | ❌ | ✅ | | Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ⭕ | -Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide. +Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this +guide. Backend availability does not imply full-model checkpoint validation. -**Note**: Intel XPU only supports AutoRound MXFP8 for offline mode. Use AutoRound quantized checkpoints or online mode for XPU. +The native `DiffusionMXFP8Config` currently dispatches only to Intel XPU and +Ascend NPU implementations. CUDA reaches `NotImplementedError`, so the NVIDIA +rows are marked unsupported even though this is an implementation gap rather +than a general claim about NVIDIA hardware capabilities. On NVIDIA, use +[FP8](fp8.md) for load-time quantization or [ModelOpt](modelopt.md) FP8 for +pre-quantized checkpoints. Adding native CUDA MXFP8 requires a registered CUDA +linear method, a compatible checkpoint contract, and model-level validation. + +Native `mxfp8` is available online on Intel XPU and online or offline on +Ascend. Intel XPU only supports AutoRound MXFP8 for offline mode. The AutoRound +column is a separate checkpoint format and compute path. ## Model Type Support ### Diffusion Model (Wan2.2) -| Model | Mode | Notes | -|-------|------|-------| -| Wan2.2-T2V-A14B | Online + Offline | MoE cascade; quantizes two transformers (`transformer` + `transformer_2`) | -| Wan2.2-I2V-A14B | Online + Offline | MoE cascade; quantizes two transformers (`transformer` + `transformer_2`) | -| Wan2.2-TI2V-5B | Online + Offline | Single transformer | +| Model | Mode | Scope | Validation | +|-------|------|-------|------------| +| Wan2.2-T2V-A14B | Online + Offline | MoE cascade: `transformer` + `transformer_2` | Validated outside scheduled full-model offline NPU CI | +| Wan2.2-I2V-A14B | Online + Offline | MoE cascade: `transformer` + `transformer_2` | Validated outside scheduled full-model offline NPU CI | +| Wan2.2-TI2V-5B | Online + Offline | Single diffusion transformer | Validated outside scheduled full-model offline NPU CI | ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) @@ -218,3 +233,7 @@ omni = Omni(model="") `--output-dir`), regenerate it with the current script. The old script wrote a separate `quantization_config.json` that is not read by vLLM-Omni; the current script injects the config directly into `transformer/config.json`. +4. The native loader, merge mappings, and Wan cascade config propagation have + unit coverage. The current Buildkite configuration does not schedule a + named full-model offline MXFP8 checkpoint on NPU, so this path is + **Validated**, not **CI-backed**. diff --git a/docs/user_guide/quantization/online.md b/docs/user_guide/quantization/online.md index 6103be9a0..2da41c441 100644 --- a/docs/user_guide/quantization/online.md +++ b/docs/user_guide/quantization/online.md @@ -6,7 +6,7 @@ Online quantization means vLLM-Omni computes quantized weights and scales while loading the model. Use it when you want memory savings without preparing a separate quantized checkpoint. -This mode is different from pre-quantized checkpoint formats such as GGUF, +This mode is different from pre-quantized checkpoint formats such as ModelOpt, AutoRound, msModelSlim, or serialized Int8 checkpoints. Those formats are prepared before serving and are documented in their method-specific guides. For MXFP8 and MXFP4, use this page for load-time quantization from BF16 @@ -15,18 +15,22 @@ by msModelSlim and the merge tools. ## Hardware Support -| Device | FP8 W8A8 | Int8 W8A8 | MXFP8 W8A8 | MXFP4 W4A4 | -|--------|----------|-----------|------------|------------| -| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ⭕ | ⭕ | -| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ⭕ | ⭕ | -| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ⭕ | ⭕ | -| AMD ROCm | ⭕ | ⭕ | ⭕ | ⭕ | -| Intel XPU | ⭕ | ⭕ | ✅ | ⭕ | -| Ascend NPU | ❌ | ✅ | ✅ | ✅ | +| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | MXFP8 W8A8 | MXFP4 W4A4 | +|--------|----------|-----------|------------------|------------|------------| +| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ❌ | ❌ | +| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ❌ | ❌ | +| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ❌ | ❌ | +| AMD ROCm (gfx950) | ⭕ | ⭕ | ❌ | ❌ | ✅ | +| Intel XPU | ⭕ | ⭕ | ❌ | ✅ | ❌ | +| Ascend NPU | ❌ | ✅ | ❌ | ✅ | ✅ | -Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this -guide. FP8 on Ampere may use a weight-only path where available. MXFP8 and -MXFP4 are documented for the Ascend NPU path. +Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this +guide. These entries describe method-level backend availability, not validation +for every model. FP8 on Ampere may use a weight-only path where available. +On ROCm gfx950, MXFP4 means online single-scale `mxfp4`; online and offline +`mxfp4_dualscale` are Ascend-only. The native MXFP8/MXFP4 configs do not +currently register CUDA implementations; use FP8 or ModelOpt FP8/NVFP4 on +NVIDIA instead. ## Model Type Support @@ -36,8 +40,9 @@ MXFP4 are documented for the Ascend NPU path. |--------|-------|----------------|--------| | FP8 W8A8 | [FP8](fp8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models | | Int8 W8A8 | [Int8](int8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image | +| BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Z-Image-Turbo; Qwen-Image and Wan2.2 are not validated | Validated for Z-Image-Turbo | | MXFP8 W8A8 | [MXFP8](mxfp8.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, Wan2.2-TI2V-5B | Validated on Ascend NPU and Intel XPU | -| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Ascend NPU only; TI2V-5B is not supported | +| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Validated on Ascend NPU; the ROCm gfx950 backend is integrated but not model-validated; TI2V-5B is unsupported | ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) @@ -59,6 +64,7 @@ from vllm_omni import Omni omni_fp8 = Omni(model="", quantization="fp8") omni_int8 = Omni(model="", quantization="int8") +omni_bnb = Omni(model="Tongyi-MAI/Z-Image-Turbo", quantization="bitsandbytes") omni_mxfp8 = Omni(model="", quantization="mxfp8") omni_mxfp4 = Omni(model="", quantization="mxfp4") omni_mxfp4_dualscale = Omni(model="", quantization="mxfp4_dualscale") @@ -69,6 +75,7 @@ CLI: ```bash vllm serve --omni --quantization fp8 vllm serve --omni --quantization int8 +vllm serve Tongyi-MAI/Z-Image-Turbo --omni --quantization bitsandbytes vllm serve --omni --quantization mxfp8 vllm serve --omni --quantization mxfp4 vllm serve --omni --quantization mxfp4_dualscale @@ -85,14 +92,19 @@ config = build_quant_config({ }) ``` +Component routing is an extension mechanism, not automatic support for every +pipeline component. See [Quantization Scope](overview.md#quantization-scope) +before applying it to an encoder, VAE, or decoder. + ## Parameters | Parameter | Methods | Description | |-----------|---------|-------------| -| `method` | FP8, Int8, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` | -| `ignored_layers` | FP8, Int8, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 | +| `method` | FP8, Int8, BitsAndBytes, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"bitsandbytes"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` | +| `ignored_layers` | FP8, Int8, BitsAndBytes, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 | | `activation_scheme` | FP8, Int8 | The runtime value `"dynamic"` selects online activation scaling | | `weight_block_size` | FP8 | Optional block-wise FP8 weight quantization size | +| `quant_type` | BitsAndBytes | Weight format: `"nf4"` (default) or `"fp4"` | | `num_bf16_fallback_layers` | MXFP4 DualScale | Leading transformer blocks to keep in BF16 for online `mxfp4_dualscale`; defaults to `5` | ## Validation and Notes diff --git a/docs/user_guide/quantization/overview.md b/docs/user_guide/quantization/overview.md index 9ceadda5c..1b4cac132 100644 --- a/docs/user_guide/quantization/overview.md +++ b/docs/user_guide/quantization/overview.md @@ -16,72 +16,123 @@ For the internal architecture and backend extension points, see the | Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion Flash Attention tensors during inference. | FP8 FA | | Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale | -## Hardware Support +Online quantization starts from a normal BF16/FP16 checkpoint and repeats the +conversion on each fresh model load. It is the simplest path for experiments. +Pre-quantized checkpoints store the packed weights and scales produced by a +separate tool; they are easier to reuse across deployments and can carry +calibrated or mixed-precision policies that load-time conversion cannot +reproduce. -| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | ModelOpt | MXFP8 W8A8 | MXFP4 W4A4 | AutoRound | msModelSlim | -|--------|----------|-----------|------------------|----------|------------|------------|-----------|-------------| -| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ | -| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ | -| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ⭕ | ✅ | ❌ | -| AMD ROCm | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ⭕ | ❌ | -| Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ✅ | ❌ | -| Ascend NPU | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | +## Support Levels -Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this -guide. FP8 on Ampere may use a weight-only path where available. +Quantization support is specific to a model, component, checkpoint format, and +hardware backend. The tables in this guide use the following levels: -## Model Type Support +| Level | Meaning | +|-------|---------| +| **CI-backed** | A named model and quantized checkpoint are exercised by scheduled full-model hardware CI. | +| **Validated** | A named model and checkpoint have end-to-end or quality-validation evidence, but the check is not necessarily scheduled in CI. | +| **Integrated** | The config, loader, quantized layers, or model wiring exists, but this guide has no completed end-to-end validation for the named model and checkpoint. | +| **Not validated** | There is no current support recommendation for this model/component combination. | +| **Unsupported** | The combination is intentionally rejected or known not to work. | -### Diffusion Model (Qwen-Image, Wan2.2) +A method being available on a device does not make every model on that device +supported. Likewise, successfully detecting a checkpoint's +`quantization_config` establishes loader compatibility, not output quality. -These models run a diffusion transformer as the primary inference module. The -default quantization target is the transformer; tokenizer, scheduler, text -encoder, and VAE stay on the base checkpoint unless a method guide says -otherwise. +## Hardware × Quantization Method -| Method | Guide | Mode | Example models | Status | -|--------|-------|------|----------------|--------| -| FP8 W8A8 | [FP8](fp8.md) | Online W8A8 or checkpoint FP8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models | -| Int8 W8A8 | [Int8](int8.md) | Online or serialized W8A8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image | -| BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Online W4 weight-only | Z-Image-Turbo; Qwen-Image/Wan2.2 not validated | Validated for Z-Image-Turbo | -| ModelOpt | [ModelOpt](modelopt.md) | Pre-quantized FP8 checkpoints | Qwen-Image, Z-Image, FLUX.2, HunyuanImage-3.0 | Validated for ModelOpt FP8 diffusion checkpoints | -| MXFP8 W8A8 | [MXFP8](mxfp8.md) | Online W8A8 or offline pre-quantized | Wan2.2-T2V-A14B, I2V-A14B, TI2V-5B | Ascend NPU only; validated for Wan2.2 | -| MXFP4 W4A4 | [MXFP4](mxfp4.md) | `mxfp4`: online single-scale only; `mxfp4_dualscale`: online or offline dual-scale (offline recommended) | Wan2.2-T2V-A14B, I2V-A14B | Ascend NPU only; validated for Wan2.2 A14B cascade models; TI2V-5B not supported; offline `mxfp4_dualscale` uses calibrated `mul_scale` for best accuracy | -| AutoRound | [AutoRound](autoround.md) | Pre-quantized W4A16 checkpoints | FLUX.1-dev; Qwen-Image/Wan2.2 not validated | Checkpoint-driven | -| msModelSlim | [msModelSlim](msmodelslim.md) | Pre-quantized Ascend checkpoints | Wan2.2 recipe; HunyuanImage-3.0 inference target | Ascend/NPU path | +This matrix uses concrete inference paths as columns. Producer toolkits are not +separate methods: NVIDIA ModelOpt produces the ModelOpt FP8/NVFP4 formats, +Intel AutoRound produces AutoRound checkpoints, and Ascend msModelSlim produces +checkpoints consumed by the native MXFP or compatible Ascend paths. -### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) +| Hardware | [Online FP8](fp8.md) | [Int8](int8.md) | [BitsAndBytes W4](bitsandbytes.md) | [ModelOpt FP8](modelopt.md) | [ModelOpt NVFP4](modelopt.md#supported-modelopt-checkpoint-formats) | [AutoRound W4A16](autoround.md) | [OCP MXFP8](mxfp8.md) | [OCP MXFP4](mxfp4.md) | +|----------|-----------------------|------------------|------------------------------------------|--------------------------------|-------------------------------------------------------------------------|--------------------------------------|--------------------------|--------------------------| +| NVIDIA Blackwell (SM 100+) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | +| NVIDIA Hopper (SM 90) | ✅ | ✅ | ✅ | ✅ | ✅ (Marlin fallback) | ✅ | ❌ | ❌ | +| NVIDIA Ada (SM 89) | ✅ | ✅ | ✅ | ✅ | ⭕ | ✅ | ❌ | ❌ | +| NVIDIA Ampere (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ | ❌ | +| AMD ROCm (gfx950 for MXFP4) | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ❌ | ✅ (online only) | +| Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ✅ | ❌ | +| Ascend NPU | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | -These models combine an AR language model with audio, vision, talker, or TTS -stages. Quantization is scoped to the AR language-model stage when the -checkpoint contains a supported `quantization_config`; the non-AR stages stay -in BF16 unless the model guide explicitly adds support. +Legend: `✅` backend available, `❌` rejected or no implementation, `⭕` not +verified in this guide. This matrix does not establish model-level support; use +the model-and-stage tables below for that. -| Method | Guide | Scope | Example models | Status | -|--------|-------|-------|----------------|--------| -| ModelOpt | [ModelOpt](modelopt.md) | Thinker or language-model checkpoint config | Qwen3-Omni thinker | ModelOpt checkpoint path | -| Int8 | [Int8](int8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated | -| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated | -| MXFP8 | [MXFP8](mxfp8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated | -| MXFP4 | [MXFP4](mxfp4.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated | -| AutoRound | [AutoRound](autoround.md) | Thinker or language-model checkpoint config | Qwen2.5-Omni, Qwen3-Omni | Supported through AutoRound checkpoints | -| msModelSlim | [msModelSlim](msmodelslim.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated | +The Int8 column covers online and serialized Int8 paths. OCP MXFP8 is online on +Intel XPU and online/offline on Ascend; Intel's separate AutoRound MXFP8 +checkpoint path is described in the [MXFP8 guide](mxfp8.md). OCP MXFP4 on AMD +means online single-scale `mxfp4` on gfx950, while MXFP4 DualScale is +Ascend-only. NVIDIA's NVFP4 and OCP MXFP4 have different checkpoint metadata, +scale layouts, and kernel contracts. Mixed ModelOpt FP8/NVFP4 checkpoints also +require a model-specific mixed-precision policy and validation. FP8 on Ampere +may use a weight-only path where available. -### Multi-Stage Diffusion Model (BAGEL, GLM-Image) +## Current Model and Stage Support -These models split generation across multiple stages. Quantization must be -attached to the intended stage rather than applied globally. +The current offline checkpoint guidance is transformer-first. A listed +checkpoint quantizes the stage named in the **Scope** column; it does not imply +that the model's encoders, VAE, decoder, or other stages are also quantized. -| Method | Guide | Scope | Example models | Status | -|--------|-------|-------|----------------|--------| -| FP8 | [FP8](fp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation | -| Int8 | [Int8](int8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation | -| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Stage-specific transformer or DiT module | BAGEL, GLM-Image | Not validated | -| ModelOpt | [ModelOpt](modelopt.md) | Checkpoint-defined diffusion stage | BAGEL, GLM-Image | Requires model-specific validation | -| MXFP8 | [MXFP8](mxfp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated | -| MXFP4 | [MXFP4](mxfp4.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated | -| AutoRound | [AutoRound](autoround.md) | Checkpoint-defined stage | BAGEL, GLM-Image | No validated checkpoint listed | -| msModelSlim | [msModelSlim](msmodelslim.md) | Ascend-generated stage weights | GLM-Image | Requires model-specific adaptation | +### Vendor and Runtime Ecosystems + +The producer and runtime backend are related, but they are not the same support +claim. A producer can emit several checkpoint formats, and each format still +needs a compatible vLLM-Omni loader, model-stage wiring, and validation. A +runtime may also provide online quantization without an offline producer path. + +| Ecosystem | Current paths | Current vLLM-Omni scope | Model evidence | +|-----------|---------------|-------------------------|----------------| +| NVIDIA | ModelOpt FP8, NVFP4, and mixed FP8/NVFP4 | Diffusion transformer or Qwen-Omni thinker | Qwen-Image and other named diffusion checkpoints are validated; Qwen3-Omni NVFP4 has H100 CI for the Marlin fallback | +| Intel | AutoRound W4A16 and AutoRound MXFP8 | Diffusion/world-model transformer or Qwen-Omni thinker | Wan A14B and GLM-Image W4A16 are CI-backed; Qwen-Image, Qwen-Omni, and Cosmos3-Super have validation outside scheduled CI | +| AMD ROCm | Online OCP MXFP4 through AITER on gfx950; no pre-quantized MXFP4 or AMD Quark workflow is integrated | Diffusion transformer linear layers | Integrated backend with dispatch and weight-allocation unit coverage; no named-model quality validation or scheduled gfx950 full-model CI | +| Ascend | msModelSlim output through native MXFP8/MXFP4 merge workflows or a compatible generic Ascend format | Wan diffusion transformers | Native Wan MXFP8/MXFP4 paths are validated outside scheduled full-model NPU CI; other model families remain integrated or unvalidated | + +### Pre-Quantized Model Matrix + +| Model family | Method | Scope | Level | Notes | +|--------------|--------|-------|-------|-------| +| Qwen-Image | [ModelOpt](modelopt.md) FP8 or mixed FP8/NVFP4 | Diffusion transformer | Validated | Named Qwen-Image 2512 checkpoints and recipes are available. Text encoder and VAE stay BF16. | +| Qwen-Image | [AutoRound](autoround.md) W4A16 | Diffusion transformer | Validated | Uses `INC4AI/Qwen-Image-AutoRound-W4A16`; text encoder and VAE stay BF16. | +| Wan2.2 I2V/T2V A14B | [AutoRound](autoround.md) W4A16 | Both diffusion transformers | CI-backed | The high-noise and low-noise DiTs are quantized; text encoder and VAE stay BF16. | +| Wan2.2 TI2V 5B | [AutoRound](autoround.md) W4A16 | Diffusion transformer | Validated | Named checkpoint available; not covered by the scheduled Wan AutoRound job. | +| Wan2.2 | [MXFP8](mxfp8.md) | One or both diffusion transformers | Validated | Native Ascend and AutoRound checkpoint formats are integrated; no scheduled full-model offline NPU job. | +| Wan2.2 I2V/T2V A14B | [MXFP4 DualScale](mxfp4.md) | Both diffusion transformers | Validated | Offline calibrated checkpoints are recommended; no scheduled full-model offline NPU job. | +| Qwen2.5-Omni | [AutoRound](autoround.md) W4A16 | Thinker language model | Validated | Encoders and output/audio stages stay BF16. | +| Qwen3-Omni | [ModelOpt](modelopt.md) FP8 | Thinker language model | Validated | Tested outside scheduled CI. | +| Qwen3-Omni | [ModelOpt](modelopt.md) NVFP4 | Thinker language model | CI-backed | H100 CI exercises the Marlin fallback; the native Blackwell FP4 path has separate recipe validation. | +| Qwen3-Omni | [AutoRound](autoround.md) W4A16 | Thinker language model | Validated | Audio encoder, vision encoder, talker, and Code2Wav stay BF16. | +| GLM-Image | [AutoRound](autoround.md) W4A16 | Diffusion transformer stage | CI-backed | Text-to-image and image-to-image full-model smoke coverage. | +| Cosmos3-Super | [AutoRound](autoround.md) W4A16 | World-model transformer | Validated | Manually validated serving path; VAE and other components stay BF16. | +| Cosmos3-Nano | [AutoRound](autoround.md) W4A16 | World-model transformer | Integrated | Named checkpoint is listed, but no model-specific end-to-end test is maintained in-tree. | +| MiniMax H3 | None | None | Not validated | No offline vendor checkpoint path is documented; online FP8 targets the DiT only. | + +The method pages contain the complete checkpoint lists, including FLUX, +Z-Image, and HunyuanImage-3.0 variants. + +### Online Quantization + +| Model family | Methods | Scope | Level | +|--------------|---------|-------|-------| +| Qwen-Image | FP8, Int8 | Diffusion transformer | Validated | +| Z-Image | Int8, BitsAndBytes W4 | Diffusion transformer | Validated | +| Wan2.2 | MXFP8 | One or both diffusion transformers | Validated on the documented XPU/NPU paths | +| Wan2.2 A14B | MXFP4 | Both diffusion transformers | Validated on Ascend; the ROCm gfx950 online backend is integrated but not model-validated | +| MiniMax H3 | FP8 | DiT and optional reference DiT | Validated | +| Qwen3-Omni | Generic online methods, such as FP8 | Thinker language model only | Integrated; not model-validated as an online recommendation | + +See [Online Quantization](online.md) for configuration and hardware details. + +### Unlisted and Long-Tail Models + +For models not listed above, generic quantized linear layers or compatible +checkpoint metadata are only an integration starting point. The default support +target is the model's primary transformer, DiT, or AR language-model stage. +Text/audio/vision encoders, VAEs, vocoders, decoders, and output stages remain +BF16 until a model-specific integration supplies quant-aware layers, checkpoint +mapping, end-to-end output validation, and backend coverage. !!! note "Online quantization" means vLLM-Omni computes the quantization data while @@ -92,8 +143,8 @@ attached to the intended stage rather than applied globally. ### Diffusion Model (Qwen-Image, Wan2.2) -The default target is the diffusion transformer. Component routing is available -through `build_quant_config()`: +The default target is the diffusion transformer. `build_quant_config()` can +construct a component router: ```python from vllm_omni.quantization import build_quant_config @@ -104,11 +155,19 @@ config = build_quant_config({ }) ``` +!!! warning "Routing syntax is not a support claim" + A component dictionary only selects a quantization config by module prefix. + It does not make an encoder or VAE quantizable. The model must construct + that component from quantization-aware layers, and an offline checkpoint + must contain matching packed weights and scales. Current user guidance + keeps diffusion text encoders and VAEs in BF16. + | Component | Default quantized? | Notes | |-----------|--------------------|-------| -| Diffusion transformer | Yes | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim | +| Diffusion transformer | Yes, for listed model/method pairs | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim | +| Second diffusion transformer | Yes, for listed Wan2.2 A14B methods | `transformer` and `transformer_2` read their own offline checkpoint metadata; arbitrary mixed methods are not currently validated | | Text encoder | No | Keep BF16 unless a method-specific guide documents support | -| VAE | No | Keep BF16; storage-only paths are method-specific | +| VAE | No | Keep BF16 | | Scheduler/tokenizer | No | Loaded from the base model repository | ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS) @@ -121,6 +180,18 @@ config = build_quant_config({ | Talker or TTS stage | No | BF16 unless model-specific support is documented | | Code2Wav | No | BF16 | +An explicit `ComponentQuantizationConfig` can be used by model developers to +route different configs to `audio_tower`, `visual`, and `language_model`, but +the supported Qwen-Omni checkpoints currently quantize only the thinker +language model. + +### World Model (Cosmos3) + +| Component | Default quantized? | Notes | +|-----------|--------------------|-------| +| World-model transformer | AutoRound checkpoints only | Cosmos3-Super is validated; Cosmos3-Nano is integrated but not end-to-end tested in-tree | +| VAE and guardrail components | No | BF16 | + ### Multi-Stage Diffusion Model (BAGEL, GLM-Image) | Component | Default quantized? | Notes | @@ -141,6 +212,7 @@ build_quant_config("fp8") build_quant_config({"method": "fp8", "activation_scheme": "static"}) build_quant_config("bitsandbytes") build_quant_config("auto-round", bits=4, group_size=128) +# Component routing syntax; model-specific support is still required. build_quant_config({"transformer": {"method": "fp8"}, "vae": None}) build_quant_config(None) ```