Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Hongsheng Liu
|
00dab2ac95 |
No files matched your search
+4
-1
@@ -14,7 +14,10 @@ A: At first, you can check current [issues](https://github.com/vllm-project/vllm
|
||||
|
||||
> Q: Does vLLM-Omni support AWQ or any other quantization?
|
||||
|
||||
A: We plan to introduce GGUF FP8 prequantized models and online FP8 quantization in version 0.16.0. Support for other quantization types will follow in future releases. For details, please see our [Q1 quantization roadmap](https://github.com/vllm-project/vllm-omni/issues/1057).
|
||||
A: AWQ is not currently listed as a validated vLLM-Omni path. Quantization
|
||||
support depends on the model, pipeline stage, checkpoint format, and hardware
|
||||
backend. See the current [quantization support matrix](../user_guide/quantization/overview.md)
|
||||
for online methods, pre-quantized checkpoints, and model-level validation.
|
||||
|
||||
> Q: Does vLLM-Omni support multimodal streaming input and output?
|
||||
|
||||
|
||||
@@ -92,7 +92,7 @@ current support.
|
||||
|--------|--------------|-------------|----------|
|
||||
| **[FP8](quantization/fp8.md)** | `quantization="fp8"` | FP8 W8A8 on validated transformer stages | Memory reduction, inference speedup |
|
||||
| **[INT8](quantization/int8.md)** | `quantization="int8"` | INT8 W8A8 on validated transformer stages | Memory reduction, broad GPU compatibility |
|
||||
| **[GGUF](quantization/gguf.md)** | `quantization="gguf"` | Native GGUF transformer-only weights (Q4, Q8, etc.) | Memory reduction on consumer GPUs |
|
||||
| **[BitsAndBytes W4](quantization/bitsandbytes.md)** | `quantization="bitsandbytes"` | Online NF4/FP4 weight-only quantization on validated CUDA transformer stages | Reducing transformer weight memory without preparing a separate checkpoint |
|
||||
|
||||
## Supported Models
|
||||
|
||||
|
||||
@@ -143,7 +143,8 @@ python text_to_video.py \
|
||||
- `--enable-layerwise-offload`: enable layerwise offloading on DiT modules.
|
||||
- `--frame-rate`: generation FPS for pipelines that require it (e.g., LTX2).
|
||||
- `--audio-sample-rate`: fallback audio sample rate when the pipeline returns audio.
|
||||
- `--quantization`: quantization method (`fp8` for FP8, `gguf` for GGUF).
|
||||
- `--quantization`: online quantization method, for example `fp8`; supported
|
||||
pre-quantized checkpoints are normally auto-detected from checkpoint metadata.
|
||||
- `--flow-shift`: scheduler flow_shift parameter.
|
||||
- `--lora-path`: path to PEFT LoRA adapter folder or checkpoint file.
|
||||
- `--lora-scale`: scale factor for LoRA weights.
|
||||
|
||||
@@ -90,8 +90,8 @@ vllm serve "Qwen/Qwen-Image" \
|
||||
--quantization-config '{"method": "fp8"}'
|
||||
```
|
||||
|
||||
Other vLLM-Omni quantization methods, such as `gguf`, `modelopt`, `mxfp4`,
|
||||
`mxfp8`, serialized checkpoints, static FP8 configs, and layer-name skip lists
|
||||
Other vLLM-Omni quantization methods, such as `modelopt`, `mxfp4`, `mxfp8`,
|
||||
serialized checkpoints, static FP8 configs, and layer-name skip lists
|
||||
such as `ignored_layers`, are intentionally not translated by this compatibility
|
||||
shortcut. Use Diffusers-native configuration through `diffusers_load_kwargs` or
|
||||
a native vLLM-Omni pipeline for those cases.
|
||||
|
||||
@@ -10,6 +10,10 @@ checkpoint's `config.json` and auto-detects
|
||||
AutoRound is static quantization: no `--quantization` flag is needed at
|
||||
inference time when the checkpoint already contains the quantization config.
|
||||
|
||||
The validation terms on this page follow the
|
||||
[support levels](overview.md#support-levels) defined in the overview. A listed
|
||||
checkpoint is not enough by itself to establish end-to-end model support.
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | Support |
|
||||
@@ -21,46 +25,57 @@ inference time when the checkpoint already contains the quantization config.
|
||||
| Intel XPU | ✅ |
|
||||
| Ascend NPU | ❌ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
|
||||
guide. AutoRound is Intel-supported.
|
||||
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
|
||||
guide. This table is method-level backend availability. AutoRound is an Intel
|
||||
project; vLLM selects a compatible CUDA or XPU compute backend at load time.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
### Diffusion Model (Qwen-Image, Wan2.2)
|
||||
|
||||
| Model | Checkpoint | Scope | Scheme | Backend |
|
||||
|-------|------------|-------|--------|---------|
|
||||
| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Qwen-Image | Not listed | Diffusion transformer | W4A16 | Not validated |
|
||||
| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Model | Checkpoint | Scope | Scheme | Validation |
|
||||
|-------|------------|-------|--------|------------|
|
||||
| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | Validated |
|
||||
| Qwen-Image | `INC4AI/Qwen-Image-AutoRound-W4A16` | Diffusion transformer | W4A16 | Validated with end-to-end generation and quality comparison |
|
||||
| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
|
||||
| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
|
||||
| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | Validated; not in the scheduled Wan AutoRound job |
|
||||
|
||||
CUDA execution uses GPTQ-Marlin or another compatible vLLM AutoRound backend;
|
||||
Intel XPU uses the Intel-supported backend. The text encoder and VAE stay in
|
||||
BF16 for all entries above.
|
||||
|
||||
### World Model (Cosmos3)
|
||||
|
||||
| Model | Checkpoint | Scope | Scheme | Backend |
|
||||
|-------|------------|-------|--------|---------|
|
||||
| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Model | Checkpoint | Scope | Scheme | Validation |
|
||||
|-------|------------|-------|--------|------------|
|
||||
| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | Integrated; no model-specific end-to-end test is maintained in-tree |
|
||||
| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | Validated with a manual serving run |
|
||||
|
||||
The Cosmos VAE and guardrail components stay in BF16.
|
||||
|
||||
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
||||
|
||||
| Model | Checkpoint | Scope | Scheme | Backend |
|
||||
|-------|------------|-------|--------|---------|
|
||||
| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Language-model stage | W4A16 | AutoRound |
|
||||
| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | AutoRound |
|
||||
| Model | Checkpoint | Scope | Scheme | Validation |
|
||||
|-------|------------|-------|--------|------------|
|
||||
| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
|
||||
| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
|
||||
| Qwen3-TTS | Not listed | TTS language-model stage | W4A16 | Not validated |
|
||||
|
||||
AutoRound support is checkpoint-driven. A model is supported when its
|
||||
checkpoint uses a compatible INC/AutoRound config and the target stage maps to
|
||||
vLLM-Omni's runtime module names.
|
||||
|
||||
For Qwen-Omni, only the thinker language model uses the AutoRound checkpoint.
|
||||
Audio and vision encoders, the talker, and waveform-decoder stages stay in
|
||||
BF16.
|
||||
|
||||
### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
|
||||
|
||||
| Model | Scope | Status | Notes |
|
||||
|-------|-------|--------|-------|
|
||||
| GLM-Image | Diffusion transformer | ✅ | `Intel/GLM-Image-int4-AutoRound` |
|
||||
| BAGEL | Checkpoint-defined diffusion or transformer stage | Not validated | Requires a compatible AutoRound checkpoint |
|
||||
| Model | Checkpoint | Scope | Validation |
|
||||
|-------|------------|-------|------------|
|
||||
| GLM-Image | `Intel/GLM-Image-int4-AutoRound` | Diffusion transformer | CI-backed |
|
||||
| BAGEL | Not listed | Checkpoint-defined diffusion or transformer stage | Not validated |
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -119,6 +134,14 @@ The checkpoint should contain a config like:
|
||||
At load time, vLLM-Omni builds an `OmniINCConfig`, remaps checkpoint block names
|
||||
to runtime module names, and selects the matching vLLM compute backend.
|
||||
|
||||
Checkpoint auto-detection and successful layer construction establish the
|
||||
**Integrated** level. Promote a model to **Validated** only after the named
|
||||
checkpoint completes an end-to-end generation or multimodal request and its
|
||||
output passes a model-appropriate sanity or quality check. Compare against the
|
||||
BF16 baseline for quality-sensitive image and video paths. Promote it to
|
||||
**CI-backed** only when that named checkpoint is selected by scheduled hardware
|
||||
CI.
|
||||
|
||||
Example checkpoint creation:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -9,6 +9,10 @@ stay in BF16/FP16. No pre-quantized checkpoint is required.
|
||||
This is an online (dynamic) path: load the normal HuggingFace checkpoint and
|
||||
quantize during model loading via the `bitsandbytes` CUDA kernels.
|
||||
|
||||
The validation terms on this page follow the
|
||||
[support levels](overview.md#support-levels). BitsAndBytes kernel support does
|
||||
not imply that every diffusion transformer has been quality-validated.
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | Support |
|
||||
@@ -21,7 +25,7 @@ quantize during model loading via the `bitsandbytes` CUDA kernels.
|
||||
| Intel XPU | ❌ |
|
||||
| Ascend NPU | ❌ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported. Non-CUDA platforms raise in
|
||||
Legend: `✅` backend available, `❌` unsupported. Non-CUDA platforms raise in
|
||||
`get_quant_method()`; CUDA requires compute capability 7.5+ (`SM 75+`).
|
||||
|
||||
Requires the optional `bitsandbytes` package (`pip install bitsandbytes`).
|
||||
@@ -30,11 +34,11 @@ Requires the optional `bitsandbytes` package (`pip install bitsandbytes`).
|
||||
|
||||
### Diffusion Model (Qwen-Image, Wan2.2)
|
||||
|
||||
| Model | HF models | CUDA | Mode | Recommendation |
|
||||
|-------|-----------|:----:|------|----------------|
|
||||
| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Yes | Online W4 weight-only | All heavy linear layers; sensitive embedders stay BF16 |
|
||||
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Not validated | Online W4 weight-only | Compare vs BF16 before enabling |
|
||||
| Wan2.2 | Wan2.2 diffusion pipelines | Not validated | Online W4 weight-only | Validate before enabling in docs |
|
||||
| Model | HF models | Scope | Level | Recommendation |
|
||||
|-------|-----------|-------|-------|----------------|
|
||||
| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Diffusion transformer | Validated | All heavy linear layers; sensitive embedders stay BF16 |
|
||||
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Diffusion transformer | Not validated | Compare with BF16 before enabling |
|
||||
| Wan2.2 | Wan2.2 diffusion pipelines | One or both diffusion transformers | Not validated | Validate quality and cascade behavior before enabling |
|
||||
|
||||
Other diffusion models may work if their transformer uses supported linear
|
||||
layers, but they are not validated in this guide.
|
||||
@@ -104,6 +108,10 @@ Multi-GPU tensor parallelism (`tensor_parallel_size` > 1) is not validated for
|
||||
BitsAndBytes in diffusion models; each rank quantizes its own weight shard
|
||||
independently.
|
||||
|
||||
Z-Image-Turbo is validated outside scheduled full-model BitsAndBytes CI. The
|
||||
real CUDA kernel smoke tests cover quantization and matrix multiplication, but
|
||||
do not establish model-level quality for unlisted architectures.
|
||||
|
||||
If quality regresses, use `ignored_layers` to keep sensitive projections in BF16
|
||||
(for example `to_out` or `w2`).
|
||||
|
||||
|
||||
@@ -13,6 +13,11 @@ components, such as the tokenizer, scheduler, text encoder, vision/audio
|
||||
encoder, and VAE, on the base checkpoint unless a model-specific guide says
|
||||
otherwise.
|
||||
|
||||
The validation terms on this page follow the
|
||||
[support levels](overview.md#support-levels) defined in the overview. ModelOpt
|
||||
format detection and checkpoint adaptation are shared runtime integrations;
|
||||
each model and stage still requires its own end-to-end validation.
|
||||
|
||||
!!! note
|
||||
ModelOpt checkpoints are pre-quantized checkpoints. Do not pass
|
||||
`--quantization fp8` for these checkpoints. The checkpoint
|
||||
@@ -58,11 +63,11 @@ stay unquantized unless the model-specific recipe validates otherwise.
|
||||
| Intel XPU | ⭕ |
|
||||
| Ascend NPU | ❌ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
|
||||
The optional CUTLASS FP8 runtime override requires CUDA SM89+. ModelOpt NVFP4
|
||||
and mixed FP8/NVFP4 diffusion checkpoints are currently validated on Blackwell
|
||||
CUDA systems in the recipes below; other CUDA generations require separate
|
||||
backend and quality validation.
|
||||
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
|
||||
guide. The optional CUTLASS FP8 runtime override requires CUDA SM89+. ModelOpt
|
||||
NVFP4 and mixed FP8/NVFP4 diffusion checkpoints are currently validated on
|
||||
Blackwell CUDA systems in the recipes below; other CUDA generations require
|
||||
separate backend and quality validation.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
@@ -79,6 +84,9 @@ backend and quality validation.
|
||||
| HunyuanImage-3.0 | `feizhai123/hunyuan-image3-modelopt-mixed-experts-nvfp4-dense-fp8` | MoE diffusion transformer | Validated for ModelOpt mixed FP8/NVFP4 checkpoints |
|
||||
| Wan2.2 | Not available | Diffusion transformer | Not validated |
|
||||
|
||||
The validated diffusion entries above have named checkpoint and recipe
|
||||
evidence. They are not all exercised by scheduled full-model CI.
|
||||
|
||||
For full serving commands and benchmark context, see
|
||||
[`recipes/Qwen/Qwen-Image.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen-Image.md)
|
||||
and
|
||||
@@ -88,8 +96,8 @@ and
|
||||
|
||||
| Model | Scope | Status |
|
||||
|-------|-------|--------|
|
||||
| Qwen3-Omni | Thinker language-model stage | ModelOpt FP8 checkpoint path |
|
||||
| Qwen3-Omni | Thinker language-model stage (W4A4 NVFP4) | Validated; see [Qwen3-Omni NVFP4 W4A4](#qwen3-omni-nvfp4-w4a4-thinker) below |
|
||||
| Qwen3-Omni | Thinker language-model stage (FP8) | Validated outside scheduled CI |
|
||||
| Qwen3-Omni | Thinker language-model stage (W4A4 NVFP4) | CI-backed for the H100 Marlin fallback; native Blackwell path is recipe-validated |
|
||||
| Qwen3-TTS | TTS language-model stage | Not validated |
|
||||
|
||||
Audio encoder, vision encoder, talker, and code2wav stages stay in BF16 unless
|
||||
@@ -102,9 +110,9 @@ Qwen3-Omni-30B-A3B-Instruct thinker language model. The thinker text body
|
||||
(attention + MoE experts) is quantized to NVFP4 with FP8 per-tensor input
|
||||
scales; the audio encoder, vision encoder, talker, and code2wav stay in BF16.
|
||||
|
||||
| Variant | HF checkpoint | Hardware |
|
||||
|---------|---------------|----------|
|
||||
| W4A4 NVFP4 (full thinker) | `YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip` | sm_100+ (Blackwell, FlashInfer FP4 GEMM) |
|
||||
| Variant | HF checkpoint | Hardware and validation |
|
||||
|---------|---------------|-------------------------|
|
||||
| W4A4 NVFP4 (full thinker) | `YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip` | sm_100+ native FlashInfer FP4 GEMM; H100 Marlin fallback is CI-backed |
|
||||
|
||||
Calibration uses ModelOpt `mtq.NVFP4_DEFAULT_CFG` with the `awq_clip` algorithm
|
||||
on 1024 ultrachat samples chat-templated through the Qwen3-Omni tokenizer.
|
||||
@@ -232,3 +240,6 @@ omni = Omni(
|
||||
5. Keep `--quantization fp8` for online FP8 from BF16 checkpoints; use this
|
||||
ModelOpt path only when the checkpoint already contains ModelOpt quantized
|
||||
weights and scales.
|
||||
6. The Qwen3-Omni full-model CI smoke exercises the H100 Marlin fallback. A
|
||||
scheduled Blackwell test is still required before treating the native
|
||||
FlashInfer FP4 kernel path as CI-backed.
|
||||
@@ -3,12 +3,17 @@
|
||||
## Overview
|
||||
|
||||
[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend compression
|
||||
toolkit for producing pre-quantized model checkpoints. In vLLM-Omni, these
|
||||
checkpoints run through the Ascend/NPU path with `--quantization ascend`.
|
||||
toolkit for producing pre-quantized model checkpoints. vLLM-Omni consumes its
|
||||
output either through a compatible generic Ascend checkpoint path or through
|
||||
the native MXFP8 and MXFP4 merge workflows documented in this guide.
|
||||
|
||||
msModelSlim is static quantization: quantized weights are generated offline
|
||||
before vLLM-Omni inference starts.
|
||||
|
||||
The validation terms on this page follow the
|
||||
[support levels](overview.md#support-levels). A producer recipe or hardware
|
||||
backend alone is an integration point, not model-level validation.
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | Support |
|
||||
@@ -20,23 +25,28 @@ before vLLM-Omni inference starts.
|
||||
| Intel XPU | ❌ |
|
||||
| Ascend NPU | ✅ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
|
||||
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
|
||||
guide.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
### Diffusion Model (Qwen-Image, Wan2.2)
|
||||
|
||||
| Model | Base model | Scope | Hardware | Notes |
|
||||
|-------|------------|-------|----------|-------|
|
||||
| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Ascend NPU | Upstream msModelSlim provides a Wan2.2 quantization recipe; vLLM-Omni inference validation is not listed |
|
||||
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Ascend NPU | Not validated in this guide |
|
||||
| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Ascend A2/A3 NPU | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation |
|
||||
| Model | Base model | Scope | Level | Notes |
|
||||
|-------|------------|-------|-------|-------|
|
||||
| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Integrated | Upstream msModelSlim provides a producer recipe; use the [MXFP8](mxfp8.md) or [MXFP4](mxfp4.md) workflow for vLLM-Omni's native merged formats |
|
||||
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Not validated | No supported checkpoint workflow is documented |
|
||||
| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Integrated | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation and validate against BF16 |
|
||||
|
||||
Public Hugging Face quantized weights are not available yet. Use the
|
||||
[HunyuanImage-3.0 msModelSlim adaptation](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8)
|
||||
to generate the checkpoint manually.
|
||||
|
||||
msModelSlim is also the producer used by the native offline MXFP8 and MXFP4
|
||||
DualScale workflows. Those workflows preprocess and annotate each transformer
|
||||
checkpoint and are auto-detected at inference; follow their method pages rather
|
||||
than passing `--quantization ascend`.
|
||||
|
||||
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
||||
|
||||
| Model | Scope | Status | Notes |
|
||||
@@ -53,25 +63,32 @@ to generate the checkpoint manually.
|
||||
|
||||
## Configuration
|
||||
|
||||
For a compatible generic Ascend checkpoint, select the Ascend quantization
|
||||
method explicitly:
|
||||
|
||||
Offline inference:
|
||||
|
||||
```bash
|
||||
python text_to_image.py --model <quantized-model-path> --quantization ascend
|
||||
python text_to_image.py --model <compatible-ascend-checkpoint> --quantization ascend
|
||||
```
|
||||
|
||||
Online serving:
|
||||
|
||||
```bash
|
||||
vllm serve <quantized-model-path> --omni --quantization ascend
|
||||
vllm serve <compatible-ascend-checkpoint> --omni --quantization ascend
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `--quantization` | str | - | Use `ascend` for msModelSlim-produced checkpoints |
|
||||
| `--quantization` | str | - | Use `ascend` only for a compatible generic Ascend-format checkpoint; omit it for native merged MXFP8/MXFP4 checkpoints |
|
||||
| `model` | str | - | Path to the quantized checkpoint generated by Ascend tooling |
|
||||
|
||||
Do not pass `--quantization ascend` for a Wan checkpoint processed by the native
|
||||
MXFP8 or MXFP4 merge tools. Their transformer metadata selects `mxfp8` or
|
||||
`mxfp4_dualscale` automatically.
|
||||
|
||||
Example msModelSlim command for a Wan2.2 W8A8 checkpoint:
|
||||
|
||||
```bash
|
||||
@@ -93,3 +110,6 @@ For HunyuanImage-3.0, use the Hunyuan-specific adaptation linked above.
|
||||
tooling; it is not a load-time CUDA quantizer.
|
||||
3. Keep the quantized checkpoint aligned with the same model architecture and
|
||||
stage config used for BF16 inference.
|
||||
4. No named full-model msModelSlim checkpoint is currently scheduled in NPU
|
||||
CI. Treat new producer outputs as **Integrated** until an end-to-end BF16
|
||||
comparison establishes **Validated** support.
|
||||
@@ -13,6 +13,10 @@ vLLM-Omni provides two quantization methods with different scale structures:
|
||||
| `mxfp4` | Single-scale (per-32 fine only) | Online only | Quick accuracy baseline; no checkpoint prep needed |
|
||||
| `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | Production; better accuracy; offline recommended |
|
||||
|
||||
The validation terms on this page follow the
|
||||
[support levels](overview.md#support-levels). Current model guidance covers
|
||||
Wan diffusion transformers only; the text encoder and VAE stay in BF16.
|
||||
|
||||
!!! tip "Recommended: `mxfp4_dualscale` offline"
|
||||
For production deployments, use the `mxfp4_dualscale` offline mode with a
|
||||
pre-quantized checkpoint produced by msModelSlim. Offline checkpoints load
|
||||
@@ -34,26 +38,45 @@ vLLM-Omni provides two quantization methods with different scale structures:
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | Support |
|
||||
|--------|---------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ⭕ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ⭕ |
|
||||
| AMD ROCm (gfx950 / MI355X) | ✅ |
|
||||
| Intel XPU | ⭕ |
|
||||
| Ascend NPU (Atlas 950 A5) | ✅ |
|
||||
| Device | Online `mxfp4` | Online `mxfp4_dualscale` | Offline `mxfp4_dualscale` |
|
||||
|--------|-----------------|---------------------------|----------------------------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ❌ | ❌ | ❌ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ | ❌ | ❌ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ❌ | ❌ | ❌ |
|
||||
| AMD ROCm (gfx950 / MI355X) | ✅ | ❌ | ❌ |
|
||||
| Intel XPU | ❌ | ❌ | ❌ |
|
||||
| Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ✅ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
|
||||
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
|
||||
guide. Backend availability does not imply full-model checkpoint validation.
|
||||
|
||||
!!! note "Why NVIDIA is marked unsupported"
|
||||
This table describes the methods currently wired into vLLM-Omni, not every
|
||||
low-precision format NVIDIA hardware could execute. `DiffusionMXFP4Config`
|
||||
has runtime implementations only for Ascend and ROCm gfx950; on CUDA it
|
||||
raises `NotImplementedError` because no CUDA MXFP4 linear method or
|
||||
checkpoint loader is registered.
|
||||
|
||||
On NVIDIA, use [ModelOpt](modelopt.md) NVFP4 for the documented 4-bit
|
||||
checkpoint path. NVFP4 and OCP MXFP4 have different checkpoint metadata,
|
||||
scale layouts, and kernel contracts, so they are not interchangeable. A
|
||||
native CUDA MXFP4 path would require a CUDA linear implementation, loader
|
||||
contract, and model-level output validation.
|
||||
|
||||
The ROCm path dispatches online single-scale `mxfp4` to AITER on gfx950. It has
|
||||
kernel and dispatch coverage, but this guide does not yet claim end-to-end
|
||||
quality validation for a named Wan checkpoint on ROCm. DualScale and
|
||||
pre-quantized MXFP4 checkpoints are not supported on ROCm.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
### Diffusion Model (Wan2.2)
|
||||
|
||||
| Model | Online | Offline | Notes |
|
||||
|-------|--------|---------|-------|
|
||||
| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade (`transformer` + `transformer_2`); both transformers quantized with the same config |
|
||||
| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade; same scheme as T2V-A14B |
|
||||
| Wan2.2-TI2V-5B | ❌ | ❌ | Parameter count too small; W4A4 causes unacceptable accuracy loss |
|
||||
| Model | Online (Ascend) | Offline (Ascend) | Validation and notes |
|
||||
|-------|-----------------|------------------|----------------------|
|
||||
| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | Validated outside scheduled full-model offline NPU CI; both cascade transformers are quantized |
|
||||
| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | Validated outside scheduled full-model offline NPU CI; same cascade scheme as T2V-A14B |
|
||||
| Wan2.2-TI2V-5B | ❌ | ❌ | Unsupported because W4A4 causes unacceptable accuracy loss |
|
||||
|
||||
The choice between `mxfp4` and `mxfp4_dualscale` in **online mode** is about
|
||||
quantization quality, not model compatibility — both work on cascade (A14B) and
|
||||
@@ -83,12 +106,13 @@ by the merge script); there is no offline `mxfp4` single-scale format.
|
||||
|
||||
### `mxfp4` — Single-Scale Online Mode
|
||||
|
||||
Online mode requires no pre-processing. vLLM-Omni quantizes BF16 weights to
|
||||
MXFP4 at load time using `npu_dynamic_mx_quant`. A single block scale
|
||||
Online mode requires no pre-processing. On Ascend, vLLM-Omni quantizes BF16
|
||||
weights at load time using `npu_dynamic_mx_quant`. On ROCm gfx950, the same
|
||||
`mxfp4` config dispatches to the AITER online kernel. A single block scale
|
||||
(`float8_e8m0fnu`, one per 32 K elements) is computed on the fly; no
|
||||
calibration `mul_scale` is available. Applies equally to single-transformer
|
||||
and cascade (A14B) models — both transformers in a cascade receive the same
|
||||
quantization config automatically.
|
||||
calibration `mul_scale` is available. The config applies equally to
|
||||
single-transformer and cascade (A14B) models, but named Wan model validation is
|
||||
currently limited to the documented Ascend path.
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
@@ -111,10 +135,10 @@ vllm serve <your-model> --omni --quantization mxfp4
|
||||
|
||||
### `mxfp4_dualscale` — DualScale Online Mode
|
||||
|
||||
Online DualScale mode computes both fine and coarse scales on the fly from BF16
|
||||
weights using `npu_dynamic_dual_level_mx_quant`. Applies equally to
|
||||
single-transformer and cascade (A14B) models. Compared to `mxfp4` online,
|
||||
DualScale provides better quantization accuracy at higher compute cost.
|
||||
Ascend-only online DualScale mode computes both fine and coarse scales on the
|
||||
fly from BF16 weights using `npu_dynamic_dual_level_mx_quant`. It applies
|
||||
equally to single-transformer and cascade (A14B) models. Compared to `mxfp4`
|
||||
online, DualScale provides better quantization accuracy at higher compute cost.
|
||||
|
||||
The default configuration keeps the leading 5 transformer blocks in BF16
|
||||
(`num_bf16_fallback_layers=5`). Accuracy evaluation on Wan2.2-A14B shows this
|
||||
@@ -341,6 +365,11 @@ unless they appear in `ignored_layers`.
|
||||
quantization. Use `ignored_layers` and `num_bf16_fallback_layers` to trade
|
||||
off compression vs. accuracy for precision-sensitive layers.
|
||||
|
||||
8. The native methods, merge mappings, and per-transformer cascade metadata
|
||||
have unit coverage. The current Buildkite configuration does not schedule a
|
||||
named full-model offline MXFP4 checkpoint on NPU, so this path is
|
||||
**Validated**, not **CI-backed**.
|
||||
|
||||
## Adapting MXFP4 for a New Model
|
||||
|
||||
This section is aimed at developers who want to add MXFP4 support to a model
|
||||
|
||||
@@ -15,30 +15,45 @@ This method supports three modes:
|
||||
| **Offline (Native)** | msModelSlim-exported MXFP8 weights converted to diffusers format via `merge_mxfp8_checkpoint.py` — weights and scales are loaded directly from the preprocessed checkpoint |
|
||||
| **Offline (AutoRound)** | AutoRound MXFP8 checkpoints with `data_type="mx_fp"` — auto-detected from `config.json` |
|
||||
|
||||
The validation terms on this page follow the
|
||||
[support levels](overview.md#support-levels). Current model guidance covers
|
||||
Wan diffusion transformers only; the text encoder and VAE stay in BF16.
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | Online | Offline (Native) | Offline (AutoRound) |
|
||||
|--------|--------|------------------|---------------------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ⭕ | ⭕ | ⭕ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ | ⭕ | ⭕ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ⭕ | ⭕ | ⭕ |
|
||||
| AMD ROCm | ⭕ | ⭕ | ⭕ |
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ❌ | ❌ | ⭕ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ | ❌ | ⭕ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ❌ | ❌ | ⭕ |
|
||||
| AMD ROCm | ❌ | ❌ | ⭕ |
|
||||
| Intel XPU | ✅ | ❌ | ✅ |
|
||||
| Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ⭕ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
|
||||
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
|
||||
guide. Backend availability does not imply full-model checkpoint validation.
|
||||
|
||||
**Note**: Intel XPU only supports AutoRound MXFP8 for offline mode. Use AutoRound quantized checkpoints or online mode for XPU.
|
||||
The native `DiffusionMXFP8Config` currently dispatches only to Intel XPU and
|
||||
Ascend NPU implementations. CUDA reaches `NotImplementedError`, so the NVIDIA
|
||||
rows are marked unsupported even though this is an implementation gap rather
|
||||
than a general claim about NVIDIA hardware capabilities. On NVIDIA, use
|
||||
[FP8](fp8.md) for load-time quantization or [ModelOpt](modelopt.md) FP8 for
|
||||
pre-quantized checkpoints. Adding native CUDA MXFP8 requires a registered CUDA
|
||||
linear method, a compatible checkpoint contract, and model-level validation.
|
||||
|
||||
Native `mxfp8` is available online on Intel XPU and online or offline on
|
||||
Ascend. Intel XPU only supports AutoRound MXFP8 for offline mode. The AutoRound
|
||||
column is a separate checkpoint format and compute path.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
### Diffusion Model (Wan2.2)
|
||||
|
||||
| Model | Mode | Notes |
|
||||
|-------|------|-------|
|
||||
| Wan2.2-T2V-A14B | Online + Offline | MoE cascade; quantizes two transformers (`transformer` + `transformer_2`) |
|
||||
| Wan2.2-I2V-A14B | Online + Offline | MoE cascade; quantizes two transformers (`transformer` + `transformer_2`) |
|
||||
| Wan2.2-TI2V-5B | Online + Offline | Single transformer |
|
||||
| Model | Mode | Scope | Validation |
|
||||
|-------|------|-------|------------|
|
||||
| Wan2.2-T2V-A14B | Online + Offline | MoE cascade: `transformer` + `transformer_2` | Validated outside scheduled full-model offline NPU CI |
|
||||
| Wan2.2-I2V-A14B | Online + Offline | MoE cascade: `transformer` + `transformer_2` | Validated outside scheduled full-model offline NPU CI |
|
||||
| Wan2.2-TI2V-5B | Online + Offline | Single diffusion transformer | Validated outside scheduled full-model offline NPU CI |
|
||||
|
||||
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
||||
|
||||
@@ -218,3 +233,7 @@ omni = Omni(model="<autoround-mxfp8-model>")
|
||||
`--output-dir`), regenerate it with the current script. The old script
|
||||
wrote a separate `quantization_config.json` that is not read by vLLM-Omni;
|
||||
the current script injects the config directly into `transformer/config.json`.
|
||||
4. The native loader, merge mappings, and Wan cascade config propagation have
|
||||
unit coverage. The current Buildkite configuration does not schedule a
|
||||
named full-model offline MXFP8 checkpoint on NPU, so this path is
|
||||
**Validated**, not **CI-backed**.
|
||||
@@ -6,7 +6,7 @@ Online quantization means vLLM-Omni computes quantized weights and scales while
|
||||
loading the model. Use it when you want memory savings without preparing a
|
||||
separate quantized checkpoint.
|
||||
|
||||
This mode is different from pre-quantized checkpoint formats such as GGUF,
|
||||
This mode is different from pre-quantized checkpoint formats such as ModelOpt,
|
||||
AutoRound, msModelSlim, or serialized Int8 checkpoints. Those formats are
|
||||
prepared before serving and are documented in their method-specific guides.
|
||||
For MXFP8 and MXFP4, use this page for load-time quantization from BF16
|
||||
@@ -15,18 +15,22 @@ by msModelSlim and the merge tools.
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | FP8 W8A8 | Int8 W8A8 | MXFP8 W8A8 | MXFP4 W4A4 |
|
||||
|--------|----------|-----------|------------|------------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ⭕ | ⭕ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ⭕ | ⭕ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ⭕ | ⭕ |
|
||||
| AMD ROCm | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Intel XPU | ⭕ | ⭕ | ✅ | ⭕ |
|
||||
| Ascend NPU | ❌ | ✅ | ✅ | ✅ |
|
||||
| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | MXFP8 W8A8 | MXFP4 W4A4 |
|
||||
|--------|----------|-----------|------------------|------------|------------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| AMD ROCm (gfx950) | ⭕ | ⭕ | ❌ | ❌ | ✅ |
|
||||
| Intel XPU | ⭕ | ⭕ | ❌ | ✅ | ❌ |
|
||||
| Ascend NPU | ❌ | ✅ | ❌ | ✅ | ✅ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
|
||||
guide. FP8 on Ampere may use a weight-only path where available. MXFP8 and
|
||||
MXFP4 are documented for the Ascend NPU path.
|
||||
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
|
||||
guide. These entries describe method-level backend availability, not validation
|
||||
for every model. FP8 on Ampere may use a weight-only path where available.
|
||||
On ROCm gfx950, MXFP4 means online single-scale `mxfp4`; online and offline
|
||||
`mxfp4_dualscale` are Ascend-only. The native MXFP8/MXFP4 configs do not
|
||||
currently register CUDA implementations; use FP8 or ModelOpt FP8/NVFP4 on
|
||||
NVIDIA instead.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
@@ -36,8 +40,9 @@ MXFP4 are documented for the Ascend NPU path.
|
||||
|--------|-------|----------------|--------|
|
||||
| FP8 W8A8 | [FP8](fp8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
|
||||
| Int8 W8A8 | [Int8](int8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
|
||||
| BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Z-Image-Turbo; Qwen-Image and Wan2.2 are not validated | Validated for Z-Image-Turbo |
|
||||
| MXFP8 W8A8 | [MXFP8](mxfp8.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, Wan2.2-TI2V-5B | Validated on Ascend NPU and Intel XPU |
|
||||
| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Ascend NPU only; TI2V-5B is not supported |
|
||||
| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Validated on Ascend NPU; the ROCm gfx950 backend is integrated but not model-validated; TI2V-5B is unsupported |
|
||||
|
||||
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
||||
|
||||
@@ -59,6 +64,7 @@ from vllm_omni import Omni
|
||||
|
||||
omni_fp8 = Omni(model="<your-model>", quantization="fp8")
|
||||
omni_int8 = Omni(model="<your-model>", quantization="int8")
|
||||
omni_bnb = Omni(model="Tongyi-MAI/Z-Image-Turbo", quantization="bitsandbytes")
|
||||
omni_mxfp8 = Omni(model="<your-model>", quantization="mxfp8")
|
||||
omni_mxfp4 = Omni(model="<your-model>", quantization="mxfp4")
|
||||
omni_mxfp4_dualscale = Omni(model="<your-model>", quantization="mxfp4_dualscale")
|
||||
@@ -69,6 +75,7 @@ CLI:
|
||||
```bash
|
||||
vllm serve <your-model> --omni --quantization fp8
|
||||
vllm serve <your-model> --omni --quantization int8
|
||||
vllm serve Tongyi-MAI/Z-Image-Turbo --omni --quantization bitsandbytes
|
||||
vllm serve <your-model> --omni --quantization mxfp8
|
||||
vllm serve <your-model> --omni --quantization mxfp4
|
||||
vllm serve <your-model> --omni --quantization mxfp4_dualscale
|
||||
@@ -85,14 +92,19 @@ config = build_quant_config({
|
||||
})
|
||||
```
|
||||
|
||||
Component routing is an extension mechanism, not automatic support for every
|
||||
pipeline component. See [Quantization Scope](overview.md#quantization-scope)
|
||||
before applying it to an encoder, VAE, or decoder.
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Methods | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `method` | FP8, Int8, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` |
|
||||
| `ignored_layers` | FP8, Int8, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 |
|
||||
| `method` | FP8, Int8, BitsAndBytes, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"bitsandbytes"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` |
|
||||
| `ignored_layers` | FP8, Int8, BitsAndBytes, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 |
|
||||
| `activation_scheme` | FP8, Int8 | The runtime value `"dynamic"` selects online activation scaling |
|
||||
| `weight_block_size` | FP8 | Optional block-wise FP8 weight quantization size |
|
||||
| `quant_type` | BitsAndBytes | Weight format: `"nf4"` (default) or `"fp4"` |
|
||||
| `num_bf16_fallback_layers` | MXFP4 DualScale | Leading transformer blocks to keep in BF16 for online `mxfp4_dualscale`; defaults to `5` |
|
||||
|
||||
## Validation and Notes
|
||||
|
||||
@@ -16,72 +16,123 @@ For the internal architecture and backend extension points, see the
|
||||
| Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion Flash Attention tensors during inference. | FP8 FA |
|
||||
| Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
|
||||
|
||||
## Hardware Support
|
||||
Online quantization starts from a normal BF16/FP16 checkpoint and repeats the
|
||||
conversion on each fresh model load. It is the simplest path for experiments.
|
||||
Pre-quantized checkpoints store the packed weights and scales produced by a
|
||||
separate tool; they are easier to reuse across deployments and can carry
|
||||
calibrated or mixed-precision policies that load-time conversion cannot
|
||||
reproduce.
|
||||
|
||||
| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | ModelOpt | MXFP8 W8A8 | MXFP4 W4A4 | AutoRound | msModelSlim |
|
||||
|--------|----------|-----------|------------------|----------|------------|------------|-----------|-------------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
|
||||
| AMD ROCm | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ⭕ | ❌ |
|
||||
| Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
|
||||
| Ascend NPU | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
## Support Levels
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
|
||||
guide. FP8 on Ampere may use a weight-only path where available.
|
||||
Quantization support is specific to a model, component, checkpoint format, and
|
||||
hardware backend. The tables in this guide use the following levels:
|
||||
|
||||
## Model Type Support
|
||||
| Level | Meaning |
|
||||
|-------|---------|
|
||||
| **CI-backed** | A named model and quantized checkpoint are exercised by scheduled full-model hardware CI. |
|
||||
| **Validated** | A named model and checkpoint have end-to-end or quality-validation evidence, but the check is not necessarily scheduled in CI. |
|
||||
| **Integrated** | The config, loader, quantized layers, or model wiring exists, but this guide has no completed end-to-end validation for the named model and checkpoint. |
|
||||
| **Not validated** | There is no current support recommendation for this model/component combination. |
|
||||
| **Unsupported** | The combination is intentionally rejected or known not to work. |
|
||||
|
||||
### Diffusion Model (Qwen-Image, Wan2.2)
|
||||
A method being available on a device does not make every model on that device
|
||||
supported. Likewise, successfully detecting a checkpoint's
|
||||
`quantization_config` establishes loader compatibility, not output quality.
|
||||
|
||||
These models run a diffusion transformer as the primary inference module. The
|
||||
default quantization target is the transformer; tokenizer, scheduler, text
|
||||
encoder, and VAE stay on the base checkpoint unless a method guide says
|
||||
otherwise.
|
||||
## Hardware × Quantization Method
|
||||
|
||||
| Method | Guide | Mode | Example models | Status |
|
||||
|--------|-------|------|----------------|--------|
|
||||
| FP8 W8A8 | [FP8](fp8.md) | Online W8A8 or checkpoint FP8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
|
||||
| Int8 W8A8 | [Int8](int8.md) | Online or serialized W8A8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
|
||||
| BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Online W4 weight-only | Z-Image-Turbo; Qwen-Image/Wan2.2 not validated | Validated for Z-Image-Turbo |
|
||||
| ModelOpt | [ModelOpt](modelopt.md) | Pre-quantized FP8 checkpoints | Qwen-Image, Z-Image, FLUX.2, HunyuanImage-3.0 | Validated for ModelOpt FP8 diffusion checkpoints |
|
||||
| MXFP8 W8A8 | [MXFP8](mxfp8.md) | Online W8A8 or offline pre-quantized | Wan2.2-T2V-A14B, I2V-A14B, TI2V-5B | Ascend NPU only; validated for Wan2.2 |
|
||||
| MXFP4 W4A4 | [MXFP4](mxfp4.md) | `mxfp4`: online single-scale only; `mxfp4_dualscale`: online or offline dual-scale (offline recommended) | Wan2.2-T2V-A14B, I2V-A14B | Ascend NPU only; validated for Wan2.2 A14B cascade models; TI2V-5B not supported; offline `mxfp4_dualscale` uses calibrated `mul_scale` for best accuracy |
|
||||
| AutoRound | [AutoRound](autoround.md) | Pre-quantized W4A16 checkpoints | FLUX.1-dev; Qwen-Image/Wan2.2 not validated | Checkpoint-driven |
|
||||
| msModelSlim | [msModelSlim](msmodelslim.md) | Pre-quantized Ascend checkpoints | Wan2.2 recipe; HunyuanImage-3.0 inference target | Ascend/NPU path |
|
||||
This matrix uses concrete inference paths as columns. Producer toolkits are not
|
||||
separate methods: NVIDIA ModelOpt produces the ModelOpt FP8/NVFP4 formats,
|
||||
Intel AutoRound produces AutoRound checkpoints, and Ascend msModelSlim produces
|
||||
checkpoints consumed by the native MXFP or compatible Ascend paths.
|
||||
|
||||
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
||||
| Hardware | [Online FP8](fp8.md) | [Int8](int8.md) | [BitsAndBytes W4](bitsandbytes.md) | [ModelOpt FP8](modelopt.md) | [ModelOpt NVFP4](modelopt.md#supported-modelopt-checkpoint-formats) | [AutoRound W4A16](autoround.md) | [OCP MXFP8](mxfp8.md) | [OCP MXFP4](mxfp4.md) |
|
||||
|----------|-----------------------|------------------|------------------------------------------|--------------------------------|-------------------------------------------------------------------------|--------------------------------------|--------------------------|--------------------------|
|
||||
| NVIDIA Blackwell (SM 100+) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| NVIDIA Hopper (SM 90) | ✅ | ✅ | ✅ | ✅ | ✅ (Marlin fallback) | ✅ | ❌ | ❌ |
|
||||
| NVIDIA Ada (SM 89) | ✅ | ✅ | ✅ | ✅ | ⭕ | ✅ | ❌ | ❌ |
|
||||
| NVIDIA Ampere (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ | ❌ |
|
||||
| AMD ROCm (gfx950 for MXFP4) | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ❌ | ✅ (online only) |
|
||||
| Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ✅ | ❌ |
|
||||
| Ascend NPU | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
|
||||
These models combine an AR language model with audio, vision, talker, or TTS
|
||||
stages. Quantization is scoped to the AR language-model stage when the
|
||||
checkpoint contains a supported `quantization_config`; the non-AR stages stay
|
||||
in BF16 unless the model guide explicitly adds support.
|
||||
Legend: `✅` backend available, `❌` rejected or no implementation, `⭕` not
|
||||
verified in this guide. This matrix does not establish model-level support; use
|
||||
the model-and-stage tables below for that.
|
||||
|
||||
| Method | Guide | Scope | Example models | Status |
|
||||
|--------|-------|-------|----------------|--------|
|
||||
| ModelOpt | [ModelOpt](modelopt.md) | Thinker or language-model checkpoint config | Qwen3-Omni thinker | ModelOpt checkpoint path |
|
||||
| Int8 | [Int8](int8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
| MXFP8 | [MXFP8](mxfp8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
| MXFP4 | [MXFP4](mxfp4.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
| AutoRound | [AutoRound](autoround.md) | Thinker or language-model checkpoint config | Qwen2.5-Omni, Qwen3-Omni | Supported through AutoRound checkpoints |
|
||||
| msModelSlim | [msModelSlim](msmodelslim.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
|
||||
The Int8 column covers online and serialized Int8 paths. OCP MXFP8 is online on
|
||||
Intel XPU and online/offline on Ascend; Intel's separate AutoRound MXFP8
|
||||
checkpoint path is described in the [MXFP8 guide](mxfp8.md). OCP MXFP4 on AMD
|
||||
means online single-scale `mxfp4` on gfx950, while MXFP4 DualScale is
|
||||
Ascend-only. NVIDIA's NVFP4 and OCP MXFP4 have different checkpoint metadata,
|
||||
scale layouts, and kernel contracts. Mixed ModelOpt FP8/NVFP4 checkpoints also
|
||||
require a model-specific mixed-precision policy and validation. FP8 on Ampere
|
||||
may use a weight-only path where available.
|
||||
|
||||
### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
|
||||
## Current Model and Stage Support
|
||||
|
||||
These models split generation across multiple stages. Quantization must be
|
||||
attached to the intended stage rather than applied globally.
|
||||
The current offline checkpoint guidance is transformer-first. A listed
|
||||
checkpoint quantizes the stage named in the **Scope** column; it does not imply
|
||||
that the model's encoders, VAE, decoder, or other stages are also quantized.
|
||||
|
||||
| Method | Guide | Scope | Example models | Status |
|
||||
|--------|-------|-------|----------------|--------|
|
||||
| FP8 | [FP8](fp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
|
||||
| Int8 | [Int8](int8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
|
||||
| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Stage-specific transformer or DiT module | BAGEL, GLM-Image | Not validated |
|
||||
| ModelOpt | [ModelOpt](modelopt.md) | Checkpoint-defined diffusion stage | BAGEL, GLM-Image | Requires model-specific validation |
|
||||
| MXFP8 | [MXFP8](mxfp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
|
||||
| MXFP4 | [MXFP4](mxfp4.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
|
||||
| AutoRound | [AutoRound](autoround.md) | Checkpoint-defined stage | BAGEL, GLM-Image | No validated checkpoint listed |
|
||||
| msModelSlim | [msModelSlim](msmodelslim.md) | Ascend-generated stage weights | GLM-Image | Requires model-specific adaptation |
|
||||
### Vendor and Runtime Ecosystems
|
||||
|
||||
The producer and runtime backend are related, but they are not the same support
|
||||
claim. A producer can emit several checkpoint formats, and each format still
|
||||
needs a compatible vLLM-Omni loader, model-stage wiring, and validation. A
|
||||
runtime may also provide online quantization without an offline producer path.
|
||||
|
||||
| Ecosystem | Current paths | Current vLLM-Omni scope | Model evidence |
|
||||
|-----------|---------------|-------------------------|----------------|
|
||||
| NVIDIA | ModelOpt FP8, NVFP4, and mixed FP8/NVFP4 | Diffusion transformer or Qwen-Omni thinker | Qwen-Image and other named diffusion checkpoints are validated; Qwen3-Omni NVFP4 has H100 CI for the Marlin fallback |
|
||||
| Intel | AutoRound W4A16 and AutoRound MXFP8 | Diffusion/world-model transformer or Qwen-Omni thinker | Wan A14B and GLM-Image W4A16 are CI-backed; Qwen-Image, Qwen-Omni, and Cosmos3-Super have validation outside scheduled CI |
|
||||
| AMD ROCm | Online OCP MXFP4 through AITER on gfx950; no pre-quantized MXFP4 or AMD Quark workflow is integrated | Diffusion transformer linear layers | Integrated backend with dispatch and weight-allocation unit coverage; no named-model quality validation or scheduled gfx950 full-model CI |
|
||||
| Ascend | msModelSlim output through native MXFP8/MXFP4 merge workflows or a compatible generic Ascend format | Wan diffusion transformers | Native Wan MXFP8/MXFP4 paths are validated outside scheduled full-model NPU CI; other model families remain integrated or unvalidated |
|
||||
|
||||
### Pre-Quantized Model Matrix
|
||||
|
||||
| Model family | Method | Scope | Level | Notes |
|
||||
|--------------|--------|-------|-------|-------|
|
||||
| Qwen-Image | [ModelOpt](modelopt.md) FP8 or mixed FP8/NVFP4 | Diffusion transformer | Validated | Named Qwen-Image 2512 checkpoints and recipes are available. Text encoder and VAE stay BF16. |
|
||||
| Qwen-Image | [AutoRound](autoround.md) W4A16 | Diffusion transformer | Validated | Uses `INC4AI/Qwen-Image-AutoRound-W4A16`; text encoder and VAE stay BF16. |
|
||||
| Wan2.2 I2V/T2V A14B | [AutoRound](autoround.md) W4A16 | Both diffusion transformers | CI-backed | The high-noise and low-noise DiTs are quantized; text encoder and VAE stay BF16. |
|
||||
| Wan2.2 TI2V 5B | [AutoRound](autoround.md) W4A16 | Diffusion transformer | Validated | Named checkpoint available; not covered by the scheduled Wan AutoRound job. |
|
||||
| Wan2.2 | [MXFP8](mxfp8.md) | One or both diffusion transformers | Validated | Native Ascend and AutoRound checkpoint formats are integrated; no scheduled full-model offline NPU job. |
|
||||
| Wan2.2 I2V/T2V A14B | [MXFP4 DualScale](mxfp4.md) | Both diffusion transformers | Validated | Offline calibrated checkpoints are recommended; no scheduled full-model offline NPU job. |
|
||||
| Qwen2.5-Omni | [AutoRound](autoround.md) W4A16 | Thinker language model | Validated | Encoders and output/audio stages stay BF16. |
|
||||
| Qwen3-Omni | [ModelOpt](modelopt.md) FP8 | Thinker language model | Validated | Tested outside scheduled CI. |
|
||||
| Qwen3-Omni | [ModelOpt](modelopt.md) NVFP4 | Thinker language model | CI-backed | H100 CI exercises the Marlin fallback; the native Blackwell FP4 path has separate recipe validation. |
|
||||
| Qwen3-Omni | [AutoRound](autoround.md) W4A16 | Thinker language model | Validated | Audio encoder, vision encoder, talker, and Code2Wav stay BF16. |
|
||||
| GLM-Image | [AutoRound](autoround.md) W4A16 | Diffusion transformer stage | CI-backed | Text-to-image and image-to-image full-model smoke coverage. |
|
||||
| Cosmos3-Super | [AutoRound](autoround.md) W4A16 | World-model transformer | Validated | Manually validated serving path; VAE and other components stay BF16. |
|
||||
| Cosmos3-Nano | [AutoRound](autoround.md) W4A16 | World-model transformer | Integrated | Named checkpoint is listed, but no model-specific end-to-end test is maintained in-tree. |
|
||||
| MiniMax H3 | None | None | Not validated | No offline vendor checkpoint path is documented; online FP8 targets the DiT only. |
|
||||
|
||||
The method pages contain the complete checkpoint lists, including FLUX,
|
||||
Z-Image, and HunyuanImage-3.0 variants.
|
||||
|
||||
### Online Quantization
|
||||
|
||||
| Model family | Methods | Scope | Level |
|
||||
|--------------|---------|-------|-------|
|
||||
| Qwen-Image | FP8, Int8 | Diffusion transformer | Validated |
|
||||
| Z-Image | Int8, BitsAndBytes W4 | Diffusion transformer | Validated |
|
||||
| Wan2.2 | MXFP8 | One or both diffusion transformers | Validated on the documented XPU/NPU paths |
|
||||
| Wan2.2 A14B | MXFP4 | Both diffusion transformers | Validated on Ascend; the ROCm gfx950 online backend is integrated but not model-validated |
|
||||
| MiniMax H3 | FP8 | DiT and optional reference DiT | Validated |
|
||||
| Qwen3-Omni | Generic online methods, such as FP8 | Thinker language model only | Integrated; not model-validated as an online recommendation |
|
||||
|
||||
See [Online Quantization](online.md) for configuration and hardware details.
|
||||
|
||||
### Unlisted and Long-Tail Models
|
||||
|
||||
For models not listed above, generic quantized linear layers or compatible
|
||||
checkpoint metadata are only an integration starting point. The default support
|
||||
target is the model's primary transformer, DiT, or AR language-model stage.
|
||||
Text/audio/vision encoders, VAEs, vocoders, decoders, and output stages remain
|
||||
BF16 until a model-specific integration supplies quant-aware layers, checkpoint
|
||||
mapping, end-to-end output validation, and backend coverage.
|
||||
|
||||
!!! note
|
||||
"Online quantization" means vLLM-Omni computes the quantization data while
|
||||
@@ -92,8 +143,8 @@ attached to the intended stage rather than applied globally.
|
||||
|
||||
### Diffusion Model (Qwen-Image, Wan2.2)
|
||||
|
||||
The default target is the diffusion transformer. Component routing is available
|
||||
through `build_quant_config()`:
|
||||
The default target is the diffusion transformer. `build_quant_config()` can
|
||||
construct a component router:
|
||||
|
||||
```python
|
||||
from vllm_omni.quantization import build_quant_config
|
||||
@@ -104,11 +155,19 @@ config = build_quant_config({
|
||||
})
|
||||
```
|
||||
|
||||
!!! warning "Routing syntax is not a support claim"
|
||||
A component dictionary only selects a quantization config by module prefix.
|
||||
It does not make an encoder or VAE quantizable. The model must construct
|
||||
that component from quantization-aware layers, and an offline checkpoint
|
||||
must contain matching packed weights and scales. Current user guidance
|
||||
keeps diffusion text encoders and VAEs in BF16.
|
||||
|
||||
| Component | Default quantized? | Notes |
|
||||
|-----------|--------------------|-------|
|
||||
| Diffusion transformer | Yes | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim |
|
||||
| Diffusion transformer | Yes, for listed model/method pairs | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim |
|
||||
| Second diffusion transformer | Yes, for listed Wan2.2 A14B methods | `transformer` and `transformer_2` read their own offline checkpoint metadata; arbitrary mixed methods are not currently validated |
|
||||
| Text encoder | No | Keep BF16 unless a method-specific guide documents support |
|
||||
| VAE | No | Keep BF16; storage-only paths are method-specific |
|
||||
| VAE | No | Keep BF16 |
|
||||
| Scheduler/tokenizer | No | Loaded from the base model repository |
|
||||
|
||||
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
||||
@@ -121,6 +180,18 @@ config = build_quant_config({
|
||||
| Talker or TTS stage | No | BF16 unless model-specific support is documented |
|
||||
| Code2Wav | No | BF16 |
|
||||
|
||||
An explicit `ComponentQuantizationConfig` can be used by model developers to
|
||||
route different configs to `audio_tower`, `visual`, and `language_model`, but
|
||||
the supported Qwen-Omni checkpoints currently quantize only the thinker
|
||||
language model.
|
||||
|
||||
### World Model (Cosmos3)
|
||||
|
||||
| Component | Default quantized? | Notes |
|
||||
|-----------|--------------------|-------|
|
||||
| World-model transformer | AutoRound checkpoints only | Cosmos3-Super is validated; Cosmos3-Nano is integrated but not end-to-end tested in-tree |
|
||||
| VAE and guardrail components | No | BF16 |
|
||||
|
||||
### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
|
||||
|
||||
| Component | Default quantized? | Notes |
|
||||
@@ -141,6 +212,7 @@ build_quant_config("fp8")
|
||||
build_quant_config({"method": "fp8", "activation_scheme": "static"})
|
||||
build_quant_config("bitsandbytes")
|
||||
build_quant_config("auto-round", bits=4, group_size=128)
|
||||
# Component routing syntax; model-specific support is still required.
|
||||
build_quant_config({"transformer": {"method": "fp8"}, "vae": None})
|
||||
build_quant_config(None)
|
||||
```
|
||||
|
||||
Reference in new issue
Block a user