docs: clarify quantization support matrix
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
c98517ce7f
commit
00dab2ac95
12 files changed
+358
-160
No files matched your search
@@ -10,6 +10,10 @@ checkpoint's `config.json` and auto-detects
|
||||
AutoRound is static quantization: no `--quantization` flag is needed at
|
||||
inference time when the checkpoint already contains the quantization config.
|
||||
|
||||
The validation terms on this page follow the
|
||||
[support levels](overview.md#support-levels) defined in the overview. A listed
|
||||
checkpoint is not enough by itself to establish end-to-end model support.
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | Support |
|
||||
@@ -21,46 +25,57 @@ inference time when the checkpoint already contains the quantization config.
|
||||
| Intel XPU | ✅ |
|
||||
| Ascend NPU | ❌ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
|
||||
guide. AutoRound is Intel-supported.
|
||||
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
|
||||
guide. This table is method-level backend availability. AutoRound is an Intel
|
||||
project; vLLM selects a compatible CUDA or XPU compute backend at load time.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
### Diffusion Model (Qwen-Image, Wan2.2)
|
||||
|
||||
| Model | Checkpoint | Scope | Scheme | Backend |
|
||||
|-------|------------|-------|--------|---------|
|
||||
| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Qwen-Image | Not listed | Diffusion transformer | W4A16 | Not validated |
|
||||
| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Model | Checkpoint | Scope | Scheme | Validation |
|
||||
|-------|------------|-------|--------|------------|
|
||||
| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | Validated |
|
||||
| Qwen-Image | `INC4AI/Qwen-Image-AutoRound-W4A16` | Diffusion transformer | W4A16 | Validated with end-to-end generation and quality comparison |
|
||||
| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
|
||||
| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
|
||||
| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | Validated; not in the scheduled Wan AutoRound job |
|
||||
|
||||
CUDA execution uses GPTQ-Marlin or another compatible vLLM AutoRound backend;
|
||||
Intel XPU uses the Intel-supported backend. The text encoder and VAE stay in
|
||||
BF16 for all entries above.
|
||||
|
||||
### World Model (Cosmos3)
|
||||
|
||||
| Model | Checkpoint | Scope | Scheme | Backend |
|
||||
|-------|------------|-------|--------|---------|
|
||||
| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
|
||||
| Model | Checkpoint | Scope | Scheme | Validation |
|
||||
|-------|------------|-------|--------|------------|
|
||||
| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | Integrated; no model-specific end-to-end test is maintained in-tree |
|
||||
| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | Validated with a manual serving run |
|
||||
|
||||
The Cosmos VAE and guardrail components stay in BF16.
|
||||
|
||||
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
||||
|
||||
| Model | Checkpoint | Scope | Scheme | Backend |
|
||||
|-------|------------|-------|--------|---------|
|
||||
| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Language-model stage | W4A16 | AutoRound |
|
||||
| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | AutoRound |
|
||||
| Model | Checkpoint | Scope | Scheme | Validation |
|
||||
|-------|------------|-------|--------|------------|
|
||||
| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
|
||||
| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
|
||||
| Qwen3-TTS | Not listed | TTS language-model stage | W4A16 | Not validated |
|
||||
|
||||
AutoRound support is checkpoint-driven. A model is supported when its
|
||||
checkpoint uses a compatible INC/AutoRound config and the target stage maps to
|
||||
vLLM-Omni's runtime module names.
|
||||
|
||||
For Qwen-Omni, only the thinker language model uses the AutoRound checkpoint.
|
||||
Audio and vision encoders, the talker, and waveform-decoder stages stay in
|
||||
BF16.
|
||||
|
||||
### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
|
||||
|
||||
| Model | Scope | Status | Notes |
|
||||
|-------|-------|--------|-------|
|
||||
| GLM-Image | Diffusion transformer | ✅ | `Intel/GLM-Image-int4-AutoRound` |
|
||||
| BAGEL | Checkpoint-defined diffusion or transformer stage | Not validated | Requires a compatible AutoRound checkpoint |
|
||||
| Model | Checkpoint | Scope | Validation |
|
||||
|-------|------------|-------|------------|
|
||||
| GLM-Image | `Intel/GLM-Image-int4-AutoRound` | Diffusion transformer | CI-backed |
|
||||
| BAGEL | Not listed | Checkpoint-defined diffusion or transformer stage | Not validated |
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -119,6 +134,14 @@ The checkpoint should contain a config like:
|
||||
At load time, vLLM-Omni builds an `OmniINCConfig`, remaps checkpoint block names
|
||||
to runtime module names, and selects the matching vLLM compute backend.
|
||||
|
||||
Checkpoint auto-detection and successful layer construction establish the
|
||||
**Integrated** level. Promote a model to **Validated** only after the named
|
||||
checkpoint completes an end-to-end generation or multimodal request and its
|
||||
output passes a model-appropriate sanity or quality check. Compare against the
|
||||
BF16 baseline for quality-sensitive image and video paths. Promote it to
|
||||
**CI-backed** only when that named checkpoint is selected by scheduled hardware
|
||||
CI.
|
||||
|
||||
Example checkpoint creation:
|
||||
|
||||
```bash
|
||||
|
||||
Reference in new issue
Block a user