160 lines
6.0 KiB
Markdown
160 lines
6.0 KiB
Markdown
# AutoRound Quantization
|
|
|
|
## Overview
|
|
|
|
[AutoRound](https://github.com/intel/auto-round) produces pre-quantized
|
|
checkpoints for LLMs, VLMs, diffusion models, and world models. vLLM-Omni reads the
|
|
checkpoint's `config.json` and auto-detects
|
|
`quantization_config.quant_method = "auto-round"`.
|
|
|
|
AutoRound is static quantization: no `--quantization` flag is needed at
|
|
inference time when the checkpoint already contains the quantization config.
|
|
|
|
The validation terms on this page follow the
|
|
[support levels](overview.md#support-levels) defined in the overview. A listed
|
|
checkpoint is not enough by itself to establish end-to-end model support.
|
|
|
|
## Hardware Support
|
|
|
|
| Device | Support |
|
|
|--------|---------|
|
|
| NVIDIA Blackwell GPU (SM 100+) | ✅ |
|
|
| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ |
|
|
| NVIDIA Ampere GPU (SM 80+) | ✅ |
|
|
| AMD ROCm | ⭕ |
|
|
| Intel XPU | ✅ |
|
|
| Ascend NPU | ❌ |
|
|
|
|
Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
|
|
guide. This table is method-level backend availability. AutoRound is an Intel
|
|
project; vLLM selects a compatible CUDA or XPU compute backend at load time.
|
|
|
|
## Model Type Support
|
|
|
|
### Diffusion Model (Qwen-Image, Wan2.2)
|
|
|
|
| Model | Checkpoint | Scope | Scheme | Validation |
|
|
|-------|------------|-------|--------|------------|
|
|
| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | Validated |
|
|
| Qwen-Image | `INC4AI/Qwen-Image-AutoRound-W4A16` | Diffusion transformer | W4A16 | Validated with end-to-end generation and quality comparison |
|
|
| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
|
|
| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
|
|
| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | Validated; not in the scheduled Wan AutoRound job |
|
|
|
|
CUDA execution uses GPTQ-Marlin or another compatible vLLM AutoRound backend;
|
|
Intel XPU uses the Intel-supported backend. The text encoder and VAE stay in
|
|
BF16 for all entries above.
|
|
|
|
### World Model (Cosmos3)
|
|
|
|
| Model | Checkpoint | Scope | Scheme | Validation |
|
|
|-------|------------|-------|--------|------------|
|
|
| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | Integrated; no model-specific end-to-end test is maintained in-tree |
|
|
| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | Validated with a manual serving run |
|
|
|
|
The Cosmos VAE and guardrail components stay in BF16.
|
|
|
|
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
|
|
|
| Model | Checkpoint | Scope | Scheme | Validation |
|
|
|-------|------------|-------|--------|------------|
|
|
| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
|
|
| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
|
|
| Qwen3-TTS | Not listed | TTS language-model stage | W4A16 | Not validated |
|
|
|
|
AutoRound support is checkpoint-driven. A model is supported when its
|
|
checkpoint uses a compatible INC/AutoRound config and the target stage maps to
|
|
vLLM-Omni's runtime module names.
|
|
|
|
For Qwen-Omni, only the thinker language model uses the AutoRound checkpoint.
|
|
Audio and vision encoders, the talker, and waveform-decoder stages stay in
|
|
BF16.
|
|
|
|
### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
|
|
|
|
| Model | Checkpoint | Scope | Validation |
|
|
|-------|------------|-------|------------|
|
|
| GLM-Image | `Intel/GLM-Image-int4-AutoRound` | Diffusion transformer | CI-backed |
|
|
| BAGEL | Not listed | Checkpoint-defined diffusion or transformer stage | Not validated |
|
|
|
|
## Configuration
|
|
|
|
Python API:
|
|
|
|
```python
|
|
from vllm_omni import Omni
|
|
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
|
|
|
omni = Omni(model="vllm-project-org/FLUX.1-dev-AutoRound-w4a16")
|
|
|
|
outputs = omni.generate(
|
|
"A cat sitting on a windowsill",
|
|
OmniDiffusionSamplingParams(num_inference_steps=28),
|
|
)
|
|
outputs[0].save_images("output.png")
|
|
```
|
|
|
|
CLI:
|
|
|
|
```bash
|
|
python examples/offline_inference/text_to_image/text_to_image.py \
|
|
--model vllm-project-org/FLUX.1-dev-AutoRound-w4a16 \
|
|
--prompt "A cat sitting on a windowsill" \
|
|
--num-inference-steps 28 \
|
|
--output outputs/flux_w4a16.png
|
|
```
|
|
|
|
## Parameters
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `quant_method` | str | Must be `"auto-round"` |
|
|
| `bits` | int | Quantized weight bit width, usually `4` |
|
|
| `group_size` | int | Quantization group size |
|
|
| `packing_format` | str | AutoRound packing format, for example `auto_round:auto_gptq` |
|
|
| `block_name_to_quantize` | str | Checkpoint block names that should map to runtime module names |
|
|
|
|
The checkpoint should contain a config like:
|
|
|
|
```json
|
|
{
|
|
"quantization_config": {
|
|
"quant_method": "auto-round",
|
|
"bits": 4,
|
|
"group_size": 128,
|
|
"sym": true,
|
|
"packing_format": "auto_round:auto_gptq",
|
|
"block_name_to_quantize": "transformer_blocks,single_transformer_blocks"
|
|
}
|
|
}
|
|
```
|
|
|
|
## Validation and Notes
|
|
|
|
At load time, vLLM-Omni builds an `OmniINCConfig`, remaps checkpoint block names
|
|
to runtime module names, and selects the matching vLLM compute backend.
|
|
|
|
Checkpoint auto-detection and successful layer construction establish the
|
|
**Integrated** level. Promote a model to **Validated** only after the named
|
|
checkpoint completes an end-to-end generation or multimodal request and its
|
|
output passes a model-appropriate sanity or quality check. Compare against the
|
|
BF16 baseline for quality-sensitive image and video paths. Promote it to
|
|
**CI-backed** only when that named checkpoint is selected by scheduled hardware
|
|
CI.
|
|
|
|
Example checkpoint creation:
|
|
|
|
```bash
|
|
auto-round \
|
|
--model black-forest-labs/FLUX.1-dev \
|
|
--scheme W4A16 \
|
|
--batch_size 1 \
|
|
--disable_opt_rtn \
|
|
--dataset coco2014 \
|
|
--iters 0
|
|
```
|
|
|
|
Use the generated output directory directly as the `model` argument. See the
|
|
[AutoRound documentation](https://github.com/intel/auto-round) for all
|
|
available schemes and options.
|