vllm-omni/docs/user_guide/quantization/autoround.md

AutoRound Quantization

Overview

AutoRound produces pre-quantized checkpoints for LLMs, VLMs, diffusion models, and world models. vLLM-Omni reads the checkpoint's config.json and auto-detects quantization_config.quant_method = "auto-round".

AutoRound is static quantization: no --quantization flag is needed at inference time when the checkpoint already contains the quantization config.

The validation terms on this page follow the support levels defined in the overview. A listed checkpoint is not enough by itself to establish end-to-end model support.

Hardware Support

Device	Support
NVIDIA Blackwell GPU (SM 100+)	✅
NVIDIA Ada/Hopper GPU (SM 89+)	✅
NVIDIA Ampere GPU (SM 80+)	✅
AMD ROCm	⭕
Intel XPU	✅
Ascend NPU	❌

Legend: ✅ backend available, ❌ unsupported, ⭕ not verified in this guide. This table is method-level backend availability. AutoRound is an Intel project; vLLM selects a compatible CUDA or XPU compute backend at load time.

Model Type Support

Diffusion Model (Qwen-Image, Wan2.2)

Model	Checkpoint	Scope	Scheme	Validation
FLUX.1-dev	vllm-project-org/FLUX.1-dev-AutoRound-w4a16	Diffusion transformer	W4A16	Validated
Qwen-Image	INC4AI/Qwen-Image-AutoRound-W4A16	Diffusion transformer	W4A16	Validated with end-to-end generation and quality comparison
Wan2.2-I2V	Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound	Both diffusion transformers	W4A16	CI-backed
Wan2.2-T2V	Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound	Both diffusion transformers	W4A16	CI-backed
Wan2.2-TI2V	Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound	Diffusion transformer	W4A16	Validated; not in the scheduled Wan AutoRound job

CUDA execution uses GPTQ-Marlin or another compatible vLLM AutoRound backend; Intel XPU uses the Intel-supported backend. The text encoder and VAE stay in BF16 for all entries above.

World Model (Cosmos3)

Model	Checkpoint	Scope	Scheme	Validation
Cosmos3-Nano	Intel/Cosmos3-Nano-int4-AutoRound	World-model transformer	W4A16	Integrated; no model-specific end-to-end test is maintained in-tree
Cosmos3-Super	Intel/Cosmos3-Super-int4-AutoRound	World-model transformer	W4A16	Validated with a manual serving run

The Cosmos VAE and guardrail components stay in BF16.

Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)

Model	Checkpoint	Scope	Scheme	Validation
Qwen2.5-Omni-7B	Intel/Qwen2.5-Omni-7B-int4-AutoRound	Thinker language-model stage	W4A16	Validated outside scheduled CI
Qwen3-Omni-30B-A3B-Instruct	Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound	Thinker language-model stage	W4A16	Validated outside scheduled CI
Qwen3-TTS	Not listed	TTS language-model stage	W4A16	Not validated

AutoRound support is checkpoint-driven. A model is supported when its checkpoint uses a compatible INC/AutoRound config and the target stage maps to vLLM-Omni's runtime module names.

For Qwen-Omni, only the thinker language model uses the AutoRound checkpoint. Audio and vision encoders, the talker, and waveform-decoder stages stay in BF16.

Multi-Stage Diffusion Model (BAGEL, GLM-Image)

Model	Checkpoint	Scope	Validation
GLM-Image	Intel/GLM-Image-int4-AutoRound	Diffusion transformer	CI-backed
BAGEL	Not listed	Checkpoint-defined diffusion or transformer stage	Not validated

Configuration

Python API:

from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams

omni = Omni(model="vllm-project-org/FLUX.1-dev-AutoRound-w4a16")

outputs = omni.generate(
    "A cat sitting on a windowsill",
    OmniDiffusionSamplingParams(num_inference_steps=28),
)
outputs[0].save_images("output.png")

CLI:

python examples/offline_inference/text_to_image/text_to_image.py \
  --model vllm-project-org/FLUX.1-dev-AutoRound-w4a16 \
  --prompt "A cat sitting on a windowsill" \
  --num-inference-steps 28 \
  --output outputs/flux_w4a16.png

Parameters

Field	Type	Description
quant_method	str	Must be "auto-round"
bits	int	Quantized weight bit width, usually 4
group_size	int	Quantization group size
packing_format	str	AutoRound packing format, for example auto_round:auto_gptq
block_name_to_quantize	str	Checkpoint block names that should map to runtime module names

The checkpoint should contain a config like:

{
  "quantization_config": {
    "quant_method": "auto-round",
    "bits": 4,
    "group_size": 128,
    "sym": true,
    "packing_format": "auto_round:auto_gptq",
    "block_name_to_quantize": "transformer_blocks,single_transformer_blocks"
  }
}

Validation and Notes

At load time, vLLM-Omni builds an OmniINCConfig, remaps checkpoint block names to runtime module names, and selects the matching vLLM compute backend.

Checkpoint auto-detection and successful layer construction establish the Integrated level. Promote a model to Validated only after the named checkpoint completes an end-to-end generation or multimodal request and its output passes a model-appropriate sanity or quality check. Compare against the BF16 baseline for quality-sensitive image and video paths. Promote it to CI-backed only when that named checkpoint is selected by scheduled hardware CI.

Example checkpoint creation:

auto-round \
  --model black-forest-labs/FLUX.1-dev \
  --scheme W4A16 \
  --batch_size 1 \
  --disable_opt_rtn \
  --dataset coco2014 \
  --iters 0

Use the generated output directory directly as the model argument. See the AutoRound documentation for all available schemes and options.