vllm-omni/recipes/Qwen/Qwen-Image-2.1.md

Qwen-Image-2.1

Text-to-image and image-conditioned generation with Qwen-Image 2.1

Summary

• Vendor: Qwen
• Model: Qwen/Qwen-Image-2.1
• Task: Text-to-image generation and image-conditioned generation (editing)
• Mode: Offline inference and online serving with optional step-wise execution
• Maintainer: Community

Model id: Qwen/Qwen-Image-2.1 follows the reference diffusers documentation and may change before the official release — the final checkpoint name is subject to the official release.

When to use this recipe

Use this recipe as a starting point for running Qwen/Qwen-Image-2.1. A single pipeline class, QwenImage21Pipeline, handles both pure text-to-image requests (no condition image) and image-conditioned requests with up to 4 condition images: prompt and condition images are encoded together by a Qwen3-VL text encoder, so editing-style requests go through the same serving path as text-to-image.

References

• Related offline examples: examples/offline_inference/text_to_image/text_to_image.py, examples/offline_inference/image_to_image/image_edit.py
• Supported models table: docs/models/supported_models.md
• Feature compatibility matrix: docs/user_guide/diffusion_features.md
• Sibling recipes: Qwen-Image.md, Qwen-Image-Edit.md

Offline Inference

Text-to-image

python examples/offline_inference/text_to_image/text_to_image.py \
  --model Qwen/Qwen-Image-2.1 \
  --prompt "A ceramic teapot on a wooden table" \
  --output qwen_image_21_t2i.png \
  --num-inference-steps 50 \
  --cfg-scale 1.0

Image-conditioned generation (editing)

python examples/offline_inference/image_to_image/image_edit.py \
  --model Qwen/Qwen-Image-2.1 \
  --color-format RGBA \
  --seed 42 \
  --image qwen_bear.png \
  --prompt "Let this mascot dance under the moon, surrounded by floating stars" \
  --output qwen_image_21_edit.png \
  --num-inference-steps 50 \
  --cfg-scale 1.0

or via the bundled launcher:

bash examples/offline_inference/image_to_image/run_qwen_image_21.sh

Up to 4 condition images can be passed via --image:

python examples/offline_inference/image_to_image/image_edit.py \
  --model Qwen/Qwen-Image-2.1 \
  --color-format RGBA \
  --seed 42 \
  --image input1.png input2.png \
  --prompt "Combine these images into a single scene" \
  --output qwen_image_21_multi.png \
  --num-inference-steps 50 \
  --cfg-scale 1.0

Use --color-format RGBA to preserve transparency in condition images. The shared example otherwise loads inputs as RGB. Add --width and --height (multiples of 32) to set the output size; when omitted, Qwen-Image 2.1 derives it from the last condition image's aspect ratio at approximately 1024×1024. Use --num-outputs-per-prompt to generate multiple images and --vae-use-tiling to reduce VAE memory usage. The launcher accepts these options as well.

Online Serving

Start the server:

vllm serve Qwen/Qwen-Image-2.1 --omni --port 8091

To enable the step-wise runtime:

vllm serve Qwen/Qwen-Image-2.1 --omni \
  --port 8091 \
  --step-execution \
  --max-num-seqs 8

Step execution batches compatible requests at the same KV-cache phase. The pipeline tags every request with a batch_compatibility_key (conditioning mode plus condition-image layout), so a newly arriving request that would otherwise join a batch already past its first denoising step is isolated by the scheduler and deferred until the in-flight batch drains — it no longer fails the whole batch. --max-num-seqs controls batch capacity.

CUDA Graph decode

Qwen-Image-2.1 automatically uses CUDA Graph for supported fixed-shape denoising decode steps when enable_cuda_graph_decode=True (the default) and enforce_eager=False. Prefill remains eager. This works with both request execution and --step-execution.

To disable graph capture while keeping the rest of the compilation setup:

vllm serve Qwen/Qwen-Image-2.1 --omni --no-enable-cuda-graph-decode

The same control is available as Omni(..., enable_cuda_graph_decode=False) and as enable_cuda_graph_decode: false on a stage in the deployment YAML. --enforce-eager / enforce_eager: true remains the broader switch: it disables both graph capture and automatic torch.compile. The two optimizations stack: the DiT blocks are regionally torch.compile'd and graph capture records the compiled (fused) kernels, while inductor's own cudagraphs stay disabled so the two graph layers never nest. Measured on a single H200 (1024×1024, 50 steps, seed 42, BF16, true_cfg_scale=1.0, warmup 1 + median of 3): compile+graph 5.93 s, compile-only 6.88 s, graph-only 9.52 s, eager 8.28 s end-to-end per image; combo and compile-only vs eager are 38.10 dB PSNR (compile fusion drift), and graph-only vs eager is bit-identical.

The autoregressive engine's compilation_config.cudagraph_mode does not control this diffusion path; use enable_cuda_graph_decode (or enforce_eager for everything) to force eager decode execution.

TP/SP/ring parallelism, HSDP, cache hooks, quantized KV caches, dynamic LoRA, padded text masks, and a second in-flight request whose cache aliases an already-owned graph key fall back to eager decode. Model-level CPU offload does not disable decode graphs while enable_cuda_graph_decode stays on (the default). DiT weights stay on fixed device storage so captured pointers remain valid, and that storage is not released while the text encoder runs. Pass --no-enable-cuda-graph-decode (or enable_cuda_graph_decode: false) for the eager offload path, which moves DiT weights off the device and frees that VRAM. Peak memory for the default graph-plus-offload combination was not re-measured in this PR. Graphs keep separate entries for different image layouts and copy request-owned prefix K/V into static buffers before replay; each graph key has a single live owner at a time, so concurrent same-shape requests never overwrite each other's prefix. Those buffers require additional memory; graph entries are bounded by the transformer's cache limit.

Verification

For a direct API smoke test:

curl http://localhost:8091/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen-Image-2.1",
    "prompt": "A ceramic teapot on a wooden table",
    "size": "1024x1024",
    "num_inference_steps": 50,
    "true_cfg_scale": 1.0,
    "seed": 42
  }'

For image-conditioned generation, pass condition images through the OpenAI-compatible multimodal endpoint (/v1/chat/completions with image_url content parts, or /v1/images/edits), same as Qwen-Image-Edit.md.

Parameters

Parameter	Default	Notes
true_cfg_scale	1.0	Disables the negative-prompt branch by default. Set above 1 with a negative_prompt to enable true CFG. Qwen-Image 2.1 supports true CFG only — there is no guidance_scale parameter.
num_inference_steps	50	Number of denoising steps.
height / width	1024×1024	Must be multiples of 32. For image-conditioned requests, leaving them unset derives the output size from the last condition image's aspect ratio at ~1024×1024.
negative_prompt	None	Ignored at the default CFG scale of 1.0; used only when true_cfg_scale > 1.
condition images	—	Up to 4 input images per request; more than 4 raises an error.
seed	—	Fix for reproducible outputs.

The default CFG scale of 1.0 runs only the positive-prompt branch at each denoising step. To enable negative-prompt guidance, explicitly set --cfg-scale (offline) or true_cfg_scale (API) above 1 and provide a negative prompt. This changes the generated output as well as compute cost.

Prefix KV cache

The transformer caches the text/condition-image prefix across denoising steps (a prefill/decode split keyed by CFG branch). The cache is enabled automatically when the checkpoint declares causal_condition: true and is re-created per generation, so no state leaks between requests. It changes the compute path — cached prefix vs. full recompute at every step — so exact numerical reproduction against a no-cache reference (e.g. the diffusers pipeline) requires matching the cache setting on both sides. The prefix KV cache is orthogonal to weight quantization and works unchanged under FP8.

FP8 quantization

Online FP8 quantization of the transformer is supported; the fused to_qkv projections load correctly, and the prefix KV cache path is unaffected. Only block-internal linears are eligible — the boundary projections (img_in/txt_in/modulation/proj_out/time_text_embed) stay in BF16 by construction:

python examples/offline_inference/text_to_image/text_to_image.py \
  --model Qwen/Qwen-Image-2.1 \
  --prompt "A ceramic teapot on a wooden table" \
  --output qwen_image_21_fp8.png \
  --num-inference-steps 50 --cfg-scale 1.0 \
  --quantization fp8 --ignored-layers "img_mlp"

ignored_layers entries are matched as name patterns for this model (e.g. img_mlp matches every block's img_mlp.proj/img_mlp.gate_layer/ img_mlp.out); exact full prefixes like transformer.transformer_blocks.3.attn.to_qkv also work.

A per-module sensitivity sweep (fixed prompt embeds / latent / timestep, 5 timesteps across the trajectory, relative L2 of the noise prediction vs BF16) ranks the groups: img_mlp is the most sensitive (3.8% max error on its own, driven by img_mlp.proj 2.8% and img_mlp.out 2.5%), then attn.to_out (2.7%), then attn.to_qkv (1.6%); errors are largest at the late (low-sigma) steps. End-to-end on H200, single GPU, same prompt as Quality Comparison ("A ceramic teapot on a wooden table", 1024×1024, 50 steps, seed 42, eager):

Config	Quantized block linears	PSNR vs BF16	Steady-state time/image	Peak memory
BF16	0 / 160	—	7.34 s	36.9 GB
FP8 all layers	160 / 160	33.0 dB	6.06 s	30.4 GB
FP8, ignored_layers=["img_mlp"]	64 / 160	34.3 dB	6.98 s	34.9 GB

Keeping img_mlp in BF16 is still the higher-fidelity DiT setting vs all-layer FP8. On H200, all-layer FP8 is both leaner and faster than BF16 at this shape.

Quantization

Online FP8 is supported per component via quantization_config:

from vllm_omni import Omni

omni = Omni(
    model="Qwen/Qwen-Image-2.1",
    quantization_config={
        # DiT: skip the sensitive image-stream MLPs if quality regresses.
        "transformer": {"method": "fp8", "ignored_layers": ["img_mlp"]},
        # Qwen3-VL text encoder: only the language-model linear layers are
        # quantized; the vision tower and the (unused) LM head stay BF16.
        "text_encoder": {"method": "fp8"},
    },
)

Either component can be quantized on its own. Measured on H200 at 1024×1024 (seed 42, 50 steps, same teapot prompt, eager): text-encoder FP8 lowers peak GPU memory from 36.9 GB (BF16) to 30.4 GB (−6.5 GB); T2I PSNR vs BF16 32.1 dB. DiT FP8-all is 30.4 GB / 6.06 s. Combined DiT+text-encoder FP8 was not re-measured on H200. The edit path routes condition images through the BF16 vision tower, so it is unaffected by text-encoder FP8. See docs/user_guide/quantization/fp8.md for the scope rules.

FP8 prefix KV storage

The cached prefix K/V can be stored in FP8 E4M3 (with per-token-per-head fp32 scales), halving the prefix-cache memory — relevant for long prompts (up to 8192 text tokens) and up to 4 condition images (~4096 latent tokens each). Prefill quantizes once; decode dequantizes per step, so attention compute is unchanged. Enable it through the model-specific extras config (e.g. in the deploy YAML's stage extras or as an Omni(...) engine kwarg):

extras:
  prefix_kv_cache_dtype: "fp8"

Accepted values: "fp8" / "fp8_e4m3" (K and V quantized), "fp8_v" (V only — K stays in the native dtype), and None / "auto" (default, native dtype — behavior unchanged). This is independent of diffusion_kv_cache_dtype, which quantizes attention Q/K/V compute per forward pass on supported backends.

Measured on one NVIDIA H200, T2I 1024x1024 (seed 42, 50 steps, true CFG 4.0, eager, PSNR vs. the bf16 baseline on the same GPU): "fp8" 34.2 dB, "fp8_v" 44.0 dB. The error is dominated by K quantization: post-RoPE keys are the precision-sensitive half of the cache, so "fp8_v" buys back about 9.8 dB at 75% (instead of 50%) of the original cache size. The residual error is inherent e4m3 precision accumulated coherently over the denoising trajectory; finer scale granularity (per-tensor/per-head/per-token were compared) or Hadamard-rotated V did not improve it. The saving scales with prefix length. These sizes are calculated at the limit (8192 text tokens + 4 condition images, ~24.6k prefix tokens), not measured on the 1024 T2I run above: ~12.9 GB per CFG branch in bf16, ~6.6 GB in "fp8" and ~9.7 GB in "fp8_v". That is where this option matters. Treat it as an opt-in for memory-bound long-prompt / multi-image workloads, not a free lunch; "fp8_v" is the better default trade-off when quality matters.

Note: a quantized prefix cache is not CUDA-graph capturable — requests using this option stay on the eager decode path (a warning is logged once).

Why not the scheduler-managed paged KV (DiffusionKVCacheMode.PAGED_SCHEDULER)?

Evaluated and deliberately not adopted (see the qwen21-p0-kv commit message for the full analysis): the paged scheduler targets cross-request KV management — a statically sized page pool, block sharing/dedup, and eviction — whereas this prefix cache lives exactly as long as one generation and is sized once per request. Migrating would require routing decode attention through the worker paged adapter (writing fresh target K/V into pages every step), per-request page tables, and re-validating the block-causal piecewise-span masking and Ulysses/CFG constraints, for no memory benefit over the per-request dense tensors. The FP8 storage above captures the actual memory win at ~1% of the integration cost.

Quality Comparison (head vs diffusers)

The reference is the diffusers Qwen-Image 2.1 pipeline (huggingface/diffusers#14804). Both sides must pin the same checkpoint, inputs, seed, precision (BF16), and hardware.

vLLM-Omni (this recipe):

python examples/offline_inference/text_to_image/text_to_image.py \
  --model Qwen/Qwen-Image-2.1 \
  --prompt "A ceramic teapot on a wooden table" \
  --seed 42 \
  --output qwen_image_21_vllm_omni.png \
  --num-inference-steps 50 \
  --cfg-scale 1.0

diffusers reference:

import torch
from diffusers import QwenImage21Pipeline  # diffusers PR #14804

pipe = QwenImage21Pipeline.from_pretrained(
    "Qwen/Qwen-Image-2.1", torch_dtype=torch.bfloat16
).to("cuda")
image = pipe(
    prompt="A ceramic teapot on a wooden table",
    num_inference_steps=50,
    true_cfg_scale=1.0,
    generator=torch.Generator(device="cuda").manual_seed(42),
    height=1024,
    width=1024,
).images[0]
image.save("qwen_image_21_diffusers.png")

Metric	vLLM-Omni	diffusers (#14804)	Status
Output image (1024×1024, 50 steps, seed 42, BF16, H200)	generated (eager + default CUDA-graph)	generated	measured
PSNR / LPIPS vs reference	eager vs Diffusers 47.17 dB; default compile+graph vs Diffusers 38.02 dB; default compile+graph vs eager 38.10 dB	—	measured (H200)
Latency (time/image, same GPU, H200)	default compile+graph 6.04 s (120.8 ms/step); eager 7.31 s (146.2 ms/step)	7.74 s (154.9 ms/step)	measured

The prefix KV cache changes the compute path (cached prefix vs. full recompute at every step), so exact numerical parity is not expected; match the cache setting on both sides or compare with PSNR/LPIPS rather than exact pixels. Eager stays numerically close to Diffusers (~47 dB). The default serving path stacks regional torch.compile with CUDA-graph decode; on H200 that combo lands at ~38.1 dB vs eager (~1.21× on the Quality Comparison timed run). Graph-only vs eager is bit-identical; the ~38 dB drift is the compile fusion, not CUDA-graph replay. See "CUDA Graph decode" for the four-mode H200 breakdown (warmup 1 + median of 3).

Parity / feature measurements (same prompt/seed/steps/CFG, BF16, eager decode unless noted; 4× NVIDIA H200):

• TP=2/4 output parity vs single-card — measured (1024×1024): TP1 vs TP2 39.32 dB, TP1 vs TP4 38.37 dB, TP2 vs TP4 45.77 dB. Latency: TP1 7.30 s, TP2 4.89 s, TP4 5.12 s per image.
• VAE tiling vs non-tiled decode — measured: 1024×1024 nontile vs --vae-use-tiling 43.55 dB (peak mem ~37.7 GB → ~32.7 GB); 2048×2048 nontile vs tiled 44.14 dB (peak mem ~57.9 GB → ~33.3 GB; gen ~36.8 s vs ~37.1 s).
• CUDA-graph four-mode vs eager — measured on H200 (1024×1024, 50 steps, seed 42, BF16, warmup 1 + median of 3): compile+graph 5.93 s, compile-only 6.88 s, graph-only 9.52 s, eager 8.28 s. Combo / compile-only vs eager 38.10 dB; graph-only vs eager bit-identical. Quality Comparison table uses a separate single-run A/B vs Diffusers (compile+graph 6.04 s, eager 7.31 s).

Known Limitations

• Cache acceleration backends (cache_dit, tea_cache) are not supported; QwenImage21Pipeline is registered in _NO_CACHE_ACCELERATION.
• Ulysses sequence parallelism is wired; ring attention is not a supported path. A short H200 smoke (1024×1024, 8 steps, seed 42, true CFG 1.0, eager) finished: Ulysses SP=2 vs SP=1 is 46.5 dB. That size's target length is divisible by 4, so SP padding was not exercised, and this is not the 50-step parity next to the TP tables. The features matrix therefore stays ❓. Ring SP=2 also ran, at 29.9 dB vs SP=1, which is why it stays unsupported.
• Only true CFG is exposed; there is no guidance_scale knob.
• VAE tiling (--vae-use-tiling, also implied by --vae-patch-parallel-size > 1) decodes in 512px tiles with 384px stride (32 latent pixels per tile at 16x spatial compression). On CUDA OOM the tile size is automatically halved (keeping 25% overlap) and decoding retried, down to 128px tiles. Tile sizes are not yet exposed as engine arguments.