vllm-omni/docs/user_guide/diffusion/multi_replica_serving.md

Diffusion Multi-Replica Serving

Diffusion stages can be replicated in online serving by setting runtime.num_replicas on a stage_type: diffusion stage. Each replica owns an independent diffusion engine process and receives requests through round-robin routing with request affinity.

When to Use

Use multi-replica serving when one diffusion process does not saturate the available GPUs, or when you want higher request throughput for many independent image/video generation requests. It improves concurrency throughput; it does not make a single request faster.

Configuration

A replica consumes the devices required by that diffusion stage. For single-device diffusion pipelines, set runtime.devices to one device per replica:

stages:
  - stage_id: 0
    stage_type: diffusion
    runtime:
      devices: "0,1,2,3"
      num_replicas: 4
    engine_args:
      model: Qwen/Qwen-Image
      parallel_config:
        world_size: 1

For multi-device diffusion pipelines, parallel_config.world_size is treated as the number of devices per replica. For example, four 2-GPU replicas need eight devices:

stages:
  - stage_id: 0
    stage_type: diffusion
    runtime:
      devices: "0,1,2,3,4,5,6,7"
      num_replicas: 4
    engine_args:
      model: Wan-AI/Wan2.2-T2V-A14B-Diffusers
      parallel_config:
        world_size: 2

The runtime splits the device list into contiguous chunks per replica. With the second example, replica 0 uses 0,1, replica 1 uses 2,3, replica 2 uses 4,5, and replica 3 uses 6,7.

Serving

Start the server with the stage config:

vllm serve Qwen/Qwen-Image \
  --omni \
  --port 8099 \
  --stage-configs-path /path/to/diffusion_multi_replica.yaml

Then send normal OpenAI-compatible diffusion chat requests. The orchestrator routes new requests across replicas and keeps updates/abort calls on the same replica for each request.

Benchmarking

Benchmark each replica count by changing runtime.num_replicas and runtime.devices, restarting the server, then running:

python benchmarks/diffusion/diffusion_benchmark_serving.py \
  --base-url http://localhost:8099 \
  --model Qwen/Qwen-Image \
  --task t2i \
  --dataset random \
  --num-prompts 8 \
  --request-rate inf \
  --num-inference-steps 2 \
  --width 256 --height 256 \
  --output-file diffusion_replica_4.json

For a quick scaling sweep, test num_replicas values 1, 2, 3, and 4 with the same prompt count and generation parameters. Compare throughput and end-to-end latency from the JSON outputs.

Notes and Limitations

• Multi-replica serving is currently supported for diffusion stages. Single-stage mode still rejects non-diffusion replicas.
• runtime.devices must provide exactly num_replicas * devices_per_replica devices.
• Request-level latency may stay similar or slightly increase because each request still runs on one replica.
• Throughput improves only when enough concurrent requests are issued and each replica has enough GPU memory.