vllm-omni/docs/user_guide/diffusion/request_batching.md

Request-Level Batching

Request-level batching lets diffusion serving combine multiple compatible logical requests into one pipeline forward() call. Each prompt remains a separate request with its own request_id, sampling parameters, seed, output, error, and abort state. The scheduler decides which requests can run together.

!!! warning "Prompt List Semantics"

Diffusion request-level batching does not support a top-level packed
list-prompt request. Submit multiple prompts as independent requests and let
the scheduler batch compatible in-flight requests. Multimodal payloads stay
inside a single prompt dict, for example
`{"prompt": "...", "multi_modal_data": {"image": image}}`.

Enablement

Request-level execution is the default diffusion request mode when step_execution=False. max_num_seqs controls the maximum number of compatible requests in one scheduler wave:

• max_num_seqs=1 is the serial request path.
• max_num_seqs>1 allows fused request-level batching for pipelines that declare request-batch support.

Increase max_num_seqs above 1 to allow the request scheduler to keep more than one compatible request active:

vllm serve Qwen/Qwen-Image --omni \
  --port 8091 \
  --max-num-seqs 4

For bursty online traffic, you can also set a small admission wait window. This lets the engine wait briefly before the first schedule() of a new wave so nearby compatible requests can arrive and share the same fused forward pass:

vllm serve Qwen/Qwen-Image --omni \
  --port 8091 \
  --max-num-seqs 4 \
  --request-batch-max-wait-ms 20

--request-batch-max-wait-ms 0 is the default and disables admission waiting, so there is no added wait latency.

For deploy YAMLs, configure the diffusion stage engine args:

stage_args:
  - stage_id: 0
    stage_type: diffusion
    engine_args:
      max_num_seqs: 4
      request_batch_max_wait_ms: 20

Compatibility

Only pipelines that declare request-batch support can use the fused multi-request pipeline.forward(batch) path. Pipelines that do not support this contract can still use request-level execution with max_num_seqs=1, which is the serial request path. Setting max_num_seqs>1 for a pipeline without request-batch support fails early during engine initialization.

The scheduler batches only compatible requests. Compatibility is based on shape-sensitive and guidance-sensitive sampling fields, including resolution, frame count, CFG settings, output count, and LoRA identity. Requests with different LoRA adapters or scales are kept in separate batches so the worker activates one adapter per batch.

Request-level batching applies when step_execution=False. For the separate step-wise runtime, see Step Execution.

Tuning

• max_num_seqs caps the number of active compatible requests in one scheduler wave. 1 means serial request execution; values above 1 enable fused request-level batching when the pipeline supports it.
• request_batch_max_wait_ms is an upper bound on extra admission wait before a new wave starts. Keep it small for latency-sensitive serving; values such as 10 to 50 ms are a practical starting range for bursty HTTP ingress.
• 0 disables admission waiting and preserves the lowest first-request latency.
• FIFO ordering is conservative: an incompatible request at the front of the waiting queue can block later compatible requests from joining the current batch.

Python API

When constructing Omni, pass the same engine arguments:

from vllm_omni.entrypoints.omni import Omni

omni = Omni(
    model="Qwen/Qwen-Image",
    max_num_seqs=4,
    request_batch_max_wait_ms=20.0,
)

outputs = omni.generate(
    [
        "a cup of coffee on a table",
        "a toy dinosaur on a sandy beach",
        "a fox waking up in bed and yawning",
    ]
)

Omni.generate([...]) submits each list item as its own logical diffusion request. The runtime may batch those requests internally when their sampling parameters are compatible.

For Contributors

For implementation details and model-author guidance, see Request-Level Batching for Diffusion.