bjf-frzandHongsheng Liu
be667a8067
Signed-off-by: bjf-frz <frz123db@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
120 lines
4.2 KiB
Markdown
120 lines
4.2 KiB
Markdown
# Request-Level Batching
|
|
|
|
Request-level batching lets diffusion serving combine multiple compatible
|
|
logical requests into one pipeline `forward()` call. Each prompt remains a
|
|
separate request with its own `request_id`, sampling parameters, seed, output,
|
|
error, and abort state. The scheduler decides which requests can run together.
|
|
|
|
!!! warning "Prompt List Semantics"
|
|
|
|
Diffusion request-level batching does not support a top-level packed
|
|
list-prompt request. Submit multiple prompts as independent requests and let
|
|
the scheduler batch compatible in-flight requests. Multimodal payloads stay
|
|
inside a single prompt dict, for example
|
|
`{"prompt": "...", "multi_modal_data": {"image": image}}`.
|
|
|
|
## Enablement
|
|
|
|
Request-level execution is the default diffusion request mode when
|
|
`step_execution=False`. `max_num_seqs` controls the maximum number of compatible
|
|
requests in one scheduler wave:
|
|
|
|
- `max_num_seqs=1` is the serial request path.
|
|
- `max_num_seqs>1` allows fused request-level batching for pipelines that
|
|
declare request-batch support.
|
|
|
|
Increase `max_num_seqs` above `1` to allow the request scheduler to keep more
|
|
than one compatible request active:
|
|
|
|
```bash
|
|
vllm serve Qwen/Qwen-Image --omni \
|
|
--port 8091 \
|
|
--max-num-seqs 4
|
|
```
|
|
|
|
For bursty online traffic, you can also set a small admission wait window. This
|
|
lets the engine wait briefly before the first `schedule()` of a new wave so
|
|
nearby compatible requests can arrive and share the same fused forward pass:
|
|
|
|
```bash
|
|
vllm serve Qwen/Qwen-Image --omni \
|
|
--port 8091 \
|
|
--max-num-seqs 4 \
|
|
--request-batch-max-wait-ms 20
|
|
```
|
|
|
|
`--request-batch-max-wait-ms 0` is the default and disables admission waiting,
|
|
so there is no added wait latency.
|
|
|
|
For deploy YAMLs, configure the diffusion stage engine args:
|
|
|
|
```yaml
|
|
stage_args:
|
|
- stage_id: 0
|
|
stage_type: diffusion
|
|
engine_args:
|
|
max_num_seqs: 4
|
|
request_batch_max_wait_ms: 20
|
|
```
|
|
|
|
## Compatibility
|
|
|
|
Only pipelines that declare request-batch support can use the fused
|
|
multi-request `pipeline.forward(batch)` path. Pipelines that do not support this
|
|
contract can still use request-level execution with `max_num_seqs=1`, which is
|
|
the serial request path. Setting `max_num_seqs>1` for a pipeline without
|
|
request-batch support fails early during engine initialization.
|
|
|
|
The scheduler batches only compatible requests. Compatibility is based on
|
|
shape-sensitive and guidance-sensitive sampling fields, including resolution,
|
|
frame count, CFG settings, output count, and LoRA identity. Requests with
|
|
different LoRA adapters or scales are kept in separate batches so the worker
|
|
activates one adapter per batch.
|
|
|
|
Request-level batching applies when `step_execution=False`. For the separate
|
|
step-wise runtime, see [Step Execution](step_execution.md).
|
|
|
|
## Tuning
|
|
|
|
- `max_num_seqs` caps the number of active compatible requests in one scheduler
|
|
wave. `1` means serial request execution; values above `1` enable fused
|
|
request-level batching when the pipeline supports it.
|
|
- `request_batch_max_wait_ms` is an upper bound on extra admission wait before a
|
|
new wave starts. Keep it small for latency-sensitive serving; values such as
|
|
`10` to `50` ms are a practical starting range for bursty HTTP ingress.
|
|
- `0` disables admission waiting and preserves the lowest first-request latency.
|
|
- FIFO ordering is conservative: an incompatible request at the front of the
|
|
waiting queue can block later compatible requests from joining the current
|
|
batch.
|
|
|
|
## Python API
|
|
|
|
When constructing `Omni`, pass the same engine arguments:
|
|
|
|
```python
|
|
from vllm_omni.entrypoints.omni import Omni
|
|
|
|
omni = Omni(
|
|
model="Qwen/Qwen-Image",
|
|
max_num_seqs=4,
|
|
request_batch_max_wait_ms=20.0,
|
|
)
|
|
|
|
outputs = omni.generate(
|
|
[
|
|
"a cup of coffee on a table",
|
|
"a toy dinosaur on a sandy beach",
|
|
"a fox waking up in bed and yawning",
|
|
]
|
|
)
|
|
```
|
|
|
|
`Omni.generate([...])` submits each list item as its own logical diffusion
|
|
request. The runtime may batch those requests internally when their sampling
|
|
parameters are compatible.
|
|
|
|
## For Contributors
|
|
|
|
For implementation details and model-author guidance, see
|
|
[Request-Level Batching for Diffusion](../../design/feature/diffusion_request_level_batching.md).
|