anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[docs] consolidate diffusion execution modes (#5599)

Browse Source 
Signed-off-by: bjf-frz <frz123db@gmail.com>
This commit is contained in:
bjf-frz authored and GitHub committed 2026-08-05 22:41:18 +08:00
1 parent 589afcef6c
commit 9f978923f2
18 files changed
+475 -682
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/.nav.yml
+1 -4
View File
@@ -71,8 +71,7 @@ nav:
- CPU Offloading: user_guide/diffusion/cpu_offload_diffusion.md
- LoRA: user_guide/diffusion/lora.md
- Custom Pipeline: features/custom_pipeline.md
- Request-Level Batching: user_guide/diffusion/request_batching.md
- Step Execution: user_guide/diffusion/step_execution.md
- Execution Modes: user_guide/diffusion/execution_modes.md
- Quantization:
- Overview: user_guide/quantization/overview.md
- Online Quantization: user_guide/quantization/online.md
@@ -120,8 +119,6 @@ nav:
- design/feature/async_chunk.md
- design/feature/omni_async_output_materialization.md
- design/feature/vae_parallel.md
- design/feature/diffusion_step_execution.md
- design/feature/diffusion_request_level_batching.md
- design/feature/diffusion_continuous_batching.md
- Module Design:
- design/module/ar_module.md
docs/configuration/stage_configs.md
+2 -2
View File
@@ -387,8 +387,8 @@ that support request-level batching with `step_execution` disabled.
Use this together with `max_num_seqs > 1` for bursty serving traffic. `0`
disables admission waiting and preserves the lowest first-request latency.
For diffusion request-level batching tuning, see
[Request-Level Batching](../user_guide/diffusion/request_batching.md).
For diffusion execution and batching tuning, see
[Diffusion Execution Modes](../user_guide/diffusion/execution_modes.md).
Default: `0.0`
docs/contributing/model/adding_diffusion_model.md
+4 -4
View File
@@ -755,7 +755,8 @@ omni = Omni(model="your-model", ulysses_degree=2, ring_degree=2)
### Step Execution
See detailed design guide: [How to add step execution support](../../design/feature/diffusion_step_execution.md)
See the detailed
[Diffusion Continuous Batching design guide](../../design/feature/diffusion_continuous_batching.md).
Use this only when your pipeline can be split into stable request-scoped and
step-scoped phases. The reference implementation is
@@ -769,9 +770,8 @@ step-scoped phases. The reference implementation is
Do not enable `step_execution=True` until those four methods are implemented
and validated against the request-level path.
If you want the pipeline to work with the experimental batched step-wise path
(`max_num_seqs > 1`), also see:
[Continuous Batching for Step-Wise Diffusion](../../design/feature/diffusion_continuous_batching.md).
The same design guide covers the experimental batched step-wise path used when
`max_num_seqs > 1`.
If you expose this in example scripts or recipes, keep it opt-in. Surface
runtime features like `step_execution` as optional flags instead of silently
docs/design/feature/diffusion_continuous_batching.md
+196 -129
View File
@@ -1,165 +1,232 @@
# Continuous Batching for Step-Wise Diffusion
# Diffusion Continuous Batching
!!! warning "Experimental Feature"
This feature is experimental. It currently applies only to native
diffusion pipelines running with `step_execution=True`.
This document describes the batching extension built on top of
[Diffusion Step Execution](diffusion_step_execution.md). The base
step-execution contract is unchanged. The batching work is mainly in the
scheduler and runner layers.
## Why It Helps
Step-wise execution breaks a long denoise loop into scheduler-visible units.
That gives the runtime a place to admit other compatible requests between
steps instead of waiting for an entire request to finish.
This matters most in low-MFU or bursty serving scenarios:
- one request's denoise step may not fully saturate the GPU
- several compatible requests can share the same denoise forward pass
- throughput and device utilization can improve without changing request-local
scheduler state
This is **not** a guaranteed single-request latency win. The main benefit is
usually higher utilization and better throughput when the workload contains
multiple in-flight compatible requests.
This document describes the unified diffusion execution architecture, including
request execution, request-level batching, step execution, continuous batching,
and output delivery. For user-facing configuration and CLI examples, see
[Diffusion Execution Modes](../../user_guide/diffusion/execution_modes.md).
## Overview
With continuous batching enabled:
`DiffusionEngine` selects an execution mode from `step_execution` and uses
`max_num_seqs` as that mode's scheduler capacity:
- the scheduler may keep multiple compatible requests active at the same time
- the runner packs request-local step state into one `InputBatch`
- `denoise_step()` runs on that batch
- `step_scheduler()` and `post_decode()` still run per request
| Configuration | Engine mode | Scheduler | Execution |
|---|---|---|---|
| `step_execution=False`, `max_num_seqs=1` | `REQUEST_BATCH` | `RequestScheduler` | One complete request-level `forward()` |
| `step_execution=False`, `max_num_seqs>1` | `REQUEST_BATCH` | `RequestScheduler` | One fused `forward()` over compatible requests |
| `step_execution=True`, `max_num_seqs=1` | `STEP_BATCH` | `StepScheduler` | One request advanced one denoise step per scheduler tick |
| `step_execution=True`, `max_num_seqs>1` | `STEP_BATCH` | `StepScheduler` | Compatible requests advanced together in step waves |
The current implementation is conservative:
Serial request execution is handled inside `REQUEST_BATCH`; it is not a
separate engine mode. Similarly, continuous batching is the multi-request
configuration of `STEP_BATCH`, not an independent execution mode.
- only compatible requests are batched together
- per-request progress and completion remain independent
The engine performs the following initialization:
Here, "continuous batching" means the step-wise path enabled by
`step_execution=True`. Request-mode `DiffusionRequestBatch` is static
request-level batching for one full pipeline `forward()` call; it does not
admit or remove requests between denoise steps.
1. Resolve model-specific pre- and post-processing hooks.
2. Select `REQUEST_BATCH` or `STEP_BATCH`.
3. Construct the configured `DiffusionExecutor`.
4. Construct `RequestScheduler` or `StepScheduler`, unless a scheduler was
explicitly injected.
5. Bind `execute_batch` or `execute_step`.
## Enablement
`DiffusionEngine.make_engine()` owns startup warmup after engine construction.
This keeps construction, backend selection, and warmup failure cleanup in one
lifecycle.
Use `--step-execution` as the feature gate, then increase `--max-num-seqs`
above `1` if you want batching:
## Unified Output Stream
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--step-execution \
--max-num-seqs 8
All asynchronous diffusion requests use the same per-request output stream,
regardless of whether the caller exposes streaming results to the user.
```python
async for output in engine.step_streaming(request):
...
```
`--max-num-seqs 1` keeps the step-wise path without enabling batching.
When a request is admitted, the engine creates an
`asyncio.Queue[DiffusionOutput]` keyed by its request ID. The busy loop sends
both intermediate chunks and final results through that queue:
For a reproducible replay flow using the bundled serving benchmark, see the
Qwen-Image replay commands in
[`benchmarks/diffusion/README.md`](gh-file:benchmarks/diffusion/README.md)
and
[`benchmarks/diffusion/performance_dashboard/qwen_image_serving_performance.md`](gh-file:benchmarks/diffusion/performance_dashboard/qwen_image_serving_performance.md).
```text
request
-> scheduler.add_request()
-> scheduler.schedule()
-> execute_batch() or execute_step()
-> scheduler.update_from_output()
-> per-request output queue
-> step_streaming()
```
## Scheduler
Streaming callers forward every yielded output. Non-streaming callers consume
the same generator to completion and return only its final output. This avoids
maintaining separate future-based and queue-based result paths.
The scheduler derives its batch capacity from `max_num_seqs` through
`max_num_running_reqs`.
`DiffusionEngine.step()` and
`async_add_req_and_wait_for_response()` remain deprecated compatibility
wrappers. New integrations should consume `step_streaming()` or
`async_add_req_and_stream_response()`.
Batch admission is gated by
[`StepBatchSamplingParamsKey`](gh-file:vllm_omni/diffusion/sched/interface.py),
which is built from shape-sensitive and CFG-sensitive sampling fields. This is
the core correctness rule for batching: requests are only co-batched when they
share the same denoise tensor contract.
### Completion and Cancellation
There are three important details:
Output delivery and scheduler lifecycle are related but separate:
- `num_inference_steps` is not part of the key, so requests with different
total step counts can still share a batch
- requests also do not need to be at the same current denoise progress; active
requests can continue batching even when their current step indices diverge
- admission is still FIFO, so an incompatible request at the head of the
waiting queue blocks later compatible requests
- A terminal scheduler result always finalizes request state.
- A disconnected or cancelled consumer removes only its delivery queue.
- The scheduler may still finish and clean up the request after its queue has
been removed.
- Engine shutdown sends an error output to all remaining streams.
Today that compatibility rule is still shape-sensitive. `height`, `width`,
`num_frames`, and CFG-related fields remain part of the key, so different
resolutions or incompatible guidance settings do **not** co-batch yet. The
key also covers LoRA identity (`lora_int_id`, `lora_scale`), so requests
targeting different adapters or scales run in separate batches and the
worker can activate exactly one adapter per step.
This separation prevents cancelled consumers from leaving active request IDs in
the scheduler.
The scheduler batching unit is one logical `OmniDiffusionRequest`. In the
step-wise path, runtime tensor batching is represented as `StepInputBatch`. For
request-mode prompt semantics, see
[Request-Level Batching](../../user_guide/diffusion/request_batching.md).
## Request-Batch Execution
## Runner
Request-batch execution runs the complete diffusion pipeline for each scheduler
wave. Each `OmniDiffusionRequest` remains an independent logical request with
its own prompt, sampling parameters, seed, request ID, output, error, and abort
state.
The runner keeps persistent per-request execution state in
[`StepRequestState`](gh-file:vllm_omni/diffusion/worker/utils.py),
while the scheduler owns a separate lightweight
[`SchedulerRequestState`](gh-file:vllm_omni/diffusion/sched/interface.py)
for queueing and lifecycle tracking.
With `max_num_seqs=1`, `execute_batch()` handles the serial request path. With a
larger value, a pipeline can opt into fused execution by declaring:
For each step, the runner builds an
[`InputBatch`](gh-file:vllm_omni/diffusion/worker/input_batch.py) from the
active request states:
```python
supports_request_batch = True
```
- prompt embeddings and masks are normalized and padded
- dynamic tensors such as `latents` and `timesteps` are gathered each step
- buffers are reused when batch composition stays the same
Its `forward()` method must accept `DiffusionRequestBatch` and return one
`DiffusionOutput` per request. The runner validates the result count and maps
outputs back to their original request IDs with `BatchRunnerOutput`.
The step-wise batched path is:
Current in-tree request-batch implementations include the Flux, LTX-2, SD3, and
Qwen-Image pipelines. Treat the pipeline capability flag in the source as
authoritative because support continues to expand.
1. Run `prepare_encode()` for newly admitted requests.
### Request-Batch Data Flow
- `DiffusionSchedulerOutput` carries scheduled request IDs and payloads.
- `DiffusionRequestBatch` provides the pipeline-facing static batch.
- `BatchRunnerOutput` carries per-request results back to the engine.
Request-local setup, such as seed handling and output/error mapping, remains
per request. Cache refresh, homogeneous LoRA activation, and
`pipeline.forward(req_batch)` can be shared by the batch.
### Compatibility and Admission
`RequestBatchSamplingParamsKey` contains fields that affect the tensor contract,
including shape, guidance, output count, and LoRA identity. Only compatible
requests can share a fused forward call.
Admission is FIFO and conservative. An incompatible request at the front of the
waiting queue can prevent later compatible requests from joining the current
wave.
`request_batch_max_wait_ms` optionally delays the first schedule operation of a
new wave so bursty arrivals can accumulate. It applies only when:
- the pipeline supports request batching;
- step execution is disabled;
- no requests are currently running; and
- the configured wait is greater than zero.
The wait ends when the queue reaches capacity, stabilizes briefly, reaches its
deadline, or the engine stops.
## Step-Batch Execution
Step execution exposes denoising progress to the scheduler. A supporting
pipeline implements four stateful operations:
| Operation | Responsibility |
|---|---|
| `prepare_encode(state)` | Validate input, encode prompts, initialize latents and timesteps, and create request-local scheduler state |
| `denoise_step(input_batch, *, states=...)` | Run one denoise forward for the scheduler-provided request states |
| `step_scheduler(state, noise_pred)` | Update latents and advance request progress |
| `post_decode(state)` | Decode and postprocess a completed request or output boundary |
Persistent request state lives in `StepRequestState`. Pipeline-specific fields
that do not belong in the shared contract should be stored in `state.extra`.
Queueing and lifecycle metadata remain in the scheduler's request state.
Current native pipelines that explicitly enable step execution include
Qwen-Image, HunyuanImage3, and Helios. Step execution alone does not imply
continuous-batching support: Qwen-Image accepts batched step states.
HunyuanImage3 accepts batched step states only when its resolved self-attention
backend is `TORCH_SDPA`; otherwise it rejects groups larger than one request.
Configure `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` or
`diffusion_attention_config.default.backend=TORCH_SDPA` when
`max_num_seqs>1`. See the
[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md)
for its validated configuration. Helios supports only a single active step
request and must use `max_num_seqs=1`.
### Continuous Batching
When `max_num_seqs>1`, `StepScheduler` can keep multiple compatible requests
active. The runner gathers their state into `InputBatch`, performs one batched
denoise forward, then applies scheduler updates per request:
1. Receive transferred KV payloads, then run `prepare_encode()` for newly
admitted requests.
2. Build or refresh `InputBatch`.
3. Run one batched `denoise_step(input_batch)`.
4. Slice the batched `noise_pred` back per request.
5. Run per-request `step_scheduler()`.
6. Run `post_decode()` only for requests that finished denoising.
7. Scatter updated latents back into persistent request state with
[`scatter_latents()`](gh-file:vllm_omni/diffusion/worker/input_batch.py).
3. Run one batched `denoise_step(input_batch, states=states)`.
4. Slice noise predictions back to each request.
5. Run each request's `step_scheduler()`.
6. Run `post_decode()` at a chunk boundary or request completion.
7. Scatter updated latents back into persistent request state.
This keeps the shared work limited to the denoise forward pass while preserving
request-local scheduler state and outputs.
`StepBatchSamplingParamsKey` protects the batched tensor contract. Requests can
have different total step counts and current step indices, but shape-sensitive,
CFG-sensitive, and LoRA fields must be compatible. FIFO head-of-line blocking
also applies here.
## Engine
Chunk-capable pipelines may emit intermediate results. Final-only step
pipelines emit when the request finishes even if `streaming_output=True`.
[`DiffusionEngine`](gh-file:vllm_omni/diffusion/diffusion_engine.py) provides
the background loop and async add-request path needed for multiple requests to
accumulate in the scheduler.
## Model Author Guidelines
When `step_execution=True`, the engine routes work through the step-wise
executor path. The continuous batching behavior is defined by scheduler-side
compatibility gating and runner-side `StepInputBatch` packing.
When converting a request-level pipeline:
## Current Limitations
- Reuse the same helpers as `forward()` to avoid behavior drift.
- Copy request-scoped scheduler state into `state.scheduler`.
- Advance `state.step_index` only in `step_scheduler()`.
- Keep model forward work in `denoise_step()`.
- Keep latent mutation in `step_scheduler()`.
- Keep final decoding equivalent to the tail of `forward()`.
- Store masks, condition latents, and other request-local tensors in the
request state rather than on the shared pipeline.
- Experimental feature; use `max_num_seqs=1` for the conservative single-request
step path.
- Only native pipelines that already support `step_execution=True`.
- Only homogeneous batches keyed by `StepBatchSamplingParamsKey` are supported.
- `cache_backend`, KV transfer, and other request-mode extras are not wired
into the batched step-wise path yet.
- Future work can relax the current same-shape restriction with richer
heterogeneous batching policies such as bucketing or padded execution for
different resolutions.
Before setting `supports_step_execution = True`, validate output parity for the
same seed and sampling parameters, scheduler isolation across concurrent
requests, abort cleanup, progress reporting, and all supported CFG paths.
## Related Files
Before setting `supports_request_batch = True`, validate the batch input
contract, one-output-per-request result shape, request identity and error
mapping, seeded concurrency, LoRA compatibility, and tensor IPC.
- Scheduler base:
[`vllm_omni/diffusion/sched/base_scheduler.py`](gh-file:vllm_omni/diffusion/sched/base_scheduler.py)
- Scheduler interface:
[`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py)
- Step scheduler:
[`vllm_omni/diffusion/sched/step_scheduler.py`](gh-file:vllm_omni/diffusion/sched/step_scheduler.py)
- Runner:
[`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py)
- Input batch:
[`vllm_omni/diffusion/worker/input_batch.py`](gh-file:vllm_omni/diffusion/worker/input_batch.py)
- Tests:
[`tests/diffusion/test_diffusion_scheduler.py`](gh-file:tests/diffusion/test_diffusion_scheduler.py)
## Limitations
- Request batching requires an explicit pipeline capability when
`max_num_seqs>1`.
- Step execution requires the four-method stateful pipeline contract.
- Both batching paths currently require homogeneous compatibility keys.
- FIFO scheduling can reduce batching opportunities.
- `request_batch_max_wait_ms` trades first-request latency for burst
coalescing.
- All diffusion cache backends are currently unsupported in step mode.
- KV transfer is supported for newly admitted step requests; some other
request-mode extras remain unsupported.
- Step continuous batching remains experimental; use `max_num_seqs=1` for the
conservative step path.
## Related Implementation
- Engine: [`vllm_omni/diffusion/diffusion_engine.py`](gh-file:vllm_omni/diffusion/diffusion_engine.py)
- Model contract: [`vllm_omni/diffusion/models/interface.py`](gh-file:vllm_omni/diffusion/models/interface.py)
- Scheduler interface: [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py)
- Request scheduler: [`vllm_omni/diffusion/sched/request_scheduler.py`](gh-file:vllm_omni/diffusion/sched/request_scheduler.py)
- Step scheduler: [`vllm_omni/diffusion/sched/step_scheduler.py`](gh-file:vllm_omni/diffusion/sched/step_scheduler.py)
- Runner: [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py)
- Request batch: [`vllm_omni/diffusion/worker/request_batch.py`](gh-file:vllm_omni/diffusion/worker/request_batch.py)
- Step input batch: [`vllm_omni/diffusion/worker/input_batch.py`](gh-file:vllm_omni/diffusion/worker/input_batch.py)
docs/design/feature/diffusion_request_level_batching.md
-175
View File
@@ -1,175 +0,0 @@
# Request-Level Batching for Diffusion
This document describes the request-mode batching path for diffusion pipelines.
For end-user enablement and tuning, see
[Request-Level Batching](../../user_guide/diffusion/request_batching.md).
This is separate from
[Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md).
Request-level batching runs one full pipeline `forward()` over a static batch of
compatible requests. Step-wise continuous batching admits work between denoise
steps when `step_execution=True`.
## Why It Helps
The request-level design avoids coupling several logical requests to one request
object. This keeps request identity, abort/error handling, and per-request
metadata unambiguous while still allowing one fused pipeline forward pass for
bursty or concurrent traffic.
## Overview
With request-level batching enabled:
- each `OmniDiffusionRequest` contains one `prompt` and one `request_id`
- the scheduler groups compatible waiting requests into one scheduler wave
- `DiffusionRequestBatch` wraps the scheduled requests for pipeline `forward()`
- batch-capable pipelines return `list[DiffusionOutput]`, one output per
request
- `BatchRunnerOutput` maps each result back to its original `request_id`
Pipelines opt in with `supports_request_batch = True` and a `forward()` method
that accepts `DiffusionRequestBatch` and returns `list[DiffusionOutput]`.
Pipelines that do not opt in keep the existing per-request execution path.
## Enablement
Request-level batching is the request-mode path, so `step_execution` must remain
disabled. Increase `max_num_seqs` above `1` to let the scheduler keep multiple
compatible requests active:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4
```
For bursty online ingress, `request_batch_max_wait_ms` can add a bounded
admission wait before the first `schedule()` of a scheduler wave:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4 \
--request-batch-max-wait-ms 20
```
`request_batch_max_wait_ms=0` disables this wait and is the default.
## Request Contract
`OmniDiffusionRequest` represents one logical request. It owns one prompt,
sampling parameters, request id, and request-local metadata. Runtime batches are
formed by the scheduler and represented separately from the request payload.
Runtime batching is represented by:
- [`DiffusionSchedulerOutput`](gh-file:vllm_omni/diffusion/sched/interface.py)
for scheduled request ids and request payloads
- [`DiffusionRequestBatch`](gh-file:vllm_omni/diffusion/worker/request_batch.py)
for the pipeline-facing request batch
- [`BatchRunnerOutput`](gh-file:vllm_omni/diffusion/worker/utils.py) for
per-request results
`DiffusionRequestBatch` intentionally exposes compatibility properties such as
`prompts`, `sampling_params`, `request_id`, and `kv_sender_info` so migrated
pipelines can stay close to upstream code while using a batch-aware contract.
## Scheduler
The scheduler derives its capacity from `max_num_seqs` through
`max_num_running_reqs`. It exposes waiting/running queue counters so the engine
can decide whether admission wait is useful before scheduling a new wave.
Batch compatibility is controlled by
[`RequestBatchSamplingParamsKey`](gh-file:vllm_omni/diffusion/sched/interface.py). The key
contains shape-sensitive and guidance-sensitive fields, including output count
and LoRA identity. Requests with incompatible shapes, CFG settings, output
counts, LoRA adapters, or LoRA scales are kept in separate batches.
Admission is conservative:
- the scheduler only batches compatible requests
- FIFO ordering is preserved
- an incompatible request at the head of the waiting queue blocks later
compatible requests
## Engine
[`DiffusionEngine`](gh-file:vllm_omni/diffusion/diffusion_engine.py) resolves
request-batch capability during initialization from the configured pipeline
class, including custom pipeline classes.
The capability check uses the pipeline class attribute
`supports_request_batch = True`. Pipelines that set this attribute must implement
a request-batch-compatible `forward()` contract and return one
`DiffusionOutput` per request; the runner validates that return shape at runtime.
When the selected pipeline is batch-capable and `step_execution=False`, request
mode routes scheduler waves through the batch executor path. Otherwise it keeps
the per-request executor path.
The optional admission wait runs only when:
- request batching is supported
- `step_execution=False`
- `request_batch_max_wait_ms > 0`
- no requests are currently running
The wait exits early when the waiting queue reaches capacity, when the queue is
stable for a short window, when the deadline expires, or when the engine stops.
## Executor And Runner
The executor exposes two request-mode entries:
- `execute_request`: one worker call per scheduled request
- `execute_batch`: one worker call for the whole `DiffusionSchedulerOutput`
On the batch path, the worker builds a `DiffusionRequestBatch` and runs the
pipeline once. Request-local setup remains per request:
- KV transfer metadata
- random generator and seed handling
- request output/error/abort mapping
Shared batch setup happens once per batch when possible:
- cache refresh
- LoRA activation for the homogeneous adapter key
- pipeline `forward(req_batch)`
Large tensor IPC still uses the shared-memory packing path. The packer traverses
both normal `RunnerOutput.result` wrappers and nested batch results so batched
outputs do not fall back to pickle IPC for tensor payloads.
## Current Limitations
- Only pipelines that declare the request-batch contract use fused batch
execution.
- Batches are homogeneous under `RequestBatchSamplingParamsKey`; heterogeneous resolution or
incompatible guidance settings do not co-batch yet.
- FIFO scheduling can reduce batching opportunities when an incompatible
request is at the front of the queue.
- `request_batch_max_wait_ms` improves burst coalescing but can add latency to
the first request in a scheduler wave. Keep it small for latency-sensitive
serving.
- Step-wise continuous batching is documented separately and only applies when
`step_execution=True`.
## Related Files
- Request object and request batch:
[`vllm_omni/diffusion/request.py`](gh-file:vllm_omni/diffusion/request.py)
- Scheduler interface:
[`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py)
- Scheduler base:
[`vllm_omni/diffusion/sched/base_scheduler.py`](gh-file:vllm_omni/diffusion/sched/base_scheduler.py)
- Engine:
[`vllm_omni/diffusion/diffusion_engine.py`](gh-file:vllm_omni/diffusion/diffusion_engine.py)
- Worker runner:
[`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py)
- Executor interface:
[`vllm_omni/diffusion/executor/abstract.py`](gh-file:vllm_omni/diffusion/executor/abstract.py)
- Tests:
[`tests/diffusion/test_diffusion_engine.py`](gh-file:tests/diffusion/test_diffusion_engine.py)
docs/design/feature/diffusion_step_execution.md
-131
View File
@@ -1,131 +0,0 @@
# Adding Step Execution Support for Diffusion Pipelines
This guide documents vLLM-Omni's stepwise diffusion contract for model authors
and contributors implementing `step_execution=True` support for a diffusion
pipeline.
For end-user enablement, supported models, and current limitations, see
[Step Execution](../../user_guide/diffusion/step_execution.md).
This document describes the base step-execution contract only. For the
experimental batching policy layered on top of the step-wise path, see
[Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md).
## Current Support Scope
`step_execution` is **not** a generic diffusion toggle. It only works for
pipelines that implement the segmented stateful contract in
[`vllm_omni/diffusion/models/interface.py`](gh-file:vllm_omni/diffusion/models/interface.py).
This page is intentionally author-facing. Treat runtime enablement
(`step_execution=True` in Python or `--step-execution` in serving) as an
opt-in user knob layered on top of the implementation contract below.
Current in-tree support:
| Pipeline | Example models | Step execution |
|----------|----------------|----------------|
| `QwenImagePipeline` | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes |
| All other diffusion pipelines | `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, `QwenImageLayeredPipeline`, GLM-Image, Wan, Flux, etc. | No |
Current engine/runtime limitations:
- Continuous batching with `max_num_seqs > 1` is experimental and documented in
[Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md).
Keep `max_num_seqs=1` for the conservative single-request step path.
- `cache_backend` is not supported in step mode.
- Request-mode extras such as KV transfer are not wired into step mode yet.
- Unsupported pipelines now fail early during model loading instead of failing on the first request.
## Execution Contract
Step mode is driven by four pipeline methods plus the shared mutable request
state object:
- `prepare_encode(state)`: one-time request preparation.
- `denoise_step(state)`: compute the noise prediction for the current step.
- `step_scheduler(state, noise_pred)`: mutate latents and advance step state.
- `post_decode(state)`: decode the final output after denoising is complete.
The state lives in
[`vllm_omni/diffusion/worker/utils.py`](gh-file:vllm_omni/diffusion/worker/utils.py)
as `StepRequestState`. Store request-scoped tensors there, or use
`state.extra` for model-specific fields that do not justify extending the core
dataclass.
The worker-side step loop lives in
[`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py):
1. `prepare_encode()` runs once for a new request.
2. `denoise_step()` runs every scheduler tick.
3. `step_scheduler()` mutates `state.latents` and advances `state.step_index`.
4. `post_decode()` runs exactly once after `state.denoise_completed` becomes true.
## Recommended Split
When converting an existing request-level `forward()` pipeline, keep the split
strict and mechanical:
| Request-level phase | Stepwise method | What belongs there |
|---------------------|-----------------|--------------------|
| Input validation, prompt encoding, latent init, timestep prep, per-request scheduler creation | `prepare_encode()` | Anything that should happen once per request |
| Transformer forward / noise prediction | `denoise_step()` | Pure denoise computation for the current timestep |
| `scheduler.step(...)` and `step_index += 1` | `step_scheduler()` | Only latent/state mutation for one step |
| VAE decode / postprocess | `post_decode()` | Final decode only |
Keep the stepwise path reusing the same helpers as the request-level path
whenever possible. Reimplementing the denoise loop from scratch is the easiest
way to introduce behavioral drift.
## Qwen-Image Reference
[`pipeline_qwen_image.py`](gh-file:vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py)
is the reference implementation and is split correctly for the current
contract:
- `prepare_encode()` reuses `_prepare_generation_context()` so prompt encoding,
latent init, timestep creation, CFG setup, and shape bookkeeping stay aligned
with `forward()`.
- `prepare_encode()` deep-copies `self.scheduler` **after**
`prepare_timesteps()` so request-specific scheduler state is isolated.
- `denoise_step()` reuses `_build_denoise_kwargs()` plus
`predict_noise_maybe_with_cfg()`, so sequential CFG, CFG-parallel, and
non-CFG behavior stay identical to the request-level path.
- `step_scheduler()` only calls
`scheduler_step_maybe_with_cfg(..., per_request_scheduler=state.scheduler)`
and increments `state.step_index`.
- `post_decode()` reuses `_decode_latents()`, so the final image decode matches
the normal `forward()` path.
That decomposition is the target pattern for future models.
## Rules For New Pipelines
- Do not keep request-scoped scheduler state on `self.scheduler`. Copy it into
`state.scheduler` during `prepare_encode()`.
- Do not mutate `state.step_index` inside `denoise_step()`. Only
`step_scheduler()` should advance the step.
- Do not decode partial outputs in `denoise_step()` or `step_scheduler()`.
- If the request-level pipeline has condition latents, masks, or edit-specific
tensors, store them in `state` or `state.extra`, not in global pipeline
attributes.
- Preserve CFG behavior by sharing the same helper path used by `forward()`.
- Keep `post_decode()` equivalent to the tail of `forward()`.
## Validation Checklist
Before marking a pipeline as `supports_step_execution = True`, verify:
- Stepwise output matches request-level output for the same seed and sampling params.
- Per-request scheduler state is isolated across concurrent requests.
- Abort during denoise does not leak cached state.
- `step_index` reported by `RunnerOutput` matches the scheduler progress.
- CFG-parallel and non-CFG paths both work if the request-level pipeline supports them.
## Related Files
- Contract: [`vllm_omni/diffusion/models/interface.py`](gh-file:vllm_omni/diffusion/models/interface.py)
- State: [`vllm_omni/diffusion/worker/utils.py`](gh-file:vllm_omni/diffusion/worker/utils.py)
- Runner loop: [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py)
- Scheduler transport: [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py)
- Reference pipeline: [`vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py`](gh-file:vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py)
docs/design/index.md
+1 -3
View File
@@ -11,9 +11,7 @@ This section contains design documents and architecture specifications for vLLM-
- [Disaggregated Inference](feature/disaggregated_inference.md)
- [Ray-based Execution](feature/ray_based_execution.md)
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
- [Adding Step Execution Support for Diffusion Pipelines](feature/diffusion_step_execution.md)
- [Request-Level Batching for Diffusion](feature/diffusion_request_level_batching.md)
- [Continuous Batching for Step-Wise Diffusion](feature/diffusion_continuous_batching.md)
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
## Infrastructure Design Documents
docs/design/module/dit_module.md
+1 -2
View File
@@ -252,8 +252,7 @@ while True:
batch of compatible independent requests for one full pipeline forward when
the pipeline supports request-level batching; the step-wise path can also
admit or remove compatible requests between denoise steps. See
[Request-Level Batching](../feature/diffusion_request_level_batching.md) and
[Continuous Batching for Step-Wise Diffusion](../feature/diffusion_continuous_batching.md).
[Diffusion Continuous Batching](../feature/diffusion_continuous_batching.md).
---
docs/getting_started/quickstart.md
+2 -3
View File
@@ -87,9 +87,8 @@ if __name__ == "__main__":
!!! info
For diffusion request-level batching controls such as `max_num_seqs` and
`request_batch_max_wait_ms`, see
[Request-Level Batching](../user_guide/diffusion/request_batching.md).
For diffusion request batching, step execution, and streaming controls, see
[Diffusion Execution Modes](../user_guide/diffusion/execution_modes.md).
For more usages, please refer to [offline inference](../user_guide/examples/offline_inference/qwen2_5_omni.md)
docs/user_guide/diffusion/execution_modes.md
+252
View File
@@ -0,0 +1,252 @@
# Diffusion Execution Modes
vLLM-Omni supports complete-request execution and step-wise diffusion
execution. Both modes use the same asynchronous engine output stream; the
configuration controls how work is scheduled and whether intermediate outputs
are exposed.
## Choose a Mode
| Goal | CLI configuration |
|---|---|
| Serial request execution | `--max-num-seqs 1` |
| Fused request-level batching | `--max-num-seqs N` |
| Single-request step execution | `--step-execution --max-num-seqs 1` |
| Step-wise continuous batching | `--step-execution --max-num-seqs N` |
| Chunked diffusion output | `--diffusion-streaming-output` |
`N` must be greater than `1` to allow batching. The selected pipeline must
support request-level batching or batched-step execution as appropriate;
single-request step support alone is not sufficient for step-wise continuous
batching.
## Request Execution
Request execution is the default when `--step-execution` is omitted. The
pipeline performs a complete `forward()` for each scheduler wave.
### Serial Requests
Use the conservative serial path with any request-mode pipeline:
```bash
vllm serve MODEL --omni \
--port 8091 \
--max-num-seqs 1
```
### Request-Level Batching
Set `max_num_seqs` above one to combine compatible independent requests into a
single pipeline forward:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4
```
For bursty traffic, add a small admission window:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4 \
--request-batch-max-wait-ms 20
```
`--request-batch-max-wait-ms 0` is the default. A nonzero value can improve
batch formation but adds up to that much latency before a new scheduler wave.
It has no effect in step mode.
Each prompt remains a separate logical request. Do not submit a top-level list
as one packed prompt. Concurrent serving requests are batched internally when
their shapes, guidance settings, output counts, and LoRA settings are
compatible.
Only pipelines that declare request-batch support accept
`max_num_seqs>1` in request mode. Unsupported pipelines fail during engine
initialization; use `max_num_seqs=1` instead.
## Step Execution
Step execution lets the scheduler advance and abort requests between denoise
steps:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--step-execution \
--max-num-seqs 1
```
Set a larger capacity to allow compatible requests to share denoise waves:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--step-execution \
--max-num-seqs 8
```
Step continuous batching is experimental. Start with `max_num_seqs=1` when
validating a model or debugging correctness, then increase it for
multi-request throughput.
Step execution is capability-based, not a generic switch for every diffusion
model. Qwen-Image supports step-wise continuous batching. HunyuanImage3 also
supports it, but only when its resolved self-attention backend is `TORCH_SDPA`;
set `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` or configure
`diffusion_attention_config.default.backend=TORCH_SDPA` before using
`--max-num-seqs >1`. See the
[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md)
for its validated configuration. Helios supports single-request step execution only: use
`--step-execution --max-num-seqs 1` for Helios. Consult the selected pipeline's
documentation and source for the latest support status.
## Streaming Output
Use `--diffusion-streaming-output` for a pipeline that can produce intermediate
diffusion outputs:
```bash
vllm serve BestWishYsh/Helios-Distilled --omni \
--port 8000 \
--diffusion-streaming-output
```
Streaming output requires step execution. If
`--diffusion-streaming-output` is set without `--step-execution`, the engine
enables step execution automatically. Model initialization fails if the
pipeline does not implement step execution.
Chunk-capable pipelines emit intermediate and final outputs through the same
request stream. Final-only step pipelines emit only the final result even when
streaming output is enabled.
The normal non-streaming serving path uses the same internal stream but drains
it and returns only the final output. Users do not need to enable
`--diffusion-streaming-output` to benefit from the unified engine lifecycle.
## Send Requests
Execution-mode flags configure the server. They do not change the
OpenAI-compatible client request format. For example, after starting any of
the non-streaming configurations above:
```bash
curl -X POST http://localhost:8091/v1/images/generations \
-H "Content-Type: application/json" \
-d '{
"prompt": "a cat sitting on a windowsill",
"size": "1024x1024",
"num_inference_steps": 50,
"seed": 42
}' | jq -r '.data[0].b64_json' | base64 -d > cat.png
```
To create a request-level batch, send multiple independent requests
concurrently. Do not put several prompts into one `prompt` field:
```bash
for prompt in "a red fox" "a blue bird" "a green frog"; do
curl -s -X POST http://localhost:8091/v1/images/generations \
-H "Content-Type: application/json" \
-d "{\"prompt\":\"${prompt}\",\"size\":\"1024x1024\",\"seed\":42}" \
> /dev/null &
done
wait
```
The scheduler may batch these requests when their sampling parameters are
compatible. See the
[Image Generation API](../../serving/image_generation_api.md) for response
formats and additional client examples. Streaming models can use
model-specific streaming endpoints documented by their serving guide.
## Python API
The Python arguments mirror the CLI flags:
```python
from vllm_omni import Omni
omni = Omni(
model="Qwen/Qwen-Image",
max_num_seqs=4,
request_batch_max_wait_ms=20.0,
)
outputs = omni.generate(
[
"a cup of coffee on a table",
"a toy dinosaur on a sandy beach",
"a fox waking up in bed and yawning",
]
)
```
Each list item passed to `Omni.generate()` becomes an independent logical
request that the scheduler may batch with compatible items.
For step execution:
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
omni = Omni(
model="Qwen/Qwen-Image",
step_execution=True,
max_num_seqs=1,
)
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(num_inference_steps=50),
)
```
## Stage Configuration
Use the equivalent engine arguments in a deployment YAML:
```yaml
stage_args:
- stage_id: 0
stage_type: diffusion
engine_args:
step_execution: false
max_num_seqs: 4
request_batch_max_wait_ms: 20
```
For step execution, set `step_execution: true` and remove
`request_batch_max_wait_ms`.
## CLI Reference
| Flag | Default | Effect |
|---|---:|---|
| `--step-execution` | disabled | Select step-wise scheduling |
| `--max-num-seqs` | `1` for diffusion stages | Set request- or step-scheduler capacity |
| `--request-batch-max-wait-ms` | `0` | Wait for burst coalescing in request mode |
| `--diffusion-streaming-output` | disabled | Expose supported intermediate diffusion outputs and require step execution |
## Limitations and Troubleshooting
- Requests batch only when their compatibility-sensitive parameters match.
- Different LoRA adapters or scales run in separate batches.
- FIFO scheduling can cause an incompatible request to block later compatible
requests.
- All diffusion cache backends are unsupported in step mode. KV transfer is
supported for newly admitted step requests; some other request-mode extras
remain unsupported.
- If request-mode startup reports that the pipeline does not support batching,
use `--max-num-seqs 1`.
- If step-mode startup mentions `prepare_encode()`, `denoise_step()`,
`step_scheduler()`, or `post_decode()`, the pipeline does not implement the
required step contract.
For implementation details and model-author guidance, see
[Diffusion Continuous Batching](../../design/feature/diffusion_continuous_batching.md).
docs/user_guide/diffusion/request_batching.md
-119
View File
@@ -1,119 +0,0 @@
# Request-Level Batching
Request-level batching lets diffusion serving combine multiple compatible
logical requests into one pipeline `forward()` call. Each prompt remains a
separate request with its own `request_id`, sampling parameters, seed, output,
error, and abort state. The scheduler decides which requests can run together.
!!! warning "Prompt List Semantics"
Diffusion request-level batching does not support a top-level packed
list-prompt request. Submit multiple prompts as independent requests and let
the scheduler batch compatible in-flight requests. Multimodal payloads stay
inside a single prompt dict, for example
`{"prompt": "...", "multi_modal_data": {"image": image}}`.
## Enablement
Request-level execution is the default diffusion request mode when
`step_execution=False`. `max_num_seqs` controls the maximum number of compatible
requests in one scheduler wave:
- `max_num_seqs=1` is the serial request path.
- `max_num_seqs>1` allows fused request-level batching for pipelines that
declare request-batch support.
Increase `max_num_seqs` above `1` to allow the request scheduler to keep more
than one compatible request active:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4
```
For bursty online traffic, you can also set a small admission wait window. This
lets the engine wait briefly before the first `schedule()` of a new wave so
nearby compatible requests can arrive and share the same fused forward pass:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4 \
--request-batch-max-wait-ms 20
```
`--request-batch-max-wait-ms 0` is the default and disables admission waiting,
so there is no added wait latency.
For deploy YAMLs, configure the diffusion stage engine args:
```yaml
stage_args:
- stage_id: 0
stage_type: diffusion
engine_args:
max_num_seqs: 4
request_batch_max_wait_ms: 20
```
## Compatibility
Only pipelines that declare request-batch support can use the fused
multi-request `pipeline.forward(batch)` path. Pipelines that do not support this
contract can still use request-level execution with `max_num_seqs=1`, which is
the serial request path. Setting `max_num_seqs>1` for a pipeline without
request-batch support fails early during engine initialization.
The scheduler batches only compatible requests. Compatibility is based on
shape-sensitive and guidance-sensitive sampling fields, including resolution,
frame count, CFG settings, output count, and LoRA identity. Requests with
different LoRA adapters or scales are kept in separate batches so the worker
activates one adapter per batch.
Request-level batching applies when `step_execution=False`. For the separate
step-wise runtime, see [Step Execution](step_execution.md).
## Tuning
- `max_num_seqs` caps the number of active compatible requests in one scheduler
wave. `1` means serial request execution; values above `1` enable fused
request-level batching when the pipeline supports it.
- `request_batch_max_wait_ms` is an upper bound on extra admission wait before a
new wave starts. Keep it small for latency-sensitive serving; values such as
`10` to `50` ms are a practical starting range for bursty HTTP ingress.
- `0` disables admission waiting and preserves the lowest first-request latency.
- FIFO ordering is conservative: an incompatible request at the front of the
waiting queue can block later compatible requests from joining the current
batch.
## Python API
When constructing `Omni`, pass the same engine arguments:
```python
from vllm_omni.entrypoints.omni import Omni
omni = Omni(
model="Qwen/Qwen-Image",
max_num_seqs=4,
request_batch_max_wait_ms=20.0,
)
outputs = omni.generate(
[
"a cup of coffee on a table",
"a toy dinosaur on a sandy beach",
"a fox waking up in bed and yawning",
]
)
```
`Omni.generate([...])` submits each list item as its own logical diffusion
request. The runtime may batch those requests internally when their sampling
parameters are compatible.
## For Contributors
For implementation details and model-author guidance, see
[Request-Level Batching for Diffusion](../../design/feature/diffusion_request_level_batching.md).
docs/user_guide/diffusion/step_execution.md
-100
View File
@@ -1,100 +0,0 @@
# Step Execution
Step execution is an opt-in diffusion execution mode enabled with
`step_execution=True` when constructing `Omni`.
It is not a generic diffusion toggle for every pipeline. Only pipelines that
implement the stepwise contract support it today.
## Quick Start
### Python API
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
omni = Omni(
model="Qwen/Qwen-Image",
step_execution=True,
)
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(
num_inference_steps=50,
),
)
```
### Serving
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--step-execution \
--max-num-seqs 8
```
For serving, `--step-execution` enables the step-wise runtime. Continuous
batching only becomes relevant when `--max-num-seqs > 1`.
`max_num_seqs` controls the step scheduler capacity:
- `max_num_seqs=1` runs one request through the step-wise state machine.
- `max_num_seqs>1` allows compatible active requests to share step waves.
## Supported Pipelines
| Pipeline | Example models | Step execution |
|----------|----------------|----------------|
| `QwenImagePipeline` | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes |
| All other diffusion pipelines | `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, `QwenImageLayeredPipeline`, GLM-Image, Wan, Flux, etc. | No |
!!! warning "Experimental continuous batching"
When `--step-execution` is enabled and `max_num_seqs > 1` is configured,
the step-wise path can batch
compatible requests together. This is experimental. Requests with
incompatible sampling parameters are intentionally kept in separate batches,
and `max_num_seqs=1` remains the conservative single-request step path.
## Current Limitations
- Continuous batching under `step_execution` is experimental and only batches
compatible requests.
- `cache_backend` is not supported together with step execution.
- Unsupported pipelines fail early during model loading.
- Request-mode extras such as KV transfer are not wired into step mode yet.
- LoRA is supported in step mode, but each batch must use a single adapter:
requests with different `lora_request` or `lora_scale` are scheduled into
separate batches.
## When To Use It
Use step execution only when you specifically need the pipeline to run through
its stepwise request state machine. For normal diffusion inference, leave it
disabled unless your workflow depends on this mode.
For Qwen-Image online serving, the usual progression is:
- start with `--step-execution --max-num-seqs 1` if you only need the step-wise path
- increase `--max-num-seqs` after that if you want the experimental compatible-request batching behavior
If you are looking for general diffusion speedups, see
[Diffusion Features Overview](../diffusion_features.md).
## Troubleshooting
If model loading fails with a message mentioning `prepare_encode()`,
`denoise_step()`, `step_scheduler()`, and `post_decode()`, the selected
pipeline does not support step execution.
## For Model Authors
If you want to add step execution support to a new diffusion pipeline, see the
implementation guide:
[Diffusion Step Execution Design](../../design/feature/diffusion_step_execution.md).
If you also want that pipeline to participate in the experimental batched
step-wise path, see:
[Continuous Batching for Step-Wise Diffusion](../../design/feature/diffusion_continuous_batching.md).
docs/user_guide/diffusion_features.md
+6 -7
View File
@@ -80,12 +80,11 @@ denoise steps.
| Method | Description | Best For |
|--------|-------------|----------|
| **[Request-Level Batching](diffusion/request_batching.md)** | Scheduler batches compatible independent diffusion requests into one pipeline forward pass | Bursty online serving and multi-request throughput |
| **[Step Execution](diffusion/step_execution.md)** | Per-step denoise execution with mid-request abort support | Request cancellation between denoise steps, fine-grained execution control |
| **[Diffusion Execution Modes](diffusion/execution_modes.md)** | Configures serial requests, request batching, step execution, continuous batching, and streaming output | Matching latency, throughput, cancellation, and output-delivery requirements |
**Note:** Request-level batching is available for pipelines that declare the
request-batch forward contract. Step execution is currently supported by
QwenImagePipeline only. See [Supported Models](#supported-models) for details.
**Note:** Request-level batching and step execution are capability-based.
Consult the execution guide and the selected pipeline's documentation for
current support.
### Quantization Methods
@@ -206,7 +205,7 @@ The following tables show which models support each feature:
3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible.
4. CPU Offloading (Layerwise) supports single-card for now.
5. Using FP8-Quant as an example of qunatization methods.
6. Step Execution is not compatible with cache backends (TeaCache, Cache-DiT). LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).
6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).
## Multi-Thread Weight Loading
@@ -286,7 +285,7 @@ Measured on NVIDIA H800:
**Execution Modes:**
- **[Step Execution Guide](diffusion/step_execution.md)** - Per-step denoise execution with mid-request abort support
- **[Diffusion Execution Modes](diffusion/execution_modes.md)** - Configure request batching, step execution, continuous batching, and streaming output
**Startup Optimization:**
docs/user_guide/examples/offline_inference/text_to_image.md
+1 -1
View File
@@ -184,7 +184,7 @@ if __name__ == "__main__":
!!! info
For diffusion request-level batching controls such as `max_num_seqs`, see
[Request-Level Batching](../../diffusion/request_batching.md).
[Diffusion Execution Modes](../../diffusion/execution_modes.md).
### Negative Prompts
examples/offline_inference/text_to_image/README.md
+1 -1
View File
@@ -260,7 +260,7 @@ if __name__ == "__main__":
For diffusion pipelines, the input list is sliced into single-item requests
before feeding into the diffusion pipeline. For request-level batching
controls such as `max_num_seqs`, see
[Request-Level Batching](../../../docs/user_guide/diffusion/request_batching.md).
[Diffusion Execution Modes](../../../docs/user_guide/diffusion/execution_modes.md).
### Negative Prompts
mkdocs.yml
+6
View File
@@ -71,6 +71,12 @@ plugins:
- search
- autorefs
- awesome-nav
- redirects:
redirect_maps:
user_guide/diffusion/request_batching.md: user_guide/diffusion/execution_modes.md
user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md
design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md
design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md
- glightbox
- git-revision-date-localized:
# exclude files
pyproject.toml
+1
View File
@@ -108,6 +108,7 @@ docs = [
"mkdocstrings-python",
"mkdocs-gen-files",
"mkdocs-awesome-nav",
"mkdocs-redirects",
"mkdocs-glightbox",
"mkdocs-git-revision-date-localized-plugin",
"mkdocs-minify-plugin",
recipes/LTX/LTX-2.md
+1 -1
View File
@@ -332,6 +332,6 @@ bundled offline CLI do not currently expose `sigmas`.
- <https://huggingface.co/Lightricks/LTX-2.3>
- <https://huggingface.co/diffusers/LTX-2.3-Diffusers>
- [Online video generation](../../docs/user_guide/examples/online_serving/text_to_video.md)
- [Request batching](../../docs/user_guide/diffusion/request_batching.md)
- [Diffusion execution modes](../../docs/user_guide/diffusion/execution_modes.md)
- [T2V offline example](../../examples/offline_inference/text_to_video/text_to_video.md)
- [I2V offline example](../../examples/offline_inference/image_to_video/README.md)