diff --git a/docs/.nav.yml b/docs/.nav.yml index 07731935b..4ff9493c7 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -71,8 +71,7 @@ nav: - CPU Offloading: user_guide/diffusion/cpu_offload_diffusion.md - LoRA: user_guide/diffusion/lora.md - Custom Pipeline: features/custom_pipeline.md - - Request-Level Batching: user_guide/diffusion/request_batching.md - - Step Execution: user_guide/diffusion/step_execution.md + - Execution Modes: user_guide/diffusion/execution_modes.md - Quantization: - Overview: user_guide/quantization/overview.md - Online Quantization: user_guide/quantization/online.md @@ -120,8 +119,6 @@ nav: - design/feature/async_chunk.md - design/feature/omni_async_output_materialization.md - design/feature/vae_parallel.md - - design/feature/diffusion_step_execution.md - - design/feature/diffusion_request_level_batching.md - design/feature/diffusion_continuous_batching.md - Module Design: - design/module/ar_module.md diff --git a/docs/configuration/stage_configs.md b/docs/configuration/stage_configs.md index bdcce41cc..290c15b94 100644 --- a/docs/configuration/stage_configs.md +++ b/docs/configuration/stage_configs.md @@ -387,8 +387,8 @@ that support request-level batching with `step_execution` disabled. Use this together with `max_num_seqs > 1` for bursty serving traffic. `0` disables admission waiting and preserves the lowest first-request latency. -For diffusion request-level batching tuning, see -[Request-Level Batching](../user_guide/diffusion/request_batching.md). +For diffusion execution and batching tuning, see +[Diffusion Execution Modes](../user_guide/diffusion/execution_modes.md). Default: `0.0` diff --git a/docs/contributing/model/adding_diffusion_model.md b/docs/contributing/model/adding_diffusion_model.md index 4c01db413..dd7dfbd6b 100644 --- a/docs/contributing/model/adding_diffusion_model.md +++ b/docs/contributing/model/adding_diffusion_model.md @@ -755,7 +755,8 @@ omni = Omni(model="your-model", ulysses_degree=2, ring_degree=2) ### Step Execution -See detailed design guide: [How to add step execution support](../../design/feature/diffusion_step_execution.md) +See the detailed +[Diffusion Continuous Batching design guide](../../design/feature/diffusion_continuous_batching.md). Use this only when your pipeline can be split into stable request-scoped and step-scoped phases. The reference implementation is @@ -769,9 +770,8 @@ step-scoped phases. The reference implementation is Do not enable `step_execution=True` until those four methods are implemented and validated against the request-level path. -If you want the pipeline to work with the experimental batched step-wise path -(`max_num_seqs > 1`), also see: -[Continuous Batching for Step-Wise Diffusion](../../design/feature/diffusion_continuous_batching.md). +The same design guide covers the experimental batched step-wise path used when +`max_num_seqs > 1`. If you expose this in example scripts or recipes, keep it opt-in. Surface runtime features like `step_execution` as optional flags instead of silently diff --git a/docs/design/feature/diffusion_continuous_batching.md b/docs/design/feature/diffusion_continuous_batching.md index 482bd1122..2ec89b9d1 100644 --- a/docs/design/feature/diffusion_continuous_batching.md +++ b/docs/design/feature/diffusion_continuous_batching.md @@ -1,165 +1,232 @@ -# Continuous Batching for Step-Wise Diffusion +# Diffusion Continuous Batching -!!! warning "Experimental Feature" - This feature is experimental. It currently applies only to native - diffusion pipelines running with `step_execution=True`. - -This document describes the batching extension built on top of -[Diffusion Step Execution](diffusion_step_execution.md). The base -step-execution contract is unchanged. The batching work is mainly in the -scheduler and runner layers. - -## Why It Helps - -Step-wise execution breaks a long denoise loop into scheduler-visible units. -That gives the runtime a place to admit other compatible requests between -steps instead of waiting for an entire request to finish. - -This matters most in low-MFU or bursty serving scenarios: - -- one request's denoise step may not fully saturate the GPU -- several compatible requests can share the same denoise forward pass -- throughput and device utilization can improve without changing request-local - scheduler state - -This is **not** a guaranteed single-request latency win. The main benefit is -usually higher utilization and better throughput when the workload contains -multiple in-flight compatible requests. +This document describes the unified diffusion execution architecture, including +request execution, request-level batching, step execution, continuous batching, +and output delivery. For user-facing configuration and CLI examples, see +[Diffusion Execution Modes](../../user_guide/diffusion/execution_modes.md). ## Overview -With continuous batching enabled: +`DiffusionEngine` selects an execution mode from `step_execution` and uses +`max_num_seqs` as that mode's scheduler capacity: -- the scheduler may keep multiple compatible requests active at the same time -- the runner packs request-local step state into one `InputBatch` -- `denoise_step()` runs on that batch -- `step_scheduler()` and `post_decode()` still run per request +| Configuration | Engine mode | Scheduler | Execution | +|---|---|---|---| +| `step_execution=False`, `max_num_seqs=1` | `REQUEST_BATCH` | `RequestScheduler` | One complete request-level `forward()` | +| `step_execution=False`, `max_num_seqs>1` | `REQUEST_BATCH` | `RequestScheduler` | One fused `forward()` over compatible requests | +| `step_execution=True`, `max_num_seqs=1` | `STEP_BATCH` | `StepScheduler` | One request advanced one denoise step per scheduler tick | +| `step_execution=True`, `max_num_seqs>1` | `STEP_BATCH` | `StepScheduler` | Compatible requests advanced together in step waves | -The current implementation is conservative: +Serial request execution is handled inside `REQUEST_BATCH`; it is not a +separate engine mode. Similarly, continuous batching is the multi-request +configuration of `STEP_BATCH`, not an independent execution mode. -- only compatible requests are batched together -- per-request progress and completion remain independent +The engine performs the following initialization: -Here, "continuous batching" means the step-wise path enabled by -`step_execution=True`. Request-mode `DiffusionRequestBatch` is static -request-level batching for one full pipeline `forward()` call; it does not -admit or remove requests between denoise steps. +1. Resolve model-specific pre- and post-processing hooks. +2. Select `REQUEST_BATCH` or `STEP_BATCH`. +3. Construct the configured `DiffusionExecutor`. +4. Construct `RequestScheduler` or `StepScheduler`, unless a scheduler was + explicitly injected. +5. Bind `execute_batch` or `execute_step`. -## Enablement +`DiffusionEngine.make_engine()` owns startup warmup after engine construction. +This keeps construction, backend selection, and warmup failure cleanup in one +lifecycle. -Use `--step-execution` as the feature gate, then increase `--max-num-seqs` -above `1` if you want batching: +## Unified Output Stream -```bash -vllm serve Qwen/Qwen-Image --omni \ - --port 8091 \ - --step-execution \ - --max-num-seqs 8 +All asynchronous diffusion requests use the same per-request output stream, +regardless of whether the caller exposes streaming results to the user. + +```python +async for output in engine.step_streaming(request): + ... ``` -`--max-num-seqs 1` keeps the step-wise path without enabling batching. +When a request is admitted, the engine creates an +`asyncio.Queue[DiffusionOutput]` keyed by its request ID. The busy loop sends +both intermediate chunks and final results through that queue: -For a reproducible replay flow using the bundled serving benchmark, see the -Qwen-Image replay commands in -[`benchmarks/diffusion/README.md`](gh-file:benchmarks/diffusion/README.md) -and -[`benchmarks/diffusion/performance_dashboard/qwen_image_serving_performance.md`](gh-file:benchmarks/diffusion/performance_dashboard/qwen_image_serving_performance.md). +```text +request + -> scheduler.add_request() + -> scheduler.schedule() + -> execute_batch() or execute_step() + -> scheduler.update_from_output() + -> per-request output queue + -> step_streaming() +``` -## Scheduler +Streaming callers forward every yielded output. Non-streaming callers consume +the same generator to completion and return only its final output. This avoids +maintaining separate future-based and queue-based result paths. -The scheduler derives its batch capacity from `max_num_seqs` through -`max_num_running_reqs`. +`DiffusionEngine.step()` and +`async_add_req_and_wait_for_response()` remain deprecated compatibility +wrappers. New integrations should consume `step_streaming()` or +`async_add_req_and_stream_response()`. -Batch admission is gated by -[`StepBatchSamplingParamsKey`](gh-file:vllm_omni/diffusion/sched/interface.py), -which is built from shape-sensitive and CFG-sensitive sampling fields. This is -the core correctness rule for batching: requests are only co-batched when they -share the same denoise tensor contract. +### Completion and Cancellation -There are three important details: +Output delivery and scheduler lifecycle are related but separate: -- `num_inference_steps` is not part of the key, so requests with different - total step counts can still share a batch -- requests also do not need to be at the same current denoise progress; active - requests can continue batching even when their current step indices diverge -- admission is still FIFO, so an incompatible request at the head of the - waiting queue blocks later compatible requests +- A terminal scheduler result always finalizes request state. +- A disconnected or cancelled consumer removes only its delivery queue. +- The scheduler may still finish and clean up the request after its queue has + been removed. +- Engine shutdown sends an error output to all remaining streams. -Today that compatibility rule is still shape-sensitive. `height`, `width`, -`num_frames`, and CFG-related fields remain part of the key, so different -resolutions or incompatible guidance settings do **not** co-batch yet. The -key also covers LoRA identity (`lora_int_id`, `lora_scale`), so requests -targeting different adapters or scales run in separate batches and the -worker can activate exactly one adapter per step. +This separation prevents cancelled consumers from leaving active request IDs in +the scheduler. -The scheduler batching unit is one logical `OmniDiffusionRequest`. In the -step-wise path, runtime tensor batching is represented as `StepInputBatch`. For -request-mode prompt semantics, see -[Request-Level Batching](../../user_guide/diffusion/request_batching.md). +## Request-Batch Execution -## Runner +Request-batch execution runs the complete diffusion pipeline for each scheduler +wave. Each `OmniDiffusionRequest` remains an independent logical request with +its own prompt, sampling parameters, seed, request ID, output, error, and abort +state. -The runner keeps persistent per-request execution state in -[`StepRequestState`](gh-file:vllm_omni/diffusion/worker/utils.py), -while the scheduler owns a separate lightweight -[`SchedulerRequestState`](gh-file:vllm_omni/diffusion/sched/interface.py) -for queueing and lifecycle tracking. +With `max_num_seqs=1`, `execute_batch()` handles the serial request path. With a +larger value, a pipeline can opt into fused execution by declaring: -For each step, the runner builds an -[`InputBatch`](gh-file:vllm_omni/diffusion/worker/input_batch.py) from the -active request states: +```python +supports_request_batch = True +``` -- prompt embeddings and masks are normalized and padded -- dynamic tensors such as `latents` and `timesteps` are gathered each step -- buffers are reused when batch composition stays the same +Its `forward()` method must accept `DiffusionRequestBatch` and return one +`DiffusionOutput` per request. The runner validates the result count and maps +outputs back to their original request IDs with `BatchRunnerOutput`. -The step-wise batched path is: +Current in-tree request-batch implementations include the Flux, LTX-2, SD3, and +Qwen-Image pipelines. Treat the pipeline capability flag in the source as +authoritative because support continues to expand. -1. Run `prepare_encode()` for newly admitted requests. +### Request-Batch Data Flow + +- `DiffusionSchedulerOutput` carries scheduled request IDs and payloads. +- `DiffusionRequestBatch` provides the pipeline-facing static batch. +- `BatchRunnerOutput` carries per-request results back to the engine. + +Request-local setup, such as seed handling and output/error mapping, remains +per request. Cache refresh, homogeneous LoRA activation, and +`pipeline.forward(req_batch)` can be shared by the batch. + +### Compatibility and Admission + +`RequestBatchSamplingParamsKey` contains fields that affect the tensor contract, +including shape, guidance, output count, and LoRA identity. Only compatible +requests can share a fused forward call. + +Admission is FIFO and conservative. An incompatible request at the front of the +waiting queue can prevent later compatible requests from joining the current +wave. + +`request_batch_max_wait_ms` optionally delays the first schedule operation of a +new wave so bursty arrivals can accumulate. It applies only when: + +- the pipeline supports request batching; +- step execution is disabled; +- no requests are currently running; and +- the configured wait is greater than zero. + +The wait ends when the queue reaches capacity, stabilizes briefly, reaches its +deadline, or the engine stops. + +## Step-Batch Execution + +Step execution exposes denoising progress to the scheduler. A supporting +pipeline implements four stateful operations: + +| Operation | Responsibility | +|---|---| +| `prepare_encode(state)` | Validate input, encode prompts, initialize latents and timesteps, and create request-local scheduler state | +| `denoise_step(input_batch, *, states=...)` | Run one denoise forward for the scheduler-provided request states | +| `step_scheduler(state, noise_pred)` | Update latents and advance request progress | +| `post_decode(state)` | Decode and postprocess a completed request or output boundary | + +Persistent request state lives in `StepRequestState`. Pipeline-specific fields +that do not belong in the shared contract should be stored in `state.extra`. +Queueing and lifecycle metadata remain in the scheduler's request state. + +Current native pipelines that explicitly enable step execution include +Qwen-Image, HunyuanImage3, and Helios. Step execution alone does not imply +continuous-batching support: Qwen-Image accepts batched step states. +HunyuanImage3 accepts batched step states only when its resolved self-attention +backend is `TORCH_SDPA`; otherwise it rejects groups larger than one request. +Configure `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` or +`diffusion_attention_config.default.backend=TORCH_SDPA` when +`max_num_seqs>1`. See the +[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md) +for its validated configuration. Helios supports only a single active step +request and must use `max_num_seqs=1`. + +### Continuous Batching + +When `max_num_seqs>1`, `StepScheduler` can keep multiple compatible requests +active. The runner gathers their state into `InputBatch`, performs one batched +denoise forward, then applies scheduler updates per request: + +1. Receive transferred KV payloads, then run `prepare_encode()` for newly + admitted requests. 2. Build or refresh `InputBatch`. -3. Run one batched `denoise_step(input_batch)`. -4. Slice the batched `noise_pred` back per request. -5. Run per-request `step_scheduler()`. -6. Run `post_decode()` only for requests that finished denoising. -7. Scatter updated latents back into persistent request state with - [`scatter_latents()`](gh-file:vllm_omni/diffusion/worker/input_batch.py). +3. Run one batched `denoise_step(input_batch, states=states)`. +4. Slice noise predictions back to each request. +5. Run each request's `step_scheduler()`. +6. Run `post_decode()` at a chunk boundary or request completion. +7. Scatter updated latents back into persistent request state. -This keeps the shared work limited to the denoise forward pass while preserving -request-local scheduler state and outputs. +`StepBatchSamplingParamsKey` protects the batched tensor contract. Requests can +have different total step counts and current step indices, but shape-sensitive, +CFG-sensitive, and LoRA fields must be compatible. FIFO head-of-line blocking +also applies here. -## Engine +Chunk-capable pipelines may emit intermediate results. Final-only step +pipelines emit when the request finishes even if `streaming_output=True`. -[`DiffusionEngine`](gh-file:vllm_omni/diffusion/diffusion_engine.py) provides -the background loop and async add-request path needed for multiple requests to -accumulate in the scheduler. +## Model Author Guidelines -When `step_execution=True`, the engine routes work through the step-wise -executor path. The continuous batching behavior is defined by scheduler-side -compatibility gating and runner-side `StepInputBatch` packing. +When converting a request-level pipeline: -## Current Limitations +- Reuse the same helpers as `forward()` to avoid behavior drift. +- Copy request-scoped scheduler state into `state.scheduler`. +- Advance `state.step_index` only in `step_scheduler()`. +- Keep model forward work in `denoise_step()`. +- Keep latent mutation in `step_scheduler()`. +- Keep final decoding equivalent to the tail of `forward()`. +- Store masks, condition latents, and other request-local tensors in the + request state rather than on the shared pipeline. -- Experimental feature; use `max_num_seqs=1` for the conservative single-request - step path. -- Only native pipelines that already support `step_execution=True`. -- Only homogeneous batches keyed by `StepBatchSamplingParamsKey` are supported. -- `cache_backend`, KV transfer, and other request-mode extras are not wired - into the batched step-wise path yet. -- Future work can relax the current same-shape restriction with richer - heterogeneous batching policies such as bucketing or padded execution for - different resolutions. +Before setting `supports_step_execution = True`, validate output parity for the +same seed and sampling parameters, scheduler isolation across concurrent +requests, abort cleanup, progress reporting, and all supported CFG paths. -## Related Files +Before setting `supports_request_batch = True`, validate the batch input +contract, one-output-per-request result shape, request identity and error +mapping, seeded concurrency, LoRA compatibility, and tensor IPC. -- Scheduler base: - [`vllm_omni/diffusion/sched/base_scheduler.py`](gh-file:vllm_omni/diffusion/sched/base_scheduler.py) -- Scheduler interface: - [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py) -- Step scheduler: - [`vllm_omni/diffusion/sched/step_scheduler.py`](gh-file:vllm_omni/diffusion/sched/step_scheduler.py) -- Runner: - [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py) -- Input batch: - [`vllm_omni/diffusion/worker/input_batch.py`](gh-file:vllm_omni/diffusion/worker/input_batch.py) -- Tests: - [`tests/diffusion/test_diffusion_scheduler.py`](gh-file:tests/diffusion/test_diffusion_scheduler.py) +## Limitations + +- Request batching requires an explicit pipeline capability when + `max_num_seqs>1`. +- Step execution requires the four-method stateful pipeline contract. +- Both batching paths currently require homogeneous compatibility keys. +- FIFO scheduling can reduce batching opportunities. +- `request_batch_max_wait_ms` trades first-request latency for burst + coalescing. +- All diffusion cache backends are currently unsupported in step mode. +- KV transfer is supported for newly admitted step requests; some other + request-mode extras remain unsupported. +- Step continuous batching remains experimental; use `max_num_seqs=1` for the + conservative step path. + +## Related Implementation + +- Engine: [`vllm_omni/diffusion/diffusion_engine.py`](gh-file:vllm_omni/diffusion/diffusion_engine.py) +- Model contract: [`vllm_omni/diffusion/models/interface.py`](gh-file:vllm_omni/diffusion/models/interface.py) +- Scheduler interface: [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py) +- Request scheduler: [`vllm_omni/diffusion/sched/request_scheduler.py`](gh-file:vllm_omni/diffusion/sched/request_scheduler.py) +- Step scheduler: [`vllm_omni/diffusion/sched/step_scheduler.py`](gh-file:vllm_omni/diffusion/sched/step_scheduler.py) +- Runner: [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py) +- Request batch: [`vllm_omni/diffusion/worker/request_batch.py`](gh-file:vllm_omni/diffusion/worker/request_batch.py) +- Step input batch: [`vllm_omni/diffusion/worker/input_batch.py`](gh-file:vllm_omni/diffusion/worker/input_batch.py) diff --git a/docs/design/feature/diffusion_request_level_batching.md b/docs/design/feature/diffusion_request_level_batching.md deleted file mode 100644 index 8c1cdbd33..000000000 --- a/docs/design/feature/diffusion_request_level_batching.md +++ /dev/null @@ -1,175 +0,0 @@ -# Request-Level Batching for Diffusion - -This document describes the request-mode batching path for diffusion pipelines. -For end-user enablement and tuning, see -[Request-Level Batching](../../user_guide/diffusion/request_batching.md). - -This is separate from -[Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md). -Request-level batching runs one full pipeline `forward()` over a static batch of -compatible requests. Step-wise continuous batching admits work between denoise -steps when `step_execution=True`. - -## Why It Helps - -The request-level design avoids coupling several logical requests to one request -object. This keeps request identity, abort/error handling, and per-request -metadata unambiguous while still allowing one fused pipeline forward pass for -bursty or concurrent traffic. - -## Overview - -With request-level batching enabled: - -- each `OmniDiffusionRequest` contains one `prompt` and one `request_id` -- the scheduler groups compatible waiting requests into one scheduler wave -- `DiffusionRequestBatch` wraps the scheduled requests for pipeline `forward()` -- batch-capable pipelines return `list[DiffusionOutput]`, one output per - request -- `BatchRunnerOutput` maps each result back to its original `request_id` - -Pipelines opt in with `supports_request_batch = True` and a `forward()` method -that accepts `DiffusionRequestBatch` and returns `list[DiffusionOutput]`. -Pipelines that do not opt in keep the existing per-request execution path. - -## Enablement - -Request-level batching is the request-mode path, so `step_execution` must remain -disabled. Increase `max_num_seqs` above `1` to let the scheduler keep multiple -compatible requests active: - -```bash -vllm serve Qwen/Qwen-Image --omni \ - --port 8091 \ - --max-num-seqs 4 -``` - -For bursty online ingress, `request_batch_max_wait_ms` can add a bounded -admission wait before the first `schedule()` of a scheduler wave: - -```bash -vllm serve Qwen/Qwen-Image --omni \ - --port 8091 \ - --max-num-seqs 4 \ - --request-batch-max-wait-ms 20 -``` - -`request_batch_max_wait_ms=0` disables this wait and is the default. - -## Request Contract - -`OmniDiffusionRequest` represents one logical request. It owns one prompt, -sampling parameters, request id, and request-local metadata. Runtime batches are -formed by the scheduler and represented separately from the request payload. - -Runtime batching is represented by: - -- [`DiffusionSchedulerOutput`](gh-file:vllm_omni/diffusion/sched/interface.py) - for scheduled request ids and request payloads -- [`DiffusionRequestBatch`](gh-file:vllm_omni/diffusion/worker/request_batch.py) - for the pipeline-facing request batch -- [`BatchRunnerOutput`](gh-file:vllm_omni/diffusion/worker/utils.py) for - per-request results - -`DiffusionRequestBatch` intentionally exposes compatibility properties such as -`prompts`, `sampling_params`, `request_id`, and `kv_sender_info` so migrated -pipelines can stay close to upstream code while using a batch-aware contract. - -## Scheduler - -The scheduler derives its capacity from `max_num_seqs` through -`max_num_running_reqs`. It exposes waiting/running queue counters so the engine -can decide whether admission wait is useful before scheduling a new wave. - -Batch compatibility is controlled by -[`RequestBatchSamplingParamsKey`](gh-file:vllm_omni/diffusion/sched/interface.py). The key -contains shape-sensitive and guidance-sensitive fields, including output count -and LoRA identity. Requests with incompatible shapes, CFG settings, output -counts, LoRA adapters, or LoRA scales are kept in separate batches. - -Admission is conservative: - -- the scheduler only batches compatible requests -- FIFO ordering is preserved -- an incompatible request at the head of the waiting queue blocks later - compatible requests - -## Engine - -[`DiffusionEngine`](gh-file:vllm_omni/diffusion/diffusion_engine.py) resolves -request-batch capability during initialization from the configured pipeline -class, including custom pipeline classes. - -The capability check uses the pipeline class attribute -`supports_request_batch = True`. Pipelines that set this attribute must implement -a request-batch-compatible `forward()` contract and return one -`DiffusionOutput` per request; the runner validates that return shape at runtime. - -When the selected pipeline is batch-capable and `step_execution=False`, request -mode routes scheduler waves through the batch executor path. Otherwise it keeps -the per-request executor path. - -The optional admission wait runs only when: - -- request batching is supported -- `step_execution=False` -- `request_batch_max_wait_ms > 0` -- no requests are currently running - -The wait exits early when the waiting queue reaches capacity, when the queue is -stable for a short window, when the deadline expires, or when the engine stops. - -## Executor And Runner - -The executor exposes two request-mode entries: - -- `execute_request`: one worker call per scheduled request -- `execute_batch`: one worker call for the whole `DiffusionSchedulerOutput` - -On the batch path, the worker builds a `DiffusionRequestBatch` and runs the -pipeline once. Request-local setup remains per request: - -- KV transfer metadata -- random generator and seed handling -- request output/error/abort mapping - -Shared batch setup happens once per batch when possible: - -- cache refresh -- LoRA activation for the homogeneous adapter key -- pipeline `forward(req_batch)` - -Large tensor IPC still uses the shared-memory packing path. The packer traverses -both normal `RunnerOutput.result` wrappers and nested batch results so batched -outputs do not fall back to pickle IPC for tensor payloads. - -## Current Limitations - -- Only pipelines that declare the request-batch contract use fused batch - execution. -- Batches are homogeneous under `RequestBatchSamplingParamsKey`; heterogeneous resolution or - incompatible guidance settings do not co-batch yet. -- FIFO scheduling can reduce batching opportunities when an incompatible - request is at the front of the queue. -- `request_batch_max_wait_ms` improves burst coalescing but can add latency to - the first request in a scheduler wave. Keep it small for latency-sensitive - serving. -- Step-wise continuous batching is documented separately and only applies when - `step_execution=True`. - -## Related Files - -- Request object and request batch: - [`vllm_omni/diffusion/request.py`](gh-file:vllm_omni/diffusion/request.py) -- Scheduler interface: - [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py) -- Scheduler base: - [`vllm_omni/diffusion/sched/base_scheduler.py`](gh-file:vllm_omni/diffusion/sched/base_scheduler.py) -- Engine: - [`vllm_omni/diffusion/diffusion_engine.py`](gh-file:vllm_omni/diffusion/diffusion_engine.py) -- Worker runner: - [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py) -- Executor interface: - [`vllm_omni/diffusion/executor/abstract.py`](gh-file:vllm_omni/diffusion/executor/abstract.py) -- Tests: - [`tests/diffusion/test_diffusion_engine.py`](gh-file:tests/diffusion/test_diffusion_engine.py) diff --git a/docs/design/feature/diffusion_step_execution.md b/docs/design/feature/diffusion_step_execution.md deleted file mode 100644 index 88fddb215..000000000 --- a/docs/design/feature/diffusion_step_execution.md +++ /dev/null @@ -1,131 +0,0 @@ -# Adding Step Execution Support for Diffusion Pipelines - -This guide documents vLLM-Omni's stepwise diffusion contract for model authors -and contributors implementing `step_execution=True` support for a diffusion -pipeline. - -For end-user enablement, supported models, and current limitations, see -[Step Execution](../../user_guide/diffusion/step_execution.md). - -This document describes the base step-execution contract only. For the -experimental batching policy layered on top of the step-wise path, see -[Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md). - -## Current Support Scope - -`step_execution` is **not** a generic diffusion toggle. It only works for -pipelines that implement the segmented stateful contract in -[`vllm_omni/diffusion/models/interface.py`](gh-file:vllm_omni/diffusion/models/interface.py). - -This page is intentionally author-facing. Treat runtime enablement -(`step_execution=True` in Python or `--step-execution` in serving) as an -opt-in user knob layered on top of the implementation contract below. - -Current in-tree support: - -| Pipeline | Example models | Step execution | -|----------|----------------|----------------| -| `QwenImagePipeline` | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes | -| All other diffusion pipelines | `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, `QwenImageLayeredPipeline`, GLM-Image, Wan, Flux, etc. | No | - -Current engine/runtime limitations: - -- Continuous batching with `max_num_seqs > 1` is experimental and documented in - [Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md). - Keep `max_num_seqs=1` for the conservative single-request step path. -- `cache_backend` is not supported in step mode. -- Request-mode extras such as KV transfer are not wired into step mode yet. -- Unsupported pipelines now fail early during model loading instead of failing on the first request. - -## Execution Contract - -Step mode is driven by four pipeline methods plus the shared mutable request -state object: - -- `prepare_encode(state)`: one-time request preparation. -- `denoise_step(state)`: compute the noise prediction for the current step. -- `step_scheduler(state, noise_pred)`: mutate latents and advance step state. -- `post_decode(state)`: decode the final output after denoising is complete. - -The state lives in -[`vllm_omni/diffusion/worker/utils.py`](gh-file:vllm_omni/diffusion/worker/utils.py) -as `StepRequestState`. Store request-scoped tensors there, or use -`state.extra` for model-specific fields that do not justify extending the core -dataclass. - -The worker-side step loop lives in -[`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py): - -1. `prepare_encode()` runs once for a new request. -2. `denoise_step()` runs every scheduler tick. -3. `step_scheduler()` mutates `state.latents` and advances `state.step_index`. -4. `post_decode()` runs exactly once after `state.denoise_completed` becomes true. - -## Recommended Split - -When converting an existing request-level `forward()` pipeline, keep the split -strict and mechanical: - -| Request-level phase | Stepwise method | What belongs there | -|---------------------|-----------------|--------------------| -| Input validation, prompt encoding, latent init, timestep prep, per-request scheduler creation | `prepare_encode()` | Anything that should happen once per request | -| Transformer forward / noise prediction | `denoise_step()` | Pure denoise computation for the current timestep | -| `scheduler.step(...)` and `step_index += 1` | `step_scheduler()` | Only latent/state mutation for one step | -| VAE decode / postprocess | `post_decode()` | Final decode only | - -Keep the stepwise path reusing the same helpers as the request-level path -whenever possible. Reimplementing the denoise loop from scratch is the easiest -way to introduce behavioral drift. - -## Qwen-Image Reference - -[`pipeline_qwen_image.py`](gh-file:vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py) -is the reference implementation and is split correctly for the current -contract: - -- `prepare_encode()` reuses `_prepare_generation_context()` so prompt encoding, - latent init, timestep creation, CFG setup, and shape bookkeeping stay aligned - with `forward()`. -- `prepare_encode()` deep-copies `self.scheduler` **after** - `prepare_timesteps()` so request-specific scheduler state is isolated. -- `denoise_step()` reuses `_build_denoise_kwargs()` plus - `predict_noise_maybe_with_cfg()`, so sequential CFG, CFG-parallel, and - non-CFG behavior stay identical to the request-level path. -- `step_scheduler()` only calls - `scheduler_step_maybe_with_cfg(..., per_request_scheduler=state.scheduler)` - and increments `state.step_index`. -- `post_decode()` reuses `_decode_latents()`, so the final image decode matches - the normal `forward()` path. - -That decomposition is the target pattern for future models. - -## Rules For New Pipelines - -- Do not keep request-scoped scheduler state on `self.scheduler`. Copy it into - `state.scheduler` during `prepare_encode()`. -- Do not mutate `state.step_index` inside `denoise_step()`. Only - `step_scheduler()` should advance the step. -- Do not decode partial outputs in `denoise_step()` or `step_scheduler()`. -- If the request-level pipeline has condition latents, masks, or edit-specific - tensors, store them in `state` or `state.extra`, not in global pipeline - attributes. -- Preserve CFG behavior by sharing the same helper path used by `forward()`. -- Keep `post_decode()` equivalent to the tail of `forward()`. - -## Validation Checklist - -Before marking a pipeline as `supports_step_execution = True`, verify: - -- Stepwise output matches request-level output for the same seed and sampling params. -- Per-request scheduler state is isolated across concurrent requests. -- Abort during denoise does not leak cached state. -- `step_index` reported by `RunnerOutput` matches the scheduler progress. -- CFG-parallel and non-CFG paths both work if the request-level pipeline supports them. - -## Related Files - -- Contract: [`vllm_omni/diffusion/models/interface.py`](gh-file:vllm_omni/diffusion/models/interface.py) -- State: [`vllm_omni/diffusion/worker/utils.py`](gh-file:vllm_omni/diffusion/worker/utils.py) -- Runner loop: [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py) -- Scheduler transport: [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py) -- Reference pipeline: [`vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py`](gh-file:vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py) diff --git a/docs/design/index.md b/docs/design/index.md index 593191616..0da17cce4 100644 --- a/docs/design/index.md +++ b/docs/design/index.md @@ -11,9 +11,7 @@ This section contains design documents and architecture specifications for vLLM- - [Disaggregated Inference](feature/disaggregated_inference.md) - [Ray-based Execution](feature/ray_based_execution.md) - [Async Omni Output Materialization](feature/omni_async_output_materialization.md) -- [Adding Step Execution Support for Diffusion Pipelines](feature/diffusion_step_execution.md) -- [Request-Level Batching for Diffusion](feature/diffusion_request_level_batching.md) -- [Continuous Batching for Step-Wise Diffusion](feature/diffusion_continuous_batching.md) +- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md) ## Infrastructure Design Documents diff --git a/docs/design/module/dit_module.md b/docs/design/module/dit_module.md index aabdb78b0..f6883f12c 100644 --- a/docs/design/module/dit_module.md +++ b/docs/design/module/dit_module.md @@ -252,8 +252,7 @@ while True: batch of compatible independent requests for one full pipeline forward when the pipeline supports request-level batching; the step-wise path can also admit or remove compatible requests between denoise steps. See - [Request-Level Batching](../feature/diffusion_request_level_batching.md) and - [Continuous Batching for Step-Wise Diffusion](../feature/diffusion_continuous_batching.md). + [Diffusion Continuous Batching](../feature/diffusion_continuous_batching.md). --- diff --git a/docs/getting_started/quickstart.md b/docs/getting_started/quickstart.md index 15ceb395e..f9673eac7 100644 --- a/docs/getting_started/quickstart.md +++ b/docs/getting_started/quickstart.md @@ -87,9 +87,8 @@ if __name__ == "__main__": !!! info - For diffusion request-level batching controls such as `max_num_seqs` and - `request_batch_max_wait_ms`, see - [Request-Level Batching](../user_guide/diffusion/request_batching.md). + For diffusion request batching, step execution, and streaming controls, see + [Diffusion Execution Modes](../user_guide/diffusion/execution_modes.md). For more usages, please refer to [offline inference](../user_guide/examples/offline_inference/qwen2_5_omni.md) diff --git a/docs/user_guide/diffusion/execution_modes.md b/docs/user_guide/diffusion/execution_modes.md new file mode 100644 index 000000000..9611a8210 --- /dev/null +++ b/docs/user_guide/diffusion/execution_modes.md @@ -0,0 +1,252 @@ +# Diffusion Execution Modes + +vLLM-Omni supports complete-request execution and step-wise diffusion +execution. Both modes use the same asynchronous engine output stream; the +configuration controls how work is scheduled and whether intermediate outputs +are exposed. + +## Choose a Mode + +| Goal | CLI configuration | +|---|---| +| Serial request execution | `--max-num-seqs 1` | +| Fused request-level batching | `--max-num-seqs N` | +| Single-request step execution | `--step-execution --max-num-seqs 1` | +| Step-wise continuous batching | `--step-execution --max-num-seqs N` | +| Chunked diffusion output | `--diffusion-streaming-output` | + +`N` must be greater than `1` to allow batching. The selected pipeline must +support request-level batching or batched-step execution as appropriate; +single-request step support alone is not sufficient for step-wise continuous +batching. + +## Request Execution + +Request execution is the default when `--step-execution` is omitted. The +pipeline performs a complete `forward()` for each scheduler wave. + +### Serial Requests + +Use the conservative serial path with any request-mode pipeline: + +```bash +vllm serve MODEL --omni \ + --port 8091 \ + --max-num-seqs 1 +``` + +### Request-Level Batching + +Set `max_num_seqs` above one to combine compatible independent requests into a +single pipeline forward: + +```bash +vllm serve Qwen/Qwen-Image --omni \ + --port 8091 \ + --max-num-seqs 4 +``` + +For bursty traffic, add a small admission window: + +```bash +vllm serve Qwen/Qwen-Image --omni \ + --port 8091 \ + --max-num-seqs 4 \ + --request-batch-max-wait-ms 20 +``` + +`--request-batch-max-wait-ms 0` is the default. A nonzero value can improve +batch formation but adds up to that much latency before a new scheduler wave. +It has no effect in step mode. + +Each prompt remains a separate logical request. Do not submit a top-level list +as one packed prompt. Concurrent serving requests are batched internally when +their shapes, guidance settings, output counts, and LoRA settings are +compatible. + +Only pipelines that declare request-batch support accept +`max_num_seqs>1` in request mode. Unsupported pipelines fail during engine +initialization; use `max_num_seqs=1` instead. + +## Step Execution + +Step execution lets the scheduler advance and abort requests between denoise +steps: + +```bash +vllm serve Qwen/Qwen-Image --omni \ + --port 8091 \ + --step-execution \ + --max-num-seqs 1 +``` + +Set a larger capacity to allow compatible requests to share denoise waves: + +```bash +vllm serve Qwen/Qwen-Image --omni \ + --port 8091 \ + --step-execution \ + --max-num-seqs 8 +``` + +Step continuous batching is experimental. Start with `max_num_seqs=1` when +validating a model or debugging correctness, then increase it for +multi-request throughput. + +Step execution is capability-based, not a generic switch for every diffusion +model. Qwen-Image supports step-wise continuous batching. HunyuanImage3 also +supports it, but only when its resolved self-attention backend is `TORCH_SDPA`; +set `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` or configure +`diffusion_attention_config.default.backend=TORCH_SDPA` before using +`--max-num-seqs >1`. See the +[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md) +for its validated configuration. Helios supports single-request step execution only: use +`--step-execution --max-num-seqs 1` for Helios. Consult the selected pipeline's +documentation and source for the latest support status. + +## Streaming Output + +Use `--diffusion-streaming-output` for a pipeline that can produce intermediate +diffusion outputs: + +```bash +vllm serve BestWishYsh/Helios-Distilled --omni \ + --port 8000 \ + --diffusion-streaming-output +``` + +Streaming output requires step execution. If +`--diffusion-streaming-output` is set without `--step-execution`, the engine +enables step execution automatically. Model initialization fails if the +pipeline does not implement step execution. + +Chunk-capable pipelines emit intermediate and final outputs through the same +request stream. Final-only step pipelines emit only the final result even when +streaming output is enabled. + +The normal non-streaming serving path uses the same internal stream but drains +it and returns only the final output. Users do not need to enable +`--diffusion-streaming-output` to benefit from the unified engine lifecycle. + +## Send Requests + +Execution-mode flags configure the server. They do not change the +OpenAI-compatible client request format. For example, after starting any of +the non-streaming configurations above: + +```bash +curl -X POST http://localhost:8091/v1/images/generations \ + -H "Content-Type: application/json" \ + -d '{ + "prompt": "a cat sitting on a windowsill", + "size": "1024x1024", + "num_inference_steps": 50, + "seed": 42 + }' | jq -r '.data[0].b64_json' | base64 -d > cat.png +``` + +To create a request-level batch, send multiple independent requests +concurrently. Do not put several prompts into one `prompt` field: + +```bash +for prompt in "a red fox" "a blue bird" "a green frog"; do + curl -s -X POST http://localhost:8091/v1/images/generations \ + -H "Content-Type: application/json" \ + -d "{\"prompt\":\"${prompt}\",\"size\":\"1024x1024\",\"seed\":42}" \ + > /dev/null & +done +wait +``` + +The scheduler may batch these requests when their sampling parameters are +compatible. See the +[Image Generation API](../../serving/image_generation_api.md) for response +formats and additional client examples. Streaming models can use +model-specific streaming endpoints documented by their serving guide. + +## Python API + +The Python arguments mirror the CLI flags: + +```python +from vllm_omni import Omni + +omni = Omni( + model="Qwen/Qwen-Image", + max_num_seqs=4, + request_batch_max_wait_ms=20.0, +) + +outputs = omni.generate( + [ + "a cup of coffee on a table", + "a toy dinosaur on a sandy beach", + "a fox waking up in bed and yawning", + ] +) +``` + +Each list item passed to `Omni.generate()` becomes an independent logical +request that the scheduler may batch with compatible items. + +For step execution: + +```python +from vllm_omni import Omni +from vllm_omni.inputs.data import OmniDiffusionSamplingParams + +omni = Omni( + model="Qwen/Qwen-Image", + step_execution=True, + max_num_seqs=1, +) + +outputs = omni.generate( + "A cat sitting on a windowsill", + OmniDiffusionSamplingParams(num_inference_steps=50), +) +``` + +## Stage Configuration + +Use the equivalent engine arguments in a deployment YAML: + +```yaml +stage_args: + - stage_id: 0 + stage_type: diffusion + engine_args: + step_execution: false + max_num_seqs: 4 + request_batch_max_wait_ms: 20 +``` + +For step execution, set `step_execution: true` and remove +`request_batch_max_wait_ms`. + +## CLI Reference + +| Flag | Default | Effect | +|---|---:|---| +| `--step-execution` | disabled | Select step-wise scheduling | +| `--max-num-seqs` | `1` for diffusion stages | Set request- or step-scheduler capacity | +| `--request-batch-max-wait-ms` | `0` | Wait for burst coalescing in request mode | +| `--diffusion-streaming-output` | disabled | Expose supported intermediate diffusion outputs and require step execution | + +## Limitations and Troubleshooting + +- Requests batch only when their compatibility-sensitive parameters match. +- Different LoRA adapters or scales run in separate batches. +- FIFO scheduling can cause an incompatible request to block later compatible + requests. +- All diffusion cache backends are unsupported in step mode. KV transfer is + supported for newly admitted step requests; some other request-mode extras + remain unsupported. +- If request-mode startup reports that the pipeline does not support batching, + use `--max-num-seqs 1`. +- If step-mode startup mentions `prepare_encode()`, `denoise_step()`, + `step_scheduler()`, or `post_decode()`, the pipeline does not implement the + required step contract. + +For implementation details and model-author guidance, see +[Diffusion Continuous Batching](../../design/feature/diffusion_continuous_batching.md). diff --git a/docs/user_guide/diffusion/request_batching.md b/docs/user_guide/diffusion/request_batching.md deleted file mode 100644 index 35bf71725..000000000 --- a/docs/user_guide/diffusion/request_batching.md +++ /dev/null @@ -1,119 +0,0 @@ -# Request-Level Batching - -Request-level batching lets diffusion serving combine multiple compatible -logical requests into one pipeline `forward()` call. Each prompt remains a -separate request with its own `request_id`, sampling parameters, seed, output, -error, and abort state. The scheduler decides which requests can run together. - -!!! warning "Prompt List Semantics" - - Diffusion request-level batching does not support a top-level packed - list-prompt request. Submit multiple prompts as independent requests and let - the scheduler batch compatible in-flight requests. Multimodal payloads stay - inside a single prompt dict, for example - `{"prompt": "...", "multi_modal_data": {"image": image}}`. - -## Enablement - -Request-level execution is the default diffusion request mode when -`step_execution=False`. `max_num_seqs` controls the maximum number of compatible -requests in one scheduler wave: - -- `max_num_seqs=1` is the serial request path. -- `max_num_seqs>1` allows fused request-level batching for pipelines that - declare request-batch support. - -Increase `max_num_seqs` above `1` to allow the request scheduler to keep more -than one compatible request active: - -```bash -vllm serve Qwen/Qwen-Image --omni \ - --port 8091 \ - --max-num-seqs 4 -``` - -For bursty online traffic, you can also set a small admission wait window. This -lets the engine wait briefly before the first `schedule()` of a new wave so -nearby compatible requests can arrive and share the same fused forward pass: - -```bash -vllm serve Qwen/Qwen-Image --omni \ - --port 8091 \ - --max-num-seqs 4 \ - --request-batch-max-wait-ms 20 -``` - -`--request-batch-max-wait-ms 0` is the default and disables admission waiting, -so there is no added wait latency. - -For deploy YAMLs, configure the diffusion stage engine args: - -```yaml -stage_args: - - stage_id: 0 - stage_type: diffusion - engine_args: - max_num_seqs: 4 - request_batch_max_wait_ms: 20 -``` - -## Compatibility - -Only pipelines that declare request-batch support can use the fused -multi-request `pipeline.forward(batch)` path. Pipelines that do not support this -contract can still use request-level execution with `max_num_seqs=1`, which is -the serial request path. Setting `max_num_seqs>1` for a pipeline without -request-batch support fails early during engine initialization. - -The scheduler batches only compatible requests. Compatibility is based on -shape-sensitive and guidance-sensitive sampling fields, including resolution, -frame count, CFG settings, output count, and LoRA identity. Requests with -different LoRA adapters or scales are kept in separate batches so the worker -activates one adapter per batch. - -Request-level batching applies when `step_execution=False`. For the separate -step-wise runtime, see [Step Execution](step_execution.md). - -## Tuning - -- `max_num_seqs` caps the number of active compatible requests in one scheduler - wave. `1` means serial request execution; values above `1` enable fused - request-level batching when the pipeline supports it. -- `request_batch_max_wait_ms` is an upper bound on extra admission wait before a - new wave starts. Keep it small for latency-sensitive serving; values such as - `10` to `50` ms are a practical starting range for bursty HTTP ingress. -- `0` disables admission waiting and preserves the lowest first-request latency. -- FIFO ordering is conservative: an incompatible request at the front of the - waiting queue can block later compatible requests from joining the current - batch. - -## Python API - -When constructing `Omni`, pass the same engine arguments: - -```python -from vllm_omni.entrypoints.omni import Omni - -omni = Omni( - model="Qwen/Qwen-Image", - max_num_seqs=4, - request_batch_max_wait_ms=20.0, -) - -outputs = omni.generate( - [ - "a cup of coffee on a table", - "a toy dinosaur on a sandy beach", - "a fox waking up in bed and yawning", - ] -) -``` - -`Omni.generate([...])` submits each list item as its own logical diffusion -request. The runtime may batch those requests internally when their sampling -parameters are compatible. - -## For Contributors - -For implementation details and model-author guidance, see -[Request-Level Batching for Diffusion](../../design/feature/diffusion_request_level_batching.md). diff --git a/docs/user_guide/diffusion/step_execution.md b/docs/user_guide/diffusion/step_execution.md deleted file mode 100644 index 4dfedfa20..000000000 --- a/docs/user_guide/diffusion/step_execution.md +++ /dev/null @@ -1,100 +0,0 @@ -# Step Execution - -Step execution is an opt-in diffusion execution mode enabled with -`step_execution=True` when constructing `Omni`. - -It is not a generic diffusion toggle for every pipeline. Only pipelines that -implement the stepwise contract support it today. - -## Quick Start - -### Python API - -```python -from vllm_omni import Omni -from vllm_omni.inputs.data import OmniDiffusionSamplingParams - -omni = Omni( - model="Qwen/Qwen-Image", - step_execution=True, -) - -outputs = omni.generate( - "A cat sitting on a windowsill", - OmniDiffusionSamplingParams( - num_inference_steps=50, - ), -) -``` - -### Serving - -```bash -vllm serve Qwen/Qwen-Image --omni \ - --port 8091 \ - --step-execution \ - --max-num-seqs 8 -``` - -For serving, `--step-execution` enables the step-wise runtime. Continuous -batching only becomes relevant when `--max-num-seqs > 1`. - -`max_num_seqs` controls the step scheduler capacity: - -- `max_num_seqs=1` runs one request through the step-wise state machine. -- `max_num_seqs>1` allows compatible active requests to share step waves. - -## Supported Pipelines - -| Pipeline | Example models | Step execution | -|----------|----------------|----------------| -| `QwenImagePipeline` | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes | -| All other diffusion pipelines | `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, `QwenImageLayeredPipeline`, GLM-Image, Wan, Flux, etc. | No | - -!!! warning "Experimental continuous batching" - When `--step-execution` is enabled and `max_num_seqs > 1` is configured, - the step-wise path can batch - compatible requests together. This is experimental. Requests with - incompatible sampling parameters are intentionally kept in separate batches, - and `max_num_seqs=1` remains the conservative single-request step path. - -## Current Limitations - -- Continuous batching under `step_execution` is experimental and only batches - compatible requests. -- `cache_backend` is not supported together with step execution. -- Unsupported pipelines fail early during model loading. -- Request-mode extras such as KV transfer are not wired into step mode yet. -- LoRA is supported in step mode, but each batch must use a single adapter: - requests with different `lora_request` or `lora_scale` are scheduled into - separate batches. - -## When To Use It - -Use step execution only when you specifically need the pipeline to run through -its stepwise request state machine. For normal diffusion inference, leave it -disabled unless your workflow depends on this mode. - -For Qwen-Image online serving, the usual progression is: - -- start with `--step-execution --max-num-seqs 1` if you only need the step-wise path -- increase `--max-num-seqs` after that if you want the experimental compatible-request batching behavior - -If you are looking for general diffusion speedups, see -[Diffusion Features Overview](../diffusion_features.md). - -## Troubleshooting - -If model loading fails with a message mentioning `prepare_encode()`, -`denoise_step()`, `step_scheduler()`, and `post_decode()`, the selected -pipeline does not support step execution. - -## For Model Authors - -If you want to add step execution support to a new diffusion pipeline, see the -implementation guide: -[Diffusion Step Execution Design](../../design/feature/diffusion_step_execution.md). - -If you also want that pipeline to participate in the experimental batched -step-wise path, see: -[Continuous Batching for Step-Wise Diffusion](../../design/feature/diffusion_continuous_batching.md). diff --git a/docs/user_guide/diffusion_features.md b/docs/user_guide/diffusion_features.md index 2d9de3dba..3f631d5e5 100644 --- a/docs/user_guide/diffusion_features.md +++ b/docs/user_guide/diffusion_features.md @@ -80,12 +80,11 @@ denoise steps. | Method | Description | Best For | |--------|-------------|----------| -| **[Request-Level Batching](diffusion/request_batching.md)** | Scheduler batches compatible independent diffusion requests into one pipeline forward pass | Bursty online serving and multi-request throughput | -| **[Step Execution](diffusion/step_execution.md)** | Per-step denoise execution with mid-request abort support | Request cancellation between denoise steps, fine-grained execution control | +| **[Diffusion Execution Modes](diffusion/execution_modes.md)** | Configures serial requests, request batching, step execution, continuous batching, and streaming output | Matching latency, throughput, cancellation, and output-delivery requirements | -**Note:** Request-level batching is available for pipelines that declare the -request-batch forward contract. Step execution is currently supported by -QwenImagePipeline only. See [Supported Models](#supported-models) for details. +**Note:** Request-level batching and step execution are capability-based. +Consult the execution guide and the selected pipeline's documentation for +current support. ### Quantization Methods @@ -206,7 +205,7 @@ The following tables show which models support each feature: 3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible. 4. CPU Offloading (Layerwise) supports single-card for now. 5. Using FP8-Quant as an example of qunatization methods. - 6. Step Execution is not compatible with cache backends (TeaCache, Cache-DiT). LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches). + 6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches). ## Multi-Thread Weight Loading @@ -286,7 +285,7 @@ Measured on NVIDIA H800: **Execution Modes:** -- **[Step Execution Guide](diffusion/step_execution.md)** - Per-step denoise execution with mid-request abort support +- **[Diffusion Execution Modes](diffusion/execution_modes.md)** - Configure request batching, step execution, continuous batching, and streaming output **Startup Optimization:** diff --git a/docs/user_guide/examples/offline_inference/text_to_image.md b/docs/user_guide/examples/offline_inference/text_to_image.md index 7cd508d29..f1cdb9293 100644 --- a/docs/user_guide/examples/offline_inference/text_to_image.md +++ b/docs/user_guide/examples/offline_inference/text_to_image.md @@ -184,7 +184,7 @@ if __name__ == "__main__": !!! info For diffusion request-level batching controls such as `max_num_seqs`, see - [Request-Level Batching](../../diffusion/request_batching.md). + [Diffusion Execution Modes](../../diffusion/execution_modes.md). ### Negative Prompts diff --git a/examples/offline_inference/text_to_image/README.md b/examples/offline_inference/text_to_image/README.md index 985c4ff7d..6ebc7ae63 100644 --- a/examples/offline_inference/text_to_image/README.md +++ b/examples/offline_inference/text_to_image/README.md @@ -260,7 +260,7 @@ if __name__ == "__main__": For diffusion pipelines, the input list is sliced into single-item requests before feeding into the diffusion pipeline. For request-level batching controls such as `max_num_seqs`, see - [Request-Level Batching](../../../docs/user_guide/diffusion/request_batching.md). + [Diffusion Execution Modes](../../../docs/user_guide/diffusion/execution_modes.md). ### Negative Prompts diff --git a/mkdocs.yml b/mkdocs.yml index 8d5ffaa1b..cda933c44 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -71,6 +71,12 @@ plugins: - search - autorefs - awesome-nav + - redirects: + redirect_maps: + user_guide/diffusion/request_batching.md: user_guide/diffusion/execution_modes.md + user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md + design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md + design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md - glightbox - git-revision-date-localized: # exclude files diff --git a/pyproject.toml b/pyproject.toml index 4af374589..38936daaa 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -108,6 +108,7 @@ docs = [ "mkdocstrings-python", "mkdocs-gen-files", "mkdocs-awesome-nav", + "mkdocs-redirects", "mkdocs-glightbox", "mkdocs-git-revision-date-localized-plugin", "mkdocs-minify-plugin", diff --git a/recipes/LTX/LTX-2.md b/recipes/LTX/LTX-2.md index fa9ec6755..3142f9fd1 100644 --- a/recipes/LTX/LTX-2.md +++ b/recipes/LTX/LTX-2.md @@ -332,6 +332,6 @@ bundled offline CLI do not currently expose `sigmas`. - - - [Online video generation](../../docs/user_guide/examples/online_serving/text_to_video.md) -- [Request batching](../../docs/user_guide/diffusion/request_batching.md) +- [Diffusion execution modes](../../docs/user_guide/diffusion/execution_modes.md) - [T2V offline example](../../examples/offline_inference/text_to_video/text_to_video.md) - [I2V offline example](../../examples/offline_inference/image_to_video/README.md)