[Refact] Refactor diffusion output streaming (#5166)

Signed-off-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
bjf-frzandHongsheng Liu authored and GitHub committed 2026-07-23 10:06:08 +08:00
1 parent 7d5376f215
commit be667a8067
27 files changed
+628 -394

No files matched your search

+6 -1
View File
@@ -377,7 +377,12 @@ Default: `"0"`
#### `engine_args.max_num_seqs`
The maximum number of sequences for concurrent processing in this stage. For LLM stages, this controls the vLLM scheduler's maximum concurrent sequences. For all stage types, this also controls how many tasks can be batched together in the task processing loop.
The maximum number of sequences for concurrent processing in this stage. For
LLM stages, this controls the vLLM scheduler's maximum concurrent sequences.
For diffusion stages, this controls scheduler wave capacity in both request
batch mode and step batch mode. `max_num_seqs: 1` is the serial/conservative
path; values above `1` allow compatible requests to batch when the selected
diffusion pipeline supports that batching mode.
Default: `1`
@@ -139,7 +139,8 @@ compatibility gating and runner-side `StepInputBatch` packing.
## Current Limitations
- Experimental feature; use `max_num_seqs=1` for the older conservative path.
- Experimental feature; use `max_num_seqs=1` for the conservative single-request
step path.
- Only native pipelines that already support `step_execution=True`.
- Only homogeneous batches keyed by `StepBatchSamplingParamsKey` are supported.
- `cache_backend`, KV transfer, and other request-mode extras are not wired
@@ -32,7 +32,7 @@ Current engine/runtime limitations:
- Continuous batching with `max_num_seqs > 1` is experimental and documented in
[Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md).
Keep `max_num_seqs=1` if you want the older conservative behavior.
Keep `max_num_seqs=1` for the conservative single-request step path.
- `cache_backend` is not supported in step mode.
- Request-mode extras such as KV transfer are not wired into step mode yet.
- Unsupported pipelines now fail early during model loading instead of failing on the first request.
+76 -47
View File
@@ -52,7 +52,7 @@ The diffusion module follows a **multi-process, distributed architecture** with
### Responsibilities
The `DiffusionEngine` is the **orchestrator** of the diffusion inference system. It manages the lifecycle of worker processes and coordinates the execution flow.
The `DiffusionEngine` is the **orchestrator** of the diffusion inference system. It owns request admission, scheduler/executor coordination, output-stream delivery, cancellation cleanup, and startup warmup.
### Key Components
@@ -60,68 +60,84 @@ The `DiffusionEngine` is the **orchestrator** of the diffusion inference system.
```python
class DiffusionEngine:
def __init__(self, od_config: OmniDiffusionConfig):
def __init__(
self,
od_config: OmniDiffusionConfig,
scheduler: SchedulerInterface | None = None,
):
self.od_config = od_config
self.post_process_func = get_diffusion_post_process_func(od_config)
self.pre_process_func = get_diffusion_pre_process_func(od_config)
self._processes: list[mp.Process] = []
self._make_client()
self._init_process_hooks(od_config)
self.execution_mode = self._resolve_execution_mode(od_config)
self._init_executor(od_config)
self._init_scheduler(od_config, scheduler)
self._init_runtime_state()
self._init_execute_fn()
```
**Key Features**:
- **Pre/Post Processing**: Registers model-specific pre-processing and post-processing functions via registry pattern
- **Worker Management**: Launches and manages multiple worker processes (one per GPU)
- **Execution Mode Selection**: Selects one of two execution modes:
- `REQUEST_BATCH`: complete requests are scheduled through request-level execution. `max_num_seqs=1` is the serial request path; values above `1` allow compatible requests to use fused `pipeline.forward(batch)` when the pipeline supports it.
- `STEP_BATCH`: requests are advanced one denoising step at a time through `prepare_encode()`, `denoise_step()`, `step_scheduler()`, and `post_decode()`. `max_num_seqs` controls the maximum number of compatible active requests in one step wave.
- **Process Isolation**: Uses multiprocessing for true parallelism
- **Scheduler Selection**: Uses `StepScheduler` for step-batch execution and `RequestScheduler` for request-batch execution. Tests and custom integrations may inject a `SchedulerInterface` instance explicitly.
#### 1.2 Worker Launch Process
- **Executor Selection**: Builds the configured `DiffusionExecutor`, then routes scheduled work through `execute_batch` or `execute_step` according to the execution mode. `execute_batch` handles the serial `max_num_seqs=1` request path internally.
The engine launches workers using a **spawn** method:
- **Factory-Owned Warmup**: `DiffusionEngine.make_engine()` constructs the selected engine backend and then runs startup warmup through `run_startup_warmup()`.
#### 1.2 Execution Modes
```python
def _launch_workers(self, broadcast_handle):
# Creates one process per GPU
for i in range(num_gpus):
process = mp.Process(
target=worker_proc.worker_main,
args=(i, od_config, writer, broadcast_handle),
name=f"DiffusionWorker-{i}",
)
process.start()
class DiffusionExecutionMode(str, Enum):
REQUEST_BATCH = "request_batch"
STEP_BATCH = "step_batch"
```
`streaming_output=True` requires step execution. If streaming is enabled without `step_execution=True`, the engine enables step execution so the output stream can receive intermediate chunks or the final step output.
#### 1.3 Output Stream Lifecycle
```python
request_id = engine.add_request(request)
async for output in engine.get_output_stream(request_id):
yield output
```
**Design Decisions**:
- **Spawn Method**: Ensures clean state for each worker (no shared memory issues)
- **Unified Stream Semantics**: `step_streaming()` is the public async engine path. Streaming callers forward every yielded output. Non-streaming callers consume the same stream to completion and return only the final output.
- **Pipe Communication**: Uses `mp.Pipe` for initialization handshake
- **Queue Delivery**: Each admitted request gets an `asyncio.Queue[DiffusionOutput]`. The busy loop emits chunks/final outputs into that queue.
- **Device Selection**: Each worker is assigned a specific GPU (`cuda:{rank}`)
- **Cancellation Cleanup**: If a client disconnects or cancels stream consumption, `get_output_stream()` removes only the queue. Scheduler terminal state is still finalized when the scheduler reports the request as finished, so request IDs do not remain active indefinitely.
#### 1.3 Request Processing Flow
#### 1.4 Request Processing Flow
```python
def step(self, requests: list[OmniDiffusionRequest]):
async def step_streaming(self, request: OmniDiffusionRequest):
# 1. Pre-process requests
requests = self.pre_process_func(requests)
request = self.pre_process_func(request)
# 2. Send to scheduler and wait for response
output = self.add_req_and_wait_for_response(requests)
# 3. Post-process results
result = self.post_process_func(output.output)
return result
# 2. Add request and consume the unified output stream
async for output in self.async_add_req_and_stream_response(request):
# 3. Post-process each delivered output
yield self.postprocess_output(request, output, ...)
```
**Flow**:
1. **Pre-processing**: Applies model-specific transformations
2. **Scheduling**: Delegates to scheduler for distribution
2. **Scheduling**: Delegates request state transitions to the selected scheduler
3. **Post-processing**: Converts raw outputs to final format (e.g., PIL images)
3. **Execution**: Calls the selected executor function for request, request-batch, or step execution
4. **Output Delivery**: Emits intermediate chunks and final outputs through the request output stream
5. **Post-processing**: Converts raw outputs to final format (e.g., PIL images)
---
@@ -915,36 +931,49 @@ def initialize_model_parallel(
1. User Request
└─> Omni.generate(prompt)
└─> Prepare OmniDiffusionRequest
└─> DiffusionEngine.step(requests)
└─> DiffusionEngine.step_streaming(request)
2. Pre-processing
└─> pre_process_func(requests)
└─> pre_process_func(request)
└─> Model-specific transformations
3. Scheduling
└─> scheduler.add_request(request)
└─> scheduler.schedule()
└─> DiffusionEngine submits scheduled request to executor.add_req(req)
└─> scheduler.add_request(request)
└─> scheduler.schedule()
└─> DiffusionEngine selects execute_batch / execute_step
4. Worker Execution
└─> WorkerProc._worker_busy_loop()
└─> DiffusionWorker.execute_model(req)
└─> Pipeline.forward(req)
├─> encode_prompt()
├─> prepare_latents()
├─> diffuse() [loop]
│ ├─> transformer.forward() [with cache backend hooks]
│ └─> scheduler.step()
└─> vae.decode()
└─> DiffusionModelRunner selects execution mode
├─> REQUEST_BATCH
│ └─> Pipeline.forward(req_or_batch)
│ ├─> encode_prompt()
│ ├─> prepare_latents()
│ ├─> diffuse() [loop]
│ │ ├─> transformer.forward() [with cache backend hooks]
│ │ └─> scheduler.step()
│ └─> vae.decode()
└─> STEP
├─> pipeline.prepare_encode(state) [new requests]
├─> pipeline.denoise_step(input_batch)
├─> pipeline.step_scheduler(state, noise_pred)
└─> pipeline.post_decode(state) [chunk boundary or final step]
5. Result Collection
└─> Executor returns DiffusionOutput
└─> Executor returns RunnerOutput / BatchRunnerOutput
└─> scheduler.update_from_output(...)
└─> DiffusionEngine pops finished request state
├─> DiffusionEngine emits chunk/final output to request queue
└─> DiffusionEngine finalizes terminal scheduler state
6. Post-processing
└─> post_process_func(output)
└─> Convert to PIL images / final format
7. Stream Completion / Cancellation
├─> Streaming callers receive every yielded output
├─> Non-streaming callers drain the stream and return the final output
└─> Cancelled streams drop their output queue but still allow scheduler state cleanup
```
---
+15 -6
View File
@@ -15,6 +15,14 @@ error, and abort state. The scheduler decides which requests can run together.
## Enablement
Request-level execution is the default diffusion request mode when
`step_execution=False`. `max_num_seqs` controls the maximum number of compatible
requests in one scheduler wave:
- `max_num_seqs=1` is the serial request path.
- `max_num_seqs>1` allows fused request-level batching for pipelines that
declare request-batch support.
Increase `max_num_seqs` above `1` to allow the request scheduler to keep more
than one compatible request active:
@@ -51,11 +59,11 @@ stage_args:
## Compatibility
Only pipelines that declare request-batch support use the fused request-batch
path. The engine validates that the pipeline `forward()` uses the request-batch
contract and returns `list[DiffusionOutput]`. Pipelines that do not support this
contract do not use fused `pipeline.forward(batch)`; scheduled requests are
executed through per-request worker calls.
Only pipelines that declare request-batch support can use the fused
multi-request `pipeline.forward(batch)` path. Pipelines that do not support this
contract can still use request-level execution with `max_num_seqs=1`, which is
the serial request path. Setting `max_num_seqs>1` for a pipeline without
request-batch support fails early during engine initialization.
The scheduler batches only compatible requests. Compatibility is based on
shape-sensitive and guidance-sensitive sampling fields, including resolution,
@@ -69,7 +77,8 @@ step-wise runtime, see [Step Execution](step_execution.md).
## Tuning
- `max_num_seqs` caps the number of active compatible requests in one scheduler
wave.
wave. `1` means serial request execution; values above `1` enable fused
request-level batching when the pipeline supports it.
- `request_batch_max_wait_ms` is an upper bound on extra admission wait before a
new wave starts. Keep it small for latency-sensitive serving; values such as
`10` to `50` ms are a practical starting range for bursty HTTP ingress.
+6 -1
View File
@@ -39,6 +39,11 @@ vllm serve Qwen/Qwen-Image --omni \
For serving, `--step-execution` enables the step-wise runtime. Continuous
batching only becomes relevant when `--max-num-seqs > 1`.
`max_num_seqs` controls the step scheduler capacity:
- `max_num_seqs=1` runs one request through the step-wise state machine.
- `max_num_seqs>1` allows compatible active requests to share step waves.
## Supported Pipelines
| Pipeline | Example models | Step execution |
@@ -51,7 +56,7 @@ batching only becomes relevant when `--max-num-seqs > 1`.
the step-wise path can batch
compatible requests together. This is experimental. Requests with
incompatible sampling parameters are intentionally kept in separate batches,
and `max_num_seqs=1` remains the conservative default.
and `max_num_seqs=1` remains the conservative single-request step path.
## Current Limitations