[Refact] Refactor diffusion output streaming (#5166)
Signed-off-by: bjf-frz <frz123db@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
7d5376f215
commit
be667a8067
27 files changed
+628
-394
No files matched your search
@@ -377,7 +377,12 @@ Default: `"0"`
|
||||
|
||||
#### `engine_args.max_num_seqs`
|
||||
|
||||
The maximum number of sequences for concurrent processing in this stage. For LLM stages, this controls the vLLM scheduler's maximum concurrent sequences. For all stage types, this also controls how many tasks can be batched together in the task processing loop.
|
||||
The maximum number of sequences for concurrent processing in this stage. For
|
||||
LLM stages, this controls the vLLM scheduler's maximum concurrent sequences.
|
||||
For diffusion stages, this controls scheduler wave capacity in both request
|
||||
batch mode and step batch mode. `max_num_seqs: 1` is the serial/conservative
|
||||
path; values above `1` allow compatible requests to batch when the selected
|
||||
diffusion pipeline supports that batching mode.
|
||||
|
||||
Default: `1`
|
||||
|
||||
|
||||
@@ -139,7 +139,8 @@ compatibility gating and runner-side `StepInputBatch` packing.
|
||||
|
||||
## Current Limitations
|
||||
|
||||
- Experimental feature; use `max_num_seqs=1` for the older conservative path.
|
||||
- Experimental feature; use `max_num_seqs=1` for the conservative single-request
|
||||
step path.
|
||||
- Only native pipelines that already support `step_execution=True`.
|
||||
- Only homogeneous batches keyed by `StepBatchSamplingParamsKey` are supported.
|
||||
- `cache_backend`, KV transfer, and other request-mode extras are not wired
|
||||
|
||||
@@ -32,7 +32,7 @@ Current engine/runtime limitations:
|
||||
|
||||
- Continuous batching with `max_num_seqs > 1` is experimental and documented in
|
||||
[Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md).
|
||||
Keep `max_num_seqs=1` if you want the older conservative behavior.
|
||||
Keep `max_num_seqs=1` for the conservative single-request step path.
|
||||
- `cache_backend` is not supported in step mode.
|
||||
- Request-mode extras such as KV transfer are not wired into step mode yet.
|
||||
- Unsupported pipelines now fail early during model loading instead of failing on the first request.
|
||||
|
||||
@@ -52,7 +52,7 @@ The diffusion module follows a **multi-process, distributed architecture** with
|
||||
|
||||
### Responsibilities
|
||||
|
||||
The `DiffusionEngine` is the **orchestrator** of the diffusion inference system. It manages the lifecycle of worker processes and coordinates the execution flow.
|
||||
The `DiffusionEngine` is the **orchestrator** of the diffusion inference system. It owns request admission, scheduler/executor coordination, output-stream delivery, cancellation cleanup, and startup warmup.
|
||||
|
||||
### Key Components
|
||||
|
||||
@@ -60,68 +60,84 @@ The `DiffusionEngine` is the **orchestrator** of the diffusion inference system.
|
||||
|
||||
```python
|
||||
class DiffusionEngine:
|
||||
def __init__(self, od_config: OmniDiffusionConfig):
|
||||
def __init__(
|
||||
self,
|
||||
od_config: OmniDiffusionConfig,
|
||||
scheduler: SchedulerInterface | None = None,
|
||||
):
|
||||
self.od_config = od_config
|
||||
self.post_process_func = get_diffusion_post_process_func(od_config)
|
||||
self.pre_process_func = get_diffusion_pre_process_func(od_config)
|
||||
self._processes: list[mp.Process] = []
|
||||
self._make_client()
|
||||
self._init_process_hooks(od_config)
|
||||
self.execution_mode = self._resolve_execution_mode(od_config)
|
||||
self._init_executor(od_config)
|
||||
self._init_scheduler(od_config, scheduler)
|
||||
self._init_runtime_state()
|
||||
self._init_execute_fn()
|
||||
```
|
||||
|
||||
**Key Features**:
|
||||
|
||||
- **Pre/Post Processing**: Registers model-specific pre-processing and post-processing functions via registry pattern
|
||||
|
||||
- **Worker Management**: Launches and manages multiple worker processes (one per GPU)
|
||||
- **Execution Mode Selection**: Selects one of two execution modes:
|
||||
- `REQUEST_BATCH`: complete requests are scheduled through request-level execution. `max_num_seqs=1` is the serial request path; values above `1` allow compatible requests to use fused `pipeline.forward(batch)` when the pipeline supports it.
|
||||
- `STEP_BATCH`: requests are advanced one denoising step at a time through `prepare_encode()`, `denoise_step()`, `step_scheduler()`, and `post_decode()`. `max_num_seqs` controls the maximum number of compatible active requests in one step wave.
|
||||
|
||||
- **Process Isolation**: Uses multiprocessing for true parallelism
|
||||
- **Scheduler Selection**: Uses `StepScheduler` for step-batch execution and `RequestScheduler` for request-batch execution. Tests and custom integrations may inject a `SchedulerInterface` instance explicitly.
|
||||
|
||||
#### 1.2 Worker Launch Process
|
||||
- **Executor Selection**: Builds the configured `DiffusionExecutor`, then routes scheduled work through `execute_batch` or `execute_step` according to the execution mode. `execute_batch` handles the serial `max_num_seqs=1` request path internally.
|
||||
|
||||
The engine launches workers using a **spawn** method:
|
||||
- **Factory-Owned Warmup**: `DiffusionEngine.make_engine()` constructs the selected engine backend and then runs startup warmup through `run_startup_warmup()`.
|
||||
|
||||
#### 1.2 Execution Modes
|
||||
|
||||
```python
|
||||
def _launch_workers(self, broadcast_handle):
|
||||
# Creates one process per GPU
|
||||
for i in range(num_gpus):
|
||||
process = mp.Process(
|
||||
target=worker_proc.worker_main,
|
||||
args=(i, od_config, writer, broadcast_handle),
|
||||
name=f"DiffusionWorker-{i}",
|
||||
)
|
||||
process.start()
|
||||
class DiffusionExecutionMode(str, Enum):
|
||||
REQUEST_BATCH = "request_batch"
|
||||
STEP_BATCH = "step_batch"
|
||||
```
|
||||
|
||||
`streaming_output=True` requires step execution. If streaming is enabled without `step_execution=True`, the engine enables step execution so the output stream can receive intermediate chunks or the final step output.
|
||||
|
||||
#### 1.3 Output Stream Lifecycle
|
||||
|
||||
```python
|
||||
request_id = engine.add_request(request)
|
||||
async for output in engine.get_output_stream(request_id):
|
||||
yield output
|
||||
```
|
||||
|
||||
**Design Decisions**:
|
||||
|
||||
- **Spawn Method**: Ensures clean state for each worker (no shared memory issues)
|
||||
- **Unified Stream Semantics**: `step_streaming()` is the public async engine path. Streaming callers forward every yielded output. Non-streaming callers consume the same stream to completion and return only the final output.
|
||||
|
||||
- **Pipe Communication**: Uses `mp.Pipe` for initialization handshake
|
||||
- **Queue Delivery**: Each admitted request gets an `asyncio.Queue[DiffusionOutput]`. The busy loop emits chunks/final outputs into that queue.
|
||||
|
||||
- **Device Selection**: Each worker is assigned a specific GPU (`cuda:{rank}`)
|
||||
- **Cancellation Cleanup**: If a client disconnects or cancels stream consumption, `get_output_stream()` removes only the queue. Scheduler terminal state is still finalized when the scheduler reports the request as finished, so request IDs do not remain active indefinitely.
|
||||
|
||||
#### 1.3 Request Processing Flow
|
||||
#### 1.4 Request Processing Flow
|
||||
|
||||
```python
|
||||
def step(self, requests: list[OmniDiffusionRequest]):
|
||||
async def step_streaming(self, request: OmniDiffusionRequest):
|
||||
# 1. Pre-process requests
|
||||
requests = self.pre_process_func(requests)
|
||||
request = self.pre_process_func(request)
|
||||
|
||||
# 2. Send to scheduler and wait for response
|
||||
output = self.add_req_and_wait_for_response(requests)
|
||||
|
||||
# 3. Post-process results
|
||||
result = self.post_process_func(output.output)
|
||||
return result
|
||||
# 2. Add request and consume the unified output stream
|
||||
async for output in self.async_add_req_and_stream_response(request):
|
||||
# 3. Post-process each delivered output
|
||||
yield self.postprocess_output(request, output, ...)
|
||||
```
|
||||
|
||||
**Flow**:
|
||||
|
||||
1. **Pre-processing**: Applies model-specific transformations
|
||||
|
||||
2. **Scheduling**: Delegates to scheduler for distribution
|
||||
2. **Scheduling**: Delegates request state transitions to the selected scheduler
|
||||
|
||||
3. **Post-processing**: Converts raw outputs to final format (e.g., PIL images)
|
||||
3. **Execution**: Calls the selected executor function for request, request-batch, or step execution
|
||||
|
||||
4. **Output Delivery**: Emits intermediate chunks and final outputs through the request output stream
|
||||
|
||||
5. **Post-processing**: Converts raw outputs to final format (e.g., PIL images)
|
||||
|
||||
---
|
||||
|
||||
@@ -915,36 +931,49 @@ def initialize_model_parallel(
|
||||
1. User Request
|
||||
└─> Omni.generate(prompt)
|
||||
└─> Prepare OmniDiffusionRequest
|
||||
└─> DiffusionEngine.step(requests)
|
||||
└─> DiffusionEngine.step_streaming(request)
|
||||
|
||||
2. Pre-processing
|
||||
└─> pre_process_func(requests)
|
||||
└─> pre_process_func(request)
|
||||
└─> Model-specific transformations
|
||||
|
||||
3. Scheduling
|
||||
└─> scheduler.add_request(request)
|
||||
└─> scheduler.schedule()
|
||||
└─> DiffusionEngine submits scheduled request to executor.add_req(req)
|
||||
└─> scheduler.add_request(request)
|
||||
└─> scheduler.schedule()
|
||||
└─> DiffusionEngine selects execute_batch / execute_step
|
||||
|
||||
4. Worker Execution
|
||||
└─> WorkerProc._worker_busy_loop()
|
||||
└─> DiffusionWorker.execute_model(req)
|
||||
└─> Pipeline.forward(req)
|
||||
├─> encode_prompt()
|
||||
├─> prepare_latents()
|
||||
├─> diffuse() [loop]
|
||||
│ ├─> transformer.forward() [with cache backend hooks]
|
||||
│ └─> scheduler.step()
|
||||
└─> vae.decode()
|
||||
└─> DiffusionModelRunner selects execution mode
|
||||
├─> REQUEST_BATCH
|
||||
│ └─> Pipeline.forward(req_or_batch)
|
||||
│ ├─> encode_prompt()
|
||||
│ ├─> prepare_latents()
|
||||
│ ├─> diffuse() [loop]
|
||||
│ │ ├─> transformer.forward() [with cache backend hooks]
|
||||
│ │ └─> scheduler.step()
|
||||
│ └─> vae.decode()
|
||||
└─> STEP
|
||||
├─> pipeline.prepare_encode(state) [new requests]
|
||||
├─> pipeline.denoise_step(input_batch)
|
||||
├─> pipeline.step_scheduler(state, noise_pred)
|
||||
└─> pipeline.post_decode(state) [chunk boundary or final step]
|
||||
|
||||
5. Result Collection
|
||||
└─> Executor returns DiffusionOutput
|
||||
└─> Executor returns RunnerOutput / BatchRunnerOutput
|
||||
└─> scheduler.update_from_output(...)
|
||||
└─> DiffusionEngine pops finished request state
|
||||
├─> DiffusionEngine emits chunk/final output to request queue
|
||||
└─> DiffusionEngine finalizes terminal scheduler state
|
||||
|
||||
6. Post-processing
|
||||
└─> post_process_func(output)
|
||||
└─> Convert to PIL images / final format
|
||||
|
||||
7. Stream Completion / Cancellation
|
||||
├─> Streaming callers receive every yielded output
|
||||
├─> Non-streaming callers drain the stream and return the final output
|
||||
└─> Cancelled streams drop their output queue but still allow scheduler state cleanup
|
||||
```
|
||||
|
||||
---
|
||||
@@ -15,6 +15,14 @@ error, and abort state. The scheduler decides which requests can run together.
|
||||
|
||||
## Enablement
|
||||
|
||||
Request-level execution is the default diffusion request mode when
|
||||
`step_execution=False`. `max_num_seqs` controls the maximum number of compatible
|
||||
requests in one scheduler wave:
|
||||
|
||||
- `max_num_seqs=1` is the serial request path.
|
||||
- `max_num_seqs>1` allows fused request-level batching for pipelines that
|
||||
declare request-batch support.
|
||||
|
||||
Increase `max_num_seqs` above `1` to allow the request scheduler to keep more
|
||||
than one compatible request active:
|
||||
|
||||
@@ -51,11 +59,11 @@ stage_args:
|
||||
|
||||
## Compatibility
|
||||
|
||||
Only pipelines that declare request-batch support use the fused request-batch
|
||||
path. The engine validates that the pipeline `forward()` uses the request-batch
|
||||
contract and returns `list[DiffusionOutput]`. Pipelines that do not support this
|
||||
contract do not use fused `pipeline.forward(batch)`; scheduled requests are
|
||||
executed through per-request worker calls.
|
||||
Only pipelines that declare request-batch support can use the fused
|
||||
multi-request `pipeline.forward(batch)` path. Pipelines that do not support this
|
||||
contract can still use request-level execution with `max_num_seqs=1`, which is
|
||||
the serial request path. Setting `max_num_seqs>1` for a pipeline without
|
||||
request-batch support fails early during engine initialization.
|
||||
|
||||
The scheduler batches only compatible requests. Compatibility is based on
|
||||
shape-sensitive and guidance-sensitive sampling fields, including resolution,
|
||||
@@ -69,7 +77,8 @@ step-wise runtime, see [Step Execution](step_execution.md).
|
||||
## Tuning
|
||||
|
||||
- `max_num_seqs` caps the number of active compatible requests in one scheduler
|
||||
wave.
|
||||
wave. `1` means serial request execution; values above `1` enable fused
|
||||
request-level batching when the pipeline supports it.
|
||||
- `request_batch_max_wait_ms` is an upper bound on extra admission wait before a
|
||||
new wave starts. Keep it small for latency-sensitive serving; values such as
|
||||
`10` to `50` ms are a practical starting range for bursty HTTP ingress.
|
||||
|
||||
@@ -39,6 +39,11 @@ vllm serve Qwen/Qwen-Image --omni \
|
||||
For serving, `--step-execution` enables the step-wise runtime. Continuous
|
||||
batching only becomes relevant when `--max-num-seqs > 1`.
|
||||
|
||||
`max_num_seqs` controls the step scheduler capacity:
|
||||
|
||||
- `max_num_seqs=1` runs one request through the step-wise state machine.
|
||||
- `max_num_seqs>1` allows compatible active requests to share step waves.
|
||||
|
||||
## Supported Pipelines
|
||||
|
||||
| Pipeline | Example models | Step execution |
|
||||
@@ -51,7 +56,7 @@ batching only becomes relevant when `--max-num-seqs > 1`.
|
||||
the step-wise path can batch
|
||||
compatible requests together. This is experimental. Requests with
|
||||
incompatible sampling parameters are intentionally kept in separate batches,
|
||||
and `max_num_seqs=1` remains the conservative default.
|
||||
and `max_num_seqs=1` remains the conservative single-request step path.
|
||||
|
||||
## Current Limitations
|
||||
|
||||
|
||||
Reference in new issue
Block a user