anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[docs] consolidate diffusion execution modes (#5599)

Browse Source 
Signed-off-by: bjf-frz <frz123db@gmail.com>
This commit is contained in:
bjf-frz authored and GitHub committed 2026-08-05 22:41:18 +08:00
1 parent 589afcef6c
commit 9f978923f2
18 files changed
+475 -682
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/diffusion/execution_modes.md
+252
View File
@@ -0,0 +1,252 @@
# Diffusion Execution Modes
vLLM-Omni supports complete-request execution and step-wise diffusion
execution. Both modes use the same asynchronous engine output stream; the
configuration controls how work is scheduled and whether intermediate outputs
are exposed.
## Choose a Mode
| Goal | CLI configuration |
|---|---|
| Serial request execution | `--max-num-seqs 1` |
| Fused request-level batching | `--max-num-seqs N` |
| Single-request step execution | `--step-execution --max-num-seqs 1` |
| Step-wise continuous batching | `--step-execution --max-num-seqs N` |
| Chunked diffusion output | `--diffusion-streaming-output` |
`N` must be greater than `1` to allow batching. The selected pipeline must
support request-level batching or batched-step execution as appropriate;
single-request step support alone is not sufficient for step-wise continuous
batching.
## Request Execution
Request execution is the default when `--step-execution` is omitted. The
pipeline performs a complete `forward()` for each scheduler wave.
### Serial Requests
Use the conservative serial path with any request-mode pipeline:
```bash
vllm serve MODEL --omni \
--port 8091 \
--max-num-seqs 1
```
### Request-Level Batching
Set `max_num_seqs` above one to combine compatible independent requests into a
single pipeline forward:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4
```
For bursty traffic, add a small admission window:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4 \
--request-batch-max-wait-ms 20
```
`--request-batch-max-wait-ms 0` is the default. A nonzero value can improve
batch formation but adds up to that much latency before a new scheduler wave.
It has no effect in step mode.
Each prompt remains a separate logical request. Do not submit a top-level list
as one packed prompt. Concurrent serving requests are batched internally when
their shapes, guidance settings, output counts, and LoRA settings are
compatible.
Only pipelines that declare request-batch support accept
`max_num_seqs>1` in request mode. Unsupported pipelines fail during engine
initialization; use `max_num_seqs=1` instead.
## Step Execution
Step execution lets the scheduler advance and abort requests between denoise
steps:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--step-execution \
--max-num-seqs 1
```
Set a larger capacity to allow compatible requests to share denoise waves:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--step-execution \
--max-num-seqs 8
```
Step continuous batching is experimental. Start with `max_num_seqs=1` when
validating a model or debugging correctness, then increase it for
multi-request throughput.
Step execution is capability-based, not a generic switch for every diffusion
model. Qwen-Image supports step-wise continuous batching. HunyuanImage3 also
supports it, but only when its resolved self-attention backend is `TORCH_SDPA`;
set `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` or configure
`diffusion_attention_config.default.backend=TORCH_SDPA` before using
`--max-num-seqs >1`. See the
[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md)
for its validated configuration. Helios supports single-request step execution only: use
`--step-execution --max-num-seqs 1` for Helios. Consult the selected pipeline's
documentation and source for the latest support status.
## Streaming Output
Use `--diffusion-streaming-output` for a pipeline that can produce intermediate
diffusion outputs:
```bash
vllm serve BestWishYsh/Helios-Distilled --omni \
--port 8000 \
--diffusion-streaming-output
```
Streaming output requires step execution. If
`--diffusion-streaming-output` is set without `--step-execution`, the engine
enables step execution automatically. Model initialization fails if the
pipeline does not implement step execution.
Chunk-capable pipelines emit intermediate and final outputs through the same
request stream. Final-only step pipelines emit only the final result even when
streaming output is enabled.
The normal non-streaming serving path uses the same internal stream but drains
it and returns only the final output. Users do not need to enable
`--diffusion-streaming-output` to benefit from the unified engine lifecycle.
## Send Requests
Execution-mode flags configure the server. They do not change the
OpenAI-compatible client request format. For example, after starting any of
the non-streaming configurations above:
```bash
curl -X POST http://localhost:8091/v1/images/generations \
-H "Content-Type: application/json" \
-d '{
"prompt": "a cat sitting on a windowsill",
"size": "1024x1024",
"num_inference_steps": 50,
"seed": 42
}' | jq -r '.data[0].b64_json' | base64 -d > cat.png
```
To create a request-level batch, send multiple independent requests
concurrently. Do not put several prompts into one `prompt` field:
```bash
for prompt in "a red fox" "a blue bird" "a green frog"; do
curl -s -X POST http://localhost:8091/v1/images/generations \
-H "Content-Type: application/json" \
-d "{\"prompt\":\"${prompt}\",\"size\":\"1024x1024\",\"seed\":42}" \
> /dev/null &
done
wait
```
The scheduler may batch these requests when their sampling parameters are
compatible. See the
[Image Generation API](../../serving/image_generation_api.md) for response
formats and additional client examples. Streaming models can use
model-specific streaming endpoints documented by their serving guide.
## Python API
The Python arguments mirror the CLI flags:
```python
from vllm_omni import Omni
omni = Omni(
model="Qwen/Qwen-Image",
max_num_seqs=4,
request_batch_max_wait_ms=20.0,
)
outputs = omni.generate(
[
"a cup of coffee on a table",
"a toy dinosaur on a sandy beach",
"a fox waking up in bed and yawning",
]
)
```
Each list item passed to `Omni.generate()` becomes an independent logical
request that the scheduler may batch with compatible items.
For step execution:
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
omni = Omni(
model="Qwen/Qwen-Image",
step_execution=True,
max_num_seqs=1,
)
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(num_inference_steps=50),
)
```
## Stage Configuration
Use the equivalent engine arguments in a deployment YAML:
```yaml
stage_args:
- stage_id: 0
stage_type: diffusion
engine_args:
step_execution: false
max_num_seqs: 4
request_batch_max_wait_ms: 20
```
For step execution, set `step_execution: true` and remove
`request_batch_max_wait_ms`.
## CLI Reference
| Flag | Default | Effect |
|---|---:|---|
| `--step-execution` | disabled | Select step-wise scheduling |
| `--max-num-seqs` | `1` for diffusion stages | Set request- or step-scheduler capacity |
| `--request-batch-max-wait-ms` | `0` | Wait for burst coalescing in request mode |
| `--diffusion-streaming-output` | disabled | Expose supported intermediate diffusion outputs and require step execution |
## Limitations and Troubleshooting
- Requests batch only when their compatibility-sensitive parameters match.
- Different LoRA adapters or scales run in separate batches.
- FIFO scheduling can cause an incompatible request to block later compatible
requests.
- All diffusion cache backends are unsupported in step mode. KV transfer is
supported for newly admitted step requests; some other request-mode extras
remain unsupported.
- If request-mode startup reports that the pipeline does not support batching,
use `--max-num-seqs 1`.
- If step-mode startup mentions `prepare_encode()`, `denoise_step()`,
`step_scheduler()`, or `post_decode()`, the pipeline does not implement the
required step contract.
For implementation details and model-author guidance, see
[Diffusion Continuous Batching](../../design/feature/diffusion_continuous_batching.md).
docs/user_guide/diffusion/request_batching.md
-119
View File
@@ -1,119 +0,0 @@
# Request-Level Batching
Request-level batching lets diffusion serving combine multiple compatible
logical requests into one pipeline `forward()` call. Each prompt remains a
separate request with its own `request_id`, sampling parameters, seed, output,
error, and abort state. The scheduler decides which requests can run together.
!!! warning "Prompt List Semantics"
Diffusion request-level batching does not support a top-level packed
list-prompt request. Submit multiple prompts as independent requests and let
the scheduler batch compatible in-flight requests. Multimodal payloads stay
inside a single prompt dict, for example
`{"prompt": "...", "multi_modal_data": {"image": image}}`.
## Enablement
Request-level execution is the default diffusion request mode when
`step_execution=False`. `max_num_seqs` controls the maximum number of compatible
requests in one scheduler wave:
- `max_num_seqs=1` is the serial request path.
- `max_num_seqs>1` allows fused request-level batching for pipelines that
declare request-batch support.
Increase `max_num_seqs` above `1` to allow the request scheduler to keep more
than one compatible request active:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4
```
For bursty online traffic, you can also set a small admission wait window. This
lets the engine wait briefly before the first `schedule()` of a new wave so
nearby compatible requests can arrive and share the same fused forward pass:
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--max-num-seqs 4 \
--request-batch-max-wait-ms 20
```
`--request-batch-max-wait-ms 0` is the default and disables admission waiting,
so there is no added wait latency.
For deploy YAMLs, configure the diffusion stage engine args:
```yaml
stage_args:
- stage_id: 0
stage_type: diffusion
engine_args:
max_num_seqs: 4
request_batch_max_wait_ms: 20
```
## Compatibility
Only pipelines that declare request-batch support can use the fused
multi-request `pipeline.forward(batch)` path. Pipelines that do not support this
contract can still use request-level execution with `max_num_seqs=1`, which is
the serial request path. Setting `max_num_seqs>1` for a pipeline without
request-batch support fails early during engine initialization.
The scheduler batches only compatible requests. Compatibility is based on
shape-sensitive and guidance-sensitive sampling fields, including resolution,
frame count, CFG settings, output count, and LoRA identity. Requests with
different LoRA adapters or scales are kept in separate batches so the worker
activates one adapter per batch.
Request-level batching applies when `step_execution=False`. For the separate
step-wise runtime, see [Step Execution](step_execution.md).
## Tuning
- `max_num_seqs` caps the number of active compatible requests in one scheduler
wave. `1` means serial request execution; values above `1` enable fused
request-level batching when the pipeline supports it.
- `request_batch_max_wait_ms` is an upper bound on extra admission wait before a
new wave starts. Keep it small for latency-sensitive serving; values such as
`10` to `50` ms are a practical starting range for bursty HTTP ingress.
- `0` disables admission waiting and preserves the lowest first-request latency.
- FIFO ordering is conservative: an incompatible request at the front of the
waiting queue can block later compatible requests from joining the current
batch.
## Python API
When constructing `Omni`, pass the same engine arguments:
```python
from vllm_omni.entrypoints.omni import Omni
omni = Omni(
model="Qwen/Qwen-Image",
max_num_seqs=4,
request_batch_max_wait_ms=20.0,
)
outputs = omni.generate(
[
"a cup of coffee on a table",
"a toy dinosaur on a sandy beach",
"a fox waking up in bed and yawning",
]
)
```
`Omni.generate([...])` submits each list item as its own logical diffusion
request. The runtime may batch those requests internally when their sampling
parameters are compatible.
## For Contributors
For implementation details and model-author guidance, see
[Request-Level Batching for Diffusion](../../design/feature/diffusion_request_level_batching.md).
docs/user_guide/diffusion/step_execution.md
-100
View File
@@ -1,100 +0,0 @@
# Step Execution
Step execution is an opt-in diffusion execution mode enabled with
`step_execution=True` when constructing `Omni`.
It is not a generic diffusion toggle for every pipeline. Only pipelines that
implement the stepwise contract support it today.
## Quick Start
### Python API
```python
from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
omni = Omni(
model="Qwen/Qwen-Image",
step_execution=True,
)
outputs = omni.generate(
"A cat sitting on a windowsill",
OmniDiffusionSamplingParams(
num_inference_steps=50,
),
)
```
### Serving
```bash
vllm serve Qwen/Qwen-Image --omni \
--port 8091 \
--step-execution \
--max-num-seqs 8
```
For serving, `--step-execution` enables the step-wise runtime. Continuous
batching only becomes relevant when `--max-num-seqs > 1`.
`max_num_seqs` controls the step scheduler capacity:
- `max_num_seqs=1` runs one request through the step-wise state machine.
- `max_num_seqs>1` allows compatible active requests to share step waves.
## Supported Pipelines
| Pipeline | Example models | Step execution |
|----------|----------------|----------------|
| `QwenImagePipeline` | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes |
| All other diffusion pipelines | `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, `QwenImageLayeredPipeline`, GLM-Image, Wan, Flux, etc. | No |
!!! warning "Experimental continuous batching"
When `--step-execution` is enabled and `max_num_seqs > 1` is configured,
the step-wise path can batch
compatible requests together. This is experimental. Requests with
incompatible sampling parameters are intentionally kept in separate batches,
and `max_num_seqs=1` remains the conservative single-request step path.
## Current Limitations
- Continuous batching under `step_execution` is experimental and only batches
compatible requests.
- `cache_backend` is not supported together with step execution.
- Unsupported pipelines fail early during model loading.
- Request-mode extras such as KV transfer are not wired into step mode yet.
- LoRA is supported in step mode, but each batch must use a single adapter:
requests with different `lora_request` or `lora_scale` are scheduled into
separate batches.
## When To Use It
Use step execution only when you specifically need the pipeline to run through
its stepwise request state machine. For normal diffusion inference, leave it
disabled unless your workflow depends on this mode.
For Qwen-Image online serving, the usual progression is:
- start with `--step-execution --max-num-seqs 1` if you only need the step-wise path
- increase `--max-num-seqs` after that if you want the experimental compatible-request batching behavior
If you are looking for general diffusion speedups, see
[Diffusion Features Overview](../diffusion_features.md).
## Troubleshooting
If model loading fails with a message mentioning `prepare_encode()`,
`denoise_step()`, `step_scheduler()`, and `post_decode()`, the selected
pipeline does not support step execution.
## For Model Authors
If you want to add step execution support to a new diffusion pipeline, see the
implementation guide:
[Diffusion Step Execution Design](../../design/feature/diffusion_step_execution.md).
If you also want that pipeline to participate in the experimental batched
step-wise path, see:
[Continuous Batching for Step-Wise Diffusion](../../design/feature/diffusion_continuous_batching.md).