anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into codex-api-server-user-guide

Browse Source 
Signed-off-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com>

# Conflicts:
#	docs/user_guide/examples/online_serving/text_to_audio.md
#	examples/online_serving/stable_audio/README.md
This commit is contained in:
hsliu_ustc committed 2026-09-01 11:23:08 +08:00
2 parents a79dcdee5c a65e89cbd4
commit 7d41ea6183
1457 files changed
+164090 -40241
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/.nav.yml
+23 -4
View File
@@ -32,7 +32,6 @@ nav:
- Text-To-Speech: user_guide/examples/offline_inference/text_to_speech.md
- Text-To-Video: user_guide/examples/offline_inference/text_to_video.md
- X-To-Text: user_guide/examples/offline_inference/x_to_text.md
- X-To-Video-Audio: user_guide/examples/offline_inference/x_to_video_audio.md
- Online Serving:
- Image-To-Image: user_guide/examples/online_serving/image_to_image.md
- Image-To-Video: user_guide/examples/online_serving/image_to_video.md
@@ -51,6 +50,7 @@ nav:
- Overview: features/README.md
- Runtime and Stage Execution:
- Execution Modes and Streaming: user_guide/diffusion/execution_modes.md
- Prefill-Decode Disaggregation (experimental): features/pd_disaggregation.md
- Sleep Mode: features/sleep_mode.md
- Quantization:
- Overview: user_guide/quantization/overview.md
@@ -61,6 +61,7 @@ nav:
- BitsAndBytes W4: user_guide/quantization/bitsandbytes.md
- MXFP8 W8A8: user_guide/quantization/mxfp8.md
- MXFP4 W4A4: user_guide/quantization/mxfp4.md
- SVDQuant W4A4: user_guide/quantization/svdquant.md
- ModelOpt: user_guide/quantization/modelopt.md
- AutoRound: user_guide/quantization/autoround.md
- msModelSlim: user_guide/quantization/msmodelslim.md
@@ -68,7 +69,11 @@ nav:
- Overview and Compatibility:
- Overview: user_guide/diffusion_features.md
- Feature Compatibility: user_guide/feature_compatibility.md
- CPU Offloading: user_guide/diffusion/cpu_offload.md
- CPU Offloading:
- Overview: user_guide/diffusion/cpu_offload.md
- Model-Level: user_guide/diffusion/offloader/cpu_offload.md
- Layerwise: user_guide/diffusion/offloader/layerwise_offload.md
- Distributed Layerwise: user_guide/diffusion/offloader/distributed_layerwise_offload.md
- Cache Acceleration:
- TeaCache: user_guide/diffusion/cache_acceleration/teacache.md
- Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md
@@ -80,7 +85,13 @@ nav:
- Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
- Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
- VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
- Attention Backends: user_guide/diffusion/attention_backends.md
- Attention Backends:
- Overview: user_guide/diffusion/attention_backends.md
- Dense Backends: user_guide/diffusion/attention_backends/dense_backends.md
- TRTLLM: user_guide/diffusion/attention_backends/trtllm.md
- SageAttention: user_guide/diffusion/attention_backends/sage.md
- Hugging Face Hub: user_guide/diffusion/attention_backends/huggingface_hub.md
- RainFusion: user_guide/diffusion/attention_backends/rainfusion.md
- Regional Compilation: user_guide/diffusion/regional_compilation.md
- Frame Interpolation: user_guide/diffusion/frame_interpolation.md
- Startup and Loading: user_guide/diffusion/startup_and_loading.md
@@ -110,10 +121,12 @@ nav:
- Feature Design:
- Runtime and Stage Execution:
- Disaggregated Inference: design/feature/disaggregated_inference.md
- Host Weight Runtime: design/feature/host_weight_runtime.md
- Async Chunk: design/feature/async_chunk.md
- Async Diffusion Output: design/feature/async_diffusion_output.md
- Async Omni Output Materialization: design/feature/omni_async_output_materialization.md
- Automatic Prefix Caching: design/feature/prefix_caching.md
- Realtime AR-Diffusion: design/feature/realtime_ar_diffusion.md
- Communication:
- OmniConnector Implementations:
- Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md
@@ -133,11 +146,16 @@ nav:
- Tensor Parallel: design/feature/tensor_parallel.md
- VAE Patch Parallelism: design/feature/vae_parallel.md
- Attention Optimization:
- Backend Selection: design/feature/attention_backend_selection.md
- Skip-Softmax: design/feature/skip_softmax.md
- CPU Offloading:
- Overview: design/feature/offloader/README.md
- Model-Level: design/feature/offloader/cpu_offload.md
- Layerwise: design/feature/offloader/layerwise_offload.md
- Distributed Layerwise: design/feature/offloader/distributed_layerwise_offload.md
- Cache-DiT: design/feature/cache_dit.md
- TeaCache: design/feature/teacache.md
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
- Distributed Layerwise Offload: design/feature/distributed_layerwise_offload.md
- Infrastructure and Performance:
- Prometheus Metrics: design/metrics.md
- Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
@@ -160,6 +178,7 @@ nav:
- Offloader: design/module/diffusion/offloader.md
- Execution Platforms: design/module/execution_platforms.md
- Cache Management: design/module/cache_management.md
- Host Weight Runtime: design/module/host_weight_runtime.md
- Quantization: design/module/quantization.md
- Observability: design/module/observability.md
- Profiling: design/module/profiling.md
docs/assets/WeChat.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 260 KiB

After

Width:  |  Height:  |  Size: 274 KiB

docs/cli/README.md
+7 -1
View File
@@ -2,6 +2,11 @@
The CLI for vLLM-Omni inherits from vllm with some additional arguments.
Environment variables are process configuration rather than CLI-only options.
See the [Environment Variables](../configuration/environment_variables.md)
reference for ownership, precedence, evaluation time, and per-stage `env`
scoping.
## serve
Starts the vLLM-Omni OpenAI Compatible API server.
@@ -39,4 +44,5 @@ vllm bench serve --omni \
--num-prompts 5
```
See [vllm bench serve](./bench/serve.md) for the full reference of all available arguments.
See [vllm bench serve](./bench/serve.md) for serving benchmark arguments and
dataset-specific examples, including OmniInteract native-duplex sessions.
docs/cli/bench/serve.md
+41 -3
View File
@@ -2,7 +2,7 @@
The vllm bench command launches the vLLM-Omni benchmark to evaluate the performance of multimodal models.
## Notes
vLLM-Omni registers the `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, and `daily-omni` serving benchmark backends.
vLLM-Omni registers the `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, `daily-omni`, and `openai-realtime-duplex` serving benchmark backends. It also adds the `omniinteract` dataset.
## Basic Parameter Description
You can use `vllm bench serve --omni --help=all` to get descriptions of all parameters. The commonly used parameters are described below:
@@ -10,7 +10,7 @@ You can use `vllm bench serve --omni --help=all` to get descriptions of all para
Enable Omni (multimodal) mode, supporting multimodal inputs and outputs such as images, videos, and audio.
- `--backend`
Specify the backend adapter. vLLM-Omni adds `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, and `daily-omni` to the upstream vLLM backend choices.
Specify the backend adapter. vLLM-Omni adds `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, `daily-omni`, and `openai-realtime-duplex` to the upstream vLLM backend choices.
- `--model`
The model identifier to load, filled according to the models supported by vLLM-Omni.
@@ -19,7 +19,7 @@ You can use `vllm bench serve --omni --help=all` to get descriptions of all para
The API endpoint exposed externally, to which clients send their requests.
- `--dataset-name`
The name of the dataset used; random-mm indicates generating random multimodal inputs (images, videos, audio).
The name of the dataset used; random-mm indicates generating random multimodal inputs (images, videos, audio), while `omniinteract` replays official OmniInteract videos.
- `--num-prompts`
The total number of requests to send, an integer.
@@ -269,6 +269,44 @@ We use audio generation time / audio duration to calculate RTF.
</details>
### OmniInteract Realtime Benchmark
OmniInteract runs each video as one native-duplex WebSocket sample in the standard serving-benchmark lifecycle:
```bash
vllm bench serve --omni \
--backend openai-realtime-duplex \
--dataset-name omniinteract \
--dataset-path /path/to/OmniInteract \
--model openbmb/MiniCPM-o-4_5 \
--base-url http://127.0.0.1:8000 \
--endpoint /v1/realtime \
--omniinteract-ref-audio /path/to/reference.wav \
--omniinteract-output-dir ./omniinteract-artifacts \
--num-prompts 3 \
--num-warmups 0
```
`--dataset-path` accepts an extracted directory, `data.tar[.gz]`, or a Hugging Face dataset ID; omitting it uses
`lucky-lance/OmniInteract`. `--num-prompts` is the total across subsets and defaults to 3 for OmniInteract; explicit `0`
selects all and oversize values use all available cases. Reference audio is required, and OmniInteract uses the
`/v1/realtime` endpoint.
Audio is replayed as 16 kHz PCM16 in 200 ms chunks and video at 1 FPS with real-time pacing. All selected media is decoded
before timing and remains in client memory for the run, so `--max-concurrency` does not limit media preparation memory; use
explicit `--num-prompts 0` only when the client has enough RAM for the full dataset. Media commands are bounded by
`--omniinteract-media-timeout-s`, and concurrency defaults to 1. Standard request-rate, warmup, result-saving, and summary
options apply. Use `--omniinteract-require-response` only for functional E2E cases; LISTEN is a valid benchmark result.
Each completed case writes `output.wav`, `wav_transcript.json`, `events.json`, `result.json`, and a final `.done` marker under
`--omniinteract-output-dir`. The root also contains `batch_summary.json` and `official_eval_manifest.jsonl`; failed cases write
`.failed.json`. Runs sharing one output root are serialized. Completion validates transport, response lifecycle, and artifacts,
not answer accuracy. Transcript timestamps are serialized playback-queue times. Clipped or cancelled outputs are ineligible and
omitted from the official manifest; `audio_clipped_bytes` records output beyond the rounded video horizon.
TTFT, TTFP, and RTF start at client receipt of `response.created`. TPOT/ITL use engine stage-0 timing; ITL is emitted only when
every token interval is present.
### Multi-Modal Benchmark
<details class="admonition abstract" markdown="1">
docs/configuration/README.md
+6 -6
View File
@@ -4,20 +4,20 @@ This section lists the most common options for running vLLM-Omni.
For options within a vLLM Engine, please refer to the [vLLM 0.26 configuration guide](https://docs.vllm.ai/en/v0.26.0/configuration/index.html).
Currently, the main options are maintained by stage configs for each model.
Each model defines fixed topology in a registered `PipelineConfig` and runtime
overrides in a deploy YAML.
For process-level settings shared by the CLI, deploy YAML, workers, and model
integrations, see [Environment Variables](./environment_variables.md).
For a specific example, see the [Qwen2.5-Omni deploy config](gh-file:vllm_omni/deploy/qwen2_5_omni.yaml). The matching frozen pipeline topology lives at [vllm_omni/model_executor/models/qwen2_5_omni/pipeline.py](gh-file:vllm_omni/model_executor/models/qwen2_5_omni/pipeline.py).
For introduction, please check [Introduction for stage config](./stage_configs.md)
For an introduction, see [Pipeline and deploy configurations](./stage_configs.md).
## Memory Configuration
- **[GPU Memory Calculation and Configuration](./gpu_memory_utilization.md)** - Guide on how to calculate memory requirements and set up `gpu_memory_utilization` for optimal performance
## Multi-Stage Recipes
- **[Prefill-Decode Disaggregation](./pd_disaggregation.md)** - How to derive a PD-aware Qwen3-Omni stage config from the default config without introducing another bundled YAML
## Optimization Features
- **[Diffusion Features Overview](../user_guide/diffusion_features.md)** - Complete overview of all diffusion model features and supported models
docs/configuration/composable_parallel.md
+2 -2
View File
@@ -1,6 +1,6 @@
# Composable parallel strategies
In vLLM-Omni, the `composable_parallel` system layers a declarative *parallel strategy* on top of the existing stage configs. A strategy YAML maps each pipeline stage (by its `model_stage` name) to a stack of axis declarations — one per parallelism dimension — that the engine translates into per-stage parallel sizing (`tensor_parallel_size`, `data_parallel_size`, `pipeline_parallel_size`, expert-parallel toggle, replica count) and, when relevant, an orchestrator-wide load-balancer policy.
In vLLM-Omni, the `composable_parallel` system layers a declarative *parallel strategy* on top of the registered pipeline and deploy config. A strategy YAML maps each pipeline stage (by its `model_stage` name) to a stack of axis declarations — one per parallelism dimension — that the engine translates into per-stage parallel sizing (`tensor_parallel_size`, `data_parallel_size`, `pipeline_parallel_size`, expert-parallel toggle, replica count) and, when relevant, an orchestrator-wide load-balancer policy.
Strategies are intentionally narrow: each one only writes the axes it declares, leaving every other engine arg untouched. This makes a strategy file a small, reusable overlay you can swap independently of the deploy YAML (see `configuration/stage_configs.md`).
@@ -195,7 +195,7 @@ Because `omni_lb_policy` is read once at `AsyncOmniEngine` construction — not
Default: no default — required for every entry under `strategies:`.
Strategy keys MUST be `model_stage` *names* (strings), the same labels stage configs use for `engine_args.model_stage` (see [`model_stage` in `stage_configs.md`](stage_configs.md#engine_argsmodel_stage)). Integer `stage_id` values are not accepted; the resolver matches on `model_stage` exactly (`_resolve_stage` in `vllm_omni/config/composable_parallel/apply.py`).
Strategy keys MUST be `model_stage` *names* (strings), the same labels defined by [`StagePipelineConfig.model_stage`](stage_configs.md#pipeline-configuration). Integer `stage_id` values are not accepted; the resolver matches on `model_stage` exactly (`_resolve_stage` in `vllm_omni/config/composable_parallel/apply.py`).
## Common pitfalls
docs/configuration/environment_variables.md
+238
View File
@@ -0,0 +1,238 @@
# Environment Variables
Environment variables are process-level configuration shared by the CLI,
deploy YAML, Python APIs, worker processes, and model integrations. They are
therefore documented under **User Guide → Configuration**, rather than under
CLI Reference.
This page describes the public vLLM-Omni environment-variable contract on the
`latest` documentation branch. Use the documentation for your installed
vLLM-Omni release when behavior differs.
!!! important "Public contract"
Only names in [Public vLLM-Omni variables](#public-vllm-omni-variables) are
public Omni configuration. A name found in source code is not automatically
supported: model-specific, benchmark, platform, and internal variables have
separate ownership and lifecycle rules.
## Naming and implementation boundary
New public variables owned by vLLM-Omni must use the `VLLM_OMNI_` prefix.
Existing public names with older prefixes remain supported for compatibility,
but they do not establish naming precedent for new settings. Inherited vLLM
and third-party names retain the prefix chosen by their owning project.
The source [environment-variable inventory](gh-file:vllm_omni/config/environment_variable_inventory.py)
records ownership, migration disposition, and redaction metadata. Unlike
`vllm.envs`, it is not an executable value resolver: existing consumers still
own parsing, defaults, precedence, and evaluation time. The public tables below
describe those live consumer contracts. Stable model or stage behavior should
move to typed configuration, and request-varying behavior should move to a
request schema instead of adding another environment variable.
## Precedence and evaluation time
There is no repository-wide rule that makes environment variables override (or
lose to) every CLI, YAML, or request option. The tables below state precedence
for each public setting. In general:
- request fields should control behavior that can vary per request;
- typed deploy or stage configuration should control stable model and stage
behavior;
- environment variables are appropriate for process launch, platform
integration, emergency diagnostics, and compatibility fallbacks.
An environment variable can be read while a module is imported, when server or
model configuration is constructed, when a worker is spawned, or whenever a
property is accessed. Set import- and startup-time variables before invoking
`vllm serve` or importing vLLM-Omni. Changing the parent shell after workers
start does not update existing worker environments.
The lifecycle labels used below are:
- **Stable** — reviewed public configuration; changes require compatibility
consideration.
- **Experimental** — public for evaluation, but its contract may still evolve.
- **Diagnostic** — supported for investigation or temporary mitigation, not as
a normal tuning interface.
- **Deprecated** — compatibility alias; migrate to the named replacement.
## Public vLLM-Omni variables
### Diffusion backend and cache selection
| Name | Type and default | Applies to and read time | Precedence and invalid values | Lifecycle |
| --- | --- | --- | --- | --- |
| `DIFFUSION_ATTENTION_BACKEND` | Backend name or `auto`; default `auto` (platform selection) | Diffusion stages; read when `OmniDiffusionConfig` is constructed | `diffusion_attention_config.default` wins. An unknown backend fails during backend resolution. | Stable fallback |
| `DIFFUSION_CACHE_BACKEND` | `none`, `cache_dit`, `tea_cache`, `mag_cache`, `step_cache`, `stepcache`, or `step_cache_dit`; default `none` | Diffusion runner startup | Explicit `cache_backend` in config wins. Otherwise this name wins over the deprecated alias. Unsupported values raise `ValueError` during runner setup. | Stable fallback |
| `DIFFUSION_CACHE_ADAPTER` | Same values as `DIFFUSION_CACHE_BACKEND`; default `none` | Diffusion runner startup | Used only when neither explicit `cache_backend` nor `DIFFUSION_CACHE_BACKEND` is set. Unsupported values raise `ValueError`. | Deprecated; use `DIFFUSION_CACHE_BACKEND` |
| `OMNI_DIFFUSION_PROMPT_EMBED_CACHE` | Boolean: `1`, `true`, `yes`, `on`, `0`, `false`, `no`, or `off`; default disabled | Each diffusion runner; resolved during model setup | A recognized environment value overrides the explicit enable setting. An unrecognized value is ignored. | Experimental |
| `OMNI_DIFFUSION_PROMPT_EMBED_CACHE_SIZE` | Positive integer; default `32` entries | Each diffusion runner; resolved during model setup | A valid environment value overrides the explicit cache size. A non-integer logs a warning; a non-positive value is ignored. | Experimental |
| `OMNI_DIFFUSION_SESSION_STATE_MANAGER` | Boolean with the same accepted spellings as the prompt cache; default disabled | Experimental diffusion session manager; model setup | A recognized environment value overrides the explicit enable setting. An unrecognized value is ignored. | Experimental |
| `OMNI_DIFFUSION_SESSION_STATE_MANAGER_MAX_SESSIONS` | Positive integer; default `64` | Experimental diffusion session manager; model setup | A valid environment value overrides the explicit maximum. A non-integer logs a warning; a non-positive value is ignored. | Experimental |
Backend names for `DIFFUSION_ATTENTION_BACKEND` are the members of
`DiffusionAttentionBackendEnum`, such as `FLASH_ATTN`, `TORCH_SDPA`,
`SAGE_ATTN`, `FLASHINFER_ATTN`, and `TRTLLM_ATTN`. Platform support still
depends on the installed kernels and model path.
### Serving and runtime
| Name | Type and default | Applies to and read time | Precedence and invalid values | Lifecycle |
| --- | --- | --- | --- | --- |
| `SPEAKER_SAMPLES_DIR` | Filesystem path; default `~/.cache/vllm-omni/speakers` | Speech server; read when speaker storage initializes | Environment-only setting. The directory is created; filesystem errors propagate. | Stable |
| `SPEAKER_MAX_UPLOADED` | Integer; default `1000` | Speech server; read when speaker storage initializes | Environment-only setting. A non-integer logs a warning and uses `1000`; range is not otherwise validated. | Stable |
| `VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT` | Float seconds; default `600` | Diffusion engine async-output wait in `step_streaming`; resolved per call on the request path, not at import | Environment-only setting. A non-float or `<=0` value warns once and uses the default. | Experimental |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; default `0` (off) | Orchestration loop and the serving-side final-output drain; read once when the `Orchestrator` is constructed | Environment-only setting. Values are stripped and case-normalized; any unrecognized value leaves the legacy poll loop selected. | Experimental |
| `VLLM_OMNI_INPUT_WAIT_TIMEOUT_S` | Float seconds; default `600`; `<=0` disables | Full-payload input coordinator, not async-chunk transfer; read when the scheduler module imports in each worker | Environment-only setting. A non-float logs a warning and uses `600`. | Stable operational control |
| `VLLM_OMNI_ORCH_MONITOR_PATH` | Filesystem path; default `<current-working-directory>/vllm_omni_orch_monitor_<timestamp>.json` | Orchestrator monitor enabled by `--enable-orch-monitor`; read when the monitor is created | Environment-only path override. Parent directories are created; write errors are logged. | Diagnostic |
| `VLLM_OMNI_VIDEO_SYNC_TIMEOUT` | Float seconds; default `600` | Synchronous Videos API; read when the API server module imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
| `VLLM_VIDEO_ASYNC_CHUNK` | `on` or `off`; default `on` | Streaming video output; read on attribute access | Environment-only setting. Values are trimmed and case-normalized; an invalid value warns once and uses `on`. | Experimental |
| `VLLM_VIDEO_AUDIO_DELTA_MODE` | `fast` or `slow`; default `fast` | Streaming video audio deltas; read on attribute access | Environment-only setting. Values are trimmed and case-normalized; an invalid value warns once and uses `fast`. | Experimental |
### Server storage
Storage names use Pydantic's nested-settings delimiter (`__`). They are read
when `ServerSettings` is initialized, normally during server import/startup.
Invalid integer values raise a Pydantic validation error.
| Name | Type and default | Meaning | Lifecycle |
| --- | --- | --- | --- |
| `VLLM_OMNI_SERVER_STORAGE__PATH` | Path; default `/tmp/storage` | Directory for completed server files. | Stable |
| `VLLM_OMNI_SERVER_STORAGE__FILE_CONCURRENCY` | Integer; default `4` | Maximum concurrent file operations. No additional range validation is currently applied. | Stable |
| `VLLM_OMNI_SERVER_STORAGE__FILE_TTL` | Integer seconds or unset; default unset | Optional lifetime for locally stored files. | Stable |
| `VLLM_OMNI_SERVER_STORAGE__TTL_SWEEP_INTERVAL` | Integer seconds or unset | Sweep frequency. If file TTL is set and this is unset, the effective default is `300`. | Stable |
| `VLLM_OMNI_STORAGE_PATH` | Path | Deprecated alias for `VLLM_OMNI_SERVER_STORAGE__PATH`. The new name wins when both are set. | Deprecated |
| `VLLM_OMNI_STORAGE_MAX_CONCURRENCY` | Integer | Deprecated alias for `VLLM_OMNI_SERVER_STORAGE__FILE_CONCURRENCY`. The new name wins when both are set. | Deprecated |
The deprecated aliases emit `DeprecationWarning`. They do not affect file TTL
settings.
### Quantization diagnostics and performance
| Name | Type and default | Applies to and read time | Precedence and invalid values | Lifecycle |
| --- | --- | --- | --- | --- |
| `VLLM_OMNI_SKIP_NVFP4_NAN_CLAMP` | Boolean truthy spellings: `1`, `true`, `yes`, `on`; default false | ModelOpt NVFP4 compatibility patch; read when `vllm_omni.patch` imports | Environment-only escape hatch. Any other value means false. Set only to diagnose the upstream NaN-scale issue. | Diagnostic and temporary |
| `VLLM_OMNI_USE_QUACK_FP8` | Boolean truthy spellings: `1`, `true`, `yes`, `on`; unset means hardware auto-detection | FP8 scaled matrix multiplication; evaluated when quack capability is selected | A set value overrides auto-detection. Any non-truthy value forces quack off. If quack cannot load, vLLM-Omni warns and falls back to FlashInfer. | Experimental performance control |
`QUACK_CACHE_DIR` is owned by the external quack library and is not an
Omni-owned variable, even though vLLM-Omni supplies a persistent default when
it is unset.
## Per-stage environment
Deploy configurations can set arbitrary environment keys for one stage:
```yaml
stages:
- stage_id: 0
env:
HCCL_IF_BASE_PORT: 23000
MY_MODEL_CACHE: /mnt/model-cache
```
The legacy nested form under `runtime.env` is also accepted. vLLM-Omni converts
keys and values to strings, applies them while the stage engine or child process
is launched, logs keys but not values, and restores the parent process's prior
environment afterward. A child process inherits the scoped values. This keeps
sibling stages from permanently changing one another's parent environment.
This scoping has two consequences:
1. A setting read inside stage/model initialization can see the stage value.
2. A setting already cached by an import that occurred before stage launch
cannot be changed retroactively with stage `env`.
Stage `env` is an escape hatch for process and third-party settings, not a
replacement for typed stage configuration. Do not put request-varying behavior
there. Secret values may reach child processes; logs and diagnostics must emit
their keys only.
## Inherited vLLM variables
vLLM-Omni also reads variables through its aligned vLLM dependency. Refer to
the [vLLM 0.26 environment-variable reference](https://docs.vllm.ai/en/v0.26.0/configuration/env_vars.html)
for their definitions. This includes vLLM launch, cache, logging, plugin, ROCm,
XPU, ModelScope, and FlashInfer workspace settings.
An inherited name does not become Omni-owned because Omni reads or forwards it.
Where behavior differs, the Omni-specific page should document only that
difference. For example, device-visibility values can be further narrowed by a
stage's `devices` setting.
## Platform, external, and secret variables
Hardware runtimes, distributed launchers, and third-party libraries retain
ownership of their variables. Common groups include:
- device visibility (`ASCEND_RT_VISIBLE_DEVICES`, `HIP_VISIBLE_DEVICES`, and
`MUSA_VISIBLE_DEVICES`);
- distributed launch (`MASTER_ADDR`, `MASTER_PORT`, `RANK`, and `WORLD_SIZE`);
- Hugging Face cache and authentication (`HF_HOME`, `HF_TOKEN`, and related
names);
- accelerator libraries such as NCCL, FlashInfer, MindIE, MORI, and quack.
Consult the owning project's documentation for accepted values and support.
`HF_TOKEN`, `HUGGINGFACE_HUB_TOKEN`, and `OPENAI_API_KEY` are explicitly marked
for redaction in the reviewed inventory. Never print their values in environment
collection, examples, bug reports, or logs.
## Model-specific variables
The audit found 56 variables read by a single model or pipeline family. They are
not listed as public usage options here because doing so would turn implementation
escape hatches into an accidental compatibility contract.
Every audited model-specific name has a migration disposition in the
[environment-variable inventory](gh-file:vllm_omni/config/environment_variable_inventory.py):
| Disposition | Count | Required outcome |
| --- | ---: | --- |
| Promote | 33 | Move a stable setting into typed stage or model configuration. |
| Request scope | 6 | Move request-varying behavior into a declared request-option schema. |
| External | 0 | Retain only when a supported third-party library owns the contract. |
| Internalize | 11 | Keep a debug or diagnostic switch out of public documentation and configuration. |
| Deprecate/remove | 6 | Remove a compatibility escape hatch that has no continuing contract. |
The disposition is a migration target, not a statement that the existing
environment switch is stable. Promote or request-scope work should land in
model-owner-reviewed follow-up changes with validation and model-page updates.
The family-sized migration packages and their ownership gate are tracked in
[#6232](https://github.com/vllm-project/vllm-omni/issues/6232).
## Environment collection and troubleshooting
Run the repository collector when preparing a bug report:
```bash
python collect_env.py
```
The environment section includes the safe public Omni allowlist, registered
vLLM variables, and selected platform prefixes. It omits names marked for
redaction and names containing common secret terms. Review the output before
sharing it. A key supplied through stage `env` is not collected merely because
it appears in deploy configuration; the same allowlist and prefix rules apply.
If a setting appears ineffective, first check its read time in the tables
above. Restart processes after changing import- or startup-time values, verify
that the variable reaches the intended worker, and check whether an explicit
typed configuration field takes precedence.
## Benchmark and internal variables
The reviewed inventory marks 20 benchmark variables as transitional while
benchmark configuration moves to CLI/request schemas in
[issue #5376](https://github.com/vllm-project/vllm-omni/issues/5376). They are not
part of the serving/runtime contract.
Internal process-coordination and debug variables are deliberately excluded
from this page. The CI scanner resolves string literals and module-level string
constants used by direct `os` access, local wrapper helpers, aliased `os`
imports, and environment membership checks. Server-storage names generated by
Pydantic settings are checked separately against their model fields.
Names assembled from cross-module metadata and arbitrary keys supplied through
stage `env` cannot be inferred reliably. Contributors must classify those names
explicitly. CI fails when a statically resolved name has no classification or
when a reviewed public name is absent from this page.
docs/configuration/gpu_memory_utilization.md
+24 -32
View File
@@ -80,19 +80,15 @@ When multiple stages share the same GPU, you must ensure the sum of their `gpu_m
**Example: Two stages on GPU 0**
```yaml
stage_args:
stages:
- stage_id: 0
runtime:
devices: "0"
engine_args:
gpu_memory_utilization: 0.6 # Uses 60% of GPU 0
devices: "0"
gpu_memory_utilization: 0.6 # Uses 60% of GPU 0
- stage_id: 1
runtime:
devices: "0"
engine_args:
gpu_memory_utilization: 0.3 # Uses 30% of GPU 0
# Total: 90% of GPU 0 (safe, leaves 10% buffer)
devices: "0"
gpu_memory_utilization: 0.3 # Uses 30% of GPU 0
# Total: 90% of GPU 0 (safe, leaves 10% buffer)
```
**Important:** If stages run on different GPUs, each can use up to 1.0 independently.
@@ -103,14 +99,12 @@ When using `tensor_parallel_size > 1`, the model is split across multiple GPUs,
**Example: 2-way tensor parallelism**
```yaml
stage_args:
stages:
- stage_id: 0
runtime:
devices: "0,1" # Uses both GPUs
engine_args:
tensor_parallel_size: 2
gpu_memory_utilization: 0.6 # 60% per GPU
# Model is split, so each GPU uses ~30% of model memory
devices: "0,1" # Uses both GPUs
tensor_parallel_size: 2
gpu_memory_utilization: 0.6 # 60% per GPU
# Model is split, so each GPU uses ~30% of model memory
```
## Examples
@@ -118,27 +112,25 @@ stage_args:
### Qwen3-Omni-MoE on 2x H100-80GB
```yaml
stage_args:
base_config: /path/to/vllm_omni/deploy/qwen3_omni_moe.yaml
stages:
- stage_id: 0 # Thinker stage with TP=2
runtime:
devices: "0,1"
engine_args:
tensor_parallel_size: 2
gpu_memory_utilization: 0.6 # 48GB per GPU
devices: "0,1"
tensor_parallel_size: 2
gpu_memory_utilization: 0.6 # 48GB per GPU
- stage_id: 1 # Talker stage
runtime:
devices: "1"
engine_args:
gpu_memory_utilization: 0.3 # 24GB on GPU 1
devices: "1"
gpu_memory_utilization: 0.3 # 24GB on GPU 1
- stage_id: 2 # Code2Wav stage
runtime:
devices: "0"
engine_args:
gpu_memory_utilization: 0.1 # 8GB on GPU 0
devices: "0"
gpu_memory_utilization: 0.1 # 8GB on GPU 0
```
**Note:** In this configuration, stages 0 and 2 share GPU 0, but they run at different times in the pipeline, so their memory usage doesn't overlap.
**Note:** Stage 0 uses GPUs 0 and 1, so the combined utilization is `0.7` on
GPU 0 and `0.9` on GPU 1. Keep the sum of all resident stages below `1.0` on
each device.
## Troubleshooting
docs/configuration/pd_disaggregation.md
-171
View File
@@ -1,171 +0,0 @@
# Prefill-Decode (PD) Disaggregation
PD disaggregation splits the Qwen3-Omni thinker into separate prefill and decode
stages so prompt processing and token generation can run on different workers.
This is documented as a stage-config recipe instead of a bundled YAML because the
deployment-specific values usually change per environment:
- GPU placement
- `tensor_parallel_size`
- connector backend and connector ports
- connector IPs or bootstrap addresses
Start from the [default Qwen3-Omni stage config](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml)
and copy it to your own file, for example `qwen3_omni_pd.yaml`. Then apply the
changes below.
## Requirements
- 3+ GPUs for a basic layout: prefill, decode, and talker+code2wav
- A KV connector supported by vLLM, such as `MooncakeConnector`
- Matching `tensor_parallel_size` on the prefill and decode thinker stages
## 1. Split the thinker into prefill and decode stages
Replace the original thinker stage with two stages:
```yaml
stage_args:
- stage_id: 0
stage_type: llm
is_prefill_only: true
runtime:
devices: "0"
engine_args:
max_num_seqs: 16
model_stage: thinker
model_arch: Qwen3OmniMoeForConditionalGeneration
worker_type: ar
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
gpu_memory_utilization: 0.9
enforce_eager: true
trust_remote_code: true
engine_output_type: latent
distributed_executor_backend: "mp"
enable_prefix_caching: false
max_num_batched_tokens: 32768
hf_config_name: thinker_config
tensor_parallel_size: 1
kv_transfer_config:
kv_connector: "MooncakeConnector"
kv_role: "kv_producer"
kv_rank: 0
kv_parallel_size: 2
kv_connector_extra_config:
mooncake_bootstrap_port: 25201
final_output: false
is_comprehension: true
default_sampling_params:
temperature: 0.4
top_p: 0.9
top_k: 1
max_tokens: 2048
seed: 42
detokenize: True
repetition_penalty: 1.05
- stage_id: 1
stage_type: llm
is_decode_only: true
runtime:
devices: "1"
engine_args:
max_num_seqs: 64
model_stage: thinker
model_arch: Qwen3OmniMoeForConditionalGeneration
worker_type: ar
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
gpu_memory_utilization: 0.9
enforce_eager: true
trust_remote_code: true
engine_output_type: latent
distributed_executor_backend: "mp"
enable_prefix_caching: false
max_num_batched_tokens: 32768
hf_config_name: thinker_config
tensor_parallel_size: 1
kv_transfer_config:
kv_connector: "MooncakeConnector"
kv_role: "kv_consumer"
kv_rank: 1
kv_parallel_size: 2
kv_connector_extra_config:
mooncake_bootstrap_port: 25202
engine_input_source: [0]
final_output: true
final_output_type: text
is_comprehension: true
default_sampling_params:
temperature: 0.4
top_p: 0.9
top_k: 1
max_tokens: 2048
seed: 42
detokenize: True
repetition_penalty: 1.05
```
Notes:
- `is_prefill_only: true` marks the thinker stage that only saves KV.
- `is_decode_only: true` marks the thinker stage that resumes from remote KV.
- `kv_transfer_config` is required on both stages.
- The orchestrator forces the prefill stage to run with `max_tokens=1`, so the
prefill side only processes the prompt and exports KV.
## 2. Shift the downstream stages by one index
After inserting the extra thinker stage, renumber the remaining stages:
```yaml
- stage_id: 2
runtime:
devices: "2"
engine_input_source: [1]
sync_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_token_only
- stage_id: 3
runtime:
devices: "2"
engine_args:
max_num_seqs: 1
engine_input_source: [2]
sync_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_token_only
```
Compared with the default Qwen3-Omni config:
- the talker becomes stage `2` instead of stage `1`
- the code2wav stage becomes stage `3` instead of stage `2`
- the talker now reads from decode stage `1`
## 3. Add runtime edges for the four-stage pipeline
```yaml
runtime:
enabled: true
edges:
- from: 0
to: 1
- from: 1
to: 2
- from: 2
to: 3
```
## 4. Launch with your custom config
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--deploy-config /path/to/qwen3_omni_pd.yaml
```
## Operational Notes
- `MooncakeConnector` does not support heterogeneous TP sizes across the PD
pair. Keep prefill and decode at the same `tensor_parallel_size`.
- If the thinker requires TP=2, both thinker stages must use TP=2 and be given
separate GPU sets, for example `"0,1"` for prefill and `"2,3"` for decode.
- Choose connector ports and addresses that match your deployment. The values
shown above are examples only.
docs/configuration/stage_configs.md
+80 -311
View File
@@ -3,52 +3,104 @@
In vLLM-Omni, a model's `PipelineConfig` defines its fixed stage topology, while a deploy configuration controls how those stages run.
!!! note
Default deploy config YAMLs (for example, `vllm_omni/deploy/qwen2_5_omni.yaml`, `vllm_omni/deploy/qwen3_omni_moe.yaml`, and `vllm_omni/deploy/qwen3_tts.yaml`) are bundled and loaded automatically when `--deploy-config` is omitted. The model registry resolves the right pipeline and deploy YAML by `model_type`.
Default deploy config YAMLs (for example, `vllm_omni/deploy/qwen2_5_omni.yaml`, `vllm_omni/deploy/qwen3_omni_moe.yaml`, and `vllm_omni/deploy/qwen3_tts.yaml`) are bundled and loaded automatically when `--deploy-config` is omitted. The resolved pipeline selects its default through `default_deploy_config_name`.
## Pipeline configuration
`PipelineConfig` and its `StagePipelineConfig` entries are Python definitions
owned by the model implementation and registered in `OMNI_PIPELINES`. They
describe fixed model topology and are not accepted in deploy YAMLs.
Common `PipelineConfig` fields include:
| Field | Description |
| ------- | ------------- |
| `model_type` | Pipeline identifier used during model and config resolution. |
| `default_deploy_config_name` | Bundled deploy YAML loaded when the user does not pass `deploy_config`. |
| `model_arch` | Default Hugging Face architecture for the pipeline. |
| `hf_architectures` | Architecture names used to identify checkpoints whose `model_type` is shared. |
| `hf_config_predicate` | Optional predicate used to select between pipelines with otherwise identical HF metadata. |
| `diffusers_class_name` | Diffusers `_class_name` used to identify Diffusers-style repositories. |
| `stages` | Ordered tuple of fixed `StagePipelineConfig` definitions. |
Common `StagePipelineConfig` fields include:
| Field | Description |
| ------- | ------------- |
| `stage_id` | Stable stage identifier. |
| `model_stage` | Logical stage name used by runtime and strategy resolution. |
| `execution_type` | `LLM_AR`, `LLM_GENERATION`, or `DIFFUSION`. |
| `input_sources` | Upstream stage IDs that provide this stage's inputs. |
| `final_output` / `final_output_type` | Whether the stage produces a user-visible output and its modality. |
| `owns_tokenizer` | Whether this stage owns the pipeline tokenizer. |
| `model_arch`, `hf_config_name` | Stage-specific model architecture and nested HF config selector. |
| `engine_output_type` | Runtime output representation such as `text`, `latent`, or `audio`. |
| `custom_process_input_func` | Processor applied to this stage's incoming payload. |
| `custom_process_next_stage_input_func` | Processor used for full-payload handoff to the next stage. |
| `async_chunk_process_next_stage_input_func` | Processor used for async chunk handoff. |
| `sampling_constraints` | Model-owned sampling constraints that deploy defaults cannot override. |
To add or change topology, define and register a new pipeline variant. Use
deploy YAML only for runtime placement, resource sizing, connectors, and other
deployment overrides.
## Deploy configuration schema
The new deploy schema lives under `vllm_omni/deploy/` and is paired with a frozen `PipelineConfig` registered by the model's `pipeline.py`. Each deploy YAML has these top-level fields:
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| ------- | ------ | ---------- | --------- | ------------- |
| `base_config` | str (path) | optional | — | Overlay parent (relative or absolute). `stages:` / `platforms:` deep-merged by stage_id; other scalars overlay-wins. Intended for user-authored overlays; prod yamls stay flat. |
| `async_chunk` | bool | optional | `true` | Enable chunked streaming between stages. Pin to `false` if the pipeline runs end-to-end. |
| `session_mode` | str | optional | `"turn"` | Session behavior. MiniCPM-o 4.5 deploy YAMLs set `"duplex"` so `/v1/realtime` and chat share the same profile. |
| `active_stream_window` | int | optional | `0` | Number of active downstream stream slots; `0` preserves all-stream cycling. |
| `duplex_session` | dict | optional | runtime defaults | Full-duplex session lifecycle, buffering, replay, and capacity limits. |
| `connectors` | dict | optional | `null` | Named connector specs (`{name, extra}`). Referenced by each stage's `input_connectors` / `output_connectors`. See [Connector schema](#connector-schema). |
| `edges` | list | optional | `null` | Explicit edge list for the KV transfer graph. Auto-derived from stage inputs if omitted. |
| `stages` | list | required | — | Per-stage engine args + wiring (see [Stage fields](#stage-fields)). |
| `stages` | list | optional | `[]` | Per-stage runtime overrides matched by `stage_id`. Pipeline stages are still created from `PipelineConfig` when this list is empty. |
| `platforms` | dict | optional | `null` | Keyed by `npu` / `rocm` / `xpu`, each contains a `stages:` list with per-platform overrides applied on top of the CUDA defaults. |
| `pipeline` | str | optional | `null` | Override the auto-detected pipeline registry key (used for structural variants like `qwen2_5_omni_thinker_only`). |
| `trust_remote_code` | bool | optional | `true` | **Pipeline-wide.** Trust HF remote code on model load; applies to every stage. |
| `pipeline` | str | optional | `null` | Override the auto-detected pipeline registry key (used for structural variants like `qwen2_5_omni_thinker_only` / `qwen3_omni_moe_thinker_only`). |
| `trust_remote_code` | bool \| null | optional | `null` | **Pipeline-wide.** Trust HF remote code on model load; applies to every stage when specified. |
| `distributed_executor_backend` | str \| null | optional | `null` | **Pipeline-wide.** Distributed executor backend forwarded to vLLM (`"mp"`, `"ray"`, `"external_launcher"`). If omitted, vLLM auto-selects backend from runtime topology. |
| `dtype` | str \| null | optional | `null` | **Pipeline-wide.** Model dtype for every stage. |
| `quantization` | str \| null | optional | `null` | **Pipeline-wide.** Quantization method for every stage. |
| `enable_prefix_caching` | bool | optional | `false` | **Pipeline-wide.** Prefix cache toggle applied to every stage. |
| `enable_prefix_caching` | bool \| null | optional | `null` | **Pipeline-wide.** Prefix cache toggle applied to every stage when specified. |
| `enable_chunked_prefill` | bool \| null | optional | `null` | **Pipeline-wide.** Chunked prefill toggle applied to every stage. |
| `data_parallel_size` | int | optional | `1` | **Pipeline-wide.** DP degree for every stage. |
| `pipeline_parallel_size` | int | optional | `1` | **Pipeline-wide.** PP degree for every stage. |
| `data_parallel_size` | int \| null | optional | `null` | **Pipeline-wide.** DP degree for every stage. |
| `pipeline_parallel_size` | int \| null | optional | `null` | **Pipeline-wide.** PP degree for every stage. |
| `custom_voice_dir` | str \| null | optional | `null` | **Pipeline-wide.** Directory containing custom voice profiles for supported TTS models. |
Note: for diffusion path, `distributed_executor_backend` currently defaults to
`mp`, and `ray` / `external_launcher` are not fully supported yet.
For fields whose deploy default is `null`, the deploy layer contributes no
override. The effective value may still come from a platform section, an
explicit CLI or stage override, or the downstream vLLM engine default.
Note: for the diffusion path, an omitted `distributed_executor_backend` selects
`uni` on a single GPU (in-process worker, no MessageQueue / `/dev/shm` output
segments) and `mp` when `num_gpus > 1`. Set `mp` explicitly to keep a worker
subprocess on one GPU. `ray` / `external_launcher` are not fully supported yet.
### Stage fields
Each entry under `stages:` accepts any `StageDeployConfig` field directly (no nested `engine_args:`). Only fields whose value legitimately varies across stages live here; pipeline-wide settings (trust_remote_code, distributed_executor_backend, dtype, quantization, prefix/chunked prefill, DP/PP sizes) are declared at the top level and applied to every stage. Unknown keys fall through to `engine_extras:` and are forwarded to the engine.
Each entry under `stages:` accepts any `StageDeployConfig` field directly (no nested `engine_args:`). Only fields whose value legitimately varies across stages live here; pipeline-wide settings (trust_remote_code, distributed_executor_backend, dtype, quantization, prefix/chunked prefill, DP/PP sizes) are declared at the top level and applied to every stage. Unknown keys fall through to `engine_extras:` and are forwarded to the engine. Frequently used fields are listed below; the source-of-truth schema is `StageDeployConfig` in `vllm_omni/config/stage_config.py`.
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| ------- | ------ | ---------- | --------- | ------------- |
| `stage_id` | int | required | — | Stage identity; matched against `PipelineConfig.stages[*].stage_id`. |
| `max_num_seqs` | int | optional | `64` | Max concurrent sequences per stage. |
| `gpu_memory_utilization` | float | optional | `0.9` | Per-stage memory budget. |
| `tensor_parallel_size` | int | optional | `1` | TP degree for this stage. |
| `enforce_eager` | bool | optional | `false` | Disable CUDA graphs. |
| `max_num_batched_tokens` | int | optional | `32768` | Prefill budget. |
| `max_model_len` | int \| null | optional | `null` | Per-stage context length (auto-sets `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1` when larger than HF default). |
| `max_num_seqs` | int \| null | optional | `null` | Max concurrent sequences per stage. |
| `gpu_memory_utilization` | float \| null | optional | `null` | Per-stage total memory target; used for automatic KV-cache sizing. |
| `kv_cache_memory_bytes` | int \| null | optional | `null` | Explicit per-rank KV-cache budget in bytes; overrides automatic sizing. |
| `tensor_parallel_size` | int \| null | optional | `null` | TP degree for this stage. |
| `enforce_eager` | bool \| null | optional | `null` | Disable CUDA graphs. |
| `max_num_batched_tokens` | int \| null | optional | `null` | Per-stage prefill/token budget; also contributes to the native maximum in-flight token limit. |
| `max_model_len` | int \| null | optional | `null` | Per-sequence context or KV length; `-1` enables native cache-capacity auto-fitting, while values above the HF default auto-set `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`. |
| `async_scheduling` | bool \| null | optional | `null` | Per-stage async scheduling toggle. |
| `devices` | str | optional | `"0"` | `CUDA_VISIBLE_DEVICES`-style device list. |
| `devices` | str \| null | optional | `null` | Device list assigned to this stage. The number of device ids must equal this stage's local world size (`tensor_parallel_size` × local data-parallel size × `pipeline_parallel_size`, or `num_replicas` × that product for a replica pool); a mismatch fails early — see the note below. |
| `output_connectors` | dict \| null | optional | `null` | Keyed by `to_stage_<n>`; values are names registered under top-level `connectors:`. |
| `input_connectors` | dict \| null | optional | `null` | Keyed by `from_stage_<n>`; values are names registered under top-level `connectors:`. |
| `default_sampling_params` | dict \| null | optional | `null` | Baseline sampling params. Deep-merged with pipeline `sampling_constraints` (pipeline wins). |
| `engine_extras` | dict | optional | `{}` | Catch-all for keys not listed above; deep-merged across overlays. Also carries per-stage overrides of pipeline-wide settings (e.g. stage-specific `dtype`). |
| `engine_extras` | dict | optional | `{}` | Catch-all for engine fields not listed above; deep-merged across overlays and forwarded to the stage engine. |
**Note:** a stage's `devices` count must equal its local world size (`tensor_parallel_size` × `data_parallel_size_local` × `pipeline_parallel_size`, falling back to global `data_parallel_size` when the local size is unset), or `num_replicas` × that product for a replica pool. A mismatch fails early and names the offending stage. A top-level `--tensor-parallel-size` is broadcast to every stage, so it can make a single-GPU stage violate this contract ([issue #5003](gh-issue:5003)); fix that case with `--stage-overrides` (set `tensor_parallel_size` and `devices` together per stage) or set TP only on the multi-GPU stage.
### Connector schema
@@ -60,12 +112,12 @@ connectors:
name: <ConnectorClassName> # required — class registered in vllm_omni.distributed
extra: # optional — forwarded to the connector's __init__
<key>: <value>
...
# Additional connector-specific options
```
| Connector class | Use case | `extra` keys |
|-----------------|----------|--------------|
| `SharedMemoryConnector` | Same-host KV transfer between stages (default for bundled YAMLs). | None. All payloads use shared memory. |
| Connector class | Use case | `extra` keys |
|--------------------------|-----------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|
| `SharedMemoryConnector` | Same-host KV transfer between stages (default for bundled YAMLs). | None. All payloads use shared memory. |
| `MooncakeStoreConnector` | Cross-host KV transfer over TCP. Required for multi-node deployments. | `host`, `metadata_server`, `master`, `segment` (int bytes), `localbuf` (int bytes), `proto` (`"tcp"` / `"rdma"`). |
A stage references a connector by name in its `input_connectors` / `output_connectors`:
@@ -85,7 +137,7 @@ stages:
### CLI flags
| Flag | Description |
|------|-------------|
| ------ | ------------- |
| `--deploy-config PATH` | Load a deploy YAML. **Optional** — when omitted, the bundled `vllm_omni/deploy/<model_type>.yaml` is auto-loaded by the model registry. |
| `--stage-overrides JSON` | Per-stage JSON overrides, e.g. `'{"0":{"gpu_memory_utilization":0.5}}'`. Per-stage values always win over global flags. |
| `--async-chunk` / `--no-async-chunk` | Flip the deploy YAML's `async_chunk:` bool. Unset (default) leaves the YAML value in force. |
@@ -199,7 +251,7 @@ CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
Effective config per stage after the merge:
| Stage | Field | Final value | Source |
|-------|-------|-------------|--------|
| ------- | ------- | ------------- | -------- |
| 0 | `gpu_memory_utilization` | `0.9` | base YAML (overlay didn't touch stage 0) |
| 0 | `max_num_seqs` | `8` | per-stage CLI (`--stage-overrides`) — wins over base `32` |
| 0 | `max_model_len` | `16384` | global CLI |
@@ -223,300 +275,17 @@ when loading a deploy configuration.
Examples:
For offline inference (assuming the necessary dependencies have been imported):
```python
model_name = "Qwen/Qwen2.5-Omni-7B"
omni = Omni(model=model_name, deploy_config="/path/to/deploy_config.yaml")
```
For online serving:
```bash
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/deploy_config.yaml
```
!!! important
We are actively iterating on the definition of deployment configurations, and we welcome feedback from users and developers.
## Legacy stage configuration reference
The remainder of this page documents the deprecated `stage_args` schema for models that have not migrated to deploy configurations. The following historical Qwen2.5-Omni configuration illustrates that schema; current Qwen2.5-Omni deployments should use `vllm_omni/deploy/qwen2_5_omni.yaml` instead.
```python
# Historical stage configuration for Qwen2.5-Omni.
stage_args:
- stage_id: 0 # mark the unique id for each stage
runtime: # The disaggregated configuration
process: true # Run this stage in a separate process
devices: "0" # Logical device index for this stage (mapped through CUDA_VISIBLE_DEVICES / ASCEND_RT_VISIBLE_DEVICES if set)
engine_args: # Engine arguments for a certain engine
model_stage: thinker
max_num_seqs: 1
model_arch: Qwen2_5OmniForConditionalGeneration # The model implementation registered in model_executor/models/registry.py
worker_type: ar # The specific worker used
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler # The specific scehduler used
gpu_memory_utilization: 0.8 # The gpu memory allocation for the stage within a single chip
enforce_eager: true # Now we only support eager mode
trust_remote_code: true # Needed by huggingface config parsing
engine_output_type: latent # It claims that the stage will input latent hiddenstates besides token ids
enable_prefix_caching: false # For request with hiddenstates output, the prefix caching is not supported now
is_comprehension: true # If the stage is a text or multimodal comprehension module. If it is, the AsyncOmni will use its tokenizer as default
final_output: true # If the stage has output as part of final outputs. If it is false, which means that the stage only works as a intermediate role.
final_output_type: text # What is the final output type. It can be text and audio now.
default_sampling_params: # sampling parameters for the stage. Their meaning aligns with vLLM.
temperature: 0.0
top_p: 1.0
top_k: -1
max_tokens: 2048
seed: 42
detokenize: True
repetition_penalty: 1.1
- stage_id: 1
runtime:
process: true
devices: "1"
engine_args:
model_stage: talker
max_num_seqs: 3
model_arch: Qwen2_5OmniForConditionalGeneration
worker_type: ar
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
gpu_memory_utilization: 0.8
enforce_eager: true
trust_remote_code: true
enable_prefix_caching: false
engine_output_type: latent
engine_input_source: [0]
default_sampling_params:
temperature: 0.9
top_p: 0.8
top_k: 40
max_tokens: 2048
seed: 42
detokenize: True
repetition_penalty: 1.05
stop_token_ids: [8294]
- stage_id: 2
runtime:
process: true
devices: "0" # Example: use a different GPU than the previous stage; use "0" if single GPU
engine_args:
model_stage: code2wav
max_num_seqs: 1
model_arch: Qwen2_5OmniForConditionalGeneration
worker_type: generation
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
gpu_memory_utilization: 0.15
enforce_eager: true
trust_remote_code: true
enable_prefix_caching: false
engine_output_type: audio
engine_input_source: [1]
final_output: true
final_output_type: audio
default_sampling_params:
temperature: 0.0
top_p: 1.0
top_k: -1
max_tokens: 2048
seed: 42
detokenize: True
repetition_penalty: 1.1
# Top-level runtime config (concise): default windows and stage edges
runtime:
enabled: true
edges:
- from: 0 # thinker → talker: trigger only after receiving full input (-1)
to: 1
- from: 1 # talker → code2wav: trigger only after receiving full input (-1)
to: 2
```
## Legacy stage configuration arguments
Each stage in the `stage_args` list contains the following configuration options:
### `stage_id`
A unique identifier for each stage in the multi-stage pipeline. Stages are numbered sequentially starting from 0, and this ID is used to reference stages in inter-stage dependencies (e.g., `engine_input_source`).
### `prompt_expand_func` (Optional)
A custom Python function hook for the LLM stage (Stage 0) that expands a single incoming prompt object into multiple prompts. This is primarily used for multi-modal Classifier-Free Guidance (CFG), where it generates the necessary companion requests (like a negative text prompt) and tags them with internal roles (e.g., `cfg_text`). This ensures the upstream LLM generates the needed contextual hidden states for both the conditional and unconditional generations simultaneously.
### `cfg_kv_collect_func` (Optional)
A custom Python function hook for downstream diffusion stages (Stage 1+) to collect, map, and process the KV caches transferred from the companion requests fired by `prompt_expand_func`. It aggregates the hidden condition states cleanly (e.g., binding them as `cfg_text_past_key_values` and `cfg_text_kv_metadata`), allowing the diffusion runtime to perform CFG smoothly without redundantly evaluating text paths on the DiT workers.
### `runtime`
Configuration for disaggregated execution of the stage, controlling how the stage is deployed and executed.
#### `runtime.process`
Whether to run this stage in a separate process. When set to `true`, the stage will be executed in an isolated process, enabling better resource isolation and parallel execution across different stages. This is essential for multi-GPU deployments where different stages run on different devices.
Default: `true`
#### `runtime.devices`
Logical device indices for this stage, specified as a string. Values are **logical indices** (`0`, `1`, `2`, ...) — not physical GPU IDs — and are mapped through the platform's visibility env var (`CUDA_VISIBLE_DEVICES` on CUDA, `ASCEND_RT_VISIBLE_DEVICES` on NPU) before being applied via `torch.cuda.set_device()` (or the equivalent).
Example: if `CUDA_VISIBLE_DEVICES=0,2,4` is set in the environment, then `devices: "0"` selects physical GPU 0 (the first visible), `devices: "1"` selects physical GPU 2, and `devices: "0,1"` makes physical GPUs 0 and 2 available to the stage. If no visibility env var is set, logical and physical IDs coincide.
Default: `"0"`
#### `engine_args.max_num_seqs`
The maximum number of sequences for concurrent processing in this stage. For
LLM stages, this controls the vLLM scheduler's maximum concurrent sequences.
For diffusion stages, this controls scheduler wave capacity in both request
batch mode and step batch mode. `max_num_seqs: 1` is the serial/conservative
path; values above `1` allow compatible requests to batch when the selected
diffusion pipeline supports that batching mode.
Default: `1`
#### `engine_args.request_batch_max_wait_ms`
The maximum time, in milliseconds, that a diffusion request-mode stage may wait
before the first `schedule()` of a new scheduler wave so compatible requests can
accumulate for request-level batching. This only applies to diffusion pipelines
that support request-level batching with `step_execution` disabled.
Use this together with `max_num_seqs > 1` for bursty serving traffic. `0`
disables admission waiting and preserves the lowest first-request latency.
For diffusion execution and batching tuning, see
[Diffusion Execution Modes](../user_guide/diffusion/execution_modes.md).
Default: `0.0`
### `engine_args`
Engine arguments for configuring the LLM engine, diffusion engine, or other engine types used by this stage.
#### `engine_args.model_stage`
The name identifier for this model stage within the multi-stage architecture. This is used internally to distinguish different stages of the same model (e.g., "thinker", "talker", "code2wav" in Qwen2.5-Omni).
#### `engine_args.model_arch`
The model architecture class name that is registered in `model_executor/models/registry.py`. This specifies which model implementation to use for this stage. The class must be registered in the model registry for vLLM-Omni to locate and instantiate it.
#### `engine_args.worker_cls`
The specific worker class to use for this stage. This determines how the model computations are executed. Examples include `vllm_omni.worker.gpu_ar_worker.GPUARWorker` for autoregressive stages and `vllm_omni.worker.gpu_generation_worker.GPUGenerationWorker` for diffusion-based stages.
#### `engine_args.scheduler_cls`
The scheduler class to use for this stage. The scheduler manages request queuing, batching, and execution order. Examples include `vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler` for standard stages and `vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler` for diffusion stages.
#### `engine_args.gpu_memory_utilization`
The fraction of GPU memory to allocate for this stage within a single GPU chip. This is a value between 0.0 and 1.0, where 0.8 means 80% of the GPU memory will be used by this stage. This allows fine-grained control over memory allocation when multiple stages share the same GPU or when reserving memory for other operations.
Default: `0.8`
!!! tip "Memory Configuration Guide"
For detailed information on how to calculate memory requirements and properly configure `gpu_memory_utilization`, see the [GPU Memory Calculation and Configuration Guide](./gpu_memory_utilization.md).
#### `engine_args.enforce_eager`
Whether to enforce eager execution mode. When set to `true`, the engine will run in eager mode without using CUDA graphs or other compilation optimizations. Currently, vLLM-Omni only supports eager mode.
Default: `true`
#### `engine_args.trust_remote_code`
Whether to trust remote code when loading models from Hugging Face. This is required for models that use custom code in their configuration files. Set to `true` when loading models that require custom model implementations.
Default: `true`
#### `engine_args.engine_output_type`
Specifies the type of output produced by this stage's engine. This determines what kind of data flows to downstream stages. Possible values include `latent` (hidden states), `text` (tokenized text), and `audio` (audio waveforms). When set to `latent`, the stage outputs latent hidden states in addition to token IDs, which are consumed by downstream stages.
Default: `latent`
#### `engine_args.enable_prefix_caching`
Whether to enable prefix caching for this stage. Prefix caching can improve performance by caching KV cache for common prompt prefixes. However, for requests that output hidden states (when `engine_output_type` is `latent`), prefix caching is not currently supported and should be set to `false`.
Default: `false`
### `is_comprehension`
Whether this stage is a text or multimodal comprehension module. When set to `true`, the stage acts as a comprehension module that processes input text or multimodal content. If this is the first comprehension stage, `AsyncOmni` will use its tokenizer as the default tokenizer for the entire pipeline.
Default: `true`
### `final_output`
Whether this stage produces output that is part of the final outputs returned to the user. When set to `false`, the stage only works as an intermediate stage, processing data that flows to downstream stages but not contributing directly to the final response.
Default: `true`
### `final_output_type`
The type of final output produced by this stage. This specifies what format the output will be in when returned to the user. Currently supported values are `text` (for text generation) and `audio` (for audio generation).
Default: `text`
### `default_sampling_params`
Default sampling parameters for this stage. These parameters control the generation behavior and align with vLLM's sampling parameter semantics. These defaults are used when no explicit sampling parameters are provided in the request.
#### `default_sampling_params.temperature`
Sampling temperature for controlling randomness. Lower values (e.g., 0.0) make the output more deterministic and focused, while higher values increase randomness.
Default: `0.0`
#### `default_sampling_params.top_p`
Nucleus sampling parameter. Only tokens with cumulative probability mass up to `top_p` are considered. This helps filter out low-probability tokens while maintaining diversity.
Default: `1.0`
#### `default_sampling_params.top_k`
Top-k sampling parameter. Only the top `k` most likely tokens are considered. Set to `-1` to disable top-k filtering and consider all tokens.
Default: `-1`
#### `default_sampling_params.max_tokens`
Maximum number of tokens to generate in this stage. This limits the length of the output sequence.
Default: `2048`
#### `default_sampling_params.seed`
Random seed for reproducible generation. When set, the random number generator will be initialized with this seed to ensure consistent outputs across runs.
Default: `42`
#### `default_sampling_params.detokenize`
Whether to detokenize the output tokens into text. When set to `true`, token IDs are converted back to readable text strings.
Default: `True`
#### `default_sampling_params.repetition_penalty`
Penalty applied to tokens that have already appeared in the generated sequence. Values greater than 1.0 discourage repetition, while values less than 1.0 encourage it. A value of 1.0 applies no penalty.
Default: `1.1`
### `tts_args` (TTS stages only)
Configuration for Text-to-Speech specific parameters. This section is only applicable to TTS model stages (e.g., `qwen3_tts`).
#### `tts_args.max_instructions_length`
Maximum character length for voice style/emotion instructions. Instructions exceeding this limit will be rejected with a validation error.
Default: `500`
This value can be overridden at runtime using the `--tts-max-instructions-length` CLI parameter when starting the server.
docs/contributing/DOCS_GUIDE.md
+1 -1
View File
@@ -46,7 +46,7 @@ class Omni:
Args:
model: Model name or path
stage_configs: Optional stage configurations
deploy_config: Optional path to a deploy configuration
**kwargs: Additional arguments passed to the engine
Example:
docs/contributing/README.md
+148 -1
View File
@@ -44,6 +44,151 @@ vLLM-Omni's `pre-commit` hooks will now run automatically every time you commit.
pre-commit run --show-diff-on-failure --color=always --all-files # runs on all files (short for --all-files)
```
!!! warning
GitHub Actions `pre-commit` **skips** several local gates so a whole-tree
`--all-files` run stays green while historical debt is cleaned up. The
current `SKIP` list in
[`.github/workflows/pre-commit.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.github/workflows/pre-commit.yml)
is `check-mark`, `markdownlint-cli2`, `shellcheck`,
`check-spdx-header`, and `mypy-3.10`. Those hooks still run on **your
commit** for changed files. A passing GitHub pre-commit check does not mean
they passed locally. Hooks **not** on that list (forbidden imports,
`torch.cuda`, TTS adapter ratchet, Buildkite schema, Ruff, typos, …) run in
both places.
The hooks below are new relative to the previous Omni config (Ruff, typos,
actionlint, YAML/whitespace, DCO sign-off, and the pickle-only checker were
already there). `check-pickle-imports` is gone: pickle is now one rule inside
`check-forbidden-imports`.
| Hook id | Enforces | GitHub Actions |
| ------- | -------- | -------------- |
| `markdownlint-cli2` | Markdown in `docs/`, `recipes/`, `README.md`, `CONTRIBUTING.md` (not `.claude/` / `.cursor/` / `CLAUDE.md`) | skipped |
| `mypy-3.10` | Type-check changed `vllm_omni/` files (model trees excluded). `tests/` uses `--follow-imports skip` | skipped |
| `mypy-3.11` / `3.12` / `3.13` | Same checker, extra Python versions | not installed; `pre-commit run --hook-stage manual mypy-3.12` |
| `check-mark` | Every `tests/**/test_*.py` has a CI level mark and a hardware platform mark or helper; no direct `pytest.mark.H100` / SKU | skipped |
| `check-tts-adapter-migration` | `self._tts_model_type` branches in `serving_speech.py` must not increase | runs |
| `shellcheck` | `*.sh` quoting / undefined vars | skipped |
| `check-spdx-header` | Omni SPDX header on `.py` / `.pyi` / `.sh` / `.rs` / `.proto` | skipped |
| `check-forbidden-imports` | pickle, stdlib `re`/`base64`, Triton, TileLang, Hugging Face Hub download/API | runs |
| `check-torch-cuda-call` | Direct `torch.cuda.*` helpers outside platform adapters | runs |
| `check-buildkite` | `.buildkite/*.yml` against the Buildkite schema | runs |
Related config (not a new hook id): Ruff `TID251` also bans `librosa` and several
`torch.cuda.*` names; typos is pinned and ignores git-describe hashes (`+g` +
hex) and NumPy `writeable`. Keep the `suggestion` hook last in
`.pre-commit-config.yaml` so autofix output does not bury the SKIP tip.
#### SPDX headers
Source files must start with:
```text
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
```
Rust and proto files use `//` instead of `#`. Empty files are exempt. A stale
`Copyright contributors to the vLLM project` line is **rewritten in place** to
the Omni copyright; the hook then exits non-zero so you restage the rewrite.
#### Forbidden imports
Under `vllm_omni/`, do not use:
- stdlib `re` → `import regex as re`
- stdlib `base64` → `import pybase64` (or `import pybase64 as base64`)
- `pickle` / `cloudpickle` (also banned in most tests)
- `from huggingface_hub import snapshot_download` / `hf_hub_download` / `HfApi`
/ similar → `vllm.transformers_utils.repo_utils`
- direct `triton` / `tilelang` → `vllm.triton_utils` / `vllm.tilelang_utils`
`examples/`, `tests/`, `benchmarks/`, `apps/`, `docs/`, `tools/`, `scripts/`,
`.buildkite/`, and `.github/` may keep stdlib `re` and `base64`
(`_NON_LIBRARY_DIRS`). Pickle is stricter: tests are not exempt unless the
file is on the pickle allowlist.
#### `torch.cuda` call sites
Prefer `current_omni_platform` / `OmniPlatform`. Platform adapters under
`vllm_omni/platforms/` are allowed; other files must not add new
`torch.cuda.*` helpers (`device_count`, `synchronize`, `empty_cache`,
`device()`, `manual_seed`, and similar). `import torch` is fine.
#### shellcheck
The hook does **not** download a binary. Install `shellcheck` with a signed
package manager, then re-run:
- Debian/Ubuntu/WSL: `sudo apt-get install shellcheck`
- Fedora: `sudo dnf install ShellCheck`
- macOS: `brew install shellcheck`
- Git Bash (MINGW): `scoop install shellcheck` so `shellcheck.exe` is on PATH.
WSL is Linux: use `apt-get`, not a Windows `.exe`.
If no native binary is found, the wrapper prints those commands (and
[the upstream install page](https://github.com/koalaman/shellcheck?tab=readme-ov-file#installing))
and exits 1. AMD and other vendor scripts are **not** excluded.
#### mypy
`mypy-3.10` runs on commit for changed files. `[tool.mypy]` uses Python 3.12,
`ignore_missing_imports = true`, and `follow_imports = "silent"`. Wrapper
`tools/pre_commit/mypy.py` skips `vllm_omni/model_executor/models/` and
`vllm_omni/diffusion/models/`, and type-checks `tests/` with
`--follow-imports skip`. Extra versions:
```bash
pre-commit run --hook-stage manual mypy-3.12
```
#### Test CI marks
`check-mark` requires each collected test module under `tests/` to
have at least one CI **level** mark (`core_model`, `advanced_model`,
`full_model`, `local_model`, or `slow`) and a **hardware** platform mark
(`cpu`, `cuda`, …) or helper (`hardware_test(` / `hardware_marks(`). SKU marks
(`H100`, `L4`, …) must come from those helpers so `cards_{n}` is attached;
direct `pytest.mark.H100` is rejected. GitHub Actions skips this hook; local
commit does not. See the
[test writing guide](./ci/test_writing_guide.md).
#### TTS adapter ratchet
`check-tts-adapter-migration` counts `self._tts_model_type` comparisons in
`vllm_omni/entrypoints/openai/serving_speech.py`. The count must stay at or
below `MAX_MODEL_TYPE_BRANCHES` in
[`tools/pre_commit/check_tts_adapter.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_tts_adapter.py).
New per-model behavior belongs in
`vllm_omni/entrypoints/openai/tts_adapters/`. Removing branches **must** lower
the budget in the same change; raising it is a reviewable policy edit.
#### Markdownlint and Buildkite
`markdownlint-cli2` auto-fixes `docs/`, `recipes/`, and the root README /
CONTRIBUTING (rules in `.markdownlint.yaml`). Skill files under `.claude/` and
`.cursor/` are excluded.
`check-buildkite` validates `.buildkite/*.yml` against the official schema
after the same expansion `upload_pipeline.py` uses. Omni-only keys such as
`mirror_hardwares` are stripped or rendered first. A few files are skipped in
`SKIP_FILES` inside `tools/pre_commit/check_buildkite.py`.
#### Extending allowlists and budgets
Prefer fixing the call site. Growing an allowlist or raising a budget is a
policy change and needs review — do not edit these just to make the hook pass.
| Gate | File | What to edit |
| ---- | ---- | ------------ |
| pickle, `re`, `base64`, Hugging Face Hub, Triton/TileLang | [`tools/pre_commit/check_forbidden_imports.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_forbidden_imports.py) | `CHECK_IMPORTS[<rule>].allowed_files`. Use `allowed_dirs` only when a whole tree should stay exempt. |
| `torch.cuda` | [`tools/pre_commit/check_torch_cuda.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_torch_cuda.py) | `ALLOWED_FILES`, or `ALLOWED_PREFIXES` for a platform adapter tree. |
| TTS `_tts_model_type` branches | [`tools/pre_commit/check_tts_adapter.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_tts_adapter.py) | `MAX_MODEL_TYPE_BRANCHES` (down only, unless the PR justifies a raise). |
| Buildkite skip list | [`tools/pre_commit/check_buildkite.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_buildkite.py) | `SKIP_FILES` for pipelines that cannot be expanded. |
To skip one hook for a single commit (discouraged): `SKIP=<hook-id> git commit`.
To bypass every hook: `git commit --no-verify` (also discouraged).
### Documentation
MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file, `mkdocs.yml`.
@@ -162,12 +307,14 @@ Only specific types of PRs will be reviewed. The PR title is prefixed appropriat
Before submitting a PR, run the [precheck-pr skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/precheck-pr/SKILL.md) with the code agent for a self-review against project conventions:
The skill offers two modes:
- **Quick (~3 min):** catches showstoppers — PR title format, missing benchmark claims, rebase status
- **Full (~10 min):** thorough maintainer-grade review — dead code scan, copy-paste detection, import hygiene
The precheck covers five PR types: Bug Fix, Performance, New Model, Diffusion Model, and General. Each type has a tailored checklist that validates evidence quality (repro steps, A/B benchmarks, registry entries, etc.). See the [precheck-pr skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/precheck-pr/SKILL.md) for the full checklist.
### Local Test
Please run the L1 and L2 test cases locally first and attach the results before contacting us to add the "ready" label. Please refer to the [test instructions](./ci/test_execution_guide.md) for running the test cases.
### Automatic skip-ci (docs and pytest skip marks)
@@ -189,7 +336,7 @@ These PR shapes all trigger skip-ci:
Skip-ci does **not** apply when the diff also touches product code (for example `vllm_omni/`), or when test files change assertions, imports, fixtures, or other non-skip logic. If the diff cannot be resolved (non-PR branches outside `main`), CI runs as usual.
!!! note
Skipping L2/L3 does **not** disable the Docker image build step. Nightly (L4) upload can still run when the PR has a `nightly-test` label or on scheduled `main` builds with `NIGHTLY=1`. Bootstrap child steps live in `bootstrap-upload-steps.yml`; the hook entry step runs `upload_pipeline.py --upload <platform>/bootstrap-upload-steps.yml`, which injects `if` by step `key` from skip-ci before upload. See [CI Settings — Diff-aware CI](./ci/ci_settings.md#diff-aware-ci) and [Test System Overview](./ci/test_system_overview.md).
Skipping L2/L3 does **not** disable the Docker image build step. Nightly (L4) upload can still run when the PR has a `nightly-test` label or on scheduled `main` builds with `NIGHTLY=1`. Weekly (L5) upload can still run with `weekly-test`, `WEEKLY=1`, or `NON_CRITICAL=1`. Bootstrap child steps live in `bootstrap-upload-steps.yml`; the hook entry step runs `upload_pipeline.py --upload <platform>/bootstrap-upload-steps.yml`, which injects `if` by step `key` from skip-ci before upload. See [CI Settings — Diff-aware CI](./ci/ci_settings.md#diff-aware-ci) and [Test System Overview](./ci/test_system_overview.md).
### Code Quality
docs/contributing/ci/ci_settings.md
+114 -11
View File
@@ -8,7 +8,7 @@ For CI levels (L1–L5) and triggers, see [Test System Overview](./test_system_o
Canonical layout (prefer these paths for new changes):
```
```text
.buildkite/
├── common/ # Shared across platforms
│ ├── scripts/
@@ -48,7 +48,7 @@ Canonical layout (prefer these paths for new changes):
└── scripts/
```
**Placement rules**
### Placement rules
| Rule | Detail |
| ---- | ------ |
@@ -81,7 +81,14 @@ There are still **legacy copies** at `.buildkite/*.yaml` (without the `cuda/` pr
| L2 | `cuda/test-ready.yml` | `ready` label |
| L3 | `cuda/test-merge.yml` | `merge-test` label / main merge |
| L4 | `cuda/test-nightly.yml` | `nightly-test` label or `NIGHTLY=1` |
| L5 | `cuda/test-weekly.yml` | `weekly-test` label or `WEEKLY=1` |
| L5 | `cuda/test-weekly.yml` | `weekly-test` label, `WEEKLY=1`, or `NON_CRITICAL=1` (see step `if`s below) |
On scheduled `main` builds, `test-weekly.yml` is uploaded when **`WEEKLY=1` or `NON_CRITICAL=1`**. Inside that file:
- **Reliability** / **Perf Test** / **Simple · CPU Coverage Test** → `WEEKLY=1` (or PR label `weekly-test` for Reliability/Perf). Pipeline upload for scheduled env vars still requires `main`.
- **E2E Tests** group (slow Omni/TTS/Diffusion sweeps) → `NON_CRITICAL=1`
**`--e2e`:** when uploading ready/merge on `main` with **`WEEKLY=1`**, bootstrap passes `--e2e` so only the **E2E Test** group is kept (same flag `run_cov_split.sh` uses to enable per-model coverage).
**Upload:** `upload_pipeline.py --upload` expands uploader-only keys before Buildkite upload.
@@ -112,6 +119,8 @@ There are still **legacy copies** at `.buildkite/*.yaml` (without the `cuda/` pr
**Test YAML:** `npu/test-npu-ready.yml` (L2), `test-npu-nightly.yml` (L4).
**L2 trigger:** PR label `ready` (CUDA/AMD L2). PR labels `ready` + `npu-test` trigger NPU CI only.
**Upload:** same as CUDA—`upload_pipeline.py --upload`.
**Hardware in YAML:** `mirror_hardwares` preset (string), expanded to `agents`, top-level `image`, and `plugins`. Presets: `a2b3_npu_1`, `a2b3_npu_4`, `a3_npu_2` in `common/ci_mirror_hardwares.yml`.
@@ -204,15 +213,15 @@ Label triggers (`ready`, `merge-test`) are unchanged—diff-aware logic only red
| Diff shape | Path | Default L2/L3 |
| --- | --- | --- |
| Product code / non-whitelisted paths | normal CI | all on |
| Docs and/or qualifying skip-mark only | `skip_all` | all off (scheduled `main` NIGHTLY/WEEKLY exceptions below) |
| Docs and/or qualifying skip-mark only | `skip_all` | all off (scheduled `main` NIGHTLY/WEEKLY/`NON_CRITICAL` exceptions below) |
| Whitelisted CI YAML only | yaml-gated (`skip_l2_l3`) | per platform/level matrix |
| Docs/skip-mark **+** whitelisted CI YAML | yaml-gated (`skip_l2_l3`) | same as CI-YAML-only (does **not** widen to normal CI) |
| Non-qualifying skip-mark and no CI YAML | normal CI | all on |
| Diff unavailable | normal CI | all on |
**`skip_all` exceptions (CUDA/NPU bootstrap only):** PR labels (`nightly-test`, `merge-test`, `npu-test`, `weekly-test`, …) do **not** revive jobs. On `main`, scheduled `NIGHTLY=1` still builds the image and uploads L4 plus L2/L3 (with `--e2e`); `WEEKLY=1` still builds the image and uploads L5 (CUDA).
**`skip_all` exceptions (CUDA/NPU bootstrap only):** PR labels (`nightly-test`, `merge-test`, `weekly-test`, `ready`, …) do **not** revive jobs under `skip_all`. On `main`, scheduled `NIGHTLY=1` still builds the image and uploads L4; `WEEKLY=1` or `NON_CRITICAL=1` still builds the image and uploads L5 (CUDA). `WEEKLY=1` on `main` also uploads L2/L3 with `--e2e`.
**Yaml-gated nightly/weekly:** L4/L5 upload steps keep their normal label / `NIGHTLY` / `WEEKLY` conditions (for example PR `nightly-test` still uploads nightly). Only L2/L3 upload steps are matrix-gated.
**Yaml-gated nightly/weekly:** L4/L5 upload steps keep their normal label / `NIGHTLY` / `WEEKLY` / `NON_CRITICAL` conditions (for example PR `nightly-test` still uploads nightly). Only L2/L3 upload steps are matrix-gated.
##### Whitelisted CI YAML → platform
@@ -285,12 +294,12 @@ CUDA **L2** (`.buildkite/cuda/test-ready.yml`) and **L3** (`.buildkite/cuda/test
| `source_file_dependencies` | List of path **prefixes**. If any changed file equals a prefix or starts with `prefix/`, keep the step (or group); otherwise omit it. |
| `mirror_hardwares` | Expand to `agents` + `plugins` (+ optional `image`) from `ci_mirror_hardwares.yml`. |
**Policy**
### Policy
- **Always uploaded** (no key): groups outside **E2E Test**—Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
- **Diff-gated**: every **E2E Test** leaf job. List the smallest prefix set per step—pytest file(s), model code under `vllm_omni/model_executor/models/` or `vllm_omni/diffusion/models/`, plus `stage_input_processors/` and `vllm_omni/deploy/*.yaml` when applicable. A **group** may define the key instead; the whole group drops if no prefix matches.
**YAML examples**
### YAML examples
```yaml
- label: "Diffusion · Qwen Image Test"
@@ -312,7 +321,7 @@ CUDA **L2** (`.buildkite/cuda/test-ready.yml`) and **L3** (`.buildkite/cuda/test
mirror_hardwares: l4_4
```
**Local dry-run**
### Local dry-run
```bash
python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml
@@ -324,15 +333,109 @@ On PR builds, `upload_pipeline.py` logs `skip '…' (no changes under …)` for
### Labels and grouping
- Use **groups** for dashboard readability; keep **E2E Test** as the group name CUDA diff filtering expects for `--e2e` nightly runs on main.
- Use **groups** for dashboard readability; keep **E2E Test** as the group name CUDA diff filtering expects for `--e2e` weekly runs on main (`WEEKLY=1`).
- Prefix labels by model domain: **Omni ·**, **TTS ·**, **Diffusion ·**, **Simple ·**, etc., matching existing steps.
### Per-model coverage
Pilot (v1): one Merge-tier (L3) job in `.buildkite/cuda/test-merge.yml` uploads
per-model, per-entry-mode coverage as Buildkite artifacts — "TTS · Qwen3-TTS Base
Test". Coverage + artifact upload run only when bootstrap uploaded that job with
`--e2e` (`WEEKLY=1` and `BUILDKITE_BRANCH=main`, via `run_cov_split.sh`); otherwise
the same job runs plain pytest without `--cov`. It runs on a single GPU so the
pilot is cheap to reproduce.
**L1 package coverage (CPU):** ready/merge Simple Test jobs run without `--cov`. Scheduled weekly on `main` with `WEEKLY=1` runs **Simple · CPU Coverage Test** in `.buildkite/cuda/test-weekly.yml` (`pytest -m 'core_model and cpu'` + Cobertura XML artifact).
**Naming convention**: `coverage-<model_id>-<mode>-<step_id>.xml.gz`, where
`<model_id>` is the model's directory name under
`vllm_omni/diffusion/models/<model_id>/` or
`vllm_omni/model_executor/models/<model_id>/` (e.g. `z_image`, `qwen3_omni`),
`<mode>` is `online` or `offline`, and `<step_id>` is `BUILDKITE_STEP_ID`
(`local` outside Buildkite). `gunzip` before feeding a report to a Cobertura
consumer.
Reports are gzipped because `--cov=vllm_omni` sets coverage's `source`, which
makes it walk the package and emit one `<line>` element per statement for every
file — including the ones a single model's tests never import. That inventory,
not the coverage, is the size: a Bagel report is 7.1 MB, of which 96.5% is
`<line>` elements and 77% belongs to the 840 of 1169 files that run never
imported. A run covering zero lines produces the same 7 MB. gzip takes it to
roughly 410 KB and changes nothing about the data.
`<step_id>` is what keeps two steps that cover the same model apart. Artifacts are
scoped to the build, and a weekly build can carry ready/merge (with `--e2e`) and
weekly tiers at once, so several same-model steps land in one namespace — `qwen3_tts` already has
both a Base and a CustomVoice job, and `minicpmo_4_5` a base and a duplex one. Two
uploads on one path do not fail loudly: an exact-name download can report the path
as ambiguous, and a glob download fetches every match concurrently and renames each
onto the same destination, so whichever finishes last wins. The step id remains the
logical step identifier across retries, while Buildkite's default artifact lookup
selects only the latest attempt — `BUILDKITE_JOB_ID` would instead give every
attempt its own filename, leaving a failed attempt's partial report to be picked up
alongside the real one. A step using `parallelism`/matrix would need
`BUILDKITE_PARALLEL_JOB` added, since its jobs share a step id; neither pilot is
one.
**Opting in a new model**: replace the job's combined `pytest` command with
[`run_cov_split.sh`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/run_cov_split.sh),
which runs each mode, names the reports, uploads them, and fails the step if
either half or the upload failed:
```yaml
commands:
- |
.buildkite/common/scripts/run_cov_split.sh \
--model-id <model_id> \
--offline <offline test path> \
--online <online test path> \
-- <the job's existing pytest flags, e.g. -m '...' --run-level '...'>
```
Everything after `--` is forwarded verbatim to both runs with quoting preserved,
so an expression like `-m 'advanced_model and cuda'` survives intact and a job can
pass anything else it needs. `--offline`/`--online` are repeatable and only one is
required, so a job with tests in a single mode runs just that half. Runtime is
bounded solely by the step's `timeout_in_minutes` — raise it, since the split
loads the model once per mode.
Before splitting, confirm each half actually collects at least one test under the job's forwarded filters — pytest exits with code 5 ("no tests collected") if a half is empty, which the script reports as a failure, red-ing a job that used to pass as one combined invocation.
Also check the tests' `num_cards` against the job's `mirror_hardwares` preset.
`_cuda_marks` turns `num_cards > 1` into `skipif(device_count() < num_cards)`, so a
test asking for more GPUs than the preset provides is silently skipped and its
coverage artifact comes back empty. Splitting such a job produces a file that
looks fine and measures nothing — "Diffusion · Z Image Test" is the current
example (online tests want 4 cards, the job runs on `l4_1`), which is why it is
not a coverage pilot.
Online-mode coverage depends on the `[tool.coverage.run]` `patch`/`sigterm`
settings in `pyproject.toml`: online tests launch the server with
`subprocess.Popen` and stop it with SIGTERM, and without those settings the XML
reflects only the pytest parent process, not the server's code paths.
The upload depends on `buildkite-agent` being callable inside the container. The
`kubernetes` presets (`h100_*`, `*_npu_*`) provide it; the `docker` ones (`l4_*`)
only do because they set `mount-buildkite-agent: true`. A new docker preset that
runs a coverage job needs the same.
List both `run_cov_split.sh` and `pyproject.toml` in every opted-in job's
`source_file_dependencies` — both change what the job measures, so without them a
change there is filtered out of normal PR builds and only surfaces in a later
nightly. `tests/buildkite/test_upload_pipeline.py` covers the filter behavior with
a synthetic job (it does not pin real merge labels).
Editing only the surrounding CI YAML still does not schedule them, so a PR that
touches just the wiring needs a full E2E run (or the commands run on a GPU host)
to produce artifacts. When checking a new model's
artifacts, compare `lines-covered` between the online and offline XML rather than
just confirming both files exist.
### Validation checklist
| Check | Command / location |
| ----- | ------------------ |
| CUDA render | `python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-<level>.yml` |
| No leaked uploader keys | `… \| grep -E 'mirror_hardwares|source_file_dependencies'` → empty |
| No leaked uploader keys | Pipe the render into `grep -E 'mirror_hardwares\|source_file_dependencies'` — expect empty |
| Skip-ci / upload unit tests | `pytest tests/buildkite/` |
| Local job replay (CUDA L2+) | `tools/run_ready_jobs.sh`, `tools/run_merge_jobs.sh`, `tools/nightly/run_nightly_jobs.sh` (read YAML from `cuda/`) |
docs/contributing/ci/failures.md
+8 -1
View File
@@ -6,7 +6,7 @@ When you open a PR against vLLM-Omni, several CI checks run automatically:
| Check | Platform | What it does |
| ----- | -------- | ------------ |
| **pre-commit** | GitHub Actions | Runs linting (Ruff), formatting, spell-checking (typos), and YAML validation. |
| **pre-commit** | GitHub Actions | Runs Ruff, formatting, typos, YAML validation, forbidden imports, `torch.cuda`, TTS adapter ratchet, and Buildkite schema. Local-only gates (SPDX, shellcheck, markdownlint, mypy-3.10, test marks) are in the workflow `SKIP` env — see [Linting](../README.md#linting). |
| **Build Wheel** | GitHub Actions | Builds Python wheels for Python 3.11 and 3.12 on Ubuntu. Skipped for docs-only or Markdown-only changes (controlled by `paths-ignore` in the workflow). |
| **DCO** | GitHub | Verifies every commit has a `Signed-off-by` line. |
| **docs/readthedocs.org:vllm-omni** | Read the Docs | Builds the MkDocs documentation site. |
@@ -29,6 +29,13 @@ pre-commit run --all-files
Then commit the fixes and push.
A green GitHub pre-commit job can still miss SPDX, shellcheck, mypy-3.10,
markdownlint, and test-mark coverage because CI skips those IDs. If your
**local** commit hook fails, follow [Linting](../README.md#linting) for the
full hook list (forbidden imports, `torch.cuda`, TTS adapter ratchet, and
Buildkite schema still run in GHA). Do not grow allowlists or raise
`MAX_MODEL_TYPE_BRANCHES` without review.
### DCO failures
Every commit must include a `Signed-off-by` line. If you forgot, amend your commits:
docs/contributing/ci/test_examples/l4_functionality_tests.inc.md
+5 -5
View File
@@ -1,8 +1,8 @@
**Scope**
#### Scope
For diffusion models, the L4 functionality test covers all or common *diffusion features* that are supported by this model, including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/quantization/overview/).
**Test Case Design**
#### Test Case Design
For a *high priority* model (currently listed in [issue #1832](https://github.com/vllm-project/vllm-omni/issues/1832)), we use several test cases, each with multiple features turned on, so that each supported feature is tested in at least one test case. This is to relieve the GPU workload on the CI machine. The suggested test case combination is as follows:
@@ -34,12 +34,12 @@ For a *normal priority* model, further reduce the number of test cases.
Currently all the features are available in online serving mode. Hence, only need to add `tests/e2e/online_serving/test_{model}_expansion.py`.
**Code Style**
#### Code Style
- Validation: test that the multimodal output files of your model have the correct shapes. `OpenAIClientHandler.send_diffusion_request` should have taken care of this.
- Validation: test that the multimodal output files of your model have the correct shapes. `OnlineOmniClient.send_diffusion_request` should have taken care of this.
- Test marks: always add `full_model` and `diffusion` for L4 nightly `test_*_expansion.py` cases. Add GPU-related marks if needed. Ref: [Markers for Tests](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/test_writing_guide/#markers-for-tests).
- To maximize code reuse, you may refer to
- `tests/conftest.py` for `omni_server` (running server in subprocess) and `openai_client` fixtures (sending requests and validating output), `generate_synthetic_image` and `assert_XXX_valid` helper.
- `tests/conftest.py` for `omni_server` (running server in subprocess) and `online_client` fixtures (sending requests and validating output), `generate_synthetic_image` and `assert_XXX_valid` helper.
- `tests/helpers/mark.py` for `@hardware_test(...)` and `hardware_marks`.
- [Parametrizing tests (pytest doc)](https://docs.pytest.org/en/stable/example/parametrize.html) to reuse test function implementation for different cases.
- Doc: add a concise docstring for each test function.
docs/contributing/ci/test_examples/l4_performance_tests.inc.md
+65 -52
View File
@@ -1,14 +1,15 @@
When you want to add L4-level ***performance test*** cases, add entries to JSON files under `tests/dfx/perf/tests/` and run them via `tests/dfx/perf/scripts/run_benchmark.py` (omni / TTS) or `run_diffusion_benchmark.py` (diffusion).
**Config file layout (in-tree examples)**
#### Config file layout (in-tree examples)
| Model type | Runner | Example JSON files |
| ---------- | ------ | ------------------ |
| Omni | `run_benchmark.py` | `test_qwen3_omni_no_async_chunk.json`, `test_qwen3_omni_async_chunk.json`, `test_qwen3_omni_vllm_text.json`, `test_qwen3_omni_multi_replicas.json` |
| Omni (nightly) | `run_benchmark.py` | `test_qwen3_omni_no_async_chunk.json`, `test_qwen3_omni_async_chunk.json` (`full_model` without `slow` in `mark`) |
| Omni (weekly) | `run_benchmark.py` | `test_qwen3_omni_async_chunk.json` (CUDA only), `test_qwen3_omni_vllm_text.json`, `test_qwen3_omni_multi_replicas.json` (`slow` in `mark`; **Perf Test** in `test-weekly.yml`) |
| TTS | `run_benchmark.py` | `test_tts.json`, `test_voxcpm2.json`, `test_higgs_audio_v3.json` |
| Diffusion | `run_diffusion_benchmark.py` | `test_qwen_image_vllm_omni.json`, `test_bagel_vllm_omni.json`, `test_wan22_i2v_vllm_omni.json`, `test_cosmos3_vllm_omni.json`, … |
**How runners pick cases**
#### How runners pick cases
Without **`--test-config-file`**, each runner scans all `*.json` under `tests/dfx/perf/tests/` but only keeps its own model type:
@@ -17,9 +18,9 @@ Without **`--test-config-file`**, each runner scans all `*.json` under `tests/df
Diffusion cases are detected when the JSON has `server_type` (typically `"vllm-omni"`) or `"diffusion"` in the `mark` array. Omni / TTS JSON has neither.
**Running perf cases**
#### Running perf cases
Pass **`--test-config-file`** to run one JSON file as-is, or omit it for the bulk scan above and filter with pytest **`-m`** on each case's JSON `mark` (for example `pytest … run_benchmark.py -m "full_model and tts and H100"`).
Pass **`--test-config-file`** to load one JSON file, or omit it for the bulk scan above. In either mode, pytest **`-m`** filters each case using its JSON `mark`. A single file can contain nightly, weekly, and platform-specific case objects. For example, the nightly object in `test_qwen3_omni_async_chunk.json` declares both H100 and A3, while its `slow` weekly object is CUDA-only; there is no NPU weekly counterpart. Use `-m "H100 and full_model and not slow"`, `-m "H100 and full_model and slow"`, or `-m "npu"` to select the intended schedule and platform.
```JSON
{
@@ -31,7 +32,7 @@ Pass **`--test-config-file`** to run one JSON file as-is, or omit it for the bul
],
"server_params": {
"model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
"stage_config_name": "qwen3_omni.yaml"
"stage_config_name": "qwen3_omni_moe.yaml"
},
"benchmark_params": [
{
@@ -55,31 +56,33 @@ Pass **`--test-config-file`** to run one JSON file as-is, or omit it for the bul
}
```
**Parameter Explanation**
#### Parameter Explanation
*Overview*
##### Overview
| Field | Required | Description |
| ---------------- | -------- | --------------------------------------------------------------- |
| test_name | Yes | Unique identifier for the test case |
| mark | No | Pytest marks for this case (see **`mark` field** below). Omit only for configs not meant to be filtered by `-m`. |
| server_params | Yes | Server-side configuration parameters |
| benchmark_params | Yes | Benchmark running parameters (supports multiple configurations) |
| server_type | Diffusion only | When set (e.g. `"vllm-omni"`), routes the case to `run_diffusion_benchmark.py`. |
| benchmark_endpoint | Diffusion only | API path for the benchmark client (e.g. `/v1/videos`, `/v1/images/generations`). |
| Field | Required | Description |
| ------------------ | -------------- | -------------------------------------------- |
| test_name | Yes | Unique identifier for the test case |
| mark | No | Pytest marks; see **`mark` field** below |
| server_params | Yes | Server-side configuration parameters |
| benchmark_params | Yes | Benchmark running parameters |
| server_type | Diffusion only | Routes case to run_diffusion_benchmark.py |
| benchmark_endpoint | Diffusion only | Benchmark API path |
**`mark` field**
Omit `mark` only for configs not meant to be filtered by `-m`. `server_type` is typically `"vllm-omni"`. `benchmark_endpoint` examples: `/v1/videos`, `/v1/images/generations`.
#### `mark` field
Optional top-level field on each perf JSON **case object** (one per `test_name`). `run_benchmark.py` and `run_diffusion_benchmark.py` read it via `tests.dfx.conftest.resolve_pytest_marks` and attach the marks to the corresponding `pytest.param`, so you can filter locally with `-m`.
When `mark` is present, it must be an **array** with exactly one ``hardware_marks`` object (same shape as `@hardware_test` / `hardware_marks()` in `tests/helpers/mark.py`), followed by registered pytest marker name strings such as `full_model`, `omni`, `tts`, `diffusion`, or `local_model`.
When `mark` is present, it must be an **array** with exactly one ``hardware_marks`` object (same shape as `@hardware_test` / `hardware_marks()` in `tests/helpers/mark.py`), followed by registered pytest marker name strings such as `full_model`, `slow`, `omni`, `tts`, `diffusion`, or `local_model`.
Supported form:
| Form | Example | Effect |
| ---- | ------- | ------ |
| Array (single platform) | `"mark": [{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}}, "full_model", "diffusion"]` | Hardware marks from `hardware_marks(...)`; `num_cards` > 1 adds `distributed_*` (+ CUDA `skipif_cuda` when applicable). String entries become extra pytest marks. |
| Array (multi-platform) | `"mark": [{"hardware_marks": {"res": {"cuda": "H100", "rocm": "MI325", "npu": "A2"}, "num_cards": {"cuda": 2, "rocm": 2, "npu": 1}}}, "full_model", "omni"]` | Same as above, but declares **multiple platforms** in one case. Filter examples: `-m "full_model and H100 and cuda"`, `-m "full_model and MI325 and rocm"`. |
| Array (single platform) | `"mark": [{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}}, "full_model", "diffusion"]` | Hardware marks from `hardware_marks(...)`; `num_cards` adds `cards_{n}` (including `cards_1` when omitted). String entries become extra pytest marks. |
| Array (multi-platform) | `"mark": [{"hardware_marks": {"res": {"cuda": "H100", "rocm": "MI325"}, "num_cards": 2}}, "full_model", "omni"]` | Same as above, but declares **multiple platforms** on one item. All platforms must share the same `num_cards` (int or identical dict values). Different counts need `@hardware_test` or one `hardware_marks()` per `pytest.param`. Filter examples: `-m "full_model and H100 and cards_2 and cuda"`, `-m "full_model and MI325 and cards_2 and rocm"`. |
Recommended for L4 perf cases:
@@ -119,11 +122,11 @@ Multi-GPU diffusion (example: Cosmos3 with `cfg-parallel-size=2`):
- Diffusion perf: include `"diffusion"` in the `mark` array
- HunyuanImage local-weight cases: add `"local_model"` to the `mark` array
**Parametrization IDs**
#### Parametrization IDs
Each `(server, benchmark index)` pair becomes one `pytest.param` with id `{test_name}-{suffix}`. The suffix comes from `benchmark_params[].name` when set (for example `test_tts-p0`, `test_omni-p1`); otherwise it is derived from fields like `task` / `eval_phase`.
**Benchmark result filenames**
#### Benchmark result filenames
Result files use the **runtime** hardware label from `get_runtime_resource_label()` (detected GPU/NPU name on the machine that ran the job), **not** `mark.hardware_marks.res`. On the default H100 CI pool, `H100` is **omitted** from filenames (`resource_label_for_filename`).
@@ -132,32 +135,40 @@ Examples:
- Omni/TTS: `result_{test_name}_{optional_hw}_{dataset}_....json` under `BENCHMARK_DIR`
- Diffusion: one aggregate `diffusion_result_{config_stem}_{optional_hw}_{timestamp}.json` per source JSON file (array of all runs from that file)
**Local commands**
#### Local commands
```bash
# Bulk load + filter by JSON mark
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and H100 and diffusion"
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py -m "full_model and omni and H100"
# Single file (same as nightly CI Perf steps)
# Single file (same selectors as the CI Perf steps)
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py \
--test-config-file tests/dfx/perf/tests/test_bagel_vllm_omni.json
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
--test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json
--test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json \
-m "H100 and full_model and not slow"
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
--test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json \
-m "H100 and full_model and slow"
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
--test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json \
-m "npu"
```
See also [Markers for Tests](#markers-for-tests) for registered hardware markers (`H100`, `L4`, `cuda`, `distributed_cuda`, …).
<!-- markdownlint-disable-next-line MD051 -->
See also [Markers for Tests](#markers-for-tests) for registered hardware markers (`H100`, `L4`, `cuda`, `cards_1`, …).
**`server_params` Configuration**
#### `server_params` Configuration
*Basic Parameters*
##### Basic Parameters
| Parameter | Required | Example | Description |
| ----------------- | -------- | ---------------------------------- | ----------------------------- |
| model | Yes | "Qwen/Qwen3-Omni-30B-A3B-Instruct" | Model name or path |
| stage_config_name | Yes | "qwen3_omni.yaml" | Stage configuration file name |
| Parameter | Required | Example | Description |
| ----------------- | -------- | ------------------------- | ------------------------------ |
| model | Yes | Qwen/Qwen3-Omni-30B-A3B… | Model name or path |
| stage_config_name | Yes | qwen3_omni_moe.yaml | Deploy configuration file name |
*Dynamic Configuration (update/delete)*
##### Dynamic Configuration (update/delete)
Supports incremental modifications based on the basic configuration:
@@ -166,38 +177,40 @@ Supports incremental modifications based on the basic configuration:
| update | Update or add configuration items |
| delete | Delete specified configuration items |
**Example**:
##### Example
```
```json
"update": {
"async_chunk": true, // Enable asynchronous chunk processing
"stage_args": {
"stages": {
"0": {
"engine_args.custom_process_next_stage_input_func": "vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_async_chunk"
"max_num_seqs": 64
}
}
},
"delete": {
"stage_args": {
"2": ["custom_process_input_func"] // Delete this configuration for stage 2
"stages": {
"2": ["max_num_batched_tokens"] // Delete this deploy override for stage 2
}
}
```
**`benchmark_params` Configuration**
#### `benchmark_params` Configuration
You can add any benchmark running parameters you need here. For all optional parameters, refer to the [benchmark documentation](https://github.com/vllm-project/vllm-omni/blob/main/docs/cli/bench/serve.md). General modifications are as follows:
1. Change the --xxx-xx-xx running parameters to xxx_xx_xx format and fill them as keys in the JSON file.
2. For boolean variables in the running parameters, modify them to forms such as ignore_eos: true/false and fill them into the JSON file.
3. Optionally add a `baseline` object (see **Baseline thresholds** below). If you omit `baseline` or leave it empty, the performance test still runs but does not assert metric thresholds from this field.
4. Set `"name"` on each `benchmark_params` entry for stable pytest ids and readable result keys.
5. The qps and concurrency modes are recommended to be mutually exclusive. For detailed explanations, see the table below:
1. Change the --xxx-xx-xx running parameters to xxx_xx_xx format and fill them as keys in the JSON file.
2. For boolean variables in the running parameters, modify them to forms such as ignore_eos: true/false and fill them into the JSON file.
3. Optionally add a `baseline` object (see **Baseline thresholds** below). If you omit `baseline` or leave it empty, the performance test still runs but does not assert metric thresholds from this field.
4. Set `"name"` on each `benchmark_params` entry for stable pytest ids and readable result keys.
5. The qps and concurrency modes are recommended to be mutually exclusive. For detailed explanations, see the table below:
| Parameter | Type | Required | Example/Values | Description |
| --------------- | ----------- | -------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name | string | No | `"1024x1024_steps4"` | Stable suffix for pytest param id and logs. |
| num_prompts | int / array | Yes | 10,[10, 20, 30] | Number of requests. Supports single values or arrays. If a single value is used, it will be automatically expanded to match the number of qps or max_concurrency, e.g., [10,10,10]. If an array is used, its length must match the number of qps or max_concurrency. |
| request_rate | float / array | No | 0.5, [0.5, 1, inf] | Queries per second. Supports single values or arrays. If a single value is used, it will be automatically expanded to match the number of num_prompts, e.g., [1,1,1]. If an array is used, its length must match the number of num_prompts. |
| max_concurrency | int / array | No | 1, [1, 2, 3] | Maximum concurrent in-flight requests. Same array / expansion rules as `request_rate` (mutually exclusive with QPS mode). |
| baseline | object | No | see below | Optional thresholds. **Hardware-nested only**: top-level keys must be `[hardware-resource]` pytest markers from `pyproject.toml` (`get_hardware_mark_list()`). Under each hardware key, metric names are free-form; values are scalars or sweep-aligned lists. Example: `{"H100": {"throughput_qps": […], "custom_stage_ms": 1.0}, "A3": {…}}`. Runtime filename aliases (`_RUNTIME_DEVICE_ALIASES`) are separate and do not expand this allowlist. |
| Parameter | Type | Required | Example/Values | Description |
| --------------- | ------------- | -------- | -------------------- | ------------------------------------ |
| name | string | No | `"1024x1024_steps4"` | Stable suffix for pytest param id |
| num_prompts | int / array | Yes | 10, [10, 20, 30] | Number of requests (scalar or sweep) |
| request_rate | float / array | No | 0.5, [0.5, 1, inf] | Queries per second (QPS mode) |
| max_concurrency | int / array | No | 1, [1, 2, 3] | Max in-flight requests |
| baseline | object | No | see below | Optional hardware-nested thresholds |
`num_prompts`, `request_rate`, and `max_concurrency` accept scalars or arrays; single values expand to match sweep length. QPS and concurrency modes are mutually exclusive. `baseline` top-level keys must be hardware pytest markers from `pyproject.toml`; metric names and values are free-form (scalars or sweep-aligned lists). Omit `baseline` to run without threshold assertions.
docs/contributing/ci/test_execution_guide.md
+20 -12
View File
@@ -1,13 +1,20 @@
# Test Execution Guide
## Setting Up the Test Environment
### Creating a Container
vLLM-Omni provides an official Docker image for deployment. These images are built upon vLLM Docker images and are available on [Docker Hub](https://hub.docker.com/r/vllm/vllm-omni/tags). The version of vLLM-Omni indicates which vLLM release it is based on.
For a local test environment, you can follow the steps below to create a container:
## Installing Dependencies
### vLLM & vLLM-Omni
vLLM-Omni is built based on vLLM. You can follow [install guide](../../getting_started/installation/README.md) to build your local environment.
### Test Case Dependencies
When running test cases, you may need to install the following dependencies:
```bash
@@ -16,6 +23,7 @@ apt-get install -y espeak-ng jq
```
## Running Tests
Our test scripts use the pytest framework. First, please use `git clone https://github.com/vllm-project/vllm-omni.git` to download the vllm-omni source code. Then, in the root directory of vllm-omni, you can run the following commands in your local test environment to execute the corresponding test cases.
### CI job runners (L2–L4): logs, timing, and timeouts
@@ -25,7 +33,7 @@ Our test scripts use the pytest framework. First, please use `git clone https://
**Log layout** (default under `logs/ready_jobs`, `logs/merge_jobs`, or `logs/nightly_jobs`; override with `--log-dir`):
| Path | Description |
|------|-------------|
| ------ | ------------- |
| `jobs/*.sh` | Generated per-step bash wrappers |
| `jobs/.job_timeouts` | Job key → `timeout_in_minutes` from Buildkite YAML (when set) |
| `<job_key>.log` | Tee output for each job |
@@ -33,7 +41,7 @@ Our test scripts use the pytest framework. First, please use `git clone https://
**Timing:** while each job runs, stderr prints `finished in …` or `failed after …`. When all jobs finish, a summary is printed and written to `timing_summary.log`, for example:
```
```text
=== Job timing summary ===
TTS_VoxCPM2_Test 25m 40s OK
Omni_Qwen3-Omni_Test 8m 05s TIMED OUT
@@ -80,7 +88,7 @@ Failed jobs: 1/2
```bash
cd tests
pytest -s -v test_xxxx.py --run-level=core_model
pytest -s -v -m "core_model and distributed_cuda and L4" --run-level=core_model
pytest -s -v -m "core_model and L4 and not cards_1" --run-level=core_model
```
=== "L3 level"
@@ -111,10 +119,9 @@ Failed jobs: 1/2
```bash
pytest -s -v test_xxxx.py --run-level=advanced_model
pytest -s -v -m "advanced_model and distributed_cuda and L4" --run-level=advanced_model
pytest -s -v -m "advanced_model and L4 and not cards_1" --run-level=advanced_model
```
=== "L4 level"
**Recommended:** run CI-aligned nightly jobs from the repo root via [`tools/nightly/run_nightly_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/nightly/run_nightly_jobs.sh). The script reads [`.buildkite/cuda/test-nightly.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/test-nightly.yml), generates per-step bash wrappers, runs pytest (perf jobs first, then aggregates perf JSON into Excel), and tees logs under `logs/nightly_jobs/` (see [CI job runners](#ci-job-runners-l2l4-logs-timing-and-timeouts); `generate_nightly_perf_excel` is included in the timing summary). Requires `bash`, `python3`, and PyYAML (`pip install pyyaml`).
@@ -146,12 +153,12 @@ Failed jobs: 1/2
```bash
cd tests
pytest -s -v test_xxxx.py --run-level=full_model
pytest -s -v -m "full_model and distributed_cuda and L4" --run-level=full_model
pytest -s -v -m "full_model and L4 and not cards_1" --run-level=full_model
pytest -s -v -m "full_model and (omni or tts) and H100" --run-level=full_model
```
If you only want to run specific test cases on a particular platform, you can use:
```bash
pytest -s -v -m "full_model and distributed_cuda and L4" --run-level=full_model
pytest -s -v -m "full_model and L4 and not cards_1" --run-level=full_model
```
Note: ``run_benchmark.py`` and ``run_diffusion_benchmark.py`` accept an optional ``--test-config-file``. If omitted, each loads every ``*.json`` under ``tests/dfx/perf/tests/`` (omni/tts vs diffusion split by ``is_diffusion_perf_config``) and pytest ``-m`` filters by each case's JSON ``mark``:
```bash
@@ -160,7 +167,7 @@ Failed jobs: 1/2
pytest -sv tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json
pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_cosmos3_vllm_omni.json
```
Nightly **Perf Test** jobs in ``test-nightly.yml`` use ``--test-config-file`` only (no ``-m``). E2e L4 function tests still use ``full_model`` + ``--run-level full_model`` (see [test-nightly.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly.yml)). Example:
Nightly **Perf Test** jobs in [``test-nightly.yml``](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/test-nightly.yml) use ``--test-config-file`` only (no ``-m``). Weekly **Perf Test** in [``test-weekly.yml``](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/test-weekly.yml) runs ``test_qwen3_omni_vllm_text.json`` and ``test_qwen3_omni_multi_replicas.json`` (JSON ``mark`` includes ``slow``). E2e L4 function tests use ``full_model`` + ``--run-level full_model``. Example:
=== "L5 level"
@@ -184,14 +191,15 @@ Failed jobs: 1/2
# Reliability: HunyuanImage DiT (H100 × 4)
pytest -s -v dfx/reliability/test_reliability_hunyuan_image.py -m slow
# Reliability: VoxCPM2 (L4)
pytest -s -v dfx/reliability/test_reliability_voxcpm2.py -m slow
```
The latest L5 CI jobs (reliability + invalid-parameter weekly steps) are in [test-weekly.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/test-weekly.yml).
The latest L5 CI jobs are in [test-weekly.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/test-weekly.yml):
- **Reliability** (+ invalid-parameter), **Perf Test** (Omni vLLM Text / Multi-Replica), and **Simple · CPU Coverage Test** → `WEEKLY=1` (or PR label `weekly-test` for Reliability/Perf). Scheduled upload of those env-gated steps still requires `main`.
- **E2E Tests** (slow Omni / TTS / Diffusion sweeps) → `NON_CRITICAL=1` (pipeline upload also accepts `WEEKLY=1` / `weekly-test`; the group `if` still needs `NON_CRITICAL`)
You can find more information about markers in [Test Writing Guide — Markers for Tests](./test_writing_guide.md#markers-for-tests).
## Adding New Test Cases
Please refer to the [Test System Overview](./test_system_overview.md) and [Test Writing Guide](./test_writing_guide.md).
docs/contributing/ci/test_system_overview.md
+12 -20
View File
@@ -12,8 +12,7 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
<th>Level</th>
<th>Scope & Focus</th>
<th>Model Coverage Strategy</th>
<th>Feature Coverage Strategy</th>
<th>Interface Coverage Strategy</th>
<th>Module Strategy</th>
<th>Tags</th>
<th>Time Cost</th>
<th>Test Dir</th>
@@ -31,7 +30,6 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td><a href="https://github.com/vllm-project/vllm-omni/blob/main/.github/PULL_REQUEST_TEMPLATE.md">PR Checklist</a></td>
<td>/</td>
<td>/</td>
@@ -43,7 +41,6 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td><a href="./failures.md">CI Failures</a></td>
<td>/</td>
<td>/</td>
@@ -53,7 +50,6 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
<td>Unit tests for components like entrypoints, models</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td><code>core_model and cpu</code></td>
<td rowspan="2">&lt;15min</td>
<td>
@@ -71,9 +67,8 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
<tr>
<td><strong>L2</strong><br>(E2E across models & GPU-required UT)</td>
<td>Online (basic deployment scenarios):<br>dummy, normal inference function (output format, stream), some instance startup UT</td>
<td>High-priority models + online basic scenarios; request success, non-empty output, format match (no Whisper/accuracy)</td>
<td>High-priority features and component tests that require launching instances</td>
<td>High-priority interfaces (using random lightweight models)</td>
<td>Key models + online basic scenarios; request success, non-empty output, format match (no Whisper/accuracy)</td>
<td>Key features and modules that require launching instances (prefer random weights)</td>
<td><code>core_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td>
<strong>Model E2E:</strong><br>
@@ -95,9 +90,8 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
<tr>
<td><strong>L3</strong><br>(Important Perf & Integration & Accuracy)</td>
<td>Online & Offline (multiple deployment scenarios):<br>real model, normal inference function, normal accuracy</td>
<td>High/medium-priority models + key online/offline scenarios; real weights, Whisper/similarity, preset voice gender, basic accuracy</td>
<td>Medium-priority features and component tests that require launching instances</td>
<td>Medium-priority interfaces (using random lightweight models)</td>
<td>Key models + key online/offline scenarios; real weights, Whisper/similarity, preset voice gender, basic accuracy</td>
<td>Key features and modules that require launching instances (using real weights)</td>
<td><code>advanced_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td>&lt;30min</td>
<td>
@@ -120,9 +114,8 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
<tr>
<td><strong>L4</strong><br>(Perf & Integration & Accuracy)</td>
<td>Online: full functional scenarios + performance test + doc test + accuracy test</td>
<td>High-priority models: function, performance, accuracy, and doc testing<br>Medium-priority models: function and doc testing</td>
<td>Low-priority features and component tests that require launching instances</td>
<td>Low-priority interfaces (using real weights)</td>
<td>Key models: function, performance, accuracy, and doc testing</td>
<td>Other features and modules that require launching instances (using real weights)</td>
<td><code>full_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td>&lt;3 hour</td>
<td>
@@ -147,11 +140,10 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
<td>GPU</td>
</tr>
<tr>
<td><strong>L5</strong><br>(Stability & Reliability)</td>
<td>Online: long-term stability test + reliability test</td>
<td>Long-term stability and reliability testing for high-priority models<br>Low-priority models: function and doc testing</td>
<td><strong>L5</strong><br>(Stability & Reliability & selected Perf)</td>
<td>Online: long-term stability + reliability; coverage test</td>
<td>Long-term stability and reliability testing for key models<br>Non-critical scenario performance and accuracy testing for key models<br>Other models: function and doc testing</td>
<td>/</td>
<td>Invalid-parameter validation for high-priority interfaces</td>
<td><code>slow and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td> Depends on reality </td>
<td>
@@ -178,8 +170,8 @@ For per-level test authoring (markers, examples), see [Test Writing Guide](./tes
Before entering specific testing levels, the project establishes two common specifications aimed at standardizing the development process and quickly locating issues.
1. ***PR Checklist ([`.github/PULL_REQUEST_TEMPLATE.md`](https://github.com/vllm-project/vllm-omni/blob/main/.github/PULL_REQUEST_TEMPLATE.md))***: This template defines the self-check items that must be completed before submitting a code review (Pull Request). It ensures that each code change meets basic requirements such as code style, dependency updates, and documentation synchronization before entering the automated testing pipeline, serving as the first manual line of defense for quality assurance.
2. ***CI Failure Explanation ([CI Failures](./failures.md))***: This document archives and explains common failure patterns in the Continuous Integration (CI) pipeline, error log interpretation, and preliminary troubleshooting steps. It helps developers and testers quickly diagnose the causes of automated test failures, improving problem-solving efficiency.
1. ***PR Checklist ([`.github/PULL_REQUEST_TEMPLATE.md`](https://github.com/vllm-project/vllm-omni/blob/main/.github/PULL_REQUEST_TEMPLATE.md))***: This template defines the self-check items that must be completed before submitting a code review (Pull Request). It ensures that each code change meets basic requirements such as code style, dependency updates, and documentation synchronization before entering the automated testing pipeline, serving as the first manual line of defense for quality assurance.
2. ***CI Failure Explanation ([CI Failures](./failures.md))***: This document archives and explains common failure patterns in the Continuous Integration (CI) pipeline, error log interpretation, and preliminary troubleshooting steps. It helps developers and testers quickly diagnose the causes of automated test failures, improving problem-solving efficiency.
## Notes
docs/contributing/ci/test_writing_guide.md
+162 -226
View File
@@ -17,47 +17,43 @@ Summary:
- **Reliability:** `tests/dfx/reliability/`
- Do **not** add new top-level `tests/` directories unrelated to a component (or to `e2e` / `dfx` / `helpers` / `examples` / `buildkite`)
Deploy YAMLs: `vllm_omni/deploy/` via `tests.helpers.stage_config.get_deploy_config_path` (see [stage configs](../../configuration/stage_configs.md)).
Deploy YAMLs: `vllm_omni/deploy/` via `tests.helpers.stage_config.get_deploy_config_path` (see [pipeline and deploy configs](../../configuration/stage_configs.md)).
## Markers for Tests
By adding markers before test functions, tests can later be executed uniformly by simply declaring the corresponding marker type.
### Current Markers
Defined in `pyproject.toml`:
| Marker | Description |
| ------------------ | --------------------------------------------------------- |
| `core_model` | L1&L2 tests (run in each PR) |
| `advanced_model` | L3 tests (run on each merge to main) |
| `full_model` | L4 tests (run nightly) |
| `diffusion` | Diffusion model tests |
| `omni` | Omni multimodal model tests |
| `tts` | TTS model tests |
| `cache` | Cache backend tests |
| `parallel` | Parallelism/distributed tests |
| `cpu` | Tests that run on CPU |
| `gpu` | Tests that run on GPU * |
| `cuda` | Tests that run on CUDA * |
| `rocm` | Tests that run on AMD/ROCm * |
| `xpu` | Tests that run on Intel XPU * |
| `npu` | Tests that run on NPU/Ascend * |
| `H100` | Tests that require H100 GPU * |
| `L4` | Tests that require L4 GPU * |
| `MI325` | Tests that require MI325 GPU (AMD/ROCm) * |
| `A2` | Tests that require A2 NPU * |
| `A3` | Tests that require A3 NPU * |
| `distributed_cuda` | Tests that require multi cards on CUDA platform * |
| `distributed_rocm` | Tests that require multi cards on ROCm platform * |
| `distributed_npu` | Tests that require multi cards on NPU platform * |
| `skipif_cuda` | Skip if the num of CUDA cards is less than the required * |
| `skipif_rocm` | Skip if the num of ROCm cards is less than the required * |
| `skipif_npu` | Skip if the num of NPU cards is less than the required * |
| `slow` | Slow tests (may skip in quick CI) |
| Marker | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------- |
| `core_model` | L1&L2 tests (run in each PR) |
| `advanced_model` | L3 tests (run on each merge to main) |
| `full_model` | L4 tests (run nightly) |
| `diffusion` | Diffusion model tests |
| `omni` | Omni multimodal model tests |
| `tts` | TTS model tests |
| `cache` | Cache backend tests |
| `parallel` | Parallelism/distributed tests |
| `cpu` | Tests that run on CPU |
| `gpu` | Tests that run on GPU * |
| `cuda` | Tests that run on CUDA * |
| `rocm` | Tests that run on AMD/ROCm * |
| `xpu` | Tests that run on Intel XPU * |
| `npu` | Tests that run on NPU/Ascend * |
| `H100` | Tests that require H100 GPU * |
| `L4` | Tests that require L4 GPU * |
| `MI325` | Tests that require MI325 GPU (AMD/ROCm) * |
| `A2` | Tests that require A2 NPU * |
| `A3` | Tests that require A3 NPU * |
| `cards_{n}` | Tests that require *n* accelerator cards (`cards_1` … `cards_8`); auto-added from `num_cards` * |
| `slow` | Slow tests (may skip in quick CI) |
| `benchmark` | Benchmark tests (decorator on runner test functions; perf JSON uses `full_model` + type marker instead) |
| `local_model` | Tests requiring local / non-HF-hub model weights |
| `local_model` | Tests requiring local / non-HF-hub model weights |
\* Means those markers are auto-added by `@hardware_test` (parametrization decorator) or `hardware_marks` (only returning the list of marks for flexibility).
\* Means those markers are auto-added by `@hardware_test` (parametrization decorator) or `hardware_marks` (only returning the list of marks for flexibility). Do **not** write `pytest.mark.H100` / `pytest.mark.L4` (or other SKU names) by hand: `check-mark` rejects that so `cards_{n}` cannot be skipped.
#### Example usage for markers
@@ -80,28 +76,26 @@ def test_video_to_audio()
This decorator is intended to make hardware-aware, cross-platform test authoring easier and more robust for CI/CD environments. The `hardware_test` decorator in `vllm-omni/tests/helpers/mark.py` performs the following actions:
1. **Applies platform and resource markers**
Adds the appropriate pytest markers for each specified hardware platform (e.g., `cuda`, `rocm`, `xpu`, `npu`) and resource type (e.g., `L4`, `H100`, `MI325`, `B60`, `A2`, `A3`).
```
Adds the appropriate pytest markers for each specified hardware platform (e.g., `cuda`, `rocm`, `xpu`, `npu`) and resource type (e.g., `L4`, `H100`, `H200`, `B200`, `MI325`, `B60`, `A2`, `A3`).
```python
@pytest.mark.cuda
@pytest.mark.L4
```
2. **Handles multi-card (distributed) scenarios**
For tests requiring multiple cards, it automatically adds distributed markers such as `distributed_cuda`, `distributed_rocm`, or `distributed_npu`.
```
@pytest.mark.distributed_cuda(num_cards=num_cards)
```
2. **Handles card-count filtering**
Adds a filterable marker `cards_{n}` (including `cards_1` when `num_cards` is omitted / 1). Use this with the SKU and platform, e.g. `-m "H100 and cards_4 and cuda"` vs `-m "H100 and cards_2 and cuda"`. Any multi-card CUDA test is `-m "cuda and not cards_1"`.
When `res` lists more than one platform **and** `num_cards` differs by platform, `@hardware_test` collects **one variant per platform** so those `cards_{n}` marks do not stack. `-m "H100 and cards_4 and cuda"` then selects only the CUDA/H100 4-card variant, not a ROCm 2-card sibling. The same count on every platform stays a single item.
3. **Supports flexible card requirements**
Accepts `num_cards` as either a single integer for all platforms or as a dictionary with per-platform values. If not specified, defaults to 1 card per platform.
4. **Integrates resource validation**
On CUDA, adds a skip marker (`skipif_cuda`) if the system does not have the required number of devices.
Support for `skipif_rocm` and `skipif_npu` will be implemented later.
On CUDA and XPU, when `num_cards > 1`, attaches pytest's built-in `skipif` if the machine has fewer devices than requested. This is not a named `skipif_*` marker and is not used for `-m` filtering.
5. **Works with pytest filtering**
Allows tests to be filtered and selected at runtime using standard pytest marker expressions (e.g., `-m "distributed_cuda and L4"`).
Allows tests to be filtered and selected at runtime using standard pytest marker expressions (e.g., `-m "H100 and cards_2"`, `-m "L4 and not cards_1"`).
##### Example usage for decorator
- Single call for multiple platforms:
```python
@hardware_test(
@@ -116,13 +110,14 @@ This decorator is intended to make hardware-aware, cross-platform test authoring
num_cards=2,
)
```
- `res` must be a dict; supported resources: CUDA (L4/H100), ROCm (MI325), XPU (B60), MUSA (S5000), NPU (A2/A3)
- `res` must be a dict. SKU values come from `pyproject.toml` markers tagged `[hardware-resource]` plus the platform key (`[cuda]`, `[rocm]`, `[xpu]`, `[npu]`, `[musa]`).
- `num_cards` can be int (all platforms) or dict (per platform); defaults to 1 when missing
- Distributed markers (`distributed_cuda`, `distributed_rocm`, `distributed_npu`) are auto-added for multi-card cases
- Filtering examples:
- CUDA only: `pytest -m "distributed_cuda and L4"`
- ROCm only: `pytest -m "distributed_rocm and MI325"`
- NPU only: `pytest -m "distributed_npu"`
- CUDA H100 4-card: `pytest -m "H100 and cards_4 and cuda"`
- CUDA H100 2-card: `pytest -m "H100 and cards_2 and cuda"`
- CUDA multi-card (any count): `pytest -m "L4 and not cards_1"`
- ROCm multi-card: `pytest -m "MI325 and not cards_1"`
- NPU multi-card: `pytest -m "npu and not cards_1"`
#### Function: `hardware_marks`
@@ -178,37 +173,30 @@ If you want to add support for a new platform (e.g., "tpu" for a new accelerator
[tool.pytest.ini_options]
markers = [
# ... existing markers ...
"tpu: Tests that require TPU device",
"TPU_V3: Tests that require TPU v3 hardware",
"distributed_tpu: Tests that require multiple TPU devices",
"tpu: [hardware-platform] Tests that require TPU device",
# Add [gpu] next to [hardware-platform] if this platform should also get pytest.mark.gpu.
"TPU_V3: [hardware-resource] [tpu] Tests that require TPU v3 hardware",
"cards_1: [hardware-cards] Tests that require 1 accelerator card",
"cards_2: [hardware-cards] Tests that require 2 accelerator cards",
]
```
`check-mark` treats `[hardware-platform]` names (including `cpu`/`gpu`) as allowed `pytest.mark.<name>` values. `hardware_test(res=...)` still rejects `cpu`/`gpu` as `res` keys.
2. **Implement a marker construction function for your platform** in `vllm-omni/tests/helpers/mark.py`:
```python
# In vllm-omni/tests/helpers/mark.py
def tpu_marks(*, res: str, num_cards: int):
test_platform = pytest.mark.tpu
if res == "TPU_V3":
test_resource = pytest.mark.TPU_V3
else:
raise ValueError(
f"Invalid TPU resource type: {res}. Supported: TPU_V3")
if num_cards == 1:
return [test_platform, test_resource]
else:
test_distributed = pytest.mark.distributed_tpu(num_cards=num_cards)
# Optionally: add skipif_tpu when implemented
return [test_platform, test_resource, test_distributed]
def _tpu_marks(*, res: str, num_cards: int):
_require_sku("tpu", res)
return [pytest.mark.tpu, getattr(pytest.mark, res), _cards_mark(num_cards)]
```
3. **Update `hardware_marks` to recognize your new platform**:
In the relevant place (see the `hardware_marks` implementation), add:
3. **Register the builder** in `_marks_for_platform`:
```python
if platform == "tpu":
marks = tpu_marks(res=resource, num_cards=cards)
builders = {
# ... existing platforms ...
"tpu": _tpu_marks,
}
```
(`hardware_test` calls `hardware_marks` internally, so both will pick up the change.)
(`hardware_test` / `hardware_marks` both go through `_marks_for_platform`.)
4. **(Recommended) Add a test using your new markers**:
```python
@hardware_test(
@@ -220,12 +208,13 @@ If you want to add support for a new platform (e.g., "tpu" for a new accelerator
```
**Summary**:
- Add pytest markers for your new platform/resources
- Implement a marker function (`xxx_marks`)
- Plug into `hardware_marks`
- You're done: tests using `@hardware_test` or `hardware_marks` with your platform now automatically get the correct markers, distribution, and isolation!
See code in `vllm-omni/tests/helpers/mark.py` for existing examples (`cuda_marks`, `rocm_marks`, `npu_marks`).
- Add pytest markers for your new platform/resources
- Implement a private marker function (`_xxx_marks`)
- Register it in `_marks_for_platform`
- You're done: tests using `@hardware_test` or `hardware_marks` with your platform now automatically get the correct markers and `cards_{n}` filtering.
See code in `vllm-omni/tests/helpers/mark.py` for existing examples (`_cuda_marks`, `_rocm_marks`, `_npu_marks`).
## Test case style
@@ -237,47 +226,46 @@ L1 and L2 level testing form the foundation of the quality assurance system. L1
L2 level testing builds upon L1 by introducing GPU resources and verifying that the end-to-end (E2E) process of the model in basic deployment scenarios is smooth. For example, it uses dummy models to confirm that core interfaces like the inference pipeline, output format, and streaming response work properly. The common goal of these two levels is to provide developers with rapid feedback, discovering and fixing issues early in the development cycle.
#### 1.2 Testing Content and Scope
- ***L1 (Unit & Logic Testing)***:
- - ***Scope***: Tests internal functions and methods of core components such as `entrypoints`, `models`.
- ***Focus***: Branch coverage, exception handling, algorithm logic correctness. Does not involve external dependencies or the complete service stack.
- ***Time Cost***: Execution time is controlled within ***15 minutes*** to ensure fast feedback.
- ***L2 (Basic End-to-End Testing)***:
- - ***Scope***: Covers two basic deployment scenarios: `online` (serving) and `offline` (inference).
- ***Focus***: Uses `dummy` weights (via deploy YAML patching at `core_model`) or lightweight real models to verify that the entire chain from request input to result output works normally, including output data structure, streaming (stream) support, and **cheap payload checks** at `--run-level core_model` (see below). Also includes some unit tests that require launching independent service instances.
- ***L2 response validation (`core_model`)***: Implemented in `tests/helpers/assertions.py` and invoked by `OpenAIClientHandler` based on `--run-level`. At L2 we require **request success** plus minimal output sanity—not full accuracy:
- **Speech / TTS** (`assert_audio_speech_response`): decoded audio must be present and non-empty (or exceed `min_audio_bytes` when set in `request_config`); `response_format` must match the returned content-type (e.g. `wav`, `pcm`). Whisper transcript similarity, PCM HNR, and preset-voice gender checks run only at L3+.
- **Diffusion** (`assert_diffusion_response`): at least one non-empty image, video, or audio artifact. Resolution/frame-count parity and other deep checks run only at L3+.
- **Omni multimodal** (`assert_omni_response`): L2 asserts successful completion; keyword, transcript, and cross-modal similarity checks run only at L3+.
- ***Characteristic***: Requires ***GPU*** resources to perform model computations.
- ***L1 (Unit & Logic Testing)***:
- ***Scope***: Tests internal functions and methods of core components such as `entrypoints`, `models`.
- ***Focus***: Branch coverage, exception handling, algorithm logic correctness. Does not involve external dependencies or the complete service stack.
- ***Time Cost***: Execution time is controlled within ***15 minutes*** to ensure fast feedback.
- ***L2 (Basic End-to-End Testing)***:
- ***Scope***: Covers two basic deployment scenarios: `online` (serving) and `offline` (inference).
- ***Focus***: Uses `dummy` weights (via deploy YAML patching at `core_model`) or lightweight real models to verify that the entire chain from request input to result output works normally, including output data structure, streaming (stream) support, and **cheap payload checks** at `--run-level core_model` (see below). Also includes some unit tests that require launching independent service instances.
- ***L2 response validation (`core_model`)***: Implemented in `tests/helpers/assertions.py` and invoked by `OpenAIClientHandler` based on `--run-level`. At L2 we require **request success** plus minimal output sanity—not full accuracy:
- **Speech / TTS** (`assert_audio_speech_response`): decoded audio must be present and non-empty (or exceed `min_audio_bytes` when set in `request_config`); `response_format` must match the returned content-type (e.g. `wav`, `pcm`). Whisper transcript similarity, PCM HNR, and preset-voice gender checks run only at L3+.
- **Diffusion** (`assert_diffusion_response`): at least one non-empty image, video, or audio artifact. Resolution/frame-count parity and other deep checks run only at L3+.
- **Omni multimodal** (`assert_omni_response`): L2 asserts successful completion; keyword, transcript, and cross-modal similarity checks run only at L3+.
- ***Characteristic***: Requires ***GPU*** resources to perform model computations.
#### 1.3 Test Directory and Execution Files
A clear directory structure is key to managing test cases efficiently. See the **Test Dir** column in [Test System Overview](./test_system_overview.md) for the canonical placement rules.
- ***L1 (component / unit)***: `tests/{component_name}/…` mirroring `vllm_omni/{component_name}/` (e.g. `tests/diffusion/`, `tests/entrypoints/`). Do **not** invent new top-level `tests/` folders unrelated to a component.
- ***L2 (basic model E2E)***:
- Online serving: `tests/e2e/online_serving/test_{model_name}.py`
- Offline inference: `tests/e2e/offline_inference/test_{model_name}.py`
- ***Feature-level integration*** (cross-cutting, not a single-model smoke): `tests/e2e/features/<feature>/` (e.g. `fullduplex/`, `custom_pipeline/`, `rlhf_test/`).
- ***L1 (component / unit)***: `tests/{component_name}/…` mirroring `vllm_omni/{component_name}/` (e.g. `tests/diffusion/`, `tests/entrypoints/`). Do **not** invent new top-level `tests/` folders unrelated to a component.
- ***L2 (basic model E2E)***:
- Online serving: `tests/e2e/online_serving/test_{model_name}.py`
- Offline inference: `tests/e2e/offline_inference/test_{model_name}.py`
- ***Feature-level integration*** (cross-cutting, not a single-model smoke): `tests/e2e/features/<feature>/` (e.g. `fullduplex/`, `custom_pipeline/`, `rlhf_test/`).
#### 1.4 Execution Method and Example
- ***Trigger Timing***: **`PR with ready label`**. That is, when a developer adds a "ready for review" or similar label to a PR on platforms like GitHub, L1 and L2 tests are automatically triggered.
- ***Diff-aware step skipping***: On L2, **E2E Test** jobs may be omitted at pipeline upload when the PR diff does not touch their [`source_file_dependencies`](ci_settings.md#step-filtering) prefixes; non-E2E groups still always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
- ***Execution Environment***: L1 uses ***CPU*** environment; L2 requires ***GPU*** environment.
- ***Script Example***:
- ***Trigger Timing***: **`PR with ready label`**. That is, when a developer adds a "ready for review" or similar label to a PR on platforms like GitHub, L1 and L2 tests are automatically triggered.
- ***Diff-aware step skipping***: On L2, **E2E Test** jobs may be omitted at pipeline upload when the PR diff does not touch their [`source_file_dependencies`](ci_settings.md#step-filtering) prefixes; non-E2E groups still always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
- ***Execution Environment***: L1 uses ***CPU*** environment; L2 requires ***GPU*** environment.
- ***Script Example***:
<details>
<summary> L1 Test Examples</summary>
Examples from `tests/model_executor/models/qwen2_5_omni/test_audio_length.py`
```python
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import pytest
@@ -331,6 +319,7 @@ def test_cap_and_align_mel_length_no_mismatch(repeats, code_len, max_mel_frames)
if max_mel_frames is not None and int(max_mel_frames) > 0 and int(max_mel_frames) >= repeats:
assert target_mel_len <= int(max_mel_frames)
```
</details>
<details>
@@ -338,7 +327,7 @@ def test_cap_and_align_mel_length_no_mismatch(repeats, code_len, max_mel_frames)
You can refer to Test Examples in the L3 section to see example test cases that incorporate both L2 and L3 testing logic.
</details>
- - ***Run Command***:
- ***Run Command***:
`pytest -s -v /tests/e2e/online_serving/test_{model_name}.py`
`pytest -s -v -m 'core_model and cpu' --run-level=core_model`
@@ -351,36 +340,34 @@ L3 level testing executes after code is merged into the main branch. Its core pu
. It acts as the "quality gatekeeper" for the main branch, ensuring that no merge breaks the core capabilities of the model service. Testing needs to provide clear conclusions within a relatively short time (<30min), balancing test depth with feedback speed.
#### 2.2 Testing Content and Scope
- ***Deployment Scenarios***: Covers richer `online` and `offline` deployment configurations, which may include different hardware configurations, batch sizes, concurrency levels, etc.
- ***Core Verification***:
- 1. ***Inference Functionality***: Ensures real models can perform forward computation normally and return results.
2. ***Accuracy Compliance***: Verifies that the model's evaluation metrics (e.g., accuracy) meet the expected baseline, preventing code changes from introducing accuracy issues.
3. ***Important Performance***: Verifies whether performance (e.g., P99 latency, throughput) in core scenarios meets preset thresholds.
- ***L3 response validation (`advanced_model`)***: At `--run-level advanced_model`, `tests/helpers/assertions.py` adds semantic checks on top of L2 payload gates (see L1 & L2 above):
- **Speech / TTS** (`assert_audio_speech_response`): Whisper transcript of returned audio vs `request_config["input"]` (cosine similarity &gt; 0.9 when input text is set); optional `min_audio_bytes` floor; PCM harmonic-to-noise ratio when `response_format` is `pcm`; **preset voice gender** via `_assert_preset_voice_gender_from_audio` when `voice` matches a known preset in `_PRESET_VOICE_GENDER_MAP` (pitch/F0-based classifier; skipped for unknown voices or `pcm` output).
- **Omni multimodal** (`assert_omni_response`): non-empty text/audio outputs; `key_words` in transcript or text; text–audio similarity / containment; preset **`speaker`** gender check (same helper as TTS).
- **Diffusion** (`assert_*_diffusion_response`): image/video dimension and frame-count parity with request parameters where configured.
- **Accuracy suites**: pixel/video similarity and metric baselines under `/tests/e2e/accuracy/` (separate from inline helper assertions).
- ***Deployment Scenarios***: Covers richer `online` and `offline` deployment configurations, which may include different hardware configurations, batch sizes, concurrency levels, etc.
- ***Core Verification***:
1. ***Inference Functionality***: Ensures real models can perform forward computation normally and return results.
2. ***Accuracy Compliance***: Verifies that the model's evaluation metrics (e.g., accuracy) meet the expected baseline, preventing code changes from introducing accuracy issues.
3. ***Important Performance***: Verifies whether performance (e.g., P99 latency, throughput) in core scenarios meets preset thresholds.
- ***L3 response validation (`advanced_model`)***: At `--run-level advanced_model`, `tests/helpers/assertions.py` adds semantic checks on top of L2 payload gates (see L1 & L2 above):
- **Speech / TTS** (`assert_audio_speech_response`): Whisper transcript of returned audio vs `request_config["input"]` (cosine similarity &gt; 0.9 when input text is set); optional `min_audio_bytes` floor; PCM harmonic-to-noise ratio when `response_format` is `pcm`; **preset voice gender** via `_assert_preset_voice_gender_from_audio` when `voice` matches a known preset in `_PRESET_VOICE_GENDER_MAP` (pitch/F0-based classifier; skipped for unknown voices or `pcm` output).
- **Omni multimodal** (`assert_omni_response`): non-empty text/audio outputs; `key_words` in transcript or text; text–audio similarity / containment; preset **`speaker`** gender check (same helper as TTS).
- **Diffusion** (`assert_*_diffusion_response`): image/video dimension and frame-count parity with request parameters where configured.
- **Accuracy suites**: pixel/video similarity and metric baselines under `/tests/e2e/accuracy/` (separate from inline helper assertions).
#### 2.3 Test Directory and Execution Files
- ***Functional / model E2E***:
- Online serving: `tests/e2e/online_serving/test_{model_name}_expansion.py`
- Offline inference: `tests/e2e/offline_inference/test_{model_name}_expansion.py`
- Accuracy / similarity: `tests/e2e/accuracy/`
- (`_expansion.py` holds broader scenario coverage than the L2 smoke file.)
- ***Feature-level integration***: `tests/e2e/features/<feature>/` (not under a new top-level `tests/` directory).
- ***Functional / model E2E***:
- Online serving: `tests/e2e/online_serving/test_{model_name}_expansion.py`
- Offline inference: `tests/e2e/offline_inference/test_{model_name}_expansion.py`
- Accuracy / similarity: `tests/e2e/accuracy/`
- (`_expansion.py` holds broader scenario coverage than the L2 smoke file.)
- ***Feature-level integration***: `tests/e2e/features/<feature>/` (not under a new top-level `tests/` directory).
#### 2.4 Execution Method and Example
- ***Trigger Timing***: **`PR Merged`**. Automatically triggered after code review is approved and merged into the main branch (typically via `merge-test` label on the PR before merge).
- ***Diff-aware step skipping***: Same [`source_file_dependencies`](ci_settings.md#step-filtering) mechanism as L2—**all E2E Test** leaf steps are diff-gated; other L3 groups always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
- ***Execution Environment***: ***GPU*** servers.
- ***Script Example***:
- ***Trigger Timing***: **`PR Merged`**. Automatically triggered after code review is approved and merged into the main branch (typically via `merge-test` label on the PR before merge).
- ***Diff-aware step skipping***: Same [`source_file_dependencies`](ci_settings.md#step-filtering) mechanism as L2—**all E2E Test** leaf steps are diff-gated; other L3 groups always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
- ***Execution Environment***: ***GPU*** servers.
- ***Script Example***:
???+ example "Test Examples"
@@ -496,9 +483,9 @@ L3 level testing executes after code is merged into the main branch. Its core pu
**Audio output debugging**: Deep validation may transcribe returned audio via `convert_audio_bytes_to_text` (Whisper). If an audio keyword or text–audio similarity assertion fails, set `VLLM_OMNI_KEEP_REQUEST_MEDIA=1` before running pytest to keep the intermediate WAV files for inspection (see [Test helper environment variables](test_system_overview.md#test-helper-environment-variables)).
- ***Run Command (L3 merge)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}.py -m advanced_model --run-level=advanced_model`
- ***Run Command (L3 merge)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}.py -m advanced_model --run-level=advanced_model`
- ***Run Command (L4 nightly expansion)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}_expansion.py -m full_model --run-level=full_model`
- ***Run Command (L4 nightly expansion)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}_expansion.py -m full_model --run-level=full_model`
### L4 Level Testing - Full Functionality, Performance, and Documentation Testing
@@ -506,27 +493,25 @@ L3 level testing executes after code is merged into the main branch. Its core pu
L4 level testing is a comprehensive quality audit before a version release. It expands upon L3, executing ***full*** functional scenarios, conducting systematic ***performance stress tests***, and simultaneously verifying the correctness of accompanying ***example documentation***. Its purpose is to perform deep validation of the system during off-peak nighttime hours, providing quality trend reports for daytime development and data support for release decisions.
#### 3.2 Testing Content and Scope
- ***Full Functionality Testing***: Executes all test cases defined in `test_{model_name}_expansion.py`, covering all implemented features, positive flows, boundary conditions, and exception handling.
- ***Performance Testing***: Uses `tests/dfx/perf/tests/test_qwen3_omni_*.json` (Omni), `test_tts.json` / `test_voxcpm2.json` / `test_higgs_audio_v3.json` (TTS), and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (passed to `run_benchmark.py` or `run_diffusion_benchmark.py` via `--test-config-file` in nightly **Perf Test** steps) to drive throughput, latency, and memory benchmarks. Each JSON **case** may declare an optional top-level **`mark`** array: exactly one ``hardware_marks`` object plus pytest marker name strings (`full_model`, `omni` / `tts` / `diffusion`, …). Runners attach those marks to each parametrized `(server, benchmark index)` pair so **local** bulk runs can filter with `-m` (for example `-m "full_model and H100 and diffusion"`). Nightly CI perf jobs select workloads by **`--test-config-file`**, not `-m`. Details are in the Performance Tests example below.
- ***Documentation Testing***: Verifies whether the example code provided to users is runnable and its results match the description.
- ***Full Functionality Testing***: Executes all test cases defined in `test_{model_name}_expansion.py`, covering all implemented features, positive flows, boundary conditions, and exception handling.
- ***Performance Testing***: JSON configs under `tests/dfx/perf/tests/` drive Omni, TTS, and diffusion serving benchmarks via `run_benchmark.py` or `run_diffusion_benchmark.py`, measuring throughput, latency, and related metrics under configured workloads.
- ***Documentation Testing***: Verifies whether the example code provided to users is runnable and its results match the description.
#### 3.3 Test Directory and Execution Files
- ***Functional Testing***: Same directories as L3.
- ***Performance Test Configuration***: `tests/dfx/perf/tests/test_qwen3_omni_*.json`, `test_tts.json`, `test_voxcpm2.json`, `test_higgs_audio_v3.json`, and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (e.g. `test_qwen_image_vllm_omni.json`, `test_cosmos3_vllm_omni.json`). Optional per-case `mark` for local `-m` filtering is documented in Section 3.4 Performance Tests.
- ***Documentation Example Tests***:
- `tests/examples/online_serving/test_{model_name}.py`
- `tests/examples/offline_inference/test_{model_name}.py`
- ***Functional Testing***: Same directories as L3.
- ***Performance Test Configuration***: `tests/dfx/perf/tests/test_qwen3_omni_*.json`, `test_tts.json`, `test_voxcpm2.json`, `test_higgs_audio_v3.json`, and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (e.g. `test_qwen_image_vllm_omni.json`, `test_cosmos3_vllm_omni.json`). Optional per-case `mark` for local `-m` filtering is documented in Section 3.4 Performance Tests.
- ***Documentation Example Tests***:
- `tests/examples/online_serving/test_{model_name}.py`
- `tests/examples/offline_inference/test_{model_name}.py`
#### 3.4 Execution Method and Example
- ***Trigger Timing***: **`Nightly`**, automatically executed every night.
- ***Execution Environment***: ***GPU*** server clusters to meet the resource demands of performance testing.
- ***Script Example***:
- ***Trigger Timing***: **`Nightly`**, automatically executed every night.
- ***Execution Environment***: ***GPU*** server clusters to meet the resource demands of performance testing.
- ***Script Example***:
??? example "Test Examples: Documentation Example Tests"
@@ -540,7 +525,7 @@ L4 level testing is a comprehensive quality audit before a version release. It e
--8<-- "docs/contributing/ci/test_examples/l4_functionality_tests.inc.md"
- ***Run Command***: (Specific commands would depend on the performance testing tool and configuration defined in `nightly.json`).
- ***Run Command***: (Specific commands would depend on the performance testing tool and configuration defined in `nightly.json`).
### L5 Level Testing - Stability and Reliability Testing
@@ -548,41 +533,43 @@ L4 level testing is a comprehensive quality audit before a version release. It e
L5 level testing focuses on the performance of model services under ***long-running*** and ***abnormal fault*** scenarios. It aims to uncover deep-seated issues that only manifest under sustained pressure or extreme conditions, such as memory leaks, resource contention, gradual performance degradation, and lack of fault tolerance mechanisms. This is the final, yet crucial, line of defense for ensuring service high availability and production environment robustness.
#### 4.2 Testing Content and Scope
- ***Long-term Stability (Stability) Testing***: Uses JSON under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` and `test_wan22.json`) to run the service under moderate load for an extended period (e.g., over 12 hours), monitoring whether metrics like memory/VRAM usage, response time, and throughput degrade over time, and whether the service process remains stable.
- ***Reliability Testing***: Uses pytest suites under `tests/dfx/reliability/` to inject controlled faults against a **live** `vllm_omni serve` instance (same **`omni_server` / `omni_server_function`** fixture style as E2E). Current suites emphasize **GPU memory pressure** (CUDA sidecar “memory hog”), **worker / runtime process kill** (`SIGKILL` on `VLLM::Worker` for Qwen3-Omni, `multiprocessing.spawn` for Wan2.2 video workers, or `vLLM-Omni::` for HunyuanImage DiT workers), **large multimodal chat**, **`/v1/videos`**, **`/v1/images/generations`**, or **`/v1/audio/speech`** jobs under OOM, **`/health` → 503** and **fast-fail / non-hanging concurrent** requests after kill, and **OpenAI-style 5xx error contracts** (e.g. text vs text+audio under OOM). **Post-fault recovery** checks exist where enabled (some cases may be `skip` while issues are tracked). See the Reliability `<details>` block in Section 4.4 for file-level responsibilities and CI markers (`slow`, `hardware_test`, POSIX-only kill).
- ***Long-term Stability (Stability) Testing***: Uses JSON under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` and `test_wan22.json`) to run the service under moderate load for an extended period (e.g., over 12 hours), monitoring whether metrics like memory/VRAM usage, response time, and throughput degrade over time, and whether the service process remains stable.
- ***Reliability Testing***: Uses pytest suites under `tests/dfx/reliability/` to inject controlled faults against a **live** `vllm_omni serve` instance (same **`omni_server` / `omni_server_function`** fixture style as E2E). Current suites emphasize **GPU memory pressure** (CUDA sidecar “memory hog”), **worker / runtime process kill** (`SIGKILL` on `VLLM::Worker` for Qwen3-Omni, `multiprocessing.spawn` for Wan2.2 video workers, or `vLLM-Omni::` for HunyuanImage DiT workers), **large multimodal chat**, **`/v1/videos`**, **`/v1/images/generations`**, or **`/v1/audio/speech`** jobs under OOM, **`/health` → 503** and **fast-fail / non-hanging concurrent** requests after kill, and **OpenAI-style 5xx error contracts** (e.g. text vs text+audio under OOM). **Post-fault recovery** checks exist where enabled (some cases may be `skip` while issues are tracked). See Section 4.3 for file-level responsibilities and Section 4.4 Test Examples (Reliability) for CI markers (`slow`, `hardware_test`, POSIX-only kill).
#### 4.3 Test Directory and Execution Files
- ***Stability Test Configuration***: `tests/dfx/stability/tests/test_qwen3_omni.json`, `tests/dfx/stability/tests/test_wan22.json` (one JSON per model / runner family)
- ***Reliability Test Suite*** (`tests/dfx/reliability/`):
- `test_reliability_qwen3_omni.py` — Qwen3-Omni chat / multimodal reliability (GPU OOM, process kill, recovery, error contract under `--async-chunk` vs default).
- `test_reliability_wan22.py` — Wan2.2 I2V video API reliability (`/v1/videos` under OOM and process kill, recovery).
- `test_reliability_hunyuan_image.py` — HunyuanImage-3.0-Instruct DiT-only reliability (`/v1/images/generations` under OOM and process kill; deploy `hunyuan_image3_dit.yaml`, H100 × 4).
- `test_reliability_voxcpm2.py` — VoxCPM2 TTS reliability (`/v1/audio/speech` under OOM and process kill; deploy `voxcpm2.yaml`, L4 × 1).
- `helpers.py` — Shared primitives used by current suites: raw HTTP probes for `/v1/chat/completions` and `/health`, OpenAI-style error parsing, GPU OOM sidecar (`inject_gpu_oom` / `stop_gpu_oom_hogs`), and `pgrep`-based process-kill injector construction (`make_process_kill_fault_injector`).
- `conftest.py` — `fault_injector` and `omni_server_after_fault` / `omni_server_after_fault_function` fixtures to run a callable **after** the server is ready.
- `README.md` — Short local run commands for this directory.
- ***Stability Test Configuration***: `tests/dfx/stability/tests/test_qwen3_omni.json`, `tests/dfx/stability/tests/test_wan22.json` (one JSON per model / runner family)
- ***Reliability Test Suite*** (`tests/dfx/reliability/`):
- `test_reliability_qwen3_omni.py` — Qwen3-Omni chat / multimodal reliability (GPU OOM, process kill, recovery, error contract under `--async-chunk` vs default).
- `test_reliability_wan22.py` — Wan2.2 I2V video API reliability (`/v1/videos` under OOM and process kill, recovery).
- `test_reliability_hunyuan_image.py` — HunyuanImage-3.0-Instruct DiT-only reliability (`/v1/images/generations` under OOM and process kill; deploy `hunyuan_image3_dit.yaml`, H100 × 4).
- `helpers.py` — Shared primitives used by current suites: raw HTTP probes for `/v1/chat/completions` and `/health`, OpenAI-style error parsing, GPU OOM sidecar (`inject_gpu_oom` / `stop_gpu_oom_hogs`), and `pgrep`-based process-kill injector construction (`make_process_kill_fault_injector`).
- `conftest.py` — `fault_injector` and `omni_server_after_fault` / `omni_server_after_fault_function` fixtures to run a callable **after** the server is ready.
- `README.md` — Short local run commands for this directory.
#### 4.4 Execution Method and Example
- ***Trigger Timing***: **`Weekly`** (weekly) or **`Days before Release`** (several days before a major release). Due to long execution times, the frequency is lower.
- ***Execution Environment***: ***GPU*** servers, requiring a stable and exclusive testing environment.
- ***Script Example***:
- ***Trigger Timing***: **`Weekly`** (weekly) or **`Days before Release`** (several days before a major release). Due to long execution times, the frequency is lower.
- ***Run Command***:
- ***Stability***: `pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/test_stability_wan22.py` (or add `test_stability_<model>.py` alongside a matching JSON config)
- ***Reliability***: `pytest -s -v tests/dfx/reliability/test_reliability_<model>.py -m slow` (current suites: `qwen3_omni`, `wan22`, `hunyuan_image`).
- ***Script Example***:
<details>
<summary> Test Examples</summary>
When you want to add L5-level stability test cases, add or extend the appropriate JSON file under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` for Omni bench traffic, or `test_wan22.json` for diffusion `/v1/videos` workloads). The following illustrates the Qwen3-Omni shape:
##### Stability
When you want to add L5-level stability test cases, add or extend the appropriate JSON file under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` for Omni bench traffic, or `test_wan22.json` for diffusion `/v1/videos` workloads). Pair the JSON with `tests/dfx/stability/scripts/test_stability_<model>.py`. The following illustrates the Qwen3-Omni shape:
```json
{
"test_name": "test_qwen3_omni_stability",
"server_params": {
"model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
"stage_config_name": "qwen3_omni.yaml"
"stage_config_name": "qwen3_omni_moe.yaml"
},
"benchmark_params": [
{
@@ -601,112 +588,61 @@ When you want to add L5-level stability test cases, add or extend the appropriat
}
```
##### Parameter Explanation
###### Parameter Explanation
***Overview***
| Field | Required | Description |
| ---------------- | -------- | --------------------------------------------------------------------------- |
| test_name | Yes | Unique identifier for the stability test case |
| server_params | Yes | Server-side configuration parameters (model, stage configuration, etc.) |
| server_params | Yes | Server-side configuration parameters (model, deploy configuration, etc.) |
| benchmark_params | Yes | Stability benchmark running parameters (supports multiple configurations) |
##### server_params Configuration
###### server_params Configuration
###### Basic Parameters
Basic parameters:
| Parameter | Required | Example | Description |
| ----------------- | -------- | ---------------------------------- | ----------------------------------- |
| model | Yes | "Qwen/Qwen3-Omni-30B-A3B-Instruct" | Model name or path |
| stage_config_name | Yes | "qwen3_omni.yaml" | Stage configuration file name |
| stage_config_name | Yes | "qwen3_omni_moe.yaml" | Deploy configuration file name |
###### Dynamic Configuration (update/delete)
Supports incremental modifications based on the basic configuration:
Dynamic configuration (`update` / `delete`) supports incremental modifications based on the basic configuration:
| Operation | Description |
| --------- | ------------------------------------ |
| update | Update or add configuration items |
| delete | Delete specified configuration items |
***Example***:
You can refer to Test Examples in L4 §3.4
***Example***: You can refer to Test Examples in L4 §3.4.
##### benchmark_params Configuration
###### benchmark_params Configuration
For stability testing, the key parameters are:
- **duration_sec**: Total duration (in seconds) during which benchmark traffic is sent. The stability benchmark will keep sending batches until this duration is reached.
- **request_rate** / **max_concurrency**: Exactly one of them must be specified. They control how the traffic is generated for each batch:
- `request_rate`: Number of requests per second. The benchmark will send `num_prompts_per_batch` requests at the given rate.
- `max_concurrency`: Maximum number of concurrent requests. When this is used, `request_rate` is set to `inf` internally.
- **num_prompts_per_batch**: Number of prompts sent in each batch. Multiple batches will be executed sequentially within `duration_sec`.
- **duration_sec**: Total duration (in seconds) during which benchmark traffic is sent. The stability benchmark will keep sending batches until this duration is reached.
- **request_rate** / **max_concurrency**: Exactly one of them must be specified. They control how the traffic is generated for each batch:
- `request_rate`: Number of requests per second. The benchmark will send `num_prompts_per_batch` requests at the given rate.
- `max_concurrency`: Maximum number of concurrent requests. When this is used, `request_rate` is set to `inf` internally.
- **num_prompts_per_batch**: Number of prompts sent in each batch. Multiple batches will be executed sequentially within `duration_sec`.
All other optional parameters follow the same rules as in L4 §3.4.
</details>
##### Reliability
<details>
<summary> Reliability test suite (<code>tests/dfx/reliability</code>)</summary>
Add pytest modules under `tests/dfx/reliability/` (for example `test_reliability_qwen3_omni.py`, `test_reliability_wan22.py`, `test_reliability_hunyuan_image.py`). Reuse `helpers.py` and `conftest.py` rather than duplicating OOM / process-kill / raw HTTP plumbing.
##### Purpose and relationship to stability
Reliability tests are **short fault-injection** integration runs (L5 **(b)** in `tests/dfx/reliability/README.md`). They complement **stability** JSON-driven long runs: instead of hours of steady traffic, they **perturb** the server (GPU OOM sidecar, fatal signals on selected processes) and check **failure mode** and **latency bounds** (e.g. chat or `/v1/videos` must not hang under concurrent fault-time load).
##### Directory layout
| Path | Responsibility |
| ---- | -------------- |
| `helpers.py` | Shared helpers used by current reliability suites: raw `POST`/`GET` probes (`/v1/chat/completions`, `/health`), OpenAI error parsing (`extract_openai_error_contract_from_bytes`), GPU OOM sidecar lifecycle (`inject_gpu_oom`, `stop_gpu_oom_hogs`), and process-kill injector builder (`make_process_kill_fault_injector`). |
| `conftest.py` | Pytest fixtures: indirect `fault_injector`, `omni_server_after_fault` / `omni_server_after_fault_function` (run injector after server is ready, then yield server). |
| `test_reliability_qwen3_omni.py` | Qwen3-Omni: OOM vs **text vs text+audio** error contract, large multimodal chat under OOM, concurrent pressure, **SIGKILL** on `VLLM::Worker`, `/health` → 503 + fast-fail + concurrent chat; optional OOM recovery scenario (may be skipped while tracked in issues). |
| `test_reliability_wan22.py` | Wan2.2 I2V: large `/v1/videos` under OOM, **SIGKILL** on `multiprocessing.spawn` chain, health / fast-fail / concurrent video requests; optional recovery test (may be skipped). |
| `test_reliability_hunyuan_image.py` | HunyuanImage DiT-only: large `/v1/images/generations` under OOM, **SIGKILL** on `vLLM-Omni::` workers and serve/tree targets, health / fast-fail / concurrent image requests; some OOM/recovery cases may be skipped while tracked in issues. |
| `test_reliability_voxcpm2.py` | VoxCPM2: `/v1/audio/speech` under OOM (error contract), **SIGKILL** on `VLLM::` workers and serve/tree targets, health / fast-fail / concurrent speech requests; some OOM cases may be skipped while tracked in issues. |
| `README.md` | Minimal run / collect examples. |
##### Parametrization and markers
###### Parametrization and markers
- Each test module defines a **`RELIABILITY_SCENARIOS`** list (`test_name`, `server_params`: model, `stage_config_name` or diffusion `server_args`, etc.). **`create_reliability_omni_server_params()`** in `tests/dfx/conftest.py` resolves stage paths (including XPU substitutions where applicable) and builds **`OmniServerParams`** lists consumed by **`@pytest.mark.parametrize(..., indirect=True)`** on `omni_server` or `omni_server_function`.
- Cases are tagged **`@pytest.mark.slow`** for weekly / selective CI. GPU-heavy suites use **`@hardware_test(res={"cuda": "H100"}, num_cards=...)`** or **`@hardware_test(res={"cuda": "L4"}, num_cards=1)`** (Qwen3-Omni **2**× H100; Wan2.2 **1**× H100; HunyuanImage DiT **4**× H100; VoxCPM2 **1**× L4).
- Cases are tagged **`@pytest.mark.slow`** for weekly / selective CI. GPU-heavy suites use **`@hardware_test(res={"cuda": "H100"}, num_cards=...)`** (Qwen3-Omni **2**× H100; Wan2.2 **1**× H100; HunyuanImage DiT **4**× H100).
- **Process-kill** tests use **`@pytest.mark.skipif(os.name == "nt", ...)`** because injection uses POSIX **`pgrep` / `kill`**.
##### CI trigger
Weekly Buildkite (`.buildkite/cuda/test-weekly.yml`) runs one step per model suite (trigger: `WEEKLY=1` or PR label `weekly-test`), for example:
| Buildkite step | Test file | CI hardware |
| -------------- | --------- | ----------- |
| Reliability Test - qwen3-omni | `test_reliability_qwen3_omni.py` | H100 × 2 (`mithril-h100-pool`) |
| Reliability Test - wan22 | `test_reliability_wan22.py` | H100 × 2 (`mithril-h100-pool`) |
| Reliability Test - hunyuan-image | `test_reliability_hunyuan_image.py` | H100 × 4 (`mithril-h100-pool`) |
| Reliability Test - voxcpm2 | `test_reliability_voxcpm2.py` | L4 × 1 (`gpu_1_queue`) |
```bash
pytest -s -v tests/dfx/reliability/test_reliability_qwen3_omni.py -m "slow"
pytest -s -v tests/dfx/reliability/test_reliability_wan22.py -m "slow"
pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py -m "slow"
pytest -s -v tests/dfx/reliability/test_reliability_voxcpm2.py -m "slow"
```
##### Local commands
```bash
pytest --collect-only tests/dfx/reliability
pytest -s -v tests/dfx/reliability/test_reliability_qwen3_omni.py -m slow
pytest -s -v tests/dfx/reliability/test_reliability_wan22.py -m slow
pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py -m slow
pytest -s -v tests/dfx/reliability/test_reliability_voxcpm2.py -m slow
```
##### Adding a new model suite
###### Adding a new model suite
1. Add `test_reliability_<model>.py` under `tests/dfx/reliability/`.
2. Define **`RELIABILITY_SCENARIOS`** and pass them through **`create_reliability_omni_server_params()`** with the correct deploy or e2e stage-config directory (same pattern as existing files).
3. Reuse **`helpers`** for OOM / kill / raw HTTP; prefer **`assert_fault_exception()`** and **`resolve_oom_device_spec()`** from `tests/dfx/conftest.py` for consistent device selection vs stage YAML.
2. Define **`RELIABILITY_SCENARIOS`** and pass them through **`create_reliability_omni_server_params()`** with the correct deploy or e2e deploy-config directory (same pattern as existing files).
3. Reuse **`helpers`** for OOM / kill / raw HTTP; prefer **`assert_fault_exception()`** and **`resolve_oom_device_spec()`** from `tests/dfx/conftest.py` for consistent device selection vs deploy YAML.
4. Register **`slow`** (and **`hardware_test`** if needed); extend **`.buildkite/cuda/test-weekly.yml`** when the suite should run in weekly L5.
</details>
- - ***Stability***: `pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/test_stability_wan22.py` (or add `test_stability_<model>.py` alongside a matching JSON config)
- ***Reliability***: `pytest -s -v tests/dfx/reliability/test_reliability_<model>.py -m slow` (current suites: `qwen3_omni`, `wan22`, `hunyuan_image`, `voxcpm2`; add `test_reliability_<suite>.py` and a matching step in `.buildkite/cuda/test-weekly.yml` for new models)
docs/contributing/model/adding_omni_model.md
+25 -14
View File
@@ -9,7 +9,7 @@ This guide walks through the process of adding a new multi-stage model to vLLM-O
3. [Step-by-Step Implementation](#step-by-step-implementation)
4. [Key Components](#key-components)
5. [Model Registration](#model-registration)
6. [Stage Configuration](#stage-configuration)
6. [Pipeline and Deploy Configuration](#pipeline-and-deploy-configuration)
7. [Stage Input Processors](#stage-input-processors)
8. [Testing](#testing)
9. [Adding a Model Recipe](#adding-a-model-recipe)
@@ -316,18 +316,28 @@ _OMNI_MODELS = {
The registry uses lazy loading, so the model class is imported only when needed.
## Stage Configuration
## Pipeline and Deploy Configuration
Create a YAML configuration file in `vllm_omni/deploy/`. For a complete example, see the [Qwen3-Omni configuration file](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml).
Define fixed topology in `vllm_omni/model_executor/models/<model>/pipeline.py`
with `PipelineConfig` and `StagePipelineConfig`, then register it in
`vllm_omni/config/pipeline_registry.py`. For a complete example, see the
[Qwen3-Omni pipeline](gh-file:vllm_omni/model_executor/models/qwen3_omni/pipeline.py).
### Key Configuration Fields
### Pipeline topology fields
- **`model_stage`**: Which stage to run ("thinker", "talker", "code2wav", etc.)
- **`model_arch`**: The model architecture name (must match registry)
- **`engine_input_source`**: List of stage IDs that provide input to this stage
- **`custom_process_input_func`**: Function to process inputs from previous stages
- **`final_output`**: Whether this stage produces the final output (True/False)
- **`final_output_type`**: Type of final output ("text", "audio", "image", etc.)
- **`model_stage`**: Logical stage name ("thinker", "talker", "code2wav", etc.)
- **`execution_type`**: Runtime family (`LLM_AR`, `LLM_GENERATION`, or `DIFFUSION`)
- **`model_arch`**: Stage architecture name, when different from the pipeline default
- **`input_sources`**: Upstream stage IDs that provide input to this stage
- **`custom_process_input_func`**: Function that processes inputs from previous stages
- **`final_output`**: Whether this stage produces user-facing output
- **`final_output_type`**: Output modality ("text", "audio", "image", etc.)
- **`owns_tokenizer`**: Whether this stage owns the pipeline tokenizer
Create the matching runtime defaults in `vllm_omni/deploy/<model>.yaml` and set
`PipelineConfig.default_deploy_config_name` to that file. The deploy YAML owns
GPU placement, memory sizing, parallelism, connectors, and sampling defaults;
it must not redefine topology fields.
## Stage Input Processors
@@ -374,7 +384,7 @@ The stage transition process follows these steps:
2. **Transition Detection**: The orchestrator checks if there's a next stage and calls `process_engine_inputs()` on it
3. **Input Processing**: The stage input processor configured in stage YAML (under `vllm_omni/model_executor/stage_input_processors/`) handles the transition:
3. **Input Processing**: The stage input processor registered by the pipeline (implemented under `vllm_omni/model_executor/stage_input_processors/`) handles the transition:
```python
def process_engine_inputs(
self, stage_list: list[Any], prompt: OmniTokensPrompt | TextPrompt = None
@@ -577,7 +587,7 @@ Adding a new model to vLLM-Omni involves:
1. **Create model directory structure** with stage implementations
2. **Implement unified model class** that orchestrates stages
3. **Register model** in `registry.py`
4. **Create stage configuration** YAML file
4. **Define pipeline topology and deploy defaults** in `pipeline.py` and `vllm_omni/deploy/`
5. **Implement stage input processors** for stage transitions
6. **Write tests** to verify functionality
7. **Add model recipe** to the [vllm-project/recipes](https://github.com/vllm-project/recipes) repository (see [Adding a Model Recipe](#adding-a-model-recipe) section)
@@ -590,7 +600,8 @@ For a complete reference implementation, see:
- **Thinker**: `vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_moe_thinker.py`
- **Talker**: `vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_moe_talker.py`
- **Code2Wav**: `vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_code2wav.py`
- **Stage config**: `vllm_omni/deploy/qwen3_omni_moe.yaml`
- **Pipeline topology**: `vllm_omni/model_executor/models/qwen3_omni/pipeline.py`
- **Deploy config**: `vllm_omni/deploy/qwen3_omni_moe.yaml`
- **Input processors**: `vllm_omni/model_executor/stage_input_processors/qwen3_omni.py`
- **Registry**: `vllm_omni/model_executor/models/registry.py`
- **Testing**: `vllm_omni/tests/e2e/offline_inference/test_qwen3_omni.py`
@@ -598,4 +609,4 @@ For a complete reference implementation, see:
For more information, see:
- [Architecture Overview](../../design/architecture_overview.md)
- [Supported Models](../../models/supported_models.md)
- [Stage Configuration Guide](../../configuration/stage_configs.md)
- [Pipeline and Deploy Configuration Guide](../../configuration/stage_configs.md)
docs/contributing/model/adding_tts_model.md
+115 -117
View File
@@ -20,7 +20,7 @@ pattern, refer to MOSS-TTS-Nano.
4. [Step-by-Step Implementation](#step-by-step-implementation)
5. [Key Components](#key-components)
6. [Model Registration](#model-registration)
7. [Stage Configuration](#stage-configuration)
7. [Pipeline and Deploy Configuration](#pipeline-and-deploy-configuration)
8. [Stage Input Processors](#stage-input-processors)
9. [Online Serving Integration](#online-serving-integration)
10. [Single-Stage Models](#single-stage-models)
@@ -78,7 +78,7 @@ Profile before optimizing.
new TTS model must pass all three levels:
| Layer | Catches | Tool |
|-------|---------|------|
| ------- | --------- | ------ |
| Offline RTF / duration | Throughput regressions, missing audio, wrong sample rate | `end2end.py`, pytest e2e |
| Browser streaming playback | Delta-vs-cumulative bugs, chunk boundary glitches, TTFP regressions | Gradio demo over `/v1/audio/speech?stream=true` |
| Concurrent requests | Per-request state leaks, codec window round-robin gaps | `max_num_seqs>1` smoke with 4+ parallel prompts |
@@ -94,10 +94,10 @@ the symptom is crosstalk or truncation under load, nothing in single-request tes
vLLM-Omni supports TTS models as multi-stage pipelines where each stage runs independently
and can be placed on different devices. Qwen3-TTS has two stages:
| Stage | Name | Input | Output |
|-------|------|-------|--------|
| 0 | Code Predictor (AR) | Text tokens | Discrete RVQ codec codes |
| 1 | Code2Wav (Decoder) | RVQ codec codes | Audio waveform |
| Stage | Name | Input | Output |
| ----- | ------------------- | --------------- | ------------------------ |
| 0 | Code Predictor (AR) | Text tokens | Discrete RVQ codec codes |
| 1 | Code2Wav (Decoder) | RVQ codec codes | Audio waveform |
Each stage is a separate model class configured independently via YAML. The two stages
are connected by the `async_chunk` framework, which enables inter-stage streaming for
@@ -170,7 +170,7 @@ and Qwen3-Omni).
When adding a new TTS model, create the following structure:
```
```text
vllm_omni/model_executor/models/
your_model_name/
__init__.py
@@ -201,7 +201,7 @@ it would not be picked up — the hub README is the documented surface.
**Qwen3-TTS reference files:**
| File | Purpose |
|------|---------|
| ------ | --------- |
| `models/qwen3_tts/qwen3_tts.py` | Unified model class |
| `models/qwen3_tts/qwen3_tts_code_predictor_vllm.py` | Stage 0 - optimized AR |
| `models/qwen3_tts/qwen3_tts_code2wav.py` | Stage 1 - decoder |
@@ -427,112 +427,93 @@ _OMNI_MODELS = {
The registry uses lazy loading - model classes are only imported when needed.
## Stage Configuration
## Pipeline and Deploy Configuration
Each stage has a `worker_type` that determines how it is scheduled:
Define fixed topology in `models/<model>/pipeline.py`. The execution type
selects the scheduler and worker family:
- `worker_type: ar` - autoregressive stage, uses `OmniARScheduler` with PagedAttention
- `worker_type: generation` - non-AR stage (e.g. decoder), uses `OmniGenerationScheduler`
- `StageExecutionType.LLM_AR` uses the autoregressive runtime with PagedAttention.
- `StageExecutionType.LLM_GENERATION` uses the generation runtime for stages such as audio decoders.
Key configuration fields:
Key topology fields belong to `StagePipelineConfig`:
| Field | Description |
|-------|-------------|
| `model_stage` | Which stage to initialize (`ar_stage`, `decoder`, etc.) |
| `model_arch` | Architecture name, must match `registry.py` |
| `engine_input_source` | List of upstream stage IDs that provide input (e.g. `[0]`) |
| `engine_output_type` | Output type: `latent` for intermediate, `audio` for final |
| `custom_process_next_stage_input_func` | Async chunk processor function path (streaming only) |
| `final_output` | Whether this stage produces the final user-facing output |
| `final_output_type` | Type of final output (`audio`, `text`, etc.) |
| ------- | ------------- |
| `model_stage` | Logical stage name (`ar_stage`, `decoder`, etc.). |
| `execution_type` | Runtime family for this stage. |
| `input_sources` | Upstream stage IDs that provide input. |
| `model_arch` | Stage-specific architecture name, when it differs from the pipeline default. |
| `engine_output_type` | Output representation such as `latent` or `audio`. |
| `custom_process_next_stage_input_func` | Processor for full-payload stage handoff. |
| `async_chunk_process_next_stage_input_func` | Processor for streaming chunk handoff. |
| `final_output` / `final_output_type` | Whether this stage produces user-facing output and its modality. |
| `owns_tokenizer` | Whether this stage supplies the pipeline tokenizer. |
!!! note
New in-tree models should define frozen topology in `models/<model>/pipeline.py`,
register it in `vllm_omni/config/pipeline_registry.py`, and put deployment
knobs in `vllm_omni/deploy/<model>.yaml`. Use `--deploy-config` to load a
custom deployment YAML.
For example, a two-stage TTS topology can be defined as:
### Batch mode
```python
from vllm_omni.config.stage_config import (
PipelineConfig,
StageExecutionType,
StagePipelineConfig,
)
```yaml
# stage_configs/your_model_name.yaml
_PROC = "vllm_omni.model_executor.stage_input_processors.your_model_name"
stage_args:
- stage_id: 0
stage_type: llm
runtime:
devices: "0"
engine_args:
model_stage: ar_stage
max_num_seqs: 64
model_arch: YourTTSModelForConditionalGeneration
worker_type: ar
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
engine_output_type: latent
default_sampling_params:
temperature: 0.9
top_k: 50
max_tokens: 2048
- stage_id: 1
stage_type: llm
runtime:
devices: "0"
engine_args:
model_stage: decoder
model_arch: YourTTSModelForConditionalGeneration
worker_type: generation
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
engine_output_type: audio
engine_input_source: [0]
final_output: true
final_output_type: audio
YOUR_TTS_PIPELINE = PipelineConfig(
model_type="your_model_name",
default_deploy_config_name="your_model_name.yaml",
model_arch="YourTTSModelForConditionalGeneration",
stages=(
StagePipelineConfig(
stage_id=0,
model_stage="ar_stage",
execution_type=StageExecutionType.LLM_AR,
input_sources=(),
owns_tokenizer=True,
engine_output_type="latent",
custom_process_next_stage_input_func=f"{_PROC}.ar2decoder",
async_chunk_process_next_stage_input_func=f"{_PROC}.ar2decoder_async_chunk",
),
StagePipelineConfig(
stage_id=1,
model_stage="decoder",
execution_type=StageExecutionType.LLM_GENERATION,
input_sources=(0,),
model_arch="YourTTSDecoder",
engine_output_type="audio",
final_output=True,
final_output_type="audio",
),
),
)
```
### Streaming mode (async_chunk)
Add `async_chunk: true` at the top level and specify `custom_process_next_stage_input_func`
on Stage 0 to define how intermediate outputs are chunked and forwarded:
Register the pipeline in `vllm_omni/config/pipeline_registry.py`. Runtime
placement and sizing belong in `vllm_omni/deploy/your_model_name.yaml`:
```yaml
# stage_configs/your_model_name_async_chunk.yaml
async_chunk: true
stage_args:
stages:
- stage_id: 0
stage_type: llm
runtime:
devices: "0"
engine_args:
model_stage: ar_stage
max_num_seqs: 64
model_arch: YourTTSModelForConditionalGeneration
worker_type: ar
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
engine_output_type: latent
custom_process_next_stage_input_func: >
vllm_omni.model_executor.stage_input_processors.your_model_name.ar2decoder_async_chunk
devices: "0"
max_num_seqs: 64
default_sampling_params:
temperature: 0.9
top_k: 50
max_tokens: 2048
- stage_id: 1
stage_type: llm
runtime:
devices: "0"
engine_args:
model_stage: decoder
model_arch: YourTTSModelForConditionalGeneration
worker_type: generation
scheduler_cls: vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler
engine_output_type: audio
engine_input_source: [0]
final_output: true
final_output_type: audio
devices: "0"
max_num_seqs: 1
```
Set `async_chunk: false` for end-to-end handoff. The registered pipeline
selects the full-payload or async chunk processor; processor paths and topology
must not be moved into the deploy YAML. Use `--deploy-config` to load a custom
deployment overlay.
## Stage Input Processors
Stage input processors convert Stage 0 outputs into Stage 1 inputs. Create yours in
@@ -547,9 +528,9 @@ Understanding what's available in stage outputs:
- `stage_list[source_id].engine_outputs` - list of `EngineCoreOutput` objects
- Each `EngineCoreOutput` has `outputs` - list of `RequestOutput` objects
- Each `RequestOutput` has:
- `token_ids` - generated token IDs
- `multimodal_output` - dict with keys matching your model's `OmniOutput.multimodal_outputs`
- `prompt_token_ids` - original prompt token IDs
- `token_ids` - generated token IDs
- `multimodal_output` - dict with keys matching your model's `OmniOutput.multimodal_outputs`
- `prompt_token_ids` - original prompt token IDs
### Batch mode (non-streaming)
@@ -629,6 +610,7 @@ def ar2decoder_async_chunk(
```
Key points:
- `transfer_manager` is the `OmniChunkTransferAdapter` that owns the chunk lifecycle
- Each call appends one AR decode step's output; a chunk is emitted every `chunk_size` steps
- The final (possibly partial) chunk is flushed when `is_finished` is true
@@ -683,6 +665,7 @@ Create `vllm_omni/entrypoints/openai/tts_adapters/your_model.py`:
```python
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""YourModel serving adapter."""
from typing import TYPE_CHECKING
@@ -821,7 +804,7 @@ directly from the model's own generator.
### Directory structure
```
```text
vllm_omni/model_executor/models/your_model_name/
__init__.py
modeling_your_model_name.py # unified class: load_weights + forward + streaming
@@ -832,32 +815,40 @@ vllm_omni/deploy/your_model_name.yaml
No stage input processor is needed.
### Stage config
### Pipeline and deploy config
Use a single stage with `worker_type: ar`. The `is_comprehension: true` field and the
top-level `async_chunk: false` are required — omitting them causes silent
misclassification in the serving layer. Set `max_num_seqs` to at least 4 for
concurrent production use.
Use a single `LLM_AR` stage that owns the tokenizer and produces final audio:
```python
YOUR_TTS_PIPELINE = PipelineConfig(
model_type="your_model_name",
default_deploy_config_name="your_model_name.yaml",
model_arch="YourModelForCausalLM",
stages=(
StagePipelineConfig(
stage_id=0,
model_stage="your_model_stage_key",
execution_type=StageExecutionType.LLM_AR,
input_sources=(),
owns_tokenizer=True,
engine_output_type="audio",
final_output=True,
final_output_type="audio",
),
),
)
```
The deploy YAML controls runtime behavior. Set `max_num_seqs` to at least 4
for concurrent production use:
```yaml
# stage_configs/your_model_name.yaml
async_chunk: false
stage_args:
stages:
- stage_id: 0
stage_type: llm
is_comprehension: true # required for serving_speech.py dispatch
runtime:
devices: "0"
engine_args:
model_stage: your_model_stage_key
model_arch: YourModelForCausalLM
worker_type: ar
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
engine_output_type: audio
max_num_seqs: 4 # min 4 for concurrent requests; default 1 causes gaps
final_output: true
final_output_type: audio
devices: "0"
max_num_seqs: 4
```
### Generator-based streaming pattern
@@ -961,10 +952,17 @@ is correct. Stage the modified files and commit again — do not revert the chan
Common failures and fixes:
| Check | Cause | Fix |
|-------|-------|-----|
| ------- | ------- | ----- |
| `ruff F841` | Local variable assigned but never used | Remove the extraction or forward it to the model call |
| `ruff E402` | Module-level import not at top of file | Move import to the top-level import block |
| `ruff format` | Line length, spacing, or quote style | Accept the auto-fix, stage, and re-commit |
| `check-spdx-header` | Missing header, or copyright still says `vLLM project` | Use `Copyright contributors to the vLLM-Omni project`; restage the rewrite |
| `check-forbidden-imports` | Stdlib `re`/`base64`, pickle, Hugging Face Hub API, or direct Triton/TileLang | `import regex as re` and `pybase64`; do not grow the allowlist without review |
| `check-torch-cuda-call` | New `torch.cuda.*` call site | Use `current_omni_platform`; do not grow `ALLOWED_FILES` without review |
| `check-tts-adapter-migration` | New `_tts_model_type` branch in `serving_speech.py` | Put logic in `tts_adapters/`; lower `MAX_MODEL_TYPE_BRANCHES` when removing branches |
| `check-mark` | Test file missing level/hardware mark, or direct `pytest.mark.H100` | Add `core_model`/… plus `cpu`/`cuda`/`hardware_test(`; SKU marks only via helpers |
| `shellcheck` | Native `shellcheck` missing, or script warning | Install via apt/dnf/brew (or `shellcheck.exe` on Windows). See [Linting](../README.md#linting) |
| `mypy-3.10` / `markdownlint-cli2` / `check-buildkite` | Types, docs markdown, or Buildkite YAML | See [Linting](../README.md#linting) |
### DCO sign-off
@@ -1019,7 +1017,7 @@ Adding a TTS model to vLLM-Omni involves:
### Qwen3-TTS Reference Files
| File | Purpose |
|------|---------|
| ------ | --------- |
| `models/qwen3_tts/qwen3_tts.py` | Unified model class |
| `models/qwen3_tts/qwen3_tts_code_predictor_vllm.py` | AR stage with vLLM fused ops |
| `models/qwen3_tts/qwen3_tts_code2wav.py` | Decoder stage with `chunked_decode_streaming()` |
@@ -1031,4 +1029,4 @@ For more information, see:
- [Architecture Overview](../../design/architecture_overview.md)
- [Async Chunk Design](../../design/feature/async_chunk.md)
- [Stage Configuration Guide](../../configuration/stage_configs.md)
- [Pipeline and Deploy Configuration Guide](../../configuration/stage_configs.md)
docs/contributing/profiling.md
+9
View File
@@ -322,6 +322,15 @@ Each 1-second window records:
On shutdown the server also logs a short summary (`loop_active_pct`, per-replica queue averages/maxima).
`loop_idle` and `loop_active` count loop iterations, so their absolute scale is
tied to which orchestration loop is running. Under the default poll loop an idle
orchestrator records roughly one iteration per millisecond. Under the
event-driven loop (`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, see
[Speech API](../serving/speech_api.md#orchestration-loop-experimental)) an idle
orchestrator wakes only once per 0.5 s reconcile timeout, while a busy one still
records one iteration per routed output. Window counts and the `loop_active_pct`
summary are therefore not comparable across the two modes.
### Relationship to other diagnostics
This monitor is intentionally separate from the existing profiling tools:
docs/design/ar_diffusion_pipeline_capability.md
+10 -8
View File
@@ -4,6 +4,9 @@
> `experimental/ar_diffusion` contract. It follows the ownership and lifecycle
> direction in [#5137](https://github.com/vllm-project/vllm-omni/issues/5137),
> but it is not yet part of the stable module-design hierarchy.
>
> The end-to-end request, identity, failure, and memory contracts are documented
> in [Realtime AR-Diffusion sessions](feature/realtime_ar_diffusion.md).
`ARDiffusionEngine` is an opt-in diffusion backend for pipelines that keep
autoregressive attention KV across requests. The engine only selects
@@ -51,14 +54,13 @@ time, and its inherited batch/step entry points also fail explicitly. Supporting
those paths requires a batch-aware state-binding contract rather than silently
bypassing `bind_ar_diffusion_state()`.
The current implementation retains one session per model instance. A request
for a different `session_id` closes the resident session through the same LRU
lifecycle used by explicit close and failure cleanup before allocating the new
one. Self-attention and cross-attention memory are therefore sized for one
resident session even if the pipeline declares a larger `session_capacity`.
This prioritizes rollout throughput and prevents block-pool exhaustion before a
count-based eviction can run. Multi-session residency can be added later with a
memory-backed capacity calculation or allocation-pressure eviction.
The implementation may retain multiple sessions per model instance. It selects
the largest feasible capacity up to the pipeline's `session_capacity`, based on
self-attention pages, cross-attention allocations, scratch, and declared
model-owned state. One session is admitted whenever it fits actual free device
memory; `gpu_memory_fraction` controls additional residency. Requests beyond
that capacity evict the least-recently-used session through the same lifecycle
hook used by explicit close and failure cleanup.
Scratch storage is also capability-driven. `frames_per_block` reserves space for
an uncommitted current video block, while `max_scratch_tokens_per_branch`
docs/design/architecture_overview.md
+3 -3
View File
@@ -335,8 +335,8 @@ The single resolution boundary is implemented by
`StageConfigFactory.create_from_model()` and
`VllmOmniConfig.from_pipeline_config()` in
[`vllm_omni/config`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/config).
The legacy `stage_args` YAML path remains only for models that have not yet
migrated to `PipelineConfig` and `DeployConfig`.
The legacy `stage_args` YAML path has been removed. Model topology now resolves
through `PipelineConfig`, with runtime overrides supplied by `DeployConfig`.
In the typed path, each stage configuration derives from
`BaseVllmOmniStageConfig` and is specialized as
@@ -406,7 +406,7 @@ the source for configuration, compatibility, and implementation details.
stage contract.
* **Quantization and memory efficiency:** [Quantization](feature/quantization.md)
resolves per-pipeline or per-component quantization configurations, while
[Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
[Distributed Layerwise Offload](feature/offloader/distributed_layerwise_offload.md)
streams diffusion blocks from host memory within the existing parallel
topology.
docs/design/feature/async_chunk.md
+23 -25
View File
@@ -211,38 +211,39 @@ In sequential mode, each stage must wait for the previous stage to complete enti
## Configuration
Enable async_chunk in stage configuration YAML:
Enable async chunk mode in the deploy YAML:
```yaml
async_chunk: true
stage_args:
- stage_id: 0
engine_args:
custom_process_next_stage_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_async_chunk
- stage_id: 1
engine_args:
custom_process_next_stage_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_async_chunk
```
### Stage Configuration
The registered `PipelineConfig` owns the async handoff processor functions.
For Qwen3-Omni these are `thinker2talker_async_chunk` and
`talker2code2wav_async_chunk`; deploy YAML only selects the mode and runtime
sizing.
### Configuration ownership
- `async_chunk: bool`: Enable/disable async chunk mode
- `custom_process_next_stage_input_func: str`: Path to custom chunk processing function; receives `(transfer_manager, pooling_output, request)`. For qwen3-omni: `thinker2talker_async_chunk`, `talker2code2wav_async_chunk`
- `stage_connector_config: dict`: Connector configuration
- `worker_type: str`: Model type, e.g. `"ar"` or `"generation"` (used by OmniChunkTransferAdapter for mode-specific payload handling)
- `async_chunk_process_next_stage_input_func: str`: Pipeline-owned chunk processor path
- `execution_type: StageExecutionType`: Pipeline-owned runtime family
- `connectors: dict`: Deploy-owned connector definitions
- `max_num_seqs: int`: Maximum number of sequences for concurrent processing in the stage
### Connector Configuration
```yaml
connectors:
- from_stage: 0
to_stage: 1
spec:
name: SharedMemoryConnector
extra:
stage_id: 0
connector_of_shared_memory:
name: SharedMemoryConnector
stages:
- stage_id: 0
output_connectors:
to_stage_1: connector_of_shared_memory
- stage_id: 1
input_connectors:
from_stage_0: connector_of_shared_memory
```
### Code2Wav Batch Configuration
@@ -250,13 +251,10 @@ connectors:
For optimal performance with async_chunk, the code2wav stage should be configured with batching:
```yaml
stage_args:
stages:
- stage_id: 2 # code2wav stage
runtime:
devices: "1"
engine_args:
model_stage: code2wav
max_num_seqs: 64 # Enables batched audio generation
devices: "1"
max_num_seqs: 64 # Enables batched audio generation
```
## Related Files
docs/design/feature/async_diffusion_output.md
+17 -1
View File
@@ -117,7 +117,22 @@ After (async D2H):
### Batch Split
When `execute_model_batch` returns a `COMPUTE_DONE` with a single `async_output_id` for the entire batch, the executor splits it into per-request `async_output_id`s (format `{batch_id}/{request_id}`) via `_batch_split_map`. When `OUTPUT_READY` arrives, the pump extracts per-request results from the batch output and resolves each request's output future independently.
When `execute_model_batch` returns a `COMPUTE_DONE` with a single `async_output_id` for the entire batch, the executor splits it into per-request `async_output_id`s (format `{batch_id}/{request_id}`) via `_batch_split_map`. When `OUTPUT_READY` arrives, the pump extracts per-request results from the batch output and resolves each request's output future independently. If `OUTPUT_READY` arrives before `execute_batch` registers the split map, `execute_batch` directly adopts the early output.
### Reliability, Lifecycle & Timeout Behavior
1. **Drain Before Memory Release (`drain_async_outputs` / `_ASYNC_OUTPUT_DRAIN_TIMEOUT_S`)**:
When workers transition into sleep states (`handle_sleep_task`) or invoke memory-releasing methods, `WorkerProc.drain_async_outputs()` blocks up to `_ASYNC_OUTPUT_DRAIN_TIMEOUT_S` (10.0s) until all in-flight background D2H/SHM packing tasks finish. This acts as a synchronization barrier preventing worker sleep cycles from releasing device tensors while the side CUDA stream is still actively reading them. Draining typically completes within 10–50 ms under normal operation.
2. **Timeout Diagnostics (`_async_output_timeout()`)**:
The engine waits for `wait_output_ready()` with a timeout of `_async_output_timeout()` — 600.0s by default, overridable per run with the `VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT` environment variable (a non-numeric or non-positive value is ignored with a warning and the default applies, since this resolves on the request path). If a timeout occurs, the executor logs detailed diagnostic bookkeeping (pending futures, cached outputs, and batch split maps) to identify whether the stall occurred in the worker, pump, or waiter. The packing itself takes milliseconds, but the wait is queued behind the step's GPU work, so its wall-clock duration tracks step time; the bound is generous because a hung engine is surfaced by the worker monitor and `check_health()`, not by this timeout.
3. **Queue Serialization & Atomic Future Resolution**:
- `result_mq` writes are serialized with `_result_mq_lock` to protect the single-writer `MessageQueue` against concurrent writes from the worker main loop (`COMPUTE_DONE`) and background thread (`OUTPUT_READY`).
- `ResultPumpThread` unpacks shared-memory payloads first, and then atomically resolves waiting futures or populates `_completed_outputs` under `_futures_lock`, avoiding race conditions with concurrent `wait_output_ready()` invocations.
4. **Storage-Aware IPC Packing**:
`pack_diffusion_output_shm()` evaluates `max(view_bytes, storage_bytes)` against `_SHM_TENSOR_THRESHOLD` (1 MB) to ensure that per-request slices sharing a large batch storage are moved to POSIX shared memory rather than serialized through pickle over the MessageQueue inline buffer.
## Model Coverage
@@ -131,6 +146,7 @@ All models running in request-mode (`step_execution=False`, the default) automat
|---|---|---|
| **HunyuanImage-3.0** | Image | `False` |
| **Qwen-Image** | Image | `True` |
| **LTX-2.3** | Video / Audio | `False` |
Other models with `step_execution=False` are also supported but not yet verified.
docs/design/feature/attention_backend_selection.md
+106
View File
@@ -0,0 +1,106 @@
# Diffusion Attention Backend Selection
This document defines the selection and extension contract for diffusion
attention backends. User-facing backend choices, installation, and tuning are
in the [attention backend guides](../../user_guide/diffusion/attention_backends.md).
## Scope
The contract applies to DiT and other diffusion attention layers. It is
separate from vLLM's autoregressive attention selector.
The selection path has four responsibilities:
1. normalize user configuration into `AttentionConfig` and `AttentionSpec`;
2. resolve a spec for an attention role;
3. ask the active platform to validate an explicit backend or choose a
hardware default; and
4. load the selected `AttentionBackend` class from the registry path.
## Resolution contract
`get_attn_backend_for_role()` resolves configuration in this order:
1. exact `per_role[role]`;
2. category `per_role[role_category]`;
3. `default`;
4. platform default.
An explicit resolution returns both the backend class and its
`AttentionSpec`. A platform-default resolution returns the class and `None`.
Layers must therefore treat the spec as optional and must not infer that a
missing spec means SDPA.
Class resolution is cached by backend name, head size, and whether TRTLLM may
be selected automatically. Logging is cached separately by role and source so
the startup record identifies why each role received its backend.
## Registry and platform boundary
`DiffusionAttentionBackendEnum` maps stable configuration names to default
qualified class paths. `register_diffusion_backend()` may replace a path at
runtime without changing the public enum value.
The active `OmniPlatform` owns compatibility policy through
`get_diffusion_attn_backend_cls()`. It must:
- validate explicit selections and fail with an actionable error when the
requested kernel cannot run;
- choose only an available, compatible default when no backend is explicit;
- account for head size and other platform-visible constraints; and
- return a qualified class path implementing `AttentionBackend`.
The selector must not duplicate device capability or package-availability
policy that belongs to the platform.
## Typed backend options
Backend-specific settings remain typed fields on `AttentionSpec` rather than
unstructured keyword dictionaries:
- `quant` is consumed by FlashInfer and TRTLLM with backend-specific value
validation;
- `skip_softmax` is consumed by TRTLLM; and
- `block_sparse` is consumed by block-sparse backends such as RainFusion; and
- `fastvideo_vsa_topk` is consumed by FastVideo VSA.
A backend reads only the fields it owns and rejects incompatible values. New
options should be added to a shared typed spec only when more than one backend
shares their semantics; otherwise add a dedicated typed block.
## Model integration contract
Each diffusion attention call declares a stable role and, when appropriate, a
broader category. Model-specific roles permit precise overrides; categories
preserve common `self` or `cross` policies. A model also decides whether its
path is compatible with automatic TRTLLM selection, because only the model
knows whether masking and packing satisfy the kernel contract.
## Adding or changing a backend
An implementation change is complete only when it:
1. implements the `AttentionBackend` interface and exposes a stable enum name;
2. adds platform validation and default routing where appropriate;
3. defines and validates any typed configuration;
4. tests explicit selection, default selection, and incompatible paths;
5. documents installation, fallback behavior, and quality implications in the
corresponding user guide; and
6. updates the overview matrix without moving algorithm details back into the
landing page.
## FastVideo VSA model metadata
FastVideo VSA support is scoped to `FastVideo/FastWan2.2-TI2V-5B-Diffusers`, whose text-to-video and image-to-video modes both use `Wan22Pipeline`. It operates on a flattened DiT sequence but partitions tokens in
the original latent video grid. Wan integrations therefore attach the
post-patch `(T, H, W)` shape as `vsa_dit_seq_shape` attention metadata. The
backend validates that `T * H * W` equals the sequence length, derives the
runtime block count, and selects the configured top-k key/value blocks for
each query block.
Checkpoint-specific compensation remains model-owned. Wan layers pass an
optional learned `gate_compress` projection through attention metadata when
the checkpoint contains those weights. Checkpoints without the projection do
not allocate or execute it. When top-k selects every block, native checkpoints
route to SDPA, while FastVideo DMD checkpoints preserve the VSA all-block path
to retain checkpoint semantics.
docs/design/feature/cfg_parallel.md
+1 -1
View File
@@ -63,7 +63,7 @@ vLLM-omni provides `CFGParallelMixin` that encapsulates all CFG parallel logic.
### N-Branch CFG (3+ branches)
Some models require more than 2 CFG branches. For example, Bagel and OmniGen2 use 3 branches, DreamID Omni uses 4 branches.
Some models require more than 2 CFG branches. For example, Bagel and OmniGen2 use 3 branches.
`predict_noise_with_multi_branch_cfg()` handles these by automatically dispatching N branches across M GPUs using round-robin (rule: branch `i` → rank `i % M`):
docs/design/feature/disaggregated_inference.md
+5 -6
View File
@@ -59,19 +59,18 @@ This `metadata` must be passed through the control plane so `get()` can locate t
## Configuration Model
Define connectors in runtime:
Define connectors at the top level of the deploy YAML:
```yaml
runtime:
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
```
Wire stages to connectors:
```yaml
stage_args:
stages:
- stage_id: 0
output_connectors:
to_stage_1: connector_of_shared_memory
docs/design/feature/distributed_layerwise_offload.md
-156
View File
@@ -1,156 +0,0 @@
# Distributed Layerwise Offload
This document describes distributed layerwise offload (DLO) for diffusion
models. DLO keeps only a small number of DiT blocks on the accelerator and
streams the remaining blocks from host memory. The distributed backend can
either shard those host-side weights across an existing parallel group or keep
the standard loader's rank-local weights and avoid an additional collective.
For user-facing commands, see the
[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload.md)
and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
## Status
DLO is implemented for multi-device diffusion execution. The default
AllGather path is the primary path for DP and SP deployments. The
`--dlo-no-use-allgather` path is a rank-local compatibility mode: it is useful
for standard-loader sharding, workstation bring-up, and systems where an
additional DLO collective is undesirable, but it does not reduce host weight
storage across ranks.
The compatibility matrix below describes the current implementation. The
unit-level guards are covered, but not every parallelism combination has a
full model-and-hardware end-to-end test.
## Design
### DLO consumes the existing parallel topology
DLO does not create a new DP, TP, or SP topology. It reads the configured
`DiffusionParallelConfig` and attaches offload hooks to the DiT blocks after the
standard distributed groups have been initialized.
The DLO weight-sharding group is selected as follows:
1. Use the existing DP group when `data_parallel_size > 1`.
2. When DP is one and SP is greater than one, use the SP group.
3. Otherwise, run rank-locally without a DLO process group.
TP is deliberately not used as DLO's AllGather group. HSDP has its own
parameter-sharding lifecycle and is not allowed to be sharded a second time by
DLO's AllGather path.
### AllGather path
With the default `dlo_use_allgather=True`, each rank stores approximately
`1 / group_size` of each streamable block in pinned host memory. The next
block's shard is copied to a device buffer and reconstructed with
`all_gather_into_tensor` on a communication stream while the current block is
executing.
```text
Compute: [Block N] [Block N+1] [Block N+2]
H2D: [shard N+1] [shard N+2]
AllGather: [full N+1] [full N+2]
Buffers: [current slot] [prefetch slot] [current slot]
```
![DLO double-buffer prefetch pipeline](../figures/dlo/dlo_pipeline.gif)
The backend uses two shared device buffers, so accelerator weight residency is
bounded by the largest streamed blocks rather than the complete model.
When DP is greater than one, the engine can process one request per DP rank in
the same denoising wave. Because AllGather is a collective, all participating
requests must take the same execution path at every denoising step.
### Rank-local path without DLO AllGather
With `--dlo-no-use-allgather`, DLO forces its internal offload shard size to
one. The regular model loader remains responsible for preparing each rank's
weights, including TP-local tensors or HSDP-managed parameters. DLO then
streams those rank-local tensors block by block using H2D copies only.
This mode means:
- DP still provides independent replicas, but DLO does not shard weights
across DP ranks.
- SP still performs its normal activation/attention collectives, but DLO does
not shard weights across SP ranks.
- TP/HSDP/SP collectives, if configured, are not disabled by this flag; only
DLO's additional weight AllGather is disabled.
- Pure DP deployments keep a full host-side model copy per rank, subject to
shared OS page-cache behavior.
- The scheduler does not require a synchronized DP request wave for DLO.
This path is intentionally not implemented by reusing the AllGather mmap
loader. It relies on the standard loader so model-specific TP/HSDP transforms
remain intact.
## Parallelism compatibility
| Parallelism | DLO + AllGather | DLO without AllGather |
|---|---|---|
| **DP** | Supported primary path. DLO shards host weights across the DP group and can run DP multi-concurrency. | Supported rank-local path. DP replicas remain independent; no DLO weight sharding or cross-DP DLO collective. |
| **SP** | Supported in the implementation. With DP=1, DLO uses the SP group for host-weight sharding; SP still shards sequence/activation work. | SP remains active, but DLO keeps standard-loader rank-local weights and adds no SP weight collective. |
| **TP > 1** | Unsupported in the DLO mmap path. TP-aware loader callbacks are bypassed, so the backend rejects this configuration when it enters that path. | Standard loading is retained and TP-local tensors can be streamed. This is the intended compatibility path, but it still needs broader model and hardware validation. |
| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted by configuration. HSDP owns parameter sharding and its own gathers; DLO only stages rank-local parameters. End-to-end coverage is limited. |
### Combined dimensions
- **DP + SP:** DLO uses the DP group for weight sharding when DP is greater
than one; SP continues to use its own sequence-parallel group. If DP is one,
the SP group becomes DLO's sharding group in AllGather mode.
- **DP + TP/SP without AllGather:** standard model loading defines the
rank-local tensor layout. DLO adds no cross-DP, cross-TP, or cross-SP weight
collective.
- **HSDP + SP:** the general parallel configuration permits HSDP over SP, but
DLO must use `--dlo-no-use-allgather`. HSDP remains responsible for weight
materialization and synchronization.
- **HSDP + DP or TP:** rejected independently by the diffusion parallel
configuration.
## Request and loading constraints
AllGather DP multi-concurrency requires:
- explicit `num_inference_steps`;
- the same `num_inference_steps` for all requests in a wave; and
- identical request arguments that affect the collective execution path.
The no-AllGather path does not impose these DLO-specific synchronized-wave
requirements.
The mmap loader is used only by the supported DLO+AllGather path when the
model has a compatible checkpoint layout. Online quantization is incompatible
with that sharded mmap path; use `--dlo-no-use-allgather` or disable online
quantization. Models that do not meet the mmap requirements use the regular
loader path.
## Validation coverage
Current source-level validation includes:
- HSDP + DLO + AllGather rejection;
- HSDP + DLO without AllGather acceptance at configuration level;
- TP rejection in the DLO+AllGather mmap path;
- resident-layer requests requiring no-AllGather;
- DP request-wave validation for denoising-step compatibility;
- sharding, double-buffer, AllGather-size, and heterogeneous-block regression
tests.
The highest-value missing coverage is end-to-end numerical comparison against
ordinary layerwise offload for DP+SP, TP+no-AllGather, and HSDP+SP+no-AllGather
on the target CUDA/NCCL or CANN/HCCL hardware.
## Recommendations
- Use **DP + DLO AllGather** for the supported throughput and host-memory
scaling path.
- Use **SP + DLO AllGather** for long-sequence workloads when DP concurrency is
not the goal.
- Use **no-AllGather** to bring up TP or workstation PCIe configurations, with
the expectation of higher host-memory use and lower validation confidence.
- Prefer **HSDP alone** for production HSDP deployments until the combined
HSDP + DLO no-AllGather path has broader end-to-end coverage.
docs/design/feature/host_weight_runtime.md
+386
View File
@@ -0,0 +1,386 @@
# Host Weight Runtime
The Host Weight Runtime is a loader-facing feature for reusing immutable,
runtime-ready model weights across workers on one host. It lets a model
integration replace repeated checkpoint loading and transformation with an
exact lookup of a previously published host representation, while keeping the
canonical model source authoritative.
This feature is shared infrastructure. Diffusion, autoregressive, speech, and
future model integrations may depend on the top-level
`vllm_omni.host_weight_runtime` package without depending on one another's
loader or execution stack. The internal component and filesystem contracts are
specified in the [Host Weight Runtime module design](../module/host_weight_runtime.md).
## Status
The implementation provides contracts and a CPU local-filesystem store. The
diffusion consumer additionally defines typed,
representation-independent final-layout identity/restoration mechanics plus a
concrete BF16-with-preserved-FP32 policy for MiniMax H3 and
`black-forest-labs/FLUX.2-klein-4B`. The opt-in no-AllGather DLO integration
selects, publishes, restores, and transfers these artifacts.
V1 includes:
- exact runtime-weight identity and immutable manifests;
- coordinated local lookup and one-producer publication;
- descriptor-backed safetensors mmap leases;
- preferred and required resolution policy;
- explicit, separately reported post-load publication;
- validation, deny, quarantine, cleanup, and capacity controls; and
- typed reports for every terminal resolution outcome.
V1 does not include:
- default-on activation or consumers outside no-AllGather DLO;
- online FP8, quantized, merged-adaptation, or additional model producers;
- HWR interaction with DLO AllGather;
- a remote artifact provider or cross-node coordination;
- automatic eviction; or
- a change to DLO collective or execution behavior.
## Motivation and use cases
Model loading can create the same final host representation repeatedly. This
is especially expensive when loading performs checkpoint decoding, tensor
renaming, TP slicing, quantization, packing, or scale construction before GPU
transfer. Independent workers may then retain private CPU copies of identical
runtime weights.
The Host Weight Runtime is useful when:
- multiple same-node workers request semantically identical final weights;
- constructing those weights is materially more expensive than validating and
mapping a local artifact;
- the runtime layout can be identified exactly and reproduced deterministically;
- the workers can share immutable file-backed pages through the OS page cache;
and
- canonical loading remains available when policy permits fallback.
Typical consumers include same-node diffusion replicas, DLO host-weight
sources, and future transformed or quantized model loaders. Request routing and
replica orchestration remain outside this feature.
This feature is not zero-copy GPU execution. A lease exposes stable CPU tensor
views and mapped ranges. A separate transport still decides whether to use
registered mmap, private pinned staging, synchronous copies, or asynchronous
H2D transfer.
## Resolution behavior
The loader resolves the immutable canonical source and computes the exact
requested identity before asking the runtime for a representation.
```mermaid
flowchart TD
L["Loader resolves canonical source"] --> I["Compute exact WeightArtifactIdentity"]
I --> M{"Runtime mode"}
M -->|"disabled"| C["Canonical loader"]
M -->|"preferred or required"| A{"Validated local artifact?"}
A -->|"yes"| R["Return HostWeightLease"]
A -->|"no"| X["Remote exact artifact (future)"]
X --> P{"Registered producer available?"}
P -->|"yes"| B["Build, validate, and publish atomically"]
B --> R
P -->|"no or recoverable failure"| F{"Policy permits fallback?"}
F -->|"preferred"| C
F -->|"required"| E["Fail startup"]
P -->|"nonretryable failure"| E
C --> W{"Post-load publication enabled?"}
W -->|"yes"| B2["Publish final model through POST_LOAD_ONLY producer"]
B2 --> CL["Close validated publication lease"]
CL --> R2["Return separate publication report"]
R2 --> D
W -->|"no"| D["Keep canonical model"]
```
The modes are:
| Mode | Observable behavior |
| --- | --- |
| `disabled` | Use canonical loading directly without probing storage, identity, credentials, or topology. |
| `preferred` | Return an exact lease when available; otherwise use canonical fallback only for a miss, invalid cache entry, or typed retryable failure. |
| `required` | Fail when an exact lease cannot be acquired. |
During pre-load resolution, an unsupported capability, semantic identity
collision, producer failure, or publication failure is nonretryable and remains
visible even in preferred mode. A post-load publication failure is likewise
visible in its own report, but cannot revise the canonical-fallback outcome.
Storage policy must never disguise a semantic or configuration error as a cache
miss.
## Loader and restoration sequence
The model integration owns the end-to-end loading transaction. The runtime
does not construct a model or invoke the canonical loader.
```mermaid
sequenceDiagram
participant L as Loader adapter
participant R as HostWeightRuntime
participant S as HostWeightStore
participant P as WeightProducer
participant X as WeightRestorer
participant T as GPU transport
L->>L: Resolve canonical revision and exact identity
L->>R: resolve(identity, producer)
R->>S: lookup(exact identity)
alt validated hit
S-->>R: HostWeightLease
R-->>L: lease and LOCAL_HIT report
L->>X: plan_restore(model, lease)
X-->>L: validation-only restore plan
L->>X: commit() once
else miss with allowed producer
R->>S: get_or_build(identity, producer)
S->>P: produce(store-scoped writer)
P-->>S: final-layout tensors and metadata
S-->>R: validated HostWeightLease
R-->>L: lease and LOCAL_PRODUCTION report
L->>X: plan_restore(model, lease)
X-->>L: validation-only restore plan
L->>X: commit() once
else policy permits canonical fallback
R-->>L: CANONICAL_FALLBACK
L->>L: Run canonical loader
opt explicit post-load publication enabled
L->>R: publish_after_load(identity, POST_LOAD_ONLY producer)
R->>S: get_or_build(identity, producer)
S-->>R: validated HostWeightLease or typed failure
R->>R: close publication lease
R-->>L: separate publication report
end
else required or nonretryable failure
R-->>L: FAILED report
end
L->>T: consume final CPU tensors and any lease
T-->>L: transfer teardown complete
L->>L: close lease when one was acquired
```
`plan_restore()` must not mutate the model or lease. `commit() -> None` is the
sole one-shot model mutation. If planning fails, canonical fallback may reuse
the untouched model. If commit begins and fails, the partially hydrated model
must be discarded and canonical fallback must construct a fresh model.
`publish_after_load()` is synchronous in V1 and accepts only a
`POST_LOAD_ONLY` producer. Its report is separate from the terminal resolution,
so a publication failure cannot rewrite a successful canonical fallback. A
successful publication closes the store-returned lease inside the runtime and
warms only future startups. It does not restore, rebind, or otherwise mutate the
canonically loaded model serving the current startup.
`allow_local_build` gates producers during pre-load resolution, while
`allow_post_load_publish` independently gates this explicit post-load path.
## Exact representation identity
The store performs exact lookup. It never silently converts one representation
into another or substitutes the first backing that responds.
Identity includes:
- immutable model revision and source fingerprint;
- model component and ownership boundary;
- representation name, dtype, and format metadata;
- final tensor layout and semantic parallel coordinates;
- static adaptation identity; and
- producer implementation, manifest, and restorer schema versions.
The requested representation is selected by the loader. The store chooses
where to obtain that exact representation: validated local artifact, future
remote materialization, or a registered producer.
### Parallelism
Parallel coordinates are included only when they change bytes, shape, layout,
or component ownership.
| Dimension | Identity and sharing rule |
| --- | --- |
| DP | Exclude DP rank for replicated weights so same-node replicas can acquire the same artifact. Include a coordinate only if DP changes weight ownership. |
| TP | Include TP size and rank when each rank owns different tensor slices or packed layouts. Each distinct TP shard is a separate exact artifact. |
| SP | Include SP size and backend when they change the weight layout. Exclude SP rank when all SP ranks consume identical bytes. |
| PP | Encode component ownership or stage-local layout when PP changes which weights belong to the consumer. |
| EP | Encode expert ownership and layout whenever ranks own different expert weights. |
The runtime does not infer these rules from process-group topology. The model
adapter is responsible for constructing a correct identity.
### Quantization and adaptations
Runtime BF16, FP8, packed quantized weights, and future formats are distinct
representations. Conversion is performed by an explicitly versioned producer,
never by cache coercion. A producer must publish the final layout consumed by
its matching restorer.
Dynamic LoRA overlays are not part of a reusable base-weight artifact. A
statically merged adapter is cacheable only as a separate identity containing
the adapter fingerprint and merge semantics.
### Initial diffusion final-layout contract
The shared diffusion contract covers complete final-layout DiT parameters and
persistent buffers. Text encoders, VAEs, non-persistent derived state, and other
pipeline components remain outside the artifact. One explicit representation
policy selects allowed dtypes, tensor roles, physical layout identity, producer
ABI, manifest schema, and restoration schema.
The contract is intentionally separate from loader activation:
- `FinalLayoutRequest` contains typed loader identity/configuration fingerprints,
TP coordinate, and conservative SP semantics. It has no open metadata bag,
DP coordinate, SP rank, device identity, DLO transfer mode, registration
policy, or store path.
- `FinalLayoutArtifactSpec` binds one `WeightRepresentation` and runtime-layout
name to explicit producer/restorer schemas and a canonical, versioned
implementation ABI descriptor. Compatibility never depends on reflective
source inspection.
- `PreparedWeightSource` snapshots immutable revisions or exact local file
content plus a typed checkpoint-adapter identity before ordinary
materialization. Source replacement before or during production fails
publication. A hash-looking symlink basename is trusted only for an explicit
Hugging Face Hub source whose repository ID and
`models--.../snapshots/<revision> -> blobs/<hash>` topology validate; every
local or otherwise unverified symlink target is content-hashed.
- the tensor ownership digest records exact runtime names, kinds, shapes,
semantic roles, dtypes, and strides from a CPU or meta model skeleton;
- `FinalLayoutTensorRestorer` accepts only an exact lease identity, validates
complete policy-defined coverage without mutation, and returns a one-shot
commit plan;
- each model declares one dtype-neutral `FinalLayoutModelContract` with an
explicit implementation version and a post-commit validator; and
- `FinalLayoutBF16Producer` accepts only the matching identity context and a
finalized CPU model. It is `POST_LOAD_ONLY` and `SINGLE_PROCESS` per exact TP
coordinate. Its BF16 policy preserves model-declared FP32 parameters and
buffers, revalidates MiniMax H3 mixed-precision invariants, and revalidates
FLUX.2-klein's two block stacks, packed QKV mapping, and BF16 base layout.
Other representations reuse source identity, typed parallel identity, tensor
ownership, and exact restoration only when their policy proves those semantics.
For example, runtime FP8 needs a separate policy/producer for generated scales,
quantization metadata, and Cutlass physical layouts; it is not enabled by
changing a dtype string on the BF16 producer.
This stage makes no startup, sharing, or DLO performance claim. A following
consumer PR owns disabled/preferred/required precedence, mixed-component loader
transactions, warm-hit restoration, and transactional lease handoff. A TP2
prewarm deployment will require a matching TP2 producer cohort to populate both
TP-coordinate identities even though the store coordinates each artifact
independently.
## Host sharing and GPU transport
Every process receives its own virtual mappings and `HostWeightLease`, but
workers mapping the same local artifact can share physical file-backed pages
through the OS page cache. This avoids independent private tensor allocations;
it does not guarantee that process PSS reports exactly one model copy.
The ownership boundary is:
- the store owns immutable artifact files, publication, and lifecycle;
- the kernel owns page-cache residency and placement;
- the lease owns process-local mappings and the shared artifact lock; and
- transport owns page registration, page locking, private staging, H2D copies,
streams, device buffers, and lease release ordering.
DLO AllGather and no-AllGather remain transport and execution choices. A DLO
integration may consume a lease, but the Host Weight Runtime does not choose a
parallel collective or orchestrate DP requests. See the
[DLO feature design](offloader/distributed_layerwise_offload.md).
## Locality and NUMA policy
The V1 backend is node-local. It accepts an allowlist of known local disk
filesystems plus tmpfs/ramfs, records the detected kernel filesystem type, and
rejects known remote or unknown filesystems. NFS, CIFS, Lustre, Ceph, and other
cross-node mounts cannot silently satisfy the local backend because their
page-cache and advisory-lock behavior does not provide the required node-local
contract.
A NUMA-specific root creates a separate storage domain but does not by itself
place page-cache pages on that NUMA node. A topology-aware integration must
also pin workers, establish local first touch or prefault, perform transport
registration in the same domain, and collect residency evidence.
Tmpfs artifacts consume host memory and may consume swap. They must be
accounted as memory-backed storage rather than ordinary disk capacity.
## Concurrency and timeouts
One process per exact identity owns a build; other workers wait and then acquire
leases for the published artifact. Publication is invisible until all payloads
and metadata are validated, hashed, fsynced, and atomically renamed.
`coordination_timeout_seconds` bounds filesystem lock acquisition. It does not
cancel synchronous validation, a producer that has already started, or atomic
publication. A hung in-process producer therefore blocks its owning process and
must be handled by external process supervision. Enforceable producer
cancellation requires a future process-isolated producer contract.
Leases keep a shared artifact lock for their lifetime. A forked child may close
its inherited descriptors but cannot unlock the parent process's lease. Cleanup
uses a nonblocking exclusive lock and reports an active lease instead of
removing live mappings.
## Failure and fallback contract
| Condition | Preferred mode | Required mode |
| --- | --- | --- |
| Exact local hit | Return lease | Return lease |
| Local miss | Try later backing or canonical fallback | Try later backing or fail |
| Invalid or denied artifact | Quarantine/rebuild when possible; otherwise canonical fallback | Quarantine/rebuild or fail |
| Retryable lock, domain, or capacity failure | Canonical fallback | Fail |
| Unsupported producer or backend | Fail | Fail |
| Identity collision | Fail | Fail |
| Pre-load producer or atomic publication failure | Fail | Fail |
| Post-load publication failure after canonical fallback | Keep canonical model; report publication failure | Not reached through required resolution |
| Restore planning failure | Canonical fallback may reuse untouched model | Fail |
| Restore commit failure | Discard model; fallback requires a fresh instance | Fail |
Cache failures should normally reduce performance rather than availability,
but only when the typed failure explicitly permits fallback. Canonical source,
authentication, configuration, and semantic errors remain visible.
## Consumer integration requirements
A consumer PR must:
1. Resolve an immutable canonical model source before cache lookup.
2. Construct the exact final representation identity, including relevant
parallel, quantization, and adaptation semantics.
3. Register a deterministic producer for that identity or support lookup-only
operation.
4. Provide a validation-only restorer with a one-shot commit.
5. Keep GPU transport outside the producer, store, and restorer contracts.
6. Handle preferred fallback and required failure without reusing a model after
a failed restore commit.
7. Retain the lease until every transport operation that may access mapped
memory has completed.
8. Emit and test the terminal resolution report and any separate post-load
publication report.
Consumer validation must prove:
- output parity with canonical loading;
- exact warm-hit work avoidance;
- correct TP/SP/PP/EP ownership and layout identity;
- correct BF16, FP8, quantized, or adaptation semantics;
- shared-backing evidence when host-memory savings are claimed;
- clean unmapping, unregistration, and artifact-lock release; and
- startup, host-memory, and transport effects for the claimed deployment.
## Rollout
The rollout is intentionally staged:
1. Land the neutral contracts and local filesystem store.
2. Add one model-specific producer/restorer integration with parity evidence.
3. Connect eligible DLO or other transport paths without moving transport
ownership into the runtime.
4. Add additional final-layout and quantized producers independently.
5. Introduce remote materialization only through the same local lease contract.
The architectural decisions and deferred work are tracked in
[RFC #6414](https://github.com/vllm-project/vllm-omni/issues/6414).
docs/design/feature/offloader/README.md
+80
View File
@@ -0,0 +1,80 @@
# CPU Offloading
This document defines the shared architecture for diffusion CPU offloading.
The three strategies have separate user and design pages:
| Strategy | User guide | Design contract |
| --- | --- | --- |
| Model-level | [Guide](../../../user_guide/diffusion/offloader/cpu_offload.md) | [Design](cpu_offload.md) |
| Layerwise | [Guide](../../../user_guide/diffusion/offloader/layerwise_offload.md) | [Design](layerwise_offload.md) |
| Distributed layerwise | [Guide](../../../user_guide/diffusion/offloader/distributed_layerwise_offload.md) | [Design](distributed_layerwise_offload.md) |
## Strategy selection
`OffloadConfig.from_od_config()` converts diffusion configuration into one
`OffloadStrategy`. If callers set multiple flags, selection is deterministic:
1. `DISTRIBUTED_LAYERWISE`;
2. `LAYERWISE`;
3. `MODEL_LEVEL`; and
4. `NONE`.
The factory logs when a higher-priority strategy disables another. It derives
parallel and HSDP state from `DiffusionParallelConfig`; callers do not provide
a separate offload group size.
`get_offload_backend()` then validates platform offload support, resolves the
device, and creates exactly one backend. Returning `None` means offloading is
disabled or unsupported and must not leave partially installed hooks.
## Shared lifecycle
Every backend implements `OffloadBackend`:
- `enable(pipeline)` discovers modules, establishes initial residency, and
installs hooks;
- `disable()` removes owned hooks and resources; and
- `is_enabled()` reports lifecycle state.
`disable()` does not promise to restore the pipeline's original device
placement. The caller owns any subsequent rematerialization.
Hooks are registered through `HookRegistry` and `ModelHook`; offload backends
must use distinct hook names and remove only hooks they own.
## Discovery and topology
Pipeline component discovery prefers `SupportsComponentDiscovery` declarations:
- `_dit_modules`;
- `_encoder_modules`;
- `_vae_modules`; and
- `_resident_modules`.
Dotted paths are supported. Legacy pipelines may use the fallback scan of
well-known attribute names, but new integrations should declare components
explicitly.
Ordinary layerwise offload obtains topology from declared
`_layerwise_offload_blocks_attrs` (with a deprecated singular-name
compatibility path). Distributed layerwise offload may instead use a pipeline
`OffloadPlan`, then falls back to declared block attributes and its documented
compatibility discovery. Discovery metadata describes structure only; the
backend remains responsible for transfer, synchronization, and storage
ownership.
## Cross-strategy invariants
- At most one strategy owns offload hooks for a pipeline.
- A parameter has one authoritative host representation while offloaded.
- Device storage is not freed until dependent compute or transfer work is
complete.
- Non-persistent and model-specific buffers remain correct after movement or
checkpoint rematerialization.
- Unsupported parallel or loading combinations fail before hooks mutate the
model.
- Platform streams, events, synchronization, and cache management go through
the vLLM-Omni platform abstraction.
The diffusion [Offloader module design](../../module/diffusion/offloader.md)
describes how these feature contracts fit into the larger diffusion runtime.
docs/design/feature/offloader/cpu_offload.md
+56
View File
@@ -0,0 +1,56 @@
# Model-Level Offload
Model-level offload enforces mutual exclusion between pipeline component
groups. For commands and model support, see the
[model-level user guide](../../../user_guide/diffusion/offloader/cpu_offload.md).
## Generic hook path
`ModelLevelOffloadBackend.enable()` discovers DiTs, encoders, and VAEs. It
makes encoders and VAEs device resident for initialization, then installs
`SequentialOffloadHook` instances:
- each DiT hook offloads all encoders and other DiTs before loading itself;
- each encoder hook offloads every DiT before loading itself; and
- VAEs are not part of the mutual-exclusion hook set.
The hook moves parameters and buffers without recursively calling
`module.to()`. This avoids recursion through modules that retain references to
other transformer blocks. Host tensors are pinned when configured; DTensors
are not pinned through the ordinary tensor path.
Before forward returns control to the model, the hook synchronizes the active
platform so compute cannot race the transfer. XPU uses blocking device-to-host
copies because its allocator cannot safely evict storage while an asynchronous
copy is pending.
## Split-model delegation
If a pipeline provides callable `enable_omni_model_cpu_offload`, the backend
delegates instead of installing generic encoder/DiT hooks. This supports models
whose mutually exclusive components are nested inside one transformer, such
as a reasoner and generator.
The delegated pipeline owns its internal contexts, but must preserve the same
contract:
- only the executing component is device resident;
- enable and disable are idempotent;
- transfers use the supplied device, pinning, and HSDP settings; and
- disabling removes all model-local offload hooks or contexts.
The delegation is currently duck typed. If another independent implementation
is added, formalize the enable/disable pair as a protocol rather than expanding
attribute-name heuristics.
## Extension invariants
- New ordinary pipelines declare both DiT and encoder groups through
`SupportsComponentDiscovery`.
- Models with multiple DiTs offload the inactive DiTs as well as encoders.
- Empty or parameterless modules do not trigger transfers.
- Hook removal does not move modules; callers must not assume original
residency is restored.
Shared selection and lifecycle behavior is defined in the
[CPU Offloading design](README.md).
docs/design/feature/offloader/distributed_layerwise_offload.md
+555
View File
@@ -0,0 +1,555 @@
# Distributed Layerwise Offload
This document describes distributed layerwise offload (DLO) for diffusion
models. DLO keeps only a small number of DiT blocks on the accelerator and
streams the remaining blocks from host memory. The distributed backend can
either shard those host-side weights across an existing parallel group or keep
complete rank-local block sources and avoid an additional collective.
For user-facing commands, see the
[distributed layerwise offloading guide](../../../user_guide/diffusion/offloader/distributed_layerwise_offload.md)
and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
## Feature compatibility
Host-storage optimization and runtime compatibility are separate decisions.
When direct checkpoint mmap is unavailable, DLO can still use tensors produced
by the ordinary loader. "Compatibility path" below means that fallback is
implemented but has less end-to-end coverage than the primary path.
Legend: ✅ supported, ⚠️ compatibility path or limited validation, ❌ unsupported.
| Feature | DLO + AllGather | DLO without AllGather |
| --- | --- | --- |
| **DP** | ✅ Primary path; host weights are sharded across the DP group. | ✅ Each DP rank streams complete rank-local blocks. |
| **SP** | ✅ When DP=1, DLO uses the SP group for weight sharding. | ✅ SP remains active without a DLO weight collective. |
| **TP > 1** | ⚠️ Ordinary TP-aware loader only; no direct checkpoint mmap. | ⚠️ Ordinary TP-aware loader only; no direct checkpoint mmap. |
| **HSDP** | ❌ Rejected to avoid double-sharding parameters. | ⚠️ Limited end-to-end coverage. |
| **Per-tensor online FP8 linears** | ✅ Ordinary loader finalizes weights and scales before DLO sharding. | ✅ Ordinary loader retains complete rank-local tensors. |
| **Online INT8 linears** | ✅ Ordinary loader finalizes the int8 weight (contiguous or transposed view) and fp32 scale before DLO sharding. | ✅ Ordinary loader retains complete rank-local tensors. |
| **Online MXFP8 linears** | ✅ Ordinary loader finalizes the fp8 weight (contiguous) and e8m0 block scale (contiguous or transposed view) before DLO sharding. | ✅ Ordinary loader retains complete rank-local tensors. |
| **Other online quantization methods** | ❌ Rejected until runtime packing and scale layouts are validated. | ⚠️ Allowed through the ordinary loader; validation is method-specific. |
| **Model-level or standard layerwise CPU offload** | ❌ Disabled because DLO takes priority. | ❌ Disabled because DLO takes priority. |
| **Resident leading layers** | ❌ Rejected. | ✅ Requires eligible resident paths in the model's `OffloadPlan`. |
See [Parallelism compatibility](#parallelism-compatibility) and
[Request and loading constraints](#request-and-loading-constraints) for the
detailed contracts and validation boundaries.
## Status
DLO is implemented for multi-device diffusion execution. The default
AllGather path is the primary path for DP and SP deployments. The
`--dlo-no-use-allgather` path streams complete blocks independently and adds no
DLO weight collective.
Host storage is selected separately from the transfer protocol. The loader can
produce a direct-checkpoint mmap plan for a proven-compatible runtime layout;
otherwise it uses the ordinary loader. Consequently, no-AllGather replicas on
the same node can share immutable checkpoint pages when direct mmap is
selected, while the ordinary-loader fallback still keeps a private runtime
copy per process.
The Phase A shared-mmap support boundary is TP1. TP greater than one is an
ordinary-loader compatibility path: DLO can consume the resulting TP-local
tensors, but those configurations do not use checkpoint mmap and must not be
used to claim shared-mmap host-memory savings.
The compatibility matrix below describes the current implementation. The
unit-level guards are covered, but not every parallelism combination has a
full model-and-hardware end-to-end test.
## Design
### DLO consumes the existing parallel topology
DLO does not create a new DP, TP, or SP topology. It reads the configured
`DiffusionParallelConfig` and attaches offload hooks to the DiT blocks after the
standard distributed groups have been initialized.
The DLO weight-sharding group is selected as follows:
1. Use the existing DP group when `data_parallel_size > 1`.
2. When DP is one and SP is greater than one, use the SP group.
3. Otherwise, run rank-locally without a DLO process group.
TP is deliberately not used as DLO's AllGather group. HSDP has its own
parameter-sharding lifecycle and is not allowed to be sharded a second time by
DLO's AllGather path.
### The loader owns host-weight planning
Before it decides whether ordinary weight materialization can be skipped, the
diffusion loader builds one `HostWeightPlan`. A direct-checkpoint mmap plan is
accepted only when preflight proves all of the following:
- every required DiT parameter and persistent buffer has exactly one source;
- runtime names, checkpoint keys, shapes, and dtypes match;
- the runtime topology is TP1 without HSDP or online quantization; and
- every custom loader operation is represented by a loader-owned checkpoint
adapter.
The exact plan object is handed to DLO. The backend does not rescan checkpoint
files, repeat the capability decision, or reconstruct names from its block
topology. If preflight fails, the loader materializes weights normally and DLO
consumes those runtime tensors.
The plan owns only dedicated DiT component sources. If a pipeline also exposes
ordinary sources for a text encoder or another non-DiT component, the loader
still consumes those sources and includes their loaded names in its strict
coverage check. Only the source prefixes covered by the plan skip ordinary
materialization. A source that mixes DiT and non-DiT weights fails closed to
the complete ordinary-loader path because it cannot be skipped safely as a
unit.
This boundary keeps checkpoint semantics out of DLO and avoids model-pipeline
flags such as `_supports_mmap_loading` or parameter attributes for mmap-only
transforms. Model-specific direct-layout knowledge, when required, lives in a
checkpoint adapter beside the ordinary loader.
### AllGather path
With the default `dlo_use_allgather=True`, each rank stores approximately
`1 / group_size` of each streamable block in pinned host memory. The next
block's shard is copied to a device buffer and reconstructed with
`all_gather_into_tensor` on a communication stream while the current block is
executing.
```text
Compute: [Block N] [Block N+1] [Block N+2]
H2D: [shard N+1] [shard N+2]
AllGather: [full N+1] [full N+2]
Buffers: [current slot] [prefetch slot] [current slot]
```
![DLO double-buffer prefetch pipeline](../../figures/dlo/dlo_pipeline.gif)
The backend uses two shared device buffers, so accelerator weight residency is
bounded by the largest streamed blocks rather than the complete model.
When direct checkpoint mmap is selected, the checkpoint mappings are only the
source used to prepare each rank's persistent shard. They can be closed after
shard preparation. Across the AllGather group, those private shards total
approximately one runtime model copy.
An effective DLO group size of one performs no collective, even when
`dlo_use_allgather=True`; it follows the rank-local transfer path described
below.
When DP is greater than one, the engine can process one request per DP rank in
the same denoising wave. Because AllGather is a collective, all participating
requests must take the same execution path at every denoising step.
### Rank-local path without DLO AllGather
With `--dlo-no-use-allgather`, DLO forces its internal offload shard size to
one and streams complete blocks using H2D copies only. The host backing may be
either a loader-approved checkpoint mapping or ordinary runtime tensors.
For direct mmap, each process retains immutable safetensors views and uses two
bounded pinned host staging slots. Processes on the same node that map the same
files share physical checkpoint pages through the OS page cache. This removes
the persistent private full-model copy per pure-DP process, but each process
still packs and transfers every complete block. Sharing is node-local; each
node has its own page cache.
When direct mmap preflight fails, the regular model loader remains responsible
for preparing each rank's weights, including TP-local tensors or HSDP-managed
parameters. In that fallback, each pure-DP process keeps a private full runtime
copy.
This mode means:
- DP still provides independent replicas, but DLO does not shard weights
across DP ranks.
- SP still performs its normal activation/attention collectives, but DLO does
not shard weights across SP ranks.
- TP/HSDP/SP collectives, if configured, are not disabled by this flag; only
DLO's additional weight AllGather is disabled.
- Pure DP deployments share one checkpoint-backed copy per node when direct
mmap is selected; the ordinary-loader fallback keeps one private runtime
copy per rank.
- The scheduler does not require a synchronized DP request wave for DLO.
### Component allocator-cache retention
MiniMax-H3 DLO keeps one bounded PyTorch allocator cache across its staged
encoder, DiT, and VAE component boundaries so allocations released by one
component can be reused by the next. The DiT prefetch path and its two shared
device buffers are unchanged.
After a component is offloaded, cached-but-unallocated memory is retained only
while it is at most 25% of device capacity and at least 5% of device capacity
remains physically free. Crossing either bound releases the allocator cache.
Missing allocator telemetry also releases it conservatively. Component or
staging failure forces a release, and an out-of-memory allocation gets one
retry after release. This is a component-local policy rather than a global
`empty_cache` override, so the executor's unconditional shutdown cleanup is
preserved.
Retaining allocator blocks trades higher reserved and peak device memory for
lower lifecycle latency. The benefit is device- and workload-dependent, and a
smaller or externally loaded device may cross a bound and return to the
ordinary per-component cache-release behavior. Current performance validation
covers MiniMax-H3 no-AllGather DLO on CUDA. Other MiniMax-H3 DLO transfer
topologies and platforms receive the same bounded policy but are not claimed
performance targets; other DLO models remain unchanged unless they explicitly
adopt it.
### Final-layout Host Weight Runtime consumer
> **Status:** PR2 landed in
> [PR #6486](https://github.com/vllm-project/vllm-omni/pull/6486). PR3 adds
> registered direct H2D plus generic HWR publication/source-digest
> optimizations without changing AllGather behavior.
Final-layout Host Weight Runtime (HWR) backing is opt-in and currently applies
only to no-AllGather DLO. Enable it with
`--host-weight-runtime-mode preferred` (or `required`) and
`--host-weight-runtime-root <node-local-root>`.
The modes express operator fallback policy, not different artifact formats:
- `preferred` consumes an exact hit, but on a miss it allows canonical loading
followed by post-load publication. It is the normal population path.
- `required` consumes the same exact artifact but fails startup on a miss or
unusable artifact. It never invokes canonical DiT fallback or post-load
publication and therefore cannot populate an empty store.
PR2 has no separate prewarm command. Operators populate one matching producer
cohort per node-local storage domain in `preferred` mode, then restart the same
model revision and parallel layout with `required`. TP coordinates have distinct
identities, while equivalent DP replicas share them.
The current producer/consumer boundary is the model-declared final-layout BF16
contract (MiniMax H3 and `black-forest-labs/FLUX.2-klein-4B`). The FLUX.2-klein
contract covers both transformer block stacks, constructor-stable packed QKV
mapping state, BF16 parameters, and any persistent loader-owned `beta`/`eps`
buffers. The shared HWR machinery supports ordinary-loader final layouts for
TP1 and TP2 rank identities plus SP layout identities. FLUX.2-klein evidence
in this change is TP1-only; its TP2/SP layouts rely on the existing
per-coordinate identity mechanics and remain unmeasured. Online quantization,
HSDP, LoRA/adapted weights, and non-default load formats remain ineligible.
HWR mode
`disabled`, DLO-disabled, and DLO AllGather configurations stop before source
identity or store construction and retain the existing checkpoint-mmap or
ordinary-loader path.
Eligibility is decided before constructing HWR, resolving canonical sources,
hashing identity inputs, probing a filesystem, or emitting an HWR observer
event:
```text
DLO disabled / HWR disabled / DLO AllGather
-> zero final-layout HWR interaction
-> preserve current checkpoint-mmap or ordinary-loader behavior
```
When final-layout HWR is explicitly enabled for no-AllGather DLO, runtime
outcomes are authoritative:
- `LOCAL_HIT`: plan and commit an exact restoration, then transfer the lease
transactionally to DLO.
- `CANONICAL_FALLBACK`: bypass checkpoint mmap, canonically materialize and
finalize the DiT, then call `publish_after_load()` for a future startup. The
current startup retains its canonical tensors.
- `FAILED`: fail startup. Preferred mode does not reinterpret nonretryable
identity, configuration, or compatibility failures as misses.
- Required mode cannot bootstrap an empty store through a `POST_LOAD_ONLY`
producer.
Checkpoint mmap remains an unchanged control path whenever final-layout HWR is
not selected.
Local source identity and publication avoid repeated full-payload passes while
preserving the same semantic contract:
- Canonical files without a trusted immutable Hub blob identity are hashed in
parallel. Their content IDs are cached per storage domain under the HWR root
and reused only when the path, inode, size, timestamps, symlink target, and
cache-record checksum still match. Corrupt entries are rebuilt; cache-lock or
cache-I/O failure falls back to hashing the canonical file directly.
- Producers that emit payload bytes in canonical storage-key order may declare
`ordered=True`. The filesystem writer then computes file and tensor SHA256
values during the write and overlaps payload `fsync` with later producer
work. Producers without that guarantee retain unordered writes and use a
parallel readback checksum fallback.
- Manifest and `READY` publication still wait for every payload `fsync` and
checksum to complete. These optimizations do not weaken artifact validation
or change lease ownership.
#### Registered mmap transport
After DLO takes an HWR lease, the no-AllGather backend may register the lease's
complete immutable mapped ranges under the existing `pin_cpu_memory` policy.
Registration is transport state: it does not change artifact identity, the
store, tensor ownership, or H2D payload. On success, each tensor view copies
directly into the existing rotating HBM block buffers and the two private host
staging slots are not allocated.
`--dlo-host-registration-limit-gib` is an optional per-worker preflight ceiling
over page-aligned registered bytes. Zero adds no ceiling. A disabled pinned
memory policy, unsupported platform/capability, over-budget mapping, or fully
rolled-back registration error selects the existing two-slot staging path. A
partial registration that cannot be rolled back aborts startup because closing
the lease would unmap memory still owned by the platform.
Direct checkpoint mmap remains unchanged and continues to use staging. It may
require loader-owned per-block transforms, while the HWR artifact already
contains final runtime bytes. DLO AllGather never receives an HWR final-layout
lease and therefore never enters this registration path.
#### Pre-service transaction boundary
The startup transaction does not end at restore commit. A lease-backed model is
disposable until backend setup and initial prefetch complete:
```text
UNRESOLVED
-> LEASE_OWNED_BY_LOADER
-> RESTORE_PLANNED
-> COMMIT_STARTED
-> CARRIER_OWNED
-> BACKEND_OWNED
-> BACKEND_READY
-> IN_SERVICE
```
Failure before `COMMIT_STARTED` closes the loader-owned lease. Preferred mode
may canonically load using the untouched skeleton; required mode fails.
Failure from `COMMIT_STARTED` through `BACKEND_READY` must:
1. synchronize or otherwise quiesce partial backend and initial-prefetch work;
2. release hook, staging, restore-plan, and model references;
3. close the lease through its current owner;
4. discard the restored model; and
5. in preferred mode, construct a fresh canonical model while bypassing HWR
lookup, HWR publication, and checkpoint mmap for that recovery attempt.
Required mode fails instead of constructing the fresh fallback. Once the model
enters service, failures follow normal runtime handling rather than startup
fallback.
#### Finalization phases
Cold canonical loading separates byte-changing work from shared runtime-state
finalization:
```text
Cold-only byte-changing:
casting, reordering, packing, quantization, generated scales,
model-specific weight transforms, and calibration that mutates
parameters or persistent buffers
Shared cold/warm non-byte:
validation, hook installation, eval state, bookkeeping,
and non-persistent runtime state
```
The warm path must snapshot restored tensor bytes and backing pointers before
shared finalization and prove both are unchanged afterward.
#### Lease ownership
Lease transfer is single-owner and single-take:
```text
loader owns
-> restore plan borrows
-> commit succeeds
-> runner carrier takes once
-> backend takes once before asynchronous work
-> backend drains pending H2D work
-> backend releases hooks, staging, and model references
-> backend unregisters every mapped range
-> backend closes lease
```
The runner carrier is process-local, rejects serialization, and rejects a
duplicate `take()`. If the carrier still owns the lease, runner cleanup closes
it. Once the backend takes ownership, only backend abort or teardown may drain
work and close it.
The implementation keeps the final-layout lease through backend setup and
initial prefetch. Successful registration copies directly from immutable HWR
views; otherwise the backend uses the existing two bounded rank-local staging
slots. Transactional cleanup drains device work, removes source references,
unregisters mappings, and only then closes the lease. An unregistration failure
retains both registration and lease for retry/process teardown. Preferred mode
then uses the runner's fresh canonical retry; required mode propagates the
failure.
#### PR2 and PR3 promotion gates
- Warm hit performs zero ordinary DiT materialization and zero producer calls.
- Shared warm finalization changes neither restored bytes nor backing pointers.
- Preferred `FAILED` outcomes do not silently fall back.
- Planning failure leaves the skeleton untouched.
- Any pre-service failure after commit begins discards the restored model;
preferred mode uses a fresh canonical model and required mode fails.
- Mixed components load normally and no required tensor remains on `meta`.
- Disabled mode and AllGather emit zero final-layout HWR interaction.
- The lease carrier rejects duplicate take and serialization.
- Backend setup or prefetch failure drains asynchronous work before lease close.
- Registration is all-or-nothing; rollback/unregistration failure never closes
a mapping still owned by the platform.
- Registered HWR transport bypasses host staging while preserving output and
H2D payload; unsupported registration retains the bounded staging path.
- Compatible checkpoint mmap remains an unchanged benchmark and control path.
- Prewarm uses one matching producer cohort per TP/SP topology and storage
domain.
- Benchmarks compare ordinary loading, cold publication, and warm staging with
output parity, startup latency, aggregate PSS, page-cache state, HBM, and H2D
payload.
## Parallelism compatibility
| Parallelism | DLO + AllGather | DLO without AllGather |
| --- | --- | --- |
| **DP** | Supported primary path. DLO shards host weights across the DP group and can run DP multi-concurrency. | Supported rank-local path. Compatible TP1 replicas can share checkpoint pages on each node; fallback runtime tensors remain private. |
| **SP** | Supported in the implementation. With DP=1, DLO uses the SP group for host-weight sharding; SP still shards sequence/activation work. | SP remains active, but DLO keeps standard-loader rank-local weights and adds no SP weight collective. |
| **TP > 1** | Outside the Phase A shared-mmap support scope. The loader falls back before mutation, preserves TP-local layouts, and DLO may apply DP/SP host sharding to those ordinary runtime tensors. | Outside the Phase A shared-mmap support scope. The ordinary TP-aware loader produces rank-local tensors, which DLO streams without an additional weight collective; DP replicas retain private runtime storage. |
| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted by configuration. HSDP owns parameter sharding and its own gathers; DLO only stages rank-local parameters. End-to-end coverage is limited. |
### Combined dimensions
- **DP + SP:** DLO uses the DP group for weight sharding when DP is greater
than one; SP continues to use its own sequence-parallel group. If DP is one,
the SP group becomes DLO's sharding group in AllGather mode.
- **DP + TP/SP without AllGather:** standard model loading defines the
rank-local tensor layout. DLO adds no cross-DP, cross-TP, or cross-SP weight
collective. When the model declares the final-layout contract, HWR keys the
reusable artifact by TP rank/size and SP layout.
- **HSDP + SP:** the general parallel configuration permits HSDP over SP, but
DLO must use `--dlo-no-use-allgather`. HSDP remains responsible for weight
materialization and synchronization.
- **HSDP + DP or TP:** rejected independently by the diffusion parallel
configuration.
## Request and loading constraints
AllGather DP multi-concurrency requires:
- explicit `num_inference_steps`;
- the same `num_inference_steps` for all requests in a wave; and
- identical request arguments that affect the collective execution path.
The no-AllGather path does not impose these DLO-specific synchronized-wave
requirements.
Direct checkpoint mmap can back either transfer path. It is currently limited
to proven TP1, non-HSDP, non-online-quantized layouts. Other layouts use the
ordinary loader. Per-tensor online FP8, online INT8, and online MXFP8 linears
can use DLO AllGather after the ordinary loader finalizes their runtime
weights and scales; DLO then shards and reconstructs those tensors with their
recorded layouts (they all keep plain 1-byte dtypes over ordinary strided
views: online INT8 has an int8 weight plus fp32 scale, online MXFP8 has an
fp8 weight plus e8m0 block scale, each with a contiguous or transposed-view
finalized tensor, all covered by the same physical-order packing as online
FP8). Other online methods must use `--dlo-no-use-allgather` or disable
online quantization until their runtime layouts are validated.
The Host Weight Runtime representation, publication, and no-AllGather consumer
contracts are merged; see
[RFC #6414](https://github.com/vllm-project/vllm-omni/issues/6414),
[PR #6445](https://github.com/vllm-project/vllm-omni/pull/6445), and
[PR #6486](https://github.com/vllm-project/vllm-omni/pull/6486). Registered
direct H2D is an optional transport layer over that merged lease contract.
## Validation coverage
Current source-level validation includes:
- HSDP + DLO + AllGather rejection;
- HSDP + DLO without AllGather acceptance at configuration level;
- loader preflight fallback for TP, HSDP, online quantization, unknown custom
loaders, missing keys, and shape/dtype mismatches;
- ordinary-loader fallback for per-tensor online FP8, online INT8, and online
MXFP8 linears followed by DLO sharding of finalized weights and scales;
- exact loader-to-backend plan transfer and ordinary-loader fallback;
- ordered publication hashing, overlapped durability, unordered parallel
checksum fallback, and node-local source-digest reuse/invalidation;
- rank-local mmap source retention, bounded two-slot staging, and adapter
transforms without parameter-side flags;
- read-only registration preflight, direct source-to-device copies, safe
fallback, partial rollback, retryable unregistration, and lease ordering;
- resident-layer requests requiring no-AllGather;
- DP request-wave validation for denoising-step compatibility;
- bounded component allocator-cache retention, conservative/forced release,
OOM retry, and immutable encoder non-block staging;
- sharding, double-buffer, AllGather-size, and heterogeneous-block regression
tests.
### B300 parallel-topology smoke matrix
A four-GPU B300 smoke test covered MiniMax-H3 FL2VA with the same prompt, seed,
CUDNN attention backend, 256x256 output, two denoising steps, and
`dlo_resident_layers=0`. The TP2 rows used DiT DP2xTP2 with the text encoder and
VAEs at TP1. They validate the ordinary-loader fallback only, not direct mmap
or shared-mmap host-memory savings.
| Configuration | Result | Warm E2E | Peak device memory | Host PSS |
| --- | ---: | ---: | ---: | ---: |
| DP4xTP1 AllGather | Passed, 4 concurrent requests | 2.87 s / 4 requests | 13.84 GiB | 211.99 GiB |
| DP4xTP1 no-AllGather | Passed, 1 request | 15.02 s | 13.23 GiB | 187.77 GiB |
| DP2xTP2 AllGather | Passed, 2 concurrent requests | 4.16 s / 2 requests | 12.50 GiB | 211.97 GiB |
| DP2xTP2 no-AllGather | Passed, 1 request | 3.51 s | 11.88 GiB | 314.01 GiB |
Within each topology, the AllGather and no-AllGather video and audio outputs
were byte-identical. All four runs completed without an `ERROR` or traceback
and released their device allocations. For DP4xTP1, no-AllGather direct mmap
reduced total PSS by 24.22 GiB (11.4%) and `Private_Dirty` from 211.33 to
125.32 GiB (40.7%) relative to AllGather. For DP2xTP2, preflight selected the
ordinary loader as designed; no-AllGather PSS was 314.01 GiB, about 48% above
AllGather, because DP replicas did not share checkpoint-backed runtime
weights. This is a functional and memory smoke test, not a production-quality
performance or output-quality benchmark.
### Host-memory measurement
A two-worker MiniMax-H3 FL2VA measurement on one L20X node compared the
ordinary-loader fallback with direct mmap. Both runs used
DP=2, TP=1, no DLO AllGather, BF16 weights, two denoising steps, and a
256x256 four-second request. The ordinary-loader workers were sampled after
initialization. The mmap workers were sampled after one completed request, so
the checkpoint working set had been faulted into the page cache; this is the
more conservative point for mmap.
The values below come from `/proc/<worker>/smaps_rollup` and include the whole
worker, not only the DiT. The stable rank-to-rank difference comes from other
pipeline components, so each worker should be compared with the same worker in
the other storage mode.
| Worker | Ordinary RSS | mmap RSS | Ordinary PSS | mmap PSS | PSS reduction |
| --- | ---: | ---: | ---: | ---: | ---: |
| DP worker 0 | 168.27 GiB | 132.76 GiB | 167.84 GiB | 101.43 GiB | 66.40 GiB |
| DP worker 1 | 116.19 GiB | 79.97 GiB | 115.73 GiB | 48.64 GiB | 67.09 GiB |
| **Two-worker total** | — | — | **283.56 GiB** | **150.08 GiB** | **133.48 GiB (47.1%)** |
The direct-mmap workers each reported 62.45 GiB `Shared_Clean` but only
31.20 GiB `Pss_File`, which is the proportional charge expected when the same
resident checkpoint pages are mapped by two workers. `Private_Dirty` also fell
from 167.53 to 70.24 GiB for worker 0 and from 115.40 to 17.44 GiB for worker
1, a reduction of about 97–98 GiB per worker. RSS understates this benefit
because it counts a shared physical page in every process that maps it; summed
PSS is the appropriate node-memory comparison.
The highest-value missing coverage is broader end-to-end numerical and
lifecycle comparison against ordinary layerwise offload for DP+SP,
HSDP+SP+no-AllGather, and TP greater than one across additional models and
target CUDA/NCCL or CANN/HCCL hardware. That broader TP coverage does not
change the Phase A direct-mmap TP1 support boundary.
## Recommendations
- Use **DP + DLO AllGather** for the supported throughput and host-memory
scaling path.
- Use **SP + DLO AllGather** for long-sequence workloads when DP concurrency is
not the goal.
- Use **no-AllGather** when independent replica execution is required. TP1
direct-mmap deployments can share checkpoint pages per node; other layouts
retain the ordinary loader's private host memory behavior and are outside
the Phase A shared-mmap support scope.
- Prefer **HSDP alone** for production HSDP deployments until the combined
HSDP + DLO no-AllGather path has broader end-to-end coverage.
docs/design/feature/offloader/layerwise_offload.md
+57
View File
@@ -0,0 +1,57 @@
# Layerwise Offload
Layerwise offload streams transformer blocks through one accelerator. For
commands and model support, see the
[layerwise user guide](../../../user_guide/diffusion/offloader/layerwise_offload.md).
## Hook ring
`LayerWiseOffloadBackend` discovers every streamable block and registers one
`LayerwiseOffloadHook` per block. Hooks form a ring: each hook owns the pinned
host representation of its registered block and a reference to the next block
it must prefetch; the final hook prefetches the first block for the next
denoising iteration.
During initialization, parameters and buffers are grouped by dtype and copied
into contiguous pinned CPU tensors. Original tensor storage is replaced with
an empty or meta placeholder while identity and module structure are retained.
Metadata records names, offsets, shapes, and dtypes for rematerialization.
## Forward lifecycle
For each block:
1. the previous hook's completion event ensures this block is materialized;
2. the copy stream asynchronously allocates and fills the next block's device
tensors;
3. parameter and buffer storage is rebound to views of those tensors;
4. the current block computes on the normal stream; and
5. its post-forward hook replaces device storage with placeholders after all
dependent work is safe.
The copy stream waits on the compute stream before reusing storage, and the
compute stream waits on the per-block prefetch event before forward. These
dependencies, not incidental global synchronization, define correctness.
## Topology contract
A DiT declares ordered block-container attributes using
`_layerwise_offload_blocks_attrs` or an `OffloadPlan`. Each resolved item must
be an executable `nn.Module`; ordering must match forward execution. Multiple
containers are concatenated in execution order.
Non-block DiT modules, encoders, VAEs, and declared resident modules are not
streamed by this backend.
## Invariants and limitations
- Each offloaded tensor has one pinned host backing store.
- Parameter identity is preserved when storage is rebound.
- Blocks with heterogeneous shapes and dtypes use their recorded metadata;
implementations must not assume identical blocks.
- The ring is initialized before the first forward so block zero is available.
- This backend is single-device; distributed weight reconstruction belongs to
[Distributed Layerwise Offload](distributed_layerwise_offload.md).
Shared selection and lifecycle behavior is defined in the
[CPU Offloading design](README.md).
docs/design/feature/omni_connectors/mooncake_store_connector.md
+12 -13
View File
@@ -32,26 +32,25 @@ mooncake_master \
## Configuration
Define the connector in runtime:
Define the connector at the top level of the deploy YAML:
```yaml
runtime:
connectors:
connector_of_mooncake:
name: MooncakeStoreConnector
extra:
host: "127.0.0.1"
metadata_server: "http://<MASTER_IP>:8080/metadata"
master: "<MASTER_IP>:50051"
segment: 512000000
localbuf: 64000000
proto: "tcp"
connectors:
connector_of_mooncake:
name: MooncakeStoreConnector
extra:
host: "127.0.0.1"
metadata_server: "http://<MASTER_IP>:8080/metadata"
master: "<MASTER_IP>:50051"
segment: 512000000
localbuf: 64000000
proto: "tcp"
```
Wire stages to the connector:
```yaml
stage_args:
stages:
- stage_id: 0
output_connectors:
to_stage_1: connector_of_mooncake
docs/design/feature/omni_connectors/mooncake_transfer_engine_connector.md
+13 -14
View File
@@ -21,26 +21,25 @@ InfiniBand/RoCE NICs).
## Configuration
Define the connector in runtime:
Define the connector at the top level of the deploy YAML:
```yaml
runtime:
connectors:
rdma_connector:
name: MooncakeTransferEngineConnector
extra:
host: "auto" # Auto-detect local RDMA IP
zmq_port: 50051 # ZMQ base port (see "Port Offset Scheme" below)
protocol: "rdma" # "rdma" or "tcp"
device_name: "" # RDMA device (e.g., "mlx5_0"), empty for auto-detect
memory_pool_size: 4294967296 # 4 GB (CPU); use 2147483648 (2 GB) for GPU
memory_pool_device: "cpu" # "cpu" for pinned memory (recommended), "cuda" for GPUDirect RDMA
connectors:
rdma_connector:
name: MooncakeTransferEngineConnector
extra:
host: "auto" # Auto-detect local RDMA IP
zmq_port: 50051 # ZMQ base port (see "Port Offset Scheme" below)
protocol: "rdma" # "rdma" or "tcp"
device_name: "" # RDMA device (e.g., "mlx5_0"), empty for auto-detect
memory_pool_size: 4294967296 # 4 GB (CPU); use 2147483648 (2 GB) for GPU
memory_pool_device: "cpu" # "cpu" for pinned memory (recommended), "cuda" for GPUDirect RDMA
```
Wire stages to the connector:
```yaml
stage_args:
stages:
- stage_id: 0
output_connectors:
to_stage_1: rdma_connector
@@ -415,7 +414,7 @@ A receiver connector:
- allocates receive buffers from its own pool
- requests metadata or transfer service from the sender
The role is not inferred dynamically. It is injected by the stage configuration layer:
The role is not inferred dynamically. It is injected from the deploy connector wiring:
- incoming edge for a stage -> `role="receiver"`
- outgoing edge for a stage -> `role="sender"`
docs/design/feature/omni_connectors/shared_memory_connector.md
+3 -4
View File
@@ -13,10 +13,9 @@ segment name is returned in metadata.
## Configuration
```yaml
runtime:
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
```
## Notes
docs/design/feature/omni_connectors/yuanrong_connector.md
+9 -10
View File
@@ -65,23 +65,22 @@ dscli stop --worker_address "${WORKER_IP}:31501"
## Configuration
Define the connector in runtime:
Define the connector at the top level of the deploy YAML:
```yaml
runtime:
connectors:
connector_of_yuanrong:
name: YuanrongConnector
extra:
host: "127.0.0.1"
port: 31501
get_sub_timeout_ms: 1000
connectors:
connector_of_yuanrong:
name: YuanrongConnector
extra:
host: "127.0.0.1"
port: 31501
get_sub_timeout_ms: 1000
```
Wire stages to the connector:
```yaml
stage_args:
stages:
- stage_id: 0
output_connectors:
to_stage_1: connector_of_yuanrong
docs/design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
+31 -38
View File
@@ -94,26 +94,23 @@ For Ascend P2P, configure the connector with `protocol: "ascend"` and an NPU
memory pool:
```yaml
runtime:
enabled: true
defaults:
connectors:
yuanrong_te_connector:
name: YuanrongTransferEngineConnector
extra:
host: "auto"
zmq_port: 50051
rpc_port: "auto"
protocol: "ascend"
device_name: "auto"
memory_pool_size: 1073741824
memory_pool_device: "npu"
edges:
- from: 0
to: 1
window_size: -1
max_inflight: 1
connectors:
yuanrong_te_connector:
name: YuanrongTransferEngineConnector
extra:
host: "auto"
zmq_port: 50051
rpc_port: "auto"
protocol: "ascend"
device_name: "auto"
memory_pool_size: 1073741824
memory_pool_device: "npu"
edges:
- from: 0
to: 1
window_size: -1
```
Important fields:
@@ -141,31 +138,27 @@ For AR 4-way TP sending KV cache to DiT 4-way TP, set the stage TP topology in
both stage `omni_kv_config` blocks:
```yaml
stage_args:
stages:
- stage_id: 0
runtime:
devices: "0,1,2,3"
engine_args:
tensor_parallel_size: 4
omni_kv_config:
need_send_cache: true
rank_mapping:
from_tp: 4
to_tp: 4
devices: "0,1,2,3"
tensor_parallel_size: 4
omni_kv_config:
need_send_cache: true
rank_mapping:
from_tp: 4
to_tp: 4
output_connectors:
to_stage_1: yuanrong_te_connector
- stage_id: 1
runtime:
devices: "4,5,6,7"
engine_args:
parallel_config:
tensor_parallel_size: 4
omni_kv_config:
need_recv_cache: true
rank_mapping:
from_tp: 4
to_tp: 4
devices: "4,5,6,7"
parallel_config:
tensor_parallel_size: 4
omni_kv_config:
need_recv_cache: true
rank_mapping:
from_tp: 4
to_tp: 4
input_connectors:
from_stage_0: yuanrong_te_connector
```
docs/design/feature/prefix_caching.md
+1 -1
View File
@@ -41,7 +41,7 @@ With this in mind, consider the set of blocks in a 2D layout, where the row repr
### Example
!!! note "Note 3"
Prefix caching in vLLM-Omni currently is only supported on AutoRegressive stages with one kv-cache group. It can be enabled/disabled per-stage via the `enable_prefix_caching` parameter in the model's stage config.
Prefix caching in vLLM-Omni currently is only supported on AutoRegressive stages with one kv-cache group. Configure it with the pipeline-wide `enable_prefix_caching` field in the deploy config.
The way in which vLLM-Omni ties into vLLM's prefix caching is best understood by example. Say that we have the following:
docs/design/feature/quantization.md
+20 -9
View File
@@ -76,6 +76,12 @@ equivalent. `auto-round` and `auto_round` use the Omni INC/AutoRound adapter.
ModelOpt checkpoint metadata is detected from its method, producer metadata, or
`quant_algo`, then mapped to the corresponding vLLM ModelOpt configuration.
Unless the input is a per-component dictionary, the resolved configuration is
passed unchanged to every quantization-aware component constructed by the
pipeline. It affects only layers supported by that component's quantization
method; it does not rewrite arbitrary `torch.nn` modules. A per-component
dictionary narrows this global scope through runtime layer-prefix routing.
The factory uses Omni-specific overrides for methods whose diffusion behavior
or platform dispatch is not provided by the vLLM registry. The current override
set includes INT8, BitsAndBytes, MXFP8, MXFP4, MXFP4 dual-scale, and INC/
@@ -154,9 +160,11 @@ quantizable vLLM layers. Each method configuration is responsible for:
Model code must keep precision-sensitive or unsupported modules unquantized.
The shared `safe_quant_config()` helper is used by model integrations for
normalization and modulation layers where a generic FP8 configuration would be
unsafe. Text encoders, audio/vision encoders, VAEs, schedulers, and tokenizers
are not implicitly quantized just because the main transformer is quantized;
they require explicit component routing and method support.
unsafe. Encoders exposed through the quantization factory receive a global
configuration just like the main transformer, but only their supported vLLM
quantizable layers are affected. Ordinary `torch.nn` encoder layers, VAEs,
schedulers, and tokenizers are not rewritten automatically. A component map can
select a narrower scope, but it cannot add quantization support to a layer.
Weight quantization and quantized KV cache are separate paths. The former is
resolved through `quantization_config`; runtime attention/KV-cache quantization
@@ -182,12 +190,15 @@ configured parallelism strategy:
- `WeightsMapper` transformations must be applied before matching ignored-layer
lists or component prefixes.
Distributed layerwise offload has an additional boundary: its sharded mmap
AllGather path is not compatible with online quantization because the path
expects already materialized checkpoint weights. Use the rank-local
`--dlo-no-use-allgather` path or a compatible pre-quantized checkpoint when
combining these features. See the [DLO design document](distributed_layerwise_offload.md)
for the full compatibility matrix.
Distributed layerwise offload has an additional boundary: direct checkpoint
mmap cannot prepare online-quantized tensors because it bypasses the ordinary
loader that creates their runtime weights and scales. Per-tensor online FP8
linears can still use DLO AllGather through the ordinary loader: each rank first
finalizes the FP8 tensors, then DLO shards those runtime tensors for
reconstruction during prefetch. Other online methods remain limited to
`--dlo-no-use-allgather` until their packing and scale layouts are validated.
See the [DLO design document](offloader/distributed_layerwise_offload.md) for
the full compatibility matrix.
## Adding a quantization backend
docs/design/feature/realtime_ar_diffusion.md
+213
View File
@@ -0,0 +1,213 @@
# Realtime AR-Diffusion sessions
> **Status:** experimental. This document describes the internal execution
> contract implemented under `vllm_omni.experimental.ar_diffusion`. It does not
> define a public HTTP or WebSocket schema.
## Scope
Realtime AR-Diffusion serves world models that generate one autoregressive
latent block per request while retaining model state across requests. The
generic runtime owns identity, event ordering, session lifecycle, routing, and
KV allocation. A model adapter owns model-specific controls and conditioning.
LingBot World v2 is the first single-KV-branch integration. DreamZero uses the
same runner capability with two logical CFG branches.
## Identities
Four identifiers have distinct lifetimes:
| Identifier | Lifetime | Owner | Contract |
| --- | --- | --- | --- |
| `session_id` | One persistent world | Session manager | Selects runner KV and model-owned state. |
| `event_id` | One control/prompt update | Transport adapter | Unique and monotonically increasing within a session, including across reset. |
| `chunk_index` | One committed AR block position | Session | Contiguous from zero; reset starts again at zero. |
| `request_id` | One chunk execution | Session | Correlates the tick with its returned metadata. |
`AsyncOmni` keeps its normal UUID-suffixed engine routing ID. The tick
`request_id` remains inside the immutable AR-Diffusion snapshot and the
standard output metadata, while caller-visible `OmniRequestOutput.request_id`
continues to use AsyncOmni's existing external-ID contract. Protocol identity
therefore does not require a private entrypoint override.
## Request lifecycle
1. `ARDiffusionSession.accept_event()` validates and queues an immutable
`ARDiffusionSessionEvent`. Prompt and opaque control tracks may be updated in
the same event.
2. `ARDiffusionSession.next_chunk()` serializes chunk execution and calls
`_build_tick_locked()`. The resulting `ARDiffusionTickRequest` is a snapshot
of the current prompt, reduced controls, ordered event IDs, and chunk index.
3. `ARDiffusionOmniTickConsumer.execute_tick()` copies the stage sampling
parameters, stores the typed tick in
`sampling_params.extra_args["ar_diffusion_tick"]`, constructs the normal Omni
prompt through its `prompt_provider`, and invokes `AsyncOmni.generate()`.
4. `AsyncOmni` allocates its normal unique engine routing ID and submits one
`OmniDiffusionRequest` through the orchestrator.
5. `ARDiffusionModelRunner._request_session()` parses the typed tick.
`_get_or_create_session()` obtains runner-owned paged KV, and
`bind_ar_diffusion_state()` exposes it to the pipeline only for the duration
of `execute_model()`.
6. The model pipeline validates the tick and generates exactly one latent
block. LingBot interprets camera controls in
`LingBotCameraControlReducer` and
`LingBotWorldCausalDMDPipeline.forward()`; the generic runtime does not parse
camera or action schemas.
7. The pipeline returns one `DiffusionOutput` with the generated payload and an
`ARDiffusionChunkMetadata` mapping under `metadata["ar_diffusion"]`.
`AsyncOmni` exposes it as
`OmniRequestOutput.multimodal_output["metadata"]["ar_diffusion"]`.
8. The consumer parses the standard envelope. The session commits its reducer
state, prompt, controls, event IDs, and next chunk index only when returned
metadata exactly equals the submitted tick snapshot.
The metadata envelope is:
```json
{
"metadata": {
"ar_diffusion": {
"session_id": "world-7",
"request_id": "ar-world-7-3-...",
"chunk_index": 3,
"applied_event_ids": [10, 11]
}
}
}
```
## Transaction and failure semantics
Accepted events are not acknowledged as applied until the model output and
metadata are validated. Reducer `prepare()` is speculative; reducer `commit()`
and the session snapshot form one logical commit. A model, metadata, or reducer
failure leaves the chunk index and queued events uncommitted and transitions the
session to `FAILED`.
After a tick failure, worker state is closed because a forward may have
partially modified or already released KV. The same chunk is not retried in
place. A caller must explicitly reset the session, starting again at chunk
zero, or close it.
Explicit close follows this state machine:
```text
ACTIVE or FAILED
|
v
CLOSING ---- worker cleanup succeeds ----> CLOSED
|
+------- worker cleanup fails --------> CLEANUP_FAILED
|
+-- close retry --> CLOSING
```
`CLEANUP_FAILED` is a local tombstone. The session manager retains it, rejects
creation of the same `session_id`, and rejects reset. Only an idempotent close
retry that succeeds removes the session. Transport disconnect uses the same
path.
## KV ownership and capacity
`ARDiffusionKVCacheSpec` is the pipeline-owned immutable geometry contract. It
declares:
- TP-local self-attention geometry, AR block size, sink, and recent window;
- logical KV branches and their worker-local slots;
- fixed-length cross-attention caches;
- scratch tokens required by an uncommitted forward;
- persistent model-owned CUDA bytes per resident session; and
- the requested resident-session capacity.
The runner computes:
```text
required(capacity) =
scratch
+ managed_self_attention_pages(capacity)
+ capacity * cross_attention_bytes_per_session
+ capacity * model_owned_state_bytes_per_session
```
`gpu_memory_fraction` is a soft expansion budget:
```text
configured_budget = available_device_bytes * gpu_memory_fraction
effective_budget = max(configured_budget, required(1))
```
If `required(1)` exceeds actual available device memory, initialization fails.
Otherwise the runner always admits one viable session and uses the configured
fraction to select additional resident sessions up to the model-declared cap.
All model-owned reservations are deducted before the paged self-attention pool
is allocated.
LingBot reports its persistent full-horizon image-condition tensor. DreamZero
reports the measured 603 MiB per-session Wan VAE causal-convolution state.
## Concurrency and routing limits
The current runner executes one request at a time:
- `max_num_seqs` must be one;
- request batching and diffusion step execution are rejected;
- one AR block is generated per engine request; and
- an AR-Diffusion stage must have exactly one replica.
The single-replica restriction is intentional. Multi-replica support requires
session-affine routing so every tick, reset, and close reaches the worker that
owns the session. Random or round-robin routing is not correct.
Multiple resident sessions may share one runner. Capacity selection accounts
for all persistent pools, and the runner performs LRU eviction through the same
model lifecycle hook used by explicit close.
## Model boundary
The generic runtime transports controls as opaque
`ARDiffusionControlInput(track, schema, data)` values. Model adapters validate
and reduce those values. This keeps target/velocity, key state, pose,
intrinsics, and other model-specific conventions out of the session protocol.
For LingBot World v2:
- `lingbot_world/actions.py` parses key-state and trajectory controls;
- `lingbot_world/camera.py` loads or constructs camera trajectories and Plücker embeddings;
- `lingbot_world/transformer.py` implements checkpoint-compatible causal attention; and
- `lingbot_world/pipeline.py` constructs conditioning, owns small non-KV session state, and
produces one block plus the standard metadata envelope.
## Non-goals
This contract does not currently provide:
- a public HTTP or WebSocket request schema (the structured-interaction
frontend is tracked separately in
[#5527](https://github.com/vllm-project/vllm-omni/pull/5527));
- camera/action semantics shared by every world model;
- session migration or replication across workers;
- retry of an ambiguous partially executed chunk;
- cross-stage KV transfer; or
- stateful streaming VAE decode.
Those features can be layered on top without adding LingBot-specific identity
or lifecycle fields to the generic runtime.
## Required regression coverage
CPU contract tests cover:
- separation of tick correlation IDs from internal engine routing IDs;
- event ordering, metadata equality, and reducer fail-closed commit;
- close/disconnect failure tombstones and cleanup retry;
- capacity one under the default fraction for shipped LingBot geometry;
- capacity expansion under a tuned fraction;
- DreamZero model-owned state accounting;
- runner session reuse, reset, close, LRU eviction, and forward cleanup; and
- LingBot model registration, imports, camera/action reduction, and one-block
output metadata.
GPU validation must additionally exercise real model loading, action input,
prompt switching, rolling-window replacement, reset, close, finite output
latents, and exact metadata/request identity.
docs/design/feature/skip_softmax.md
+4 -2
View File
@@ -2,8 +2,10 @@
Skip-Softmax is the sparse-attention mode of the `TRTLLM_ATTN` backend. Usage — the config keys
and how to pick an operating point — is in
[Diffusion Attention Backends](../../user_guide/diffusion/attention_backends.md). This page explains
the algorithm.
[TRTLLM Attention](../../user_guide/diffusion/attention_backends/trtllm.md#skip-softmax).
The shared selector contract is documented in
[Diffusion Attention Backend Selection](attention_backend_selection.md). This
page explains the algorithm.
## Motivation
docs/design/feature/vae_parallel.md
+69 -49
View File
@@ -22,59 +22,60 @@ We use **Qwen-Image** as the reference implementation for decode parallel, and *
**VAE Patch Parallelism** is an acceleration technique for both **encoding** and **decoding**. Instead of processing the entire tensor at once, the tensor is:
+ Split into multiple spatial tiles
- Split into multiple spatial tiles
+ Distributed across multiple ranks
- Distributed across multiple ranks
+ Encoded/Decoded in parallel
- Encoded/Decoded in parallel
+ Merged to reconstruct the final output
- Merged to reconstruct the final output
This approach:
+ Distributes computation across multiple devices
- Distributes computation across multiple devices
+ Reduces peak memory usage per device
- Reduces peak memory usage per device
+ Accelerates encoding/decoding latency
- Accelerates encoding/decoding latency
### When to Use Encode vs Decode Parallel
| Operation | Use Case | Example |
|-----------|----------|---------|
| **Decode Parallel** | Text-to-Image, Text-to-Video | Latent → Image/Video |
| **Encode Parallel** | Image-to-Video (I2V) | Image → Latent (for conditioning) |
| Operation | Use Case | Example |
| ------------------- | ---------------------------- | --------------------------------- |
| **Decode Parallel** | Text-to-Image, Text-to-Video | Latent → Image/Video |
| **Encode Parallel** | Image-to-Video (I2V) | Image → Latent (for conditioning) |
### Architecture
We introduce **DistributedVaeExecutor** as the core component responsible for distributed VAE encoding/decoding.
The executor is model-agnostic and accepts three function parameters:
+ split – Partition the latent into tiles
- split – Partition the latent into tiles
+ exec – Decode a single tile
- exec – Decode a single tile
+ merge – Combine decoded tiles into the final output
- merge – Combine decoded tiles into the final output
#### Execution Flow
+ Call split(z) to generate a list of TileTask and a GridSpec
- Call split(z) to generate a list of TileTask and a GridSpec
+ Dispatch tasks across ranks using workload-based balancing
- Dispatch tasks across ranks using workload-based balancing
+ Each rank executes exec(task) on its assigned tiles
- Each rank executes exec(task) on its assigned tiles
+ Gather decoded tile results to rank 0
- Gather decoded tile results to rank 0
+ Rank 0 performs merge(...)
- Rank 0 performs merge(...)
+ (Optional) Broadcast final result to all ranks
- (Optional) Broadcast final result to all ranks
This design separates:
+ Distributed execution logic
- Distributed execution logic
+ Model-specific tiling and merging logic
- Model-specific tiling and merging logic
#### Why split / exec / merge is necessary?
@@ -82,23 +83,23 @@ The latent tensor cannot be arbitrarily partitioned.
During decoding:
+ Each output pixel may depend on neighboring pixels
- Each output pixel may depend on neighboring pixels
+ The receptive field is model-dependent
- The receptive field is model-dependent
Therefore:
+ Tiles must include overlap
- Tiles must include overlap
+ Merge must perform blending to avoid seams
- Merge must perform blending to avoid seams
## Step-by-Step Implementation (Decode)
### Step 1: Implement DistributedAutoencoderKLQwenImage
`QwenImagePipeline` use `AutoencoderKLQwenImage` for vae, so implement a distributed version:
```
```python
class DistributedAutoencoderKLQwenImage(AutoencoderKLQwenImage, DistributedVaeMixin):
@classmethod
def from_pretrained(cls, *args: Any, **kwargs: Any):
@@ -106,28 +107,35 @@ class DistributedAutoencoderKLQwenImage(AutoencoderKLQwenImage, DistributedVaeMi
model.init_distributed()
return model
```
**Key points**:
+ Inherit both AutoencoderKLQwenImage and DistributedVaeMixin
+ Call init_distributed() after loading weights
- Inherit both AutoencoderKLQwenImage and DistributedVaeMixin
- Call init_distributed() after loading weights
### Step 2: Implement split/exec/merge
Reuse `AutoencoderKLQwenImage.tiled_decode` logic and divide it into three stages. And we need return tiles with `GridSpec` and `TileTask`:
```
```python
class GridSpec:
split_dims: tuple[int, ...] # Tensor dimensions being split (e.g., (2, 3) for (B, C, H, W))
grid_shape: tuple[int, ...] # Tile grid layout (num_rows, num_cols)
tile_spec: dict = field(default_factory=dict) # Metadata required for merging
output_dtype: torch.dtype | None = None # Final output dtype
```
```
```python
class TileTask:
tile_id: int # task id
grid_coord: tuple[int, ...] # Tile position in grid
tensor: torch.Tensor | list[torch.Tensor] # The tile tensor
workload: int | float = 1 # Used for load balancing (e.g., tile area)
```
And tiled base split/exec/merge as follow:
```
```python
def tile_split(self, z: torch.Tensor) -> tuple[list[TileTask], GridSpec]:
# mostly copy from AutoencoderKL
_, _, num_frames, height, width = z.shape
@@ -210,11 +218,14 @@ def tile_merge(self, coord_tensor_map: dict[tuple[int, ...], torch.Tensor], grid
```
### Step 3: Override tiled_decode
We need to override tiled_decode, the main logic is:
+ check distributed is enabled
+ select split/exec/merge
+ Invoke self.distributed_executor.execute to decode
```
- check distributed is enabled
- select split/exec/merge
- Invoke self.distributed_executor.execute to decode
```python
def tiled_decode(self, z: torch.Tensor, return_dict: bool = True):
if not self.is_distributed_enabled():
return super().tiled_decode(z, return_dict=return_dict)
@@ -230,11 +241,14 @@ def tiled_decode(self, z: torch.Tensor, return_dict: bool = True):
return DecoderOutput(sample=result)
```
`broadcast_result` is set to True or False depending on the model; when enabled, the result will be used even on ranks other than 0.
### Step 4: Modify Pipeline
Change vae model from AutoencoderKLQwenImage to DistributedAutoencoderKLQwenImage
```
```diff
class YourModelPipeline(nn.Module):
def __init__(
self,
@@ -259,8 +273,8 @@ For models that require VAE encoding (e.g., Image-to-Video), you can also parall
Similar to decode, split the input tensor into tiles. Key considerations:
+ **Patchify handling**: If the model uses `patch_size`, scale tile parameters accordingly
+ **Temporal chunking**: Video VAEs may have temporal compression (e.g., 4x)
- **Patchify handling**: If the model uses `patch_size`, scale tile parameters accordingly
- **Temporal chunking**: Video VAEs may have temporal compression (e.g., 4x)
```python
def encode_tile_split(self, x: torch.Tensor) -> tuple[list[TileTask], GridSpec]:
@@ -406,23 +420,28 @@ def tiled_encode(self, x: torch.Tensor) -> torch.Tensor:
**Key differences from decode parallel:**
| Aspect | Decode Parallel | Encode Parallel |
|--------|-----------------|-----------------|
| -------- | ----------------- | ----------------- |
| `broadcast_result` | Often `False` (only rank 0 needs output) | `True` (all ranks need latents for diffusion) |
| Patchify | Applied in merge (unpatchify) | Handled by parent `_encode()` before `tiled_encode()` |
| Temporal chunking | Frame-by-frame | Chunk-based (e.g., 1 + 4n frames) |
## Testing
Verify numerical consistency between:
+ vae_patch_parallel_size = 1
+ vae_patch_parallel_size = N
Verify numerical consistency between:
- vae_patch_parallel_size = 1
- vae_patch_parallel_size = N
Example:
```python
torch.allclose(output_1, output_n, atol=1e-5)
```
Testing requirements:
+ Fix random seed
+ Use identical tiling strategy
- Fix random seed
- Use identical tiling strategy
```python
m = Omni(
@@ -434,6 +453,7 @@ m = Omni(
),
)
```
When vae_patch_parallel_size is larger than the DiT world size, it will automatically fall back to using the DiT world size instead.
## Reference Implementations
@@ -441,7 +461,7 @@ When vae_patch_parallel_size is larger than the DiT world size, it will automati
Complete examples in the codebase:
| Model | Path | Decode Parallel | Encode Parallel |
|-------|------|-----------------|-----------------|
| ------- | ------ | ----------------- | ----------------- |
| **Z-Image** | `vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl.py` | ✅ | ❌ |
| **Wan2.2** | `vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl_wan.py` | ✅ | ✅ |
| **Qwen-Image** | `vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl_qwenimage.py` | ✅ | ❌ |
@@ -456,7 +476,7 @@ The tile-parallel executor above assigns independent spatial **tiles** to ranks.
### How it differs from tile parallel
| Aspect | Tile parallel (`"tile"`) | Spatially-sharded (`"spatial_shard_height"`/`"spatial_shard_width"`) |
|--------|--------------------------|-----------------------------------------------|
| -------- | -------------------------- | ----------------------------------------------- |
| Unit of work | Independent overlapping tiles | A single global feature map sharded along H or W |
| Cross-rank communication | Gather tiles to rank 0, stitch + blend | Per-conv **halo exchange** of boundary rows/cols (P2P) |
| Output assembly | Blend overlapping tiles | All-gather shards on rank 0, trim padding (matches `broadcast_result=False`) |
@@ -481,7 +501,7 @@ serve.py (--vae-parallel-mode) / OmniEngineArgs
### Notes
- The decoder is patched **in place** the first time spatial-shard decode runs and is bound to a single split dimension for the lifetime of the VAE instance.
- Numerical correctness vs. single-GPU decode is covered by `tests/diffusion/distributed/test_wan_spatial_shard.py::test_spatial_shard_decode_matches_reference` (multi-GPU, nightly `full_model` + `distributed_cuda`).
- Numerical correctness vs. single-GPU decode is covered by `tests/diffusion/distributed/test_wan_spatial_shard.py::test_spatial_shard_decode_matches_reference` (multi-GPU, nightly `full_model` + `cards_2`).
---
docs/design/index.md
+19 -9
View File
@@ -18,10 +18,12 @@ implementation contract; it is not, by itself, a general support claim.
### Runtime and stage execution
- [Disaggregated Inference](feature/disaggregated_inference.md)
- [Host Weight Runtime](feature/host_weight_runtime.md)
- [Async Chunk](feature/async_chunk.md)
- [Async Diffusion Output](feature/async_diffusion_output.md)
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
- [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
- [Realtime AR-Diffusion Sessions](feature/realtime_ar_diffusion.md)
### Communication
@@ -53,15 +55,22 @@ implementation contract; it is not, by itself, a general support claim.
#### Attention optimization
The [Diffusion Attention Backends](../user_guide/diffusion/attention_backends.md)
guide lists all selectable backends and their platform defaults. Skip-Softmax
is the backend-related optimization with a standalone feature-design contract:
guides list selectable backends, platform defaults, installation, and tuning.
The design contracts separate selection mechanics from backend algorithms:
- [Attention Backend Selection](feature/attention_backend_selection.md)
- [Skip-Softmax](feature/skip_softmax.md)
#### CPU offloading
- [Overview and Shared Contracts](feature/offloader/README.md)
- [Model-Level Offload](feature/offloader/cpu_offload.md)
- [Layerwise Offload](feature/offloader/layerwise_offload.md)
- [Distributed Layerwise Offload](feature/offloader/distributed_layerwise_offload.md)
- [Cache-DiT](feature/cache_dit.md)
- [TeaCache](feature/teacache.md)
- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
## Infrastructure and Performance
@@ -80,14 +89,15 @@ is the backend-related optimization with a standalone feature-design contract:
- [Model Integration](module/model_integration.md)
- [Autoregressive Runtime](module/ar_runtime.md)
- Diffusion
- [Overview](module/diffusion/index.md)
- [Runtime](module/diffusion/diffusion_runtime.md)
- [Model Integration](module/diffusion/diffusion_model_integration.md)
- [Continuous Batching](module/diffusion/continuous_batching.md)
- [Parallelism](module/diffusion/parallelism.md)
- [Offloader](module/diffusion/offloader.md)
- [Overview](module/diffusion/index.md)
- [Runtime](module/diffusion/diffusion_runtime.md)
- [Model Integration](module/diffusion/diffusion_model_integration.md)
- [Continuous Batching](module/diffusion/continuous_batching.md)
- [Parallelism](module/diffusion/parallelism.md)
- [Offloader](module/diffusion/offloader.md)
- [Execution Platforms](module/execution_platforms.md)
- [Cache Management](module/cache_management.md)
- [Host Weight Runtime](module/host_weight_runtime.md)
- [Quantization](module/quantization.md)
- [Observability](module/observability.md)
- [Profiling](module/profiling.md)
docs/design/metrics.md
+76 -30
View File
@@ -6,7 +6,7 @@ This document describes how vLLM-Omni exposes Prometheus metrics for multi-stage
- Expose pipeline-level request and latency metrics that span the full multi-stage execution (orchestrator scope).
- Preserve all upstream vLLM per-engine metrics (`vllm:*`) for stages backed by an AR LLM engine, and reshape their `engine` label into `stage` + `replica` so multi-replica deployments gain per-replica visibility automatically.
- Expose per-modality SLO metrics that the upstream `vllm:*` families do not capture — audio TTFP / RTF / duration / frames / streaming continuity / silent-loss.
- Expose per-modality SLO metrics that the upstream `vllm:*` families do not capture — audio TTFP / RTF / duration / frames / streaming continuity / silent-loss, plus image and diffusion workload, queueing, execution, and memory signals.
- Expose per-replica-edge cross-stage transfer metrics so the slack between E2E latency and the sum of per-stage `gen_time` (queueing, serialization, network) becomes attributable.
- Keep the metrics collection overhead low enough that it does not regress TTFA or throughput.
@@ -20,7 +20,7 @@ vLLM's `unregister_vllm_metrics()` function strips every `prometheus_client` col
### The Problem
vLLM-Omni runs multiple engine instances (stages × replicas) within a single process, coordinated by an Orchestrator. The pipeline needs its own metrics — aggregate request counts, end-to-end latency across all stages, per-modality SLO signals, and cross-stage transfer attribution — that do not exist in upstream vLLM. All pipeline-level metrics use the `vllm:omni_` prefix to distinguish them from upstream per-engine metrics. At import time (see `vllm_omni/patch.py`) we replace `unregister_vllm_metrics()` with a scoped version that still strips upstream `vllm:*` collectors before each new `PrometheusStatLogger` registers (so multi-engine processes don't crash on duplicate registration), but preserves anything prefixed `vllm:omni_` / `vllm_omni`.
vLLM-Omni runs multiple engine instances (stages × replicas) within a single process, coordinated by an Orchestrator. The pipeline needs its own metrics — aggregate request counts, end-to-end latency across all stages, per-modality SLO signals, and cross-stage transfer attribution — that do not exist in upstream vLLM. All pipeline-level metrics use the `vllm_omni:` prefix to distinguish them from upstream per-engine metrics. At import time (see `vllm_omni/patch.py`) we replace `unregister_vllm_metrics()` with a scoped version that still strips upstream `vllm:*` collectors before each new `PrometheusStatLogger` registers (so multi-engine processes don't crash on duplicate registration), but preserves anything prefixed `vllm_omni:` / `vllm_omni`.
Upstream per-engine metrics retain the `vllm:` prefix but are now registered by `OmniPrometheusStatLogger`, a thin subclass of upstream's `PrometheusStatLogger` that reshapes the single `engine` label into a `stage` + `replica` pair (see "OmniPrometheusStatLogger wrap" below).
@@ -38,7 +38,7 @@ Upstream per-engine metrics retain the `vllm:` prefix but are now registered by
|
+--------+--------+--------+--------+--------+
| |
vllm:omni_* vllm:*
vllm_omni:* vllm:*
collectors collectors
| |
+----+--------+ +-----------+ +----------+ +-----------+
@@ -58,17 +58,30 @@ Upstream per-engine metrics retain the `vllm:` prefix but are now registered by
### Data Flow
There are four independent paths for metric collection.
There are five independent paths for metric collection.
**Path 1: Pipeline-level metrics (`vllm:omni_*`)**
**Path 1: Pipeline-level metrics (`vllm_omni:*`)**
`OmniPrometheusMetrics` registers the Gauge / Counter / Histogram collectors at import time. It is instantiated once per entrypoint, labeled with the model name. The entrypoint calls its methods as requests progress:
- `set_running(n)` / `set_waiting(n)` — updated after each request completes. The running count comes from `OmniRequestCounter`, a simple counter incremented/decremented by the Orchestrator as it tracks requests. Waiting is derived as `total - running`.
- `request_succeeded(e2e_seconds, finished_reason="stop")` — recorded when a request finishes at the final stage. `finished_reason` is extracted from `engine_outputs.outputs[0].finish_reason` (vLLM `CompletionOutput` convention) and increments `vllm:omni_requests_success_total{finished_reason}`.
- `request_failed()` — recorded by the cleanup path when a request exits without natural completion. Internally maps to `finished_reason="abort"` so a single Counter family covers both natural and aborted completion.
- `request_succeeded(e2e_seconds, finished_reason="stop")` — recorded when a request finishes at the final stage. `finished_reason` is extracted from `engine_outputs.outputs[0].finish_reason` (vLLM `CompletionOutput` convention) and increments `vllm_omni:requests_success_total{finished_reason}`.
- `request_failed()` — recorded by the cleanup path when a request exits without natural completion. Internally maps to `finished_reason="abort"` on the existing completion Counter.
- `inc_requests_failed(reason)` — increments the dedicated failure-attribution Counter once per request. Reasons are normalized to the bounded set `client_abort`, `client_disconnect`, `stage_error`, and `unknown`.
**Path 2: Audio modality metrics (`vllm:omni_audio_*`)**
**Path 2: Image and diffusion metrics**
Image and diffusion metrics use two cooperating collectors. `OmniPrometheusMetrics` owns pipeline/stage workload families, while `OmniModalityMetrics` owns per-replica diffusion breakdowns. Finished stage messages are consumed once in `OmniBase._process_single_result`, which prevents replayed terminal messages from incrementing Counters or Histograms twice.
- `stage_gen_time_s`, `stage_in_queue_s`, image workload, inference-step, and per-stage peak-memory values are emitted from `StageRequestStats` when a stage finishes. `image_pixels` and `image_count_total` are restricted to image outputs; `num_inference_steps` is restricted to diffusion stages.
- Diffusion engine timings are accumulated per request by `OrchestratorAggregator.accumulate_diffusion_metrics`. Engine-side millisecond values are converted to seconds before `observe_modality_at_finalize(...)` dispatches the per-replica Histogram observations.
- `request_queue_wait_s` is emitted once at final request completion from `pipeline_timings["queue_wait_ms"]`. A present value of `0` is a valid observation; a missing key produces no sample.
- `stage_waiting_requests` is the sum of the latest scheduler snapshots for all live replicas in a stage. AR snapshots arrive through `SchedulerStats`; diffusion snapshots ride normal diffusion output metrics. A dead replica's snapshot is removed. Because this deliberately reuses existing message paths, the gauge can retain its last value while a replica produces no output.
- `kv_wait_s` measures the AR scheduler interval from entering the KV-transfer-free wait state until extraction acknowledgement. The sample is carried to the Orchestrator through the existing engine output and labeled by connector type. Abort/error/removal paths discard incomplete timers rather than emitting partial waits.
Zero and missing values have different meanings. Valid zero-duration queue observations are recorded, while unavailable optional measurements are omitted instead of being exported as synthetic zeroes. `peak_memory_mb`, `diffusion_forward_s`, `diffusion_kv_load_s`, `vae_decode_s`, and `kv_wait_s` therefore appear only when their data source is active.
**Path 3: Audio modality metrics (`vllm_omni:audio_*`)**
`OmniModalityMetrics` registers seven audio families with `{model_name, stage, replica}` (plus an extra `threshold_ms` / `reason` label on the two extra-cardinality Counters). Three observation sites:
@@ -76,7 +89,7 @@ There are four independent paths for metric collection.
- `observe_audio_first_packet(...)` — called from the OpenAI SSE audio branch in `serving_chat.py` on the first audio packet for a request. The once-per-request guard is held by `ClientRequestState.first_audio_ts`. The `request_arrival_ts` anchor is stored in `ClientRequestState` by `async_omni.generate()`, computed at request entry.
- `observe_audio_streaming_finalize(...)` — called from `serving_chat.py` after the streaming chunk loop exhausts. It runs the per-chunk player simulation from `vllm_omni/benchmarks/audio_continuity.py` to compute the worst-case underrun and emits `audio_underrun_s` plus (when the request stayed below the threshold) `audio_continuity_ok_total{threshold_ms}`. Per-chunk PCM byte counts and arrival timestamps are recorded by the same audio branch that updates `first_audio_ts`.
**Path 3: Cross-stage transfer metrics (`vllm:omni_transfer_*`)**
**Path 4: Cross-stage transfer metrics (`vllm_omni:transfer_*`)**
`OmniTransferMetrics` registers four Histogram families with `{model_name, from_stage, from_replica, to_stage, to_replica}` labels. Each observation corresponds to one physical transfer hop (one chunk between adjacent stages), not the per-request accumulated total — so the histograms track per-transfer distribution.
@@ -87,7 +100,7 @@ The hook lives in `OrchestratorAggregator.record_transfer_tx` and `record_transf
Defensive fail-safe: if `transfer_emitter` or `replica_resolver` is missing, or the resolver returns `None` for either side, the emit is skipped silently (the underlying `TransferEdgeStats` accumulation is unaffected).
**Path 4: Per-engine metrics (`vllm:*`, stage/replica wrap)**
**Path 5: Per-engine metrics (`vllm:*`, stage/replica wrap)**
The Orchestrator instantiates `OmniPrometheusStatLogger` (a thin subclass of upstream `vllm.v1.metrics.loggers.PrometheusStatLogger`) and feeds it scheduler stats and iteration stats after processing each batch of engine outputs. This populates the standard ~37 vLLM metric families (TTFT, ITL, TPOT, KV cache usage, etc.) using the same upstream code path — but with the `engine` label reshaped into `stage` + `replica` so multi-replica deployments produce distinct series per replica. See the next section for the wrap mechanics.
@@ -97,9 +110,9 @@ The Orchestrator runs in a background thread. The API server (OmniBase) runs in
### Metric Registration and Lifecycle
All `vllm:omni_*` collectors are registered once when their owning class (`OmniPrometheusMetrics` / `OmniModalityMetrics` / `OmniTransferMetrics`) is imported. Per-`(stage, replica)` labels are bound lazily on first observation to avoid registering label sets for combinations that never produce data.
All `vllm_omni:*` collectors are registered once when their owning class (`OmniPrometheusMetrics` / `OmniModalityMetrics` / `OmniTransferMetrics`) is imported. Per-`(stage, replica)` labels are bound lazily on first observation to avoid registering label sets for combinations that never produce data.
The `prometheus_client` default registry holds all collectors. FastAPI's `/metrics` endpoint serves the default registry, so `vllm:omni_*` and the wrapped `vllm:*` metrics appear in the same scrape response alongside `http_*` and `process_*` metrics from the instrumentator and the Python client runtime.
The `prometheus_client` default registry holds all collectors. FastAPI's `/metrics` endpoint serves the default registry, so `vllm_omni:*` and the wrapped `vllm:*` metrics appear in the same scrape response alongside `http_*` and `process_*` metrics from the instrumentator and the Python client runtime.
## OmniPrometheusStatLogger Wrap
@@ -130,11 +143,11 @@ A reverse map `(stage_id, replica_id) -> flat_idx` is maintained on the Orchestr
## Gating: `--log-stats`
All metrics — both the 15 `vllm:omni_*` families and the ~65 upstream `vllm:*` wrap families — are gated by the user's `--log-stats` CLI flag (default off). The flag is plumbed from `OmniBase.__init__(log_stats=...)` through `AsyncOmniEngine` to the stage-spawn helpers and to the three Prometheus metric classes (`OmniPrometheusMetrics` / `OmniModalityMetrics` / `OmniTransferMetrics`), and is forwarded to `Orchestrator._init_metrics_state(...)`.
All metrics — both the omni-specific `vllm_omni:*` families and the upstream `vllm:*` wrap families — are gated by the user's `--log-stats` CLI flag (default off). The flag is plumbed from `OmniBase.__init__(log_stats=...)` through `AsyncOmniEngine` to the stage-spawn helpers and to the three Prometheus metric classes (`OmniPrometheusMetrics` / `OmniModalityMetrics` / `OmniTransferMetrics`), and is forwarded to `Orchestrator._init_metrics_state(...)`.
Behavior with `--log-stats=off` (default):
- The three `Omni*Metrics` classes register their module-level Gauge / Counter / Histogram families at import time (prometheus_client requires up-front registration), but each `observe / inc / set` method early-returns. The per-label child series for `vllm:omni_*` stay materialized but never have data written to them.
- The three `Omni*Metrics` classes register their module-level Gauge / Counter / Histogram families at import time (prometheus_client requires up-front registration), but each `observe / inc / set` method early-returns. The per-label child series for `vllm_omni:*` stay materialized but never have data written to them.
- `OmniPrometheusStatLogger` is not constructed in `_init_metrics_state`, so the ~65 upstream `vllm:*` wrap families are not registered in the default registry at all.
- The engine core's `Scheduler.make_stats()` also short-circuits inside upstream (`if not self.log_stats: return None`), so no `SchedulerStats` is produced per step — the per-iteration cost is bounded by the existing upstream gate.
@@ -162,10 +175,43 @@ second.
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `vllm:omni_num_requests_running` | Gauge | `model_name` | Requests currently executing across all stages |
| `vllm:omni_num_requests_waiting` | Gauge | `model_name` | Requests queued but not yet scheduled |
| `vllm:omni_requests_success_total` | Counter | `model_name`, `finished_reason` | Total requests by completion reason ({stop, length, abort, ...}); aborts cover client-disconnect / cancellation paths in addition to upstream `FinishReason.ABORT` |
| `vllm:omni_e2e_request_latency_s` | Histogram | `model_name` | Pipeline-global end-to-end latency in seconds |
| `vllm_omni:num_requests_running` | Gauge | `model_name` | Requests currently executing across all stages |
| `vllm_omni:num_requests_waiting` | Gauge | `model_name` | Requests queued but not yet scheduled |
| `vllm_omni:requests_success_total` | Counter | `model_name`, `finished_reason` | Total requests by completion reason ({stop, length, abort, ...}); aborts cover client-disconnect / cancellation paths in addition to upstream `FinishReason.ABORT` |
| `vllm_omni:e2e_request_latency_s` | Histogram | `model_name` | Pipeline-global end-to-end latency in seconds |
### Image and diffusion service-level metrics
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `vllm_omni:stage_gen_time_s` | Histogram | `model_name`, `stage`, `stage_type` | Stage submit to finished output; includes in-stage queueing |
| `vllm_omni:request_queue_wait_s` | Histogram | `model_name` | Orchestration-layer queue wait; a present zero is recorded |
| `vllm_omni:stage_waiting_requests` | Gauge | `model_name`, `stage` | Sum of the latest waiting snapshots across live replicas in the stage |
| `vllm_omni:stage_in_queue_s` | Histogram | `model_name`, `stage` | Diffusion scheduler wait from enqueue to initial execution admission |
| `vllm_omni:num_inference_steps` | Histogram | `model_name` | Denoising-step distribution for diffusion stages |
| `vllm_omni:image_count_total` | Counter | `model_name` | Cumulative number of generated image outputs |
| `vllm_omni:image_pixels` | Histogram | `model_name` | Pixel-count distribution for image outputs |
| `vllm_omni:peak_memory_mb` | Gauge | `model_name`, `stage` | Positive per-stage peak-memory samples reported by engine outputs |
| `vllm_omni:requests_failed_total` | Counter | `model_name`, `reason` | Request failures with bounded reason values (`client_abort`, `client_disconnect`, `stage_error`, `unknown`) |
| `vllm_omni:kv_wait_s` | Histogram | `model_name`, `connector_type` | Completed AR KV-transfer wait intervals, grouped by connector backend |
### Diffusion execution breakdown
Labels: `{model_name, stage, replica}`.
| Metric | Type | Description |
|--------|------|-------------|
| `vllm_omni:diffusion_exec_s` | Histogram | Core diffusion-step execution time per request |
| `vllm_omni:diffusion_exec_per_step_s` | Histogram | Core diffusion execution divided by inference-step count |
| `vllm_omni:diffusion_preprocess_s` | Histogram | Diffusion input preprocessing time |
| `vllm_omni:diffusion_postprocess_s` | Histogram | Diffusion output postprocessing time |
| `vllm_omni:vae_decode_s` | Histogram | VAE decode time when the engine reports it |
| `vllm_omni:diffusion_forward_s` | Histogram | Forward-only denoise-loop time when pipeline profiling is enabled |
| `vllm_omni:diffusion_kv_load_s` | Histogram | Diffusion-side KV receive/load time when upstream KV is used |
| `vllm_omni:image_ttfp_s` | Histogram | Image-stage submit to first materialized image output |
| `vllm_omni:denoise_step_latency_s` | Histogram | Mean diffusion forward time per denoise step for image output stages |
The optional breakdown and memory families are sparse by design: when an engine does not produce the source measurement, no sample is emitted. This differs from queue timings, where an explicitly measured zero is meaningful and is retained.
### Audio (7)
@@ -173,13 +219,13 @@ Labels: `{model_name, stage, replica}` plus the listed extra label.
| Metric | Type | Extra label | Description |
|--------|------|-------------|-------------|
| `vllm:omni_audio_ttfp_s` | Histogram | — | Time from request arrival to first audio packet/frame |
| `vllm:omni_audio_duration_s` | Histogram | — | Audio content duration (`audio_frames / sample_rate`) |
| `vllm:omni_audio_rtf` | Histogram | — | Real-time factor `stage_gen_time_s / audio_duration_s` (SLO `< 1`); uses `RTF_BUCKETS` |
| `vllm:omni_audio_frames_total` | Counter | — | Cumulative audio frames generated |
| `vllm:omni_audio_underrun_s` | Histogram | — | Per-request worst-case player deficit; `> 0` indicates listener heard silent gaps |
| `vllm:omni_audio_continuity_ok_total` | Counter | `threshold_ms` | Incremented when the request's worst underrun stayed below `threshold_ms` |
| `vllm:omni_audio_skipped_requests_total` | Counter | `reason` | Silent-loss counter — code2wav rejected malformed codec input and returned `200 OK` with empty audio |
| `vllm_omni:audio_ttfp_s` | Histogram | — | Time from request arrival to first audio packet/frame |
| `vllm_omni:audio_duration_s` | Histogram | — | Audio content duration (`audio_frames / sample_rate`) |
| `vllm_omni:audio_rtf` | Histogram | — | Real-time factor `stage_gen_time_s / audio_duration_s` (SLO `< 1`); uses `RTF_BUCKETS` |
| `vllm_omni:audio_frames_total` | Counter | — | Cumulative audio frames generated |
| `vllm_omni:audio_underrun_s` | Histogram | — | Per-request worst-case player deficit; `> 0` indicates listener heard silent gaps |
| `vllm_omni:audio_continuity_ok_total` | Counter | `threshold_ms` | Incremented when the request's worst underrun stayed below `threshold_ms` |
| `vllm_omni:audio_skipped_requests_total` | Counter | `reason` | Silent-loss counter — code2wav rejected malformed codec input and returned `200 OK` with empty audio |
### Cross-stage transfer (4)
@@ -187,10 +233,10 @@ Labels: `{model_name, from_stage, from_replica, to_stage, to_replica}`.
| Metric | Type | Description |
|--------|------|-------------|
| `vllm:omni_transfer_size_bytes` | Histogram | Per-transfer payload size in bytes |
| `vllm:omni_transfer_tx_s` | Histogram | Sender-side time (serialize + submit to connector) |
| `vllm:omni_transfer_rx_s` | Histogram | Receiver-side time (recv + deserialize) |
| `vllm:omni_transfer_in_flight_s` | Histogram | Network in-flight time (TX done → RX recv start) |
| `vllm_omni:transfer_size_bytes` | Histogram | Per-transfer payload size in bytes |
| `vllm_omni:transfer_tx_s` | Histogram | Sender-side time (serialize + submit to connector) |
| `vllm_omni:transfer_rx_s` | Histogram | Receiver-side time (recv + deserialize) |
| `vllm_omni:transfer_in_flight_s` | Histogram | Network in-flight time (TX done → RX recv start) |
### LLM stage-level (wrapped `vllm:*`)
@@ -203,7 +249,7 @@ After the wrap, every upstream `vllm:*` family — TTFT, ITL, TPOT, e2e latency,
- `SECONDS_FAST_BUCKETS` (0.001 s – 60 s) for fine-grained cross-stage transfer and audio-underrun values that need millisecond-level resolution.
- Counters use the `_total` suffix (auto-appended by `prometheus_client`).
- Sizes use the `_bytes` suffix.
- All omni-specific families are prefixed `vllm:omni_`. The upstream `unregister_vllm_metrics()` function is monkey-patched to a scoped version that still strips upstream `vllm:*` collectors (so multi-engine init within one process does not crash on duplicate registration) but preserves anything prefixed `vllm:omni_` / `vllm_omni`.
- All omni-specific families are prefixed `vllm_omni:`. The upstream `unregister_vllm_metrics()` function is monkey-patched to a scoped version that still strips upstream `vllm:*` collectors (so multi-engine init within one process does not crash on duplicate registration) but preserves anything prefixed `vllm_omni:` / `vllm_omni`.
## Logging vs. Prometheus
docs/design/module/diffusion/offloader.md
+6 -1
View File
@@ -18,7 +18,7 @@ validation_paths:
- tests/diffusion/offloader/**
upstream_refs:
- torch.nn.Module.to
last_reviewed: 2026-07-16
last_reviewed: 2026-08-25
---
# Diffusion offloader
@@ -48,6 +48,11 @@ and correctness required by the selected execution mode.
**Rule:** Retained host and device copies MUST respect configured limits and
provide deterministic teardown.
Allocator-cache retention MUST remain local to an explicit component owner,
declare both cache and physical-free-memory bounds, release conservatively
when telemetry is unavailable, and force release on failure or memory
pressure. It MUST NOT replace unconditional executor shutdown cleanup.
## Safe-change guide
Test enabled and disabled paths, repeated execution, asynchronous transfer,
docs/design/module/engine_orchestration.md
+16
View File
@@ -37,6 +37,7 @@ validation_paths:
- tests/engine/test_async_omni_engine_input.py
- tests/engine/test_async_omni_engine_outputs.py
- tests/engine/test_async_omni_engine_abort.py
- tests/e2e/offline_inference/test_qwen3_omni.py
- tests/engine/test_async_omni_engine_stage_init.py
- tests/engine/test_orchestrator.py
- tests/engine/test_orchestrator_error_handling.py
@@ -66,6 +67,12 @@ the boundary affected by the in-flight stage client/process refactor in
[#5441](https://github.com/vllm-project/vllm-omni/pull/5441). Names and
responsibilities proposed only by that PR are not current contracts.
The orchestration loop also has an opt-in event-driven mode
(`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, default off) proposed in
[#5221](https://github.com/vllm-project/vllm-omni/pull/5221). It changes poll
cadence only: the routing, ordering, and terminal-state contracts below hold
identically on both loops.
## Ownership boundary
This document owns `AsyncOmniEngine`, `Orchestrator`, request-state creation,
@@ -107,6 +114,15 @@ Numbers become append-only after normative promotion.
Test routing, output ordering, queue correlation, cancellation, failure
propagation, shutdown ordering, and representative multi-stage execution.
AR abort of a final-stage LLM request must deliver a terminal output with
`finish_reason="abort"` and the generated prefix so collocated training can
resume; diffusion abort remains whole-sample retry. Aborting a parallel-sampling
child must drop `parent_requests` once no children remain. EngineCore control
RPCs (`sleep` / `wake_up` / `pause_scheduler` / `resume_scheduler`) must
propagate worker exceptions rather than returning `{"supported": False}`.
When a request has multiple final output stages, only the last abort message
is request-terminal. Output-processor abort state is committed only after the
physical EngineCore abort succeeds.
Changes that cross into stage runtime or public error behavior require review
from that contract's owners.
docs/design/module/entrypoints.md
+8 -1
View File
@@ -43,6 +43,7 @@ depends_on:
validation_paths:
- tests/entrypoints/test_omni_entrypoints.py
- tests/entrypoints/test_async_omni.py
- tests/entrypoints/test_async_omni_pause_sleep_routing.py
- tests/entrypoints/test_async_omni_duplex.py
- tests/entrypoints/test_serve.py
- tests/entrypoints/test_stream_finish_reason.py
@@ -119,7 +120,13 @@ promotion.
Test request validation, protocol conversion, model-adapter routing,
streaming, session identity, disconnect/cancellation, and error mapping for
each affected offline or serving entrypoint.
each affected offline or serving entrypoint. Sleep must wait for in-flight
`generate()` admission before EngineCore offload; `wake_up` does not resume
admission — callers must `resume_generation()`. Sleeping tags are tracked per
stage so `wake_up(stage_ids=[0])` does not skip a later `wake_up(stage_ids=[1])`.
Streaming input pumps take an admission slot immediately before each EngineCore
ADD or update, not while waiting for the next client chunk. Frontend abort
keeps `request_states` until `generate()` consumes the terminal output.
## Promotion gate
docs/design/module/host_weight_runtime.md
+341
View File
@@ -0,0 +1,341 @@
---
title: Host Weight Runtime
kind: module
status: draft
owners:
- "@hsliuustc0106"
primary_code_paths:
- vllm_omni/host_weight_runtime/**
related_code_paths:
- vllm_omni/diffusion/model_loader/**
- vllm_omni/diffusion/offloader/**
depends_on:
- model_integration.md
- quantization.md
validation_paths:
- tests/host_weight_runtime/**
upstream_refs:
- https://github.com/vllm-project/vllm-omni/issues/6414
last_reviewed: 2026-08-25
---
# Host Weight Runtime
The Host Weight Runtime is a loader-adjacent, transport-independent foundation
for reusing immutable, runtime-ready model weights. It separates the semantic
question—"are these exactly the weights this loader requested?"—from the
physical question—"where is the validated host representation stored?"
The runtime can save repeated checkpoint loading, transformation, and peak host
memory when several workers need the same final representation. It also creates
a second, versioned path for reconstructing a valid model, so a cache miss or
recoverable store failure normally falls back to the canonical loader.
This module does not enable host-weight caching for any model by itself. A model
integration must still provide an exact identity, a producer, and a restorer.
For feature behavior and consumer integration requirements, see the
[Host Weight Runtime feature design](../feature/host_weight_runtime.md).
## Component boundary
```mermaid
flowchart LR
LA["Model loader adapter"] -->|"exact identity + policy"| RT["HostWeightRuntime"]
RT -->|"lookup / get_or_build"| ST["HostWeightStore"]
PR["WeightProducer"] -->|"runtime-ready tensors"| ST
ST -->|"validated mmap views"| LE["HostWeightLease"]
LE --> RE["WeightRestorer"]
RE --> MO["Restored host model"]
MO --> TR["DLO or another GPU transport"]
CS["Canonical model source"] --> PR
LA -. "fallback" .-> CS
```
The contracts have deliberately narrow ownership:
- `HostWeightRuntime` applies resolution and fallback policy and emits one
terminal resolution report. Explicit post-load publication emits a separate
report and closes its store-returned lease before returning; it does not
revise the completed resolution.
The runtime does not own a model or call the canonical loader.
- `HostWeightStore` owns artifact lookup, coordinated construction,
publication, validation, quarantine, and lifecycle.
- `HostWeightLease` owns process-local tensor views, mapped-file resources, and
the shared artifact lock that stabilizes their lifetime.
- `WeightProducer` creates one declared final-layout representation. It writes
only through a store-scoped writer and cannot publish paths directly.
- `WeightRestorer` validates a lease-to-model plan without mutation. Its
one-shot `commit() -> None` is the sole model-mutating phase. Concrete
restorers remain with the model or quantization integration.
- GPU transport owns page registration, private staging, H2D scheduling, and
lease release. DLO must not parse manifests or manage store artifacts.
## Exact semantic identity
Storage lookup is exact; the store never coerces one representation into
another. `WeightArtifactIdentity` includes:
- immutable model revision and source-file fingerprint;
- component and ownership boundary;
- runtime representation, dtype, and representation metadata;
- final tensor layout and semantic parallel coordinates;
- static adaptation identity; and
- producer version, implementation fingerprint, manifest schema, and restorer
schema.
Loader integrations may persist content digests for canonical source shards in
the selected storage domain. Reuse requires the same path, inode, size,
timestamps, symlink target, and cache-record checksum. Corrupt records are
rebuilt, and coordination or cache-I/O failure falls back to hashing the
canonical source directly. This is an identity-computation optimization, not a
second source of truth or an artifact substitution rule.
The identity excludes transfer policy such as registered mmap versus private
pinned staging. Those paths move the same host representation and must not
produce duplicate artifacts.
Parallel coordinates follow semantics rather than process topology:
- TP size and rank are included when they change tensor contents or shape.
- DP rank is excluded for replicated weights.
- SP size and backend are included when they affect layout. SP rank is excluded
only when all SP ranks consume an identical representation.
- PP, EP, or future parallel modes must be added by the model adapter whenever
they change component ownership or tensor layout.
Dynamic LoRA overlays are not cached as base weights. A statically merged
adapter is a distinct artifact identity with its own fingerprint.
## Deterministic resolution
The loader first resolves the canonical source and computes the exact requested
identity. The runtime then follows this order:
```text
validated local artifact
-> exact remote artifact materialized locally (future)
-> registered local producer
-> canonical loader fallback, if policy permits
```
An already published local artifact wins because it avoids network transfer and
is already in node-accessible storage. "Wins" means the highest-priority valid
representation, not whichever backing responds first.
Configuration modes are:
- `disabled`: return the canonical-direct outcome without probing a root,
credentials, topology, or identity;
- `preferred`: recoverable store failures may return canonical fallback; and
- `required`: fail when no exact lease can be acquired.
Failure-to-action policy is centralized in `HostWeightRuntime`. A clean miss or
invalid cache artifact may fall back in preferred mode. A typed failure may do
so only when it is marked retryable; unsupported capabilities, identity
collisions, producer failures, and publication failures remain visible as
startup failures.
`coordination_timeout_seconds` bounds acquisition of lookup and build locks; it
is not a hard wall-clock deadline for in-process validation, production, or
atomic publication. Once this V1 implementation becomes the producer, the
synchronous producer and publication run to completion. A hung producer blocks
its owning process and requires external process supervision; enforceable
producer cancellation requires a future process-isolated producer contract. A
waiter timing out never cancels another process's valid build.
Remote providers are represented by protocols but are explicitly unsupported
in the first implementation. Pre-load `resolve()` invokes only
`PRE_LOAD_SAFE` producers. A loader that completed canonical materialization
may explicitly call `publish_after_load()` with a `POST_LOAD_ONLY` producer
when `allow_post_load_publish` is enabled.
`allow_local_build` controls producer use during pre-load resolution;
`allow_post_load_publish` independently controls the explicit post-load entry
point. This permits lookup-only warm starts that populate a missing artifact
only after the canonical loader has established the current model.
Post-load publication is synchronous in V1, but its outcome is independent of
the already completed resolution. A store failure is returned in a separate
publication report and cannot turn a valid canonical model into a failed
resolution. On success, the runtime closes the validated store lease before
returning the report. Post-load publication only warms future startups; it does
not restore, rebind, or otherwise mutate the canonically loaded model serving
the current startup.
## Restoration transaction boundary
`WeightRestorer.plan_restore()` is validation-only: it must not modify the model
or lease. The returned plan may be committed exactly once, `commit()` returns
`None`, and commit is the sole phase allowed to mutate the model. Registration,
staging, and GPU transfer remain transport responsibilities rather than restore
side effects.
If planning fails, preferred-mode canonical fallback may reuse the untouched
model. If commit begins and then fails, the model is considered partially
hydrated and must be discarded; canonical fallback must construct a fresh model
instance instead of reusing it.
## Filesystem domain and artifact lifecycle
A storage domain describes physical locality and policy; it is not part of
semantic identity. The first implementation supports node- or NUMA-scoped
roots on verified local disk filesystems or tmpfs. The detected kernel
filesystem type is preserved in `domain.json` and inspection output. Known
remote filesystems and unknown filesystem types are rejected rather than being
silently treated as local disk, because cross-node page cache and advisory-lock
semantics do not satisfy this backend's contract. A NUMA root alone does not
guarantee page placement; CPU pinning, local first-touch/prefault, and residency
evidence belong to a later topology-aware integration.
```text
<domain>/
├── domain.json
├── domain-policy.json
├── locks/
│ ├── <identity>.build.lock
│ └── <identity>.artifact.lock
├── artifacts/<identity>/
│ ├── READY.json
│ ├── manifest.json
│ └── *.safetensors
├── tmp/
├── quarantine/
└── deny/
```
Lock files live outside replaceable artifact directories. Every cooperative
mutation follows one lock order: build lock, then exclusive artifact lock.
Lookup acquires a shared artifact lock, validates `READY` and the manifest,
opens payloads with non-following descriptor-based access, creates mmap-backed
views, and transfers that shared lock to the lease.
Lease acquisition rechecks the producer and restorer schemas against the exact
identity. Malformed safetensors payloads are reported as typed invalid
artifacts and denied rather than escaping the resolution policy as raw parser
exceptions.
Each `FileLock` records its creator PID. A forked child closes an inherited lock
descriptor without issuing `LOCK_UN`, because `flock` state belongs to the
shared open-file description and unlocking in the child would also release the
parent's active lease lock.
Publication follows this lifecycle:
```text
ABSENT -> invisible temporary build -> fully hashed and fsynced
-> atomic rename -> READY
READY -> denied -> quarantined -> explicit removal
```
`READY.json` is written last in the temporary directory. Readers therefore see
either a complete validated artifact or a miss, never an in-progress build.
Failed process-local builds are left invisible and a later build owner removes
their stale temporary directories. Explicit cleanup is nonblocking and reports
an active builder or lease instead of unlinking live mappings.
## Validation and capacity
Every publication computes full file and tensor hashes. Local lookup supports:
- `manifest_and_metadata`, which validates identity, immutable publication
metadata, file descriptors, sizes, tensor keys, shapes, dtypes, and strides;
and
- `full_checksum`, which additionally reads and hashes every payload file.
An ordered producer may compute file and tensor digests while payload bytes are
written and overlap each closed payload's `fsync` with later producer work. An
unordered producer retains parallel readback hashing. In both cases manifest
and `READY` publication wait for every payload digest and `fsync`; the atomic
publication and validation contract is unchanged.
Filesystem verity is modeled but unsupported. Invalid artifacts receive an
external deny marker before their shared lookup lock is released; a build owner
then quarantines the immutable directory under the exclusive lock.
Lookup checks the marker both before validation and after constructing the
lease so a weaker overlapping lookup cannot return a lease after a stronger
validation has denied the artifact.
If publishing the external deny marker fails, lookup returns a retryable
storage failure and marks the artifact directory with a persistent invalidation
mode before releasing the shared lock. Cooperating readers check that mode both
while opening and immediately before returning a lease. Lookup then makes a
bounded attempt to take the exclusive artifact lock and quarantine the entry;
the quarantine transition removes the internal mode marker. If a cooperating
builder replaces the marked inode while the locks are exchanged, containment
preserves that new publication instead of quarantining it. If neither the
external marker nor the directory-mode marker can be persisted, cross-process
exclusion cannot be guaranteed on an unavailable filesystem and the domain
must remain failed rather than treating the artifact as an ordinary invalid
cache entry.
New publications carry the internal invalidation mode through the staging
rename. Hardening clears it and is the logical authority transition for a
process crash; the temporary- and artifact-parent syncs then make the rename
durable, with an artifact-inode sync first making the hardened mode durable. A
process that dies before hardening leaves a persistently denied
entry, while one that dies after successful hardening leaves an authoritative
ready entry. Any synchronous failure after rename is marked invalid and
contained in quarantine before `PUBLICATION_FAILED` is returned, so that
nonretryable result cannot later resolve as a ready hit. If the filesystem
refuses both the inode marker and the quarantine move, the store instead returns
a retryable storage failure with `outcome_uncertain` details; as with deny-marker
double failure, cross-process exclusion cannot be guaranteed until storage
recovers. Once the artifact-parent sync succeeds, a later artifact- or
build-lock release error is logged and lookup resolves the committed entry
instead of reporting a contradictory publication failure. The same authority
rule applies when a competing publisher already installed byte-identical
semantic content; staging-cleanup errors are retried without downgrading that
validated ready entry to publication failure.
Before widening a hardened artifact for a lifecycle move, the store durably
sets the same invalidation mode. A process crash before rename therefore leaves
the writable source denied. After rename, the store re-hardens the destination
but retains the marker until both parent-directory syncs make the move durable;
only then does it durably clear the marker. An interrupted move therefore leaves
a marked transition that the next exact-key lifecycle operation re-hardens.
Explicit cleanup uses the locked move and retries exact-key
`.cleanup.*` tombstones left by an interrupted removal; a later build performs
the same tombstone reconciliation before producing replacement content.
An operational failure while inspecting a noncooperative competing publication
is a retryable storage failure. The competing entry remains authoritative and
is not mislabeled as corrupt or quarantined merely because its manifest could
not be read transiently.
`inspect_domain()` and `inspect_artifact()` return typed, read-only snapshots.
Inspection validation does not create a deny marker. Reported lock activity is
point-in-time evidence only; kernel locks are not a persistent owner registry.
Capacity policy covers all store-owned bytes, including ready, temporary, and
quarantined data. The local writer preflights `max_artifact_bytes`,
`max_store_bytes`, and `min_free_bytes`, and preallocates payloads where the
filesystem supports it. `ENOSPC` remains a normal typed store failure because
concurrent preflight is inherently racy. Automatic eviction and strict
concurrent reservations are not implemented.
For tmpfs, artifact bytes are host-memory consumption and may consume swap;
they must not be reported operationally as ordinary disk capacity. The store
owns files and reports capacity. The kernel controls page-cache residency, and
transport controls registration or page locking.
## First implementation scope
The foundation implements a CPU local filesystem store and
`SINGLE_PROCESS` production for one exact artifact. It intentionally adds no:
- public CLI or DLO configuration;
- BF16, FP8, or model-specific producer/restorer;
- change to DLO AllGather or no-AllGather behavior;
- CUDA registration, pinned staging, or H2D scheduling;
- remote provider, group-cohort producer, artifact bundle, eviction, or
cross-node coordination.
Consumer PRs must prove ordinary-loader parity, warm-hit work avoidance,
parallel-layout correctness, clean teardown, shared-backing evidence, and
representation-specific startup and memory behavior. Generic foundation tests
remain CPU-only and make no production performance claim.
The complete rationale and staged rollout are tracked in
[RFC #6414](https://github.com/vllm-project/vllm-omni/issues/6414).
docs/design/module/model_integration.md
+1 -1
View File
@@ -42,7 +42,7 @@ and execution behavior to stable runtime contracts.
### MODEL-INV-001: Registration is explicit
**Rule:** A model integration MUST declare how its model class, loader, input
processor, and stage configuration are selected.
processor, and pipeline configuration are selected.
### MODEL-INV-002: Model code does not route stages
docs/design/module/vllm_omni_config.md
+61 -7
View File
@@ -10,16 +10,15 @@ owners:
primary_code_paths:
- vllm_omni/config/**
- vllm_omni/deploy/**
- vllm_omni/model_executor/stage_configs/**
related_code_paths:
- vllm_omni/platforms/*/stage_configs/**
- vllm_omni/model_executor/models/*/pipeline.py
related_code_paths: []
depends_on: []
validation_paths:
- tests/config/**
upstream_refs:
- vllm.config
last_reviewed: 2026-08-07
last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
last_reviewed: 2026-08-11
last_verified_commit: e356708da4cd39992405686dacfc32b18fcccc7f
---
# vLLM-Omni configuration
@@ -39,11 +38,66 @@ The PR review proposal to capture environment-derived values in configuration
objects at initialization is an open design question, not a current
invariant.
## Current structured-stage reuse behavior
The current RFC #4021 implementation uses upstream vLLM config classes as the
schema source for AR and generation stages:
- `OmniStageLoadConfig` inherits `vllm.config.LoadConfig`.
- `OmniStageCacheConfig` inherits `vllm.config.CacheConfig`.
- `OmniStageSchedulerConfig` inherits `vllm.config.SchedulerConfig`.
- `OmniStageParallelConfig` inherits `vllm.config.ParallelConfig`.
This inheritance preserves the existing load, cache, scheduler, and parallel
concern boundaries while reducing duplicate downstream declarations. It does
not, by itself, make every inherited field an effective vLLM-Omni runtime
option. Construction and runtime consumption are separate surfaces:
| Surface | Current behavior |
| --- | --- |
| Structured schema | Upstream dataclass fields are inherited. Their defaults, default factories, and applicable Pydantic validation can participate when the structured object is constructed. |
| Omni input ownership | Pipeline/deploy/CLI construction continues to accept only fields with an existing structured owner. Direct construction of an inherited sub-config can expose a wider upstream schema. |
| Engine projection | For AR and generation stages, reusable fields are discovered by intersecting each upstream config dataclass with upstream `EngineArgs`. Only constructor-explicit values are emitted; inherited defaults remain deferred to terminal materialization. |
| Terminal materialization | The engine-owning process constructs the final upstream `VllmConfig` and performs model-, platform-, rank-, port-, and backend-dependent initialization. |
The generated projection maps include known upstream naming differences such
as `cache_dtype` to `kv_cache_dtype`, `policy` to `scheduling_policy`, and
`data_parallel_master_ip` to `data_parallel_address`. A field added to both an
upstream concern config and `EngineArgs` therefore enters the AR/generation
projection without requiring a second downstream allowlist. Explicit inherited
fields that have no projection raise an error rather than being silently
dropped.
Ownership exclusions remain deliberate. Stage topology owns `scheduler_cls`,
cache owns `disable_hybrid_kv_cache_manager`, and Omni runtime owns
`distributed_executor_backend` and `worker_cls`; vLLM's private API-process
fields are terminal internals. These exclusions prevent one input from being
constructed or projected through two config concerns. Effective-engine-argument
tests cover both the dynamic mapping and this exclusion boundary.
`CompilationConfig` and `ProfilerConfig` are direct structured-stage fields:
`stage.compilation_config` and `stage.profiler_config`. They accept mapping/YAML
inputs and complete upstream value objects. Mapping inputs are validated and
materialized as their concrete upstream types when the structured config is
constructed, so downstream consumers see one resolved representation.
Prebuilt upstream objects retain their type across the structured projection
boundary.
Quantization remains on its existing Omni-owned transport contract. This
reuse step does not adopt upstream `QuantizationConfigArgs`, because the
current engine-specific quantization split needs to be resolved separately.
Diffusion keeps its existing engine-field ownership and projection surface.
Shared Python inheritance must not be treated as evidence that an LLM-only
parallel input is effective for a diffusion stage.
## Safe-change guide while deferred
Changes should trace every affected producer to its runtime consumer and test
the structured, legacy, CLI, and deployment paths that are actually supported.
Do not infer a stable contract from this placeholder page.
the structured pipeline, CLI, and deployment paths that are actually supported.
For inherited vLLM fields, tests should separately cover constructor behavior
and effective `EngineArgs` projection. Do not infer a stable contract from
this placeholder page.
## Promotion gate
docs/design/qwen3_omni_tts_performance_optimization.md
+44 -56
View File
@@ -111,7 +111,7 @@ This post walks through each optimization in the same order they are typically e
For both Qwen3-Omni and Qwen3-TTS, batching is a pipeline-level optimization:
- Requests are grouped per stage using `runtime.max_batch_size`
- Requests are grouped per stage using `stages[].max_num_seqs`
- Each stage executes batch inference with its own scheduler/worker
- Stage outputs are routed to downstream stages with per-request mapping preserved
@@ -171,7 +171,7 @@ At concurrency 10, batching alone brings Qwen3-TTS RTF from 2.19 (slower than re
In decode-heavy serving, repeatedly launching many small kernels from CPU can become a visible overhead. CUDA Graph reduces this overhead by capturing and replaying stable execution graphs.
In stage configs, this is represented by `enforce_eager: false` for stages where graph capture is desired (Thinker/Talker), while Code2Wav keeps eager mode depending on stage behavior.
In deploy configs, this is represented by `enforce_eager: false` for stages where graph capture is desired (Thinker/Talker), while Code2Wav keeps eager mode depending on stage behavior.
### CUDA Graph results on top of batching
@@ -320,7 +320,7 @@ In the async chunk pipeline, the standard `codec_chunk_frames` is 25 (each chunk
**Dynamic initial chunk sizing (default behavior):**
Rather than using a fixed initial chunk size, vLLM-Omni dynamically selects it based on current server load. The initial chunk size is chosen from power-of-2 steps [2, 4, 8, 16] based on load factor (`active_requests / max_batch_size`):
Rather than using a fixed initial chunk size, vLLM-Omni dynamically selects it based on current server load. The initial chunk size is chosen from power-of-2 steps [2, 4, 8, 16] based on load factor (`active_requests / max_num_seqs`):
| Server load | Initial chunk frames | Rationale |
| --- | --- | --- |
@@ -349,16 +349,15 @@ This overrides the dynamic calculation for that request.
**Config (server-side):**
```yaml
runtime:
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
extra:
codec_streaming: true
codec_chunk_frames: 25 # standard chunk size (~2s of audio)
codec_left_context_frames: 25
# initial chunk is computed dynamically by default
# set initial_codec_chunk_frames: 2 to force a fixed value
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
extra:
codec_streaming: true
codec_chunk_frames: 25 # standard chunk size (~2s of audio)
codec_left_context_frames: 25
# initial chunk is computed dynamically by default
# set initial_codec_chunk_frames: 2 to force a fixed value
```
The 64 ms TTFP result reported above for Qwen3-TTS at concurrency 1 uses the dynamic initial chunk, which picks `initial_codec_chunk_frames=2` at low load. At higher concurrency the dynamic sizing increases the initial chunk to maintain decode efficiency.
@@ -408,7 +407,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
Notes:
- `runtime.max_batch_size` controls stage-level batching.
- `stages[].max_num_seqs` controls stage-level batching.
- Thinker/Talker commonly use `enforce_eager: false` for CUDA Graph paths.
- Code2Wav often remains eager (`enforce_eager: true`) depending on runtime behavior.
- Qwen3-Omni defaults `VLLM_USE_FLASHINFER_MOE_FP16=0`. The Triton has been more stable & faster
@@ -426,33 +425,26 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
```yaml
async_chunk: true
stage_args:
stages:
- stage_id: 0 # thinker
runtime:
max_batch_size: 64
engine_args:
enforce_eager: false
max_num_batched_tokens: 32768
custom_process_next_stage_input_func: >-
vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_async_chunk
max_num_seqs: 64
enforce_eager: false
max_num_batched_tokens: 32768
- stage_id: 1 # talker
runtime:
max_batch_size: 64
engine_args:
enforce_eager: false
max_num_batched_tokens: 32768
custom_process_next_stage_input_func: >-
vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_async_chunk
max_num_seqs: 64
enforce_eager: false
max_num_batched_tokens: 32768
- stage_id: 2 # code2wav
runtime:
max_batch_size: 64
engine_args:
enforce_eager: true
max_num_batched_tokens: 51200
max_num_seqs: 64
enforce_eager: true
max_num_batched_tokens: 51200
```
The async handoff processors are fixed by the registered Qwen3-Omni
`PipelineConfig`; deploy YAML only selects `async_chunk` and runtime sizing.
#### Reproduce Qwen3-Omni benchmarks
```bash
@@ -483,7 +475,7 @@ vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
The default config (`qwen3_tts.yaml`) enables the full optimization stack:
- Batching with `max_batch_size: 10` on the Talker stage
- Batching with `max_num_seqs: 10` on the Talker stage
- CUDA Graph on the Talker (`enforce_eager: false`)
- Async chunk with streaming transport
@@ -500,33 +492,29 @@ vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
```yaml
async_chunk: true
stage_args:
stages:
- stage_id: 0 # Talker (AR decoder)
runtime:
max_batch_size: 10
engine_args:
enforce_eager: false
max_num_batched_tokens: 512
custom_process_next_stage_input_func: >-
vllm_omni.model_executor.stage_input_processors.qwen3_tts.talker2code2wav_async_chunk
max_num_seqs: 10
enforce_eager: false
max_num_batched_tokens: 512
- stage_id: 1 # Code2Wav (vocoder)
runtime:
max_batch_size: 1
engine_args:
enforce_eager: true
max_num_batched_tokens: 8192
max_num_seqs: 1
enforce_eager: true
max_num_batched_tokens: 8192
runtime:
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
extra:
codec_streaming: true
codec_chunk_frames: 25
codec_left_context_frames: 25
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
extra:
codec_streaming: true
codec_chunk_frames: 25
codec_left_context_frames: 25
```
The async Talker-to-Code2Wav processor is defined by the registered Qwen3-TTS
pipeline rather than the deploy YAML.
#### Reproduce Qwen3-TTS benchmarks
```bash
docs/features/README.md
+11 -5
View File
@@ -20,13 +20,19 @@ an implementation contract has a user-facing workflow.
| Choose serial, batched, step-wise, or streaming diffusion execution | [Execution Modes and Streaming](../user_guide/diffusion/execution_modes.md) | [Diffusion Continuous Batching](../design/feature/diffusion_continuous_batching.md), [Async Diffusion Output](../design/feature/async_diffusion_output.md) |
| Reclaim stage memory without restarting the server | [Sleep Mode](sleep_mode.md) | Runtime lifecycle behavior is documented in the user guide |
The [Prefill-Decode Disaggregation design reference](pd_disaggregation.md)
describes the experimental Qwen3-Omni runtime topology and its current
integration boundaries. It is not yet a supported launch recipe.
Some runtime designs are deliberately not promoted as standalone User Guide
features yet:
- [Disaggregated Inference](../design/feature/disaggregated_inference.md) and
- Generic, OmniConnector-based
[Disaggregated Inference](../design/feature/disaggregated_inference.md) is not
yet a standalone User Guide feature. Its
[OmniConnector implementations](../design/index.md#communication) describe
topology and transport contracts. Practical deployment configuration remains
under [Pipeline and deploy configurations](../configuration/stage_configs.md).
transport contracts, while practical connector configuration remains under
[Pipeline and deploy configurations](../configuration/stage_configs.md).
- [Async Chunk](../design/feature/async_chunk.md) and
[Async Omni Output Materialization](../design/feature/omni_async_output_materialization.md)
are model- and pipeline-dependent. Use the selected model's deploy
@@ -48,10 +54,10 @@ defines the shared configuration and backend extension points.
| Goal | User guide | Related design contract |
| --- | --- | --- |
| Compare acceleration methods and supported combinations | [Overview](../user_guide/diffusion_features.md), [Feature Compatibility](../user_guide/feature_compatibility.md) | [Diffusion acceleration designs](../design/index.md#diffusion-acceleration) |
| Move weights between host and device memory | [CPU Offloading](../user_guide/diffusion/cpu_offload.md) | [Distributed Layerwise Offload](../design/feature/distributed_layerwise_offload.md) |
| Move weights between host and device memory | [CPU Offloading](../user_guide/diffusion/cpu_offload.md) | [CPU Offloading](../design/feature/offloader/README.md) |
| Reuse denoising computation | [Cache Acceleration](../user_guide/diffusion/cache_acceleration/cache_dit.md) | [Cache-DiT](../design/feature/cache_dit.md), [TeaCache](../design/feature/teacache.md) |
| Distribute diffusion work across devices | [Parallelism](../user_guide/diffusion/parallelism/overview.md) | [Parallelism designs](../design/index.md#parallelism) |
| Select dense, sparse, or quantized attention paths | [Attention Backends](../user_guide/diffusion/attention_backends.md) | [Skip-Softmax](../design/feature/skip_softmax.md) |
| Select dense, sparse, or quantized attention paths | [Attention Backends](../user_guide/diffusion/attention_backends.md) | [Attention Backend Selection](../design/feature/attention_backend_selection.md) |
| Compile repeated diffusion regions | [Regional Compilation](../user_guide/diffusion/regional_compilation.md) | User-facing optimization guide |
| Add generated video frames | [Frame Interpolation](../user_guide/diffusion/frame_interpolation.md) | User-facing extension guide |
| Reduce diffusion model startup time | [Startup and Loading](../user_guide/diffusion/startup_and_loading.md) | User-facing loading guide |
docs/features/custom_pipeline.md
+1 -1
View File
@@ -101,7 +101,7 @@ outputs = omni.generate(
)
# Access custom trajectory data
output = outputs[0].request_output
output = outputs[0]
print(f"Trajectory timesteps shape: {output.metrics['trajectory_timesteps'].shape}")
print(f"Trajectory latents shape: {output.latents.shape}")
```
docs/features/pd_disaggregation.md
+140
View File
@@ -0,0 +1,140 @@
# Prefill-Decode (PD) Disaggregation (experimental)
!!! warning "Experimental design reference"
PD disaggregation is only partially integrated on the current `main` branch.
The runtime contains PD detection and routing scaffolding, but there is no
supported deploy overlay or end-to-end validated launch recipe yet. Treat this
page as a description of the current design, not as production guidance.
Prefill-decode disaggregation splits the Qwen3-Omni Thinker into two logical
stages: a prefill worker that processes the prompt and produces KV cache, and a
decode worker that imports that cache and generates tokens. The Talker and
Code2Wav stages remain downstream; PD does not disaggregate the entire
Qwen3-Omni pipeline.
## Current design
### Runtime topology
```mermaid
flowchart LR
client["Client request"] --> orchestrator["Omni Orchestrator"]
orchestrator -->|"original prompt<br/>max_tokens = 1"| prefill["Thinker prefill<br/>KV producer"]
prefill -->|"KV blocks"| connector[("vLLM KV connector")]
prefill -->|"KV routing metadata"| orchestrator
orchestrator -->|"original prompt<br/>remote KV parameters"| decode["Thinker decode<br/>KV consumer"]
connector -->|"remote KV load"| decode
decode -->|"generated text"| client
```
The orchestrator owns the request lifecycle, while the vLLM KV connector owns
the bulk KV transfer. The original prompt is submitted to both Thinker stages:
the decode worker still needs the request tokens and multimodal features to
construct its request, while the imported KV cache prevents it from repeating
the completed prefill computation.
For the full Qwen3-Omni pipeline, splitting the Thinker adds one logical stage:
| Stage | Role | Output or handoff |
| --- | --- | --- |
| 0 | Thinker prefill | Saves prompt KV and prompt-side multimodal output |
| 1 | Thinker decode | Loads remote KV, generates text, and produces decode-side conditioning |
| 2 | Talker | Converts combined Thinker conditioning into codec codes |
| 3 | Code2Wav | Converts codec codes into audio |
Text remains a final output of the decode stage, and audio remains a final
output of Code2Wav.
### Request lifecycle
1. At startup, PD detection looks for one stage marked `is_prefill_only` and
one downstream stage marked `is_decode_only`. The decode stage must list the
prefill stage as an input source.
2. The orchestrator clones the Thinker sampling parameters for prefill, forces
`max_tokens=1`, clears stop conditions, and sets producer-side
`kv_transfer_params`. This makes prefill finish by length after exporting KV.
3. When prefill finishes, the orchestrator captures its KV routing metadata and
Qwen multimodal output under the original request ID.
4. The orchestrator submits the original prompt to decode with consumer-side
`kv_transfer_params`. Those parameters identify the transfer, prefill engine,
bootstrap endpoint, and, when required by the connector, remote request.
5. The decode engine imports the KV cache and continues generation. The normal
Omni stage path then carries Thinker conditioning through Talker and
Code2Wav.
### Control and data paths
PD uses two different connector layers. They solve different handoffs and are
not interchangeable.
| Path | Payload | Owner |
| --- | --- | --- |
| PD control path | Request ID, original prompt, `transfer_id`, bootstrap address, engine ID, and connector-specific request metadata | Omni Orchestrator |
| PD data path | Thinker attention KV blocks | vLLM KV connector |
| Downstream model path | Thinker hidden states and embeddings, Talker codec codes, and audio tensors | vLLM-Omni stage connectors and processors |
### Detection and validation
The current runtime accepts one logical prefill/decode pair per pipeline. Each
stage may use tensor parallelism, but both sides of the pair must satisfy these
invariants:
- both stages define `kv_transfer_config`
- prefill uses `kv_producer` or `kv_both`
- decode uses `kv_consumer` or `kv_both`
- both stages use the same KV connector
- `kv_buffer_device` and `kv_buffer_size` agree when set on both stages
- both stages use the same `tensor_parallel_size`
These checks happen before request processing so an inconsistent PD topology
fails during engine initialization.
## What is implemented today
| Area | Current state |
| --- | --- |
| Pair detection and topology validation | Implemented in the shared PD helper |
| Prefill sampling preparation | Implemented for sync and async entrypoints |
| Prefill-to-decode routing | Implemented in the current Orchestrator |
| Deploy-based configuration | Not yet migrated; the supported `stages:` deploy schema cannot currently expand Qwen3-Omni into the four-stage PD topology |
| Qwen3 Thinker-to-Talker handoff | Bridge state and embedding-merge helpers exist, but the merge is not wired into the live stage processor |
| Connector compatibility | Incomplete; the Orchestrator currently requires `remote_request_id`, while the pinned Mooncake connector completes prefill without returning that metadata |
| Validation coverage | PD entrypoint tests and the PD deploy overlay in online end-to-end tests are temporarily disabled during migration |
| Performance | No non-regression or throughput/latency result is established for the current path |
Because these pieces are still converging, the previous legacy `stage_args` YAML
and `vllm serve` command have been removed. The serve CLI now accepts the
deploy-based `stages:` schema, so retaining the legacy recipe would imply a
launch path that current `main` rejects.
## Configuration contract under development
The intended deploy integration starts from the
[default Qwen3-Omni deploy config](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml)
and expands its frozen three-stage pipeline into the four stages shown above.
A complete integration must:
- split the Thinker definition without duplicating the Talker or Code2Wav model
contracts
- remap downstream stage IDs, input sources, and stage connectors
- inject producer and consumer `kv_transfer_config` values into the two Thinker
stages
- allocate separate device sets for prefill and decode
- preserve both final outputs: decode text and Code2Wav audio
- provide connector-specific routing metadata without hard-coding one connector
contract
The required number of GPUs depends on tensor-parallel sizes and whether Talker
and Code2Wav share a device. A basic TP=1 layout uses one device for prefill, one
for decode, and one for the two downstream audio stages.
## Implementation map
- PD detection, validation, and sampling-parameter preparation:
[`PDDisaggregationMixin`](gh-file:vllm_omni/entrypoints/pd_utils.py)
- startup detection and connector metadata:
[`AsyncOmniEngine`](gh-file:vllm_omni/engine/async_omni_engine.py)
- request-scoped prefill-to-decode routing:
[`Orchestrator`](gh-file:vllm_omni/engine/orchestrator.py)
- Qwen3 downstream conditioning helpers:
[Qwen3-Omni stage input processor](gh-file:vllm_omni/model_executor/stage_input_processors/qwen3_omni.py)
docs/getting_started/installation/README.md
+1 -1
View File
@@ -1,7 +1,7 @@
# Installation
!!! important
vLLM-Omni is released against the matching upstream vLLM major/minor version. vLLM-Omni 0.26.x requires vLLM 0.26.x; the stable v0.26.0 instructions pin vLLM 0.26.0.
vLLM-Omni is released against the matching upstream vLLM major/minor version. vLLM-Omni 0.28.x requires vLLM 0.28.x; the stable v0.28.0 instructions pin vLLM 0.28.0.
vLLM-Omni supports the following hardware platforms:
docs/getting_started/installation/gpu.md
+1 -1
View File
@@ -34,7 +34,7 @@ vLLM-Omni is a Python library that supports the following GPU variants. The libr
### Pre-built wheels
Note: Pre-built wheels are currently available for vLLM-Omni 0.11.0rc1, 0.12.0rc1, 0.14.0rc1, 0.14.0, 0.16.0, 0.18.0, 0.20.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, and 0.26.0. If you need a newer unreleased revision, please [build from source](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/gpu/#build-wheel-from-source).
Note: Pre-built wheels are currently available for vLLM-Omni 0.11.0rc1, 0.12.0rc1, 0.14.0rc1, 0.14.0, 0.16.0, 0.18.0, 0.20.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, and 0.26.0; the 0.28.0 wheels are published together with the v0.28.0 release this branch produces. If you need a newer unreleased revision, please [build from source](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/gpu/#build-wheel-from-source).
=== "NVIDIA CUDA"
docs/getting_started/installation/gpu/cuda.inc.md
+9 -9
View File
@@ -20,7 +20,7 @@ Therefore, it is recommended to install vLLM and vLLM-Omni with a **fresh new**
vLLM-Omni is built based on vLLM. Please install it with command below.
```bash
uv pip install vllm==0.26.0 --torch-backend=auto
uv pip install vllm==0.28.0 --torch-backend=auto
```
#### Installation of vLLM-Omni
@@ -39,13 +39,13 @@ uv pip install 'vllm-omni[demo]'
# --8<-- [start:build-wheel-from-source]
#### Installation of vLLM
If you do not need to modify source code of vLLM, you can directly install the stable 0.26.0 release version of the library
If you do not need to modify source code of vLLM, you can directly install the stable 0.28.0 release version of the library
```bash
uv pip install vllm==0.26.0 --torch-backend=auto
uv pip install vllm==0.28.0 --torch-backend=auto
```
The 0.26.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.
The 0.28.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.
#### Installation of vLLM-Omni
Since vllm-omni is rapidly evolving, it's recommended to install it from source
@@ -66,12 +66,12 @@ If you want to check, modify or debug with source code of vLLM, install the libr
```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.26.0
git checkout v0.28.0
```
Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set `VLLM_PRECOMPILED_WHEEL_LOCATION` as your local absolute path of whl file.
```bash
#For CUDA 13.0 (the default for v0.26.0; the wheel filename has no `+cu130` suffix)
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.26.0/vllm-0.26.0-cp38-abi3-manylinux_2_28_x86_64.whl
#For CUDA 13.0 (the default for v0.28.0; the wheel filename has no `+cu130` suffix)
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.28.0/vllm-0.28.0-cp38-abi3-manylinux_2_28_x86_64.whl
```
Install vllm with command below (If you have no existing PyTorch).
```bash
@@ -102,7 +102,7 @@ docker run --runtime nvidia --gpus 2 \
--env "HF_TOKEN=$HF_TOKEN" \
-p 8091:8091 \
--ipc=host \
vllm/vllm-omni:v0.26.0 \
vllm/vllm-omni:v0.28.0 \
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
```
@@ -124,7 +124,7 @@ If you want to specify the base vLLM version:
```bash
DOCKER_BUILDKIT=1 docker build \
-f docker/Dockerfile.cuda \
--build-arg BASE_IMAGE=vllm/vllm-openai:v0.26.0 \
--build-arg BASE_IMAGE=vllm/vllm-openai:v0.28.0 \
-t vllm-omni-cuda .
```
docs/getting_started/installation/gpu/rocm.inc.md
+7 -7
View File
@@ -13,7 +13,7 @@ vLLM-Omni current recommends the steps in under setup through Docker Images.
vLLM-Omni is built based on vLLM. Please install it with command below.
```bash
uv pip install vllm==0.26.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.26.0/rocm723
uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
```
#### Installation of vLLM-Omni
@@ -34,13 +34,13 @@ uv pip install onnxruntime-rocm
# --8<-- [start:build-wheel-from-source]
#### Installation of vLLM
If you do not need to modify source code of vLLM, you can directly install the stable 0.26.0 release version of the library
If you do not need to modify source code of vLLM, you can directly install the stable 0.28.0 release version of the library
```bash
uv pip install vllm==0.26.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.26.0/rocm723
uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
```
The pre-built 0.26.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
The pre-built 0.28.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
#### Installation of vLLM-Omni
Since vllm-omni is rapidly evolving, it's recommended to install it from source
@@ -58,7 +58,7 @@ If you want to check, modify or debug with source code of vLLM, install the libr
```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.26.0
git checkout v0.28.0
python3 -m pip install -r requirements/rocm.txt
python3 setup.py develop
```
@@ -130,7 +130,7 @@ docker run --rm \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=$HF_TOKEN" \
-p 8091:8091 \
vllm/vllm-omni-rocm:v0.26.0 \
vllm/vllm-omni-rocm:v0.28.0 \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
```
@@ -149,7 +149,7 @@ docker run --rm -it \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=$HF_TOKEN" \
--entrypoint bash \
vllm/vllm-omni-rocm:v0.26.0
vllm/vllm-omni-rocm:v0.28.0
```
# --8<-- [end:pre-built-images]
docs/getting_started/installation/gpu/xpu.inc.md
+13
View File
@@ -23,6 +23,19 @@ vLLM-Omni currently recommends using the Docker image setup steps below.
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.xpu -t vllm-omni-xpu --shm-size=4g .
```
This layers vLLM-Omni on top of the published `vllm/vllm-openai-xpu:<VLLM_VERSION>`
base image, which Docker pulls automatically. To target a different vLLM release,
pass `--build-arg VLLM_VERSION=<tag>`. If that tag has not been published yet, build
the base from upstream's own Dockerfile first and point the build at it:
```bash
git clone --depth 1 --branch "$VLLM_VERSION" https://github.com/vllm-project/vllm /tmp/vllm
DOCKER_BUILDKIT=1 docker build -t vllm-openai-xpu:local --target vllm-openai \
-f /tmp/vllm/docker/Dockerfile.xpu /tmp/vllm
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.xpu -t vllm-omni-xpu --shm-size=4g \
--build-arg VLLM_BASE=vllm-openai-xpu:local .
```
#### Launch the docker image
##### Launch with OpenAI API Server
docs/getting_started/quickstart.md
+6 -7
View File
@@ -19,10 +19,10 @@ uv venv --python 3.12 --seed
source .venv/bin/activate
# On CUDA
uv pip install vllm==0.26.0 --torch-backend=auto
uv pip install vllm==0.28.0 --torch-backend=auto
# On ROCm
uv pip install vllm==0.26.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.26.0/rocm723
uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
@@ -34,7 +34,7 @@ For additional installation methods — please see the [installation guide](inst
!!! note
It is important to install the same major & minor version of vLLM and vLLM Omni, otherwise things may not work as expected. If the versions are misaligned, you will see a warning when you import vLLM Omni.
If you are seeing strange behavior with the `vllm` command not handling the `--omni` flag correctly, you most likely have a version mismatch with vLLM < `0.26.0` and vLLM Omni `0.26.0`, as vLLM Omni no longer hijacks the vLLM entrypoint. Updating vLLM should resolve this issue.
If you are seeing strange behavior with the `vllm` command not handling the `--omni` flag correctly, you most likely have a version mismatch with vLLM < `0.28.0` and vLLM Omni `0.28.0`, as vLLM Omni no longer hijacks the vLLM entrypoint. Updating vLLM should resolve this issue.
## Offline Inference
@@ -47,7 +47,7 @@ if __name__ == "__main__":
omni = Omni(model="Tongyi-MAI/Z-Image-Turbo")
prompt = "a cup of coffee on the table"
outputs = omni.generate(prompt)
images = outputs[0].request_output.images
images = outputs[0].images
images[0].save("coffee.png")
```
@@ -66,7 +66,7 @@ from vllm_omni.entrypoints.omni import Omni
if __name__ == "__main__":
omni = Omni(
model="Tongyi-MAI/Z-Image-Turbo",
# stage_configs_path="./stage-config.yaml", # See below
# deploy_config="./deploy-config.yaml", # Optional deploy override
)
prompts = [
"a cup of coffee on a table",
@@ -75,8 +75,7 @@ if __name__ == "__main__":
]
omni_outputs = omni.generate(prompts)
for i_prompt, prompt_output in enumerate(omni_outputs):
this_request_output = prompt_output.request_output
this_images = this_request_output.images
this_images = prompt_output.images
for i_image, image in enumerate(this_images):
image.save(f"p{i_prompt}-img{i_image}.jpg")
print("saved to", f"p{i_prompt}-img{i_image}.jpg")
docs/mkdocs/hooks/generate_examples.py
-1
View File
@@ -37,7 +37,6 @@ GENERAL_EXAMPLE_SLUGS = frozenset(
"text_to_speech",
"text_to_video",
"x_to_text",
"x_to_video_audio",
}
)
docs/models/supported_models.md
+31 -17
View File
@@ -21,43 +21,54 @@ th {
|--------------|--------|-------------------|------------|---------|-----|-----------|--------|
| `Qwen3OmniMoeForConditionalGeneration` | Qwen3-Omni | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen3-Omni.md) |
| `Qwen2_5OmniForConditionalGeneration` | Qwen2.5-Omni | `Qwen/Qwen2.5-Omni-7B`, `Qwen/Qwen2.5-Omni-3B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `MingFlashOmniForConditionalGeneration` + `MingImagePipeline` | Ming-flash-omni-2.0 (omni-speech + imagegen<sup>1</sup>) | `Jonathan1909/Ming-flash-omni-2.0` | ✅︎ | | | | — |
| `MingFlashOmniForConditionalGeneration` + `MingImagePipeline` | Ming-flash-omni-2.0 (omni-speech + imagegen<sup>1</sup>) | `Jonathan1909/Ming-flash-omni-2.0` | ✅︎ | | | | — |
| `BagelForConditionalGeneration` | BAGEL (DiT-only) | `ByteDance-Seed/BAGEL-7B-MoT` | ✅︎ | ✅︎ | | ✅︎ | — |
| `InternVLAA1Pipeline` | InternVLA-A1 | `InternRobotics/InternVLA-A1-3B` | ✅︎ | ✅︎ | | | — |
| `Gr00tN1d7Pipeline` | GR00T N1.7 | `nvidia/GR00T-N1.7-3B` | ✅︎ | | | | — |
| `HunyuanImage3ForCausalMM` | HunyuanImage3.0 (DiT-only) | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `QwenImagePipeline` | Qwen-Image | `Qwen/Qwen-Image` | ✅︎ | | | | [Published](https://recipes.vllm.ai/Qwen/Qwen-Image) |
| `QwenImagePipeline` | Qwen-Image-2512 | `Qwen/Qwen-Image-2512` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen-Image-2512.md) |
| `QwenImagePipeline` | Qwen-Image | `Qwen/Qwen-Image` | ✅︎ | | | ✅︎ | [Published](https://recipes.vllm.ai/Qwen/Qwen-Image) |
| `QwenImagePipeline` | Qwen-Image-2512 | `Qwen/Qwen-Image-2512` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen-Image-2512.md) |
| `QwenImageEditPipeline` | Qwen-Image-Edit | `Qwen/Qwen-Image-Edit` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `QwenImageEditPlusPipeline` | Qwen-Image-Edit-2509 | `Qwen/Qwen-Image-Edit-2509` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `QwenImageLayeredPipeline` | Qwen-Image-Layered | `Qwen/Qwen-Image-Layered` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `QwenImageEditPlusPipeline` | Qwen-Image-Edit-2511 | `Qwen/Qwen-Image-Edit-2511` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `GlmImagePipeline` | GLM-Image | `zai-org/GLM-Image` | ✅︎ | ✅︎ | | | — |
| `ZImagePipeline` | Z-Image | `Tongyi-MAI/Z-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `Krea2Pipeline` | Krea 2 (Raw + Turbo) | `krea/Krea-2-Raw`, `krea/Krea-2-Turbo` | ✅︎ | | | | — |
| `Krea2Pipeline` | Krea 2 (Raw + Turbo) | `krea/Krea-2-Raw`, `krea/Krea-2-Turbo` | ✅︎ | | | ✅︎ | — |
| `WanPipeline` | Wan2.1-T2V, Wan2.2-T2V, Wan2.2-TI2V | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`, `Wan-AI/Wan2.1-T2V-14B-Diffusers`, `Wan-AI/Wan2.2-T2V-A14B-Diffusers`, `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `WanImageToVideoPipeline` | Wan2.2-I2V | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `LingBotWorldCausalDMDPipeline` | LingBot-World 2.0 v1 (experimental) | `robbyant/lingbot-world-v2-14b-causal-fast-diffusers` | Experimental | | | | — |
| `Cosmos3OmniDiffusersPipeline` | Cosmos3 T2I, T2V, I2V, V2V, T2V with sound, action policy | `nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `SanaWmPipeline` | SANA-WM | `BBBBruce/SANA-WM_bidirectional-stage1-diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-WM.md) |
| `Wan22S2VPipeline` | Wan2.2-S2V | `Wan-AI/Wan2.2-S2V-14B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `Wan22VACEPipeline` | Wan2.1-VACE | `Wan-AI/Wan2.1-VACE-1.3B-diffusers`, `Wan-AI/Wan2.1-VACE-14B-diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `Wan22VACEPipeline` | Wan2.2-VACE | `Pyros13/Wan2.2-VACE-Fun-A14B-Diffusers` | ✅︎ | | | | — |
| `Wan22VACEPipeline` | Wan2.2-VACE | `Pyros13/Wan2.2-VACE-Fun-A14B-Diffusers` | ✅︎ | | | ✅︎ | — |
| `LTX2Pipeline` | LTX-2 / LTX-2.3 one-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` | ✅︎ | ✅︎ | | | — |
| `LTX2DistilledPipeline` | LTX-2 distilled two-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled` | ✅︎ | ✅︎ | | | — |
| `LTX2TwoStagePipeline` | LTX-2 / LTX-2.3 ordinary two-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` + matching Lightricks LoRA and upsampler | ✅︎ | ✅︎ | | | — |
| `LTX2DistilledOneStagePipeline` | LTX-2 / LTX-2.3 merged-distilled one-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` | ✅︎ | ✅︎ | | | — |
| `LTX2DistilledTwoStagePipeline` | LTX-2 / LTX-2.3 merged-distilled two-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` + matching Lightricks upsampler | ✅︎ | ✅︎ | | | — |
| `LTX2Pipeline` | LTX-2.5 Full/SFT one-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) |
| `LTX2TwoStagePipeline` | LTX-2.5 Full/SFT two-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` + `Lightricks/LTX-2.5` LoRA | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) |
| `LTX2DistilledOneStagePipeline` | LTX-2.5 merged-distilled one-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) |
| `LTX2DistilledTwoStagePipeline` | LTX-2.5 merged-distilled two-stage T2V and I2V | `Lightricks/LTX-2.5-Diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) |
| `LingBotVideoPipeline` | LingBot-Video dense and MoE T2I, T2V, TI2V | `robbyant/lingbot-video-dense-1.3b`, `robbyant/lingbot-video-moe-30b-a3b` | ✅︎ | | | | — |
| `SanaVideoPipeline` | SANA-Video-2B T2V (native 480p/720p; Diffusers adapter validated at both resolutions) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) |
| `SanaImageToVideoPipeline` | SANA-Video-2B I2V (native and Diffusers adapter validated at 480p/720p) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) |
| `MiniMaxH3Pipeline` | MiniMax H3 T2VA, first/last-keyframe FL2VA, and mixed-reference Ref2VA | `MiniMaxAI/MiniMax-H3` | ✅︎ | ✅︎ | | | [Published](https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3) |
| `DreamZeroPipeline` | DreamZero-DROID | `GEAR-Dreams/DreamZero-DROID` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `Pi0Pipeline` | π0 (Pi-Zero) VLA | `lerobot/pi0_base` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/lerobot/Pi0.md) |
| `HeliosPipeline`, `HeliosPyramidPipeline` | Helios | `BestWishYsh/Helios-Base`, `BestWishYsh/Helios-Mid`, `BestWishYsh/Helios-Distilled` | ✅︎ | ✅︎ | ✅︎ | | — |
| `MagiHumanPipeline` | MagiHuman | `SII-GAIR/daVinci-MagiHuman-Base-1080p` | ✅︎ | ✅︎ | | | — |
| `OvisImagePipeline` | Ovis-Image | `OvisAI/Ovis-Image` | ✅︎ | ✅︎ | | ✅︎ | — |
| `LongcatImagePipeline` | LongCat-Image | `meituan-longcat/LongCat-Image` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `LongCatImageEditPipeline` | LongCat-Image-Edit | `meituan-longcat/LongCat-Image-Edit` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `LongCatVideoAvatarPipeline` | LongCat-Video-Avatar-1.5 A2V/AI2V (native single-speaker and multi-speaker AI2V/AVC) | `meituan-longcat/LongCat-Video-Avatar-1.5` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/meituan-longcat/LongCat-Video-Avatar-1.5.md) |
| `BooguImagePipeline` | Boogu-Image | `Boogu/Boogu-Image-0.1-Base`, `Boogu/Boogu-Image-0.1-Edit` | ✅︎ | | | | — |
| `BooguImagePipeline` | Boogu-Image | `Boogu/Boogu-Image-0.1-Base`, `Boogu/Boogu-Image-0.1-Edit`, `Boogu/Boogu-Image-0.1-Edit-Turbo` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Boogu/Boogu-Image.md) |
| `StableDiffusionXLPipeline` | Stable-Diffusion-XL | `stabilityai/stable-diffusion-xl-base-1.0` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ | — |
| `CosyVoice3Model` | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✅︎ | ✅︎ | | ✅︎ | — |
| `OmniVoiceModel` | OmniVoice | `k2-fsa/OmniVoice` | ✅︎ | | | | — |
| `VoxCPM2TalkerForConditionalGeneration` | VoxCPM2 | `openbmb/VoxCPM2` | ✅︎ | | | | — |
| `OmniVoiceModel` | OmniVoice | `k2-fsa/OmniVoice` | ✅︎ | | | ✅︎ | — |
| `VoxCPM2TalkerForConditionalGeneration` | VoxCPM2 | `openbmb/VoxCPM2` | ✅︎ | | | ✅︎ | — |
| `DotsTTSForConditionalGeneration` | dots.tts | `rednote-hilab/dots.tts-soar` | ✅︎ | | | | — |
| `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | | — |
| `MammothModa2ForConditionalGeneration` | MammothModa2-Dev (AR-only image understanding) | `bytedance-research/MammothModa2-Dev` | ✅︎ | | | | — |
| `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
@@ -66,13 +77,11 @@ th {
| `FluxPipeline` | FLUX.1-schnell | `black-forest-labs/FLUX.1-schnell` | ✅︎ | ✅︎ | | ✅︎ | — |
| `OmniGen2Pipeline` | OmniGen2 | `OmniGen2/OmniGen2` | ✅︎ | ✅︎ | | ✅︎ | — |
| `StableAudioPipeline` | Stable-Audio-Open | `stabilityai/stable-audio-open-1.0` | ✅︎ | ✅︎ | | ✅︎ | — |
| `SoulXSingerPipeline` | SoulX-Singer (SVS) | `Soul-AILab/SoulX-Singer` | ✅︎ | | | | — |
| `SoulXSingerSVCPipeline` | SoulX-Singer-SVC | `Soul-AILab/SoulX-Singer` (`model-svc.pt`) | ✅︎ | | | | — |
| `AudioXPipeline` | AudioX | `zhangj1an/AudioX` | ✅︎ | ✅︎ | | | — |
| `Qwen3TTSForConditionalGeneration` | Qwen3-TTS-12Hz-1.7B-CustomVoice | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | ✅︎ | | | | [Published](https://recipes.vllm.ai/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) |
| `MiniMaxMusic3ForConditionalGeneration` | MiniMax Music 3 (text-to-music) | `MiniMaxAI/MiniMax-Music3` | ✅︎ | | | | — |
| `Qwen3TTSForConditionalGeneration` | Qwen3-TTS-12Hz-1.7B-CustomVoice | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | ✅︎ | | | ✅︎ | [Published](https://recipes.vllm.ai/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) |
| `Qwen3TTSForConditionalGeneration` | Qwen3-TTS-12Hz-1.7B-VoiceDesign | `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `Qwen3TTSForConditionalGeneration` | Qwen3-TTS-12Hz-1.7B-Base | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `MingTTSForConditionalGeneration` | Ming-omni-tts dense 0.5B | `inclusionAI/Ming-omni-tts-0.5B` | ✅︎ | | | | — |
| `MingTTSForConditionalGeneration` | Ming-omni-tts dense 0.5B | `inclusionAI/Ming-omni-tts-0.5B` | ✅︎ | | | ✅︎ | — |
| `GLMTTSForConditionalGeneration` | GLM-TTS | `zai-org/GLM-TTS` | ✅︎ | | | | — |
| `MossTTSNanoForCausalLM` | MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | ✅︎ | | | | — |
| `MossTTSDelayModel` | MOSS-TTS, MOSS-TTSD, MOSS-SoundEffect, MOSS-VoiceGenerator | `OpenMOSS-Team/MOSS-VoiceGenerator` | ✅︎ | | | | — |
@@ -86,17 +95,18 @@ th {
| `MiMoV2ASRForCausalLM` | MiMo-V2.5-ASR | `XiaomiMiMo/MiMo-V2.5-ASR` | ✅︎ | ✅︎ | | | — |
| `Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | ✅︎ | ✅︎ | | | — |
| `FishSpeechSlowARForConditionalGeneration` | Fish Speech S2 Pro | `fishaudio/s2-pro` | ✅︎ | ✅︎ | | ✅︎ | — |
| `DreamIDOmniPipeline` | DreamID-Omni | `XuGuo699/DreamID-Omni` | ✅︎ | ✅︎ | | | — |
| `SenseNovaU1Pipeline` | SenseNova-U1 (DiT-only) | `SenseNova/SenseNova-U1-8B-MoT` | ✅︎ | | | | — |
| `LancePipeline` | Lance | `bytedance-research/Lance` | ✅︎ | | | | — |
| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | | — |
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | | — |
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
|`DyninOmniForConditionalGeneration` | Dynin-Omni | `snu-aidas/Dynin-Omni` | ✅︎ | | | | — |
|`DyninOmniForConditionalGeneration` | Dynin-Omni | `snu-aidas/Dynin-Omni` | ✅︎ | | | ✅︎ | — |
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | — |
| `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — |
|`HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
| `HiDreamO1ImagePipeline` | HiDream-O1-Image | `HiDream-ai/HiDream-O1-Image` | ✅︎ | | | | — |
The `Recipe` column links the corresponding deployment recipe. `Published` is
preferred when a corresponding `recipes.vllm.ai` page exists; otherwise the
@@ -109,6 +119,10 @@ support metadata until their recipe is audited.
✅︎ indicates the model is supported on that backend. Empty cells mean not listed as supported on that backend.
See the [LingBot-World 2.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Robbyant/LingBot-World-2.0.md)
for offline and experimental realtime usage, supported controls, validation,
and current limitations.
<sup>H3</sup> MiniMax H3 AMD GPU support is validated on gfx942 (MI300X) and
gfx950 (MI350) in BF16, using the AITER `FLASH_ATTN` diffusion attention backend.
Other AMD SKUs (e.g. MI325X) are not listed until their
docs/serving/image_edit_api.md
+8 -1
View File
@@ -123,7 +123,14 @@ When `stream=false` or omitted, the endpoint returns the standard image edit res
}
],
"output_format": null,
"size": null
"size": null,
"metrics": {
"stage_durations": {
"queue_wait_ms": 1.0,
"stage_0_gen_ms": 2.0
},
"peak_memory_mb": 1024.0
}
}
```
docs/serving/speech_api.md
+61 -3
View File
@@ -131,7 +131,7 @@ Content-Type: application/json
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `ref_audio` | string | null | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) |
| `ref_audio` | string | null | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`). Local files fold `mtime_ns` and `size` into cache keys to automatically reload on-disk edits; HTTP URLs and base64 URIs remain cached by string locator. |
| `ref_text` | string | null | Transcript of reference audio |
| `x_vector_only_mode` | bool | null | Use speaker embedding only (no ICL) |
@@ -141,7 +141,20 @@ The response shape depends on the streaming parameters:
**Non-streaming (default).** With `stream=false` and no `stream_format`, returns the
complete clip as binary audio data with an appropriate `Content-Type` header (e.g.
`audio/wav`). The raw-bytes body has no JSON carrier, so no `usage` is reported.
`audio/wav`). Because the raw-bytes body has no JSON carrier, successful
non-streaming responses from non-diffusion speech servers report usage through
response headers:
| Header | Description |
| --- | --- |
| `x-vllm-omni-input-tokens` | Total input tokens (`text_tokens` + `audio_tokens`). |
| `x-vllm-omni-output-tokens` | Generated codec/audio tokens. |
| `x-vllm-omni-total-tokens` | `input_tokens` + `output_tokens`. |
| `x-vllm-omni-input-text-tokens` | Tokens from the synthesized text (`input` plus `instructions`). |
| `x-vllm-omni-input-audio-tokens` | Reference-audio codec frames, non-zero only for in-context voice cloning. |
Diffusion-mode speech servers route through a separate response path and do not
emit these headers.
**Raw audio stream** (`stream_format="audio"`). Streams raw audio bytes (PCM or
WAV) as they are decoded.
@@ -675,7 +688,7 @@ for result in response.json()["results"]:
|-----------|--------|---------|-------------|
| `tts_batch_max_items` | engine kwarg | 32 | Maximum number of items per batch request |
All items are fanned out to `generate()` concurrently. The engine's stage worker automatically batches them up to the configured `max_batch_size` and queues the rest — no client-side throttling needed.
All items are fanned out to `generate()` concurrently. The engine's stage worker automatically batches them up to the configured `max_num_seqs` and queues the rest — no client-side throttling needed.
For best throughput, set both stages' `max_num_seqs` above 1 via `--stage-overrides`. On the current Qwen3-TTS CustomVoice benchmark, stage 1 performed best at `max_num_seqs: 10`:
@@ -798,6 +811,51 @@ If you encounter OOM errors:
Use `/v1/audio/voices` to list available voices for the loaded model.
## Orchestration Loop (experimental)
Multi-stage omni deployments route stage outputs through a single orchestrator
loop. By default that loop polls every stage replica on a 1 ms cadence. An
opt-in event-driven mode replaces the poll with one reader task per live stage
replica awaiting its client directly, and switches the serving-side
final-output drain to a condition-variable wakeup at the same time.
**Configuration (environment variables):**
| Variable | Default | Description |
|----------|---------|-------------|
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `0` (off) | Switches the orchestration loop and the final-output drain from the legacy 1 ms poll to event-driven wakeups. Enabled by `1`, `true`, `yes`, or `on`, matched case-insensitively after surrounding whitespace is stripped; any other value leaves it off. |
Set it on the process that runs the orchestrator (stage 0 of an omni
deployment) before starting the server:
```bash
export VLLM_OMNI_EVENT_DRIVEN_ORCH=1
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--omni \
--port 8091
```
The server logs the selected loop mode and its reader/poller counts once at
startup, so you can confirm which loop is live.
Routing, output ordering, and terminal-state behavior are identical on both
loops; only the poll cadence changes. Leaving the variable unset keeps the
legacy poll loop, which is the supported default.
**Known limitations:**
- The measured serving A/B (idle CPU 2.43% to 0.07%; TTFP p99 -32% at
concurrency 8) predates the rebuild on the per-replica fault-isolation work
in [#4285](https://github.com/vllm-project/vllm-omni/pull/4285). That work
changed dead-replica handling and reader/poller lifecycle rather than the
steady-state output path, and the parity suite covers it, but the serving
A/B has not been re-run on the current head.
- The diffusion-poller branch is covered by unit tests only. Deployments whose
stages all run as standard engine cores never exercise it, including GLM-TTS,
which deploys its DiT without `stage_type: diffusion`.
- Concurrency 1 and 32 measured at parity with the legacy loop. At 32 the
latency is admission-bound, which this mode does not address.
## Development
Enable debug logging:
docs/serving/video_stream_api.md
+1 -1
View File
@@ -10,7 +10,7 @@ Each server instance runs a single model specified at startup with `vllm serve <
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
--deploy-config vllm_omni/deploy/qwen3_omni.yaml \
--deploy-config vllm_omni/deploy/qwen3_omni_moe.yaml \
--omni \
--port 8000 \
--trust-remote-code
docs/serving/videos_api.md
+23 -5
View File
@@ -161,11 +161,29 @@ are not implemented yet. Models may expose additional V2V controls through
`extra_params`. For example, Cosmos3 supports
`condition_frame_indexes_vision` and `condition_video_keep` to select which
decoded reference frames are used as clean conditioning. Cosmos3 transfer mode
also accepts `edge`, `blur`, `depth`, `seg`, or `wsm` control hints plus
transfer options such as `control_path`, `control_guidance`,
`control_guidance_interval`, `num_video_frames_per_chunk`,
`num_conditional_frames`, `show_control_condition`, and `show_input`; see the
Cosmos3 recipe for complete examples.
also accepts `edge`, `blur`, `depth`, `seg`, or `wsm` control hints. Each hint
may specify its own `control_path` and `control_weight`. Request-level transfer
options include
`control_guidance`, `control_guidance_interval`,
`emphasize_control_in_prompt`, `num_video_frames_per_chunk`,
`num_conditional_frames`, `show_control_condition`, and `show_input`. Transfer
uses its transfer-specific system prompt and, by default, appends a
control-adherence directive to the positive prompt. It adds duration/FPS and
resolution metadata to both CFG branches but does not add a negative prompt
automatically. The Cosmos3 recipe includes an optional reference negative
prompt and shows how to pass it. Set `emphasize_control_in_prompt`,
`use_duration_template`, or `use_resolution_template` to `false` to disable the
corresponding addition. `negative_metadata_mode` accepts `same`, `inverse`, or
`none` and defaults to `same` for transfer. See the Cosmos3 recipe for complete
examples.
HTTP redirects for `image_reference.image_url` follow vLLM's
`VLLM_MEDIA_URL_ALLOW_REDIRECTS` setting. Before starting the server, set it to
`1` (the default) to allow redirects or `0` to reject them. A redirect target
can differ from the original host, and vLLM-Omni does not yet fully implement
upstream vLLM's media URL allowlist protection. Deployments should therefore
treat remote media URLs as untrusted and choose this setting as part of their
URL access policy.
### Speech-to-Video
docs/user_guide/diffusion/attention_backends.md
+114 -413
View File
@@ -1,494 +1,195 @@
# Diffusion Attention Backends
This document describes the diffusion attention backends available in vLLM-Omni, how to select them globally and per-role, the per-platform defaults, and how to use SageAttention.
Use this page to select and configure a diffusion attention backend. Backend-
specific installation, tuning, and compatibility details live in separate
guides so that the selection contract stays easy to scan.
## Overview
Diffusion attention backend selection applies to DiT attention in image and
video generation models. It does **not** change autoregressive LLM attention,
which uses vLLM's attention selector.
Diffusion attention backend selection is resolved in `vllm_omni.diffusion.attention.selector`. It looks up the backend from a structured `AttentionConfig` carried on `OmniDiffusionConfig` and falls back to the platform default when nothing is configured.
For the internal selector, registry, and platform contract, see
[Attention Backend Selection](../../design/feature/attention_backend_selection.md).
This backend is used by diffusion attention layers such as the DiT attention in video and image generation models. It does **not** affect autoregressive (LLM) attention paths — those go through vLLM's own attention backend selector.
## Choose a guide
The full set of backends and their platform defaults is in the **Backend Options** and **Platform Defaults** sections below. If no attention backend is configured, vLLM-Omni asks the current platform to choose the default.
| Need | Guide |
| --- | --- |
| Select a conservative or platform-native dense kernel | [Dense Backends](attention_backends/dense_backends.md) |
| Use FlashInfer trtllm-gen FMHA, Skip-Softmax, or TRTLLM SAGE quantization | [TRTLLM Attention](attention_backends/trtllm.md) |
| Install and use SageAttention 2.2 or SageAttention3 | [SageAttention](attention_backends/sage.md) |
| Match training or rollout kernels loaded from Hugging Face | [Hugging Face Hub Backends](attention_backends/huggingface_hub.md) |
| Use block-sparse video attention on Ascend NPU | [RainFusion](attention_backends/rainfusion.md) |
| Use FastVideo VSA with FastWan2.2-TI2V-5B on CUDA | [FastVideo VSA](attention_backends/fastvideo_vsa.md) |
## Backend Options
| Value | Notes |
|---|---|
| `TRTLLM_ATTN` | FlashInfer's trtllm-gen FMHA (TensorRT-LLM's generated kernels, vendored by FlashInfer). Dense BF16, GQA native, `head_dim=128`. Datacenter Blackwell only (sm_100 / sm_103). Packed paths can provide `cu_seqlens` directly. Supports optional **Skip-Softmax** sparse attention — see below. Requires `flashinfer`. |
| `FLASH_ATTN` | Wraps FlashAttention 4 on Blackwell when `flash-attn-4` is installed, then falls back to FlashAttention 3/2. Default on Hopper / Ada / Ampere when a compatible FlashAttention package is installed. |
| `CUDNN_ATTN` | Pins `sdpa_kernel([CUDNN_ATTENTION])`. Default on Blackwell (sm_10x / sm_12x) with cuDNN ≥ 9.5. Wins on mask-heavy DiTs (HunyuanVideo-1.5: 2× e2e vs SDPA). |
| `FLASHINFER_ATTN` | Uses FlashInfer's batch-prefill wrapper and supports mixed Q/K and V input dtypes through backend-specific configuration. Used as Blackwell fallback when cuDNN is unavailable. Requires `flashinfer` >= 0.6.16rc1 for mixed-dtype configurations. |
| `TORCH_SDPA` | PyTorch `scaled_dot_product_attention` with the default backend dispatcher. Most conservative; always available. |
| `SAGE_ATTN` | SageAttention 2.2 — INT8-quantized attention with FP16 accumulation. Lossy but typically visually indistinguishable on diffusion outputs. Requires `sageattention`. |
| `SAGE_ATTN_3` | Requires `sageattn3` from `SageAttention/sageattention3_blackwell`. CUDA only, intended for Blackwell GPUs, with GQA/MQA requests falling back to PyTorch SDPA. |
| `FLASH_ATTN_HUB` | FlashAttention 2 from HuggingFace `kernels` library. Useful for train/rollout alignment. |
| `FLASH_ATTN_3_HUB` | FlashAttention 3 from HuggingFace `kernels` library. CUDA Hopper (sm_90+) only; falls back to `FLASH_ATTN_HUB` on older GPUs. |
| `RAINFUSION_ATTN` | MindIE-SD **RainFusion** block-sparse video attention — see [below](#rainfusion_attn-backend-and-block-sparse-video-attention). Ascend NPU only; requires `mindiesd`. Delegates to `FLASH_ATTN` for anything that is not a packed video sequence. |
## Backend options
| Value | Family | Primary use | Detail |
| --- | --- | --- | --- |
| `TORCH_SDPA` | Dense | Conservative reference; always available | [Dense Backends](attention_backends/dense_backends.md#torch_sdpa) |
| `FLASH_ATTN` | Dense | FlashAttention 4/3/2 depending on the installed package and GPU | [Dense Backends](attention_backends/dense_backends.md#flash_attn) |
| `CUDNN_ATTN` | Dense | Mask-heavy DiTs on Blackwell with cuDNN 9.5 or newer | [Dense Backends](attention_backends/dense_backends.md#cudnn_attn) |
| `FLASHINFER_ATTN` | Dense or quantized | FlashInfer batch prefill; optional mixed Q/K and V dtypes | [Dense Backends](attention_backends/dense_backends.md#flashinfer_attn) |
| `TRTLLM_ATTN` | Dense, sparse, or quantized | Datacenter Blackwell with `head_dim=128` and compatible packed paths | [TRTLLM Attention](attention_backends/trtllm.md) |
| `SAGE_ATTN` | Quantized | SageAttention 2.2 INT8 attention | [SageAttention](attention_backends/sage.md#sage_attn) |
| `SAGE_ATTN_3` | Quantized | SageAttention3 on Blackwell | [SageAttention](attention_backends/sage.md#sage_attn_3) |
| `FLASH_ATTN_HUB` | Hub kernel | FlashAttention 2 from Hugging Face `kernels` | [Hugging Face Hub Backends](attention_backends/huggingface_hub.md) |
| `FLASH_ATTN_3_HUB` | Hub kernel | FlashAttention 3 from Hugging Face `kernels` on Hopper or newer | [Hugging Face Hub Backends](attention_backends/huggingface_hub.md) |
| `RAINFUSION_ATTN` | Block sparse | MindIE-SD RainFusion video attention on Ascend NPU | [RainFusion](attention_backends/rainfusion.md) |
| `FASTVIDEO_VSA` | Block sparse | FastVideo variable sparse self-attention for FastWan2.2-TI2V-5B on CUDA | [FastVideo VSA](attention_backends/fastvideo_vsa.md) |
## Configuration
Diffusion attention backends can be configured three ways, in priority order:
Backend selection is resolved in this order:
1. **`--diffusion-attention-config`** — structured per-role config (highest priority).
2. **`--diffusion-attention-backend` / `DIFFUSION_ATTENTION_BACKEND` env var** — global shorthand that sets the default backend.
3. **Platform default** — used when nothing is configured.
1. `--diffusion-attention-config` per-role configuration.
2. `--diffusion-attention-backend` or `DIFFUSION_ATTENTION_BACKEND` as a
global default.
3. The current platform's default.
`--diffusion-attention-backend` is shorthand for `--diffusion-attention-config.default.backend`. It may be combined with `--diffusion-attention-config.per_role.*` overrides, but is mutually exclusive with `--diffusion-attention-config.default.backend`.
`--diffusion-attention-backend` is shorthand for
`--diffusion-attention-config.default.backend`. Do not pass it together with an
explicit `default.backend` in the structured configuration.
### Global default
Set the default backend for every diffusion attention layer:
```bash
# CLI flag
vllm-omni serve <model> --diffusion-attention-backend SAGE_ATTN
vllm-omni serve <model> --diffusion-attention-backend FLASH_ATTN
# Environment variable (also recognized for backwards compatibility)
export DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN
# Backwards-compatible environment variable
export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN
```
### Per-role configuration
Roles are free-form strings declared by each diffusion model. The two common categories are `"self"` and `"cross"`; model-specific roles (e.g. `"ltx2.audio_to_video"`) may also be declared. A role string is matched in this order:
Roles are declared by each diffusion model. Common categories are `self` and
`cross`; a model may also use a more specific role such as
`ltx2.audio_to_video`. Resolution order is:
1. Exact `per_role[role]` match
2. `per_role[role_category]` fallback (e.g. `"ltx2.audio_to_video"` → `"cross"`)
3. `default`
4. Platform default
Use vLLM-style dotted flags or one JSON blob:
1. Exact `per_role[role]` match.
2. Category `per_role[role_category]` match.
3. `default`.
4. Platform default.
```bash
# Dotted flags
vllm-omni serve <model> \
--diffusion-attention-config.default.backend FLASH_ATTN \
--diffusion-attention-config.per_role.cross.backend TORCH_SDPA
--diffusion-attention-config.default.backend FLASH_ATTN \
--diffusion-attention-config.per_role.cross.backend TORCH_SDPA
# JSON
# Equivalent JSON
vllm-omni serve <model> \
--diffusion-attention-config '{"default":{"backend":"FLASH_ATTN"},"per_role":{"cross":{"backend":"TORCH_SDPA"}}}'
--diffusion-attention-config \
'{"default":{"backend":"FLASH_ATTN"},"per_role":{"cross":{"backend":"TORCH_SDPA"}}}'
```
A backend that needs configuration exposes it as a typed field on the spec:
`TRTLLM_ATTN`'s Skip-Softmax (see [below](#trtllm_attn-backend-and-skip-softmax)) and
`RAINFUSION_ATTN`'s RainFusion block (see [below](#rainfusion_attn-backend-and-block-sparse-video-attention)):
```bash
--diffusion-attention-config.default.backend TRTLLM_ATTN \
--diffusion-attention-config.default.skip_softmax.target_sparsity 0.5
```
### Programmatic API
When constructing `OmniDiffusionConfig` directly:
### Python API
```python
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, AttnQuantSpec, OmniDiffusionConfig
from vllm_omni.diffusion.data import (
AttentionConfig,
AttentionSpec,
OmniDiffusionConfig,
)
config = OmniDiffusionConfig(
diffusion_attention_config=AttentionConfig(
default=AttentionSpec(backend="FLASH_ATTN"),
per_role={
"cross": AttentionSpec(backend="TORCH_SDPA"),
},
per_role={"cross": AttentionSpec(backend="TORCH_SDPA")},
),
...,
)
```
A plain dict is also accepted and normalized to `AttentionConfig`.
Backend-specific typed blocks are documented with their consumers:
#### FlashInfer QK16/V8 quantized attention
- `quant`: [FlashInfer](attention_backends/dense_backends.md#flashinfer-quantized-attention)
and [TRTLLM SAGE](attention_backends/trtllm.md#sage-quantization).
- `skip_softmax`: [TRTLLM Skip-Softmax](attention_backends/trtllm.md#skip-softmax).
- `block_sparse`: [RainFusion](attention_backends/rainfusion.md#configuration).
- `fastvideo_vsa_topk`: [FastVideo VSA](attention_backends/fastvideo_vsa.md#choose-top-k).
`FLASHINFER_ATTN` accepts quantization options through `AttentionSpec.quant`. For QK16/V8:
```python
config = OmniDiffusionConfig(
diffusion_attention_config=AttentionConfig(
default=AttentionSpec(
backend="FLASHINFER_ATTN",
quant=AttnQuantSpec(
dtype_qk="bfloat16",
dtype_vo="fp8_e4m3",
),
),
),
...,
)
```
`dtype_qk` configures the Q and K input dtypes. `dtype_vo` configures the V input dtype.
## Platform Defaults
## Platform defaults
### Blackwell (sm_100 / sm_103 / sm_120 / sm_121)
Auto-route preference, in order:
The CUDA auto-route preference is:
1. `TRTLLM_ATTN` — on **datacenter** Blackwell (sm_100 / sm_103) when `flashinfer` is installed, the model's `head_dim` is 128, **and the model declares a compatible packed/mask-free path**
2. `CUDNN_ATTN` — when cuDNN ≥ 9.5 is available (ships in PyTorch 2.5+ wheels)
3. `FLASHINFER_ATTN` — when `flashinfer` is installed but cuDNN < 9.5
4. `FLASH_ATTN` — when `flash-attn` is installed with the Blackwell CUTE kernel
5. `TORCH_SDPA` — last resort
1. `TRTLLM_ATTN` on datacenter Blackwell (sm_100/sm_103) when FlashInfer is
available, `head_dim=128`, and the model declares a compatible packed or
mask-free path.
2. `CUDNN_ATTN` when cuDNN 9.5 or newer is available.
3. `FLASHINFER_ATTN` when FlashInfer is available but cuDNN is too old.
4. `FLASH_ATTN` when a compatible package is installed.
5. `TORCH_SDPA`.
`TRTLLM_ATTN` is skipped on workstation Blackwell (sm_120 / sm_121) and for any `head_dim != 128`, so those GPUs keep the `CUDNN_ATTN` route described below.
`TRTLLM_ATTN` is not auto-selected on workstation Blackwell
(sm_120/sm_121), for other head dimensions, or for paths that require an
unsupported mask.
`TRTLLM_ATTN` outranks `CUDNN_ATTN` on datacenter Blackwell for compatible packed/mask-free pipelines. Workstation Blackwell (sm_120 / sm_121) and pipelines that require attention masks retain their normal fallback.
### Hopper, Ada, and Ampere
The startup log line `Defaulting to diffusion attention backend CUDNN_ATTN (Blackwell sm_120, cuDNN 91002)` confirms the route.
The CUDA auto-route uses `FLASH_ATTN` when available and otherwise falls back
to `TORCH_SDPA`. `CUDNN_ATTN` and `FLASHINFER_ATTN` remain explicit options.
**Why CUDNN_ATTN by default**: on mask-heavy diffusion models (HunyuanVideo-1.5, Qwen-Image), cuDNN's pinned FMHA kernel sidesteps a PyTorch SDPA dispatch quirk where the unpinned dispatcher picks `EFFICIENT_ATTENTION` (~25 ms) for masked calls instead of cuDNN (~11 ms). The pin gives 2× e2e on HV-1.5 with no regression on lighter models.
Other platforms validate an explicit backend and choose their own default
through the platform implementation. Check the startup log to confirm the
resolved backend.
### Hopper (sm_90) / Ada (sm_89) / Ampere (sm_80–sm_86)
## Choosing a backend manually
Auto-route preference:
Override the platform default when you need:
1. `FLASH_ATTN` — when `flash-attn` is installed
2. `TORCH_SDPA` — fallback
- a correctness reference (`TORCH_SDPA`);
- a backend-specific workaround;
- training/rollout kernel alignment (Hub backends); or
- an explicitly validated sparse or quantized speedup.
`CUDNN_ATTN` and `FLASHINFER_ATTN` are still selectable via env var on these GPUs but are not in the auto-route — FlashAttention 2 is the well-tuned path on pre-Blackwell hardware.
The startup log prints the resolved backend and whether it came from explicit
configuration or platform defaulting. If no resolution message appears, check
earlier logs for diffusion-stage initialization failures.
## TRTLLM_ATTN Backend and Skip-Softmax
## Reference benchmark
`TRTLLM_ATTN` runs FlashInfer's trtllm-gen FMHA. Dense it is a BF16 backend like the others; on top of
that it can enable **Skip-Softmax**, a sparse-attention mode that trades a little fidelity for speed
(algorithm: [design doc](../../design/feature/skip_softmax.md)).
The following BF16 results were measured on an sm_120 RTX Pro 6000 Blackwell
with the same prompt and seed across runs. Treat them as reference results, not
portable guarantees.
Enable it through the typed `skip_softmax` block on the attention spec:
| Key | Valid values | Meaning |
|---|---|---|
| `target_sparsity` | finite, `[0, 1]` | Operating point on a calibrated curve. Needs a calibration for the model. |
| `threshold` | finite, `≥ 0` | Direct threshold, no calibration needed. Mutually exclusive with `target_sparsity`. |
| `disabled_until_timestep` | finite, `[0, 1]` | Holds early, high-noise steps dense; skip turns on once the normalized timestep `t ≤ D`. |
```bash
vllm-omni serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--diffusion-attention-config '{"default": {"backend": "TRTLLM_ATTN",
"skip_softmax": {"target_sparsity": 0.65, "disabled_until_timestep": 0.86}}}'
```
Programmatically the same block is a typed `SkipSoftmaxSpec` (values validated at construction):
```python
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, SkipSoftmaxSpec
AttentionConfig(
default=AttentionSpec(
backend="TRTLLM_ATTN",
skip_softmax=SkipSoftmaxSpec(target_sparsity=0.65, disabled_until_timestep=0.86),
),
)
```
**Start at `target_sparsity=0.65, disabled_until_timestep=0.86`** and raise `target_sparsity` only
as far as your quality bar allows — the output diverges from dense as it climbs. The gain grows with
sequence length, since Skip-Softmax only accelerates attention. Dense BF16 is `TRTLLM_ATTN` with no
`skip_softmax` block.
Requires datacenter Blackwell with `head_dim == 128`; elsewhere, or without FlashInfer, selecting
`TRTLLM_ATTN` raises rather than silently degrading.
## TRTLLM_ATTN SAGE Quantization
On top of dense/Skip-Softmax, `TRTLLM_ATTN` can run **SAGE** (SageAttention) — the trtllm-gen
FMHA kernel with per-block quantized Q/K and per-channel fp8 V, so the two attention matmuls run
in low precision. Enable it through the typed `quant` block on the attention spec:
| Key | Valid values | Meaning |
|---|---|---|
| `dtype_qk` | `int8`, `fp8_e4m3` | Quantization dtype for Q and K. Absent ⇒ dense (SAGE off). |
| `q_block_size` | `1`, `4`, `16` | Per-token block size for Q scales (default `1`). |
| `k_block_size` | `1`, `4`, `16` | Per-token block size for K scales (default `16`). |
V is always quantized per-channel to fp8_e4m3 and K-smoothing is applied inside the routine, so
neither is a user knob. Only block sizes `1`, `4`, `16` have compiled kernels.
The `quant` block is a shared `AttnQuantSpec` consumed by more than one backend; each reads the
fields that apply and rejects the rest. `TRTLLM_ATTN` (SAGE) reads `dtype_qk` (`int8` / `fp8_e4m3`),
`q_block_size`, and `k_block_size`; `FLASHINFER_ATTN` reads `dtype_qk` (`float16` / `bfloat16`),
`dtype_vo`, and `flashinfer_backend` (see its section). A `dtype_qk` from the wrong set raises.
```bash
vllm-omni serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--diffusion-attention-config '{"default": {"backend": "TRTLLM_ATTN",
"quant": {"dtype_qk": "fp8_e4m3", "q_block_size": 1, "k_block_size": 16}}}'
```
```python
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, AttnQuantSpec
AttentionConfig(
default=AttentionSpec(
backend="TRTLLM_ATTN",
quant=AttnQuantSpec(dtype_qk="fp8_e4m3", q_block_size=1, k_block_size=16),
),
)
```
**Requirements.** Needs FlashInfer ≥ 0.6.16rc1. Kernel
availability is arch-dependent: `fp8_e4m3` QK has kernels on both **SM100** (B200) and **SM103**
(B300); `int8` QK kernels are compiled for **SM100 only**.
## RAINFUSION_ATTN Backend and Block-Sparse Video Attention
`RAINFUSION_ATTN` runs MindIE-SD's RainFusion (`rf_v2`) kernel on Ascend NPU. It pools each
128-token block of keys, ranks them per query block, and computes attention against only the
top-scoring ones. Video tokens are rearranged into `(t, h, w)` order first, so the kept blocks are
spatiotemporal neighbours rather than arbitrary rows of the packed sequence.
Only the video segment is sparsified. The prefix rows (text, visual conditions, audio) and the
first-frame blocks are always kept dense, which is why the realized sparsity is lower than the
nominal `sparsity` you configure. Anything the kernel cannot handle — a warmup denoise step, an
exempt layer, a sequence with no published video geometry, a video segment under 32 blocks —
delegates to `FLASH_ATTN`, so a model can select this backend unconditionally.
Enable it through the typed `block_sparse` block on the attention spec, shared by every
block-sparse backend:
| Key | Valid values | Meaning |
|---|---|---|
| `sparsity` | finite, `[0, 1]` | Nominal fraction of key blocks dropped per query block. `0` disables sparsity. Defaults to `0.8`. |
| `start_step` | `≥ 0` | Keep the first N denoise steps dense. Layout is decided early, so these steps dominate structural fidelity. |
| `skip_layers` | index selector, e.g. `"0-3,38"` | DiT blocks that always stay dense. |
```bash
vllm-omni serve MiniMaxAI/MiniMax-H3 \
--diffusion-attention-config '{"default": {"backend": "RAINFUSION_ATTN",
"block_sparse": {"sparsity": 0.8, "start_step": 0}}}'
```
Programmatically the same block is a typed `BlockSparseSpec` (values validated at construction):
```python
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, BlockSparseSpec
AttentionConfig(
default=AttentionSpec(
backend="RAINFUSION_ATTN",
block_sparse=BlockSparseSpec(sparsity=0.8, start_step=0, skip_layers="0-1"),
),
)
```
**Tune in the order `start_step` → `sparsity` → `skip_layers`.** Raise `start_step` first: it is the
cheapest way to recover structure, because the early high-noise steps decide global layout while the
later steps only refine texture. Then walk `sparsity` up to your quality bar. Reach for
`skip_layers` last, once an A/B against dense at the same seed points at specific blocks.
Requires Ascend NPU with `mindiesd`; selecting it on any other platform raises. It is also
incompatible with ring sequence parallelism, since `rf_v2` needs the whole key sequence to rank
blocks — use Ulysses SP (`ring_degree=1`).
### Which geometries run sparse
MindIE-SD rf_v2 handles arbitrary video grids by spatially rearranging video first and, when
necessary, promoting a real-video suffix to its always-kept prefix segment. It never pads: rf_v2
does not consume an attention mask for padded keys. The remaining video rows stay sparse, while the
promoted rows remain visible to every query and key.
For example, MiniMax-H3 at 1344x768 has grid `(62, 24, 42)`: its 62496 video rows leave a 32-row
128-block residue, and its two non-8-wide latent columns form a 2928-row rearranged suffix.
MindIE-SD promotes 2976 rows, leaving 59520 sparse rows (`465 × 128`). This keeps the block mask
and kernel tiling aligned, at the cost of retaining more blocks.
8x8-aligned spatial grids remain preferable for performance. A latent `h` or `w` not divisible by 8
creates an always-kept residual suffix and lowers realized sparsity. "Multiple of 8" on the latent
grid means **width and height that are multiples of 256**. This path requires a MindIE-SD release
that implements protected video tails for rf_v2.
## End-to-End Benchmark (BF16, sm_120 RTX Pro 6000 Blackwell)
Same prompt and seed across runs. `Total generation time` from `text_to_video.py` / `text_to_image.py`.
| Model | Shape | TORCH_SDPA | CUDNN_ATTN | FLASHINFER_ATTN |
|---|---|---|---|---|
| Model | Shape | `TORCH_SDPA` | `CUDNN_ATTN` | `FLASHINFER_ATTN` |
| --- | --- | ---: | ---: | ---: |
| HunyuanVideo-1.5 (T2V) | 480p / 33f / 50 steps | 147.05 s | **73.02 s** | 127.84 s |
| Wan 2.2 14B (T2V) | 480p / 33f / 40 steps | 117.75 s | 117.17 s | **115.07 s** |
| Qwen-Image (T2I) | 1024² / 50 steps | 17.41 s | **15.14 s** | 16.02 s |
| FLUX.2-dev (T2I) | 1024² / 50 steps, TP=2 | 53.62 s | **53.30 s** | 54.94 s |
Pattern: mask-heavy DiTs (HV-1.5, Qwen-Image) favor `CUDNN_ATTN`; lighter-mask DiTs and TP-saturated configs (Wan 2.2, FLUX.2 TP=2) tie within noise.
Mask-heavy DiTs favored `CUDNN_ATTN`; lighter-mask or TP-saturated workloads
were close enough that users should benchmark their exact model and shape.
## Known Limitations
## Compatibility anchors
### LTX-2.0: `CUDNN_ATTN` crashes under torch.compile
The following headings preserve links to sections that moved into dedicated
guides.
LTX-2's audio attention has a symbolic head_dim under torch.compile tracing. cuDNN's SDPA backend selector rejects symbolic dims and Dynamo aborts compilation. Tracked in [#3121](https://github.com/vllm-project/vllm-omni/issues/3121).
## TRTLLM_ATTN Backend and Skip-Softmax
**Workaround**: explicitly select `FLASHINFER_ATTN` or `TORCH_SDPA` for LTX-2.0:
See [TRTLLM Attention](attention_backends/trtllm.md#skip-softmax).
```bash
DIFFUSION_ATTENTION_BACKEND=FLASHINFER_ATTN python examples/offline_inference/text_to_video/text_to_video.py \
--model Lightricks/LTX-2 ...
```
## TRTLLM_ATTN SAGE Quantization
### FlashAttention-4 on Blackwell
See [TRTLLM Attention](attention_backends/trtllm.md#sage-quantization).
Install the optional CUDA 13 extra to use the CuTe-based FA4 path:
## RAINFUSION_ATTN Backend and Block-Sparse Video Attention
```bash
pip install 'vllm-omni[fa4]'
```
`FLASH_ATTN` prefers `flash_attn.cute` on Blackwell and falls back to FA3/FA2
when it is unavailable. Version `4.0.0b18` is required; earlier beta
wheels had known JIT failures on Blackwell.
## Choosing a Backend Manually
### When to override the default
- **Quality validation**: compare a new backend against `TORCH_SDPA` as the reference, since SDPA's default dispatcher is the most extensively tested.
- **Lossy speedup hunting**: try `SAGE_ATTN` (INT8 quantized) on diffusion outputs — typically indistinguishable visually but always validate.
- **Workaround for known issues**: see Known Limitations above.
### Verifying which backend is in use
The startup log prints one of:
```
Using diffusion attention backend 'CUDNN_ATTN' # explicit override
Defaulting to diffusion attention backend CUDNN_ATTN ... # auto-route
Defaulting to diffusion attention backend SDPA # nothing else available
```
If you don't see one of these, the model didn't reach diffusion stage init — check earlier logs for failures.
See [RainFusion](attention_backends/rainfusion.md).
## SageAttention Installation
vLLM-Omni expects SageAttention to be installed into the same Python environment as vLLM-Omni.
Build from source:
```bash
git clone https://github.com/thu-ml/SageAttention.git
cd SageAttention
export EXT_PARALLEL=4 NVCC_APPEND_FLAGS="--threads 8" MAX_JOBS=32
pip install . --no-build-isolation
```
Quick check:
```bash
python -c "import sageattention; print(sageattention.__file__)"
```
See [SageAttention](attention_backends/sage.md#installation).
## SageAttention3 Installation
vLLM-Omni expects SageAttention3 to be installed into the same Python environment as vLLM-Omni.
Build from source:
```bash
git clone https://github.com/thu-ml/SageAttention.git
cd SageAttention/sageattention3_blackwell
python setup.py install
```
Quick check:
```bash
python -c "import sageattn3; print(sageattn3.__file__)"
```
Notes:
- `SAGE_ATTN_3` is only selected on CUDA when `sageattn3` is importable and the GPU is Blackwell-class.
- SageAttention3's Blackwell kernel assumes `Hq == Hkv`. In vLLM-Omni, GQA/MQA diffusion requests fall back to PyTorch SDPA for correctness.
See [SageAttention](attention_backends/sage.md#sageattention3-installation).
## HuggingFace Kernels Hub Backends
To achieve perfect numerical consistency between **training** (typically using HuggingFace Diffusers with Hub kernels) and **serving/rollout** (using vLLM-Omni), you can use the Hub-based attention backends. This eliminates numerical drift / sampling divergence caused by executing different local kernel versions during rollout.
The following backend options are supported:
- `FLASH_ATTN_HUB` (HuggingFace `kernels-community/flash-attn2`)
- `FLASH_ATTN_3_HUB` (HuggingFace `kernels-community/flash-attn3`, Hopper sm_90+ only)
### Installation
To use these backends, you must install the `kernels` library:
```bash
pip install kernels==0.14.1
```
If the `kernels` library is not available in the environment, vLLM-Omni will log a warning and fall back gracefully to the corresponding local backend implementations (`FLASH_ATTN`). On CUDA GPUs below Hopper (compute capability < 9.0), `FLASH_ATTN_3_HUB` falls back to `FLASH_ATTN_HUB`.
### Usage
Select a Hub backend using the global CLI flag or environment variables:
```bash
# Environment variable
export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN_HUB
# CLI flag
vllm-omni serve <model> --diffusion-attention-backend FLASH_ATTN_HUB
```
## Usage Examples
### Default (auto-route)
```bash
python examples/offline_inference/text_to_video/text_to_video.py \
--model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v \
--prompt "A dog running across a field of golden wheat." \
--height 480 --width 832 --num-frames 33 \
--num-inference-steps 50 --seed 42 --guidance-scale 6.0 \
--output hv15.mp4
```
On Blackwell this picks `CUDNN_ATTN` automatically. Check the log for the `Defaulting to ...` line.
### Explicit backend selection
```bash
DIFFUSION_ATTENTION_BACKEND=FLASHINFER_ATTN python examples/offline_inference/text_to_video/text_to_video.py \
--model Lightricks/LTX-2 \
--prompt "A dog running across a field of golden wheat." \
--height 480 --width 832 --num-frames 33 \
--num-inference-steps 40 --seed 42 --guidance-scale 4.0 \
--output ltx2.mp4
```
### SageAttention (lossy)
```bash
DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN python examples/offline_inference/text_to_video/text_to_video.py \
--model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v \
--prompt "A dog running across a field of golden wheat." \
--height 480 --width 832 --num-frames 33 \
--num-inference-steps 30 --seed 42 --guidance-scale 6.0 \
--tensor-parallel-size 2 \
--output hv15_sage.mp4
```
Example: Wan2.2 TI2V 5B
```bash
DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN python examples/offline_inference/text_to_video/text_to_video.py \
--model Wan-AI/Wan2.2-TI2V-5B-Diffusers \
--prompt "A dog running across a field of golden wheat." \
--height 704 --width 1280 --num-frames 49 \
--num-inference-steps 30 --seed 42 --guidance-scale 5.0 \
--tensor-parallel-size 2 \
--output outputs/wan22_sage.mp4
```
### Enable SageAttention3
Example:
```bash
DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN_3 python examples/offline_inference/text_to_video/text_to_video.py \
--model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v \
--prompt "A dog running across a field of golden wheat." \
--height 480 --width 832 --num-frames 33 \
--num-inference-steps 30 --seed 42 --guidance-scale 6.0 \
--tensor-parallel-size 2 \
--output outputs/hv15_sage3.mp4
```
See [Hugging Face Hub Backends](attention_backends/huggingface_hub.md).
docs/user_guide/diffusion/attention_backends/dense_backends.md
+110
View File
@@ -0,0 +1,110 @@
# Dense Attention Backends
Use these backends when you want dense attention without backend-specific
sparsity. Start with `TORCH_SDPA` for correctness comparisons, then benchmark
the fastest compatible kernel for your model, shape, and hardware.
For selection precedence and per-role configuration, see the
[attention backend overview](../attention_backends.md).
## `TORCH_SDPA`
`TORCH_SDPA` calls PyTorch `scaled_dot_product_attention` and lets PyTorch's
dispatcher choose the implementation. It is always available and is the most
conservative reference when validating another backend.
```bash
vllm-omni serve <model> --diffusion-attention-backend TORCH_SDPA
```
## `FLASH_ATTN`
`FLASH_ATTN` uses the installed FlashAttention implementation. On Blackwell it
prefers the CuTe-based FlashAttention 4 path and falls back to
FlashAttention 3 or 2 when that path is unavailable. On Hopper, Ada, and
Ampere it is the preferred automatic route when a compatible package is
installed.
```bash
vllm-omni serve <model> --diffusion-attention-backend FLASH_ATTN
```
### FlashAttention 4 on Blackwell
Install the optional CUDA 13 extra:
```bash
pip install 'vllm-omni[fa4]'
```
Version `4.0.0b18` is required; earlier beta wheels had known JIT failures on
Blackwell. If the CuTe path is unavailable, the backend falls back to the
compatible FlashAttention 3 or 2 path.
## `CUDNN_ATTN`
`CUDNN_ATTN` pins PyTorch SDPA to `CUDNN_ATTENTION`. It is particularly useful
for mask-heavy DiTs and is automatically preferred on Blackwell when cuDNN
9.5 or newer is available and the higher-priority TRTLLM route is not
compatible.
```bash
vllm-omni serve <model> --diffusion-attention-backend CUDNN_ATTN
```
### LTX-2.0 limitation
LTX-2 audio attention has a symbolic head dimension during `torch.compile`
tracing. The cuDNN SDPA selector rejects that symbolic dimension and Dynamo
aborts compilation. This is tracked in
[issue #3121](https://github.com/vllm-project/vllm-omni/issues/3121).
Use `FLASHINFER_ATTN` or `TORCH_SDPA` as a workaround:
```bash
DIFFUSION_ATTENTION_BACKEND=FLASHINFER_ATTN \
python examples/offline_inference/text_to_video/text_to_video.py \
--model Lightricks/LTX-2 ...
```
## `FLASHINFER_ATTN`
`FLASHINFER_ATTN` uses FlashInfer's batch-prefill wrapper. It is an explicit
option on CUDA platforms and an automatic Blackwell fallback when FlashInfer
is installed but cuDNN is too old for `CUDNN_ATTN`.
```bash
vllm-omni serve <model> --diffusion-attention-backend FLASHINFER_ATTN
```
### FlashInfer quantized attention
The backend accepts an `AttentionSpec.quant` block. For QK16/V8, keep Q and K
in FP16 or BF16 and use FP8 E4M3 for V:
```python
from vllm_omni.diffusion.data import (
AttentionConfig,
AttentionSpec,
AttnQuantSpec,
OmniDiffusionConfig,
)
config = OmniDiffusionConfig(
diffusion_attention_config=AttentionConfig(
default=AttentionSpec(
backend="FLASHINFER_ATTN",
quant=AttnQuantSpec(
dtype_qk="bfloat16",
dtype_vo="fp8_e4m3",
),
),
),
...,
)
```
`dtype_qk` controls Q and K; `dtype_vo` controls V. Mixed-dtype
configurations require FlashInfer 0.6.16rc1 or newer. The shared quantization
schema is also consumed by TRTLLM, but each backend validates its own allowed
fields and values; see [TRTLLM SAGE quantization](trtllm.md#sage-quantization).
docs/user_guide/diffusion/attention_backends/fastvideo_vsa.md
+125
View File
@@ -0,0 +1,125 @@
# FastVideo VSA
FastVideo Variable Sparse Attention (VSA) accelerates the `FastVideo/FastWan2.2-TI2V-5B-Diffusers` model's
self-attention by partitioning the post-patch latent grid into spatiotemporal
blocks. For every query block, VSA scores the key/value blocks and computes
attention only against the selected top-k blocks.
The supported checkpoint provides both text-to-video and image-to-video modes through `Wan22Pipeline`; the separate Wan I2V-14B, S2V, and VACE pipelines are outside this backend's supported scope.
VSA is a CUDA-only, explicitly selected backend. It requires the
`fastvideo-kernel` package and currently supports non-causal self-attention
with equal query and key/value sequence lengths. Unsupported shapes, masks,
dtypes, sequence-parallel execution, or kernel failures fall back to
`TORCH_SDPA` and emit a warning with the reason.
## Enable the backend
For online serving, select the backend with the existing attention backend
flag. Use `--fastvideo-vsa-topk` to set the number of key/value blocks retained
for every query block:
```bash
vllm-omni serve <model> \
--diffusion-attention-backend FASTVIDEO_VSA \
--fastvideo-vsa-topk 64
```
The backwards-compatible environment variable selects the backend with the
default `topk=64`:
```bash
export DIFFUSION_ATTENTION_BACKEND=FASTVIDEO_VSA
vllm-omni serve <model>
```
To tune top-k, pass the CLI backend and top-k flags together as shown above,
or use the equivalent structured configuration:
```bash
vllm-omni serve <model> \
--diffusion-attention-config \
'{"default":{"backend":"FASTVIDEO_VSA","fastvideo_vsa_topk":64}}'
```
Do not combine `--diffusion-attention-backend` with an explicit
`diffusion_attention_config.default.backend`. The top-k value must be positive
and is valid only when the default backend is `FASTVIDEO_VSA`.
For a deploy YAML stage, use either the shorthand fields:
```yaml
stages:
- stage_id: 0
diffusion_attention_backend: FASTVIDEO_VSA
fastvideo_vsa_topk: 64
```
or the structured configuration:
```yaml
stages:
- stage_id: 0
diffusion_attention_config:
default:
backend: FASTVIDEO_VSA
fastvideo_vsa_topk: 64
```
## Choose top-k
At runtime the backend logs the sequence shape and derived block count:
```text
FASTVIDEO_VSA routing: seq_len=27280, dit_seq_shape=(31, 22, 40),
block_size=(4, 8, 8), num_blocks=120, topk=64, keep_ratio=53.3%,
checkpoint_mode=native, route=VSA
```
Use `num_blocks` as the upper bound when tuning top-k. Top-k is the number of
key/value blocks retained for each query block, not the number of tokens or
the total number of blocks processed by the layer.
- A smaller top-k increases sparsity and may reduce attention computation, but
it can remove relevant blocks and reduce visual quality or temporal
consistency. Routing and padding overhead can also make a smaller value
slower for some shapes.
- A larger top-k retains more context and generally approaches dense-attention
quality, but reduces the potential speedup and increases memory traffic.
- `topk > num_blocks` is invalid for the runtime shape and falls back to SDPA.
- For a native checkpoint, `topk == num_blocks` routes to SDPA because scoring
every block provides no sparsity benefit.
- For a FastVideo DMD checkpoint, `topk == num_blocks` stays on the VSA
all-block path so the checkpoint keeps its trained compensation semantics.
There is no universally optimal value. Resolution, frame count, GPU, kernel
version, and checkpoint all affect both quality and latency. Start from the
logged `num_blocks`, test several keep ratios on the target workload, and
compare output quality against the same checkpoint running dense attention.
## Checkpoint behavior
The backend does not expose a user-selectable gate mode. Wan checkpoints that
contain learned `to_gate_compress` weights use that projection automatically.
When those weights are absent, the unused projection is removed and VSA uses
the sparse branch without learned compensation.
FastVideo DMD checkpoints use their fixed distilled timestep schedule. Native
Wan checkpoints keep their normal scheduler and inference-step configuration;
selecting VSA does not turn a native checkpoint into a distilled model.
## Verify routing and fallback
Check the startup and first-forward logs instead of assuming that selecting
the backend guarantees sparse execution:
- `route=VSA` means top-k block selection is active.
- `route=VSA_ALL_BLOCKS` means the FastVideo DMD checkpoint retained all
blocks through the VSA kernel.
- `route=SDPA` or `FASTVIDEO_VSA falling back to SDPA: ...` means dense SDPA
executed; the warning includes the reason.
The current kernel requires CUDA tensors in FP16 or BF16, 256-token blocks,
standard `head_size**-0.5` scaling, equal Q/K/V head counts, no attention mask,
and no active sequence-parallel context. NPU and XPU paths do not execute the
FastVideo VSA CUDA kernel.
docs/user_guide/diffusion/attention_backends/huggingface_hub.md
+32
View File
@@ -0,0 +1,32 @@
# Hugging Face Hub Attention Backends
Hub backends load the same published kernels commonly used with Hugging Face
Diffusers. They are useful when training and serving must use matching kernel
implementations to reduce numerical drift and sampling divergence.
| Backend | Kernel | Platform |
| --- | --- | --- |
| `FLASH_ATTN_HUB` | `kernels-community/flash-attn2` | Compatible CUDA GPUs |
| `FLASH_ATTN_3_HUB` | `kernels-community/flash-attn3` | Hopper sm_90 or newer |
## Installation
```bash
pip install kernels==0.14.1
```
If `kernels` is unavailable, vLLM-Omni warns and falls back to the
corresponding local FlashAttention implementation. On CUDA devices older than
Hopper, `FLASH_ATTN_3_HUB` falls back to `FLASH_ATTN_HUB`.
## Usage
```bash
export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN_HUB
# Equivalent CLI selection
vllm-omni serve <model> --diffusion-attention-backend FLASH_ATTN_HUB
```
For per-role kernel alignment, configure the Hub backend on only the relevant
role; see the [attention backend overview](../attention_backends.md#per-role-configuration).
docs/user_guide/diffusion/attention_backends/rainfusion.md
+80
View File
@@ -0,0 +1,80 @@
# RainFusion Attention
`RAINFUSION_ATTN` runs MindIE-SD RainFusion (`rf_v2`) block-sparse video
attention on Ascend NPU. It pools 128-token key blocks, ranks them per query
block, and attends to the highest-scoring blocks after arranging video tokens
in `(t, h, w)` order.
Only the video segment is sparse. Prefix rows such as text, visual conditions,
and audio, plus first-frame blocks, remain dense. Unsupported calls—including
warmup steps, skipped layers, missing video geometry, and video segments below
32 blocks—delegate to `FLASH_ATTN`, so compatible models can select RainFusion
globally.
## Configuration
| Key | Valid values | Meaning |
| --- | --- | --- |
| `sparsity` | finite, `[0, 1]` | Nominal dropped-key-block fraction; default `0.8`; `0` disables sparsity |
| `start_step` | integer, `>= 0` | Number of early denoise steps kept dense |
| `end_step` | integer, `>= 0` | Number of final denoise steps kept dense (tail fallback) |
| `precision` | `"bf16"`, `"fp8"`, `"mix"` | Kernel precision mode; default `"bf16"`. Requires MindIE-SD with `sparse_attention(precision=...)` |
| `skip_layers` | selector such as `"0-3,38"` | DiT blocks kept dense |
```bash
vllm-omni serve MiniMaxAI/MiniMax-H3 \
--diffusion-attention-config '{"default":{"backend":"RAINFUSION_ATTN",\
"block_sparse":{"sparsity":0.8,"start_step":0}}}'
```
```python
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, BlockSparseSpec
config = AttentionConfig(
default=AttentionSpec(
backend="RAINFUSION_ATTN",
block_sparse=BlockSparseSpec(
sparsity=0.8,
start_step=0,
skip_layers="0-1",
),
),
)
```
Tune `start_step`, then `sparsity`, then `skip_layers`. Increasing the dense
early-step window is usually the cheapest way to recover global structure;
use layer exclusions only after a same-seed dense comparison identifies
sensitive blocks.
## Requirements and compatibility
RainFusion requires Ascend NPU and `mindiesd`. Selecting it on another
platform raises. It is incompatible with ring sequence parallelism because
the kernel needs the complete key sequence for ranking; use Ulysses sequence
parallelism with `ring_degree=1`.
The `precision` knob requires a MindIE-SD release whose `sparse_attention`
accepts `precision=`; older releases accept it through `**kwargs` but
silently ignore it and run the BF16 path. `RAINFUSION_ATTN` raises at runtime
when a non-`bf16` precision is requested without that support, so pin a
compatible MindIE-SD release when using `precision="mix"` or `"fp8"`.
## Geometry handling
RainFusion handles arbitrary video grids by rearranging video spatially and,
when necessary, promoting a real-video suffix to the always-kept prefix. It
does not pad because `rf_v2` does not consume a padding attention mask.
For MiniMax-H3 at 1344x768, grid `(62, 24, 42)` produces 62,496 video rows.
The implementation promotes 2,976 rows and leaves 59,520 sparse rows
(`465 x 128`) so the mask and kernel tiling remain aligned.
Spatial grids aligned to 8x8 are still preferable. A latent height or width
not divisible by 8 creates an always-kept suffix and reduces realized
sparsity. At the image level, that means width and height divisible by 256.
Protected video tails require a compatible MindIE-SD release.
For common configuration and selector behavior, see the
[attention backend overview](../attention_backends.md) and the
[backend selection design](../../../design/feature/attention_backend_selection.md).
docs/user_guide/diffusion/attention_backends/sage.md
+63
View File
@@ -0,0 +1,63 @@
# SageAttention
SageAttention backends provide lossy low-precision attention for diffusion
models. Validate output quality against `TORCH_SDPA` at the same seed before
using either backend in production.
## `SAGE_ATTN`
`SAGE_ATTN` uses SageAttention 2.2 with INT8-quantized attention and FP16
accumulation.
### Installation
Install SageAttention into the same environment as vLLM-Omni:
```bash
git clone https://github.com/thu-ml/SageAttention.git
cd SageAttention
export EXT_PARALLEL=4 NVCC_APPEND_FLAGS="--threads 8" MAX_JOBS=32
pip install . --no-build-isolation
```
Verify the installation:
```bash
python -c "import sageattention; print(sageattention.__file__)"
```
Select it globally:
```bash
vllm-omni serve <model> --diffusion-attention-backend SAGE_ATTN
```
## `SAGE_ATTN_3`
`SAGE_ATTN_3` uses the SageAttention3 Blackwell implementation.
### SageAttention3 installation
```bash
git clone https://github.com/thu-ml/SageAttention.git
cd SageAttention/sageattention3_blackwell
python setup.py install
```
Verify the installation:
```bash
python -c "import sageattn3; print(sageattn3.__file__)"
```
```bash
vllm-omni serve <model> --diffusion-attention-backend SAGE_ATTN_3
```
`SAGE_ATTN_3` requires CUDA, an importable `sageattn3`, and a Blackwell-class
GPU. Its kernel assumes the query-head count equals the key/value-head count.
GQA and MQA diffusion calls therefore fall back to PyTorch SDPA for
correctness.
For common configuration and platform routing, see the
[attention backend overview](../attention_backends.md).
docs/user_guide/diffusion/attention_backends/trtllm.md
+103
View File
@@ -0,0 +1,103 @@
# TRTLLM Attention
`TRTLLM_ATTN` runs FlashInfer's trtllm-gen FMHA kernels. It supports dense
BF16 attention plus optional Skip-Softmax sparsity or SAGE quantization.
For common selection and per-role configuration, see the
[attention backend overview](../attention_backends.md).
## Requirements
Selecting `TRTLLM_ATTN` requires:
- datacenter Blackwell, sm_100 or sm_103;
- `head_dim=128`;
- FlashInfer; and
- a model-declared packed or mask-free compatible path.
It is not supported on workstation Blackwell (sm_120/sm_121). An incompatible
explicit selection raises instead of silently changing the backend.
## Dense mode
Select `TRTLLM_ATTN` without `skip_softmax` or `quant` to run dense BF16:
```bash
vllm-omni serve <model> --diffusion-attention-backend TRTLLM_ATTN
```
## Skip-Softmax
Skip-Softmax trades some fidelity for speed by skipping low-contribution
attention work. See the [feature design](../../../design/feature/skip_softmax.md)
for the algorithm and implementation contract.
| Key | Valid values | Meaning |
| --- | --- | --- |
| `target_sparsity` | finite, `[0, 1]` | Point on a model-calibrated curve |
| `threshold` | finite, `>= 0` | Direct threshold; mutually exclusive with `target_sparsity` |
| `disabled_until_timestep` | finite, `[0, 1]` | Keep early high-noise steps dense until normalized `t <= D` |
```bash
vllm-omni serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--diffusion-attention-config '{"default":{"backend":"TRTLLM_ATTN",\
"skip_softmax":{"target_sparsity":0.65,"disabled_until_timestep":0.86}}}'
```
```python
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, SkipSoftmaxSpec
config = AttentionConfig(
default=AttentionSpec(
backend="TRTLLM_ATTN",
skip_softmax=SkipSoftmaxSpec(
target_sparsity=0.65,
disabled_until_timestep=0.86,
),
),
)
```
Start with `target_sparsity=0.65` and `disabled_until_timestep=0.86`. Increase
sparsity only after comparing quality against dense output at the same seed.
The benefit grows with sequence length because only attention is accelerated.
## SAGE quantization
TRTLLM SAGE quantizes Q/K per block and V per channel so both attention
matrix multiplications use low precision.
| Key | Valid values | Meaning |
| --- | --- | --- |
| `dtype_qk` | `int8`, `fp8_e4m3` | Q/K quantization dtype; absent means dense |
| `q_block_size` | `1`, `4`, `16` | Q scale block size; default `1` |
| `k_block_size` | `1`, `4`, `16` | K scale block size; default `16` |
V is always quantized per channel to FP8 E4M3 and K-smoothing is internal.
Only block sizes with compiled kernels are accepted.
```bash
vllm-omni serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--diffusion-attention-config '{"default":{"backend":"TRTLLM_ATTN",\
"quant":{"dtype_qk":"fp8_e4m3","q_block_size":1,"k_block_size":16}}}'
```
```python
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, AttnQuantSpec
config = AttentionConfig(
default=AttentionSpec(
backend="TRTLLM_ATTN",
quant=AttnQuantSpec(
dtype_qk="fp8_e4m3",
q_block_size=1,
k_block_size=16,
),
),
)
```
This path requires FlashInfer 0.6.16rc1 or newer. FP8 Q/K kernels are
available on sm_100 and sm_103; INT8 Q/K kernels are compiled for sm_100 only.
The shared `AttnQuantSpec` is backend typed: values intended for
`FLASHINFER_ATTN` are rejected here and vice versa.
docs/user_guide/diffusion/cpu_offload.md
+59 -374
View File
@@ -1,389 +1,74 @@
# CPU Offloading for Diffusion Models
## Overview
vLLM-Omni provides three CPU-offload strategies for diffusion models. Choose
the coarsest strategy that meets your memory target: finer-grained streaming
saves more device memory but adds more transfer and setup overhead.
vLLM-Omni provides two offloading strategies to reduce GPU memory usage for diffusion models:
For the shared factory, discovery, and lifecycle contract, see the
[CPU Offloading design](../../design/feature/offloader/README.md).
1. **Model-level (Sequential) Offloading**: Mutual exclusion between DiT model and encoder - only one is on GPU at a time.
2. **Layerwise (Blockwise) Offloading**: Keeps only one transformer block on GPU at a time with compute-memory overlap.
## Choose a strategy
Both strategies use pinned memory for faster CPU-GPU transfers. The strategies are **mutually exclusive** for now - if both are enabled, layerwise takes priority.
| Strategy | Device residency | Parallel scope | Guide |
| --- | --- | --- | --- |
| Model-level (sequential) | One pipeline component group at a time | Single device | [Model-Level Offloading](offloader/cpu_offload.md) |
| Layerwise (blockwise) | One transformer block, with next-block prefetch | Single device | [Layerwise Offloading](offloader/layerwise_offload.md) |
| Distributed layerwise | Fixed two-block device buffer; optional host-weight sharding and AllGather | Multiple GPU/NPU ranks | [Distributed Layerwise Offloading](offloader/distributed_layerwise_offload.md) |
All strategies use pinned host memory for faster transfers where applicable.
Configuration priority is:
1. Distributed layerwise offloading.
2. Layerwise offloading.
3. Model-level offloading.
Treat the flags as mutually exclusive. If more than one is enabled, the
higher-priority strategy is selected.
## Quick selection
- Use [model-level offloading](offloader/cpu_offload.md) when swapping whole
encoders and DiTs is enough to fit and phase-boundary transfers are
acceptable.
- Use [layerwise offloading](offloader/layerwise_offload.md) for compute-heavy video
DiTs where block transfers can overlap computation.
- Use [distributed layerwise offloading](offloader/distributed_layerwise_offload.md)
when a multi-rank deployment also needs bounded device residency and,
optionally, sharded host weights.
## Supported models
| Architecture | Example models | DiT class | Model-level | Layerwise | Distributed layerwise | Layerwise block attributes |
| --- | --- | --- | :---: | :---: | :---: | --- |
| Flux2Pipeline | `black-forest-labs/FLUX.2-dev` | `Flux2Transformer2DModel` | yes | yes | — | `transformer_blocks`, `single_transformer_blocks` |
| LongCatImagePipeline | `meituan-longcat/LongCat-Image` | `LongCatImageTransformer2DModel` | — | yes | — | `transformer_blocks`, `single_transformer_blocks` |
| NextStep11Pipeline | `stepfun-ai/NextStep-1.1` | `NextStepModel` | — | yes | — | `layers` |
| OvisImagePipeline | `AIDC-AI/Ovis-Image-7B` | `OvisImageTransformer2DModel` | — | yes | — | `transformer` |
| QwenImagePipeline | `Qwen/Qwen-Image` | `QwenImageTransformer2DModel` | yes | yes | — | `transformer_blocks` |
| StableDiffusionXLPipeline | `stabilityai/stable-diffusion-xl-base-1.0` | `SDXLUNet2DConditionModel` | yes | yes | — | `down_blocks`, `up_blocks` |
| StableDiffusion3Pipeline | `stabilityai/stable-diffusion-3.5-medium` | `SD3Transformer2DModel` | — | yes | — | `transformer_blocks` |
| Wan22I2VPipeline | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | `WanTransformer3DModel` | yes | yes | — | `blocks` |
| Wan22Pipeline | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | `WanTransformer3DModel` | yes | yes | — | `blocks` |
| BagelPipeline | `ByteDance-Seed/BAGEL-7B-MoT` | `Qwen2MoTModel` | — | yes | — | `layers`, customized modules |
| Cosmos3OmniDiffusersPipeline | `nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super` | `Cosmos3VFMTransformer`, `Cosmos3LanguageModel` | yes | yes | yes | `layers`, `gen_layers` |
Model-level support requires discoverable DiT and encoder components.
Layerwise support requires transformer block topology. Distributed support
reuses that topology but still requires validation for the model, checkpoint,
and parallel configuration.
## Compatibility anchors
These headings preserve links to sections that moved into dedicated guides.
## Model-level (Sequential) Offloading
### How It Works
Model-level offloading implements mutual exclusion between DiT transformer and encoder modules using pre forward hooks:
- **When encoders run**: DiT transformer is offloaded to CPU
- **When DiT runs**: Encoders are offloaded to CPU, if more than one dit models, only one loaded on GPU, others get offloaded to CPU.
- **VAE**: Stays resident on GPU
Before each module's forward pass, the hook automatically moves it to GPU while offloading the other module group to CPU. Transfers use pinned memory for speed.
### Usage
**Python API:**
```python
from vllm_omni import Omni
m = Omni(model="Wan-AI/Wan2.2-T2V-A14B-Diffusers", enable_cpu_offload=True)
```
**CLI:**
```bash
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --enable-cpu-offload
```
### To Support a Model
Implement the `SupportsComponentDiscovery` protocol to declare which
submodules serve as pipeline components (used by offloading, HSDP
sharding, and other framework features):
```python
from typing import ClassVar
from vllm_omni.diffusion.models.interface import SupportsComponentDiscovery
class MyPipeline(nn.Module, SupportsComponentDiscovery):
_dit_modules: ClassVar[list[str]] = ["transformer"]
_encoder_modules: ClassVar[list[str]] = ["text_encoder", "vision_model"]
_vae_modules: ClassVar[list[str]] = ["vae"]
_resident_modules: ClassVar[list[str]] = [] # optional
def __init__(self):
super().__init__()
self.transformer = ... # DiT — stays on GPU during denoising
self.text_encoder = ... # Encoder — offloaded to CPU during denoising
self.vision_model = ... # Encoder — offloaded to CPU during denoising
self.vae = ... # VAE — always on GPU
```
- `_dit_modules`: attribute names of denoising submodules (kept on GPU
during the diffusion loop).
- `_encoder_modules`: attribute names of encoder/vision submodules
(offloaded to CPU during the diffusion loop).
- `_vae_modules`: attribute names of VAE(s) (always kept on GPU, not
part of the mutual exclusion hooks).
- `_resident_modules`: attribute names of small submodules that must
stay on GPU during layerwise offloading (e.g. embedders, connectors).
Optional — defaults to `[]`.
All attribute names support dotted paths for nested submodules
(e.g. `"pipe.transformer"`, `"bagel.time_embedder"`).
Both DiT and encoder lists are needed because the offload hooks use
mutual exclusion: when one group runs, the other moves to CPU.
### Limitations
- Cold start latency increases
- Adds overhead from CPU-GPU transfers between encoder and denoising phases
- Support single GPU only for now
### Component offloading for split models (e.g. Cosmos3)
Some models split their transformer into mutually-exclusive *components* that run
in different phases of a single forward pass rather than as separate pipeline
components -- e.g. Cosmos3's understanding (reasoner) component runs once per
generation while the generation (generator) component runs every denoising step.
Such models have no separate text encoder to swap against, so the transformer
owns a small model-local offload path and wraps each phase with
`with self._offload_context(name):`
```python
class Cosmos3VFMTransformer(nn.Module):
def forward(self, ...):
with self._offload_context("reasoner"):
... # understanding pass, runs once
with self._offload_context("generator"):
... # denoising pass, runs every step
```
Model-level offloading then keeps exactly one component GPU-resident at a time
(the other on CPU), reusing the same `SequentialOffloadHook` `.to()` movers. The
pipeline opts in by exposing `enable_omni_model_cpu_offload` (which drives the
transformer's `enable_model_cpu_offload` and pins the VAE). Layerwise offloading
works for these models too -- each component declares its own block container via
`_layerwise_offload_blocks_attrs`.
See [Model-Level Offloading](offloader/cpu_offload.md).
## Layerwise (Blockwise) Offloading
### How It Works
Layerwise offloading keeps only one transformer block on GPU at a time.
As each block completes, the next block is prefetched to GPU while the current block is freed. The pre and forward hooks utilized by layerwise offloading apply a separate CUDA stream (`copy_stream`) to overlap weight transfer with computation, and retain flattened tensors in pinned CPU memory for block parameters re-materialization. Encoders, VAE, and non-block DiT modules (embeddings, norms) always stay on GPU.
**Execution Flow:**
| Block | Pre-forward Hook | Forward | Post-forward Hook |
|-------|------------------|---------|-------------------|
| block-0 | Prefetch block-1 (async) | Compute block-0 | Free block-0 |
| block-1 | Prefetch block-2 (async) | Compute block-1 | Free block-1 |
| ... | ... | ... | ... |
| block-(n-1) | **Prefetch block-0** (async) | Compute block-(n-1) | Free block-(n-1) |
Each transformer block has a `LayerwiseOffloadHook` that prefetches the next block before forward and frees the current block after forward.
Layerwise offloading is primarily recommended for large **video generation models** where the compute cost per block is high enough to effectively overlap with memory prefetch operations. For example, Wan2.2 T2V and I2V pipelines.
### Usage
**Python API:**
```python
from vllm_omni import Omni
# Text-to-video
m = Omni(model="Wan-AI/Wan2.2-T2V-A14B-Diffusers", enable_layerwise_offload=True)
# Or image-to-video
m = Omni(model="Wan-AI/Wan2.2-I2V-A14B-Diffusers", enable_layerwise_offload=True)
```
**CLI:**
```bash
# Text-to-video
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --enable-layerwise-offload
# Or image-to-video
vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni --enable-layerwise-offload
```
### To Support a Model
Models must define the blocks attribute name for layerwise offloading:
```python
class WanTransformer3DModel(nn.Module):
_layerwise_offload_blocks_attrs = ["blocks"] # Attribute names containing transformer blocks
def __init__(self):
self.blocks = nn.ModuleList([...]) # Transformer blocks
```
For models with multiple block types:
```python
class Flux2Transformer2DModel(nn.Module):
_layerwise_offload_blocks_attrs = ["transformer_blocks", "single_transformer_blocks"]
```
### Limitations
- Cold start latency increases because offloaded components must be moved to CPU
during setup; layerwise offload may add extra weight consolidation and pinning
work.
- Performance depends on compute cost and H2D bandwidth as well
- Support single GPU only for now
See [Layerwise Offloading](offloader/layerwise_offload.md).
## Distributed Layerwise Offloading
### How It Works
Distributed layerwise offloading extends single-GPU layerwise offloading to
multi-device deployments. In the default mode, each rank stores only
**1/dp_size** of the model weights in host memory; full layer weights are
reconstructed at runtime via **AllGather** on a dedicated communication stream,
overlapped with computation via a fixed double-buffer scheme. The
`--dlo-no-use-allgather` mode instead streams standard-loader rank-local
weights independently and does not shard weights across ranks.
For the implementation design and compatibility matrix, see
[Distributed Layerwise Offload](../../design/feature/distributed_layerwise_offload.md).
**Key features:**
- **Weight sharding + AllGather**: each rank stores 1/dp_size of weights,
reconstructed per-layer via `all_gather_into_tensor`
- **Fixed double-buffer**: exactly 2 transformer blocks on each device at any
time, regardless of model size
- **DP multi-concurrency**: N concurrent requests processed in parallel
(AllGather only gathers weight shards, which are request-independent)
- **mmap weight loading**: weights loaded as mmap views pointing to shared OS
page cache, eliminating O(dp_size × model_size) RSS during model creation
- **Platform-agnostic**: works on NVIDIA GPU (CUDA/NCCL) and Ascend NPU
(CANN/HCCL) via vLLM-Omni's platform abstraction
**Execution Flow (per device):**
```
Compute Stream: [Layer N] [Layer N+1] [Layer N+2]
H2D Stream: [H2D Shard N+1] [H2D Shard N+2]
AllGather: [AG N+1] [AG N+2]
Slot Usage: Slot0: Layer N Slot1: Layer N+1
```
### Usage
**CLI:**
```bash
# 4× GPU/NPU with AllGather (recommended)
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--data-parallel-size 4
# Without DLO AllGather (each rank streams standard-loader rank-local weights)
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--data-parallel-size 4 \
--dlo-no-use-allgather
# With SP instead of DP (long sequences)
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--usp 4
```
**Python API:**
```python
from vllm_omni import Omni
m = Omni(
model="/path/to/model",
enable_distributed_layerwise_offload=True,
dlo_use_allgather=True, # default
)
```
### CLI Flags
| Flag | Description | Default |
|------|-------------|---------|
| `--enable-distributed-layerwise-offload` | Enable DLO | `false` |
| `--data-parallel-size N` | Number of DP ranks; DLO weight-sharding group when AllGather is enabled | `1` |
| `--dlo-use-allgather` | Shard + AllGather (saves CPU, requires concurrent requests) | `true` |
| `--dlo-no-use-allgather` | Use standard-loader rank-local weights; no DLO sharding or AllGather | `false` |
### How model weights are loaded (mmap path)
When DLO + AllGather is active, the offloader:
1. **Saves non-persistent buffers** (e.g. RoPE `inv_freq`, timestep `freqs`)
from the normally-created transformer
2. **Converts to meta device** via `to_empty(device="meta")`, releasing random
initialization weights
3. **Loads checkpoint weights as mmap views** via `safe_open().get_tensor()`,
which return views into the OS page cache (shared across all ranks, 0 RSS)
4. **Calls `post_load_weights()`** to apply model-specific dtype conversions
(e.g. Cosmos3's `time_embedder` → FP32)
5. **Restores non-persistent buffers** from saved copies
This approach requires **zero model-specific code changes** — no pipeline or
transformer modifications are needed.
### OffloadPlan (declarative topology metadata)
Models can optionally declare an `OffloadPlan` class variable to provide
topology metadata (block attribute names, submodules to offload) without
any offload-specific logic:
```python
from vllm_omni.diffusion.offloader import OffloadPlan
class MyPipeline(nn.Module):
_dit_modules = ["transformer"]
_offload_plan = OffloadPlan(
block_attrs={"transformer": ("blocks",)},
offload_submodules={"context_encoder": "layers"},
)
```
When not declared, the offloader falls back to `_layerwise_offload_blocks_attrs`
and heuristic attribute search. This is backward-compatible — existing models
work without any changes.
### DP Multi-concurrency
When `--data-parallel-size > 1` and AllGather is enabled, the scheduler batches
up to `dp_size` requests per denoise step. Each DP rank processes a different
request while AllGather synchronizes only weight shards (request-independent).
**Requirements for concurrent requests:**
- `num_inference_steps` must be specified explicitly (None is not allowed)
- All concurrent requests must have the same `num_inference_steps` value
- These constraints exist because AllGather is a collective that requires
every rank to participate at each denoise step
### Limitations
- Online quantization (FP8) is incompatible with mmap loading — falls back to
regular loading only when the model is outside the DLO+AllGather mmap path;
online quantization with DLO+AllGather is rejected. Use
`--dlo-no-use-allgather` or disable online quantization.
- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path because TP-aware
weight-loader callbacks are bypassed. The no-AllGather path retains the
standard TP loader and is experimental.
- HSDP + AllGather is rejected because it would double-shard parameters. HSDP
with no-AllGather is accepted at configuration level, but has limited
end-to-end validation.
- `num_inference_steps=None` is not allowed in DP multi-concurrency mode
**Module Discovery**
The offloader discovers pipeline components in two ways:
1. **Protocol-based** (preferred): If the pipeline implements
`SupportsComponentDiscovery`, its `_dit_modules`, `_encoder_modules`,
`_vae_modules`, and `_resident_modules` class variables are used
directly. All attribute names support dotted paths (e.g.
`"pipe.transformer"`, `"bagel.time_embedder"`) for nested submodules.
2. **Fallback attribute scan**: Otherwise, the offloader scans for
well-known attribute names:
- **DiT modules**: `transformer`, `transformer_2`, `dit`, `sr_dit`, `language_model`, `transformer_blocks`, `model`
- **Encoders**: `text_encoder`, `text_encoder_2`, `text_encoder_3`, `image_encoder`
- **VAE**: `vae`, `audio_vae`
**Hook System**
Both strategies use vLLM-Omni's hook registry system (`HookRegistry` and `ModelHook`) to register pre/post forward callbacks on modules, enabling automatic swapping without modifying model code.
**Backend Architecture**
```
OffloadBackend (base class)
├── ModelLevelOffloadBackend → uses SequentialOffloadHook (.to() swap)
│ (delegates to a pipeline's enable_omni_model_cpu_offload
│ for split models like Cosmos3)
├── LayerWiseOffloadBackend → uses LayerwiseOffloadHook
│ (single-GPU, full weights on host)
└── DistributedLayerwiseOffloadBackend → uses DistributedLayerwiseOffloadHook
(multi-GPU, sharded AllGather or rank-local mode)
```
Factory function `get_offload_backend()` selects the appropriate backend based on
configuration.
For split models, `ModelLevelOffloadBackend.enable()` detects a pipeline's
`enable_omni_model_cpu_offload` hook and delegates to it; Cosmos3 then swaps its
reasoner/generator components inside the model forward pass.
## Supported Models
| Architecture | Example Models | DiT Class | Model-Level Offload | Layerwise Offload | Distributed Layerwise Offload | Blocks Attrs (Layerwise specific) |
|--------------|----------------|-----------|---------------------|-------------------|-------------------------------|-----------------------------------|
| Flux2Pipeline | `black-forest-labs/FLUX.2-dev` | `Flux2Transformer2DModel` | ✓ | ✓ | - | `"transformer_blocks"`, `"single_transformer_blocks"` |
| LongCatImagePipeline | `meituan-longcat/LongCat-Image` | `LongCatImageTransformer2DModel` | - | ✓ | - | `"transformer_blocks"`, `"single_transformer_blocks"` |
| NextStep11Pipeline | `stepfun-ai/NextStep-1.1` | `NextStepModel` | - | ✓ | - | `"layers"` |
| OvisImagePipeline | `AIDC-AI/Ovis-Image-7B` | `OvisImageTransformer2DModel` | - | ✓ | - | `"transformer"` |
| QwenImagePipeline | `Qwen/Qwen-Image` | `QwenImageTransformer2DModel` | ✓ | ✓ | - | `"transformer_blocks"` |
| StableDiffusionXLPipeline | `stabilityai/stable-diffusion-xl-base-1.0` | `SDXLUNet2DConditionModel` | ✓ | ✓ | - | `"down_blocks"`, `"up_blocks"` |
| StableDiffusion3Pipeline | `stabilityai/stable-diffusion-3.5-medium` | `SD3Transformer2DModel` | - | ✓ | - | `"transformer_blocks"` |
| Wan22I2VPipeline | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | `WanTransformer3DModel` | ✓ | ✓ | - | `"blocks"` |
| Wan22Pipeline | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | `WanTransformer3DModel` | ✓ | ✓ | - | `"blocks"` |
| SoulXSingerPipeline / SoulXSingerSVCPipeline | `Soul-AILab/SoulX-Singer` | `DiffLlama` (`cfm_decoder.model.diff_estimator`) | ✓ | ✓ | - | `"layers"` |
| BagelPipeline | `ByteDance-Seed/BAGEL-7B-MoT` | `Qwen2MoTModel` | - | ✓ | - | `"layers"`, `"customized modules"` |
| Cosmos3OmniDiffusersPipeline | `nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super` | `Cosmos3VFMTransformer`, `Cosmos3LanguageModel` | ✓ | ✓ | ✓ | `"layers"`, `"gen_layers"` |
**Notes:**
- Model-Level Offloading is expected to be supported by all common diffusion models (DiT and encoders) naturally
- Layerwise Offloading requires DiT class to define `_layerwise_offload_blocks_attrs` pointing to transformer blocks
- Distributed Layerwise Offloading reuses the layerwise block topology, but the
model and parallelism combination still needs validation. AllGather mode
additionally requires a compatible checkpoint/mmap layout; no-AllGather mode
retains the standard loader's rank-local tensors. See the
[DLO design](../../design/feature/distributed_layerwise_offload.md) and
[Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)
for usage examples.
See [Distributed Layerwise Offloading](offloader/distributed_layerwise_offload.md).
docs/user_guide/diffusion/execution_modes.md
+15 -9
View File
@@ -101,8 +101,16 @@ set `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` or configure
`--max-num-seqs >1`. See the
[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md)
for its validated configuration. Helios supports single-request step execution only: use
`--step-execution --max-num-seqs 1` for Helios. Consult the selected pipeline's
documentation and source for the latest support status.
`--step-execution --max-num-seqs 1` for Helios. MiniMax H3 supports step-wise
continuous batching by packing co-batched requests into one sequence that keeps
a separate attention document per request; that layout needs a backend which
honors the packed `cu_seqlens` metadata, so run it with
`--diffusion-attention-backend FLASH_ATTN` (other backends stay correct but
fall back to one transformer forward per request). Batching H3 does not improve
its throughput — see the measured numbers in the
[MiniMax-H3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md)
— so keep `--max-num-seqs 1` unless you need step-level scheduling. Consult the
selected pipeline's documentation and source for the latest support status.
## Streaming Output
@@ -207,18 +215,16 @@ outputs = omni.generate(
)
```
## Stage Configuration
## Deploy Configuration
Use the equivalent engine arguments in a deployment YAML:
```yaml
stage_args:
stages:
- stage_id: 0
stage_type: diffusion
engine_args:
step_execution: false
max_num_seqs: 4
request_batch_max_wait_ms: 20
step_execution: false
max_num_seqs: 4
request_batch_max_wait_ms: 20
```
For step execution, set `step_execution: true` and remove
docs/user_guide/diffusion/lora.md
+35 -2
View File
@@ -12,15 +12,17 @@ LoRA adapters are lightweight, model-specific fine-tuning weights that can be ap
## LoRA Adapter Format
### PEFT (Parameter-Efficient Fine-Tuning) format (default)
A typical PEFT-format LoRA adapter directory structure:
```
```text
lora_adapter/
├── adapter_config.json
└── adapter_model.safetensors
```
The `adapter_config.json` file contains metadata about the LoRA adapter, including:
- `r`: LoRA rank
- `lora_alpha`: LoRA alpha scaling factor
- `target_modules`: List of module names to apply LoRA to
@@ -67,7 +69,7 @@ For distilled few-step LoRAs, pass `lora_backend="distill"` together with one or
For Qwen-Image and Wan pipelines, the distill backend calls `pipeline.load_lora_weights(...)` during worker initialization.
| Pipeline | Supported distilled LoRA repo | Notes |
|----------|--------------------------------|-------|
| ---------- | -------------------------------- | ------- |
| `QwenImagePipeline` | `lightx2v/Qwen-Image-2512-Lightning` | Used with Qwen-Image-2512 Lightning-style few-step inference. |
| `Wan22Pipeline` | `lightx2v/Wan2.1-Distill-Loras`, `lightx2v/Wan2.2-Distill-Loras` | Wan2.1 uses one LoRA file. For dual-transformer Wan2.2 MoE, pass high-noise then low-noise LoRA files. |
| `Wan22I2VPipeline` | `lightx2v/Wan2.2-Distill-Loras` | For dual-transformer Wan2.2 MoE, pass high-noise then low-noise LoRA files. |
@@ -240,6 +242,37 @@ Notes:
- Output quality and speed depend on the replacement checkpoints and sampling params you choose.
- If you only need to fuse distilled LoRAs into a Wan2.2 checkpoint at load time (without the full LightX2V convert + assemble pipeline), you can instead pass them directly via `--lora-backend distill --lora-path <high>.safetensors <low>.safetensors`. See the [Distill backend](#distill-backend-fuse-distilled-lora-at-init) section above.
## MiniMax-H3 adapter-declared schedules
MiniMax-H3 supports two few-step mechanisms that must not be conflated:
- **Checkpoint-pinned schedule**: a merged release writes `base_schedule` into
`model_index.json`. Requests must pass `num_inference_steps` as the interval
count (for example `4` for `[1.0, 0.7, 0.4, 0.15, 0.0]`).
- **Runtime Turbo LoRA**: LightX2V Turbo artifacts request five sigma points and
enforce `flow_shift=6`, `audio_flow_shift=3`.
- **Runtime native LoRA**: FlashGen-style artifacts declare
`key_format=minimax-h3-native` and embed `base_schedule` in safetensors
metadata. When active, the adapter schedule overrides the base checkpoint and
requests must use the interval-count contract (`num_inference_steps=4`).
Request-mode generation may omit the field and take the count from the adapter
schedule; step execution requires it explicitly, because the step scheduler
derives the total step count from the request before the adapter schedule is
known.
Native artifacts also declare `qkv_layout=grouped`. The H3 loader reorders fused
`qkv_proj` LoRA rows with the same `_reorder_grouped_qkv_to_qkv` helper used for
base-weight loading, then binds the packed Q/K/V slices through the legacy PEFT
manager without modifying `DiffusionLoRAManager`.
Both runtime adapters run with distributed layerwise offload in request-mode
generation, where the manager keeps the LoRA A/B buffers resident on the compute
device while DLO streams the base blocks. MiniMax-H3 step execution still
rejects DLO, so `--step-execution` cannot be combined with
`--enable-distributed-layerwise-offload`. Model-level CPU offload and standard
layerwise offload are rejected in every mode, because the dynamic LoRA tensors
are neither parameters nor registered buffers and therefore do not participate
in those weight lifecycles.
## See Also
docs/user_guide/diffusion/offloader/cpu_offload.md
+81
View File
@@ -0,0 +1,81 @@
# Model-Level Offloading
Model-level, or sequential, offloading keeps only the pipeline component group
currently executing on the accelerator. It is the simplest offload strategy
and is selected with `--enable-cpu-offload`.
## How it works
Pre-forward hooks enforce mutual exclusion between DiT and encoder modules:
- before an encoder runs, the DiT moves to CPU;
- before a DiT runs, encoders and other DiTs move to CPU; and
- VAE modules remain on the accelerator.
Pinned host memory reduces transfer overhead. Transfers occur at phase
boundaries, so cold-start and encoder-to-denoiser transitions become slower.
## Usage
```python
from vllm_omni import Omni
omni = Omni(
model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
enable_cpu_offload=True,
)
```
```bash
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--omni --enable-cpu-offload
```
## Model integration
Pipelines should implement `SupportsComponentDiscovery`:
```python
from typing import ClassVar
from vllm_omni.diffusion.models.interface import SupportsComponentDiscovery
class MyPipeline(nn.Module, SupportsComponentDiscovery):
_dit_modules: ClassVar[list[str]] = ["transformer"]
_encoder_modules: ClassVar[list[str]] = ["text_encoder", "vision_model"]
_vae_modules: ClassVar[list[str]] = ["vae"]
_resident_modules: ClassVar[list[str]] = []
```
All entries may be dotted paths. DiT and encoder lists are both required for
mutual exclusion. VAE modules are pinned but not swapped; resident modules are
small modules that must stay on the accelerator for layerwise paths.
## Split-model components
Some models, such as Cosmos3, split one transformer into mutually exclusive
components that run in different phases. The pipeline exposes
`enable_omni_model_cpu_offload`, and the backend delegates to the model-local
contexts:
```python
class Cosmos3VFMTransformer(nn.Module):
def forward(self, ...):
with self._offload_context("reasoner"):
...
with self._offload_context("generator"):
...
```
This preserves the same invariant—exactly one component is device resident—
while reusing sequential `.to()` movers.
## Limitations
- Single device only.
- Higher cold-start latency.
- Transfers between encoder and denoising phases add latency.
See the [model-level design](../../../design/feature/offloader/cpu_offload.md)
for lifecycle and extension invariants.
docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md
+285
View File
@@ -0,0 +1,285 @@
# Distributed Layerwise Offloading
Distributed layerwise offloading (DLO) extends block streaming to multi-device
deployments. With AllGather enabled, each rank stores roughly `1 / dp_size` of
the host weights and reconstructs each layer at runtime. Without AllGather,
each rank streams a complete block independently. Compatible TP1 deployments
can share checkpoint-backed host pages among processes on the same node;
otherwise DLO streams the ordinary loader's rank-local tensors.
See the [DLO feature design](../../../design/feature/offloader/distributed_layerwise_offload.md)
for the implementation contract and compatibility matrix.
## Execution model
DLO overlaps three operations with a fixed two-block device buffer:
```text
Compute stream: [Layer N] [Layer N+1] [Layer N+2]
H2D stream: [H2D shard N+1] [H2D shard N+2]
AllGather: [AG N+1] [AG N+2]
Slots: slot 0: Layer N slot 1: Layer N+1
```
AllGather communicates only request-independent weight shards, so data-
parallel ranks may process different requests concurrently.
## Usage
```bash
# Four ranks with sharded host weights and AllGather
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--data-parallel-size 4
# Standard-loader rank-local weights, without DLO AllGather
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--data-parallel-size 4 \
--dlo-no-use-allgather
# Sequence parallel deployment
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--usp 4
```
```python
from vllm_omni import Omni
omni = Omni(
model="/path/to/model",
enable_distributed_layerwise_offload=True,
dlo_use_allgather=True,
)
```
## Flags
| Flag | Meaning | Default |
| --- | --- | --- |
| `--enable-distributed-layerwise-offload` | Enable DLO | `false` |
| `--data-parallel-size N` | DP ranks and AllGather weight-sharding group | `1` |
| `--dlo-use-allgather` | Shard host weights and reconstruct with AllGather | `true` |
| `--dlo-no-use-allgather` | Stream complete rank-local blocks without a DLO weight collective | `false` |
| `--dlo-resident-layers N` | Keep N leading main-DiT blocks on device; requires no-AllGather and model-declared resident paths | `0` |
| `--host-weight-runtime-mode {disabled,preferred,required}` | HWR policy: no interaction, populate on a miss, or require an exact hit | `disabled` |
| `--host-weight-runtime-root PATH` | Writable node-local HWR store shared by workers in one storage domain; required for `preferred` and `required` | unset |
| `--dlo-host-registration-limit-gib N` | Optional per-worker ceiling for registering an HWR mmap; zero adds no ceiling | `0` |
## Host-weight loading
The diffusion loader chooses host storage before DLO is enabled. It first
attempts to build a complete, validated direct-checkpoint mmap plan. If names,
coverage, shape, dtype, topology, or loader-callback compatibility cannot be
proven, it runs the ordinary model loader instead. DLO consumes that result and
does not make a second checkpoint-compatibility decision.
The shared-mmap optimization in this phase is supported only with TP1. TP
greater than one falls back before model mutation to ordinary TP-aware loading.
DLO may still consume those TP-local tensors, but this is a compatibility path:
it does not share checkpoint-backed runtime weights across DP replicas and
provides no shared-mmap host-memory guarantee.
The mmap plan skips only dedicated DiT weight sources. Other component sources,
such as a text encoder loaded through the shared diffusion loader, continue to
use their ordinary component loader. A checkpoint source that mixes DiT and
non-DiT weights falls back completely rather than leaving an unplanned
component uninitialized.
With direct checkpoint mmap, the loader:
1. saves non-persistent buffers such as RoPE frequencies;
2. moves the normally created transformer to the meta device;
3. loads checkpoint tensors as mmap views backed by the shared OS page cache;
4. applies any loader-owned bounded layout adapters while packing blocks;
5. restores saved non-persistent buffers; and
6. preserves `post_load_weights()` and `validate_loaded_weights()` lifecycle
hooks.
For AllGather with a group larger than one, each process copies only its
persistent shard and then releases the source mapping. For no-AllGather, each
process keeps the mapping open and packs complete blocks through two bounded
pinned staging slots. Processes mapping the same files on one node share the
immutable pages; no-AllGather still performs a complete-block H2D copy in each
process.
When the effective DLO group size is one, `dlo_use_allgather=True` does not
perform a collective and uses the same rank-local transfer behavior.
### Final-layout Host Weight Runtime
HWR is an opt-in startup optimization for models that declare the final-layout
BF16 restore contract. Use it only with no-AllGather DLO:
The validated BF16 model contracts currently cover MiniMax H3 and
`black-forest-labs/FLUX.2-klein-4B`. FLUX.2-klein-9B shares the same model
class but has not been validated against this contract. FLUX.2-dev, online
FP8, HSDP, LoRA/adapted weights, and non-default load formats remain outside
the validated scope.
```bash
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--dlo-no-use-allgather \
--host-weight-runtime-mode preferred \
--host-weight-runtime-root /var/cache/vllm-omni/hwr
```
#### Choosing a mode
| Mode | Exact local artifact hit | Miss or recoverable artifact/store problem | Intended use |
| --- | --- | --- | --- |
| `disabled` | HWR is not consulted | Use the existing checkpoint-mmap or ordinary-loader path | Default compatibility path |
| `preferred` | Restore the final-layout artifact | Load canonically, serve with those tensors, and attempt to publish an artifact for the next startup | Normal deployment and store population |
| `required` | Restore the final-layout artifact | Fail startup without canonical DiT fallback or publication | Enforce a pre-populated store in controlled rollouts or CI |
Both enabled modes still fail on non-retryable configuration, identity, or
compatibility errors. `preferred` is a fallback policy for a cache miss or a
recoverable cache problem; it does not hide an invalid deployment.
#### Populating a store for `required` mode
`required` is deliberately consume-only. PR2 does not include a separate
prewarm command, so populate each node-local storage domain with one matching
`preferred` producer cohort:
1. Choose a persistent, writable root visible to every diffusion worker in the
storage domain. Do not use a process-private temporary directory.
2. Start the deployment with `preferred`, using the exact model revision,
dtype, TP size, SP configuration, and other weight-layout settings intended
for serving. No inference request is needed; publication happens during
model startup. Wait for the engine to become healthy, then shut it down
cleanly.
3. Restart with the same arguments and root, changing only the mode to
`required`. A successful startup proves that every worker acquired a valid
artifact; a miss, corrupt artifact, or incompatible identity fails startup.
4. Repeat the population start on every node or storage domain because this
store is node-local.
For example:
```bash
# First startup: canonically load and populate the exact artifacts.
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--dlo-no-use-allgather \
--host-weight-runtime-mode preferred \
--host-weight-runtime-root /var/cache/vllm-omni/hwr
# After a healthy startup and clean shutdown, enforce cache hits.
vllm serve /path/to/model --omni \
--enable-distributed-layerwise-offload \
--dlo-no-use-allgather \
--host-weight-runtime-mode required \
--host-weight-runtime-root /var/cache/vllm-omni/hwr
```
Include the same TP and SP layout flags in both commands. DP rank and DP size
are excluded from artifact identity, so the population and serving DP sizes may
differ and equivalent DP replicas share artifacts. TP rank is included, so a
TP-N deployment normally needs N rank-specific artifacts in each storage
domain; launching the matching TP cohort creates that set. If the model
revision or layout changes, run `preferred` again for the new identity before
returning to `required`.
On a cold start, the canonical loader remains authoritative and publishes a
validated final-layout artifact for later workers. A warm start restores the
DiT final tensors without ordinary DiT materialization. DLO then attempts to
register the immutable mapping for direct H2D; if registration is unavailable,
it streams through the same two bounded host staging slots. The artifact
identity includes the TP rank/size and SP layout, so TP1, TP2 rank-local shards,
and distinct SP layouts do not alias one another. Publication failure remains
separate from a `preferred` serving startup; the next `required` startup
provides the explicit artifact-availability check.
For local canonical checkpoints, the first eligible worker may hash source
shards to establish immutable identity. HWR caches those digests in the same
node-local storage domain and validates file metadata before reuse, so later
workers normally avoid repeating that read. Cold BF16 publication also hashes
ordered payloads as they are written and overlaps payload durability work with
later shards; this changes startup work only, not artifact contents or runtime
H2D behavior.
#### Registered direct H2D
Registration is attempted automatically only for an eligible warm HWR hit when
`pin_cpu_memory` is enabled. A successful path is:
```text
shared read-only HWR mmap -> existing rotating HBM block buffer -> GPU kernel
```
It removes the recurrent mmap-to-private-staging CPU copy and does not allocate
the two host staging slots. It does not reduce the H2D payload or make GPU
kernels access host memory directly.
CUDA requires read-only host-registration capability for the immutable HWR
mapping. Unsupported capability, a positive registration limit smaller than
the complete page-aligned mapping, or a safely rolled-back registration error
falls back to bounded staging. Programmatic configurations can set
`pin_cpu_memory=False` to disable registration explicitly. A successful
registration locks the complete mapped range in host memory for that worker's
lifetime, so use
`--dlo-host-registration-limit-gib` when an operator-enforced ceiling is
required.
Shutdown drains H2D work, releases hook/source references, unregisters every
range, and only then closes the HWR lease. AllGather and direct checkpoint mmap
retain their existing transfer paths.
When HWR is disabled, DLO is disabled, or AllGather is enabled, the loader does
not resolve HWR sources or construct its store.
## Declarative topology
Models may declare an `OffloadPlan` instead of embedding offload logic:
```python
from vllm_omni.diffusion.offloader import OffloadPlan
class MyPipeline(nn.Module):
_dit_modules = ["transformer"]
_offload_plan = OffloadPlan(
block_attrs={"transformer": ("blocks",)},
offload_submodules={"context_encoder": "layers"},
)
```
When no plan exists, discovery falls back to
`_layerwise_offload_blocks_attrs` and then heuristic attribute lookup.
## Data-parallel concurrency
With `data_parallel_size > 1` and AllGather enabled, the scheduler can process
up to `dp_size` requests per denoising step. Every concurrent request must set
the same explicit `num_inference_steps`; `None` is rejected because every rank
must enter each collective.
## Limitations
- Direct checkpoint mmap currently requires TP1. TP greater than one is
outside the Phase A shared-mmap support scope and falls back before model
mutation to the ordinary TP-aware loader. DLO can stream that runtime layout,
and eligible no-AllGather configurations may use HWR final-layout artifacts,
but direct checkpoint mmap provides no shared-mmap host-memory guarantee for
TP greater than one.
- HSDP plus AllGather is rejected to avoid double sharding. HSDP without
AllGather has limited end-to-end validation.
- Per-tensor online FP8 linears use the ordinary loader and can run with either
DLO transfer path. With AllGather, every rank temporarily materializes the
complete FP8 model in host memory before DLO retains only its shard. Other
online quantization methods require no-AllGather until their runtime layouts
are validated.
- Resident leading layers require `--dlo-no-use-allgather` and a model
`OffloadPlan` that declares eligible `resident_dit_paths`.
- DP concurrency requires an explicit, identical inference-step count.
Sharing quantized or otherwise unvalidated transformed layouts through a
normalized HWR producer is a follow-up design in
[RFC #6195](https://github.com/vllm-project/vllm-omni/issues/6195), not part of
the current BF16 final-layout path.
See the [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)
for an end-to-end example.
docs/user_guide/diffusion/offloader/layerwise_offload.md
+65
View File
@@ -0,0 +1,65 @@
# Layerwise Offloading
Layerwise, or blockwise, offloading keeps one transformer block on the
accelerator and prefetches the next block while the current block computes.
It is best suited to compute-heavy video DiTs whose block execution can hide
host-to-device transfers.
## Execution flow
Each block has a pre-forward and post-forward hook. Parameters are consolidated
in pinned host tensors and rematerialized for execution on a dedicated copy
stream.
| Block | Pre-forward hook | Forward | Post-forward hook |
| --- | --- | --- | --- |
| block 0 | Prefetch block 1 | Compute block 0 | Free block 0 |
| block 1 | Prefetch block 2 | Compute block 1 | Free block 1 |
| ... | ... | ... | ... |
| last block | Prefetch block 0 | Compute last block | Free last block |
Encoders, VAE modules, and non-block DiT modules such as embeddings and norms
remain device resident.
## Usage
```python
from vllm_omni import Omni
omni = Omni(
model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
enable_layerwise_offload=True,
)
```
```bash
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--omni --enable-layerwise-offload
```
## Model integration
Transformer classes declare containers of executable blocks:
```python
class WanTransformer3DModel(nn.Module):
_layerwise_offload_blocks_attrs = ["blocks"]
class Flux2Transformer2DModel(nn.Module):
_layerwise_offload_blocks_attrs = [
"transformer_blocks",
"single_transformer_blocks",
]
```
See the [layerwise design](../../../design/feature/offloader/layerwise_offload.md) for
the discovery and hook invariants. `OffloadPlan` is a separate declarative
topology path for distributed layerwise offload.
## Limitations
- Single device only.
- Setup consolidates and pins block parameters, increasing cold-start time.
- Performance depends on block compute time and host-to-device bandwidth;
lightweight blocks may not hide transfers.
docs/user_guide/diffusion/parallelism/hsdp.md
+38 -1
View File
@@ -64,6 +64,41 @@ omni = Omni(
)
```
### Combined with Online FP8 Quantization
HSDP can be combined with online FP8 quantization for diffusion models that
support both features. HSDP shards the quantized weights across GPUs to reduce
the per-GPU weight memory footprint.
```python
from vllm_omni import Omni
from vllm_omni.diffusion.data import DiffusionParallelConfig
omni = Omni(
model="<your-model>",
quantization="fp8",
parallel_config=DiffusionParallelConfig(
use_hsdp=True,
hsdp_shard_size=2,
),
)
```
For online serving:
```bash
vllm serve <your-model> \
--omni \
--quantization fp8 \
--use-hsdp \
--hsdp-shard-size 2
```
The model must support both HSDP and online FP8 quantization. HSDP remains
incompatible with Tensor Parallelism. See the
[FP8 Quantization Guide](../../quantization/fp8.md) for supported models and
hardware.
---
## Example Script
@@ -117,7 +152,8 @@ In `DiffusionParallelConfig`:
**Constraints:**
- `hsdp_replicate_size × hsdp_shard_size == world_size`
- HSDP cannot be used with Tensor Parallelism (`tensor_parallel_size` must be 1)
- HSDP cannot be used with Tensor Parallelism, ordinary Data Parallelism,
Pipeline Parallelism, or Expert Parallelism.
---
@@ -147,3 +183,4 @@ For detailed instructions on adding HSDP support to new models, see the [HSDP Co
1. ✅ **Enable HSDP** - Set `use_hsdp=True` and `hsdp_shard_size` to reduce per-GPU memory for large models
2. ✅ **Combine with SP** - Use together with `ulysses_degree` for video models requiring both memory reduction and sequence parallelism
3. ⚠️ **Incompatible with TP** - `tensor_parallel_size` must be 1 when HSDP is enabled
4. ✅ **Combine with online FP8** - Use `quantization="fp8"` or `--quantization fp8` to reduce the sharded weight memory footprint
docs/user_guide/diffusion/regional_compilation.md
+1 -1
View File
@@ -24,7 +24,7 @@ stages:
For an experimental whole-transformer compile scope, set
`--diffusion-compile-granularity full` or use
`diffusion_compile_granularity: full` in the stage configuration. Full scope may
`diffusion_compile_granularity: full` in the deploy configuration. Full scope may
still contain graph breaks; it does not force one graph. It is rejected when
HSDP, sequence parallelism, CPU offload, or layerwise offload is enabled. Use
regional scope with those features.
docs/user_guide/diffusion_features.md
+19 -7
View File
@@ -106,7 +106,10 @@ The following tables show which models support each feature:
> Notes:
> 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only. Split models like Cosmos3 (no separate text encoder) swap their reasoner/generator components for module-wise offload; see the [CPU Offload Guide](diffusion/cpu_offload.md).
> 1. CPU Offload has three strategies: model-level, layerwise, and distributed
> layerwise. The tables below show **layerwise support** only. Split models
> like Cosmos3 swap their reasoner/generator components for model-level
> offload; see the [CPU Offload Guide](diffusion/cpu_offload.md).
> 2. The **💾Quantization** column is collapsed for readability. See [Quantization](quantization/overview.md) for per-method and per-model support details.
### ImageGen
@@ -121,11 +124,11 @@ The following tables show which models support each feature:
| **FLUX.2-dev** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| **GLM-Image** | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Hidream-I1-Full** | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **HiDream-O1-Image** | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **HunyuanImage3** | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅* |
| **Krea 2** | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **LongCat-Image** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **LongCat-Image-Edit** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **MagiHuman** | ❌ | ❌ | ❌ | ❓ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **MammothModa2(T2I)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Nextstep_1(T2I)** | ❓ | ❓ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| **OmniGen2** | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
@@ -144,7 +147,7 @@ The following tables show which models support each feature:
> Notes:
> 1. Nextstep_1(T2I) does not support cache acceleration methods such as TeaCache or Cache-DiT.
> 2. `Tongyi-MAI/Z-Image-Turbo` and `SII-GAIR/daVinci-MagiHuman-Base-1080p` are distilled models with minimal NFEs; CFG-Parallel is not necessary.
> 2. `Tongyi-MAI/Z-Image-Turbo` is a distilled model with minimal NFEs; CFG-Parallel is not necessary.
> 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload swaps the nested UND reasoner and GEN generator pathways; layerwise offload remains available for blockwise GEN/UND offload.
> 4. Krea 2 currently supports single-GPU inference plus LoRA, Cache-DiT, HSDP, CPU/layerwise offload, and VAE-patch-parallel (decode). TP/SP/CFG-Parallel are not yet wired. The few-step distilled (Turbo) checkpoint uses `is_distilled=true` (fixed timestep shift `mu=1.15`); generate at 2048x2048 by default with `num_inference_steps≈8` and `guidance_scale=0`. The Raw checkpoint uses 1024x1024, `num_inference_steps=28`, and `guidance_scale=4.5`.
> 5. HunyuanImage3 supports step execution. Multi-request step execution requires `TORCH_SDPA`; see [Diffusion Execution Modes](diffusion/execution_modes.md#step-execution).
@@ -158,20 +161,29 @@ The following tables show which models support each feature:
| **Wan2.1-VACE** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **LTX-2** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **LTX-2.3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
| **LTX-2.5** | ❌ | ❓ (one-stage) | ❓ (Ulysses only) | ✅ (Full only) | ❓ | ❌ | ✅ | ✅ | ✅ (decode) | ❓ (FP8) | ❌ |
| **SANA-Video-2B T2V I2V** | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | ❌ |
| **Helios** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅* |
| **HunyuanVideo-1.5 T2V I2V** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
| **DreamID-Omni** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| **Cosmos3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
| **LongCat-Video-Avatar-1.5** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **MiniMax-H3** | ✅ (FL2VA) | ✅ | ✅ | ❌ | ✅ (DiT/TE) | ❌ | ✅ | ✅ | ✅ (tile) | ✅ (DiT) | ❌ |
| **SANA-WM** | ❌ | ❌ | ❌<sup>5</sup> | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
> Notes:
> 5. SANA-WM cannot support sequence parallelism: its bidirectional gated delta
> recurrence carries state across frames, so a rank cannot denoise a slice of
> the token sequence in isolation. Doing so would need a distributed scan or
> an all-gather before every GDN block. The remaining ❌ columns are simply
> unvalidated on this model, not known-broken.
> **Step execution note:** Helios supports single-request step execution only;
> use `max_num_seqs=1`.
**Frame Interpolation Support**
- **Supported**: Wan2.2 text-to-video, image-to-video, and TI2V pipelines
- **Not supported**: Wan2.1-VACE, LTX-2, LTX-2.3, Helios, HunyuanVideo-1.5, DreamID-Omni
- **Supported**: Wan2.2 text-to-video, image-to-video, and TI2V pipelines; SANA-Video-2B native text-to-video and image-to-video pipelines
- **Not supported**: Wan2.1-VACE, LTX-2, LTX-2.3, LTX-2.5, Helios, HunyuanVideo-1.5, DreamID-Omni, SANA-WM; SANA-Video-2B Diffusers-adapter text-to-video and image-to-video pipelines
### AudioGen
@@ -214,7 +226,7 @@ The following tables show which models support each feature:
3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible.
4. The CPU Offloading (Layerwise) row describes local layerwise offload.
Multi-device Distributed Layerwise Offload has a separate topology and
compatibility matrix in the [CPU Offloading guide](diffusion/cpu_offload.md#distributed-layerwise-offloading).
compatibility matrix in the [Distributed Layerwise Offloading guide](diffusion/offloader/distributed_layerwise_offload.md).
5. The compatibility matrix uses FP8 as the representative quantization method.
6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).
docs/user_guide/examples/offline_inference/glm_image.md
+2 -2
View File
@@ -43,7 +43,7 @@ if __name__ == "__main__":
),
],
)
outputs[0].request_output.images[0].save("output.png")
outputs[0].images[0].save("output.png")
```
## Image-to-Image (Image Editing)
@@ -72,7 +72,7 @@ if __name__ == "__main__":
),
],
)
outputs[0].request_output.images[0].save("output.png")
outputs[0].images[0].save("output.png")
```
## Generation Parameters
docs/user_guide/examples/offline_inference/image_to_video.md
+43 -3
View File
@@ -4,8 +4,8 @@ Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inf
This example demonstrates how to generate videos from images using Wan2.2,
LTX-2, and HunyuanVideo-1.5 Image-to-Video models with vLLM-Omni's
offline inference API.
LTX-2, HunyuanVideo-1.5, and SANA-Video Image-to-Video models with
vLLM-Omni's offline inference API.
## Local CLI Usage
@@ -66,9 +66,49 @@ python image_to_video.py \
See the [LTX-2 recipe](../../../../recipes/LTX/LTX-2.md) for all checkpoints,
pipeline selection, T2V, defaults, and advanced options.
### SANA-Video-2B
SANA checkpoints identify their upstream T2V pipeline in `model_index.json`.
Select the native I2V implementation explicitly:
```bash
python image_to_video.py \
--model Efficient-Large-Model/SANA-Video_2B_480p_diffusers \
--model-class-name SanaImageToVideoPipeline \
--image cherry_blossom.jpg \
--prompt "Cherry blossoms sway in the breeze as petals drift past the camera." \
--negative-prompt "blurry, low quality, temporal artifacts" \
--height 480 \
--width 832 \
--num-frames 81 \
--num-inference-steps 50 \
--guidance-scale 6.0 \
--extra-body '{"motion_score": 30}' \
--fps 16 \
--seed 42 \
--output sana_video_i2v_480p.mp4
```
For the 720p checkpoint, use
`Efficient-Large-Model/SANA-Video_2B_720p_diffusers` with
`--height 704 --width 1280`. The native I2V path supports both checkpoint
variants: the 480p checkpoint uses the Wan VAE, while the 720p checkpoint uses
the LTX-2 Video VAE.
For SANA-Video, 81 frames at 16 FPS is the standard checkpoint request
(approximately five seconds), not minute-scale long-video generation.
Minute-scale SANA generation requires the separate LongSANA/LongLive
autoregressive workflow, which this pipeline does not implement.
See the [SANA-Video recipe](../../../../recipes/NVIDIA/SANA-Video-2B.md) for
online serving, backend boundaries, and the tested hardware profile.
Key arguments:
- `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V, or LTX-2).
- `--model`: Model ID or local path (for example Wan I2V/TI2V, LTX-2, or
SANA-Video).
- `--model-class-name`: Explicit pipeline override. SANA-Video I2V requires
`SanaImageToVideoPipeline`.
- `--image`: Path to input image (required).
- `--prompt`: Text description of desired motion/animation.
- `--height/--width`: Output resolution (auto-calculated from image if not set).
docs/user_guide/examples/offline_inference/mimo_audio.md
+2 -2
View File
@@ -22,7 +22,7 @@ MiMo-Audio provides multiple task variants for audio understanding and generatio
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
See the [pipeline and deploy configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation for your hardware.
### Environment Variables
@@ -188,7 +188,7 @@ Note: This task uses hardcoded message lists in the script.
### Other
- If the model or stage config fails to load, check [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) for memory and GPU settings.
- If the model or deploy config fails to load, check the [pipeline and deploy configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) for memory and GPU settings.
- For errors when reading/writing WAV (e.g. unsupported format), ensure input files are standard WAV/MP3 and that `soundfile` is linked to a working libsndfile (see above).
## Notes
docs/user_guide/examples/offline_inference/qwen2_5_omni.md
+1 -1
View File
@@ -4,7 +4,7 @@ Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inf
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
See the [pipeline and deploy configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation for your hardware.
## Run examples
docs/user_guide/examples/offline_inference/qwen3_omni.md
+6 -5
View File
@@ -4,7 +4,8 @@ Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inf
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
Use `--deploy-config` for deployment overrides such as stage memory allocation.
See the [pipeline and deploy configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/).
## Run examples
@@ -73,8 +74,8 @@ For true stage-level concurrency -- where downstream stages (Talker, Code2Wav)
start **before** the upstream stage (Thinker) finishes -- use the async_chunk
example. This requires:
1. A stage config YAML with ``async_chunk: true`` (e.g.
``qwen3_omni_moe_async_chunk.yaml``).
1. A deploy YAML with ``async_chunk: true`` (for example, an overlay based on
``vllm_omni/deploy/qwen3_omni_moe.yaml``).
2. Hardware that matches the config (e.g. 2x H100 for the default 3-stage
config).
@@ -100,11 +101,11 @@ bash run_multiple_prompts_async_chunk.sh --max-in-flight 4
python end2end_async_chunk.py --query-type text --modalities text
```
#### Custom stage config
#### Custom deploy config
```bash
python end2end_async_chunk.py \
--query-type use_audio \
--stage-configs-path /path/to/your_async_chunk.yaml
--deploy-config /path/to/your_async_chunk.yaml
```
> **Note**: The synchronous ``end2end.py`` (using ``Omni``) is still the
docs/user_guide/examples/offline_inference/text_to_image.md
+3 -3
View File
@@ -55,7 +55,7 @@ if __name__ == "__main__":
omni = Omni(model="Qwen/Qwen-Image")
prompt = "a cup of coffee on the table"
outputs = omni.generate(prompt)
images = outputs[0].request_output.images
images = outputs[0].images
images[0].save("coffee.png")
```
@@ -179,7 +179,7 @@ if __name__ == "__main__":
]
outputs = omni.generate(prompts)
for i, output in enumerate(outputs):
output.request_output.images[0].save(f"{i}.jpg")
output.images[0].save(f"{i}.jpg")
```
!!! info
@@ -207,7 +207,7 @@ if __name__ == "__main__":
}
])
for i, output in enumerate(outputs):
output.request_output.images[0].save(f"{i}.jpg")
output.images[0].save(f"{i}.jpg")
```
You can also pass a negative prompt via the CLI argument `--negative-prompt`:
docs/user_guide/examples/offline_inference/text_to_speech.md
+3 -3
View File
@@ -69,7 +69,7 @@ python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
### Notes
- Output: 48 kHz mono WAV.
- Stage config: `vllm_omni/model_executor/stage_configs/voxcpm2.yaml` (default).
- Deploy config: `vllm_omni/deploy/voxcpm2.yaml` (default).
---
@@ -152,7 +152,7 @@ python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a streaming test." \
--streaming
```
Streaming requires `async_chunk: true` in the stage config.
Streaming requires `async_chunk: true` in the deploy config.
### Notes
- Output: 44.1 kHz mono WAV.
@@ -274,7 +274,7 @@ python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
--streaming \
--output-dir /tmp/out_stream
```
Streaming requires `async_chunk: true` in the stage config.
Streaming requires `async_chunk: true` in the deploy config.
### Batched decoding
The Code2Wav stage supports batched decoding through the SpeechTokenizer. Configure both stages with `max_num_seqs > 1` via `--stage-overrides` and pass multiple prompts via `--txt-prompts`:
docs/user_guide/examples/offline_inference/text_to_video.md
+27
View File
@@ -16,6 +16,8 @@ For backend selection and SageAttention usage, see the [Diffusion Attention Back
| `Lightricks/LTX-2` | 512x768 | 121 | 40 | video 3.0 / audio 7.0 | Model-dependent |
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | 480x832 | 121 | 50 | 6.0 | 1×A100 80GB |
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required |
| `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | 480x832 | 81 | 50 | 6.0 | BF16 DiT + FP32 Wan VAE |
| `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | 704x1280 | 81 | 50 | 6.0 | ~23.6 GiB |
## Local CLI Usage
@@ -122,6 +124,31 @@ python text_to_video.py \
--output quick_test.mp4
```
### SANA-Video-2B
```bash
python text_to_video.py \
--model Efficient-Large-Model/SANA-Video_2B_480p_diffusers \
--model-class-name SanaVideoPipeline \
--prompt "A cinematic tracking shot of a sailboat crossing the ocean at sunset." \
--height 480 \
--width 832 \
--num-frames 81 \
--num-inference-steps 50 \
--guidance-scale 6.0 \
--extra-body '{"motion_score": 30}' \
--fps 16 \
--output sana_video_480p.mp4
```
For the 720p checkpoint, switch the model to
`Efficient-Large-Model/SANA-Video_2B_720p_diffusers`, use
`--height 704 --width 1280`, and write to a different output path. SANA-Video
accepts `motion_score`, `clean_caption`, and `use_resolution_binning` through
`--extra-body`. For image-to-video, use the shared
`examples/offline_inference/image_to_video/image_to_video.py` example with
`--model-class-name SanaImageToVideoPipeline` and `--image <path>`.
## Key Arguments
### Common
docs/user_guide/examples/offline_inference/x_to_video_audio.md
-115
View File
@@ -1,115 +0,0 @@
# X-To-Video-Audio
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/x_to_video_audio>.
The `DreamID-Omni` pipeline generates short videos from text, image and video.
## Local CLI Usage
### Download the Model locally
Since DreamID-Omni combine multiple models, and without any config, so we need to download them locally.
```bash
python download_dreamid_omni.py --output-dir ./dreamid_omni
```
After download, the model directory will look like this:
```
dreamid_omni/
├── DreamID-Omni/
│ ├── dreamid_omni.safetensors
├── MMAudio/
│ ├── ext_weights/
│ │ ├── best_netG.pt
│ │ ├── v1-16.pth
├── Wan2.2-TI2V-5B/
│ ├── google/*
│ ├── models_t5_umt5-xxl-enc-bf16.pth
│ ├── Wan2.2_VAE.pth
│
├── model_index.json # create by download_dreamid_omni.py
```
### Run the Inference
```python
python x_to_video_audio.py \
--model /path/to/dreamid_omni \
--prompt "Two people walking together and singing happily" \
--image-path ./example0.png ./example1.png \
--audio-path ./example0.wav ./example1.wav \
--video-negative-prompt "jitter, bad hands, blur, distortion" \
--audio-negative-prompt "robotic, muffled, echo, distorted" \
--cfg-parallel-size 2 \
--num-inference-steps 45 \
--height 704 \
--width 1280 \
--output out_dreamid_omni_twoip.mp4
```
In the current test scenario (2 images + 2 audio inputs), the VRAM requirement is 72GB, regardless of whether cfg-parallel is enabled or disabled.
The VRAM usage can be reduced by enabling CPU offload via --enable-cpu-offload.
For multi-GPU memory reduction on the fused DreamID-Omni transformer, you can also enable HSDP:
```python
python x_to_video_audio.py \
--model /path/to/dreamid_omni \
--prompt "Two people walking together and singing happily" \
--image-path ./example0.png ./example1.png \
--audio-path ./example0.wav ./example1.wav \
--video-negative-prompt "jitter, bad hands, blur, distortion" \
--audio-negative-prompt "robotic, muffled, echo, distorted" \
--cfg-parallel-size 2 \
--num-inference-steps 45 \
--height 704 \
--width 1280 \
--use-hsdp \
--hsdp-shard-size 2 \
--output out_dreamid_omni_twoip.mp4
```
You could take reference images/audios from the test cases in the official repo: https://github.com/Guoxu1233/DreamID-Omni
For example, single IP ref resources can be found under https://github.com/Guoxu1233/DreamID-Omni/tree/main/test_case/oneip, you could download them correspondingly to your local and use them for testing.
```python
# Example usage for oneip, ref media from the official repo DreamID-Omni
python x_to_video_audio.py \
--model /path/to/dreamid_omni \
--prompt "<img1>: In the frame, a woman with black long hair is identified as <sub1>.\n**Overall Environment/Scene**: A lively open-kitchen café at night; stove flames flare, steam rises, and warm pendant lights swing slightly as staff move behind her. The shot is an upper-body close-up.\n**Main Characters/Subjects Appearance**: <sub1> is a young woman with thick dark wavy hair and a side part. She wears a fitted black top under a light apron, a thin gold chain necklace, and small stud earrings.\n**Main Characters/Subjects Actions**: <sub1> tastes the sauce with a spoon, then turns her face toward the camera while still holding the spoon, her expression shifting from focused to conflicted.\n<sub1> maintains eye contact, swallows as if choosing her words, and says, <S>I keep telling myself I’m fine,but some nights it feels like I’m just performing calm.<E>" \
--image-path 9.png \
--audio-path 9.wav \
--video-negative-prompt "jitter, bad hands, blur, distortion" \
--audio-negative-prompt "robotic, muffled, echo, distorted" \
--cfg-parallel-size 2 \
--num-inference-steps 45 \
--height 704 \
--width 1280 \
--output out_dreamid_omni_oneip.mp4
```
Key arguments:
- `--prompt`: text description (string).
- `--model`: path to the model local directory.
- `--height/--width`: output resolution (defaults 704 * 1024).
- `--image-path`: path to the input image list.
- `--audio-path`: path to the input audio list, indicate the timbre of the output video.
- `--cfg-parallel-size`: number of parallel cfg parallel (defaults 1).
- `--use-hsdp`: enable HSDP weight sharding for DreamID-Omni fused blocks.
- `--hsdp-shard-size`: number of GPUs used for HSDP sharding.
- `--hsdp-replicate-size`: number of HSDP replica groups.
- `--num-inference-steps`: number of denoising steps (defaults 45).
- `--video-negative-prompt`: negative prompt for video generation.
- `--audio-negative-prompt`: negative prompt for audio generation.
- `--enable-cpu-offload`: enable CPU offload (defaults False).
## Example materials
??? abstract "download_dreamid_omni.py"
``````py
--8<-- "examples/offline_inference/x_to_video_audio/download_dreamid_omni.py"
``````
??? abstract "x_to_video_audio.py"
``````py
--8<-- "examples/offline_inference/x_to_video_audio/x_to_video_audio.py"
``````
docs/user_guide/examples/online_serving/qwen3_omni.md
+13 -1
View File
@@ -28,6 +28,18 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--deploy-config /path/to/deploy_config_file
```
To serve **thinker-only** (text output, no talker / code2wav loaded) with Instruct
weights, pass the bundled thinker-only deploy YAML. The `pipeline:` key
`qwen3_omni_moe_thinker_only` overrides the HF `enable_audio_output` resolver:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--deploy-config vllm_omni/deploy/qwen3_omni_moe_thinking.yaml
```
Captioner / Thinking checkpoints (`enable_audio_output=false`) still auto-select
the same single-stage pipeline without `--deploy-config`.
### Launch individual stages (stage-based CLI)
Adopt the stage-based CLI architecture to independently instantiate execution processes per functional stage.
@@ -271,7 +283,7 @@ The script supports the following arguments:
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
```
If you have custom stage configs file:
If you have a custom deploy config file:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --deploy-config /path/to/deploy_config_file
```
docs/user_guide/examples/online_serving/text_to_audio.md
-193
View File
@@ -1,193 +0,0 @@
# Text-To-Audio
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/stable_audio>.
This example demonstrates how to deploy Stable Audio models for online text-to-audio generation using vLLM-Omni.
## Supported Models
| Model | Description |
|-------|-------------|
| `stabilityai/stable-audio-open-1.0` | Open-source audio generation, up to ~47 seconds, 44.1 kHz stereo |
## Start Server
### Basic Start
```bash
vllm serve stabilityai/stable-audio-open-1.0 \
--host 0.0.0.0 \
--port 8091 \
--gpu-memory-utilization 0.9 \
--trust-remote-code \
--enforce-eager \
--omni
```
## API Calls
### Method 1: Using curl
```bash
# Run all curl examples
bash curl_examples.sh
# Or execute directly
curl -X POST http://localhost:8091/v1/audio/generate \
-H "Content-Type: application/json" \
-d '{
"input": "The sound of a cat purring",
"audio_length": 10.0
}' --output cat.wav
```
### Method 2: Using Python Client
```bash
cd examples/online_serving/stable_audio
# Simple generation
python stable_audio_client.py \
--text "The sound of a cat purring"
# With custom duration
python stable_audio_client.py \
--text "A dog barking" \
--audio_length 5.0
# With all parameters
python stable_audio_client.py \
--text "Thunder and rain" \
--audio_length 15.0 \
--negative_prompt "Low quality" \
--guidance_scale 7.0 \
--num_inference_steps 100 \
--seed 42 \
--output thunder.wav
```
The Python client supports the following command-line arguments:
- `--api_url`: API endpoint URL (default: `http://localhost:8091/v1/audio/generate`)
- `--text`: Text prompt for audio generation (default: `"The sound of a cat purring"`)
- `--audio_length`: Audio length in seconds (default: `10.0`, max ~47s for `stable-audio-open-1.0`)
- `--audio_start`: Audio start time in seconds (default: `0.0`)
- `--negative_prompt`: Negative prompt for classifier-free guidance (default: `"Low quality"`)
- `--guidance_scale`: Guidance scale for diffusion (default: `7.0`)
- `--num_inference_steps`: Number of inference steps (default: `100`)
- `--seed`: Random seed for reproducibility (default: `None`)
- `--response_format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`
- `--output`: Output file path (default: `stable_audio_output.wav`)
### Method 3: Using Python httpx
```python
import httpx
response = httpx.post(
"http://localhost:8091/v1/audio/generate",
json={
"input": "The sound of ocean waves crashing on a beach",
"audio_length": 10.0,
"negative_prompt": "Low quality, distorted",
"guidance_scale": 7.0,
"num_inference_steps": 100,
},
timeout=300.0,
)
with open("ocean.wav", "wb") as f:
f.write(response.content)
```
## Request Format
### Simple Generation
```json
{
"input": "The sound of ocean waves"
}
```
### Generation with Parameters
```json
{
"input": "A piano playing a gentle melody",
"audio_length": 10.0,
"negative_prompt": "Low quality, distorted, noisy",
"guidance_scale": 8.0,
"num_inference_steps": 150,
"seed": 42,
"response_format": "wav"
}
```
## API Reference
### Endpoint
```
POST /v1/audio/generate
Content-Type: application/json
```
### Generation Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `input` | string | **required** | Text prompt describing the audio to generate |
| `model` | string | server's model | Model to use (optional, should match server if specified) |
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, opus |
| `speed` | float | 1.0 | Playback speed (0.25 - 4.0) |
| `audio_length` | float | null | Audio duration in seconds (max ~47s for `stable-audio-open-1.0`) |
| `audio_start` | float | 0.0 | Audio start time in seconds |
| `negative_prompt` | string | null | Text describing what to avoid in generation |
| `guidance_scale` | float | model default | Classifier-free guidance scale (higher = more adherence to prompt) |
| `num_inference_steps` | int | model default | Number of denoising steps (higher = better quality, slower) |
| `seed` | int | null | Random seed for reproducible generation |
### Response Format
Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/wav`).
## Tuning Tips
1. **Audio Length**: Keep under 47 seconds for `stable-audio-open-1.0`.
2. **Quality vs Speed**:
- 50 steps: Fast, decent quality (quick previews)
- 100 steps: Good balance (general purpose)
- 150+ steps: High quality, slower (final / critical audio)
3. **Guidance Scale**:
- Lower (3 - 5): More creative / varied output
- Default (7): Good balance
- Higher (10+): Strict adherence to the prompt
4. **Negative Prompts**: Use to avoid unwanted characteristics such as `"Low quality"`, `"distorted"`, `"noisy"`.
5. **Seeds**: Set a fixed seed to get deterministic, reproducible results.
## File Description
| File | Description |
|------|-------------|
| `curl_examples.sh` | Curl examples covering common use cases |
| `stable_audio_client.py` | Python client with full CLI argument support |
## Troubleshooting
1. **Audio generation model did not produce audio output**: Verify the server started successfully and the model loaded without errors.
2. **Connection refused**: Make sure the server is running on the correct port.
3. **Generation timeout**: Reduce `num_inference_steps` or `audio_length`, and check GPU memory with `nvidia-smi`.
4. **Out of memory**: Lower `--gpu-memory-utilization` or reduce `audio_length`.
5. **Audio quality issues**: Increase `num_inference_steps`, add a negative prompt, or raise `guidance_scale`.
## Example materials
??? abstract "stable_audio_client.py"
``````py
--8<-- "examples/online_serving/stable_audio/stable_audio_client.py"
``````
??? abstract "curl_examples.sh"
``````sh
--8<-- "examples/online_serving/stable_audio/curl_examples.sh"
``````
docs/user_guide/examples/online_serving/text_to_speech.md
+1 -1
View File
@@ -419,7 +419,7 @@ curl -X POST http://localhost:8091/v1/audio/speech \
"response_format": "pcm"
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
```
Raw PCM streaming requires `stream_format="audio"`, `response_format="pcm"`, and `async_chunk: true` on the stage config (default in `qwen3_tts.yaml`). `speed` is not supported when streaming.
Raw PCM streaming requires `stream_format="audio"`, `response_format="pcm"`, and `async_chunk: true` in the deploy config (default in `qwen3_tts.yaml`). `speed` is not supported when streaming.
### Streaming WebSocket
The `/v1/audio/speech/stream` endpoint accepts text incrementally and synthesizes the buffered text as one continuous request on `input.done`:
docs/user_guide/fault_injection_reliability_matrix.md
+1 -1
View File
@@ -65,7 +65,7 @@ More failure modes (for example, network jitter and network interruption) will b
## Source of Conclusions
The behaviors and conclusions above are summarized from current fault injection validation results on **`Qwen3-Omni`**, **`Wan2.2`**, **`HunyuanImage-3.0-Instruct`** (DiT-only, `/v1/images/generations`), and **`VoxCPM2`** (`/v1/audio/speech`). Automated coverage lives under `tests/dfx/reliability/` (`test_reliability_qwen3_omni.py`, `test_reliability_wan22.py`, `test_reliability_hunyuan_image.py`, `test_reliability_voxcpm2.py`) and runs weekly via `.buildkite/cuda/test-weekly.yml`.
The behaviors and conclusions above are summarized from current fault injection validation results on **`Qwen3-Omni`**, **`Wan2.2`**, and **`HunyuanImage-3.0-Instruct`** (DiT-only, `/v1/images/generations`). Automated coverage lives under `tests/dfx/reliability/` (`test_reliability_qwen3_omni.py`, `test_reliability_wan22.py`, `test_reliability_hunyuan_image.py`) and runs weekly via `.buildkite/cuda/test-weekly.yml`.
### Example: `SIGTERM` fault injection log (Qwen3-Omni)
docs/user_guide/feature_compatibility.md
+1 -1
View File
@@ -165,7 +165,7 @@ vllm serve Qwen/Qwen-Image --omni --port 8091 \
- **Tensor Parallelism — Text Encoder Not Sharded**: TP currently only shards the DiT blocks. Each TP rank retains a **full copy of the text encoder weights**, leading to significant GPU memory overhead proportional to TP degree. Tracked in [Issue #771](https://github.com/vllm-project/vllm-omni/issues/771).
- **CPU Offloading — Two Modes Are Mutually Exclusive**: Model-level offload (`enable_cpu_offload`) and layerwise offload (`enable_layerwise_offload`) cannot be used simultaneously. If both are set, layerwise takes priority and model-level is silently ignored.
- **CPU Offloading — Strategies Are Mutually Exclusive**: Distributed layerwise, layerwise, and model-level offload should not be enabled together. Selection priority is distributed layerwise, then layerwise, then model-level.
- **CPU Offloading — VAE stays on GPU**: Both offloading strategies keep the VAE on GPU at all times. For high-resolution generation, VAE decode can still cause OOM. Mitigate by combining with `vae_use_tiling=True` or VAE Patch Parallelism.
docs/user_guide/quantization/fp8.md
+56 -3
View File
@@ -80,20 +80,22 @@ warmup_quack_fp8([(14040, 2048, 6144), (14040, 2048, 2048)])
## Model Type Support
### Diffusion Model (Qwen-Image, Wan2.2)
### Diffusion Models
| Model | HF models | Online | Pre-calibrated | Recommendation | `ignored_layers` | Text-Encoder quantization |
|-------|-----------|:-------:|:------:|----------------|------------------|------------------|
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes | Yes | Skip sensitive image-stream MLPs when quality regresses | `img_mlp` | |
| Wan2.2 | Wan2.2 diffusion pipelines | Not validated | Not validated | Validate against BF16 before documenting as supported | TBD | |
| LTX-2 | `Lightricks/LTX-2`, `rootonchair/LTX-2-19b-distilled` | Yes | Not validated | Transformer only; use dynamic phase LoRA for ordinary two-stage | None | |
| LTX-2.3 | `diffusers/LTX-2.3-Diffusers`, `diffusers/LTX-2.3-Distilled-Diffusers` | Yes | Not validated | Transformer only; use dynamic phase LoRA for ordinary two-stage | None | |
| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Yes | Yes | All layers | None | ✅︎ |
| FLUX.1 | `black-forest-labs/FLUX.1-dev`, `black-forest-labs/FLUX.1-schnell` | Yes | Yes | All layers | None | |
| FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | Yes | Not validated | All layers | None | ✅︎ |
| FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B` | Yes | Yes | All layers | None | |
| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | Yes | Yes | All layers; use the Hunyuan stage config for multi-stage runs | None | |
| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | Yes | Yes | All layers; use the Hunyuan deploy config for multi-stage runs | None | |
| HunyuanVideo-1.5 | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `720p_t2v`, `480p_i2v` | Yes | Yes | All layers | None | |
| Cosmos3 | `nvidia/Cosmos3-Nano`, `nvidia/Cosmos3-Super` | Yes | Not validated | All layers | None | |
| MiniMax H3 | `MiniMaxAI/MiniMax-H3` | Yes | Not validated | DiT linears except patch, timestep, and final projections; not compatible with layerwise offload | None | |
| MiniMax-H3 | `MiniMaxAI/MiniMax-H3` (`FL2VA` / `Ref2VA`) | Yes | Not validated | `quantization="fp8"` quantizes eligible DiT and text-encoder linears; mixed-precision input/output heads stay FP32 | None | ✅︎ |
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -136,6 +138,39 @@ outputs = omni.generate(
)
```
### Global and per-component scope
A plain, global quantization configuration is passed unchanged to every
quantization-aware component constructed by a pipeline. This includes an
eligible encoder when that encoder is implemented with vLLM quantizable
layers; it does not rewrite arbitrary `torch.nn` modules. A structured
component map is the only way to narrow that scope. Pipeline integrations that
do not yet expose an encoder through the quantization factory remain DiT-only.
For a pipeline that exposes both a transformer and a quantization-aware text
encoder, the scope is:
| Configuration | Transformer | Text encoder | Components without supported quantizable layers |
|---------------|-------------|--------------|-------------------------------------------------|
| `quantization="fp8"` | FP8 | FP8 | checkpoint precision |
| `{"transformer": {"method": "fp8"}}` | FP8 | checkpoint precision | checkpoint precision |
| `{"text_encoder": {"method": "fp8"}}` | checkpoint precision | FP8 | checkpoint precision |
Use `quantization_config` for component-selective Python configuration. For
example, quantize only the text decoder:
```python
omni = Omni(
model="<your-model>",
quantization_config={"text_encoder": {"method": "fp8"}},
)
```
Component keys are runtime prefixes exposed by the pipeline integration; common
keys include `transformer` and `text_encoder`. Entries may be combined, and
`ignored_layers` can keep named eligible layers in checkpoint precision. Check
the model recipe for supported components and their runtime prefixes.
CLI:
```bash
@@ -144,6 +179,24 @@ python text_to_image.py --model <your-model> --quantization fp8 --ignored-layers
vllm serve <your-model> --omni --quantization fp8
```
### Distributed layerwise offload
Per-tensor online FP8 linears can be combined with DLO's default AllGather path:
```bash
vllm serve <your-model> \
--omni \
--quantization fp8 \
--enable-distributed-layerwise-offload
```
Direct checkpoint mmap is not used for this combination. Each rank runs the
ordinary loader to create the final FP8 weights and scales before DLO shards
them, so startup temporarily materializes the complete FP8 model in host memory
on every rank. Runtime DLO host residency is sharded normally. Use
`--dlo-no-use-allgather` when the synchronized AllGather request-wave contract
does not fit the workload.
## Parameters
| Parameter | Type | Default | Description |
Loaded 100 of 1457 files, more files were not shown because too many files have changed in this diff. Show more