Merge remote-tracking branch 'origin/main' into qwen-image-2.1

# Conflicts:
#	vllm_omni/diffusion/executor/multiproc_executor.py
#	vllm_omni/diffusion/offloader/sequential_backend.py
This commit is contained in:
Isotr0py committed 2026-09-21 07:32:04 +00:00
commit 9174554e40
1332 files changed
+160051 -60504

No files matched your search

+3
View File
@@ -71,6 +71,7 @@ nav:
- Overview and Compatibility:
- Overview: user_guide/diffusion_features.md
- Feature Compatibility: user_guide/feature_compatibility.md
- Device-side Video Postprocessing: user_guide/diffusion/device_side_video_postprocess.md
- CPU Offloading:
- Overview: user_guide/diffusion/cpu_offload.md
- Model-Level: user_guide/diffusion/offloader/module_offload.md
@@ -135,6 +136,7 @@ nav:
- Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md
- Mooncake Transfer Engine Connector: design/feature/omni_connectors/mooncake_transfer_engine_connector.md
- Mori Transfer Engine Connector: design/feature/omni_connectors/mori_transfer_engine_connector.md
- NIXL Connector: design/feature/omni_connectors/nixl_connector.md
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
@@ -159,6 +161,7 @@ nav:
- Cache-DiT: design/feature/cache_dit.md
- TeaCache: design/feature/teacache.md
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
- Mooncake KV Transfer: design/feature/mooncake-kv-transfer.md
- Infrastructure and Performance:
- Prometheus Metrics: design/metrics.md
- Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
+1 -1
View File
@@ -56,7 +56,7 @@ vLLM-Omni is flexible and easy to use with:
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
- Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
- TTS models (e.g. Qwen3-TTS, IndexTTS 2.5, dots.tts, CosyVoice3)
- TTS models (e.g. Qwen3-TTS, IndexTTS 2.5, CosyVoice3)
- Diffusion models — image, video, and audio generation (e.g. MiniMax H3, LTX-2.5, SANA-Video, Wan2.2)
- Robot-policy and action models (e.g. π0, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
+6 -1
View File
@@ -5,6 +5,9 @@
Main entry points for vLLM-Omni inference and serving.
- [vllm_omni.entrypoints.async_omni.AsyncOmni][]
- [vllm_omni.entrypoints.async_omni_base.AsyncOmniBase][]
- [vllm_omni.entrypoints.duplex_omni.DuplexOmni][]
- [vllm_omni.entrypoints.duplex_omni.DuplexSessionHandle][]
- [vllm_omni.engine.cfg_companion_tracker.CfgCompanionTracker][]
- [vllm_omni.entrypoints.cli.benchmark.base.OmniBenchmarkSubcommandBase][]
- [vllm_omni.entrypoints.cli.benchmark.main.OmniBenchmarkSubcommand][]
@@ -17,12 +20,13 @@ Main entry points for vLLM-Omni inference and serving.
## Clients
Client libraries that talk to a running vLLM-Omni server (the full-duplex Realtime client and its per-model session presets).
Client libraries for full-duplex sessions: the Realtime WebSocket client, the in-process client over `DuplexOmni`, and the per-model session presets.
- [vllm_omni.clients.duplex.AudioDelta][]
- [vllm_omni.clients.duplex.AudioFormat][]
- [vllm_omni.clients.duplex.ConnectionResumed][]
- [vllm_omni.clients.duplex.DuplexClient][]
- [vllm_omni.clients.duplex.DuplexClientBase][]
- [vllm_omni.clients.duplex.DuplexClientError][]
- [vllm_omni.clients.duplex.DuplexConnectionError][]
- [vllm_omni.clients.duplex.DuplexEvent][]
@@ -44,6 +48,7 @@ Client libraries that talk to a running vLLM-Omni server (the full-duplex Realti
- [vllm_omni.clients.duplex.TextDelta][]
- [vllm_omni.clients.duplex.TranscriptDelta][]
- [vllm_omni.clients.duplex.WebSocketTransport][]
- [vllm_omni.clients.inline_duplex.InlineDuplexClient][]
## Inputs
Binary file not shown.

Before

Width:  |  Height:  |  Size: 272 KiB

After

Width:  |  Height:  |  Size: 267 KiB

+60 -3
View File
@@ -23,7 +23,9 @@ You can use `vllm bench serve --omni --help=all` to get descriptions of all para
The API endpoint exposed externally, to which clients send their requests.
- `--dataset-name`
The name of the dataset used; random-mm indicates generating random multimodal inputs (images, videos, audio), while `omniinteract` replays official OmniInteract videos.
The name of the dataset used; `random-mm` generates random multimodal inputs (images, videos, audio),
`omniinteract` replays official OmniInteract videos, and `videomme` loads Video-MME MCQ videos
(`lmms-eval/Video-MME`, MiniCPM OmniEvalKit packing).
- `--num-prompts`
The total number of requests to send, an integer.
@@ -316,8 +318,12 @@ arrives, checkpointing the response's history position so a later committed user
outputs are ineligible and omitted from the official manifest; `audio_clipped_bytes` records output beyond the rounded video
horizon.
TTFT, TTFP, and RTF start at client receipt of `response.created`. TPOT/ITL use engine stage-0 timing; ITL is emitted only when
every token interval is present.
Per-response TTFT and TTFP start when the server begins executing the native model-turn request that owns the response. RTF
continues to use client receipt of `response.created` through the last audio packet, divided by emitted audio duration. Global
TTFT, TTFP, and RTF cover the complete input-stream window. TPOT/ITL use engine stage-0 timing; ITL is emitted only when every
token interval is present within a continuous generation segment. Model-unit pacing and gaps between generation segments are
excluded from TPOT/ITL. Raw request metrics retain `response_created_to_first_text_ms` and
`response_created_to_first_audio_ms` as client-envelope diagnostics.
The checked-in local performance configuration measures four deterministic cases from each OmniInteract subset (12 videos
total), with no benchmark warmups and a maximum concurrency of two. Each subset also sends one readiness request before its
@@ -339,6 +345,57 @@ transcript, event, and result artifacts without errors. A valid LISTEN-only case
chunks. Official-manifest eligibility is reported separately because clipped or cancelled output is a benchmark-quality
signal, not a transport failure. This local performance test does not score answer accuracy.
### Video-MME Benchmark
Video-MME (`--dataset-name videomme`) scores multiple-choice video QA. Default packing is
OmniEvalKit MiniCPM `minicpm-frames` (up to 96 sampled frames as `image_url`). Pass a local
mirror with `--dataset-path` / `--videomme-parquet` + `--videomme-video-dir`, or a Hugging
Face dataset id (`lmms-eval/Video-MME` by default; any `org/name` is accepted when
`--dataset-name videomme` is explicit). Relative `--videomme-video-dir` values are resolved
to absolute `file://` URLs. For those URLs, start the server with
`--allowed-local-media-path` covering the video root; otherwise use `--videomme-inline-local-video`.
For MiniCPM-o 4.5, the default deployment allows 64 images per request. To run the
96-frame recipe, start the server with a stage 0 override (replace `/path/to/Video-MME`
with the local video root shared by the benchmark client and server):
```bash
vllm serve openbmb/MiniCPM-o-4_5 --omni \
--allowed-local-media-path /path/to/Video-MME \
--stage-overrides '{"0":{"limit_mm_per_prompt":{"image":96,"audio":64,"video":1}}}'
```
The override applies to this server invocation only. Alternatively, use
`--videomme-max-frames 64` in the benchmark to stay within the default image limit;
this evaluates a different frame-sampling setting from the 96-frame recipe.
Run the benchmark against that server:
```bash
vllm bench serve --omni \
--backend openai-chat-omni \
--dataset-name videomme \
--dataset-path /path/to/Video-MME \
--videomme-pack-mode minicpm-frames \
--videomme-max-frames 96 \
--model openbmb/MiniCPM-o-4_5 \
--endpoint /v1/chat/completions \
--num-prompts 8 \
--max-concurrency 1 \
--save-result
```
Accuracy keys (`videomme_accuracy`, per-duration / domain / task breakdowns) are written into
the saved JSON. `videomme_accuracy` excludes HTTP failures; `videomme_accuracy_incl_http_fail`
counts them as wrong. `videomme_submitted` / `videomme_unique_question_ids` show when
oversampling made the request count differ from unique-question coverage.
`videomme_skipped_rows` counts rows skipped during sampling because of missing or unreadable
media or invalid question fields. The opt-in accuracy runner rejects skipped rows, HTTP
failures, duplicate question IDs, and missing gold answers. This checks the sampled subset;
it does not establish that a supplied local mirror contains the entire official dataset.
Use `--videomme-save-eval-items` (or `VIDEOMME_SAVE_EVAL_ITEMS=1`) for per-request
`videomme_eval_items` rows.
### Multi-Modal Benchmark
<details class="admonition abstract" markdown="1">
+69
View File
@@ -1,5 +1,67 @@
# vllm-omni serve
## Multiple API frontends
For local EngineCore pipelines with one or more stages, `--api-server-count` starts
multiple API frontend processes that share one parent-owned set of stage
engines:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--api-server-count 2 \
--port 8091
```
This mode is intended for workloads whose request parsing, multimodal input
processing, or response encoding can bottleneck a single frontend. It
currently requires local multi-process EngineCore stages with one local
process group per replica. A stage may still use multiple `num_replicas`.
It cannot be combined with diffusion, headless or
remote stages, intra-stage data parallelism, Ray, fault tolerance, elastic
expert parallelism, sleep mode, or runtime LoRA updating. Runtime voice upload
and deletion are also disabled because those mutations are process-local;
built-in voices, inline reference audio, and voices restored at startup remain
available. The `/v1/omni/sleep` and `/v1/omni/wakeup` control routes return HTTP
409 for the same reason: their bookkeeping is process-local while stage engines
are shared.
A single-stage model pipeline is supported through the normal launch command;
this is distinct from the distributed `--stage-id` / `--headless` launch mode.
Every stage in the pipeline must be an EngineCore stage. Mixed AR-to-DiT
pipelines, including MiniMax-H3, remain unsupported even when their first
stage is an EngineCore stage. Diffusion support requires per-frontend channels
and shared-engine lifecycle handling in the diffusion backend.
### Workloads and performance validation
Each API process owns its frontend processor, Orchestrator, request state,
and per-replica EngineCore channels. Requests return through the channel of
the originating API process; model weights and GPU stage engines are shared.
This distributes frontend and orchestration work across CPU processes.
| Workload | Potential benefit | Limitation |
| ---------- | ------------------- | ------------ |
| Concurrent cold image/video inputs on supported EngineCore pipelines | Parallel frontend processing when one API process is CPU-limited | A single cold request and GPU computation may be unchanged |
| Many replicas producing frequent text/audio outputs | Less API/Orchestrator work and GIL contention per process | Routing remains serial within each Orchestrator |
| Warm inputs or low-concurrency, GPU-limited generation | Additional frontend capacity | More API processes may add overhead without improving throughput |
Here, a cold input means media or preprocessing caches miss, not that model
weights must be loaded or GPU kernels compiled. Each frontend has its own
processor/cache state, so cache duplication, CPU thread contention, and memory
usage also matter. Multi-API serving does not itself change the output queue,
move response encoding to an executor, or change the model's numerical scheduler.
Compare `--api-server-count 1`, `2`, and higher counts with the same hardware,
model/stage configuration, media, sampling parameters, output lengths, and
offered load. Keep cold-cache and warmed-cache runs separate and report repeated
measurements. Record CPU and request counts for every API worker, input and
output queue delay, GPU utilization, first-output latency, and end-to-end latency.
Confirm requests reach every worker: connection reuse can concentrate traffic
on one process. Queue-only or single-API experiments do not establish a
multi-API throughput gain. The multi-replica workload in
[#4680](https://github.com/vllm-project/vllm-omni/issues/4680) remains a relevant
validation target.
## Stage-based CLI quickstart
The stage-based CLI is designed for deployments that require launching each pipeline stage in an isolated process
@@ -61,3 +123,10 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
## Arguments
--8<-- "docs/generated/argparse_omni/omni_serve.inc.md"
Async video jobs use process-local metadata and task handles. With multiple API
workers, creating, listing, retrieving, downloading, and deleting these jobs
returns HTTP 409. This restriction is independent of the current diffusion
stage launch restriction: supporting multi-worker diffusion also requires a
shared job lifecycle. Missing frontend topology state returns HTTP 503 for
operations that require process-local state protection.
@@ -70,6 +70,7 @@ The lifecycle labels used below are:
| `OMNI_DIFFUSION_PROMPT_EMBED_CACHE_SIZE` | Positive integer; default `32` entries | Each diffusion runner; resolved during model setup | A valid environment value overrides the explicit cache size. A non-integer logs a warning; a non-positive value is ignored. | Experimental |
| `OMNI_DIFFUSION_SESSION_STATE_MANAGER` | Boolean with the same accepted spellings as the prompt cache; default disabled | Experimental diffusion session manager; model setup | A recognized environment value overrides the explicit enable setting. An unrecognized value is ignored. | Experimental |
| `OMNI_DIFFUSION_SESSION_STATE_MANAGER_MAX_SESSIONS` | Positive integer; default `64` | Experimental diffusion session manager; model setup | A valid environment value overrides the explicit maximum. A non-integer logs a warning; a non-positive value is ignored. | Experimental |
| `VLLM_OMNI_AR_DIFFUSION_KV_GATHER` | `1` enables; default `0` (off) | AR-diffusion KV manager at allocation and attention at dispatch; set before worker startup | Only the exact value `1` enables contiguous KV gathering. History staging additionally requires `ARDiffusionKVConfig.reuse_history_staging`. | Experimental |
Backend names for `DIFFUSION_ATTENTION_BACKEND` are the members of
`DiffusionAttentionBackendEnum`, such as `FLASH_ATTN`, `TORCH_SDPA`,
@@ -82,6 +83,7 @@ depends on the installed kernels and model path.
| --- | --- | --- | --- | --- |
| `SPEAKER_SAMPLES_DIR` | Filesystem path; default `~/.cache/vllm-omni/speakers` | Speech server; read when speaker storage initializes | Environment-only setting. The directory is created; filesystem errors propagate. | Stable |
| `SPEAKER_MAX_UPLOADED` | Integer; default `1000` | Speech server; read when speaker storage initializes | Environment-only setting. A non-integer logs a warning and uses `1000`; range is not otherwise validated. | Stable |
| `VLLM_OMNI_ABORT_TIMEOUT` | Float seconds; default `2` | Engine abort wait for Videos DELETE and `generate()` cancel/error cleanup; read when `async_omni` imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
| `VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT` | Float seconds; default `600` | Diffusion engine async-output wait in `step_streaming`; resolved per call on the request path, not at import | Environment-only setting. A non-float or `<=0` value warns once and uses the default. | Experimental |
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; default `0` (off) | Orchestration loop and the serving-side final-output drain; read once when the `Orchestrator` is constructed | Environment-only setting. Values are stripped and case-normalized; any unrecognized value leaves the legacy poll loop selected. | Experimental |
| `VLLM_OMNI_INPUT_WAIT_TIMEOUT_S` | Float seconds; default `600`; `<=0` disables | Full-payload input coordinator, not async-chunk transfer; read when the scheduler module imports in each worker | Environment-only setting. A non-float logs a warning and uses `600`. | Stable operational control |
@@ -90,6 +92,13 @@ depends on the installed kernels and model path.
| `VLLM_VIDEO_ASYNC_CHUNK` | `on` or `off`; default `on` | Streaming video output; read on attribute access | Environment-only setting. Values are trimmed and case-normalized; an invalid value warns once and uses `on`. | Experimental |
| `VLLM_VIDEO_AUDIO_DELTA_MODE` | `fast` or `slow`; default `fast` | Streaming video audio deltas; read on attribute access | Environment-only setting. Values are trimmed and case-normalized; an invalid value warns once and uses `fast`. | Experimental |
### NIXL stage transfer
| Name | Type and default | Applies to and read time | Precedence and invalid values | Lifecycle |
| --- | --- | --- | --- | --- |
| `VLLM_OMNI_NIXL_LEASE_S` | Float seconds; default `3600` | Unclaimed producer payload expiry; connector construction | Nonempty environment value overrides `lease_seconds`; empty falls back to config/default. Invalid floats raise `ValueError`. Claimed READ allocations never expire by time. | Experimental |
| `VLLM_OMNI_NIXL_XFER_TIMEOUT_S` | Float seconds; default `300` | Receiver READ wait; connector construction | Nonempty environment value overrides `transfer_timeout_s`; empty falls back to config/default. Invalid floats raise `ValueError`. Timeout does not cancel DMA or release active allocations. | Experimental |
### Server storage
Storage names use Pydantic's nested-settings delimiter (`__`). They are read
@@ -112,6 +121,7 @@ settings.
| Name | Type and default | Applies to and read time | Precedence and invalid values | Lifecycle |
| --- | --- | --- | --- | --- |
| `VLLM_OMNI_FUSED_QK_NORM_ROPE_MIN_TOKENS` | Non-negative integer; unset or empty means each consumer's own default (Boogu-Image: `2048`) | Token gate of the shared `fused_qk_norm_rope` op: a rotary table is packed for the fused kernel only when it spans at least this many positions (`B*S`); read at forward time | Environment-only. `0` = always fuse; a very large value disables the fused path. Any other value raises `ValueError` on the first forward. The crossover is host-dependent; re-benchmark before overriding. | Experimental performance control |
| `VLLM_OMNI_SKIP_NVFP4_NAN_CLAMP` | Boolean truthy spellings: `1`, `true`, `yes`, `on`; default false | ModelOpt NVFP4 compatibility patch; read when `vllm_omni.patch` imports | Environment-only escape hatch. Any other value means false. Set only to diagnose the upstream NaN-scale issue. | Diagnostic and temporary |
| `VLLM_OMNI_USE_QUACK_FP8` | Boolean truthy spellings: `1`, `true`, `yes`, `on`; unset means hardware auto-detection | FP8 scaled matrix multiplication; evaluated when quack capability is selected | A set value overrides auto-detection. Any non-truthy value forces quack off. If quack cannot load, vLLM-Omni warns and falls back to FlashInfer. | Experimental performance control |
+91 -3
View File
@@ -17,6 +17,7 @@ Common `PipelineConfig` fields include:
| ------- | ------------- |
| `model_type` | Pipeline identifier used during model and config resolution. |
| `default_deploy_config_name` | Bundled deploy YAML loaded when the user does not pass `deploy_config`. |
| `duplex_plugin` | Dotted path of the model's `DuplexModelPlugin`. Set only for full-duplex models; it makes `vllm-omni serve` run the model through `DuplexOmni` (a server whose every surface runs on a duplex session) and the engine host a `DuplexOrchestrator` with the plugin loaded. |
| `model_arch` | Default Hugging Face architecture for the pipeline. |
| `hf_architectures` | Architecture names used to identify checkpoints whose `model_type` is shared. |
| `hf_config_predicate` | Optional predicate used to select between pipelines with otherwise identical HF metadata. |
@@ -38,7 +39,7 @@ Common `StagePipelineConfig` fields include:
| `custom_process_input_func` | Processor applied to this stage's incoming payload. |
| `custom_process_next_stage_input_func` | Processor used for full-payload handoff to the next stage. |
| `async_chunk_process_next_stage_input_func` | Processor used for async chunk handoff. |
| `sampling_constraints` | Model-owned sampling constraints that deploy defaults cannot override. |
| `sampling_constraints` | Model-owned sampling constraints that deploy defaults cannot override. Scalar values replace deploy defaults; required `stop_token_ids` extend and deduplicate them. |
To add or change topology, define and register a new pipeline variant. Use
deploy YAML only for runtime placement, resource sizing, connectors, and other
@@ -52,7 +53,7 @@ The new deploy schema lives under `vllm_omni/deploy/` and is paired with a froze
| ------- | ------ | ---------- | --------- | ------------- |
| `base_config` | str (path) | optional | — | Overlay parent (relative or absolute). `stages:` / `platforms:` deep-merged by stage_id; other scalars overlay-wins. Intended for user-authored overlays; prod yamls stay flat. |
| `async_chunk` | bool | optional | `true` | Enable chunked streaming between stages. Pin to `false` if the pipeline runs end-to-end. |
| `session_mode` | str | optional | `"turn"` | Session behavior. MiniCPM-o 4.5 deploy YAMLs set `"duplex"` so `/v1/realtime` and chat share the same profile. |
| `session_mode` | str | optional | `"turn"` | Session behavior. For pipelines with a `duplex_plugin`, explicitly select `"duplex"` for the duplex engine or `"turn"` for the ordinary online serving stack. The shipped MiniCPM-o 4.5 default remains `"duplex"`. See [Full Duplex](../serving/full_duplex_api.md#enable-full-duplex). |
| `active_stream_window` | int | optional | `0` | Number of active downstream stream slots; `0` preserves all-stream cycling. |
| `duplex_session` | dict | optional | runtime defaults | Full-duplex session lifecycle, buffering, replay, and capacity limits. |
| `connectors` | dict | optional | `null` | Named connector specs (`{name, extra}`). Referenced by each stage's `input_connectors` / `output_connectors`. See [Connector schema](#connector-schema). |
@@ -97,7 +98,7 @@ Each entry under `stages:` accepts any `StageDeployConfig` field directly (no ne
| `devices` | str \| null | optional | `null` | Device list assigned to this stage. The number of device ids must equal this stage's local world size (`tensor_parallel_size` × local data-parallel size × `pipeline_parallel_size`, or `num_replicas` × that product for a replica pool); a mismatch fails early — see the note below. |
| `output_connectors` | dict \| null | optional | `null` | Keyed by `to_stage_<n>`; values are names registered under top-level `connectors:`. |
| `input_connectors` | dict \| null | optional | `null` | Keyed by `from_stage_<n>`; values are names registered under top-level `connectors:`. |
| `default_sampling_params` | dict \| null | optional | `null` | Baseline sampling params. Deep-merged with pipeline `sampling_constraints` (pipeline wins). |
| `default_sampling_params` | dict \| null | optional | `null` | Baseline sampling params. Merged with pipeline `sampling_constraints`; scalar constraints win, while required `stop_token_ids` are appended and deduplicated. |
| `engine_extras` | dict | optional | `{}` | Catch-all for engine fields not listed above; deep-merged across overlays and forwarded to the stage engine. |
**Note:** a stage's `devices` count must equal its local world size (`tensor_parallel_size` × `data_parallel_size_local` × `pipeline_parallel_size`, falling back to global `data_parallel_size` when the local size is unset), or `num_replicas` × that product for a replica pool. A mismatch fails early and names the offending stage. A top-level `--tensor-parallel-size` is broadcast to every stage, so it can make a single-GPU stage violate this contract ([issue #5003](gh-issue:5003)); fix that case with `--stage-overrides` (set `tensor_parallel_size` and `devices` together per stage) or set TP only on the multi-GPU stage.
@@ -289,3 +290,90 @@ vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/depl
!!! important
We are actively iterating on the definition of deployment configurations, and we welcome feedback from users and developers.
## Qwen3-TTS with Model Runner V2
Qwen3-TTS can opt into the native CUDA Model Runner V2 pipeline on vLLM
0.29.0. Select one of these deployment profiles:
| Profile | Runner | Code2Wav graph batches | Intended use |
| --- | --- | --- | --- |
| `qwen3_tts.yaml` | V1 | Existing defaults | Existing deployment / regression control |
| `qwen3_tts_mrv2.yaml` | V2 | B1 | Native runner validation |
| `qwen3_tts_high_concurrency_mrv2.yaml` | V2 | B1, B2 | Opt-in throughput tuning |
| `qwen3_tts_high_concurrency_mrv2_b4.yaml` | V2 | B1, B2, B3, B4 | Experimental throughput / buffered playback |
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni \
--deploy-config vllm_omni/deploy/qwen3_tts_mrv2.yaml
```
The V2 profiles bound Talker prefill to 512 tokens per step and select the
Talker AR runner and the Code2Wav generation
runner together. Native inter-stage delivery carries codec payloads directly;
request-owned snapshots preserve buffers through asynchronous completion and
CUDA graph reuse. Terminal completion waits for upstream stage metrics before
releasing request state. Platform sections retain V1 on NPU, XPU, ROCm and MUSA;
this change does not qualify MRV2 on those backends or enable other model families.
Selecting V2 for a stage without `supports_native_mrv2_data_plane` emits a warning
when pipeline and deployment settings are merged. That stage retains the legacy
transport path; the warning does not establish support for that combination.
MRV2 model hooks use capability declarations rather than architecture or stage
names. Omni lifecycle flags select the model state, and `_returns_tuple` declares
the capture output contract. The optional batched predictor hook is `mtp`, with
an explicit `mtp_output_key` (a string or two-part payload key), optional
`mtp_validity_key`, and `mtp_graph_safe`/`mtp_disable_graph` capture controls.
Its inputs are token IDs, embeddings, previous hidden states and per-row
conditioning; it returns updated embeddings and prediction codes. Models may
supply `mtp_sampling_params` and `get_mtp_seed(sampling_params)` for model-local explicit seeds, and declare
`mtp_accepts_per_row_generators`, `mtp_accepts_req_infos`, or `mtp_sample_uniforms`
(with `mtp_sample_steps` and `mtp_sample_vocab_size`) as needed. Qwen3-TTS retains
its existing `talker_mtp` entry point for V1.
### Included performance work
- Request snapshots have a fast path for immutable scalar leaves, including
waveform lists. Mutable containers, aliases and cycles keep deep-copy
semantics. This is not the historical shallow `dict(prompt)` experiment.
- The API reference cache holds owned float32 arrays with a default capacity
of 1024 entries and a 512 MiB waveform-payload budget. Cache hits return
independent lists. Qwen3-TTS model artifact caches also default to 1024
entries. These caches have different owners and eviction policies; equal
entry limits do not make them a single coherent cache.
- Talker state stays on GPU where its lifecycle allows it. BOS/EOS projections
are cached in their projection dtype and invalidated on weight loading.
- Code2Wav packs CPU codec inputs before device transfer. Optional B2/B4 graph
buckets batch compatible requests without sharing their per-request state.
- Native output materialization runs in a bounded worker and drains before
closing the data plane. The existing control-only shortcut avoids launching
model work for a step that has only lifecycle events.
MTP prefix re-prefill remains disabled in the high-concurrency V2 profile.
Adding more graph shapes increases compilation cost and has not established a
stable end-to-end gain for this extracted version.
### Decoder batches and playback
Changing `decode_cudagraph_batch_sizes` selects the captured batch buckets.
Keep `decode_batch_max_size` consistent with the intended maximum too; the
stateful graph path currently groups according to the captured buckets, while
the stateless path also uses the explicit maximum.
B4 remains experimental. First-packet latency alone does not establish
uninterrupted playback. Validate inter-chunk arrival times, buffering, WER and
speaker similarity before adopting either batching preset for a production
workload. Floating-point decoder outputs can differ across batch sizes; this PR
does not claim bitwise or quality equivalence.
### Optional MPS deployment
NVIDIA MPS is an optional operator setting for colocated CUDA processes, not a
YAML option or a library default. This PR does not establish a throughput or
first-packet latency benefit from MPS. Measure the exact deployment with and
without MPS before enabling it.
Use only assigned GPUs and an independent MPS pipe directory. A private MPS
server does not provide exclusive GPU ownership or MIG isolation. For a
single-GPU deployment, explicitly place both stages on that GPU; the supplied
high-concurrency profile places its two stages on different GPUs by default.
+37 -31
View File
@@ -15,7 +15,8 @@ Canonical layout (prefer these paths for new changes):
│ │ ├── skip_ci.py # skip-ci decision (docs / skip-mark / CI YAML paths)
│ │ ├── upload_pipeline.py # Bootstrap + test-pipeline uploader (CUDA/NPU)
│ │ └── resolve_skip_ci.sh # Shell helpers for AMD/Intel bootstrap
│ └── ci_mirror_hardwares.yml # CUDA uploader presets (referenced by name only)
│ ├── ci_mirror_hardwares.yml # CUDA/NPU hardware presets (referenced by name only)
│ └── ci_source_file_dependencies.yml # L2–L5 path-prefix presets (referenced by name only)
├── cuda/ # Primary NVIDIA CUDA CI
│ ├── pipeline.yml # Bootstrap entry (hook upload)
│ ├── bootstrap-upload-steps.yml # Bootstrap child steps (upload_pipeline --upload)
@@ -34,6 +35,7 @@ Canonical layout (prefer these paths for new changes):
├── amd/
│ ├── test-amd-ready.yml # L2 job definitions (template input)
│ ├── test-amd-merge.yml # L3 job definitions
│ ├── test-amd-nightly.yml # Nightly full-model jobs (experimental)
│ ├── test-template-amd-omni.j2 # Renders final pipeline.yaml
│ └── scripts/
│ ├── bootstrap-amd-omni.sh # Entry: skip-ci → Jinja → upload
@@ -63,7 +65,7 @@ Canonical layout (prefer these paths for new changes):
| -------- | ---------------- | -------------- | ---------------- | -------------------- |
| **CUDA** | `cuda/pipeline.yml` | `test-ready.yml`, `test-merge.yml`, `test-nightly.yml`, `test-weekly.yml` | `upload_pipeline.py --upload` (expands uploader-only keys) | omit `mirror_hardwares` (from `-m`) / preset string + `MIRROR_HW` |
| **NPU** | `npu/pipeline-npu.yml` | `test-npu-ready.yml`, `test-npu-nightly.yml` | `upload_pipeline.py --upload` | `mirror_hardwares: a2b3_npu_1` / `a2b3_npu_4` / `a3_npu_2` |
| **AMD** | `amd/scripts/bootstrap-amd-omni.sh` | `test-amd-ready.yml`, `test-amd-merge.yml` | Jinja (`test-template-amd-omni.j2`) → `pipeline upload` | `agent_pool` + `mirror_hardwares: [amdproduction]` (array, template filter) |
| **AMD** | `amd/scripts/bootstrap-amd-omni.sh` | `test-amd-ready.yml`, `test-amd-merge.yml`, `test-amd-nightly.yml` | Jinja (`test-template-amd-omni.j2`) → `pipeline upload` | `agent_pool` + `mirror_hardwares: [amdproduction]` (array, template filter) |
| **Intel** | `intel/scripts/bootstrap-intel-omni.sh` | `intel/pipeline-intel.yml` (steps inline) | Direct `pipeline upload` | Inline `agents.queue` on each step |
## Platform configuration style
@@ -108,19 +110,21 @@ Canonical layout (prefer these paths for new changes):
Inferred form: unset/`empty` `MIRROR_HW` matches `H100` or `L4` in `-m` (both present → H100); no match skips the step. `MIRROR_HW=b200` must appear in `-m` or the step is skipped. The uploader does **not** rewrite pytest `-m`; B200 collection comes from the YAML expression and from tests that declare `B200` in `hardware_test` / `hardware_marks` (for example `res={"cuda": ["H100", "B200"]}`). `MIRROR_HW` must be empty or `b200` (case-insensitive); unknown values (for example `b20o`) **fail the upload**. A CUDA preset string such as `mirror_hardwares: h100_4` is skipped when `MIRROR_HW=b200`. Unset `MIRROR_HW` keeps string presets. NPU presets ignore `MIRROR_HW`. A `mirror_hardwares` name that is not a key in `ci_mirror_hardwares.yml` **fails the upload**.
**Path filter in YAML:** `source_file_dependencies: <preset>` (string, or a list of preset names)—preset names in [`common/ci_source_file_dependencies.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/ci_source_file_dependencies.yml). Required on **E2E Test** leaf jobs in L2/L3, and on L4/L5 leaf jobs that should be diff-gated on PR labels; see [Step filtering](#step-filtering).
**Conventions**
- **`depends_on`:** leaf jobs depend on `upload-ready-pipeline`, `upload-merge-pipeline`, etc.
- **`group` / `label`:** `:card_index_dividers:` groups; labels like `Diffusion · Qwen Image Test`.
- **`commands`:** `timeout … pytest …` with markers and `--run-level` for the pipeline level.
- **`source_file_dependencies`:** required on **E2E Test** leaf jobs in L2/L3; see [Step filtering](#step-filtering).
- **`source_file_dependencies`:** required on **E2E Test** leaf jobs in L2/L3, and on L4/L5 leaf jobs that should be diff-gated on PR labels; name a job key from `ci_source_file_dependencies.yml` (see [Step filtering](#step-filtering)). The key must list every script the step runs; `commands` is not scanned.
**Adding a job**
1. Pick the level file (`test-ready.yml` / `test-merge.yml` / `test-nightly.yml` / `test-weekly.yml`).
2. Add a step under the right **group** (usually **E2E Test** for model pytest).
3. Set `label`, `commands`, `mirror_hardwares`, `depends_on: upload-ready-pipeline` (or `upload-merge-pipeline` / `upload-nightly-pipeline` / `upload-weekly-pipeline`).
4. For L2/L3 E2E, add `source_file_dependencies` (pytest + model + deploy YAML prefixes).
3. Set `label`, `commands`, `mirror_hardwares`, `depends_on: upload-<level>-pipeline`.
4. For L2–L5 jobs that should be diff-gated, add a job key in `common/ci_source_file_dependencies.yml` and set `source_file_dependencies: <key>` on the step. YAML anchors hold model business-code paths only. The job key (`{model}_function` / `_perf` / `_accuracy` / `_reliability` / `_doc` / `_cov`) aliases that anchor and lists the pytest / example / benchmark scripts the step actually runs. The uploader does **not** parse `commands` for path filters—omitting a script from the key means editing that script will not select the job.
5. Dry-run:
```bash
@@ -144,6 +148,8 @@ Canonical layout (prefer these paths for new changes):
- A5: `a5_npu_2`, `a5_npu_4`, `a5_npu_8`, `a5_npu_16`
- 310P: `310p_npu_1`, `310p_npu_2`, `310p_npu_4`
**Path filter in YAML:** same `source_file_dependencies` presets as CUDA, on **L4** leaf jobs in `test-npu-nightly.yml`. Filtering applies on the PR `nightly-test` label; `main` + `NIGHTLY=1` keeps every job. If no job-key prefix matches, a change to the pipeline YAML being uploaded, or to any path under `source_filter_fallback`, keeps every job in that upload. See [Step filtering](#step-filtering).
**Conventions**
- **`depends_on: upload-ready-pipeline`** (or `upload-nightly-pipeline`) ties jobs to bootstrap upload keys.
@@ -153,7 +159,7 @@ Canonical layout (prefer these paths for new changes):
1. Edit `test-npu-ready.yml` (L2) or `test-npu-nightly.yml` (L4).
2. Add a step with `mirror_hardwares` (add a new preset in `ci_mirror_hardwares.yml` first if needed).
3. Set `commands` to your pytest file and markers.
3. Set `commands` to your pytest file and markers. For L4, set `source_file_dependencies` to a job key from `ci_source_file_dependencies.yml` that aliases the model anchor and lists that pytest file.
4. Dry-run:
```bash
@@ -166,9 +172,9 @@ Canonical layout (prefer these paths for new changes):
**Bootstrap:** [`amd/scripts/bootstrap-amd-omni.sh`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/scripts/bootstrap-amd-omni.sh)—skip-ci, diff filtering, Jinja render, then `buildkite-agent pipeline upload`.
**Test YAML (data):** `amd/test-amd-ready.yml` (L2 / `ready`), `test-amd-merge.yml` (L3 / `merge-test` and `main`). On a PR carrying both labels, AMD combines both suites behind one image build. PR builds without either tier label retain the legacy L2 fallback for compatibility with the `amd-test` trigger. `DEBUG_TEST_YAML` remains an explicit override. AMD has no L4 file yet, so `nightly-test` does not select an additional suite.
**Test YAML (data):** `amd/test-amd-ready.yml` (L2 / `ready`), `test-amd-merge.yml` (L3 / `merge-test` and ordinary `main`), and experimental `test-amd-nightly.yml` (`full_model` / `nightly-test` or `main` with `NIGHTLY=1`). Multiple PR tier labels combine their suites behind one image build. PR builds that reach the bootstrap without a tier label retain the legacy ready-suite fallback. `DEBUG_TEST_YAML` accepts `ready`, `merge`, and `nightly` as an explicit override. During nightly burn-in every AMD nightly leaf is non-blocking.
**Trigger boundary:** AMD label handling has two layers. First, the external `vllm-omni-amd-ci` Buildkite pipeline condition decides whether a GitHub label event creates a build. Only after that build starts does this repository's bootstrap inspect all current PR labels and select L2, L3, or both. Repository-side selection therefore cannot make a `merge-test` event start AMD CI by itself. The external condition must admit both `ready` and `merge-test`, while preserving the legacy `amd-test` trigger and non-PR `main` / scheduled builds. For PR builds without a `DEBUG_TEST_YAML` override, an unreadable label set fails the bootstrap instead of silently selecting the wrong tier. Debug overrides skip label lookup and are validated directly; invalid values, including whitespace-only values, fail the bootstrap. Keep `nightly-test` out of the AMD condition until an AMD L4 suite exists; otherwise it would start a build without providing nightly coverage.
**Trigger boundary:** AMD label handling has two layers. First, the external `vllm-omni-amd-ci` Buildkite pipeline condition decides whether a GitHub label event creates a build. Only after that build starts does this repository's bootstrap inspect all current PR labels and select the ready, merge, nightly, or combined suites. Repository-side selection therefore cannot start AMD CI by itself. AMD-only nightly validation uses both `amd-test` and `nightly-test`: the external condition should accept an event for either label only when the complete PR label set contains both. `amd-test` admits the AMD pipeline, while `nightly-test` makes the bootstrap select `NIGHTLY_TESTS:test-amd-nightly.yml`. Existing `ready` / `merge-test` behavior and non-PR `main` / scheduled builds remain separate. For PR builds without a `DEBUG_TEST_YAML` override, an unreadable label set fails the bootstrap instead of silently selecting the wrong tier. Debug overrides skip label lookup and are validated directly; invalid values, including whitespace-only values, fail the bootstrap. Until the paired-label condition is configured, use an authorized manual Buildkite build with `DEBUG_TEST_YAML=nightly` and verify that the bootstrap reports only the nightly suite.
**Rendering:** [`test-template-amd-omni.j2`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/test-template-amd-omni.j2) wraps data steps with `amd-build` image build and `amd_<agent_pool>` queues. Do **not** hand-edit generated `pipeline.yaml`.
@@ -184,7 +190,7 @@ Canonical layout (prefer these paths for new changes):
**Adding a job**
1. Edit `test-amd-ready.yml` or `test-amd-merge.yml`.
1. Edit `test-amd-ready.yml`, `test-amd-merge.yml`, or `test-amd-nightly.yml`.
2. Copy a neighboring block: `label`, `agent_pool`, `mirror_hardwares`, `commands`, optional `grade`.
3. Regenerate via bootstrap / Jinja; update `skip_ci.py` if you add a new YAML path.
@@ -213,7 +219,7 @@ PR diffs drive **two independent skip layers**. Both read changed files from git
| Layer | Script | When | What is skipped | Where you configure |
| ----- | ------ | ---- | --------------- | ------------------- |
| **Bootstrap** | [`skip_ci.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/skip_ci.py) + [`upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py) | Before child test pipelines upload (`cuda/pipeline.yml`, `npu/pipeline-npu.yml`, AMD/Intel bootstraps) | Entire default CI, or whole L2/L3 upload for a platform | Whitelists in `skip_ci.py`; bootstrap `if` injected by step `key` in `upload_pipeline.py` |
| **Step filter** | [`upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py) | While uploading CUDA L2/L3 YAML | Individual Buildkite steps inside `test-ready.yml` / `test-merge.yml` | `source_file_dependencies` on each step or group |
| **Step filter** | [`upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py) | While uploading CUDA L2–L5 YAML | Individual Buildkite steps inside `test-ready.yml` / `test-merge.yml` / `test-nightly.yml` / `test-weekly.yml` | `source_file_dependencies` key on each step (presets in `ci_source_file_dependencies.yml`) |
**Changed files** (both layers):
@@ -240,7 +246,7 @@ Label triggers (`ready`, `merge-test`) are unchanged—diff-aware logic only red
| Non-qualifying skip-mark and no CI YAML | normal CI | all on |
| Diff unavailable | normal CI | all on |
**`skip_all` exceptions (CUDA/NPU bootstrap only):** PR labels (`nightly-test`, `merge-test`, `weekly-test`, `ready`, …) do **not** revive jobs under `skip_all`. On `main`, scheduled `NIGHTLY=1` still builds the image and uploads L4; `WEEKLY=1` or `NON_CRITICAL=1` still builds the image and uploads L5 (CUDA). `WEEKLY=1` on `main` also uploads L2/L3 with `--e2e`.
**`skip_all` exceptions:** For CUDA/NPU, PR labels (`nightly-test`, `merge-test`, `weekly-test`, `ready`, …) do **not** revive jobs under `skip_all`; on `main`, scheduled `NIGHTLY=1` still builds the image and uploads L4, while `WEEKLY=1` or `NON_CRITICAL=1` still builds the image and uploads L5 (CUDA). `WEEKLY=1` on `main` also uploads L2/L3 with `--e2e`. AMD applies `skip_ci.py gate` only to selected ready/merge specs, so an explicitly selected AMD nightly spec (`nightly-test`, `NIGHTLY=1`, or a nightly debug override) remains runnable even for a docs-only diff.
**Yaml-gated nightly/weekly:** L4/L5 upload steps keep their normal label / `NIGHTLY` / `WEEKLY` / `NON_CRITICAL` conditions (for example PR `nightly-test` still uploads nightly). Only L2/L3 upload steps are matrix-gated.
@@ -257,6 +263,7 @@ Register new files in `L2_YAML_FILES`, `L3_YAML_FILES`, or `L45_YAML_FILES` in `
| L3 | `.buildkite/cuda/test-merge.yml` | cuda |
| L3 | `.buildkite/amd/test-amd-merge.yml` | amd |
| L4/L5 | `.buildkite/cuda/test-nightly.yml`, `test-weekly.yml` | cuda |
| L4/L5 | `.buildkite/amd/test-amd-nightly.yml` | amd |
| L4/L5 | `.buildkite/npu/test-npu-nightly.yml` | npu |
##### Yaml-gated category branches
@@ -289,7 +296,7 @@ Docs/skip-mark mixed with any row below follows the same matrix.
| CUDA ready + AMD merge | G4 | `cuda/l2` + `amd/l3` |
| CUDA ready + AMD ready | G2 | `cuda/l2` + `amd/l2` |
| CUDA merge + AMD merge | G3 | `cuda/l3` + `amd/l3` |
| Only nightly / weekly / npu-nightly YAML | G1 | (none) |
| Only nightly / weekly / amd-nightly / npu-nightly YAML | G1 | (none) |
| CUDA ready + nightly | G5 | `cuda/l2` |
| CUDA merge + nightly | G6 | `cuda/l3` |
| CUDA ready + merge + nightly | G7 | `cuda/l2` + `cuda/l3` |
@@ -306,40 +313,38 @@ Unit coverage: `tests/buildkite/test_skip_ci.py`.
#### Step filtering {#step-filtering}
CUDA **L2** (`.buildkite/cuda/test-ready.yml`) and **L3** (`.buildkite/cuda/test-merge.yml`) only. Bootstrap upload entry: `upload_pipeline.py --upload .buildkite/cuda/bootstrap-upload-steps.yml`.
CUDA **L2** (`.buildkite/cuda/test-ready.yml`), **L3** (`.buildkite/cuda/test-merge.yml`), **L4** (`.buildkite/cuda/test-nightly.yml`), **L5** (`.buildkite/cuda/test-weekly.yml`), and NPU **L4** (`.buildkite/npu/test-npu-nightly.yml`). Bootstrap upload entry: `upload_pipeline.py --upload .buildkite/cuda/bootstrap-upload-steps.yml` (or `npu/bootstrap-upload-steps.yml`).
**When filtering applies:** **PR label** uploads (`ready`, `merge-test`, `nightly-test`, `weekly-test`) and **post-merge `main` L3** (`test-merge.yml`). Scheduled **`main` + env** uploads that pass `--all` (`NIGHTLY=1`, `WEEKLY=1` / `NON_CRITICAL=1`) or `--e2e` (`WEEKLY=1` on L2/L3) keep every selected job and still strip the key. If **no** job-key prefix matches the diff, a change to the pipeline YAML being uploaded, or any path under the `source_filter_fallback` key in `ci_source_file_dependencies.yml`, keeps every job in that upload. `source_filter_fallback` is not a job key — do not attach it to a step. When any job-key prefix already matches, normal filtering wins (the fallback does not expand the selection).
**Uploader-only keys** — removed before Buildkite sees the YAML; never used at runtime on agents:
| Key | Purpose |
| --- | ------- |
| `source_file_dependencies` | List of path **prefixes**. If any changed file equals a prefix or starts with `prefix/`, keep the step (or group); otherwise omit it. |
| `source_file_dependencies` | Named preset (or list of presets) from [`ci_source_file_dependencies.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/ci_source_file_dependencies.yml). The uploader expands only that key; it does **not** scan `commands`. If any changed file equals a listed prefix or starts with `prefix/`, keep the step (or group); otherwise omit it. Inline path-prefix lists are still accepted. |
| `mirror_hardwares` | Expand to `agents` + `plugins` (+ optional `image`) from `ci_mirror_hardwares.yml`. |
### Policy
- **Always uploaded** (no key): groups outside **E2E Test**—Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
- **Diff-gated**: every **E2E Test** leaf job. List the smallest prefix set per step—pytest file(s), model code under `vllm_omni/model_executor/models/` or `vllm_omni/diffusion/models/`, plus `stage_input_processors/` and `vllm_omni/deploy/*.yaml` when applicable. A **group** may define the key instead; the whole group drops if no prefix matches.
- **Always uploaded** (no key): L2/L3 groups outside **E2E Test**—Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
- **Diff-gated**: every **E2E Test** leaf job in L2/L3, L4 nightly leaf jobs (CUDA and NPU), and L5 weekly jobs whose `if` includes the `weekly-test` label (Reliability / Perf). Jobs gated only by `WEEKLY=1` or `NON_CRITICAL=1` have no `source_file_dependencies`. Registry layout: YAML anchors are model business code only (`vllm_omni/model_executor/models/`, `vllm_omni/diffusion/models/`, plus `stage_input_processors/` and `vllm_omni/deploy/*.yaml` when applicable). Job keys alias the anchor and **also** list the scripts that job runs (pytest files, `run_cov_split.sh --offline` / `--online` targets, example README runners). Feature keys stay cross-cutting. A **group** may define the key instead; the whole group drops if no prefix matches.
### YAML examples
```yaml
- label: "Diffusion · Qwen Image Test"
source_file_dependencies:
- tests/e2e/online_serving/test_qwen_image.py
- vllm_omni/diffusion/models/qwen_image/
source_file_dependencies: diffusion_qwen_image_function
# diffusion_qwen_image_function: *qwen_image + tests/e2e/online_serving/test_qwen_image.py
commands:
- pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model' ...
mirror_hardwares: h100_1
- label: "TTS · Qwen3-TTS CustomVoice Test"
source_file_dependencies:
- tests/e2e/online_serving/test_qwen3_tts_customvoice.py
- vllm_omni/model_executor/models/qwen3_tts/
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
- vllm_omni/deploy/qwen3_tts.yaml
source_file_dependencies: tts_qwen3_tts_function
# tts_qwen3_tts_function: *qwen3_tts + tests/e2e/online_serving/test_qwen3_tts_customvoice.py
commands:
- pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py ...
mirror_hardwares: l4_4
mirror_hardwares: l4_1
```
### Local dry-run
@@ -443,15 +448,16 @@ plugin, which provides the agent. `l4_*` jobs run on the EKS `l4-k8s` queue
still needs `mount-buildkite-agent: true`.
List both `run_cov_split.sh` and `pyproject.toml` in every opted-in job's
`source_file_dependencies` — both change what the job measures, so without them a
`source_file_dependencies` preset — both change what the job measures, so without them a
change there is filtered out of normal PR builds and only surfaces in a later
nightly. `tests/buildkite/test_upload_pipeline.py` covers the filter behavior with
a synthetic job (it does not pin real merge labels).
Editing only the surrounding CI YAML still does not schedule them, so a PR that
touches just the wiring needs a full E2E run (or the commands run on a GPU host)
to produce artifacts. When checking a new model's
artifacts, compare `lines-covered` between the online and offline XML rather than
just confirming both files exist.
If no job-key prefix matches, editing the pipeline YAML being uploaded, or any
path under the `source_filter_fallback` registry key, keeps every job in that
upload (including coverage jobs) so the wiring can be validated before merge.
When a job-key prefix already matches, bypass does not expand the selection. A
change to a different pipeline YAML still does not schedule them. When checking a new model's artifacts, compare `lines-covered`
between the online and offline XML rather than just confirming both files exist.
### Validation checklist
@@ -182,6 +182,8 @@ CUDA **L4** (`full_model`) jobs are defined once in [`.buildkite/cuda/test-night
- **Nightly:** scheduled `main` builds with `NIGHTLY=1` (and PR labels such as `nightly-test` / `omni-test`). Default fleet is typically H100.
- **Pre-release:** the same L4 suite runs again on extra GPU SKUs before a release (for example B200 via `MIRROR_HW=b200`), so `full_model` coverage is not limited to the nightly machine type.
AMD has an experimental, non-blocking `full_model` nightly vertical slice in [`.buildkite/amd/test-amd-nightly.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/test-amd-nightly.yml). It selects ROCm Qwen3-Omni function-expansion, accuracy, documentation-example, and AITER-on smoke coverage on the MI300 pool. AMD-only PR validation uses `amd-test` to admit the external AMD pipeline and `nightly-test` to select this suite; scheduled `main` builds select it with `NIGHTLY=1`. Keep it non-blocking while the lane accumulates stability and completion-time evidence.
How count-form `mirror_hardwares` and `MIRROR_HW` pick a SKU: [CI Settings](./ci_settings.md).
### L2 / L3 diff-aware CI (CUDA)
+17 -18
View File
@@ -131,13 +131,13 @@ for the validated batching, attention, and quantization combinations.
### MiniMax-H3: conditioning, denoising, and decode
MiniMax-H3 illustrates logical component and phase boundaries between text
encoding, task-specific denoising, and media decoding:
`Text Encoder → DiT → VAE Decoder`. The current registered topology co-locates
these components in one diffusion stage. These boundaries are useful for
profiling and offload; independent placement would require a future
multi-stage `PipelineConfig`. The stage mapping follows the implementation and
serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3)
MiniMax-H3 combines text and reference-media encoding, task-specific denoising,
and media decoding. Standard serving co-locates these components in one diffusion
stage. The opt-in
[disaggregated topology](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3-Disaggregated.md)
runs Qwen and the video/audio VAE encoders in Stage 0, then sends their conditioning
to a Stage 1 containing the DiTs and VAE decoders. The stage mapping follows the
implementation and serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3)
notes that a detailed technical report is forthcoming.
#### Model architecture by component
@@ -149,16 +149,15 @@ notes that a detailed technical report is forthcoming.
| **VAE Decoder** | Video VAE and audio VAE decode the generated latents and assemble the synchronized output | Decoded video frames and stereo waveform, with FPS and audio sample rate metadata |
Both DiT partitions remain behind one task-selection point; `task` chooses
which partition runs. In the current implementation, the three logical
components execute inside one diffusion stage. This is different from
Qwen3-Omni's three independently scheduled stages, even though both models
produce synchronized audio and video-like outputs.
which partition runs.
#### Representative component execution
The following table describes the standard single-stage deployment.
| Logical component | Batching | Attention and execution | Parallelism | Quantization |
| --- | --- | --- | --- | --- |
| **Text Encoder** | The current co-located pipeline follows the diffusion request scheduler; independent prompt/reference batching would require a separate multi-stage topology | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks; independent placement is not part of the current topology | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder |
| **Text Encoder** | Follows the diffusion request scheduler | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder |
| **DiT** | The current H3 implementation executes one generation request per diffusion batch; use concurrency for service-level throughput | cuDNN attention for the validated two-GPU consumer profile; TRTLLM attention or FlashAttention-4 on supported Blackwell profiles | TP2 plus text-encoder TP and Ulysses/VAE parallel groups on multi-GPU profiles; DLO/CPU offload trade memory for transfer time | Online FP8 applies to eligible DiT linears and is incompatible with layerwise offload |
| **VAE Decoder** | Decode follows each generation request; tile/patch work can be distributed even when denoising is not batched | VAE decode kernels rather than DiT attention | VAE patch parallelism and native tiled decode within the diffusion stage | BF16/FP32 baseline; the documented H3 FP8 path leaves both VAEs unchanged |
@@ -324,7 +323,7 @@ inputs, fields, and ownership rules are described below.
flowchart TB
layer1["Layer 1 · Authoring inputs<br/>PipelineConfig + DeployConfig"]
layer2["Layer 2 · Production resolution boundary<br/>resolve_omni_config()<br/>(StageConfigFactory + VllmOmniConfig internally)"]
layer3["Layer 3 · Startup hand-off<br/>OmniConfigResolution<br/>PipelineConfig + temporary OmegaConf stage bridge"]
layer3["Layer 3 · Startup hand-off<br/>OmniConfigResolution<br/>PipelineConfig + typed stage configs"]
layer4["Layer 4 · Runtime launch planning<br/>StageRuntime"]
layer5["Layer 5 · Engine materialization<br/>VllmConfig / OmniDiffusionConfig"]
@@ -336,10 +335,10 @@ The production resolution boundary is `resolve_omni_config()` in
It delegates typed construction to `StageConfigFactory.create_from_model()` and
`VllmOmniConfig.from_pipeline_config()`, then returns an
`OmniConfigResolution` consumed by both `AsyncOmniEngine` and headless startup.
Until `StageRuntime` consumes typed stage configs directly, this envelope carries
the effective `PipelineConfig` alongside OmegaConf-compatible `stage_configs` as
a temporary runtime bridge; both views describe the same resolved topology,
including injected stages.
This envelope carries the effective `PipelineConfig` alongside typed
`stage_configs`, which `StageRuntime` consumes directly. Both views describe the
same resolved topology, including injected stages. Backend arguments are
projected from these typed stages when each engine is initialized.
The legacy `stage_args` YAML path has been removed. Model topology now resolves
through `PipelineConfig`, with runtime overrides supplied by `DeployConfig`.
@@ -361,7 +360,7 @@ The important ownership rules are:
3. CLI and Python overrides are applied at the resolution boundary, with
per-stage overrides taking precedence over global values where supported.
4. `OmniConfigResolution` is the sole startup hand-off. Its `pipeline_config`
and temporary `stage_configs` compatibility view must describe the same
and typed `stage_configs` must describe the same
topology.
5. `StageRuntime` owns launch planning and replica lifecycle. `ReplicaInitPlan`
is runtime-private state, not a user configuration object.
@@ -76,6 +76,34 @@ preserve common `self` or `cross` policies. A model also decides whether its
path is compatible with automatic TRTLLM selection, because only the model
knows whether masking and packing satisfy the kernel contract.
### Model-owned implementation specializations
Use `Attention(..., impl_overrides={backend_name: implementation_class})`
when a model needs different computation within an existing backend contract.
The model defines the implementation in its own directory and passes it when
constructing the layer:
```python
# MyModelVSAImpl is a model-owned subclass of FastVideoVSAImpl.
self.attention = Attention(
num_heads=num_heads,
head_size=head_size,
softmax_scale=head_size**-0.5,
causal=False,
role="self",
impl_overrides={"FASTVIDEO_VSA": MyModelVSAImpl},
)
```
The override is applied after role/platform selection, keyed by the selected
backend's `get_name()`. It must subclass the **selected implementation**;
incompatible platform implementations raise `TypeError`. Unselected overrides
have no effect. Backend capabilities, constructor options, and shared
parallel dispatch remain unchanged and must be honored by the specialization.
Use a new backend for a new provider/capability contract. `custom_attention`
instead owns communication and requires `skip_sequence_parallel=True`.
## Adding or changing a backend
An implementation change is complete only when it:
@@ -91,8 +119,9 @@ An implementation change is complete only when it:
## FastVideo VSA model metadata
FastVideo VSA support is scoped to `FastVideo/FastWan2.2-TI2V-5B-Diffusers`, whose text-to-video and image-to-video modes both use `Wan22Pipeline`. It operates on a flattened DiT sequence but partitions tokens in
the original latent video grid. Wan integrations therefore attach the
FastWan's text-to-video and image-to-video modes both use `Wan22Pipeline`.
It operates on a flattened DiT sequence but partitions tokens in the original
latent video grid. Wan integrations therefore attach the
post-patch `(T, H, W)` shape as `vsa_dit_seq_shape` attention metadata. The
backend validates that `T * H * W` equals the sequence length, derives the
runtime block count, and selects the configured top-k key/value blocks for
@@ -104,3 +133,8 @@ the checkpoint contains those weights. Checkpoints without the projection do
not allocate or execute it. When top-k selects every block, native checkpoints
route to SDPA, while FastVideo DMD checkpoints preserve the VSA all-block path
to retain checkpoint semantics.
MiniMax-H3 supplies `MiniMaxH3VSAImpl` through `impl_overrides`.
`models/minimax_h3/attention/fastvideo_h3.py` owns its prefix layout, video tiling, and
learned compression gate. Pooling, block-map construction, and tile64 provider
calls are shared through `attention/ops/block_sparse.py`.
@@ -9,6 +9,7 @@ Backend-specific setup lives in separate docs:
- [MooncakeStoreConnector](omni_connectors/mooncake_store_connector.md)
- [MooncakeTransferEngineConnector](omni_connectors/mooncake_transfer_engine_connector.md)
- [MoriTransferEngineConnector](omni_connectors/mori_transfer_engine_connector.md)
- [NixlConnector](omni_connectors/nixl_connector.md)
- [YuanrongConnector](omni_connectors/yuanrong_connector.md)
- [YuanrongTransferEngineConnector](omni_connectors/yuanrong_transfer_engine_connector.md)
@@ -25,6 +26,7 @@ Current connectors operate in D2H2D (device to host to device) mode.
| Multi node (Mooncake Store) | MooncakeStoreConnector | TCP-based, requires Mooncake Master + metadata server. |
| Multi node (Mooncake RDMA) | MooncakeTransferEngineConnector | RDMA/TCP direct transfer with managed memory pool. Fastest. |
| Multi node (Mori RDMA) | MoriTransferEngineConnector | RDMA direct transfer via Mori IOEngine. |
| Multi node (NIXL) | NixlConnector | NIXL `READ` over UCX; the only backend covering Intel XPU. |
| Multi node (Yuanrong) | YuanrongConnector | Requires Yuanrong Datasystem + etcd. |
| Ascend NPU P2P (Yuanrong TE) | YuanrongTransferEngineConnector | Uses Yuanrong TransferEngine directly. Configure NPU device IPv4 and `memory_pool_device: "npu"`. |
+14 -2
View File
@@ -42,6 +42,11 @@ V1 does not include:
## Motivation and use cases
For investigating CPU memory retained by dependency tensor materialization,
see the [standalone safetensors diagnostic](https://github.com/vllm-project/vllm-omni/blob/main/benchmarks/host_weight_runtime/README.md).
It distinguishes repeated dependency calls from reuse of cached views and does
not establish per-request HWR leakage.
Model loading can create the same final host representation repeatedly. This
is especially expensive when loading performs checkpoint decoding, tensor
renaming, TP slicing, quantization, packing, or scale construction before GPU
@@ -313,8 +318,15 @@ One process per exact identity owns a build; other workers wait and then acquire
leases for the published artifact. Publication is invisible until all payloads
and metadata are validated, hashed, fsynced, and atomically renamed.
`coordination_timeout_seconds` bounds filesystem lock acquisition. It does not
cancel synchronous validation, a producer that has already started, or atomic
`coordination_timeout_seconds` bounds domain-initialization and lookup/build
lock acquisition. Store construction and each later resolution or publication
operation have separate budgets from the same wait policy, rather than one
end-to-end startup deadline. A domain-init timeout follows the retryable domain
failure policy below. After contention ends, a fresh runtime construction can
retry initialization; the timed-out runtime retains its original failure.
The coordination budget does not cancel filesystem I/O, synchronous validation,
a producer that has already started, or atomic
publication. A hung in-process producer therefore blocks its owning process and
must be handled by external process supervision. Enforceable producer
cancellation requires a future process-isolated producer contract.
@@ -0,0 +1,132 @@
# Model-local KV caches
Some models keep their attention KV as HuggingFace `transformers` cache objects
instead of using the engine's paged KV manager. Each such model declares what it
holds by implementing `model_local_kv_specs()`, and
`OmniGPUModelRunner.load_model` sums and logs the result after load.
## What is and is not already accounted for
This distinction decides what the protocol is for, and an earlier revision of
this page got it wrong by claiming none of this memory was accounted for.
CUDA-graph capture happens *during* `load_model` -- MiMo in its constructor,
Qwen3-TTS in `load_weights`. `determine_available_memory()` then samples
per-process GPU memory through NVML and subtracts it from the requested budget
(`vllm_omni/worker/base.py`). So the **graph-resident** portion is already
charged. Subtracting the declared total from the KV budget would double-count
it.
What is *not* charged is the **per-request** portion, which is allocated after
profiling. That half is not small: Qwen3-TTS's codec decoder is 4.44 MiB per
request, so a server at `max_num_seqs=256` reaches roughly 1.1 GiB that no
per-stage number predicted.
The consumer therefore reports and does not subtract.
## What the four models declare
| Model | Cache | Bounded by | Per row | Rows |
|---|---|---|---|---|
| Qwen3-TTS codec decoder (async-chunk only) | sliding `DynamicCache` | `sliding_window - 1 = 71` | 4.44 MiB | `max_num_seqs`, x2 for the working copy |
| Qwen3-TTS graph pool | retained captures | same | 4.44 MiB | fixed, from captured shapes |
| MiMo-Audio local transformer | `DynamicCache` | `group_size + max(delay_pattern) = 11` | 704 KiB | `max_num_seqs` |
| MiMo-Audio graph pool | captured | same | 704 KiB | fixed 261 (bucket sum) |
| MiniCPM-o Whisper encoder (streaming only) | `EncoderDecoderCache` | `embed_positions` rows = 1500 | 140.62 MiB | fixed 1, per session |
| ming_flash_omni talker | `StaticCache` | hardcoded `max_cache_len = 2048` | 24.00 MiB | fixed 1 |
**No cache is bounded by `max_model_len`.** Each has its own mechanism: a
sliding window, a decode-loop trip count, an encoder frame limit, a constant.
**ming's geometry does not exist in this repo.** `self._llm_config` is a
`Qwen2Config` from the checkpoint, so layers, kv-heads, head-dim and dtype are
only knowable after load. That is what makes this a runtime query rather than a
table.
## Lifetime is not multiplicity
`scope` records when memory appears and goes. It does **not** decide how wide
the allocation is, and an earlier revision that derived scaling from it was
wrong in two different directions at once:
- ming's talker is per-call but serialized -- `forward` takes
`runtime_additional_information[0]` and runs the whole AR loop inline -- so
scaling it by `max_num_seqs` over-reported it by that factor.
- MiniCPM-o's encoder is per *session*, not per sequence, so `max_num_seqs` is
simply the wrong number for it.
So width is declared explicitly, by naming the driver. `RowDriver` has two
values, because two are all the known caches need:
- `FIXED` -- a count the model controls, with `rows_fixed` and a required
`rows_reason`. `rows_fixed=1` on a cache that looks per-request is a claim a
reviewer must be able to check.
- `MAX_NUM_SEQS` -- one row per in-flight sequence; only the engine knows the
value.
A third value belongs here when a model actually has a cache the engine widens
by some other number. An earlier revision added one speculatively, for a single
declarer, and paid for it with a class hierarchy, an engine-capacity object and
a conditional-annotation field.
`rows` counts row-equivalents at peak and deliberately does not distinguish
"one allocation N rows wide" from "N allocations of one row". Those cost the
same, and a revision that tried to carry both got the object topology wrong in
three of four declarers: MiniCPM-o keeps one cache object per session while
asserting batch size 1, and both graph pools keep one distinct object per
captured bucket. The layout belongs in `allocation_note`, where it cannot be
mistaken for arithmetic.
## Declaring nothing is a valid declaration
A declaration must be inert when its path is. Qwen3-TTS holds KV only in
async-chunk mode -- stateless decoding runs `_forward_exact`, which calls the
transformer without `use_cache` -- so the stateless path declares nothing.
Getting this wrong is easy in both directions, and both have happened here: one
revision hung the declaration off the CUDA-graph wrapper and reported zero under
`enforce_eager`, where the eager async-chunk path does allocate; the next
declared unconditionally and invented roughly a gigabyte for stateless
deployments.
Where the model cannot see the deciding setting, it declares the honest unit
and says so in `allocation_note`. MiniCPM-o's encoder cache exists only on the
streaming path and is one object per session; the session cap is not visible
from the model, so it declares one session rather than scaling by a number it
would have to guess at.
## Finding the declarers
`collect_model_local_kv_specs()` walks `named_modules()`. Two wrinkles:
- The runner's `self.model` may be a `CUDAGraphWrapper`, which is not an
`nn.Module` but forwards attribute access to its runnable. The walk therefore
works either way, but entering through both the wrapper and its runnable would
count a root-level declarer twice. The collector calls the supported
`unwrap()` accessor first.
- An owner that is not a submodule at all (Qwen3-TTS's graph wrapper, ming's
`MingAudioGenerator`) is reached by one explicit hop from the module that
holds it.
## Adding a model
Return one entry per distinct allocation. Prefer `spec_from_hf_config()` and
pass the same config object the cache is built from; it raises rather than
guessing when an attribute is missing. `capacity_source` is diagnostic text --
never branch on it.
## Known limits
Row counts taken from live state after warmup miss caches `transformers` has
not yet materialized, which makes Qwen3-TTS's graph-pool figure a floor for the
xvec path.
MiMo's graph-pool cache has no reachable Python object -- it is a local inside
the captured frame -- so its width comes from the captured bucket list rather
than from tensors. Its two entries are also mutually exclusive per call: a call
either replays a bucket or runs eagerly. The sum is the right ceiling for the
process and the wrong number for a single request.
MiMo captures every bucket gated only on `torch.cuda.is_available()`
(`mimo_audio_llm.py:670`), so those 179.44 MiB stay resident even when the
deploy config sets `enforce_eager: true`. Gating it is a one-line change that
needs a model boot to verify, so it is noted rather than bundled here.
@@ -0,0 +1,89 @@
# Mooncake KV Transfer
## PR summary
This change adds an opt-in AR-to-DiT KV handoff using vLLM's native
`KVTransferConfig`, Scheduler-owned diffusion KV pages, and
`MooncakeConnector`. The existing `OmniKVTransferManager` path remains the
fallback when `kv_transfer_config` is absent.
The orchestrator creates one transfer ticket per request and resolves the
bound AR replica's Mooncake endpoint after generation. DiT allocates the
destination pages, waits for every worker rank to finish loading, and releases
the source and destination pages only after the connector lifecycle completes.
Request registration failures are rolled back before Worker dispatch. Receive
timeouts and cancellation terminate the request without recycling in-flight
pages: the scheduler retains them until every CFG row has completed on every
rank. Late completion is polled only while such requests remain. If a peer
never completes, its pages remain reserved until shutdown; other requests can
continue using the remaining capacity. Invalid pages, missing Worker ranks,
and failures without a proven rollback still stop the replica before teardown.
The example sets `transfer_timeout: 60.0` seconds explicitly. Cancellation
queued during a synchronous receive is processed when that receive completes
or times out; it does not interrupt the remote write.
## Reference configuration
| Stage | Replicas | Devices | Parallelism | KV cache |
| --- | ---: | --- | --- | ---: |
| AR | 2 | 0-3 | TP=2 per replica | 2 GiB per worker |
| DiT | 2 | 4-7 | TP=2 per replica, EP enabled | 1 GiB per worker |
The acceptance workload uses 512x512 images, two denoising steps, guidance
scale 2.0, and `max_num_seqs=1`.
For a four-GPU debug run, set both stages to one replica and assign AR TP=2 to
devices `0,1` and DiT TP=2 to devices `2,3`.
## Experiment
The final comparison used one node with eight NVIDIA RTX PRO 6000 Blackwell
Server Edition GPUs, TCP Mooncake transport, the same 50-prompt/seed T2I
dataset, concurrency 8, and two independent 100-request runs per
implementation. Each side completed 200/200 requests without an OOM or request
failure.
| Implementation | Run 1 (req/s) | Run 2 (req/s) | Mean (req/s) |
| --- | ---: | ---: | ---: |
| `OmniKVTransferManager` | 4.4103 | 4.4497 | 4.4300 |
| Native `MooncakeConnector` | 4.4457 | 4.4699 | 4.4578 |
Native throughput is **0.63% higher** than the legacy mean. The absolute
performance difference is below the 1% acceptance threshold.
For accuracy, a non-zero transferred prefix and the resulting attention output
were compared bit-for-bit with the legacy manager path:
```text
prefix_bits_equal True
attention_bits_equal True
max_abs_diff 0.0
```
The one-AR/one-DiT TP=2 debug topology also completed 20/20 sequential
requests. A separate real-TCP transport check transferred both CFG rows from a
TP=1 producer to a strict-Ulysses SP=2 consumer with exact values on both
target ranks.
## Scope and limitations
- The fully validated end-to-end topology is 2x AR TP=2 plus 2x DiT TP=2.
- Heterogeneous TP=1 to SP=2 reshaping is transport-tested, but arbitrary mixed
TP/CFG/SP end-to-end combinations have not been exhaustively validated.
- CFG-parallel end-to-end deployment is not part of this acceptance run.
- Native AR-to-DiT transfer currently requires `async_chunk: false`.
- TCP is covered by the acceptance run. The supplementary single-node RDMA
measurements use a separate short workload; they do not establish RDMA
support for multi-node deployment, which remains out of scope.
- The native path currently validates the two-stage `AR(0) -> DiT(1)` graph. Other stage graphs are rejected during startup until
their producer/consumer edge mapping is implemented.
- Sleep mode is disabled for the native paged path because registered KV pages
must remain mapped for the connector lifetime.
- Ascend NPU adaptation and its performance validation are maintained in a
separate follow-up PR; the GPU acceptance results above do not validate NPU.
- The homogeneous piecewise CUDA batching optimization is retained. A native
paged-attention CUDA graph capture/replay regression covers two CFG rows,
strided QKV, mixed causal/image spans, changed inputs, and output stability.
It must be run on CUDA hardware; CPU unit tests do not establish graph or
end-to-end performance acceptance.
+45 -8
View File
@@ -41,7 +41,8 @@ Every backend implements `OffloadBackend`:
- `is_enabled()` reports lifecycle state.
`disable()` does not promise to restore the pipeline's original device
placement. The caller owns any subsequent rematerialization.
placement. It must restore usable parameter and buffer storage before releasing
backend-owned host weights. The caller owns subsequent device placement.
An enable failure must remove every partially installed hook. Rank-local
backends restore ordinary tensors so the pipeline can be retried. A failed
@@ -54,9 +55,19 @@ must use distinct hook names and remove only hooks they own.
## Discovery and topology
`resolve_offload_plan(pipeline, config)` is the single topology entry point.
Every backend calls it once when `enable()` starts and then consumes the
resolved components, block stacks, resident heads, and staged-residency flags
it returns. Backends do not re-read model declarations.
The generic model-level and ordinary layerwise paths call it once when
`enable()` starts and consume its component selection and block stacks.
Ordinary layerwise offload also respects a resolved `skip_reason`, preserving
the legacy missing-DiT no-op without placing auxiliary components. Non-block
DiT and encoder state is placed by resolved block tensor identity, so the
backend does not rediscover block containers by attribute name.
Two compatibility paths remain outside this completed generic cutover:
model-level pipelines implementing `SupportsModelCpuOffload` still own their
custom phase lifecycle (RFC #6648 J4), and distributed layerwise offload still
reads declarations for nested-submodule staging and loader-owned mmap paths
(J3). The custom model-level protocol is delegated to before generic plan
resolution; it must not be subjected to the generic encoder/DiT swap checks.
Pipeline component discovery prefers `SupportsComponentDiscovery` declarations:
@@ -73,10 +84,22 @@ Block topology comes from the pipeline `OffloadPlan` first: `block_attrs` maps
each DiT path to its ordered block containers, `encoder_block_attrs` declares
streamable encoder stacks, `on_demand_component_paths` marks pipeline-managed
residency, `resident_dit_paths` marks the DiTs that may hold resident layers,
and `encoder_dlo_weight_replication` marks the encoders whose loader-produced
weights are safe for AllGather. DiTs absent from the plan fall back to
`_layerwise_offload_blocks_attrs` (including the deprecated singular-name
compatibility path).
`encoder_dlo_weight_replication` marks the encoders whose loader-produced
weights are safe for AllGather, and `offload_submodules` maps a DiT submodule
that owns its residency to its own block container. DiTs absent from the plan
fall back to `_layerwise_offload_blocks_attrs` (including the deprecated
singular-name compatibility path).
An undeclared DiT submodule joins the plan only when it is large enough that
keeping it resident would defeat streaming; its block container is then found
by scanning well-known attribute names, which warns once per submodule class.
Size is computed from parameter shapes and dtypes, including meta parameters,
so resolving before mmap loading preserves the same residency decision.
Block ownership validation includes nested components: a block shared with a
parent, sibling, or encoder is rejected before any hooks or storage are changed.
A submodule with no block container must implement the `load_to_device` /
`offload_to_cpu` lifecycle, and the resolver rejects it before any component of
the pipeline is placed or hooked.
The resolver is pure: it moves no tensor, installs no hook, writes no module
attribute, and reads no process group — multi-rank facts come from
@@ -102,3 +125,17 @@ remains responsible for transfer, synchronization, and storage ownership.
The diffusion [Offloader module design](../../module/diffusion/offloader.md)
describes how these feature contracts fit into the larger diffusion runtime.
## Backend acceptance tests
`tests/diffusion/offloader/test_backend_plan_contract.py` covers the two
behaviors the generic cutover changed. A DiT attribute aliasing a streamed
block must keep the ring's host residency, so placement cannot follow attribute
names; the CUDA case fails against the previous name-based placement. Both
backends must then run with their selector helpers and the model declaration
made unreadable, proving execution consumes the resolved plan alone.
Selection errors, rollback, residency, and enable/disable cycles stay in
`test_plan_resolver.py`, `test_layerwise_backend.py` and
`test_sequential_backend.py`. Model-specific phases and executed tied aliases
remain the J4 contract suite.
@@ -318,18 +318,18 @@ The same behavior applies to the Base and VoiceDesign Qwen3-TTS checkpoints
because they use the same Talker implementation.
!!! warning
Do not pass `--no-async-chunk` or enable prefix caching when you want this
optimization. Either change causes the runner to fall back to synchronous
output construction. Enabling async chunk alone cannot activate the feature
for a model that has not opted into async Omni output.
Do not pass `--no-async-chunk` when you want this optimization. Disabling
async chunk causes the runner to fall back to synchronous output
construction. Enabling async chunk alone cannot activate the feature for a
model that has not opted into async Omni output.
!!! note "Prefix cache compatibility"
Async Omni output materialization and Omni prefix caching cannot currently
run together. `_should_use_async_omni_output()` returns `False` whenever
`self.omni_prefix_cache` is present, so prefix caching selects synchronous
output materialization. Supporting both features requires snapshotting or
otherwise synchronizing prefix-cache merge and update state before the
background output path can consume it safely.
Prefix cache and async Omni output can run together. `save_outputs` returns
a `step_id`; the output builder later calls `materialize(sid, req_ids)` with
the save-time request list, which may be a step late relative to the live
`input_batch`. `_should_use_async_omni_output()` does not disable itself
when the cache is present. See
[Automatic Prefix Caching](prefix_caching.md#implementation).
## Compatibility and Fallbacks
@@ -341,10 +341,9 @@ runtime conditions hold:
| AR async scheduling is enabled | The optimization relies on the scheduler advancing while the prior output is materialized |
| `async_chunk` is enabled | The feature targets incremental downstream Omni payloads |
| The model stage opts in with `use_async_omni_output` | Models must declare that their output lifecycle is safe to defer |
| Omni prefix cache is disabled | Prefix-cache merge and update ordering currently requires synchronous materialization |
| Speculative decoding is disabled | Speculative output state is not included in this deferred path |
| Routed-expert output is disabled | Routed-expert extraction currently requires the synchronous path |
| Postprocess is absent or explicitly runs eagerly | State needed by the next decode step must be updated before the runner returns |
| Postprocess is absent or explicitly runs eagerly | State needed by the next decode step must be updated before the runner returns. Eager postprocess sees the scheduled slice (`None, None` combined tensors), not the prefix-cache-merged full prompt; it must only consume the tail |
These checks are evaluated per stage on runners based on `GPUARModelRunner`.
An unsupported combination does not prevent serving; it only falls back to
@@ -380,6 +379,8 @@ so Omni payload materialization remains synchronous.
- `tests/worker/test_gpu_ar_model_runner.py`: Snapshot, guard, connector
ordering, and background error-propagation tests.
- [Async Chunk](async_chunk.md): Inter-stage chunking and scheduling design.
- [Automatic Prefix Caching](prefix_caching.md#implementation): Prefix-cache
`step_id` / `materialize` contract used by the deferred output builder.
- [Qwen3-Omni optimization blog](https://vllm.ai/blog/2026-07-01-qwen3-omni-optimization):
Optimization context and controlled performance results.
- [PR #4476](https://github.com/vllm-project/vllm-omni/pull/4476): Feature
@@ -0,0 +1,135 @@
# NixlConnector
## When to Use
Multi-node or intra-node stage transfer over NIXL, which brokers RDMA/shared-memory
transports through a single API. Useful when the deployment already standardises on
NIXL for KV transfer, or when the target accelerator has no Mooncake/Mori backend --
NIXL reaches Intel XPU through UCX's Level Zero support.
## Mechanism
Uses vLLM's `NixlWrapper` (`vllm.distributed.nixl_utils`) to register the producer's
tensors and let the consumer pull them with a NIXL `READ`.
- Data Plane: NIXL agent-to-agent `READ`, GPU-to-GPU where the backend allows it.
- Control Plane: a ZMQ ROUTER socket atomically claims the source allocation before
returning descriptors. Forwarded `put()` metadata includes the generation and
endpoint; the consumer claims that generation before submitting any READ.
Transfer metadata uses schema version 1. Tensor descriptors are grouped by NIXL
memory type (`DRAM`, `VRAM`, and so on), while each descriptor retains its global
tensor index so mixed CPU/accelerator payloads can be reconstructed in order.
Payloads are not restricted to tensors. A single tensor or a list of tensors is
transferred as-is; a nested structure has its tensor leaves extracted and shipped
alongside a msgpack-encoded skeleton; anything else is msgpack-encoded into one
uint8 tensor. Mapping and sequence structure is preserved. `msgspec.Struct` values,
including `OmniPayloadStruct`, are normalised to mappings so their tensor leaves use
the direct NIXL path rather than CPU object serialisation.
Each NIXL agent uses strict thread synchronization. A consumer performs one short
metadata query per scheduler poll (10ms by default), which prevents an unavailable
producer from blocking unrelated requests. The regular transfer-completion handshake
keeps the configured transfer timeout.
## Installation
CUDA hosts can install the published wheel:
```bash
pip install nixl
```
For Intel XPU, use `docker/Dockerfile.xpu`. Its upstream vLLM XPU base image
provides a matching torch-xpu, UCX with Level Zero support, and NIXL runtime.
## Configuration
```yaml
connectors:
nixl_connector:
name: NixlConnector
extra:
host: "auto"
zmq_port: 50061
backends: ["UCX"]
stages:
- stage_id: 0
output_connectors:
to_stage_1: nixl_connector
- stage_id: 1
input_connectors:
from_stage_0: nixl_connector
```
Parameters:
- `host`: address the producer binds its handshake socket to (`"auto"` to detect).
- `zmq_port`: handshake port. If a producer omits it, an ephemeral port is opened
for ownership claims and returned in `put()` metadata. The value in the deploy YAML is a
base port. Purpose, replica, rank, and producer-stage offsets are added centrally,
so colocated connectors and tensor-parallel workers receive distinct endpoints
without one config entry per rank.
- `sender_host` / `sender_zmq_port`: consumer-side override naming the producer's
handshake endpoint. Only needed when the consumer cannot learn it from metadata.
In replicated deployments, the producer endpoint is carried with each request and
selected at `get()` time; it is not mutable connector-wide state.
Explicit sender ports are preserved, not recomputed using the receiver replica.
Intermediate NIXL stages retain their incoming receiver configuration and a
separate outgoing bind configuration, so one worker can receive and publish.
- `backends`: NIXL backends to register memory with. Defaults to `["UCX"]`.
- `receive_device`: forces where received tensors land. By default the consumer
keeps the producer's device *type* but uses its own current device of that type.
- `memory_type`: overrides the NIXL memory type, otherwise `DRAM` for CPU tensors
and `VRAM` for accelerator tensors.
- `lease_seconds`: how long a `put()` payload stays registered while waiting to be
read (default 3600). The consumer reports completion, so this only bounds payloads
nobody has claimed. An internal reaper releases only unclaimed registrations.
`VLLM_OMNI_NIXL_LEASE_S` overrides it.
- `transfer_timeout_s`: how long a `get()` waits for its `READ` to complete
(default 300). NIXL 1.3 has no transfer cancellation API, so a timed-out transfer's
buffers and registrations remain owned by the connector until NIXL reports a
terminal state; `close()` waits for that state before releasing them.
`VLLM_OMNI_NIXL_XFER_TIMEOUT_S` overrides it.
### Source ownership and failure limits
Claims and expiry use the same lock. Completion identifies both a payload
generation and a unique READ claim; duplicate or stale ACKs cannot release a
different reader or replacement payload. Replacing a claimed key is rejected.
Terminal errors also release claims; timeout, unknown status, and active sibling
transfers retain the complete destination bundle and source claim until terminal.
Zero-element tensors retain their shape/dtype/skeleton positions but never create
DMA descriptors or registrations.
A lost metadata response, lost completion ACK, or abandoned consumer can retain a
claim indefinitely. NIXL 1.3 cannot prove remote cancellation, so neither TTL nor
`cleanup()` frees those allocations. Producer `close()` rejects new work and
retains its agent, listener and claimed allocations; a later `close()` finishes
teardown after claims drain. Permanently abandoned claims remain until process
exit. This favors memory safety over bounded shutdown/memory usage. Only trusted
peers may access this unauthenticated control plane. Both endpoints must use the
claim-aware protocol; rolling compatibility with older GET_META clients is not
provided. Legacy externally-owned metadata without a generation is accepted only
under the caller's lifetime guarantee, not managed as a leased allocation here.
## Validation
The native CUDA smoke test requires NIXL and at least two visible CUDA devices:
```bash
UCX_RCACHE_MAX_UNRELEASED=1024 pytest -o addopts='' -q -s \
tests/distributed/omni_connectors/test_nixl_connector_native.py
```
It starts independent producer and consumer processes, loads the real UCX backend,
performs the ZMQ `GET_META` handshake and a native NIXL `READ` from GPU 0 to GPU 1,
checks mixed CPU/GPU tensor leaves from an `OmniPayloadStruct`, sends `XFER_DONE`,
and verifies that producer registrations are released. This validates same-host,
cross-GPU operation. Inter-node UCX transport is supported by the design but was not
validated by this test.
For more details, refer to the [NIXL repository](https://github.com/ai-dynamo/nixl).
+178 -15
View File
@@ -1,6 +1,5 @@
# Automatic Prefix Caching in Omni Models
---
## Table of Contents
@@ -9,6 +8,8 @@
- [High-Level Approach](#high-level-approach)
- [Example](#example)
- [What About Multimodal Inputs?](#what-about-multimodal-inputs)
- [Implementation](#implementation)
- [Related Files](#related-files)
---
@@ -27,6 +28,7 @@ vLLM implements automatic prefix caching for managing its kv-cache, which is bes
This document describes vLLM-Omni's mechanism for caching tensor outputs that are meant to be passed between stages, when requests have common prefixes, similar to the way in which vLLM has prefix caching for the kv-cache. This works in conjunction with vLLM's multimodal encoder caching, but is distinct. See the final section for a concrete example for how they tie together in practice.
### High-Level Approach
!!! note "Note 2"
Prior to reading this section, it's recommended to take a look at the design documents in vLLM for [Automatic Prefix Caching](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/), which will make some of the concepts more clear.
@@ -38,8 +40,10 @@ The main focus of vLLM-Omni's approach to prefix caching stage outputs is to bui
With this in mind, consider the set of blocks in a 2D layout, where the row represents the index of blocks being considered, and the columns represent the slots corresponding to tokens within each block. Since we know the `num_blocks` and `block_size` from our kv cache config, if we want to cache a tensor with feature size `D`, we can preallocate a CPU tensor of size `(num_blocks, block_size, D)`, and use the same block index and slot mapping to retrieve the corresponding feature vector.
Host footprint: each cached key costs `num_blocks × block_size × D × dtype_bytes` of **pinned** CPU memory (pinned so device→host can overlap compute), allocated on the first `save_outputs` that sees the key — the first real request pays the `cudaHostAlloc`. Measured on a Qwen3-Omni deployment: thinker `__hidden_states__` `[15092, 16, 2048]` bf16 ≈ 0.92 GiB (plus the same again for each `hidden_states.layer_*` key a model exposes), Qwen3-TTS talker `codes.audio` `[16180, 16, 16]` ≈ 33 MiB. Budget host RAM for the stage accordingly.
### Example
!!! note "Note 3"
Prefix caching in vLLM-Omni currently is only supported on AutoRegressive stages with one kv-cache group. Configure it with the pipeline-wide `enable_prefix_caching` field in the deploy config.
@@ -56,7 +60,7 @@ The prefix cache flow is then outlined below.
2. Say we process the request `The quick brown fox was tired and slept beneath the shady tree`, which is 12 tokens and evenly divides into 3 blocks as shown below.
```
```text
[ The quick brown fox ] [ was tired and slept ] [beneath the shady tree ]
Block 1: |<--- block tokens ---->|
Block 2: |<------- prefix ------>| |<--- block tokens --->|
@@ -65,10 +69,9 @@ Block 3: |<------------------ prefix -------------------->| |<--- block tokens -
When the request processes, we inspect the multimodal outputs and identify the `mm_feature` tensor, which will be of shape `(seq_len, feature_dim)`, i.e., `(12, 16)` in this example. We note that the first axis is dependent on the `seq_len` and add a new cache_tensor of shape `(num_blocks, block_size, feature_dim)` to our multimodal cache for tensors.
1. If we lay out the cache as a 2D tensor of shape (`num_blocks`, `block_size`), we'll have something like the following:
3. If we lay out the cache as a 2D tensor of shape (`num_blocks`, `block_size`), we'll have something like the following:
```
```text
0: [ The quick brown fox ]
1: [ was tired and slept ]
2: [beneath the shady tree ]
@@ -78,7 +81,8 @@ When the request processes, we inspect the multimodal outputs and identify the `
```
Or, if we flatten it down to 1D,
```
```text
0: The
1: quick
2: brown
@@ -90,7 +94,8 @@ Or, if we flatten it down to 1D,
```
which we can think of as row indices into the hidden states tensor if we view it as the 2D shape `(num_blocks x block_size, feature_dim)`. That is, the analogous flattened (from 3D -> 2D) mapping of the cache for hidden states becomes the following.
```
```text
0: <hidden states vector of len 2 corresponding to 'The'>
1: <hidden states vector of len 2 corresponding to 'quick'>
2: <hidden states vector of len 2 corresponding to 'brown'>
@@ -103,10 +108,9 @@ which we can think of as row indices into the hidden states tensor if we view it
Similarly, for the multimodal outputs cache, the flattened coordinates are the same, but the `mm_feature` maps to vectors of length `16` instead of the hidden size of `2`. Note that in practice, we may have multiple multimodal output tensors per forward pass, which may have different names and different feature dimensions.
1. Now, say that we receive a new request `The quick brown fox jumped over the dog`.
4. Now, say that we receive a new request `The quick brown fox jumped over the dog`.
```
```text
[ The quick brown fox ] [ jumped over the dog ]
Block 1: |<--- block tokens ---->|
Block 2: |<------- prefix ------>| |<--- block tokens --->|
@@ -116,7 +120,7 @@ Here, we will have a cache hit for `Block 1` which will be detected by vLLM base
Since we have the block indices / slot mappings from the kv cache manager, we can simply mirror the mappings and leverage the same indices for the cached hidden states and multimodal outputs. This allows us to look up the correct tensors from our externally maintained 3D caches.
```
```text
0: [ The quick brown fox ] < already in the cache
1: [ was tired and slept ]
2: [beneath the shady tree ]
@@ -129,11 +133,11 @@ Since we have the block indices / slot mappings from the kv cache manager, we ca
Finally, to pass the full hidden states and multimodal outputs to the next stage, we simply concatenate the cached contents with the corresponding new tensors computed from the current forward call.
### What About Multimodal Inputs?
It's also useful to consider the case about how Omni prefix caching is handled when we have multimodal inputs that don't cleanly end on block boundaries, as well as how this works with multimodal encoder caching in vLLM. For example:
```
```text
[ Im0 Im1 Im2 Im3 ] [ Im4 Im5 foo <empty> ]
Block 1: |<--- block tokens ---->|
Block 2: |<------- prefix ------>| |<--- block tokens --->|
@@ -147,9 +151,9 @@ In reality, this isn't a big problem for correctness, because vLLM also maintain
- The hash describing the image data starting at position 0 and with length 6
- In vLLM's encoder cache, a mapping from the image hash above to the encoder output
To understand what happens, say we get the following input as a second request:
```
```text
[ Im0 Im1 Im2 Im3 ] [ Im4 Im5 bar baz ]
Block 1: |<--- block tokens ---->|
Block 2: |<------- prefix ------>| |<--- block tokens --->|
@@ -162,3 +166,162 @@ Because we have multimodal data in a scheduled span that isn't fully precomputed
When we pass our multimodal tensors to the language model component in the same stage, we'll then expect the same outputs, because the prefix caching behaviors in vLLM-Omni / vLLM match, so the LLM will use vLLM's KV cache manager's prefix caching to correctly handle the attention information for `Block 1` while calculating the outputs for `Block 2`, giving us the correct results for processing `Block 2` with the context of `Block 1`.
Finally, we look up the output hidden states/multimodal tensors corresponding to the prefix cache hit `Block 1` and concatenate it with the forward pass result to get the final result, which is expected to be identical to the full hidden states when prefix caching is disabled.
### Implementation
The block/slot model is `vllm_omni/core/prefix_cache/`.
`OmniPrefixCacheManager` owns slot occupancy, the request-task table,
hit spans, and merge.
`OmniPrefixCacheController` moves data: a reusable `StagingBufferPool` for
this step's device→host copy, and writes into the durable `PrefixBlockPool`.
The state lock covers those tables only.
Miss is not an error (this step's forward slice only). A hit span whose
hidden rows are absent is fatal. For mm keys the rule is looser: a model may
emit a key only for some requests, so a hit span with no rows behind an mm key
reads zeros from the pool for those positions rather than raising. Abort still
writes: once a hash entered this step's batch it must land in the cache.
`enable_prefix_caching` is refused on KV-consumer stages (`kv_role` of
`kv_consumer` or `kv_both`). KV received from a producer is reported as
`num_computed_tokens` too, and the manager cannot tell it from a local hit.
Producer-only stages are unaffected. Pooling stages never save, so they get
no cache. This gate is one function, `stage_prefix_cache_config`, called by
both the GPU and the NPU model runner at kv-cache init.
Which stages may set `enable_prefix_caching: true`:
| Stage | `enable_prefix_caching` | Why |
| --- | --- | --- |
| AR stage with one full-attention kv group whose hidden states / per-token mm feed the next stage (Qwen3-Omni thinker and talker) | supported | The case the cache is built for: full-prompt hidden states are merged from the pool on a hit. |
| AR stage that sets `requires_full_prefix_cached_hidden_states = False`, optionally with `deferred_prefix_cache_mm_keys` (Qwen3-TTS talker, Higgs v3 talker) | supported | Hidden is not cached; deferred codec rows are written once on finish. |
| Pooling stage | ignored | Never saves; the gate returns no config. |
| `kv_role: kv_consumer` / `kv_both` | refused at kv-cache init (`OmniPrefixCacheUnmatchError`) | Producer KV shows up as `num_computed_tokens` and is indistinguishable from a local hit. |
| Speculative decoding on the stage | refused at kv-cache init | Under async scheduling vLLM keeps `num_computed_tokens_cpu` optimistic (all drafts accepted) during the forward and corrects it afterwards; `step_slots_cpu` would mirror rows at the wrong slots. Not verified; refused as a whole. |
| `prefix_match_unit` smaller than `block_size` | refused at kv-cache init | Sub-block hits make `num_computed_tokens` unaligned; the hit registry only mirrors whole blocks. |
| Hybrid / sliding-window / multi-group kv cache (e.g. a talker with `attention_type: sliding_recompute`) | refused at kv-cache init | The cache mirrors exactly one full-attention block table. |
| Attention backend whose kernel block size differs from `--block-size` (FlashInfer / FlashMLA / CutlassMLA with a block size they do not list natively), or decode context parallel | refused at first step (`FullAttentionGroupView`) | `step_slots_cpu` computes `table[req, pos // block_size] * block_size + pos % block_size` over allocator block ids; hybrid kernel blocks and DCP token striping change that row layout. FlashAttention / Triton accept any multiple of 16 and never split blocks. |
| Codec decoder / Code2Wav stages (Qwen3-Omni stage 2, Qwen3-TTS stage 1) | keep `false` | Nothing downstream consumes their hidden states; the cache would only add device→host copies. Not validated. |
| Diffusion stages | n/a | No vLLM KV cache to mirror. |
Hit spans come from `scheduled_new_reqs` only, as in the pre-refactor cache:
- A new request with a (partial) prefix hit is the normal path: the hit
blocks are read from the pool, the rest is this step's rows, and the
divergent tail is written under its own blocks. The gather starts on the
prefetch thread at `new_step_starts`; a same-step hit (vLLM hashes blocks
at schedule time, so `b` can hit blocks `a` computes in the same forward)
cannot plan until `a`'s write is registered and starts at `save_outputs`
instead. Either way it runs before the next step can hand those blocks
to a new tenant. vLLM frees blocks one step before `finished_req_ids`
arrives, so a finished request whose last step materializes late may
still find a hit slot reassigned. Each `(slot, key)` carries a write
version, bumped whenever a new write claims it. A planned read captures
that version and is registered in `_pending_reads`; a later write that
reclaims those slots copy-on-writes the still-`COMMITTED` rows into the
ref before it overwrites the pool, so a delayed fetch serves the original
tenant. A version mismatch with no preserved copy raises for live and
finished alike (the pool rows are a newer tenant's). Production defaults
stay opt-in until preempt/resume hit spans are reconstructed.
- `async_chunk` continuation: when the next upstream chunk arrives, the same
request id re-enters `scheduled_new_reqs` with `num_computed_tokens` equal
to what it already computed itself. Ids already in `live_reqs` are skipped
for hit marking: those rows were delivered in earlier steps and re-emitting
them would duplicate output. A `delivered_upto` span for this case is
Phase 2.
- Preemption + reschedule: vLLM resets `num_computed_tokens` to 0 on
preemption and re-runs prefix matching on resume, so the resumed request
can come back with a fresh hit. With the V1 model runner it arrives
through `scheduled_cached_reqs` (id in `resumed_req_ids`, `new_block_ids`
replaces the table); with the V2 runner it re-enters `scheduled_new_reqs`
while still in `live_reqs`. Neither path marks an omni hit span: the
resumed request gets only the rows it recomputes, and its still-open
deferred write keeps appending (a slot written twice keeps the later
chunk). Cache integrity holds either way — the hit blocks already have
rows, from this request or the one it hit. Stages that need full prompt
hidden states should be sized so preemption does not occur while prefix
caching is on. Same as before this refactor; tracked for Phase 2.
Two write paths, split by `ModelCachePolicy.deferred_keys`:
- Immediate (`JOIN_NEXT_STEP`): save launches a whole-step device→host into
a staging slot; the committer waits that event and writes the CPU pool.
The next save waits `done` (the pool write): a reused slot must never
leave a pending pool write behind, or a delayed hit read of the old rows
would find nothing recoverable.
- Deferred (`JOIN_ON_FINISH`): mm whose first dim is this step's token count
stays on a per-request GPU clone; `_WriteChunk`s append across steps;
finish/abort (or GPU-byte-budget pressure) forces the copy. One open
WriteTask per request; a budget flush may close it mid-request, in which
case the next save opens a new one (`write_n` + 1) and a hit reads both,
so one long request cannot pin the whole budget.
A `WriteTask` moves through `TaskState` only via `transition()`, one step at
a time along a strict chain:
`PENDING` (registered, not queued) → `QUEUED` (on a copy queue) →
`COPYING` (one thread owns the copy stage) → `HOST_READY` (host rows ready) →
`WRITTEN` (in the CPU pool); `FAILED` is reachable from any non-terminal
state. Skipping a step raises. The worker claims `COPYING` in the same `_wake`
critical section as the queue pop, so `escalate` never re-queues a task it can
see is already claimed. `host_ready` / `done` are wait primitives set by the
transitions into `HOST_READY` / `WRITTEN` / `FAILED`. GPU-clone bytes are
charged once per clone on a `_BudgetTicket` pinned by every task that views
it, and uncharged when the last holder releases (idempotent per tid).
Mm whose first dim equals the unpadded scheduled length *or* the CUDA-graph
padded length is registered on first sighting. Talker `codes.audio` is a
cat of scheduled rows and stays unpadded while hidden is padded; both must
open a pool key, whether immediate or deferred.
Leftover mm (lists, `codes.ref`, any tensor whose first dim is not this
step's token count) is copied to CPU at save without truncating that first
dim. That leftover copy is the async-builder read replica for this step; it
does not write the pool or carry abort/preempt occupancy.
```python
cache.register_policy(ModelCachePolicy.from_model(model)) # load_model
cache.new_step_starts(scheduler_output) # before _update_states
sid = cache.save_outputs(hidden, mm_outputs, num_tokens_unpadded=n,
num_tokens_padded=n_pad)
outs = cache.materialize(sid, req_ids) # or discard_step(sid)
```
Each step id is consumed exactly once. `req_ids` must be a subset of the save
snapshot. At most `staging_depth` unused step ids may exist at once: every
`save_outputs` claims one staging slot, including saves with only leftover
mm that copy no device→host page. A slot is also held by each immediate
write that views it, until the committer's pool write. A later save waits
for `materialize`/`discard_step` or that pool write to free a slot;
`staging_claim_timeout_s` then errors with the unused ids and the task count.
`join`/`join_host_ready` use the same bound and raise with the stuck task's
id and state instead of hanging the caller.
`materialize` may run on the async output builder after the engine has
entered the next step; leftover mm (not written to the pool) is copied
to CPU at `save_outputs` so the builder never reads live CUDA-graph
buffers. See
[Async Omni Output Materialization](omni_async_output_materialization.md).
The cache is constructed only on the last pipeline-parallel rank
(`_ensure_omni_prefix_cache`). Other ranks skip it: they never call
`save_outputs`, so a hit table there would resolve to absent slots.
Threads, locks, and what each may block on:
| Thread | Role | May block on | Must not hold while blocked |
| --- | --- | --- | --- |
| Engine | `new_step_starts`, `save_outputs` | previous-step `join` (`done`) / finished-write `join_host_ready`; `reserve()` GPU-byte flush; staging-slot claim; `dispatch()` / finish-abort `escalate()` (eager mode: the copy + pool write run inline) | `_state_lock` |
| Async output builder | `materialize` (may overlap the next engine step) | this step's `step_d2h_event`; `join` (`done`); deferred `fetch_host` | `_state_lock` |
| Committer | `_worker_loop`: wait device→host / deferred copy / pool write | `_wake.wait`; `step_d2h_event` or copy-stream sync | never takes `_state_lock` |
| Prefetch pool | hit-span gather during forward | `join` (`done`); deferred `fetch_host` | `_state_lock` |
| Lock | Covers | Does not cover |
| --- | --- | --- |
| manager `_state_lock` | occupancy tables, step contexts, hit spans, task registration, pool-key publish (`install_key`) | join, GPU-byte flush, copy, `step_d2h_event` wait, pool-key allocation, eager `dispatch()` / `escalate()` |
| controller `_lock` / `_wake` | task registry, queues, GPU-clone byte budget | device→host / pool-write body (released before `synchronize`) |
| `WriteTask.lock` | `state`, `reassigned`, `append_chunk` | waiting on `host_ready` / `done` (those are events) |
### Related Files
- `vllm_omni/core/prefix_cache/`
- `vllm_omni/worker/gpu_model_runner.py` (`_ensure_omni_prefix_cache`)
- `tests/core/test_prefix_cache.py`
- [Async Omni Output Materialization](omni_async_output_materialization.md)
+32 -2
View File
@@ -143,8 +143,20 @@ fraction to select additional resident sessions up to the model-declared cap.
All model-owned reservations are deducted before the paged self-attention pool
is allocated.
LingBot reports its persistent full-horizon image-condition tensor. DreamZero
reports the measured 603 MiB per-session Wan VAE causal-convolution state.
LingBot advances a session-owned causal Wan VAE encoder for each condition
block: the opening pixel frame is followed by four zero pixel frames per later
latent frame. It retains one current condition block, with separate committed
and in-flight encoder histories so a failed block cannot overwrite committed
context. The state reservation includes both encoder histories, derived from
the encoder's convolution input grids, and the streaming decoder cache. These
caches are released through the same close/reset hooks. Condition encoding
currently requires the unpatched, non-tiled Wan encoder path; the existing
decoder's tiling fallback remains independent. Real-weight GPU parity with a
full-horizon encode still requires validation because the pointwise posterior
projection now runs per block.
DreamZero reports the measured 603 MiB per-session Wan VAE causal-convolution
state.
## Concurrency and routing limits
@@ -178,6 +190,24 @@ For LingBot World v2:
- `lingbot_world/pipeline.py` constructs conditioning, owns small non-KV session state, and
produces one block plus the standard metadata envelope.
## Ulysses sequence parallelism
LingBot supports pure Ulysses sequence parallelism for both direct and
AR-Diffusion execution. With Ulysses degree greater than one, hidden tokens,
camera features, token-expanded timestep modulation, and RoPE tables are sharded
together. Without SP, timestep modulation retains the frame-broadcast path.
Self-attention performs the sequence-to-head all-to-all before reading or writing
paged KV, while static text K/V uses the same local head shard. Text K/V shards
own compact storage; cross-attention exchanges query/output layouts to use these
shards. This retains the shared cache geometry and reduces text K/V storage at
the cost of two all-to-all calls per layer.
The output head projects local tokens before gathering the flow values. For
the 14B model this reduces the gathered width from 5120 to 64; frame modulation
uses each shard's global token offset, including shards that split a frame. Only
`ulysses_mode="strict"` is supported; `advanced_uaa`, Ring, and AllGather-KV modes
remain unsupported for this model.
## Non-goals
This contract does not currently provide:
+4 -2
View File
@@ -77,7 +77,9 @@ Use it when plain Ulysses-SP would otherwise fail because:
- `auto_pad=True` pads sequence tokens in `_sp_plan` and requires attention backends that support `attention_mask`.
- `advanced_uaa` does not depend on mask-based token padding inside Ulysses attention. It is therefore a better fit for non-divisible head counts and uneven Ulysses shard sizes.
- `auto_pad=True` remains incompatible with Ring attention because the ring backend does not consume `attention_mask`.
- `auto_pad=True` remains incompatible with Ring attention because the ring backend does not consume arbitrary
`attention_mask` values. A model with contiguous suffix padding may instead publish `valid_kv_length`; Ring trims
each circulated K/V block against that global valid prefix.
- `advanced_uaa` is still experimental and hybrid mode remains limited by Ring's equal-shape requirement.
---
@@ -416,7 +418,7 @@ _sp_plan = {
| Constraint | Description |
|------------|-------------|
| **Attention Backend Compatibility** | The attention backends must support `attention_mask`. Currently only `TORCH_SDPA` and `FLASH_ATTN` (default for diffusion models) are supported. |
| **Ring Attention Limitation** | Ring attention does not support `attention_mask`. Therefore, when using `auto_pad=True`, the combination of Ulysses + Ring attention is not feasible. |
| **Ring Attention Limitation** | Ring attention does not support arbitrary `attention_mask` values, so `auto_pad=True` remains incompatible. Contiguous suffix padding is supported when the model publishes the global `valid_kv_length` prefix. |
1. Enable `auto_pad=True` for all sequence-dimension inputs in `_sp_plan`:
```python
+110 -2
View File
@@ -20,7 +20,7 @@ The target is the engine-native path:
-> PersonaPlex Talker
-> streaming PersonaPlex Code2Wav
-> PersonaPlex data-plane projector
-> response.audio.delta + response.audio_transcript.delta
-> response.output_audio.delta + response.output_audio_transcript.delta
```
The standalone `/api/chat` and `/v1/audio/duplex` server that accompanied the
@@ -197,7 +197,7 @@ model-neutral native results:
It owns per-request audio and text cursors so a cumulative output cannot replay
old audio. The generic projector turns each native result into the Realtime
`response.audio.delta` / `response.audio_transcript.delta` pair (and
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair (and
`response.output_text.delta` for text) under one `response_id`; the full
mapping is the name map in the
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide.
@@ -271,3 +271,111 @@ The required evidence is:
Audio that is empty, all zero, non-finite, or only a protocol `listen` event is
not a successful end-to-end result.
### Paced multi-session load driver
The existing driver keeps its two-session admission and slot-reuse validation
when `--sessions` is omitted. Passing `--sessions N` selects a separate,
bounded load mode; it does not replace the lifecycle test or change server
admission capacity. Start an existing PersonaPlex server configured to admit
the requested number of sessions, then run from a source checkout with the
vLLM-Omni client dependencies installed:
```bash
python -m tests.e2e.online_serving.personaplex_realtime_duplex \
--model nvidia/personaplex-7b-v1 \
--input-wav /path/to/input.wav \
--sessions 2 --load-frames 1000 --drain-s 2 \
--server-revision SERVER_COMMIT --server-hardware A100-80GB \
--output-dir /tmp/personaplex-load-n2
```
Use a new output directory for every run. The driver refuses to overwrite an
existing `load-result.json`. The server labels are supplied by the operator,
not detected or verified by the client. Record the server command, package
versions, checkpoint revision and hardware alongside a published measurement.
All attempted sessions finish admission before admitted sessions share a
common input start. A rejected session remains a failed row in the report;
it is not removed from the denominator. Each session receives at most
`--load-frames` input frames, without looping the WAV, and uses 80 ms pacing.
A stalled send cannot cause a catch-up burst. The timeline records send
schedule lateness so an overloaded client is visible rather than mistaken
for a faster server. This mode measures steady concurrency, not join/leave
interference; the original lifecycle mode still checks slot reuse.
Load mode does not probe an additional N+1 session for `resource_exhausted`;
the lifecycle mode owns that admission-limit check. A per-session
`client_pacing_warning` and the summary's `client_pacing_warning_sessions`
identify send-lateness p99 above one nominal frame (80 ms). This is a warning,
not an extra acceptance gate: read it alongside server-facing measurements
so a slow client is not mistaken for a slow server.
The fixed drain window is part of the measurement configuration. Audio
arriving during cleanup is excluded, so closing cannot rescue a session that
missed its observation deadline. Cleanup is attempted for admitted sessions
even when audio validation fails. Protocol errors, missing/invalid audio,
shared response IDs and cleanup failures make the report fail and the CLI
exit nonzero. Cancellation closes transports and propagates to the caller.
`--cleanup-timeout-s` (default 5 s) bounds session-close send and acknowledgement
together, independently of `--timeout-s`. The subsequent WebSocket close
handshake has the same separate bound; an unresponsive close acknowledgement
cannot hold cancellation for the full inference timeout.
`load-result.json` retains per-session send and audio-packet timelines using
the client's monotonic clock. It does not export handshake credentials or
raw audio payloads. Metric definitions are explicit:
Both modes validate each nonempty audio packet's strict base64 encoding, PCM16
sample alignment, 24 kHz sample rate and response identity. Valid aggregate
audio cannot hide a malformed packet. In load mode, rejected packets make the session fail
and appear in `invalid_audio_packets` with their event index, receive time and
fixed diagnostic code; they do not contribute samples or timing intervals.
Empty audio deltas remain legal and contribute no samples. This check does
not change the public client's decoding. The lifecycle scenario and dispatch
remain the same, but malformed packets that aggregate checks once tolerated
now fail that mode as well.
The probe accepts both the current OpenAI Realtime audio event name
`response.output_audio.delta` and the legacy `response.audio.delta`. Validation
uses the wire payload directly rather than relying on a particular client
library's alias table, so a server-side event-name migration cannot silently
turn real audio into a zero-output measurement.
| Field | Meaning |
| --- | --- |
| `client_audio_packet_interval_ms` | Consecutive nonempty packet arrivals: median, linear p99, max, and zero-based index of the first maximal interval. Interval index 0 ends at packet 1. One packet can contain multiple codec frames; this is not server tick latency. |
| `client_first_audio_after_stream_start_ms` | First audio receipt minus first input send start; admission is excluded. |
| `client_stream_rtf` | First input send start to last audio receipt, divided by received audio duration. This includes real-time input pacing and is not an inference-only RTF. |
| `client_send_lateness_ms` | Send start minus its original shared schedule, clipped at zero. |
| `frame_deficit` | Integer input-frame minus output-frame count from the shared session validation; missing when valid whole-frame accounting is unavailable. |
| `output_frames`, `voiced_frames`, `silent_frames`, `frame_coverage_ratio`, `audio_rms`, `audio_chunks` | Session-validation statistics retained on success and on acceptance failure once computed. |
| `acceptance_check` | Failed local check, with numeric diagnostics in `error`; other exceptions retain phase/type only, without remote handshake or metadata text. |
| `audio_underrun_s`, `audio_underrun_event_count`, `audio_continuity_ok` | Shared `compute_continuity_stats` on valid PCM16 packet arrivals, with its default 100 ms underrun threshold; absent audio gives missing values rather than a continuity pass. |
No output gives missing latency/RTF values, not zero. By default RTF is
diagnostic; `--max-client-rtf` adds an explicit ceiling. The existing frame
coverage, audible-frame and scheduler-data-plane checks still apply. A longer
`--drain-s` can demonstrate eventual completion under backlog, but it must not
be reported as real-time capacity when the fixed-window run failed or client
RTF exceeded the stated target. Likewise, the first live session can pay
one-time model/codec/JIT costs; either use the server's configured duplex
warmup path or label that run as cold-start evidence before comparing steady
capacity. Passing client-only metrics does not certify model quality or a
real-time capacity target on another GPU. Begin with N=1 and N=2; higher N is a
capacity experiment, not a promised property of an A100 or of this patch.
Continuity is a client-arrival diagnostic, not proof of speaker playback and
not an additional pass/fail gate. Packet-interval and send-lateness summaries
retain their documented linear p99; the older client interval helper reports
rounded nearest-rank p95 instead and is not substituted for that schema.
Deterministic driver and localhost WebSocket checks need no model or GPU:
```bash
python -m pytest tests/e2e/online_serving/test_personaplex_load_driver.py \
-m 'core_model and cpu' --run-level core_model -q
```
These checks test the driver against controlled protocol responses. Run the
load-driver command against real weights separately before reporting model-serving
performance. Server tick timestamps and generic TTFT/TPOT aggregation remain
the responsibility of the existing duplex metrics paths, not this driver.
+427 -1056
View File
File diff suppressed because it is too large. Load diff
+2
View File
@@ -26,6 +26,7 @@ implementation contract; it is not, by itself, a general support claim.
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
- [Runner-to-model Prefill/Decode Phase Contract](feature/preprocess_phase_contract.md)
- [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
- [Model-local KV Caches](feature/model_local_kv_caches.md)
- [Realtime AR-Diffusion Sessions](feature/realtime_ar_diffusion.md)
### Communication
@@ -35,6 +36,7 @@ implementation contract; it is not, by itself, a general support claim.
- [Mooncake Store Connector](feature/omni_connectors/mooncake_store_connector.md)
- [Mooncake Transfer Engine Connector](feature/omni_connectors/mooncake_transfer_engine_connector.md)
- [Mori Transfer Engine Connector](feature/omni_connectors/mori_transfer_engine_connector.md)
- [NIXL Connector](feature/omni_connectors/nixl_connector.md)
- [Shared Memory Connector](feature/omni_connectors/shared_memory_connector.md)
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
+31 -14
View File
@@ -60,7 +60,7 @@ Upstream per-engine metrics retain the `vllm:` prefix but are now registered by
There are five independent paths for metric collection.
**Path 1: Pipeline-level metrics (`vllm_omni:*`)**
#### Path 1: Pipeline-level metrics (`vllm_omni:*`)
`OmniPrometheusMetrics` registers the Gauge / Counter / Histogram collectors at import time. It is instantiated once per entrypoint, labeled with the model name. The entrypoint calls its methods as requests progress:
@@ -69,7 +69,7 @@ There are five independent paths for metric collection.
- `request_failed()` — recorded by the cleanup path when a request exits without natural completion. Internally maps to `finished_reason="abort"` on the existing completion Counter.
- `inc_requests_failed(reason)` — increments the dedicated failure-attribution Counter once per request. Reasons are normalized to the bounded set `client_abort`, `client_disconnect`, `stage_error`, and `unknown`.
**Path 2: Image and diffusion metrics**
#### Path 2: Image and diffusion metrics
Image and diffusion metrics use two cooperating collectors. `OmniPrometheusMetrics` owns pipeline/stage workload families, while `OmniModalityMetrics` owns per-replica diffusion breakdowns. Finished stage messages are consumed once in `OmniBase._process_single_result`, which prevents replayed terminal messages from incrementing Counters or Histograms twice.
@@ -81,15 +81,15 @@ Image and diffusion metrics use two cooperating collectors. `OmniPrometheusMetri
Zero and missing values have different meanings. Valid zero-duration queue observations are recorded, while unavailable optional measurements are omitted instead of being exported as synthetic zeroes. `peak_memory_mb`, `diffusion_forward_s`, `diffusion_kv_load_s`, `vae_decode_s`, and `kv_wait_s` therefore appear only when their data source is active.
**Path 3: Audio modality metrics (`vllm_omni:audio_*`)**
#### Path 3: Audio modality metrics (`vllm_omni:audio_*`)
`OmniModalityMetrics` registers seven audio families with `{model_name, stage, replica}` (plus an extra `threshold_ms` / `reason` label on the two extra-cardinality Counters). Three observation sites:
- `observe_modality_at_finalize(...)` — called from `omni_base._process_single_result` inside the existing `e2e_done` finalize guard. For `output_type == "audio"` it emits `audio_frames_total`, `audio_duration_s`, `audio_rtf` (or `audio_skipped_requests_total{reason="no_audio_data"}` when no audio was produced). Sample rate is resolved from `engine_outputs.multimodal_output` via `definitions.resolve_audio_sample_rate(...)` (fallback chain mirrors `serving_chat.py`'s audio response path).
- `observe_audio_first_packet(...)` — called from the OpenAI SSE audio branch in `serving_chat.py` on the first audio packet for a request. The once-per-request guard is held by `ClientRequestState.first_audio_ts`. The `request_arrival_ts` anchor is stored in `ClientRequestState` by `async_omni.generate()`, computed at request entry.
- `observe_audio_streaming_finalize(...)` — called from `serving_chat.py` after the streaming chunk loop exhausts. It runs the per-chunk player simulation from `vllm_omni/benchmarks/audio_continuity.py` to compute the worst-case underrun and emits `audio_underrun_s` plus (when the request stayed below the threshold) `audio_continuity_ok_total{threshold_ms}`. Per-chunk PCM byte counts and arrival timestamps are recorded by the same audio branch that updates `first_audio_ts`.
- `observe_audio_first_packet(...)` — called from the OpenAI SSE audio branch in `serving_chat.py` and the streaming TTS output paths in `serving_speech.py` on the first non-empty audio payload for a request. Streaming Speech observes the first PCM payload rather than a header-only WAV chunk. HTTP carries the middleware request timestamp through response generation; WebSocket uses the start of each sentence synthesis request. Non-streaming Speech does not emit TTFP and uses `e2e_request_latency_s` for overall latency instead.
- `observe_audio_streaming_finalize(...)` — called from `serving_chat.py` and `serving_speech.py` after a streaming chunk loop exhausts normally. It runs the per-chunk player simulation from `vllm_omni/benchmarks/audio_continuity.py` to compute the worst-case underrun and emits `audio_underrun_s` plus (when the request stayed below the threshold) `audio_continuity_ok_total{threshold_ms}`. Per-chunk PCM byte counts and arrival timestamps are recorded by the same output paths that update TTFP. Speech records raw PCM payloads before raw/SSE/WebSocket encoding, excludes WAV headers and empty chunks, and does not emit continuity samples for failed, cancelled, or non-streaming requests.
**Path 4: Cross-stage transfer metrics (`vllm_omni:transfer_*`)**
#### Path 4: Cross-stage transfer metrics (`vllm_omni:transfer_*`)
`OmniTransferMetrics` registers four Histogram families with `{model_name, from_stage, from_replica, to_stage, to_replica}` labels. Each observation corresponds to one physical transfer hop (one chunk between adjacent stages), not the per-request accumulated total — so the histograms track per-transfer distribution.
@@ -100,7 +100,7 @@ The hook lives in `OrchestratorAggregator.record_transfer_tx` and `record_transf
Defensive fail-safe: if `transfer_emitter` or `replica_resolver` is missing, or the resolver returns `None` for either side, the emit is skipped silently (the underlying `TransferEdgeStats` accumulation is unaffected).
**Path 5: Per-engine metrics (`vllm:*`, stage/replica wrap)**
#### Path 5: Per-engine metrics (`vllm:*`, stage/replica wrap)
The Orchestrator instantiates `OmniPrometheusStatLogger` (a thin subclass of upstream `vllm.v1.metrics.loggers.PrometheusStatLogger`) and feeds it scheduler stats and iteration stats after processing each batch of engine outputs. This populates the standard ~37 vLLM metric families (TTFT, ITL, TPOT, KV cache usage, etc.) using the same upstream code path — but with the `engine` label reshaped into `stage` + `replica` so multi-replica deployments produce distinct series per replica. See the next section for the wrap mechanics.
@@ -174,7 +174,7 @@ second.
### Pipeline (4)
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| -------- | ------ | -------- | ------------- |
| `vllm_omni:num_requests_running` | Gauge | `model_name` | Requests currently executing across all stages |
| `vllm_omni:num_requests_waiting` | Gauge | `model_name` | Requests queued but not yet scheduled |
| `vllm_omni:requests_success_total` | Counter | `model_name`, `finished_reason` | Total requests by completion reason ({stop, length, abort, ...}); aborts cover client-disconnect / cancellation paths in addition to upstream `FinishReason.ABORT` |
@@ -183,7 +183,7 @@ second.
### Image and diffusion service-level metrics
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| -------- | ------ | -------- | ------------- |
| `vllm_omni:stage_gen_time_s` | Histogram | `model_name`, `stage`, `stage_type` | Stage submit to finished output; includes in-stage queueing |
| `vllm_omni:request_queue_wait_s` | Histogram | `model_name` | Orchestration-layer queue wait; a present zero is recorded |
| `vllm_omni:stage_waiting_requests` | Gauge | `model_name`, `stage` | Sum of the latest waiting snapshots across live replicas in the stage |
@@ -200,7 +200,7 @@ second.
Labels: `{model_name, stage, replica}`.
| Metric | Type | Description |
|--------|------|-------------|
| -------- | ------ | ------------- |
| `vllm_omni:diffusion_exec_s` | Histogram | Core diffusion-step execution time per request |
| `vllm_omni:diffusion_exec_per_step_s` | Histogram | Core diffusion execution divided by inference-step count |
| `vllm_omni:diffusion_preprocess_s` | Histogram | Diffusion input preprocessing time |
@@ -218,7 +218,7 @@ The optional breakdown and memory families are sparse by design: when an engine
Labels: `{model_name, stage, replica}` plus the listed extra label.
| Metric | Type | Extra label | Description |
|--------|------|-------------|-------------|
| -------- | ------ | ------------- | ------------- |
| `vllm_omni:audio_ttfp_s` | Histogram | — | Time from request arrival to first audio packet/frame |
| `vllm_omni:audio_duration_s` | Histogram | — | Audio content duration (`audio_frames / sample_rate`) |
| `vllm_omni:audio_rtf` | Histogram | — | Real-time factor `stage_gen_time_s / audio_duration_s` (SLO `< 1`); uses `RTF_BUCKETS` |
@@ -227,12 +227,29 @@ Labels: `{model_name, stage, replica}` plus the listed extra label.
| `vllm_omni:audio_continuity_ok_total` | Counter | `threshold_ms` | Incremented when the request's worst underrun stayed below `threshold_ms` |
| `vllm_omni:audio_skipped_requests_total` | Counter | `reason` | Silent-loss counter — code2wav rejected malformed codec input and returned `200 OK` with empty audio |
### Speech streaming (2)
Labels: `{model_name}` plus the listed extra label.
| Metric | Type | Extra label | Description |
| -------- | ------ | ------------- | ------------- |
| `vllm_omni:speech_stream_aborted_total` | Counter | `reason` | Interrupted audio generators, including before first PCM; reasons: `cancelled`, `closed`, `engine_dead`, `error` |
| `vllm_omni:speech_stream_completed_total` | Counter | — | Normally completed audio generators; does not confirm client receipt |
Scope: started Speech audio generators (raw/SSE/WebSocket), counted once per
generation; WebSocket counts each sentence. Chat, non-streaming Speech, and
failures before generator execution are excluded. Interrupted streams do not
emit continuity or underrun samples.
Interruption ratio: `aborted / (aborted + completed)`, using increments over the
same time window and summing all reasons per model.
### Cross-stage transfer (4)
Labels: `{model_name, from_stage, from_replica, to_stage, to_replica}`.
| Metric | Type | Description |
|--------|------|-------------|
| -------- | ------ | ------------- |
| `vllm_omni:transfer_size_bytes` | Histogram | Per-transfer payload size in bytes |
| `vllm_omni:transfer_tx_s` | Histogram | Sender-side time (serialize + submit to connector) |
| `vllm_omni:transfer_rx_s` | Histogram | Receiver-side time (recv + deserialize) |
@@ -245,8 +262,8 @@ After the wrap, every upstream `vllm:*` family — TTFT, ITL, TPOT, e2e latency,
## Naming Convention
- All time-bearing metrics use the `_s` suffix (values in seconds). Two bucket families are used:
- `SECONDS_BUCKETS` (0.05 s – 300 s) for e2e / generation / TTFP style values.
- `SECONDS_FAST_BUCKETS` (0.001 s – 60 s) for fine-grained cross-stage transfer and audio-underrun values that need millisecond-level resolution.
- `SECONDS_BUCKETS` (0.05 s – 300 s) for e2e / generation / TTFP style values.
- `SECONDS_FAST_BUCKETS` (0.001 s – 60 s) for fine-grained cross-stage transfer and audio-underrun values that need millisecond-level resolution.
- Counters use the `_total` suffix (auto-appended by `prometheus_client`).
- Sizes use the `_bytes` suffix.
- All omni-specific families are prefixed `vllm_omni:`. The upstream `unregister_vllm_metrics()` function is monkey-patched to a scoped version that still strips upstream `vllm:*` collectors (so multi-engine init within one process does not crash on duplicate registration) but preserves anything prefixed `vllm_omni:` / `vllm_omni`.
+9 -2
View File
@@ -16,6 +16,7 @@ required_reviewers:
- "@yenuo26"
- "@NickCao"
primary_code_paths:
- vllm_omni/engine/omni_engine_base.py
- vllm_omni/engine/async_omni_engine.py
- vllm_omni/engine/async_engine_utils.py
- vllm_omni/engine/orchestrator.py
@@ -28,8 +29,12 @@ related_code_paths:
- vllm_omni/engine/stage_pool.py
- vllm_omni/outputs/output_processor.py
- vllm_omni/distributed/omni_coordinator/**
# Duplex kernel design is owned by ../fullduplex.md; listed here because the
# orchestrator hosts the duplex control plane.
# The duplex siblings (DuplexOmniEngine / DuplexOrchestrator) and the
# engine-resident session framework are owned by ../fullduplex.md; listed
# here because they fill the template seams of OmniEngineBase and
# OrchestratorBase.
- vllm_omni/engine/duplex_omni_engine.py
- vllm_omni/engine/duplex_orchestrator.py
- vllm_omni/engine/duplex/**
depends_on:
- input_output_modality_contracts.md
@@ -48,6 +53,8 @@ validation_paths:
- tests/engine/test_cfg_companion_lifecycle.py
- tests/engine/test_rpc_result_router.py
- tests/engine/duplex/**
- tests/engine/test_duplex_orchestrator.py
- tests/engine/test_duplex_omni_engine.py
- tests/engine/test_duplex_import_boundary.py
upstream_refs:
- vllm.v1.engine.EngineCoreRequest
+61 -3
View File
@@ -19,7 +19,9 @@ required_reviewers:
primary_code_paths:
- vllm_omni/entrypoints/omni.py
- vllm_omni/entrypoints/async_omni.py
- vllm_omni/entrypoints/async_omni_base.py
- vllm_omni/entrypoints/omni_base.py
- vllm_omni/entrypoints/duplex_omni.py
- vllm_omni/entrypoints/cli/**
- vllm_omni/entrypoints/openai/**
- vllm_omni/entrypoints/openpi/**
@@ -27,14 +29,14 @@ primary_code_paths:
- vllm_omni/entrypoints/stage_utils.py
- vllm_omni/entrypoints/utils.py
- vllm_omni/entrypoints/duplex/**
- vllm_omni/entrypoints/duplex_request_client.py
- vllm_omni/clients/**
- vllm_omni/protocol/**
primary_path_exceptions:
- path: vllm_omni/entrypoints/openai/errors.py
owner: error_contracts.md
- path: vllm_omni/entrypoints/duplex/**
owner: ../fullduplex.md
- path: vllm_omni/entrypoints/duplex_request_client.py
- path: vllm_omni/entrypoints/duplex_omni.py
owner: ../fullduplex.md
- path: vllm_omni/clients/**
owner: ../fullduplex.md
@@ -42,7 +44,9 @@ related_code_paths:
- vllm_omni/errors.py
- vllm_omni/inputs/**
- vllm_omni/outputs/**
- vllm_omni/engine/omni_engine_base.py
- vllm_omni/engine/async_omni_engine.py
- vllm_omni/engine/duplex_omni_engine.py
- vllm_omni/config/**
- vllm_omni/deploy/**
depends_on:
@@ -50,10 +54,13 @@ depends_on:
- error_contracts.md
- engine_orchestration.md
validation_paths:
- tests/protocol/**
- tests/entrypoints/test_omni_entrypoints.py
- tests/entrypoints/test_async_omni.py
- tests/entrypoints/test_async_omni_pause_sleep_routing.py
- tests/entrypoints/test_async_omni_duplex.py
- tests/entrypoints/test_duplex_omni.py
- tests/entrypoints/duplex/**
- tests/entrypoints/openai_api/test_duplex_api_server.py
- tests/entrypoints/test_serve.py
- tests/entrypoints/test_stream_finish_reason.py
- tests/entrypoints/openai/**
@@ -89,11 +96,43 @@ This document owns offline API semantics, CLI and serve composition,
supported OpenAI-compatible routes, request validation and normalization,
response conversion, streaming/session behavior, and engine handoff.
When `--api-server-count` is greater than one, the serve composition root
starts the frontend process manager while a parent-owned `StageRuntime`
launches the shared local stage engines. Each frontend receives only its own
stage-client channel configuration and does not launch or retire backend
processes. CLI validation rejects unsupported distributed, diffusion,
fault-tolerant, elastic-EP, Ray, and runtime-LoRA combinations before startup.
It does not own configuration precedence, cross-stage routing, stage
placement, payload implementation, or semantic error classification.
`entrypoints/openai/errors.py` is an explicit primary-path exception owned by
`error_contracts.md`.
It also owns `vllm_omni/protocol/**`, the shared wire codec promoted by RFC
[#6592](https://github.com/vllm-project/vllm-omni/issues/6592) P0a. It follows the three tiers
`docs/serving/realtime_duplex_api.md` already defines. `protocol/realtime/**`
is Tier 1, identical to OpenAI: 30 server events, 10 client commands, their
typed fields and pure wire rendering, audio-format negotiation,
conversation-item rules and the error-envelope shape. `protocol/duplex/**` is
Tier 2 (OpenAI names carrying vLLM-Omni extensions, each a subclass of its
Tier 1 twin declaring only what it adds) and Tier 3 (ours alone, including our
error-code vocabulary), kept separate so a GA-only consumer is not handed a
vocabulary its clients never send. That package sits
outside `entrypoints/` on purpose --- the engine depends on it, so it belongs
to neither layer --- but deciding what a Realtime client may put on the wire is
this document's subject, and the same reasoning already places
`vllm_omni/errors.py`, `inputs/` and `outputs/` under contract documents rather
than under a consumer.
It is deliberately **not** owned by `../fullduplex.md`, even though duplex is
the codec's only consumer today. RFC #6592 asks for a named owner precisely so
a second Realtime consumer does not need duplex review to change the codec
(#6592 open question 8). Ownership here is review scope, not a dependency: the
codec may not import the engine, the entrypoints or the model code, which
`tests/protocol/realtime/test_protocol_import_boundary.py` enforces. When a
second consumer lands (#6592 P0b) the package is expected to graduate to its
own module document with owners drawn from both consumers.
## Candidate invariants
These identifiers are proposals while the document is `draft`.
@@ -103,6 +142,22 @@ These identifiers are proposals while the document is `draft`.
**Rule:** Entrypoints MUST NOT implement cross-stage routing or stage lifecycle
policy.
### ENTRY-INV-002: The shared Realtime codec does not depend on its consumers
**Rule:** `vllm_omni/protocol/**` MUST NOT import `vllm_omni.engine`,
`vllm_omni.entrypoints`, `vllm_omni.model_executor`, `vllm_omni.worker` or
`vllm_omni.clients`; `protocol/duplex/**` MAY depend on `protocol/realtime/**`
but never the reverse, and a Tier 1 class MUST NOT carry a vLLM-Omni extension
field; and each protocol/codec behavior MUST have exactly one
implementation that every consumer uses. A wire type MUST NOT carry a
consumer's internal representation: the duplex mailbox rendering
(`DuplexCommand.payload()`, whose channel differs from the client event for
`session.update` and the `conversation.item.*` commands) stays engine-side.
A duplex consumer (`engine/duplex/**`, `entrypoints/duplex/**`, the duplex
clients) MUST import `vllm_omni.protocol.duplex` and MUST NOT import
`vllm_omni.protocol.realtime` directly, so the tier boundary has exactly one
extension point.
### ENTRY-INV-100: Public requests are normalized once
**Rule:** Public protocol values MUST be validated and converted to internal
@@ -136,6 +191,9 @@ stage so `wake_up(stage_ids=[0])` does not skip a later `wake_up(stage_ids=[1])`
Streaming input pumps take an admission slot immediately before each EngineCore
ADD or update, not while waiting for the next client chunk. Frontend abort
keeps `request_states` until `generate()` consumes the terminal output.
Multi-API serving must preserve one client rank and one channel set per
frontend, include every frontend in readiness/failure observation, and keep
stage lifecycle ownership in the parent composition root.
## Promotion gate
+10 -2
View File
@@ -136,8 +136,16 @@ so only when it is marked retryable; unsupported capabilities, identity
collisions, producer failures, and publication failures remain visible as
startup failures.
`coordination_timeout_seconds` bounds acquisition of lookup and build locks; it
is not a hard wall-clock deadline for in-process validation, production, or
`coordination_timeout_seconds` bounds acquisition of the domain-initialization
lock and lookup/build locks. Store construction and each later resolution or
publication operation receive separate coordination budgets from the same
`WaitPolicy`; they do not share an end-to-end startup deadline. A domain-init
lock timeout is a retryable domain failure: preferred mode may fall back and
required mode fails. Retrying initialization requires constructing a fresh
runtime after contention ends; a waiter never cancels or unlocks the owner.
The coordination budget is not a hard wall-clock deadline for filesystem I/O,
in-process validation, production, or
atomic publication. Once this V1 implementation becomes the producer, the
synchronous producer and publication run to completion. A hung producer blocks
its owning process and requires external process supervision; enforceable
+20
View File
@@ -77,6 +77,22 @@ readiness, one pool per logical stage, replica identity and membership,
selection and affinity, stage-client/process lifecycle, liveness, draining,
and shutdown.
Local EngineCore initialization and multi-API serving use
`StageRuntime.launch_stage_engines(num_api_servers=1)` as their common launch
entry. It resolves plans when not supplied by the runtime's device-group
scheduler, coordinates device locks and environment overlays, starts engines,
and rolls back resources on startup or attachment failure. Single-API clients
attach locally after readiness and take ownership of the engine resources;
there is no additional API subprocess. Diffusion and remote attachment retain
their backend-specific initialization.
For local multi-API serving, the parent runtime owns and launches each stage
engine once. It supplies a distinct input/output channel for every API
frontend and stage replica; frontend workers attach clients to those engines
without acquiring stage-process ownership. Readiness observation and shutdown
remain parent-owned. This topology currently excludes diffusion and
remote/headless stages, intra-stage data parallelism, and Ray backends.
It does not own cross-stage request policy, model scheduler policy, connector
transfer semantics, configuration precedence, or public error rendering.
@@ -114,6 +130,10 @@ affinity, membership changes, abort routing, process failure, draining, and
repeated shutdown. Cross-stage policy changes belong in
`engine_orchestration.md`.
For multi-API changes, additionally exercise per-frontend channel assignment,
shared-engine startup handshakes, frontend failure observation, and parent
cleanup without duplicate stage shutdown.
## Promotion gate
- Reconcile terminology and ownership after #5441 reaches a final state.
+30 -2
View File
@@ -13,7 +13,6 @@ It can send model inference requests to either a locally running vLLM-Omni servi
!!! tip
If you run both ComfyUI and vLLM-Omni on the same device, you can create separate virtual environments and use different Python versions for them.
## Installation
Copy the `apps/ComfyUI-vLLM-Omni` folder to the `custom_nodes` subfolder of your ComfyUI installation. Your directory should look like `ComfyUI/custom_nodes/ComfyUI-vLLM-Omni`.
@@ -21,9 +20,10 @@ Copy the `apps/ComfyUI-vLLM-Omni` folder to the `custom_nodes` subfolder of your
If you are running ComfyUI during copying, you should restart ComfyUI to load this extension.
!!! tip
You can use utility websites such as https://download-directory.github.io/ to download a subdirectory of a repo. Also checkout community discussions (e.g., https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repository) for more info.
You can use utility websites such as <https://download-directory.github.io/> to download a subdirectory of a repo. Also checkout community discussions (e.g., <https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repository>) for more info.
On the device and virtual environment you run ComfyUI, launch ComfyUI with
```bash
cd ComfyUI
@@ -35,6 +35,7 @@ python main.py --cpu
```
On the device and virtual environment you run vLLM-Omni, start a model service with
```bash
vllm serve The_Model_ID_to_Serve --omni --port 8000
```
@@ -50,12 +51,27 @@ This extension offers the following nodes based on the output modalities:
- **Generate Video** for text-to-video and image-to-video tasks
- **Multimodality Understanding** for multimodality-to-text and multimodality-to-audio tasks
- **TTS** and **TTS Voice Clone** for TTS tasks
- **Generate Music** for text-to-music tasks
This extension also offers example workflows (at **ComfyUI sidebar -> Templates -> vLLM-Omni**)
!!! info
The node UI and feature designs are intended to match vLLM-Omni online serving interfaces. It cannot offer more than what the interfaces support.
Every node carries the vLLM-Omni mark in its title bar and is tinted by what it outputs, so a graph is readable at a glance:
| Colour | Nodes | What they produce |
| --- | --- | --- |
| Blue | Generate Image, Generate Video, Multimodality Understanding, TTS, TTS Voice Clone | A generated image, video, audio, or text. These are the only nodes that reach a server. |
| Amber | AR / Diffusion / Multi-Stage Sampling Params | Sampling parameters that apply to any model |
| Purple | Qwen TTS Params, Wan Video Params, MiniMax-H3 Video Params | Parameters that only one model family accepts |
| Red | LoRA, FastH3 Deployment | Which weights the server is expected to have loaded |
| Teal | Video References | Reference media |
Recolouring a node by hand (right click -> Colors) overrides its tint, and the choice is kept.
**Generate Video** takes a clip length in seconds (`duration`), not a frame count. Frames stay the wire unit and are derived with the node's `fps`, so the length is always measured against the rate that is actually served; models that accept only certain frame counts still round to their own lattice server-side. Graphs saved before this widget existed stored `num_frames` in its place and are converted on load, using the fps recorded alongside it -- the browser console names every node it rewrites.
To build a simple workflow yourself,
- Drag a generation node onto the canvas.
@@ -66,6 +82,18 @@ To build a simple workflow yourself,
- For some multi-stage models like BAGEL, [only one stage's sampling parameters are exposed and tunable via vLLM-Omni's online serving API](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/examples/online_serving/bagel/). Thus, these models are treated as single-stage ones. Please check the vLLM-Omni documentation on how to correctly set each model's sampling parameters.
- For multi-stage models where all stages are either autoregression or diffusion, you can also connect only a single Sampling Params node, indicating that this set of sampling parameters will be used for all stages.
## MiniMax H3 text-to-video workflow
The **MiniMax H3 Text to Video** template uses the existing remote Generate Video,
H3 Params, sampling, and Remote LoRA nodes. It includes Base settings and an
optional Turbo preset, with native resolution, 24 FPS, and `17k+5` frame counts.
ComfyUI receives the generated video and audio from the server; it does not load
H3 weights locally. Configure the server-side LoRA path before enabling Turbo.
See the [H3 workflow guide](https://github.com/vllm-project/vllm-omni/blob/main/apps/ComfyUI-vLLM-Omni/docs/minimax-h3-t2v.md)
for server setup, template import, Turbo configuration, recorded validation,
and saved-video/audio checks.
## Examples & Screenshots
Please read the [ComfyUI integration's README](https://github.com/vllm-project/vllm-omni/tree/main/apps/ComfyUI-vLLM-Omni) for more info.
+6 -3
View File
@@ -1,7 +1,7 @@
# Custom Pipeline Extension Guide
Transformer already support Custom Pipeline via
https://github.com/huggingface/diffusers/blob/main/docs/source/en/using-diffusers/custom_pipeline_overview.md
<https://github.com/huggingface/diffusers/blob/main/docs/source/en/using-diffusers/custom_pipeline_overview.md>
This guide demonstrates how to use the newly added features for extending vLLM-Omni's diffusion pipeline with custom functionality.
@@ -20,8 +20,9 @@ Three main features enable custom pipeline extension:
`WorkerWrapperBase` is a wrapper class that creates `DiffusionWorker` instances with optional extension support. It enables dynamic inheritance, allowing you to add custom methods and functionality to workers without modifying the base worker class.
**Key capabilities:**
- Dynamic worker class extension via `worker_extension_cls`
- Support for custom pipeline initialization via `custom_pipeline_args`
- Support for custom pipeline initialization via `custom_pipeline_args["pipeline_class"]`
- Method delegation to underlying worker
- Attribute access forwarding
@@ -42,6 +43,7 @@ The `diffusion_load_format` parameter controls the initial diffusion model load.
`CustomPipelineWorkerExtension` is a mixin class that extends `DiffusionWorker` with the ability to re-initialize the pipeline with a custom implementation.
**Key method:**
- `re_init_pipeline(custom_pipeline_args)`: Re-initializes the pipeline with custom arguments, properly cleaning up the old pipeline first
**Location:** `vllm_omni/diffusion/worker/diffusion_worker.py`
@@ -150,6 +152,7 @@ omni = Omni(
},
worker_extension_cls=MyCustomExtension,
# Note: worker_extension_cls is an internal parameter
# CustomPipelineWorkerExtension will automatically init pipeline when custom_pipeline_args is provided
# CustomPipelineWorkerExtension will automatically init pipeline when
# custom_pipeline_args includes "pipeline_class".
)
```
@@ -1,11 +1,23 @@
<!-- markdownlint-disable MD001 MD025 -->
# --8<-- [start:requirements]
- GPU: Validated on gfx942 (It should be supported on the AMD GPUs that are supported by vLLM.)
# --8<-- [end:requirements]
# --8<-- [start:set-up-using-python]
vLLM-Omni current recommends the steps in under setup through Docker Images.
For ROCm, vLLM-Omni currently recommends the setup steps through Docker Images.
vLLM-Omni depends on the matching major/minor release of vLLM. The 0.29
development line uses vLLM 0.29.x. Published 0.28.0 wheels and images use vLLM
0.28.x.
The Dockerfile's `BASE_IMAGE` pin applies only to Docker builds. The
`vllm-omni` package does not install vLLM as a dependency, so non-Docker source
installs must install the matching ROCm vLLM release explicitly before
installing vLLM-Omni, as shown below.
# --8<-- [start:pre-built-wheels]
@@ -14,6 +26,7 @@ vLLM-Omni current recommends the steps in under setup through Docker Images.
These pre-built wheel instructions install the published vLLM-Omni 0.28.0 release. For the 0.29 development line, use the source-install instructions below.
vLLM-Omni is built based on vLLM. Please install it with command below.
```bash
uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
```
@@ -36,6 +49,7 @@ uv pip install onnxruntime-rocm
# --8<-- [start:build-wheel-from-source]
#### Installation of vLLM
If you do not need to modify source code of vLLM, you can directly install the stable 0.29.0 release version of the library
```bash
@@ -45,7 +59,9 @@ uv pip install vllm==0.29.0+rocm723 --extra-index-url https://wheels.vllm.ai/roc
The pre-built 0.29.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
#### Installation of vLLM-Omni
Since vllm-omni is rapidly evolving, it's recommended to install it from source
```bash
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
@@ -71,15 +87,31 @@ python3 setup.py develop
#### Build docker image
The source build defaults to the published upstream base image
`vllm/vllm-openai-rocm:v0.29.0`, aligned with the vLLM release used by CI.
Older or custom bases must provide the vLLM APIs checked by the Dockerfile's
image-build canary. This upstream base is distinct from the prebuilt
`vllm/vllm-omni-rocm` images discussed below; their published-tag availability
is tracked in [#7405](https://github.com/vllm-project/vllm-omni/issues/7405).
```bash
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm-omni-rocm .
```
To select the upstream ROCm base explicitly, pass `BASE_IMAGE`:
```bash
DOCKER_BUILDKIT=1 docker build \
-f docker/Dockerfile.rocm \
--build-arg BASE_IMAGE=vllm/vllm-openai-rocm:v0.29.0 \
-t vllm-omni-rocm .
```
#### Launch the docker image
##### Launch with OpenAI API Server
```
```bash
docker run --rm \
--group-add=video \
--ipc=host \
@@ -97,7 +129,7 @@ vllm-omni-rocm \
##### Launch with interactive session for development
```
```bash
docker run --rm -it \
--network=host \
--group-add=video \
@@ -119,7 +151,9 @@ vllm-omni-rocm
vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as [vllm/vllm-omni-rocm](https://hub.docker.com/r/vllm/vllm-omni-rocm/tags). The version of vLLM-Omni indicates which release of vLLM it is based on.
#### Launch vLLM-Omni Server
Here's an example deployment command that has been verified on 2 x MI300's:
```bash
docker run --rm \
--group-add=video \
@@ -136,8 +170,10 @@ docker run --rm \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
```
#### Launch an interactive terminal with prebuilt docker image.
#### Launch an interactive terminal with prebuilt docker image
If you want to run in dev environment you can launch the docker image as follows:
```bash
docker run --rm -it \
--network=host \
@@ -10,25 +10,25 @@ The recommended way to use vLLM-Omni on NPU is through the vLLM-Ascend pre-built
=== "A2"
```bash
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.29.0
```
=== "A3"
```bash
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0-a3
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.29.0-a3
```
=== "A5"
```bash
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0-a5
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.29.0-a5
```
=== "310P (Experimental)"
```bash
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0-310p
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.29.0-310p
```
```bash
@@ -53,7 +53,7 @@ docker run --rm \
# Inside the container, install vLLM-Omni from source
cd /vllm-workspace
git clone -b v0.28.0 https://github.com/vllm-project/vllm-omni.git
git clone -b v0.29.0rc1 https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
pip install -v -e . --no-build-isolation
# or VLLM_OMNI_TARGET_DEVICE=npu pip install -v -e .
@@ -71,10 +71,10 @@ We are keeping [issue #886](https://github.com/vllm-project/vllm-omni/issues/886
# --8<-- [start:installation-main]
You can also install vLLM-Omni from the latest main branch if you want to use the latest features or bug fixes. Use the aligned vLLM-Ascend v0.28.0 image described above, and check [issue #886](https://github.com/vllm-project/vllm-omni/issues/886) for the status of the latest vLLM-Omni commit on NPU.
You can also install vLLM-Omni from the latest main branch if you want to use the latest features or bug fixes. Use the aligned vLLM-Ascend v0.29.0 image described above, and check [issue #886](https://github.com/vllm-project/vllm-omni/issues/886) for the status of the latest vLLM-Omni commit on NPU.
```bash
# Inside the vLLM-Ascend v0.28.0 container, install vLLM-Omni from main
# Inside the vLLM-Ascend v0.29.0 container, install vLLM-Omni from main
cd /vllm-workspace
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
@@ -103,7 +103,7 @@ Here's an example deployment command that has been verified on 4 x NPUs:
=== "A2"
```bash
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0
docker run --rm \
--name vllm-omni-a2 \
--shm-size=64g \
@@ -127,7 +127,7 @@ Here's an example deployment command that has been verified on 4 x NPUs:
=== "A3"
```bash
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0-a3
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0-a3
docker run --rm \
--name vllm-omni-a3 \
--shm-size=64g \
@@ -151,7 +151,7 @@ Here's an example deployment command that has been verified on 4 x NPUs:
=== "A5"
```bash
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0-a5
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0-a5
docker run --rm \
--name vllm-omni-a5 \
--shm-size=64g \
@@ -175,7 +175,7 @@ Here's an example deployment command that has been verified on 4 x NPUs:
=== "310P (Experimental)"
```bash
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0-310p
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0-310p
docker run --rm \
--name vllm-omni-310p \
--shm-size=64g \
+2 -2
View File
@@ -25,8 +25,8 @@ CATEGORIES = {
"clients": {
"name": "Clients",
"description": (
"Client libraries that talk to a running vLLM-Omni server "
"(the full-duplex Realtime client and its per-model session presets)."
"Client libraries for full-duplex sessions: the Realtime WebSocket client, "
"the in-process client over `DuplexOmni`, and the per-model session presets."
),
},
"inputs": {
+2
View File
@@ -2,6 +2,7 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import ast
import json
import logging
import re
import sys
@@ -124,6 +125,7 @@ def extract_omni_serve_subparser_init():
"make_arg_parser": lambda parser: parser, # no-op for doc
"_ensure_vllm_platform": lambda: None, # no-op for doc
"_nonneg_finite_float": float,
"_json_object": json.loads,
"_parse_stage_overrides": __import__("json").loads,
"VLLM_SUBCMD_PARSER_EPILOG": "",
"logger": logger,
+1 -1
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Reduce browser search-index memory for generated API reference pages."""
from material.plugins.search.plugin import SearchIndex
+6 -4
View File
@@ -59,6 +59,7 @@ th {
| `Magi2Pipeline` | MAGI-2 native Preview-stage T2VA and I2VA (272p/540p) | `sand-ai/MAGI-2-preview` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SandAI/MAGI-2-preview-L20X.md) |
| `DreamZeroPipeline` | DreamZero-DROID | `GEAR-Dreams/DreamZero-DROID` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `Pi0Pipeline` | π0 (Pi-Zero) VLA | `lerobot/pi0_base` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/lerobot/Pi0.md) |
| `Pi05Pipeline` | π0.5 VLA | `lerobot/pi05_base` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/lerobot/Pi05.md) |
| `HeliosPipeline`, `HeliosPyramidPipeline` | Helios | `BestWishYsh/Helios-Base`, `BestWishYsh/Helios-Mid`, `BestWishYsh/Helios-Distilled` | ✅︎ | ✅︎ | ✅︎ | | — |
| `OvisImagePipeline` | Ovis-Image | `OvisAI/Ovis-Image` | ✅︎ | ✅︎ | | ✅︎ | — |
| `LongcatImagePipeline` | LongCat-Image | `meituan-longcat/LongCat-Image` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
@@ -69,9 +70,10 @@ th {
| `StableDiffusionXLPipeline` | Stable-Diffusion-XL | `stabilityai/stable-diffusion-xl-base-1.0` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ | — |
| `CosyVoice3Model` | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✅︎ | ✅︎ | | ✅︎ | — |
| `AuKForConditionalGeneration` | AuK / AuK-Flash (instruction-driven speech generation and editing) | `tencent/AuK`, `tencent/AuK-Flash` (with `Qwen/Qwen2.5-Omni-3B`; assemble with `tools/prepare_auk_checkpoint.py`) | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/AuK-H100.md) |
| `OmniVoiceModel` | OmniVoice | `k2-fsa/OmniVoice` | ✅︎ | | | ✅︎ | — |
| `VoxCPM2TalkerForConditionalGeneration` | VoxCPM2 | `openbmb/VoxCPM2` | ✅︎ | | | ✅︎ | — |
| `DotsTTSForConditionalGeneration` | dots.tts | `dots-studio/dots.tts-soar` | ✅︎ | | | | — |
| `BreezeForConditionalGeneration` | Breeze-TTS-2 (TTS, voice clone, voice direction) | `BreezeBlue/Breeze-TTS-2` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md) |
| `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | | — |
| `MammothModa2ForConditionalGeneration` | MammothModa2-Dev (AR-only image understanding) | `bytedance-research/MammothModa2-Dev` | ✅︎ | | | | — |
| `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
@@ -86,7 +88,7 @@ th {
| `Qwen3TTSForConditionalGeneration` | Qwen3-TTS-12Hz-1.7B-Base | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `MingTTSForConditionalGeneration` | Ming-omni-tts dense 0.5B | `inclusionAI/Ming-omni-tts-0.5B` | ✅︎ | | | ✅︎ | — |
| `GLMTTSForConditionalGeneration` | GLM-TTS | `zai-org/GLM-TTS` | ✅︎ | | | | — |
| `MossTTSNanoForCausalLM` | MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | ✅︎ | | | | — |
| `MossTTSNanoForCausalLM` | MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenMOSS/MOSS-TTS-Nano-NPU.md) |
| `MossTTSDelayModel` | MOSS-TTS, MOSS-TTSD, MOSS-SoundEffect, MOSS-VoiceGenerator | `OpenMOSS-Team/MOSS-VoiceGenerator` | ✅︎ | | | | — |
| `MossTTSRealtime` | MOSS-TTS-Realtime | `OpenMOSS-Team/MOSS-TTS-Realtime` | ✅︎ | | | | — |
| `HiggsAudioV2ForConditionalGeneration` | Higgs-Audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✅︎ | | | | — |
@@ -105,12 +107,12 @@ th {
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | | — |
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
| `DyninOmniForConditionalGeneration` | Dynin-Omni | `snu-aidas/Dynin-Omni` | ✅︎ | | | ✅︎ | — |
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | — |
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) |
| `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — |
| `HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
| `HiDreamO1ImagePipeline` | HiDream-O1-Image | `HiDream-ai/HiDream-O1-Image` | ✅︎ | | | | — |
| `AnimaPipeline` | Anima | `circlestone-labs/Anima` | ✅︎ | ✅︎ | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/circlestone-labs/Anima.md) |
The `Recipe` column links the corresponding deployment recipe. `Published` is
preferred when a corresponding `recipes.vllm.ai` page exists; otherwise the
+12 -5
View File
@@ -145,15 +145,22 @@ interchangeably.
| Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) |
| Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) |
| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) |
| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` or `WS /v1/duplex` | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) |
| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`) | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) |
| Generated video chunks | `WS /v1/realtime/video` | Start a diffusion request and receive fragmented MP4 | [Streaming Video Output](streaming_video_output_api.md) |
| Robot policy inference | `WS /v1/realtime/robot/openpi` | Send MessagePack observations and receive action arrays | [OpenPI Robot Policy](openpi_api.md) |
All six routes are model- or configuration-dependent. In particular,
`/v1/realtime` is not full duplex unless the client sets `duplex=1` and the
deployment explicitly enables duplex sessions. Clients should also verify the
duplex capability payload because the query-parameter form falls back to the
ordinary realtime handler when duplex is unavailable.
`/v1/realtime` is full duplex when the model is a duplex model (its pipeline
declares a `duplex_plugin` and the deploy configuration sets
`session_mode: duplex`); a stock Realtime client needs no vendor query
parameter, and `duplex=0` is the explicit opt-out. Such a server serves the
websocket route plus `POST /v1/chat/completions`, and no other turn-based
HTTP route. On a duplex server that chat route is not the turn-based path:
each request runs on a short-lived duplex session, so it holds one of
`duplex_session.max_sessions` for its lifetime and answers at the model's
real-time pace -- see [Full Duplex](full_duplex_api.md). Clients should
also verify the duplex capability payload because the query-parameter form
falls back to the ordinary realtime handler when duplex is unavailable.
## Related Endpoints
+64 -35
View File
@@ -15,40 +15,68 @@ capability gates, see the [Realtime Duplex API guide](realtime_duplex_api.md).
| Endpoint | Protocol | Recommended use |
| --- | --- | --- |
| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients |
| `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing |
Both endpoints use the same duplex engine and require the same model-side
adapter. Prefer the Realtime projection unless the native lifecycle events are
specifically required.
| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style events (the normative contract) | Applications and browser clients |
| `WS /v1/duplex` | Alias of `/v1/realtime?duplex=1` (same protocol, same handler) | Clients that prefer a dedicated path |
| Python: `DuplexOmni` / `InlineDuplexClient` | Typed commands and events in-process | Embedding the model without a server |
## Enable Full Duplex
The route becomes usable only when the deployment configuration explicitly
sets:
A model is served full duplex when its registered pipeline declares a
`duplex_plugin` (the model's `DuplexModelPlugin`) and its deploy configuration
sets `session_mode: duplex`. `vllm serve --omni` constructs `DuplexOmni`
instead of `AsyncOmni`. It serves
`/v1/realtime?duplex=1` (and its alias `/v1/duplex`),
`POST /v1/chat/completions`, `/v1/models` and `/health`; every other
turn-based HTTP route (speech, batch, embeddings, video, ...) answers "not
available". Set `session_mode: turn` to select the ordinary online serving
stack instead. The model's supported tasks and endpoint restrictions still apply.
`/v1/chat/completions` is the ordinary chat service running on the duplex
engine: a request is a turn-based generation on the same stages, served
alongside the live websocket sessions, with the request options the chat
service supports. It is wired only when the model's plugin declares
`DuplexCapabilities.supports_chat_completions`; on any other duplex model the
route reports "not available". A model that should not serve the route at all
lists it in the deploy configuration's `endpoint_restrictions`.
The deploy configuration of such a model must agree:
```yaml
session_mode: duplex
```
The selected model pipeline must also provide a duplex serving adapter.
Model-native deployments configure the corresponding engine runtime extension
and control plane as part of their registered pipeline.
A duplex-capable model must explicitly set `session_mode` to `duplex` or `turn`
in its deploy configuration, possibly through `base_config` inheritance.
Missing or invalid values fail at startup. The default MiniCPM-o deployment
continues to use duplex mode.
To run MiniCPM-o 4.5 with the turn-based engine:
```bash
vllm serve openbmb/MiniCPM-o-4_5 --omni \
--deploy-config vllm_omni/deploy/minicpmo_4_5_turn.yaml \
--trust-remote-code \
--port 8091
```
This profile inherits the default model and stage settings and overrides only
`session_mode`. It supports ordinary HTTP requests without creating a duplex
session handler. Mode selection happens at startup; a WebSocket query parameter
does not switch engines. The Python `Omni` / `AsyncOmni` APIs are unchanged.
!!! warning
`WS /v1/duplex` fails with `Duplex API is not available` when duplex is not
enabled. By contrast, `/v1/realtime?duplex=1` falls back to the ordinary
turn-based realtime handler when no duplex handler exists. Confirm that
On a deployment that is *not* duplex, `WS /v1/duplex` fails with
`Duplex API is not available` and `/v1/realtime?duplex=1` falls back to the
ordinary turn-based realtime handler. Confirm that
`session.created.session.capabilities` is present before treating the
connection as full duplex.
The current unified-runtime integrations are:
- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5.yaml`
(`session_mode: duplex`);
- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex;
- Nemotron VoiceChat, via its registered duplex plugin package.
**MiniCPM-o 4.5** (`vllm_omni/deploy/minicpmo_4_5.yaml`) is the only model
served over this endpoint today. PersonaPlex and Nemotron VoiceChat still carry
their pre-framework duplex code: their pipelines declare no `duplex_plugin`, so
they run turn-based until the follow-up PRs port them to the plugin contract
(RFC [vllm-omni#7181](https://github.com/vllm-project/vllm-omni/issues/7181)).
JoyVL is a separate HTTP interaction orchestrator and does not use these
WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
@@ -86,7 +114,7 @@ A typical `/v1/realtime?duplex=1` session follows this lifecycle:
3. Send `input_audio_buffer.append` events while microphone audio arrives.
4. Send `input_audio_buffer.commit` at a user-turn boundary when required by
the model policy.
5. Consume `response.created`, transcript deltas, `response.audio.delta`, and
5. Consume `response.created`, transcript deltas, `response.output_audio.delta`, and
`response.done` or `response.listen` events.
6. Send `playback.ack` after audio has been played when the session advertises
playback acknowledgement support.
@@ -101,22 +129,23 @@ deferred, treated as a short acknowledgement, or used to interrupt output.
The `session.created` payload includes capability fields such as
`supports_barge_in`, `supports_playback_ack`, `supports_multi_session`,
`supports_session_resume`, and `chunk_period_ms`. Treat this payload as the
runtime contract.
runtime contract and branch on the flags, never on the model name: a model
that supports native overlapping speech may still advertise
`supports_barge_in=false` when destructive output interruption and model-state
rewind are not validated for it. Capacity and session-resume behavior also
depend on the selected deployment configuration. The per-model table lives in
the [Realtime Duplex API guide](realtime_duplex_api.md#capability-negotiation-by-model).
For example, PersonaPlex supports native overlapping speech but currently
advertises `supports_barge_in=false`; destructive output interruption and
model-state rewind have not been validated for that integration. Capacity and
session-resume behavior also depend on the selected deployment configuration.
## Python API
## Native Protocol
`vllm_omni.entrypoints.duplex_omni.DuplexOmni` runs the same engine-resident
sessions in-process: `open_session()` returns a `DuplexSessionHandle` whose
`submit()` takes typed `DuplexCommand` objects and whose `events()` yields
typed `DuplexEvent` objects (each with a `to_realtime()` wire rendering).
`vllm_omni.clients.inline_duplex.InlineDuplexClient` exposes that handle
behind the `DuplexClient` API. See the
[Realtime Duplex API guide](realtime_duplex_api.md#using-the-python-api).
`WS /v1/duplex` exposes lower-level lifecycle names including
`session.create`, `input_audio_buffer.append`, `turn.signal`, `playback.ack`,
and `session.close`. It returns native session, input, response, overlap, and
error events. This protocol is experimental and may evolve with the runtime;
applications should use the provided Realtime client where possible.
See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo),
[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex),
See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo)
and the [full-duplex runtime design](../design/fullduplex.md)
for model-specific validation and architecture details.
+3 -3
View File
@@ -43,8 +43,8 @@ event is base64-encoded raw PCM16.
| Client to server | `input_audio_buffer.commit` with `final: true` | Marks the end of input |
| Server to client | `transcription.delta` | Carries incremental response text |
| Server to client | `transcription.done` | Carries final text and token usage |
| Server to client | `response.audio.delta` | Carries incremental PCM16 response audio |
| Server to client | `response.audio.done` | Marks the end of response audio |
| Server to client | `response.output_audio.delta` | Carries incremental PCM16 response audio |
| Server to client | `response.output_audio.done` | Marks the end of response audio |
| Server to client | `error` | Reports an invalid event, model, or audio payload |
A minimal client sends events in this order:
@@ -63,7 +63,7 @@ engine is consuming the stream.
## Audio Handling
- Input is mono PCM16 at 16 kHz for the Qwen3-Omni example.
- `response.audio.delta.audio` contains base64-encoded PCM16 bytes.
- `response.output_audio.delta.audio` contains base64-encoded PCM16 bytes.
- Read `sample_rate_hz` from each audio event instead of assuming an output
rate. Qwen3-Omni output is typically 24 kHz.
- Concatenate audio deltas in receive order to construct the output waveform.
+295 -144
View File
@@ -1,17 +1,47 @@
# Realtime Duplex API
vLLM-Omni serves full-duplex speech models over a WebSocket endpoint,
`/v1/realtime`, that speaks the OpenAI Realtime vocabulary plus a
`/v1/realtime?duplex=1`, that speaks the OpenAI Realtime vocabulary plus a
set of duplex extensions: the model decides when to listen and when to speak,
the user can talk over the assistant, playback progress is reported back so
history stays honest, and a dropped connection can resume the same session.
This page covers how to run a duplex deployment, how to drive it from Python
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
The handler is enabled by either a model-native `session_mode: duplex`
deployment or an explicit `duplex_session` configuration for turn-based
Server VAD. The runtime architecture is described in
[Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
The endpoint is served for models that ship a duplex plugin — including
MiniCPM-o 4.5 and Qwen3-Omni — and just for deploy configurations that declare
`session_mode: duplex`. PersonaPlex and Nemotron VoiceChat are ported to the
plugin contract in follow-up PRs and are not served over this endpoint yet. The runtime architecture
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
## Qwen3-Omni conversation history
Qwen3-Omni keeps incoming conversation items independently addressable by item
ID. Before applying the model's chat template, it merges consecutive user
items into one multimodal user message. An assistant message, including an
empty message reserved for an interrupted or unheard answer, separates turns.
The final user message places media before text instructions, following the
[official Qwen3-Omni demo's `format_history()`](https://github.com/QwenLM/Qwen3-Omni/blob/e4235853125589c789f06a2dd83e9f4126df5e9d/web_demo.py#L84-L179).
A camera image or text item may arrive after an audio commit and before
`response.create`. Those items belong to the same pending user turn: the
committed audio remains associated with its history item even when that item
is no longer last. Prompt preparation snapshots the selected inputs; later
items do not modify an already prepared request.
The duplex adapter retains its existing budgets: at most four audio payloads
(including the current input), an 8 MiB encoded-audio history budget (keeping
at least the newest payload), and at most eight prompt images. The recent
16-message window is rounded outwards to a complete turn. If any audio in an
older turn is no longer retained, the entire turn's model context, including
its images/text and assistant replies, is omitted. Images from that turn are
therefore **not** permanent session-wide model context. These limits differ
from the demo's one image-bearing turn and five audio-bearing turns; the
shared rule is grouping before pruning, rather than copying its UI budgets.
Grouping and pruning affect only the model-input view. They do not merge or
delete the source conversation items exposed to clients, and do not change
playback acknowledgement or interruption handling.
## Quick Start
@@ -26,11 +56,24 @@ vllm-omni serve openbmb/MiniCPM-o-4_5 \
```
`vllm_omni/deploy/minicpmo_4_5.yaml` declares `session_mode: duplex` and
`duplex_session.max_sessions: 4`, which mounts the WebSocket routes
`ws://<host>:8099/v1/realtime` (this page) and `ws://<host>:8099/v1/duplex`
(the native dialect for raw-protocol clients) next to the usual HTTP API.
PersonaPlex and Nemotron VoiceChat use `vllm_omni/deploy/personaplex.yaml` and
`vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml` in the same way.
`duplex_session.max_sessions: 4`. Because the MiniCPM-o 4.5 pipeline declares a
`duplex_plugin`, `vllm-omni serve` runs it through `DuplexOmni`: the server
mounts `ws://<host>:8099/v1/realtime?duplex=1` (this page; `ws://<host>:8099/v1/duplex`
is an alias of the same route), `POST /v1/chat/completions`, `/v1/models` and
`/health`, and nothing else. The remaining turn-based HTTP routes (speech,
batch, embeddings, video, ...) are not served in duplex mode.
`/v1/chat/completions` uses the ordinary chat service on the same engine
when the plugin declares `supports_chat_completions`. HTTP chat requests do
not create duplex sessions or consume session admission slots. Their options
follow the [Chat Completions API](chat_completions_api.md) and the model's
capabilities and endpoint restrictions.
For a turn-based online deployment, use `minicpmo_4_5_turn.yaml` or explicitly
set `session_mode: turn` in a deploy configuration derived from the model's
profile. This selects `AsyncOmni` and the ordinary API initialization instead
of the duplex handler. See [Full Duplex](full_duplex_api.md#enable-full-duplex)
for the command and configuration requirements.
### Run the Example Client
@@ -81,25 +124,24 @@ ships a preset:
| Preset | Input / output audio | What it sets |
| --- | --- | --- |
| `vllm_omni.clients.minicpmo_4_5.create_duplex_session_config(ref_audio=...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | `extra_body.native_duplex=True` (opt in to the model-native lane), `force_listen_count=0`, `overlap_policy="listen_only"`, `playback_commit_policy="ack_only"`; `ref_audio` is the assistant voice clip |
| `vllm_omni.clients.minicpmo_4_5.create_duplex_session_config(ref_audio=...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | `force_listen_count=0`, `overlap_policy="listen_only"`, `playback_commit_policy="ack_only"`; `ref_audio` is the assistant voice clip |
| `vllm_omni.clients.personaplex.create_duplex_session_config(voice="NATF2.pt", persona="")` | 24 kHz `pcm_f32le` / 24 kHz `pcm16` | bundled `.pt` voice prompt and the persona as `instructions` |
| `SessionConfig(...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | model-neutral defaults; pass `extra_body`, `turn_detection`, `overlap_policy`, `playback_commit_policy`, `instructions`, `voice`, `temperature` yourself |
Keyword overrides on a preset replace the corresponding `SessionConfig`
field, e.g. `create_duplex_session_config(ref_audio=..., temperature=0.6)`.
Constructor options: `session_id` (names the session the client creates; it
does not resume or take over an existing session — entering the client always
performs the `session.update` handshake), `reconnect` (a `ReconnectPolicy`,
default five attempts with 0.25–4 s jittered backoff; `None` disables
auto-resume), `heartbeat_interval_s` (default 30 s; `None` disables the lease
heartbeat), `handshake_timeout_s`, and `connect` (a custom transport factory,
used by tests and benchmarks). Resume is supported only as automatic reconnect
within the same `DuplexClient` instance (see
[Reconnect and resume](#reconnect-and-resume)); re-attaching from a new client
or process requires the wire-level `session.resume` handshake
(`resume_token`, `incarnation`, `last_received_server_event_seq`), which this
client does not expose.
Constructor options: `reconnect` (a `ReconnectPolicy`, default five
attempts with 0.25–4 s jittered backoff; `None` disables auto-resume),
`heartbeat_interval_s` (default 30 s; `None` disables the lease heartbeat),
`handshake_timeout_s`, and `connect` (a custom transport factory, used by
tests and benchmarks). The session id is allocated by the server and becomes
`client.session_id` once `session.created` arrives; a client cannot choose
it. Resume is supported only as automatic reconnect within the same
`DuplexClient` instance (see [Reconnect and resume](#reconnect-and-resume));
re-attaching from a new client or process requires the wire-level
`session.resume` handshake (`resume_token`,
`last_received_server_event_seq`), which this client does not expose.
### Stream audio and commit a turn
@@ -195,7 +237,7 @@ the stamped `event_id`.
On a transport drop the client reconnects with the `ReconnectPolicy` backoff
and sends `session.resume` from the last acknowledged `server_event_seq`.
Replayed events are deduplicated, consumers see one `ConnectionResumed`
event, and `client.incarnation` / `client.resume_token` are refreshed. The
event, and `client.resume_token` is refreshed. The
server keeps a detached session alive for `disconnect_grace_s` (30 s by
default); after that a resume fails with `session_resume_expired`. Resume is
gated by `capabilities.supports_session_resume` (MiniCPM-o 4.5 and Nemotron
@@ -238,9 +280,67 @@ definitions out of the dependency-free client. `audio_bytes(response_id)`, `coun
`openai-realtime-duplex` benchmark backend (`vllm-omni bench serve`) is built
on the same helpers.
## Using the Python API
`vllm_omni.entrypoints.duplex_omni.DuplexOmni` is the in-process Python API
of a duplex model: sessions run inside the engine, and the class only opens,
resumes and closes them and pipes typed commands in and typed events out
through a `DuplexSessionHandle`.
```python
from vllm_omni.engine.duplex.commands import AppendAudio, Commit
from vllm_omni.engine.duplex.events import AudioDelta, SessionClosed
from vllm_omni.entrypoints.duplex_omni import DuplexOmni
omni = DuplexOmni(model="openbmb/MiniCPM-o-4_5", trust_remote_code=True)
async with await omni.open_session({"modalities": ["audio", "text"], "ref_audio": ref_audio_data_url}) as session:
async def consume():
async for event in session.events(): # typed DuplexEvent objects
if isinstance(event, AudioDelta):
play(event.audio)
elif isinstance(event, ResponseDone):
break # one turn is enough here
task = asyncio.create_task(consume())
await session.submit(AppendAudio(audio=pcm16_bytes, format="pcm16", sample_rate_hz=16000))
await session.submit(Commit())
await task
# leaving the block closes the session (the last event is SessionClosed)
```
`open_session` takes a `DuplexSessionConfig` or the same Realtime `session`
object the WebSocket handshake takes; the session id is always allocated by
`DuplexOmni`. `DuplexSessionHandle` offers `submit(command)` plus one
convenience wrapper per command (`append_audio`, `commit`, `create_response`,
`cancel_response`, `barge_in`, `ack_playback`, `update`, `create_item`,
`delete_item`, `truncate_item`, `heartbeat`, ...). `events()` is a
single-consumer async iterator that ends after `session.closed` /
`session.expired`; rejected commands come back as `ErrorEvent`s on it.
Every event renders the wire JSON with `to_realtime()`, so anything written
against the WebSocket protocol works unchanged on the typed stream.
`vllm_omni.clients.inline_duplex.InlineDuplexClient` wraps a `DuplexOmni`
behind the `DuplexClient` API, so the same application code runs in-process
(no server) or over the network:
```python
from vllm_omni.clients.inline_duplex import InlineDuplexClient
from vllm_omni.clients.minicpmo_4_5 import create_duplex_session_config
config = create_duplex_session_config(ref_audio=audio_data_url(Path("reference_voice.wav")))
async with InlineDuplexClient(omni, model="openbmb/MiniCPM-o-4_5", config=config) as client:
await client.stream_pcm(question)
await client.commit()
async for response in client.responses():
...
```
`examples/online_serving/barge_in_client.py --inline` drives the barge-in
scenario this way. The inline client has no transport, so it has no
reconnect, resume, heartbeat or event-acknowledgement machinery.
## API Reference
This section is the complete wire contract of `/v1/realtime`: how
This section is the complete wire contract of `/v1/realtime?duplex=1`: how
the dialect relates to the OpenAI Realtime protocol, which surfaces each
model's capability flags gate, a catalogue of every client-to-server and
server-to-client event, and a JSON example for each. `DuplexClient` speaks
@@ -253,21 +353,26 @@ its protocol tests.
### Endpoint and transport
The endpoint is `ws(s)://<host>/v1/realtime`. Bare connections select the
session handler when the deployment declares either `session_mode: duplex`
or `duplex_session`. `?duplex=1` remains a compatibility alias, while
`?duplex=0` selects the legacy Realtime handler. Optional query parameters
are `model`, `session_id`,
`autostart` (`0` means resume-only), `resume`, and `native_duplex`
(`minicpmo45_native_duplex` is accepted as a deprecated alias and folded
into the canonical name). Every message is one JSON object per WebSocket
The endpoint is `ws(s)://<host>/v1/realtime`. On a deployment whose pipeline
declares a `duplex_plugin` and whose deploy configuration declares
`session_mode: duplex`, a bare connection is already a duplex session, so a
stock Realtime client needs no vendor query parameter. `?duplex=1` (`1`,
`true` or `on`) is the spelling this page uses and stays valid;
`?duplex=0` (`0`, `false`, `off`) explicitly opts out and selects the
turn-based Realtime handler, which a duplex server does not mount --
that connection is answered with `Realtime API is not available`.
`ws(s)://<host>/v1/duplex` is an alias that always selects the duplex
handler. Optional query parameters are `model`, `autostart` (`0` means
resume-only) and `resume`. Every message is one JSON object per WebSocket
text frame, discriminated by `type`. Inbound events are applied in arrival
order through one per-session mailbox; outbound events preserve that order
and carry a monotonically increasing `server_event_seq`, which is the replay
cursor for `session.resume`. Client events may carry an `event_id`, echoed in
`error.error.event_id` when the event is rejected.
Session identity is `(session_id, incarnation)`; `attachment_generation`
Session identity is the server-allocated `session_id` (`duplex-<uuid>`,
announced in `session.created`; a `session_id` the client puts in the session
object is ignored, and ids are never reused); `attachment_generation`
names the current socket; `epoch` is the interruption fence that advances on
every cancel, clear, or barge-in; `turn_id` counts committed model turns;
`response_id` (item id `item_<response_id>`) names one assistant utterance.
@@ -281,12 +386,13 @@ with no OpenAI counterpart.
The event vocabulary is uniform, but several surfaces are gated by the
`capabilities` object the server returns in `session.created`; a client must
branch on those flags rather than on the model name. The current model
plugins advertise:
branch on those flags rather than on the model name. MiniCPM-o 4.5 is the
only model on the plugin contract today; the other two columns record what
their integrations advertise once the follow-up PRs port them:
| Capability | MiniCPM-o 4.5 | PersonaPlex | Nemotron VoiceChat | Gated surface |
| --- | --- | --- | --- | --- |
| `implementation_level` | `model_native_duplex` (when `extra_body.native_duplex=true`) | `model_native_duplex` | `model_native_duplex` | model-owned `response.listen` / `response.speak`; the chat-fallback lane otherwise |
| `implementation_level` | `model_native_duplex` | `model_native_duplex` | `model_native_duplex` | model-owned `response.listen` / `response.speak` (constant: every duplex model is model-native) |
| `chunk_period_ms` | 1000 | 80 | 80 | the model unit that `response.listen` decisions and camera frames align to |
| `supports_session_resume` | yes | no | yes | `session.resume`, `session.resumed`, `session.replaced`, replay after a transport drop |
| `supports_barge_in` | yes | no | no | `barge_in`, `turn.signal{event:"barge_in"}`, `overlap_policy=barge_in_on_speech`, `turn_detection.server_vad` |
@@ -311,13 +417,13 @@ work unmodified. The overlap falls into the three tiers used by the event catalo
| Direction | Messages |
| --- | --- |
| Client → server | `session.update`, `input_audio_buffer.append` / `.commit` / `.clear`, `conversation.item.create` / `.retrieve` / `.truncate` / `.delete`, `response.create`, `response.cancel`, `output_audio_buffer.clear` (OpenAI: WebRTC-only; here also a WebSocket event) |
| Server → client | `error` (same `{type, code, message, event_id, param}` envelope and the same classes `invalid_request_error` / `server_error` / `rate_limit_error`), `session.created`, `session.updated`, `input_audio_buffer.committed` / `.cleared` / `.speech_started` / `.speech_stopped`, `conversation.item.created` / `.added` / `.done` / `.deleted` / `.retrieved` / `.truncated`, `conversation.item.input_audio_transcription.completed`, `response.created`, `response.done`, `response.output_item.added` / `.done`, `response.content_part.added` / `.done`, `response.audio.delta` / `.done`, `response.audio_transcript.delta` / `.done`, `response.output_text.delta` / `.done`, `response.function_call_arguments.delta` / `.done`, `rate_limits.updated`, `output_audio_buffer.cleared` |
| Server → client | `error` (same `{type, code, message, event_id, param}` envelope and the same classes `invalid_request_error` / `server_error` / `rate_limit_error`), `session.created`, `session.updated`, `input_audio_buffer.committed` / `.cleared` / `.speech_started` / `.speech_stopped`, `conversation.item.created` / `.added` / `.done` / `.deleted` / `.retrieved` / `.truncated`, `conversation.item.input_audio_transcription.completed`, `response.created`, `response.done`, `response.output_item.added` / `.done`, `response.content_part.added` / `.done`, `response.output_audio.delta` / `.done`, `response.output_audio_transcript.delta` / `.done`, `response.output_text.delta` / `.done`, `response.function_call_arguments.delta` / `.done`, `rate_limits.updated`, `output_audio_buffer.cleared` |
| Session fields | `model`, `modalities` / `output_modalities`, `instructions`, `voice`, `input_audio_format` / `output_audio_format` (beta spelling) **and** `audio.input.format` / `audio.output.format` objects (GA spelling), `turn_detection.server_vad {threshold, prefix_padding_ms, silence_duration_ms}`, `input_audio_transcription`, `input_audio_noise_reduction`, `tools`, `tool_choice`, `temperature`, `max_response_output_tokens`, `speed`, `tracing` |
| Object shapes | `realtime.session`, `realtime.item`, `realtime.response`; item content parts `input_text` / `input_audio` / `output_text` / `output_audio`; `function_call` and `function_call_output` items; `previous_item_id` chaining; `output_index` / `content_index` addressing |
| Sequencing | `response.created → output_item.added → content_part.added → deltas → *.done → content_part.done → output_item.done → response.done → rate_limits.updated`; `speech_started → speech_stopped → committed` |
The server deliberately emits and accepts **both** the beta and the GA
spellings at once (`response.audio.delta` alongside `output_modalities`,
spellings at once (`response.output_audio.delta` alongside `output_modalities`,
`conversation.item.added` alongside `conversation.item.created`) so either
generation of OpenAI client parses the stream.
@@ -327,9 +433,9 @@ A stock client ignores the extra keys; the extensions are additive.
| Message | Extension |
| --- | --- |
| `session.created` / `session.updated` | top-level `incarnation`, `attachment_generation`, `resume_token`; inside `session`: `state`, `turn_state`, `epoch`, `turn_id`, `active_request_id`, `active_response_id`, `active_response_turn_id`, `overlap_policy`, `overlap_*_ms/rms`, `playback_commit_policy`, `playback`, `capabilities`, `ref_audio`, `extra_body`, `idle_timeout_s`, `response_format` |
| `session.created` / `session.updated` | top-level `attachment_generation`, `resume_token`; inside `session`: `state`, `turn_state`, `epoch`, `turn_id`, `active_request_id`, `active_response_id`, `active_response_turn_id`, `overlap_policy`, `overlap_*_ms/rms`, `playback_commit_policy`, `playback`, `capabilities`, `ref_audio`, `extra_body`, `idle_timeout_s`, `response_format` |
| `response.created` / `response.done` / `response.listen` | `response_id` at top level; the raw duplex event under `response.metadata` (`duplex_event`); `status_details.reason` uses vLLM reasons (`barge_in`, `client_cancelled`, `new_response`, …) |
| `response.audio.delta` | `format`, `sample_rate_hz`, `metadata{session_id, epoch, model_speak, end_of_turn, audio_duration_ms, audio_text_marks, playback}` |
| `response.output_audio.delta` | `format`, `sample_rate_hz`, `metadata{session_id, epoch, model_speak, end_of_turn, audio_duration_ms, audio_text_marks, playback}` |
| `response.speak` (inserted before the first delta) | not an OpenAI event, but rides the OpenAI response envelope (`response_id`, `item_id`, `output_index`, `content_index`) |
| `input_audio_buffer.append` | `is_speech`, `video_frames` (+ `max_slice_nums`), `duration_ms`, `audio_end_ms`, per-event `format` / `sample_rate_hz` |
| `input_audio_buffer.commit` | `final`, `response_create`, `is_speech:false` (silence declaration) |
@@ -343,14 +449,9 @@ Semantic divergences hidden behind shared names:
before any commit; a commit may end in `response.listen` and no
response, and the model may open a response with no commit at all.
OpenAI: commit ⇒ item, `response.create` ⇒ exactly one response.
- With `turn_detection={"type":"server_vad"}`, omitted `interrupt_response`
defaults to `false` on turn-based models such as Qwen3-Omni, enabling
endpointing and automatic responses without interruption. Explicit `true`
is rejected there. Model-native duplex keeps the `true` default and rejects
`false`; its VAD implies `overlap_policy=barge_in_on_speech`.
This capability-dependent default differs from OpenAI's `true` default;
the effective configuration is returned in `session.created` / `session.updated`.
`semantic_vad` is unsupported.
- `turn_detection.interrupt_response=false` is rejected;
`create_response` is ignored; `semantic_vad` is unsupported; the VAD
runs per session (Silero) and implies `overlap_policy=barge_in_on_speech`.
- `rate_limits.updated` is always an empty list (compatibility only).
- Cancellation never reuses a `response_id`; truncation is driven by
`playback.ack` as well as `conversation.item.truncate`.
@@ -359,11 +460,11 @@ Semantic divergences hidden behind shared names:
| Area | Messages / fields |
| --- | --- |
| Session lifetime | `session.heartbeat` / `session.heartbeat_ack`, `session.event_ack`, `session.close` / `session.closed`, `session.resume` / `session.resumed` / `session.replaced` / `session.expired` / `session.resync_required`; `server_event_seq`, `resume_token`, `incarnation`, `attachment_generation`; query params `?duplex=1`, `autostart`, `resume`, `native_duplex` |
| Session lifetime | `session.heartbeat` / `session.heartbeat_ack`, `session.event_ack`, `session.close` / `session.closed`, `session.resume` / `session.resumed` / `session.replaced` / `session.expired` / `session.resync_required`; `server_event_seq`, `resume_token`, `attachment_generation`; query params `?duplex=1`, `autostart`, `resume` |
| Turn-taking | `response.speak`, `response.listen`, `overlap.decision`, `overlap_policy`, `barge_in`, `turn.signal`, `input.cancel`, `input.text.append`, `epoch` / `turn_id` fencing, `force_listen` |
| Playback truth | `playback.ack` / `playback.acknowledged`, `playback_commit_policy` (`ack_only` \| `commit_all_on_done`), `playback{generated_ms, sent_ms, played_ms, committed_ms}`, `history_committed`, `audio_text_marks` |
| Model negotiation | `capabilities{implementation_level, supports_input_append, supports_barge_in, supports_session_resume, chunk_period_ms, input_modes, …}`, `ref_audio`, `extra_body` (`auto_response`, `native_duplex` — deprecated alias `minicpmo45_native_duplex` is folded into it, `force_listen_count`, `duplex_initial_user_text`) |
| Diagnostics | `runtime.control`, `duplex.function_call.done`, the full error-code vocabulary (`REALTIME_ERROR_TYPES_BY_CODE` in `vllm_omni/entrypoints/duplex/realtime_state.py`) |
| Model negotiation | `capabilities{implementation_level, supports_input_append, supports_barge_in, supports_session_resume, chunk_period_ms, input_modes, …}`, `ref_audio`, `extra_body` (`auto_response`, `force_listen_count`, `duplex_initial_user_text`) |
| Diagnostics | `duplex.function_call.done`, the full error-code vocabulary (`REALTIME_ERROR_TYPES_BY_CODE` in `vllm_omni/engine/duplex/events.py`) |
#### Consequences for clients
@@ -381,14 +482,61 @@ Semantic divergences hidden behind shared names:
the Tier 3 surface on top of the same Tier 1 events, so both kinds of
client can share one deployment and one event log.
### Turn-based Chat audio metadata
For turn-based models served through Chat fallback, non-empty audio choices
produced by the Omni Chat audio encoder include `audio_metadata`, in both
streaming and complete Chat responses. Text-only choices do not require it.
| Field | Meaning |
| --- | --- |
| `format` | Encoding of the returned audio, such as `pcm` or `wav`; `pcm16` requests normalize to `pcm`. |
| `sample_rate_hz` | Sample rate used by the encoder, in samples per second. |
| `frame_count` | Samples per channel in this chunk, or in the complete waveform for a non-streaming choice. Not a byte count. |
| `channels` | Number of audio channels. |
These values describe the waveform after audio transformations. Compressed
formats may add codec padding that is not included in `frame_count`.
The encoder uses a model-reported sample rate when available; Qwen3-Omni
currently omits it and uses the existing 24 kHz default. This metadata does
not independently determine a model's sample rate.
Chat fallback requests the session's output format explicitly. Its internal
`response.output_audio.delta` carries `format`, `sample_rate_hz`, `channels`,
and `audio_duration_ms`. The Realtime endpoint exposes output `format` and
`sample_rate_hz` on `response.audio.delta`, with duration in
`metadata.audio_duration_ms`; it does not currently forward `channels`.
The duration is cumulative for the response:
integer frames are summed before converting to milliseconds and rounding down.
Non-empty audio without valid metadata, or a sample-rate change within one
response, fails that fallback response with `response_error`.
In this Chat fallback path, `response.done.response.metadata.playback` reports:
- `generated_ms`: cumulative source-waveform duration.
- `sent_ms`: cumulative audio accepted into the server output path, including
queued or journaled audio. A journaled event remains accepted if the
subsequent send fails; this is not proof of socket delivery.
- `played_ms`: cumulative playback reported by the client.
- `committed_ms`: playback position used for history reconciliation.
The native model path still records audio before sending; it does not yet
share Chat fallback's output-acceptance timing. Only client playback reports
establish what was played. The existing duration-proportional text truncation
remains an approximation, not word-level audio alignment.
### Event catalogue
Every event type the Realtime route accepts or emits, with a worked example in the next section: 21
client→server and 42 server→client. Aliases (`push_chunk`,
`input.audio.append`, `input_text.append`, `push_text`, `signal_turn`,
`audio.playback_ack`, `close_session`, `close`, `session_close`) share the
payload of their canonical event and are not listed separately; the alias
table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
client→server and 42 server→client. Aliases (`input_text.append`,
`push_text`, `signal_turn`, `audio.playback_ack`, `close_session`, `close`)
share the payload of their canonical event and are not listed
separately; the alias table is `translate_realtime_command` in
`vllm_omni/engine/duplex/realtime_commands.py`. The pre-Realtime WAV-append
aliases `push_chunk` and `input.audio.append` are **not** accepted on this
route: send `input_audio_buffer.append` with one of the supported input
formats (`pcm16`, `pcm_s16le`, `s16le`, `pcm_f32le`, `g711_ulaw`,
`g711_alaw`).
#### Client to server
@@ -398,18 +546,18 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
| `session.heartbeat` | 3 | Keepalive; refreshes the engine session lease. |
| `session.event_ack` | 3 | Acknowledges received server events by `server_event_seq`; trims the resume replay journal. |
| `session.close` | 3 | Graceful close; answered with `session.closed`. |
| `session.resume` | 3 | Re-attach a live session on a new socket (`resume_token`, `incarnation`, `last_received_server_event_seq`). |
| `session.resume` | 3 | Re-attach a live session on a new socket (`session_id`, `resume_token`, `last_received_server_event_seq`). |
| `input_audio_buffer.append` | 2 | Append one audio chunk. OpenAI name; extended with `is_speech`, `video_frames`, `duration_ms`, `audio_end_ms`, per-event `format`/`sample_rate_hz`. |
| `input_audio_buffer.commit` | 2 | Seal the buffered utterance into a user item. OpenAI name; extended with `final`, `response_create`, `is_speech:false`. |
| `input.commit` | 3 | Alias of `input_audio_buffer.commit` accepted by the runner. |
| `input_audio_buffer.clear` | 1 | Drop un-committed input audio. |
| `input.cancel` | 3 | Cancel pending input; advances `epoch`. |
| `input.text.append` | 3 | Append text to the open input item (chat-fallback lane only). |
| `input.text.append` | 3 | Append text input; model-native sessions reject it with `native_text_append_unsupported`. |
| `conversation.item.create` | 1 | Add a completed user item (text, audio, or `function_call_output`) to history. |
| `conversation.item.retrieve` | 1 | Fetch a stored item by id. |
| `conversation.item.delete` | 1 | Delete a stored item by id. |
| `conversation.item.truncate` | 1 | Truncate an assistant item's audio/transcript at `audio_end_ms`. |
| `response.create` | 1 | Explicitly request a response with per-response overrides. |
| `response.create` | 1 | Explicitly request a response with per-response overrides. Requires committed audio input; otherwise `response_create_without_input`. |
| `response.cancel` | 1 | Cancel the active (or named) response; advances `epoch`. |
| `output_audio_buffer.clear` | 1 | Discard queued output audio; advances `epoch` (OpenAI: WebRTC-only, here also WebSocket). |
| `barge_in` | 3 | Explicit hard interrupt; requires `capabilities.supports_barge_in`. |
@@ -420,10 +568,10 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
| Event type | Tier | Description |
| --- | --- | --- |
| `session.created` | 2 | Session opened. OpenAI name; adds `incarnation`, `attachment_generation`, `resume_token` and vLLM-Omni keys inside `session` (`epoch`, `turn_id`, `playback`, `capabilities`, ...). |
| `session.created` | 2 | Session opened. OpenAI name; adds `attachment_generation`, `resume_token` and vLLM-Omni keys inside `session` (`id` is the server-allocated session id, `epoch`, `turn_id`, `playback`, `capabilities`, ...). |
| `session.updated` | 2 | Echo of the effective session config after every `session.update` (same extended `session` object). |
| `session.heartbeat_ack` | 3 | Reply to `session.heartbeat`. |
| `session.closed` | 3 | Last event on the socket; `reason` ∈ `session_close`, `timeout`, `disconnect`, ... |
| `session.closed` | 3 | Last event on the socket, emitted once the engine released the session; `reason` ∈ `client_close`, `disconnect`, `timeout`, `transport_error`, `shutdown`, ... |
| `session.resumed` | 3 | Resume accepted; carries the new `attachment_generation` and rotated `resume_token`; journaled events are replayed after it. |
| `session.replaced` | 3 | Sent to the superseded socket when another socket resumes the session. |
| `session.expired` | 3 | Engine lease reaped (`disconnect_grace_expired`, `idle_ttl_expired`); socket closes after it. |
@@ -443,12 +591,12 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
| `response.output_item.added` | 1 | Assistant item (or `function_call` item) attached to the response. |
| `response.content_part.added` | 1 | Audio/text content part opened on the assistant item. |
| `response.speak` | 3 | Model chose to speak (native lane); at most once per response, before the first audio delta. |
| `response.audio.delta` | 2 | One ordered audio chunk. OpenAI name; adds `format`, `sample_rate_hz`, `metadata{epoch, model_speak, end_of_turn, audio_text_marks, playback}`. |
| `response.audio_transcript.delta` | 1 | Transcript text paired one-to-one with each audio delta. |
| `response.output_audio.delta` | 2 | One ordered audio chunk. OpenAI name; adds `format`, `sample_rate_hz`, `metadata{epoch, model_speak, end_of_turn, audio_text_marks, playback}`. |
| `response.output_audio_transcript.delta` | 1 | Transcript text paired one-to-one with each audio delta. |
| `response.output_text.delta` | 1 | Text delta for text-modality responses. |
| `response.output_text.done` | 1 | Final text of a text-modality response. |
| `response.audio.done` | 1 | Audio stream closed for the response. |
| `response.audio_transcript.done` | 1 | Full transcript (concatenation of the deltas). |
| `response.output_audio.done` | 1 | Audio stream closed for the response. |
| `response.output_audio_transcript.done` | 1 | Full transcript (concatenation of the deltas). |
| `response.content_part.done` | 1 | Content part finalized with its transcript/text. |
| `response.output_item.done` | 1 | Assistant item finalized. |
| `response.done` | 2 | Terminal event; `status` ∈ `completed` \| `cancelled` \| `failed`. OpenAI name; vLLM-Omni `status_details.reason` values and raw duplex event under `metadata`. |
@@ -459,8 +607,8 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
| `duplex.function_call.done` | 3 | Raw fallback when a model tool call is malformed. |
| `output_audio_buffer.cleared` | 1 | Reply to `output_audio_buffer.clear`. |
| `playback.acknowledged` | 3 | Reply to `playback.ack`; returns the playback ledger and `history_committed`. |
| `overlap.decision` | 3 | How an append that overlapped an active response was handled (`drop` \| `listen` \| `barge_in`). |
| `runtime.control` | 3 | Diagnostic: an engine control signal was partially unsupported. |
| `overlap.decision` | 3 | How an append that overlapped an active response was handled (`drop` \| `listen` \| `barge_in`); details under `metadata`. |
| `turn.event` | 3 | A turn-taking signal the session accepted (`event`, resulting `turn_state`). |
| `error` | 2 | Error envelope. OpenAI shape and classes; vLLM-Omni error codes. |
### Message examples
@@ -479,7 +627,6 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
"event_id": "evt_0001",
"session": {
"model": "openbmb/MiniCPM-o-4_5",
"session_id": "sess_demo_01",
"modalities": ["audio", "text"],
"instructions": "You are a concise voice assistant.",
"input_audio_format": "pcm16",
@@ -492,7 +639,6 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
"idle_timeout_s": 300,
"extra_body": {
"auto_response": true,
"native_duplex": true,
"force_listen_count": 0
}
}
@@ -504,13 +650,12 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
```json
{
"type": "session.created",
"incarnation": 0,
"attachment_generation": 1,
"resume_token": "rt_3f9c…",
"session": {
"object": "realtime.session",
"type": "realtime",
"id": "sess_demo_01",
"id": "duplex-5f2a…",
"model": "openbmb/MiniCPM-o-4_5",
"state": "open",
"turn_state": "idle",
@@ -563,7 +708,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
`session.created` and every later `session.update`)
```json
{"type": "session.updated", "session": {"object": "realtime.session", "id": "sess_demo_01", "…": "…"}}
{"type": "session.updated", "session": {"object": "realtime.session", "id": "duplex-5f2a…", "…": "…"}}
```
`C→S session.heartbeat` / `S→C session.heartbeat_ack`
@@ -573,7 +718,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
```
```json
{"type": "session.heartbeat_ack", "session_id": "sess_demo_01"}
{"type": "session.heartbeat_ack", "session_id": "duplex-5f2a…"}
```
`C→S session.event_ack`
@@ -589,7 +734,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
```
```json
{"type": "session.closed", "session_id": "sess_demo_01", "reason": "session_close"}
{"type": "session.closed", "session_id": "duplex-5f2a…", "epoch": 3, "reason": "client_close", "event": {"type": "session.closed", "reason": "client_close"}}
```
`C→S session.resume` (new socket, `?duplex=1&autostart=0`)
@@ -597,8 +742,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
```json
{
"type": "session.resume",
"session_id": "sess_demo_01",
"incarnation": 0,
"session_id": "duplex-5f2a…",
"resume_token": "rt_3f9c…",
"last_received_server_event_seq": 40
}
@@ -610,8 +754,8 @@ events with `server_event_seq > 40` are replayed in order)
```json
{
"type": "session.resumed",
"session_id": "sess_demo_01",
"incarnation": 0,
"session_id": "duplex-5f2a…",
"session": {"id": "duplex-5f2a…", "state": "open", "epoch": 3, "turn_id": 2, "...": "the engine's current session object"},
"attachment_generation": 2,
"resume_token": "rt_a71e…"
}
@@ -620,19 +764,19 @@ events with `server_event_seq > 40` are replayed in order)
`S→C session.replaced` (sent to the superseded socket, which is then closed)
```json
{"type": "session.replaced", "session_id": "sess_demo_01", "attachment_generation": 1}
{"type": "session.replaced", "session_id": "duplex-5f2a…", "attachment_generation": 1}
```
`S→C session.expired`
```json
{"type": "session.expired", "session_id": "sess_demo_01", "incarnation": 0, "reason": "disconnect_grace_expired"}
{"type": "session.expired", "session_id": "duplex-5f2a…", "reason": "disconnect_grace_expired"}
```
`S→C session.resync_required`
```json
{"type": "session.resync_required", "session_id": "sess_demo_01", "reason": "journal_gap"}
{"type": "session.resync_required", "session_id": "duplex-5f2a…", "reason": "journal_gap"}
```
#### Input
@@ -653,7 +797,10 @@ events with `server_event_seq > 40` are replayed in order)
```
with camera frames (omni video; rides the append that closes a 1 s model
unit; each entry is a **bare** base64 JPEG/PNG — no `data:` URL prefix)
unit; each entry is a **bare** base64 JPEG/PNG — no `data:` URL prefix).
Only a model whose Stage 0 interleaves a frame track at unit boundaries takes
this field; a turn model has no such boundary and rejects it, expecting
`conversation.item.create` with `input_image` content instead.
```json
{
@@ -723,7 +870,7 @@ silence/noise declaration (no runtime append; answered with `response.listen`)
"item_id": "item_9b2d…",
"event": {
"type": "input.committed",
"session_id": "sess_demo_01",
"session_id": "duplex-5f2a…",
"turn_id": 1,
"epoch": 0,
"history_len": 1,
@@ -753,7 +900,7 @@ deferred commit during an active response (`event.response_create_deferred`)
"type": "input_audio_buffer.committed",
"previous_item_id": "item_9b2d…",
"item_id": "item_c044…",
"event": {"type": "input.committed", "session_id": "sess_demo_01", "turn_id": 2, "epoch": 0, "history_len": 3, "response_create_deferred": true, "message": {"…": "…"}}
"event": {"type": "input.committed", "session_id": "duplex-5f2a…", "turn_id": 2, "epoch": 0, "history_len": 3, "response_create_deferred": true, "message": {"…": "…"}}
}
```
@@ -830,6 +977,10 @@ deferred commit during an active response (`event.response_create_deferred`)
{"type": "conversation.item.truncated", "item_id": "item_resp_01", "content_index": 0, "audio_end_ms": 1850, "event": {"…": "…"}}
```
After this acknowledgement, `conversation.item.retrieve` returns the truncated
transcript. Without text/audio alignment marks, its prefix is estimated from
the requested position and audio duration; this is not exact word alignment.
#### Response lifecycle
`C→S response.create`
@@ -862,7 +1013,7 @@ deferred commit during an active response (`event.response_create_deferred`)
"status_details": null,
"output": [],
"modalities": ["audio", "text"],
"metadata": {"duplex_event": {"type": "response.created", "session_id": "sess_demo_01", "response_id": "resp_01", "epoch": 0, "turn_id": 1}}
"metadata": {"duplex_event": {"type": "response.created", "session_id": "duplex-5f2a…", "response_id": "resp_01", "epoch": 0, "turn_id": 1}}
}
}
```
@@ -898,15 +1049,15 @@ deferred commit during an active response (`event.response_create_deferred`)
"item_id": "item_resp_01",
"output_index": 0,
"content_index": 0,
"metadata": {"session_id": "sess_demo_01", "epoch": 0, "model_speak": true}
"metadata": {"session_id": "duplex-5f2a…", "epoch": 0, "model_speak": true}
}
```
`S→C response.audio.delta` (one ~1 s unit, 24 kHz PCM16)
`S→C response.output_audio.delta` (one ~1 s unit, 24 kHz PCM16)
```json
{
"type": "response.audio.delta",
"type": "response.output_audio.delta",
"response_id": "resp_01",
"item_id": "item_resp_01",
"output_index": 0,
@@ -915,7 +1066,7 @@ deferred commit during an active response (`event.response_create_deferred`)
"format": "pcm16",
"sample_rate_hz": 24000,
"metadata": {
"session_id": "sess_demo_01",
"session_id": "duplex-5f2a…",
"epoch": 0,
"model_speak": true,
"end_of_turn": false,
@@ -926,10 +1077,10 @@ deferred commit during an active response (`event.response_create_deferred`)
}
```
`S→C response.audio_transcript.delta` (exactly one per audio delta)
`S→C response.output_audio_transcript.delta` (exactly one per audio delta)
```json
{"type": "response.audio_transcript.delta", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "delta": "It is sunny"}
{"type": "response.output_audio_transcript.delta", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "delta": "It is sunny"}
```
`S→C response.output_text.delta` / `response.output_text.done` (text modality)
@@ -942,14 +1093,14 @@ deferred commit during an active response (`event.response_create_deferred`)
{"type": "response.output_text.done", "response_id": "resp_02", "item_id": "item_resp_02", "output_index": 0, "content_index": 0, "text": "Tomorrow looks cloudy."}
```
`S→C response.audio.done` / `response.audio_transcript.done`
`S→C response.output_audio.done` / `response.output_audio_transcript.done`
```json
{"type": "response.audio.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0}
{"type": "response.output_audio.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0}
```
```json
{"type": "response.audio_transcript.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "transcript": "It is sunny, 24 degrees."}
{"type": "response.output_audio_transcript.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "transcript": "It is sunny, 24 degrees."}
```
`S→C response.content_part.done`
@@ -980,7 +1131,7 @@ deferred commit during an active response (`event.response_create_deferred`)
"status": "completed",
"status_details": null,
"output": [{"id": "item_resp_01", "object": "realtime.item", "type": "message", "role": "assistant", "status": "completed", "content": [{"type": "output_audio", "transcript": "It is sunny, 24 degrees."}]}],
"metadata": {"type": "response.done", "session_id": "sess_demo_01", "response_id": "resp_01", "epoch": 0, "committed": true, "playback": {"generated_ms": 2400, "sent_ms": 2400, "played_ms": 2400, "committed_ms": 2400}}
"metadata": {"type": "response.done", "session_id": "duplex-5f2a…", "response_id": "resp_01", "epoch": 0, "committed": true, "playback": {"generated_ms": 2400, "sent_ms": 2400, "played_ms": 2400, "committed_ms": 2400}}
}
}
```
@@ -1006,9 +1157,9 @@ deferred commit during an active response (`event.response_create_deferred`)
```json
{
"type": "response.listen",
"session_id": "sess_demo_01",
"session_id": "duplex-5f2a…",
"epoch": 0,
"response": {"object": "realtime.response", "status": "listening", "metadata": {"type": "response.listen", "session_id": "sess_demo_01", "epoch": 0, "reason": "model_listen"}}
"response": {"object": "realtime.response", "status": "listening", "metadata": {"type": "response.listen", "session_id": "duplex-5f2a…", "epoch": 0, "reason": "model_listen"}}
}
```
@@ -1055,7 +1206,7 @@ function-call response (Nemotron VoiceChat tools)
raw fallback when a model tool call is malformed (`call_id`/`name` missing)
```json
{"type": "duplex.function_call.done", "event": {"type": "function_call.done", "session_id": "sess_demo_01", "epoch": 0, "call_id": null, "name": null, "arguments": "{"}}
{"type": "duplex.function_call.done", "event": {"type": "function_call.done", "session_id": "duplex-5f2a…", "epoch": 0, "call_id": null, "name": null, "arguments": "{"}}
```
#### Playback
@@ -1071,7 +1222,7 @@ raw fallback when a model tool call is malformed (`call_id`/`name` missing)
"type": "playback.acknowledged",
"event": {
"type": "playback.acknowledged",
"session_id": "sess_demo_01",
"session_id": "duplex-5f2a…",
"epoch": 0,
"item_id": "item_resp_01",
"played_ms": 1000,
@@ -1096,20 +1247,17 @@ truncating ack after an interruption (what was actually heard)
```json
{
"type": "overlap.decision",
"session_id": "sess_demo_01",
"session_id": "duplex-5f2a…",
"epoch": 0,
"policy": "listen_only",
"action": "listen",
"reason": "long_overlap_speech",
"overlap_ms": 1400,
"buffer_audio": true,
"defer_runtime_append": true,
"force_listen": true
"metadata": {"overlap_ms": 1400, "buffer_audio": true, "defer_runtime_append": true, "force_listen": true}
}
```
```json
{"type": "overlap.decision", "session_id": "sess_demo_01", "epoch": 0, "policy": "barge_in_on_speech", "action": "barge_in", "reason": "server_vad_utterance_active"}
{"type": "overlap.decision", "session_id": "duplex-5f2a…", "epoch": 0, "policy": "barge_in_on_speech", "action": "barge_in", "reason": "server_vad_utterance_active"}
```
server-VAD session (`session.update` fragment) and the annotated append the
@@ -1123,12 +1271,6 @@ translator forwards internally (shown for reference; clients never see it)
{"type": "input_audio_buffer.append", "audio": "…", "format": "pcm16", "sample_rate_hz": 16000, "is_speech": true, "force_listen": true, "vad": {"backend": "silero", "is_speech": true, "speech_active": true, "speech_started": true, "speech_stopped": false, "speech_probability": 0.93}}
```
`S→C runtime.control` (only when an engine signal was partially unsupported)
```json
{"type": "runtime.control", "session_id": "sess_demo_01", "epoch": 1, "result": {"…": "redacted engine control result"}}
```
#### Errors
`S→C error` — Realtime shape
@@ -1154,71 +1296,80 @@ translator forwards internally (shown for reference; clients never see it)
```
```json
{"type": "error", "error": {"type": "rate_limit_error", "code": "duplex_session_busy", "message": "duplex session capacity reached (max_sessions=4)"}}
{"type": "error", "error": {"type": "rate_limit_error", "code": "resource_exhausted", "message": "duplex_session_capacity_exhausted: limit=4"}}
```
Handshake-stage errors (before the Realtime projector is bound) use the
flat native shape:
```json
{"type": "error", "error": "Unknown or expired duplex session: sess_demo_01", "code": "session_resume_expired"}
{"type": "error", "error": "Unknown or expired duplex session: duplex-5f2a…", "code": "session_resume_expired"}
```
#### Coverage
The examples above cover every event type the Realtime route emits or
accepts (verified by diffing the `type` literals in
`vllm_omni/entrypoints/duplex/` against this document). Not shown, because
they never reach a `/v1/realtime?duplex=1` client unprojected: the native
`/v1/duplex` dialect (`session.create` / `open_session` / `session.config`,
`input.committed`, `input.cancelled`, `audio.cancelled`,
`response.output_audio.delta`, `response.text.delta`, `response.message`,
`function_call.done`) — their Realtime projections are the examples above
(see the name map at the end of this section) — and the runner-internal markers `__timeout__`,
`__disconnect__`, `__replaced_attachment__`. Input aliases (`push_chunk`,
`input.audio.append`, `input_text.append`, `push_text`, `signal_turn`,
`audio.playback_ack`, `close_session`, `close`, `session_close`) share the
payload of their canonical event (see the alias table in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md)).
`vllm_omni/engine/duplex/events.py` and `vllm_omni/engine/duplex/commands.py`
against this document). Not shown, because they never reach a client
unprojected: the session-internal events the runner produces
(`input.committed`, `input.cancelled`, `audio.cancelled`,
`response.output_audio.delta`, `response.text.delta`, `function_call.done`)
— their Realtime projections are the examples above (see the name map at the
end of this section). Every public event is a typed `DuplexEvent` whose
`to_realtime()` is the wire object; every client event is parsed into a
typed `DuplexCommand` by `command_from_realtime()`.
#### Native to Realtime name map
#### Internal to Realtime name map
The native `/v1/duplex` dialect is what the session runner produces; the
Realtime projector renames and fans out these events before they reach a
`/v1/realtime?duplex=1` client.
The session runner emits these session-internal events; the Realtime
projector (`vllm_omni/engine/duplex/realtime_events.py`) renames and fans
them out into typed events before they reach a client.
| Native / internal event | Realtime projection |
| Internal event | Realtime projection |
| --- | --- |
| `session.create` (`open_session`, `session.config`) | first `session.update` |
| `input.committed` | `conversation.item.added` / `conversation.item.created`, `input_audio_buffer.committed`, `conversation.item.input_audio_transcription.completed`, `conversation.item.done` |
| `input.cancelled` | `input_audio_buffer.cleared` |
| `response.output_audio.delta` (with `audio_transcript`) | optional `response.speak`, then `response.audio.delta` and `response.audio_transcript.delta` |
| `response.output_audio.done` / `response.output_text.done` | `response.audio.done` / `response.output_text.done` |
| `response.output_audio.delta` (with `audio_transcript`) | optional `response.speak`, then `response.output_audio.delta` and `response.output_audio_transcript.delta` |
| `response.output_audio.done` / `response.output_text.done` | `response.output_audio.done` / `response.output_text.done` |
| `response.text.delta` | `response.output_text.delta` |
| `response.message` (chat-fallback raw chunk) | passed through |
| `audio.cancelled` | optional `output_audio_buffer.cleared`, then the cancelled terminal events and `response.done` with `status: "cancelled"` |
| `function_call.done` | `response.created`, `response.output_item.added` (`function_call` item), `response.function_call_arguments.delta` / `.done`, `response.output_item.done`, `response.done` |
| `playback.acknowledged` | `playback.acknowledged` (wrapped verbatim under `event`) |
| `runtime.control`, `session.resumed`, `session.replaced`, `session.expired`, `session.resync_required`, `session.heartbeat_ack` | passed through unchanged |
| `session.resumed`, `session.replaced`, `session.expired`, `session.resync_required`, `session.heartbeat_ack` | passed through unchanged |
## Known Limitations
- Only MiniCPM-o 4.5 is served over this endpoint today; PersonaPlex and
Nemotron VoiceChat arrive with the follow-up PRs that port them to the
plugin contract.
- Several surfaces are capability-gated per model (see *Capability
negotiation by model* above): PersonaPlex does not support session resume,
barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
or audio truncation; camera frames are consumed only by MiniCPM-o 4.5; tool
calls are produced only by Nemotron VoiceChat.
- `turn_detection` supports `server_vad` or `null`, not `semantic_vad`.
Turn-based models support endpointing without interruption, including
`create_response=false` for explicit response creation. Model-native duplex
requires `interrupt_response=true` and `create_response=true`.
- A response is generated from committed **audio**, so the OpenAI text-prompt
shape does not drive one: `conversation.item.create` with an `input_text`
part adds the item to history, but a following `response.create` with no
committed audio is rejected with `response_create_without_input`, and
`input.text.append` is rejected with `native_text_append_unsupported`.
Text-to-speech has its own shape here — put the text in
`extra_body.duplex_initial_user_text` on `session.update`, then stream audio
units (silence is enough). The seeded turn is what the model answers.
- `turn_detection` supports only `server_vad` with `interrupt_response=true`;
`semantic_vad`, `interrupt_response=false`, and `create_response` are not
supported.
- `input_audio_transcription` and `input_audio_noise_reduction` are accepted
and echoed but no separate transcription or noise-reduction stage runs;
transcripts come from the model.
- `rate_limits.updated` is emitted for compatibility only and always carries
an empty list.
- Session capacity is bounded by `duplex_session.max_sessions` in the deploy
configuration; admission beyond it fails with `duplex_session_busy` or
`resource_exhausted`.
- The native `/v1/duplex` route speaks the internal dialect and does not
support `session.resume`; use `/v1/realtime?duplex=1` for the contract on
this page.
configuration; admission beyond it fails with `resource_exhausted`
(`rate_limit_error`, retryable).
- A duplex deployment serves the websocket route and, when supported by the
model, ordinary `/v1/chat/completions` requests on the same engine without a
session per request (see [Start the Server](#start-the-server)). Other
turn-based HTTP routes report "not available" in duplex mode. To use the
ordinary online serving stack, select `session_mode: turn` as described in
[Full Duplex](full_duplex_api.md#enable-full-duplex).
+97 -3
View File
@@ -6,8 +6,9 @@ vLLM-Omni provides an OpenAI-compatible API for text-to-speech (TTS) generation.
- **Fish Speech S2 Pro** (`fishaudio/s2-pro`) -- Dual-AR TTS with DAC codec. Supports text-to-speech and voice cloning via reference audio. Output: 44.1 kHz.
- **Voxtral TTS** (`mistralai/Voxtral-4B-TTS-2603`) -- AR + FlowMatching TTS with preset voices. Output: 24 kHz.
- **CosyVoice3** (`FunAudioLLM/Fun-CosyVoice3-0.5B-2512`) -- 2-stage talker + flow-matching code2wav. Voice cloning via `ref_audio` + `ref_text` (no presets). Output: 24 kHz.
- **Gepard-1.0** (`nineninesix/gepard-1.0`) -- single-stage native-AR TTS with a 22.05 kHz NanoCodec. Zero-shot default voice only.
See the [Supported Models](#supported-models) section below for the full list, including OmniVoice, VoxCPM2, and MOSS-TTS-Nano.
See the [Supported Models](#supported-models) section below for the full list, including OmniVoice, VoxCPM2, MOSS-TTS-Nano, and Breeze-TTS-2.
!!! tip "Deployment recipes"
TTS deployment recipes are published at
@@ -40,11 +41,16 @@ vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091
# CosyVoice3 (voice cloning only — supply ref_audio + ref_text per request)
vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
--omni --port 8091 --trust-remote-code
# Gepard-1.0 (zero-shot default voice; packaged gepard.yaml)
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
--stage-init-timeout 900 \
--deploy-config vllm_omni/deploy/gepard.yaml
```
### Generate Speech
**Using curl:**
**Qwen3-TTS CustomVoice, using curl:**
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
@@ -56,7 +62,24 @@ curl -X POST http://localhost:8091/v1/audio/speech \
}' --output output.wav
```
**Using Python:**
**Gepard-1.0, using curl:**
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`,
`stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields
include `speed`, `language`, `instructions`, `task_type`, `ref_audio`,
`ref_text`, `extra_params`, and `word_timestamps`.
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, this is Gepard speaking.",
"voice": "default",
"seed": 7
}' --output output.wav
```
**Qwen3-TTS CustomVoice, using Python:**
```python
import httpx
@@ -498,6 +521,15 @@ are cached in-process with a shared LRU so repeated requests with the same
all TTS model types; deleting a voice invalidates every model-type slot at
once.
Decoded reference waveforms use a separate LRU cache of owned, contiguous
float32 arrays at the source sampling rate. Its byte budget counts numeric
buffers (four bytes per mono sample), excluding cache metadata. Numeric storage
avoids retaining a Python float object for every sample during garbage
collection. MOSS reference encoding consumes these arrays directly; other
list-based interfaces, including MOSS Nano, receive temporary lists that are
not retained in the resolve cache. The MOSS reference encoder also releases
completed batch inputs before waiting for more work.
### Precomputed Custom Voices
Qwen3-TTS Base and VoxCPM2 can load offline-precomputed voices at startup.
@@ -761,6 +793,28 @@ Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` ne
| ------- | ------------- |
| `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | Voice cloning from `ref_audio` + `ref_text`. No built-in voice presets — upload a voice or pass `ref_audio`/`ref_text` per request. |
### Gepard-1.0
| Model | Description |
| ----- | ----------- |
| `nineninesix/gepard-1.0` | Zero-shot native-AR TTS. 22.05 kHz mono. `voice` must be omitted or `"default"`. |
Gepard request fields:
| Field | Behavior |
| ----- | -------- |
| `input` | Required. Empty/whitespace-only returns 400. |
| `voice` | Omitted or `"default"` only. Other values 400. |
| `response_format` | `wav` default. Non-streaming: `wav`/`pcm`/`flac`/`mp3`. `opus` returns 400 (22.05 kHz is not an Opus sample rate). Streaming: `pcm`/`wav` only. |
| `stream` / `stream_format` | SSE (`speech.audio.*`) and raw `audio` byte streaming. |
| `speed` | Must be `1.0`. Gepard has no native speed control. |
| `max_new_tokens` | Frame budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms). Default 1000 from deploy YAML; adapter bounds 1..4096. |
| `seed` | Optional. Reaches the in-model 32-head sampler. The packaged YAML currently pins `seed: 42`, so serving is deterministic by default until that pin is removed. |
| `extra_params` | Any key returns 400, including `temperature`/`top_p`/`top_k`. |
| Cloning / style fields | `ref_audio`, `ref_text`, `speaker_embedding`, `instructions`, `language`, `task_type`, `word_timestamps`, and similar declared-but-unsupported fields return 400. |
See the [Gepard section of the online TTS hub](../user_guide/examples/online_serving/text_to_speech.md#gepard-10) for launch commands. Native-AR recompute preemption is a known limitation under concurrency.
### OmniVoice
| Model | Description |
@@ -777,12 +831,52 @@ hidden-state dtype before the output projection. Consequently, a float32 stage c
| ------- | ------------- |
| `openbmb/VoxCPM2` | TTS + voice cloning with built-in speaker presets and uploaded-voice support. Accepts `voice` (preset or uploaded) or `ref_audio` + optional `ref_text`. |
#### Startup LoRA adapter
To serve a fine-tuned VoxCPM2 voice, set
`voxcpm2_runtime_config.startup_lora_path` in the stage's `hf_overrides`.
Copy `vllm_omni/deploy/voxcpm2.yaml` to a local deploy config and add this key
alongside its existing runtime settings:
```yaml
# Under stages[0].engine_extras.hf_overrides.voxcpm2_runtime_config:
startup_lora_path: /absolute/path/to/adapter
```
Then launch with the modified config:
```bash
vllm serve openbmb/VoxCPM2 --omni --deploy-config /absolute/path/to/voxcpm2-lora.yaml
```
The directory must be accessible to the worker and contain the native VoxCPM2
training export: `lora_config.json` (with a `lora_config` object containing
`r`, `alpha`, enabled groups, and target module names) and
`lora_weights.safetensors`. PEFT checkpoints and pickle checkpoints are not
accepted. Missing, unexpected, non-finite, or incorrectly shaped adapter
tensors fail model loading rather than silently loading a partial adapter.
The adapter is merged into the base LM, residual LM, LocDiT, and optional
projection layers selected by its configuration, after base weight loading
and before compilation or CUDA graph capture. All requests use that adapter;
`voice` still selects a reference voice, not a LoRA adapter. Changing adapters
requires restarting the server. Runtime loading/unloading, per-request
multi-LoRA selection, and `load_format=dummy` are not supported by this path.
Weight fusion rounds to the base model's dtype, so numerical and speech-quality
parity should be checked against the upstream adapter on your deployment.
### MOSS-TTS-Nano
| Model | Description |
| ------- | ------------- |
| `OpenMOSS-Team/MOSS-TTS-Nano` | Voice cloning only. Requires `ref_audio` (or an uploaded `voice`); no built-in voice presets. `ref_text` is accepted but ignored — upstream's `voice_clone` mode does not consume a transcript. |
### Breeze-TTS-2
| Model | Description |
| ------- | ------------- |
| `BreezeBlue/Breeze-TTS-2` | Two-stage AR TTS (T5Gemma2 + Qwen3 talker with a depth decoder, bundled Qwen3-TTS codec) at 24 kHz. Four modes are selected from the request fields: plain (`input` + speaker tag `voice`, `S0`..`S9`), voice design (`instructions`), voice clone (`ref_audio` + `ref_text`, exactly one clip), and voice direction (reference + `instructions`). Greedy decoding only: `sample_rate` must be `24000`, `speed` must be `1.0`, and `guidance_scale`/`cfg_scale` other than `1.0`, `negative_prompt`, `temperature`/`top_p`/`top_k` overrides, `language`, and `speaker_embedding` are rejected. Streaming returns PCM `speech.audio.delta` events. See [`recipes/BreezeBlue/Breeze-TTS-2.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md). |
## Error Responses
### 400 Bad Request
+32 -3
View File
@@ -41,7 +41,7 @@ bytes.
| Server to client | `video.start` | Confirms the request ID, format, and accepted configuration |
| Server to client | `video.chunk_metadata` | Describes the next binary media frame |
| Server to client | Binary frame | Carries one fragmented MP4 chunk |
| Client to server | `session.interaction` | Queues an optional prompt update during generation |
| Client to server | `session.interaction` | Queues an optional interaction (e.g., prompt. camera action) during generation |
| Server to client | `session.interaction.queued` | Confirms that an interaction was queued |
| Client to server | `session.ping` | Refreshes the stall clock |
| Server to client | `session.pong` | Acknowledges a ping |
@@ -83,10 +83,39 @@ Models that implement interaction updates can change the active prompt:
}
```
Models that register the camera modality (for example LingBot-World) also
accept structural SE3 payloads on `event.multi_modal_data.camera`. Coordinates
follow Unity semantics (`+X` right, `+Y` up, `+Z` forward; same as LingBot's
native camera frame). WASD key tokens are not admitted by the engine:
```json
{
"type": "session.interaction",
"interaction": {
"event_id": "forward-1",
"event": {
"multi_modal_data": {
"camera": {
"mode": "velocity",
"data": {
"translation": [0.0, 0.0, 0.05],
"rotation": [0.0, 0.0, 0.0, 1.0]
}
}
}
}
}
}
```
`mode` is `target` (lerp to an absolute pose over `transition_chunks`) or
`velocity` (hold a per-frame SE3 delta until replaced). Prompt and camera may
be combined in one interaction event.
This feature is pipeline-dependent. An enabled endpoint does not imply that
the loaded model supports midway prompt updates.
the loaded model supports midway prompt or camera updates.
The default start timeout is 10 seconds and the server reports a stall after
about 60 seconds without generation progress or a `session.ping`. See the
[streaming video generation example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation)
for prompt-update scheduling and browser playback.
for prompt/camera-update scheduling and browser playback.
+25 -3
View File
@@ -45,11 +45,19 @@ WebSocket /v1/video/chat/stream
| Server -> Client | `response.start` | none | Query generation started. |
| Server -> Client | `response.text.delta` | `delta` | Incremental text output. |
| Server -> Client | `response.text.done` | `text` | Final text output for the query. |
| Server -> Client | `response.audio.delta` | `data`, `format` | Incremental generated audio, base64 WAV. |
| Server -> Client | `response.audio.done` | none | Audio output finished. |
| Server -> Client | `response.output_audio.delta` | `data`, `format` | Incremental generated audio, base64 WAV. |
| Server -> Client | `response.output_audio.done` | none | Audio output finished. |
| Server -> Client | `session.done` | none | Session closed. |
| Server -> Client | `error` | `message` | Recoverable protocol or generation error. |
### Frame Consumption Reporting
When buffered frames include client-supplied `frame_id` values, the server emits `video.frames.consumed` after the first engine output. Its `frame_ids`, `frames`, and `latest_pts_ms` describe the images included in that query's prompt.
Queries stride-sample the buffered frames, keeping the last frame, then exclude frames already known to have failed decoding. Excluded frames are not replaced with other buffered frames. Frames whose background decoding has not finished remain eligible through the image URL path.
**Bugfix compatibility note:** `video.frames.consumed` now excludes known decode failures that were already excluded from the prompt. Older versions could report those frames and their timestamps as consumed. The event name and fields are unchanged; an empty selection reports empty lists and `latest_pts_ms: null`.
### `session.config` Fields
| Field | Type | Default | Description |
@@ -60,10 +68,24 @@ WebSocket /v1/video/chat/stream
| `max_frames` | integer, 1-256 | `50` | Maximum retained frame buffer size. Oldest frames are evicted first. |
| `system_prompt` | string or null | null | Optional custom system prompt. |
| `use_audio_in_video` | bool | `true` | Include streamed audio chunks in multimodal video understanding when audio is present. |
| `sampling_params_list` | list or null | null | Optional per-stage sampling parameter overrides. |
| `sampling_params_list` | list or null | null | Optional per-stage parameter dictionaries. Each provided entry replaces that stage's deployment sampling settings. |
| `enable_frame_filter` | bool | `true` | Enable EVS near-duplicate frame filtering. |
| `frame_filter_threshold` | float, 0.0-1.0 | `0.95` | EVS similarity threshold. Higher keeps more frames; lower drops more near-duplicates. |
Sampling parameter bugfix: `sampling_params_list` is now forwarded to the engine;
earlier versions accepted this field but silently used deployment defaults instead.
Each provided entry constructs a fresh `SamplingParams` for that stage in pipeline
order; it is not merged with that stage's deployment defaults. Fields absent from a
provided entry use `SamplingParams` constructor defaults. Only omitted trailing
stages keep their deployment defaults. An omitted, `null`, or empty list keeps the
engine defaults. Invalid sampling parameters return an `error` when the query is
submitted.
For example, `[{"temperature": 0.2, "max_tokens": 64}]` configures the thinker while
the talker and code2wav retain their deployment defaults. With only
`[{"temperature": 0.2}]`, the thinker's `max_tokens` and `top_p` use constructor
defaults, not its YAML values.
### Legacy Aliases
The server accepts these legacy field names and rewrites them before validation. New clients should send the canonical names above.
+99
View File
@@ -73,6 +73,10 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
| `image_reference` | string | null | JSON-encoded reference image payload; do not combine with `input_reference` or `video_reference` |
| `video_reference` | string | null | JSON-encoded reference video payload; do not combine with `input_reference` or `image_reference` |
| `audio_reference` | string | null | JSON-encoded audio reference for speech-to-video: `{"audio_url": "..."}` — supports HTTP(s) URLs or base64 data URLs |
| `source_video` | file | null | MiniMax H3 latent-edit source video (`.mp4` or `.mov`, up to 512 MiB) |
| `source_audio` | file | null | Optional MiniMax H3 latent-edit source audio (`.wav` or `.mp3`, up to 512 MiB) |
| `video_noise_mask` | file | null | UTF-8 JSON MiniMax H3 video mask; `0` preserves and `1` regenerates a token |
| `audio_noise_mask` | file | null | UTF-8 JSON MiniMax H3 audio mask; `0` preserves and `1` regenerates a token |
| `width` | integer | model default | Output video width |
| `height` | integer | model default | Output video height |
| `num_frames` | integer | 1 | Number of generated frames |
@@ -109,6 +113,16 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
The final content is available from `/v1/videos/{video_id}/content` after the
job status becomes `completed`.
`queued` means the request is still waiting for diffusion scheduler admission.
The status changes to `in_progress` when the scheduler first selects the
request for execution.
`DELETE /v1/videos/{video_id}` issues a bounded engine abort
(`VLLM_OMNI_ABORT_TIMEOUT`, default 2s), then cancels the frontend
task. Cancellation cleanup is also bounded and best-effort: it confirms
the abort was queued, and the current request batch may still drain.
The job is then re-read so a completed save is not orphaned.
### Synchronous Response
`POST /v1/videos/sync` blocks until generation finishes and returns raw video
@@ -204,6 +218,49 @@ upstream vLLM's media URL allowlist protection. Deployments should therefore
treat remote media URLs as untrusted and choose this setting as part of their
URL access policy.
### MiniMax H3 Latent-Mask Editing
MiniMax H3 accepts request-scoped source media and video/audio noise masks.
At least one mask is required. A nontrivial `video_noise_mask` requires
`source_video`, while a nontrivial `audio_noise_mask` requires either
`source_audio` or a `source_video` with an audio stream. Mask values are in
`[0, 1]`: `0` preserves the source, `1` regenerates it, and fractional values
blend the two behaviors. Exact all-one masks are no-ops and do not require a
source. Source uploads without a mask are rejected. Masks may be a JSON scalar
or arrays matching the H3 latent/token grid; pixel-resolution masks must be
resized or pooled by the client before upload.
For an aligned output of `F` frames at `W x H`, the video latent grid is
`[Tv, H/16, W/16]`, where `Tv = 2 + 5 * ((F - 5) / 17)`. The video mask may be
a scalar, a flat token vector, `[Tv, H/32, W/32]`, or the full latent grid. For
the model input, timestep, and velocity, a full-grid mask is max-pooled over
each 2x2 spatial token and fractional values are rounded upward to 1/256
levels. The final x0 restore uses the original, unquantized mask, so full-grid
masks retain cell-level preservation inside a model token. The audio length is
`Ta = round(F * 40 / 24)`, and its mask may be a scalar, `[Ta]`, `[2, Ta]`, or
a flat `2 * Ta` vector. If source audio is short, its missing latent tail is
forced to mask value `1` so H3 generates that portion. Each mask must be sent
as a UTF-8 JSON file part and is limited to 8 MiB. A file may contain either a
JSON scalar or an array.
```bash
curl -s http://localhost:8091/v1/videos/sync \
-F "prompt=Partially restyle the complete clip and soundtrack" \
-F 'extra_params={"task":"t2va","duration":4.0,"aspect_ratio":"16:9"}' \
-F "source_video=@source.mp4;type=video/mp4" \
-F "source_audio=@source.wav;type=audio/wav" \
-F "video_noise_mask=@video-mask.json;type=application/json" \
-F "audio_noise_mask=@audio-mask.json;type=application/json" \
-o edited.mp4
```
For example, each mask file in the request above may contain the JSON scalar
`0.5`.
The server streams source files to temporary request-scoped storage and removes
them after synchronous or asynchronous generation finishes. Models that do not
declare latent-mask editing support reject these fields.
### Speech-to-Video
For models that support audio-driven generation (e.g., Wan2.2-S2V), pass both
@@ -234,6 +291,48 @@ curl -X POST http://localhost:8091/v1/videos/sync \
-o output.mp4
```
## Output Encoding
These `extra_params` control how the server turns decoded frames into MP4 bytes.
| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `preencode_mp4` | boolean | false | Encode the MP4 on the worker while the VAE is still decoding, instead of after the full video is materialized |
| `preencode_batch_frames` | positive integer | 17 (H3, Wan T2V/I2V); 1 (Wan S2V) | Minimum accumulated frames per worker transfer/encoding batch; used only with `preencode_mp4=true` |
| `video_codec_options` | object | null | Encoder options passed through to the H.264 encoder, such as `{"preset": "ultrafast", "threads": "0"}` |
With `preencode_mp4` enabled, each committed VAE chunk leaves the accelerator and
is encoded while later chunks are still decoding, so host transfer and CPU
encoding overlap the remaining decode instead of following it. The response is
unchanged: the same complete MP4, byte-for-byte equivalent frames.
```bash
curl -X POST http://localhost:8091/v1/videos/sync \
-F "prompt=A small robot walking through a neon city" \
-F 'extra_params={"preencode_mp4": true, "preencode_batch_frames": 33, "video_codec_options": {"preset": "ultrafast"}}' \
-o output.mp4
```
Set `preencode_batch_frames` in `extra_params` (or `extra_args` for offline
sampling) to tune batching. The worker accumulates complete VAE chunks until
it has at least this many frames, then transfers and encodes them together.
It always flushes the final partial batch. This is a threshold, not an exact
chunk length: a value of 1 submits every native chunk immediately, and a value
smaller than a native chunk does not split it. Larger values reduce transfers
but retain more frames on the accelerator and delay encoding. The VAE decode
window and output frame count stay unchanged. Wan S2V keeps its existing
per-clip behavior by default. Zero, negative, fractional, boolean, string, and
null values are rejected when pre-encoding is enabled.
`preencode_mp4` applies to the complete-MP4 response paths only. The
`/v1/realtime/video` WebSocket endpoint rejects it, because that path already
overlaps encoding through its own incremental fragmented-MP4 encoder. Wan also
rejects it together with `enable_frame_interpolation`, which needs the decoded
frames the pre-encoded path no longer materializes.
Support is per model: MiniMax-H3 and Wan 2.2 (T2V, I2V, and S2V) implement it,
and other models ignore the flag and take the full-decode path.
## Storage
Set `VLLM_OMNI_SERVER_STORAGE__PATH` to control where asynchronous video outputs are
+17 -8
View File
@@ -12,7 +12,7 @@ curl http://localhost:8000/metrics
## Metric Namespaces
| Prefix | Source | Present when |
|--------|--------|--------------|
| -------- | -------- | -------------- |
| `vllm_omni:` | vLLM-Omni orchestrator / audio modality / cross-stage transfer | Pipeline-dependent |
| `vllm:` | Upstream vLLM engine, wrapped by `OmniPrometheusStatLogger` to expose `{stage, replica}` | Pipeline includes an LLM (AR) stage |
| `http_` / `process_` | Uvicorn / Python runtime | Always |
@@ -24,7 +24,7 @@ Defined in `vllm_omni/metrics/prometheus.py`. Track request lifecycle across the
### Request counts
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| -------- | ------ | -------- | ------------- |
| `vllm_omni:num_requests_running` | Gauge | `model_name` | Pipeline-global in-flight requests (dispatched to engine, not yet finalized) |
| `vllm_omni:num_requests_waiting` | Gauge | `model_name` | Requests waiting in the Orchestrator queue |
| `vllm_omni:requests_success_total` | Counter | `model_name`, `finished_reason` | Total requests by completion reason. `finished_reason` ∈ {`stop`, `length`, `abort`, ...} mirroring upstream `vllm:request_success_total`; aborts cover client disconnect / cancellation paths in addition to upstream `FinishReason.ABORT` |
@@ -32,15 +32,15 @@ Defined in `vllm_omni/metrics/prometheus.py`. Track request lifecycle across the
### Latency
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| -------- | ------ | -------- | ------------- |
| `vllm_omni:e2e_request_latency_s` | Histogram | `model_name` | Pipeline-global end-to-end request latency in seconds |
## Audio Modality Metrics (`vllm_omni:`)
Emitted at request finalize, except for `audio_ttfp_s` (streaming-hook at the first audio packet) and `audio_underrun_s` / `audio_continuity_ok_total` (streaming finalize, after the chunk stream is exhausted). All carry `{model_name, stage, replica}` plus the listed extra label.
Emitted at request finalize, except for `audio_ttfp_s` (the first audio packet/frame from streaming Chat or Speech APIs) and `audio_underrun_s` / `audio_continuity_ok_total` (Chat or Speech streaming finalize, after the chunk stream is exhausted). Non-streaming Speech latency is represented by `e2e_request_latency_s`, not TTFP. All audio metrics carry `{model_name, stage, replica}` plus the listed extra label.
| Metric | Type | Extra label | Description |
|--------|------|-------------|-------------|
| -------- | ------ | ------------- | ------------- |
| `vllm_omni:audio_ttfp_s` | Histogram | — | Time from request arrival to first audio packet/frame |
| `vllm_omni:audio_duration_s` | Histogram | — | Audio content duration (`audio_frames / sample_rate`) |
| `vllm_omni:audio_rtf` | Histogram | — | Real-time factor (`stage_gen_time_s / audio_duration_s`); streaming TTS SLO red line `< 1`; uses `RTF_BUCKETS` |
@@ -50,13 +50,22 @@ Emitted at request finalize, except for `audio_ttfp_s` (streaming-hook at the fi
| `vllm_omni:audio_skipped_requests_total` | Counter | `reason` | Silent-loss counter — code2wav rejected malformed codec input and returned `200 OK` with empty audio |
The continuity math comes from `vllm_omni/benchmarks/audio_continuity.py::compute_continuity_stats` so the server-side observation aligns with the bench-side definition.
For Speech, continuity applies to raw audio, SSE, and WebSocket streaming. WAV headers and empty chunks are excluded; non-streaming Speech requests do not emit continuity samples.
Continuity rate for the default 100 ms threshold can be derived from the successful-continuity counter and the underrun Histogram's request count:
```promql
sum by (model_name, stage, replica) (rate(vllm_omni:audio_continuity_ok_total{threshold_ms="100"}[5m]))
/
sum by (model_name, stage, replica) (rate(vllm_omni:audio_underrun_s_count[5m]))
```
## Diffusion Metrics (`vllm_omni:`)
Per-request timing breakdowns for diffusion (image/video) stages. Emitted at request finalize when `stage_metrics.diffusion_metrics` is present. All carry `{model_name, stage, replica}`.
| Metric | Type | Description |
|--------|------|-------------|
| -------- | ------ | ------------- |
| `vllm_omni:diffusion_exec_s` | Histogram | DiT forward pass execution time per request in seconds |
| `vllm_omni:diffusion_exec_per_step_s` | Histogram | DiT forward pass execution time per denoising step in seconds (`exec_s / num_inference_steps`) |
| `vllm_omni:diffusion_preprocess_s` | Histogram | Diffusion input preprocessing time per request in seconds |
@@ -80,7 +89,7 @@ rate(vllm_omni:diffusion_preprocess_s_sum[5m])
Per-physical-transfer histograms tracking the data hop between adjacent stages. Labels `{model_name, from_stage, from_replica, to_stage, to_replica}` let dashboards attribute latency to specific replica edges. `from_replica` / `to_replica` are resolved from the orchestrator's sticky-routing binding (`stage_pool.get_bound_replica_id(request_id)`), so no extra plumbing through `TransferEdgeStats` is needed.
| Metric | Type | Description |
|--------|------|-------------|
| -------- | ------ | ------------- |
| `vllm_omni:transfer_size_bytes` | Histogram | Per-transfer payload size in bytes |
| `vllm_omni:transfer_tx_s` | Histogram | Sender-side time (serialize + submit to connector) |
| `vllm_omni:transfer_rx_s` | Histogram | Receiver-side time (recv + deserialize) |
@@ -106,7 +115,7 @@ For the full list of upstream metrics, see [the vLLM docs](https://github.com/vl
## Metric Availability by Pipeline Type
| Metric group | Multi-stage LLM (Qwen3-Omni) |
|---|---|
| --- | --- |
| `vllm_omni:` request tracking + latency | With `--log-stats` |
| `vllm_omni:` audio modality | With `--log-stats`, if pipeline has a talker stage |
| `vllm_omni:` transfer | With `--log-stats`, if pipeline has ≥ 2 stages |
@@ -1,24 +1,33 @@
# FastVideo VSA
FastVideo Variable Sparse Attention (VSA) accelerates the `FastVideo/FastWan2.2-TI2V-5B-Diffusers` model's
self-attention by partitioning the post-patch latent grid into spatiotemporal
FastVideo Variable Sparse Attention (VSA) partitions the post-patch latent grid into spatiotemporal
blocks. For every query block, VSA scores the key/value blocks and computes
attention only against the selected top-k blocks.
The supported checkpoint provides both text-to-video and image-to-video modes through `Wan22Pipeline`; the separate Wan I2V-14B, S2V, and VACE pipelines are outside this backend's supported scope.
## Supported models
MiniMax-H3 served with a FastH3 VSA adapter uses a second, H3-specific route
through the same backend. See the
[MiniMax-H3 recipe](https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3) for its geometry,
supported topologies, and commands.
| Model / checkpoint | Required adapter | Tasks | Sequence parallelism |
| --- | --- | --- | --- |
| `FastVideo/FastWan2.2-TI2V-5B-Diffusers` | None | T2V, I2V through `Wan22Pipeline` | Disabled |
| `MiniMaxAI/MiniMax-H3` | [FastH3 VSA (recipe)](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter) | T2VA | Disabled or pure Ulysses |
| Wan I2V-14B, S2V, VACE | — | Unsupported | — |
VSA is a CUDA-only, explicitly selected backend. It requires the
`fastvideo-kernel` package and currently supports non-causal self-attention
with equal query and key/value sequence lengths. Unsupported shapes, masks,
dtypes, sequence-parallel execution, or kernel failures fall back to
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
sequence-parallel context is one of those fallbacks; the H3 route supports pure
Ulysses and rejects ring or all-gather sequence parallelism at startup.
## Installation
Install vLLM-Omni from this checkout with the `vsa` extra:
```bash
uv pip install -e '.[vsa]'
```
The extra installs the tested kernel dependency automatically. Prebuilt kernels require
Linux, Python 3.12, and glibc 2.34 or newer (x86-64 or aarch64). The full
FastVideo framework and provider environment variables are not required.
`FASTVIDEO_VSA` selects the attention algorithm. On SM120, the H3 integration
currently executes FastVideo's 64-token Triton block-sparse kernel; it does not
dispatch to FlashInfer. The `vsa` extra installs `fastvideo-kernel==0.3.4` for
this execution path.
## Enable the backend
@@ -27,24 +36,18 @@ flag. Use `--fastvideo-vsa-topk` to set the number of key/value blocks retained
for every query block:
```bash
vllm-omni serve <model> \
vllm serve <model> --omni \
--diffusion-attention-backend FASTVIDEO_VSA \
--fastvideo-vsa-topk 64
```
The backwards-compatible environment variable selects the backend with the
default `topk=64`:
<details markdown="1">
<summary>Alternative configuration formats</summary>
Equivalent structured configuration:
```bash
export DIFFUSION_ATTENTION_BACKEND=FASTVIDEO_VSA
vllm-omni serve <model>
```
To tune top-k, pass the CLI backend and top-k flags together as shown above,
or use the equivalent structured configuration:
```bash
vllm-omni serve <model> \
vllm serve <model> --omni \
--diffusion-attention-config \
'{"default":{"backend":"FASTVIDEO_VSA","fastvideo_vsa_topk":64}}'
```
@@ -73,8 +76,15 @@ stages:
fastvideo_vsa_topk: 64
```
</details>
## Choose top-k
H3 uses 64-token video blocks and keeps
all prefix blocks; see the [FastH3 VSA recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
### Wan top-k behavior
At runtime the backend logs the sequence shape and derived block count:
```text
@@ -117,8 +127,16 @@ selecting VSA does not turn a native checkpoint into a distilled model.
## Verify routing and fallback
Check the startup and first-forward logs instead of assuming that selecting
the backend guarantees sparse execution:
VSA is a CUDA-only, explicitly selected backend. It requires the
`fastvideo-kernel` package and currently supports non-causal self-attention
with equal query and key/value sequence lengths. Unsupported shapes, masks,
dtypes, sequence-parallel execution, or kernel failures fall back to
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
sequence-parallel context is one of those fallbacks; the H3 route supports pure
Ulysses and rejects ring or all-gather sequence parallelism at startup.
H3 accelerator faults propagate instead of attempting dense recovery.
On the Wan route, check the startup and first-forward logs:
- `route=VSA` means top-k block selection is active.
- `route=VSA_ALL_BLOCKS` means the FastVideo DMD checkpoint retained all
@@ -126,6 +144,10 @@ the backend guarantees sparse execution:
- `route=SDPA` or `FASTVIDEO_VSA falling back to SDPA: ...` means dense SDPA
executed; the warning includes the reason.
For H3, check `FastH3 adapter active` at startup and
`FASTVIDEO_VSA H3 routing` during DiT execution, as described in the
[model recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
The Wan route requires CUDA tensors in FP16 or BF16, 256-token blocks,
standard `head_size**-0.5` scaling, equal Q/K/V head counts, no attention mask,
and no active sequence-parallel context. The MiniMax-H3 route uses 64-token
@@ -78,3 +78,35 @@ Protected video tails require a compatible MindIE-SD release.
For common configuration and selector behavior, see the
[attention backend overview](../attention_backends.md) and the
[backend selection design](../../../design/feature/attention_backend_selection.md).
## Wan2.2 T2V quantization on Ascend
`RAINFUSION_ATTN` accepts `diffusion_kv_cache_dtype: fp8` or `mxfp4` for
Wan2.2 T2V A14B.
MindIE-SD must expose `sparse_attention` with `rf_v3` and an explicit `precision`
argument. Omni passes the selected precision directly to this API; the installed
native operators must support that precision. Unsupported inputs or an
incompatible Python API raise an actionable error; Omni does not change BSA
precision automatically. Update `diffusion_kv_cache_dtype` to a precision
supported by the installed MindIE-SD, or unset it. Native execution errors also
propagate without retry.
BSA FP8 and MXFP4 retain MindIE-SD's default rotation policy.
`diffusion_kv_cache_skip_layers`/`diffusion_kv_cache_skip_steps` explicitly
select floating-point BSA and keep attention sparse. The existing `block_sparse`
warmup/layer exclusions and short-sequence threshold instead select Dense
attention, using the configured Dense precision. Leave `block_sparse.precision`
at its default when using `diffusion_kv_cache_dtype` as the shared Dense/BSA
precision setting.
Configure BSA quantization in the model's existing deployment YAML with
`backend: RAINFUSION_ATTN` and stage-level `diffusion_kv_cache_dtype: fp8` or
`mxfp4`. The precision and skip selectors are siblings of
`diffusion_attention_config`; they are not per-role `quant` fields.
For 40 denoising steps, `diffusion_kv_cache_skip_steps: "0,1,38,39"` and
`diffusion_kv_cache_skip_layers: "0,39"` are recommended starting points. These
fixed step indices do not automatically follow a changed step count. Selected
forwards remain floating-point BSA; unsupported BSA precision is an error rather
than an automatic fallback. See
[quantized attention](../../quantization/quantized_kvcache.md#wan22-t2v-quantized-attention-on-ascend)
for the shared configuration structure and index semantics.
@@ -0,0 +1,72 @@
# Device-side video postprocessing
vLLM-Omni can convert decoded WAN2.2 video from normalized floating-point
`[B, C, T, H, W]` tensors to contiguous `uint8` `[B, T, H, W, C]` frames before
the worker-to-engine device-to-host transfer. This reduces the transported video
payload by four times when the original IPC path widens bfloat16 to float32.
The optimization is disabled by default. Enable it with a structured diffusion
configuration:
```python
from vllm_omni import Omni
engine = Omni(
model="Wan-AI/Wan2.2-TI2V-5B-Diffusers",
video_output_transport={"enable_device_postprocess": True},
)
```
For the server CLI, pass the same object as JSON:
```bash
vllm serve Wan-AI/Wan2.2-TI2V-5B-Diffusers --omni \
--video-output-transport '{"enable_device_postprocess": true}'
```
For a deploy configuration, place it on the diffusion stage:
```yaml
stages:
- stage_id: 0
video_output_transport:
enable_device_postprocess: true
```
## Runtime contract
WAN2.2 emits decoded video through the typed `DiffusionOutput.media` field. The
model runner validates and splits the batch into request-local tensors before
preparing each video for transport. The worker then performs the D2H copy and
shared-memory IPC using the prepared representation. The engine uses the generic
media finalizer instead of a model-specific postprocessor.
The runtime leaves the video in normalized floating-point form when:
- `enable_device_postprocess` is false;
- the requested output type is not `np`; or
- frame interpolation still requires the floating-point tensor.
These are policy fallbacks, not malformed-contract errors. If the device cannot
allocate the temporary conversion buffer, the runner logs a warning and prepares
the request-local float representation instead. Invalid tensor layout, encoding,
value range, lifecycle state, and non-memory runtime failures still fail before
IPC.
## Precision
Device preparation converts to float32 before denormalization and quantization.
For bfloat16 WAN output this is more precise than the historical host path,
which denormalizes in bfloat16. Relative to that path, values may differ by at
most one level out of 255. Float32 inputs remain byte-identical after
quantization.
## Scope
WAN2.2 is the reference migration for the first contract version. Other video
pipelines continue using their existing output paths until they can declare the
same typed media contract without bypassing model-specific float consumers such
as safety checks or audio/video packaging.
See [RFC #6541](https://github.com/vllm-project/vllm-omni/issues/6541) for the
contract, lifecycle, batching rules, and migration plan.
+2 -2
View File
@@ -28,7 +28,7 @@ and their optimal Triton tile configurations:
Config loading order (3-tier, see ops/mot_gemm.py):
1. $VLLM_TUNED_CONFIG_FOLDER/<filename> (env override)
2. This directory: vllm_omni/diffusion/layers/mot/configs/<filename>
2. This directory: vllm_omni/diffusion/models/bagel/mot/configs/<filename>
3. Conservative default config (compiles everywhere, sub-optimal perf)
If no config file matches the current device, a warning is printed with
@@ -39,7 +39,7 @@ To generate configs for your hardware:
python benchmarks/kernels/mot_linear_benchmarks.py \
--model ByteDance-Seed/BAGEL-7B-MoT \
--tp-size 1 --dtype w16a16 --tune \
--save-dir vllm_omni/diffusion/layers/mot/configs/
--save-dir vllm_omni/diffusion/models/bagel/mot/configs/
For multi-GPU tuning (uses Ray for parallel search):
@@ -353,9 +353,14 @@ modes can resolve the same `None` value to different schedules.
Cache-DiT on any selected component. Prompt-embedding cache is also
incompatible when the text encoder uses `allgather`. Rank-local cache hits
could otherwise make ranks enter different weight collectives.
- Per-tensor online FP8 linears use the ordinary loader and can run with either
DiT transfer path. With DiT `allgather`, every rank temporarily materializes
the complete FP8 model in host memory before DLO retains only its shard.
- Per-tensor online FP8, INT8, and MXFP8 linears use the ordinary loader and
can run with either DiT transfer path. With DiT `allgather`, every rank
temporarily materializes the complete quantized model in host memory before
DLO retains only its shard. Layers too wide for the NPU quantization kernel
(`npu_quant_matmul` rejects an output dimension past 65535) stay
unquantized and load straight into host memory under offload-after-quant —
their runtime layout is a plain contiguous bf16 weight, the same layout DLO
shards on the ordinary path, so they are allowed under `allgather` too.
Other online quantization methods require rank-local transfer for the
affected component until their runtime layouts are validated.
- Resident leading layers require DiT `rank-local` transfer and a model
@@ -19,6 +19,12 @@ CFG-Parallel accelerates diffusion models by distributing positive and negative
See supported models list in [Supported Models](../../diffusion_features.md#supported-models).
Distilled Cosmos3 checkpoints, including `nvidia/Cosmos3-Super-Image2Video-4Step`,
run without CFG and reject `--cfg-parallel-size` greater than 1 at startup.
Use `--ulysses-degree 2` for two-GPU inference. These checkpoints always use
`guidance_scale=1.0`; other explicitly requested values produce a warning, and
`negative_prompt` does not affect generation.
---
## Quick Start
@@ -34,5 +34,34 @@ that provide their own `setup_compile()` implementation manage their compilation
policy independently. Compilation is lazy, so backend or graph errors can first
surface on the initial request.
## Pinning one packed shape (MiniMax-H3)
Compiled regions are keyed by input shape. MiniMax-H3 packs a request into a
row count that follows the request -- a longer prompt, more frames, a larger
frame -- and rounds it up to the next 64-row boundary, so two requests can land
on two different packed lengths.
Under the default `diffusion_compile_dynamic: true` a new packed length is
absorbed by the dynamic shape, so it is not by itself a recompilation. It still
hands the backend a shape it has not autotuned, which is the risk this knob
exists to remove: a fixed-shape deployment (`--no-diffusion-compile-dynamic`)
recompiles the transformer blocks per shape, and a first-seen large shape can
push autotuning into its worst-case memory use.
A request pins the length with `extra_args["pad_seq_len"]`:
```python
sampling_params = SamplingParams(extra_args={"pad_seq_len": 54080})
```
The value must be a positive multiple of 64 and must cover the rows the request
actually uses; the packed sequence is then padded to it instead of to the next
64-row boundary. Pick a bucket that covers every request dimension the
deployment accepts -- prompt length, frame count, frame size and reference
blocks all change the used row count. The server logs the effective length as
`MiniMax H3 packed sequence: ... pad_seq_len=... used=... seq_len=...` whenever
a request pins it. The padding rows are masked out, so their cost is the
attention and feed-forward work on those rows.
Use `--enforce-eager` to disable the model runner's generic compile setup.
Pipelines that compile internally define their own eager-mode behavior.
@@ -1,35 +0,0 @@
# HunyuanImage-3.0 Image-to-Text Inference
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/hunyuan_image3>.
This example demonstrates how to run HunyuanImage-3.0 Image-to-Text with the vLLM-Omni.
## Local CLI Usage
Download the example image:
```bash
wget https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg
```
Run example:
```bash
python image_to_text.py \
--image cherry_blossom.jpg \
--prompt "<|startoftext|>You are an assistant that understands images and outputs text.<img>Describe the content of the picture."
```
Key arguments:
- `--model`: Model used. Default is: tencent/HunyuanImage-3.0-Instruct (Optional).
- `--image`: Path to input image (required).
- `--prompt`: Text description used to guide image understanding (required).
## Example materials
??? abstract "image_to_text.py"
``````py
--8<-- "examples/offline_inference/hunyuan_image3/image_to_text.py"
``````
@@ -2,7 +2,6 @@
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni>.
## 🛠️ Installation
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
@@ -15,6 +14,19 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
```
For a local deployment with multiple API frontend processes sharing one set
of stage engines, add `--api-server-count`:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--api-server-count 2 --port 8091
```
This mode supports local EngineCore stages with one local process group per
replica, including stages configured with multiple `num_replicas`. Headless,
remote, diffusion, intra-replica data-parallel, and Ray stage deployments
remain single-frontend.
The default deployment configuration situated at `vllm_omni/deploy/qwen3_omni_moe.yaml` is resolved and loaded
automatically via the model registry, obviating the necessity for the `--deploy-config` flag in standard deployment topologies.
The bundled Qwen3-Omni setup explicitly sets `moe_backend: triton` for the Thinker and Talker stages. This keeps
@@ -22,7 +34,12 @@ both stages on vLLM's Triton unquantized MoE path and avoids the unstable FlashI
some GPUs.
Asynchronous chunk streaming is **enabled by default** within the bundled configuration.
For Thinker pipeline parallelism, configure `pipeline_parallel_size` and `devices`
on the Thinker stage. Keep Talker and Code2Wav at `pipeline_parallel_size: 1`. The
default asynchronous scheduling and audio chunk streaming can remain enabled.
To explicitly utilize a custom deployment YAML, specify the configuration path:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--deploy-config /path/to/deploy_config_file
@@ -46,7 +63,7 @@ Adopt the stage-based CLI architecture to independently instantiate execution pr
The example below pins Stage 0 to GPU 0 and Stage 1/2 to GPU 1 via
`CUDA_VISIBLE_DEVICES`.
**1. Stage 0 (Thinker + API server)**
#### 1. Stage 0 (Thinker + API server)
```bash
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
@@ -56,7 +73,7 @@ CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--omni-master-port 26000
```
**2. Stage 1 (Talker)**
#### 2. Stage 1 (Talker)
```bash
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
@@ -66,7 +83,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--omni-master-port 26000
```
**3. Stage 2 (Code2Wav)**
#### 3. Stage 2 (Code2Wav)
```bash
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
@@ -91,6 +108,9 @@ To experiment with another supported MoE backend, set `moe_backend` in the
deployment YAML for both the Thinker and Talker stages. For the stage-based
CLI, pass `--moe-backend` directly on each stage command.
For other stage settings, pass the option on that stage command. For example,
to change its GPU memory utilization:
```bash
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--stage-id 1 \
@@ -103,11 +123,12 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
### Send Multi-modal Request
Get into the example folder
```bash
cd examples/online_serving/qwen3_omni
```
#### Send request via python
#### Send request via python
```bash
python openai_chat_completion_client_for_multimodal_generation.py --query-type use_image --port 8091 --host "localhost"
@@ -122,7 +143,6 @@ The Python client supports the following command-line arguments:
- `--audio-path` (or `-a`): Path to local audio file or URL. If not provided and query-type is `use_audio`, uses default audio URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common audio formats: MP3, WAV, OGG, FLAC, M4A. Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
- `--prompt` (or `-p`): Custom text prompt/question. If not provided, uses default prompt for the selected query type. Example: `--prompt "What are the main activities shown in this video?"`
For example, to use a local video file with custom prompt:
```bash
@@ -132,19 +152,20 @@ python openai_chat_completion_client_for_multimodal_generation.py \
--prompt "What are the main activities shown in this video?"
```
#### Send request via curl
#### Send request via curl
```bash
bash run_curl_multimodal_generation.sh use_image
```
## Modality control
You can control output modalities to specify which types of output the model should generate. This is useful when you only need text output and want to skip audio generation stages for better performance.
### Supported modalities
| Modalities | Output |
|------------|--------|
| ------------ | -------- |
| `["text"]` | Text only |
| `["audio"]` | Audio only |
| `["text", "audio"]` | Text + Audio |
@@ -222,7 +243,9 @@ print(response.choices[1].message.audio) # Audio response
```
## Streaming Output
If you want to enable streaming output, please set the argument as below. The final output will be obtained just after generated by corresponding stage. We support both text streaming output and audio streaming output. Other modalities can output normally.
```bash
python openai_chat_completion_client_for_multimodal_generation.py \
--query-type use_image \
@@ -255,12 +278,14 @@ The convenience script launches both the vLLM server and Gradio demo together:
```
This script will:
1. Start the vLLM server in the background
2. Wait for the server to be ready
3. Launch the Gradio demo
4. Handle cleanup when you press Ctrl+C
The script supports the following arguments:
- `--model`: Model name/path (default: Qwen/Qwen3-Omni-30B-A3B-Instruct)
- `--server-port`: Port for vLLM server (default: 8091)
- `--gradio-port`: Port for Gradio demo (default: 7861)
@@ -271,18 +296,19 @@ The script supports the following arguments:
#### Option 2: Manual Launch (Two-Step Process)
**Step 1: Launch the vLLM API server**
##### Step 1: Launch the vLLM API server
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
```
If you have a custom deploy config file:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --deploy-config /path/to/deploy_config_file
```
**Step 2: Run the Gradio demo**
##### Step 2: Run the Gradio demo
In a separate terminal:
@@ -295,7 +321,7 @@ Then open `http://localhost:7861/` on your local browser to interact with the we
The gradio script supports the following arguments:
- `--model`: Model name/path (should match the server model)
- `--api-base`: Base URL for the vLLM API server (default: http://localhost:8091/v1)
- `--api-base`: Base URL for the vLLM API server (default: <http://localhost:8091/v1>)
- `--ip`: Host/IP for Gradio server (default: 127.0.0.1)
- `--port`: Port for Gradio server (default: 7861)
- `--share`: Share the Gradio demo publicly (creates a public link)
@@ -18,6 +18,7 @@ For the full list of supported architectures across all modalities, see
| --- | --- | --- | --- | --- | --- |
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | — |
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
| Gepard-1.0 | `nineninesix/gepard-1.0` | — (zero-shot default voice) | ✓ (PCM / WAV stream) | `default` only | — |
| higgs-audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✓ (`ref_audio`+`ref_text`) | ✓ (codec_streaming) | — | — |
| GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
@@ -92,7 +93,7 @@ curl -X POST http://localhost:8091/v1/audio/speech \
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
```
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others).
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for Gepard, 24 kHz for the others).
For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md).
@@ -120,6 +121,15 @@ vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remot
Streaming is on by default via `async_chunk: true` in `vllm_omni/deploy/cosyvoice3.yaml`. Pass `--no-async-chunk` (or `NO_ASYNC_CHUNK=1 ./cosyvoice3/run_server.sh`) for the legacy synchronous path.
Cross-request Stage-1 flow batching is opt-in. Enable it in the server environment when concurrent requests should share a flow-estimator call:
```bash
export COSYVOICE3_BATCH_FLOW=1
vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remote-code
```
Batching preserves output lengths and streaming cache alignment, but the different GEMM shapes can produce small waveform differences compared with processing each request separately. Leave `COSYVOICE3_BATCH_FLOW` unset (or set it to `0`) when request-independent numerical behavior is required. Set `COSYVOICE3_BATCH_FLOW_DEBUG=1` to log the observed group-size distribution and enable detailed Stage-1 profiler scopes; diagnostics are disabled by default to avoid per-step profiling overhead.
### CLI client
The client defaults to the official upstream zero-shot prompt, so it runs without extra flags:
@@ -203,6 +213,61 @@ python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server
---
## Gepard-1.0
Single-stage native AR TTS at 22.05 kHz mono. Zero-shot only: omit `voice` or pass `"default"`. Voice cloning from reference audio is not available yet.
### Prerequisites
Same NeMo NanoCodec install as the [offline Gepard section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#gepard-10). On a host whose CUDA toolkit cannot build kernels — no `nvcc`/`ninja`, or a consumer Blackwell (`sm_120`) card — also `export VLLM_USE_FLASHINFER_SAMPLER=0` before launch. That env is not a deploy-YAML field.
### Launch
```bash
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
--stage-init-timeout 900 \
--deploy-config vllm_omni/deploy/gepard.yaml
# or:
./gepard/run_server.sh
```
`--stage-init-timeout 900` matches the online e2e fixture and the offline example. Serve defaults to 300s, which is often too short for a cold download of the talker plus NanoCodec.
The packaged `vllm_omni/deploy/gepard.yaml` must be passed with `--deploy-config`. The checkpoint self-identifies as `qwen3_5_text`, so omitting the YAML launches a diffusion fallback instead of the Gepard pipeline. The YAML sets `async_chunk: false`, `max_num_seqs: 4`, and currently pins `seed: 42`, so serving is deterministic by default until that YAML seed is removed. Pass an explicit per-request `seed` in tests and clients rather than depending on either default.
### Sending requests
```bash
python examples/online_serving/text_to_speech/gepard/speech_client.py \
--text "Hello, this is Gepard speaking."
python examples/online_serving/text_to_speech/gepard/speech_client.py \
--text "Hello, this is Gepard speaking." --seed 7 --stream --output output.pcm
```
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`, `stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields include `speed`, `language`, `instructions`, `task_type`, `ref_audio`, `ref_text`, `extra_params`, and `word_timestamps`.
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, this is Gepard speaking.",
"voice": "default",
"seed": 7
}' --output output.wav
```
### Notes
- Output: 22.05 kHz mono. `max_new_tokens` is a **frame** budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms at 21.5 fps; adapter bounds 1..4096).
- Supported request fields: `input` (required), `voice` (`default` only), `response_format` (`wav` default; `wav/pcm/flac/mp3` non-streaming; `opus` 400 because 22.05 kHz is not an Opus sample rate; streaming `pcm/wav` only), `stream` / `stream_format`, `max_new_tokens`, `seed`.
- Unsupported: `speed`, `extra_params` (including `temperature`/`top_p`/`top_k`), `ref_audio`, `ref_text`, `speaker_embedding`, `task_type`, `instructions`, `language`, and `word_timestamps`.
- Concurrent requests at `max_num_seqs: 4` are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below `max_num_seqs` and treat preemption as out of scope until the platform fix lands.
- Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.
- For the request-field table, see the [Speech API Gepard section](https://github.com/vllm-project/vllm-omni/blob/main/docs/serving/speech_api.md#gepard-10).
---
## higgs-audio v2
2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) that streams chunks back to the client.
@@ -348,6 +413,22 @@ vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
./qwen3_tts/run_server.sh Base
```
For a local deployment with multiple API frontend processes sharing one set
of TTS stage engines, add `--api-server-count`:
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--omni --api-server-count 2 --port 8091
```
This mode supports local EngineCore stages with one local process group per
replica, including stages configured with multiple `num_replicas`. Headless,
remote, intra-replica data-parallel, and Ray stage deployments are not
supported. Runtime voice upload and deletion are disabled with multiple
API frontends because their registries are process-local. Built-in voices,
inline `ref_audio`, and voices restored from `custom_voice_dir` at startup are
supported.
### Executor backend
Single-GPU serves now default to the uniproc executor (lower IPC overhead, the Base cloning use case from [#2603](https://github.com/vllm-project/vllm-omni/issues/2603) / [#2604](https://github.com/vllm-project/vllm-omni/pull/2604)). `vllm_omni/deploy/qwen3_tts.yaml` is the only Qwen3-TTS deploy config; pass `--deploy-config <path>` to override.
@@ -413,6 +494,10 @@ checkpoint. When a request names an uploaded voice, the server infers
`task_type="Base"`. Built-in presets such as `vivian` and `ryan` remain
CustomVoice speakers and require a CustomVoice checkpoint.
The runtime upload and delete routes require a single API frontend; with
`--api-server-count > 1`, use inline `ref_audio` or restore precomputed voices
from `custom_voice_dir` at startup.
### Precomputed custom voices
For reused Base voice-cloning speakers, precompute the reference artifacts once
+85
View File
@@ -204,6 +204,91 @@ omni = Omni(
| `--linear-backend cutlass` | str | auto | Select the validated CUTLASS linear backend for supported ModelOpt NVFP4 or mixed FP8/NVFP4 diffusion stages |
| `--moe-backend cutlass` | str | auto | Select the validated CUTLASS MoE backend for supported ModelOpt mixed MoE checkpoints |
## Cosmos3 Mixed-Precision Schedule
The experimental Cosmos3 schedule can use the checkpoint-native quantized GEMM for middle denoising
steps and dense 16-bit activation execution for selected first and last steps.
The schedule supports serialized ModelOpt FP8 and NVFP4 checkpoints:
| Detected weight format | Native middle steps | A16 boundary steps |
|------------------------|---------------------|--------------------|
| FP8 | W8A8 | W8A16 |
| NVFP4 | W4A4 | W4A16 |
The native and A16 paths share one quantized weight representation. The A16
path dequantizes the live backend weight and calls `F.linear`; it does not keep
a second checkpoint-weight snapshot. Scheduled FP8 currently requires
serialized tensorwise scales and a backend that retains canonical FP8 weights.
Scheduled NVFP4 requires a supported CUTLASS-compatible native or FlashInfer
layout. SmoothQuant, Marlin-repacked FP8, and per-channel/per-token FP8
schedules fail closed.
The reasoner uses dense A16 execution by default when its weights are FP8 or
NVFP4. Set the nested `reasoner` field to `native` to retain the checkpoint-native
W8A8/W4A4 path. A BF16 reasoner remains BF16 in either mode. The schedule
currently requires tensor parallel size 1 and one active request per worker,
does not support HSDP, and does not support block-scaled FP8. Its live-weight
design is compatible with model-level and standard layer-wise offload, but
those paths still require GPU end-to-end validation. Distributed layer-wise
offload is rejected because its direct loader bypasses ModelOpt post-load
transformations.
A checkpoint can opt in by adding a versioned policy to its authoritative
`transformer/config.json`. The same policy schema is used for FP8 and NVFP4:
```json
{
"quantization_config": {
"quant_method": "modelopt",
"quant_algo": "FP8",
"runtime": {
"diffusion_step_policy": {
"schema_version": 1,
"type": "first_last_n",
"index_space": "denoising_loop_iteration",
"scope": ["transformer"],
"default_mode": "native",
"first_steps": {"count": 3, "mode": "a16"},
"last_steps": {"count": 3, "mode": "a16"},
"overlap": "a16",
"reasoner": "a16"
}
}
}
}
```
The example preserves the checkpoint's existing `quant_method` and
`quant_algo`; only `runtime` is added.
When present and valid, the checkpoint policy is the default. An explicit
runtime object overrides it. To retain ordinary checkpoint-native execution
for every step, disable only the schedule:
```bash
vllm serve /path/to/Cosmos3-Nano-modelopt \
--omni \
--additional-config '{"cosmos3_mixed_precision":{"enabled":false}}'
```
This does not disable checkpoint quantization; it prevents installation of the
mixed-step runtime.
```bash
vllm serve /path/to/Cosmos3-Nano-modelopt \
--omni \
--additional-config \
'{"cosmos3_mixed_precision":{"first_steps":3,"last_steps":3,"reasoner":"a16"}}'
```
The presence of `cosmos3_mixed_precision` supplies an explicit runtime
override; an empty object uses the defaults shown above.
The schedule uses vLLM's selected native backend and fails during loading if
its live weight layout cannot support A16 dequantization.
The runtime selects FP8 or NVFP4 from the checkpoint's ModelOpt configuration.
BF16 linears are untouched. Mixed FP8/NVFP4 checkpoints are not supported by
this schedule.
## Validation and Notes
1. Compare the ModelOpt checkpoint against the BF16 baseline with the same
+310 -104
View File
@@ -6,22 +6,25 @@ W4A4 MXFP4 (Microscaling FP4) quantizes both weights and activations to FP4
(`float4_e2m1fn_x2`, packed 2 values per byte) using the OCP MX format: groups
of 32 K-dimension elements share a single `float8_e8m0fnu` exponent scale.
On Ascend, single-scale MXFP4 also supports Universal Optimal Scaling
(UOS, `Qmax=7.25`), described in the single-scale scaling section below.
vLLM-Omni provides two quantization methods with different scale structures:
| Method | Scale structure | Mode | Use case |
|--------|----------------|------|----------|
| `mxfp4` | Single-scale (per-32 fine only) | Online only | Quick accuracy baseline; no checkpoint prep needed |
| `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | Production; better accuracy; offline recommended |
| -------- | ---------------- | ------ | ---------- |
| `mxfp4` | Single-scale (per-32 fine only) | Online + pre-quantized single-scale weights | Online baseline, or an existing single-scale checkpoint in the layout below |
| `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | W4A4 with dual-level scales; offline mode loads calibrated Smooth tensors |
!!! tip "Recommended: `mxfp4_dualscale` offline"
For production deployments, use the `mxfp4_dualscale` offline mode with a
!!! tip "Existing W4A4-only DualScale deployments"
For W4A4-only deployments, the existing `mxfp4_dualscale` offline mode uses a
pre-quantized checkpoint produced by msModelSlim. Offline checkpoints load
calibrated `mul_scale` tensors from disk, providing measurably better accuracy
than any online method. The one-time preprocessing cost amortises across all
subsequent inference runs.
calibrated `mul_scale` tensors from disk. Quality depends on the model,
calibration recipe and BF16 fallback policy; evaluate the checkpoint against
your quality target before deployment.
Use `mxfp4` online only for quick experimentation where preprocessing time
is not acceptable and accuracy loss is tolerable.
W4A8 layer/step fallback requires single-scale `mxfp4`, online or offline.
DualScale supports W4A4 and BF16 routing only.
!!! warning "Online single-scale ≠ Offline dual-scale"
`mxfp4_dualscale` offline mode uses `NPUMxfp4DualScaleLinearMethod`:
@@ -34,61 +37,244 @@ vLLM-Omni provides two quantization methods with different scale structures:
## Hardware Support
| Device | Support |
|--------|---------|
| NVIDIA Blackwell GPU (SM 100+) | ⭕ |
| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ |
| NVIDIA Ampere GPU (SM 80+) | ⭕ |
| AMD ROCm (gfx950 / MI355X) | ✅ |
| Intel XPU | ⭕ |
| Ascend NPU (Atlas 950 A5) | ✅ |
The following matrix describes these vLLM-Omni implementations, rather than
hardware support for the MXFP4 format in general.
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
| Device | Single-scale online OCP | Single-scale offline | UOS / W4A8 fallback | DualScale |
| -------- | -------- | -------- | -------- | -------- |
| Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ✅ | ✅ |
| AMD ROCm (gfx950 / MI355X) | ✅ | ❌ | ❌ | ❌ |
| Other AMD architectures | ❌ | ❌ | ❌ | ❌ |
| NVIDIA GPU | ❌ | ❌ | ❌ | ❌ |
| Intel XPU | ❌ | ❌ | ❌ | ❌ |
Legend: `✅` supported by the implementation, `❌` unsupported.
ROCm uses the existing AITER online path and requires `mxfp4_scale_alg=0`.
## Model Type Support
### Diffusion Model (Wan2.2)
| Model | Online | Offline | Notes |
|-------|--------|---------|-------|
| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade (`transformer` + `transformer_2`); both transformers quantized with the same config |
| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade; same scheme as T2V-A14B |
| Wan2.2-TI2V-5B | ❌ | ❌ | Parameter count too small; W4A4 causes unacceptable accuracy loss |
| ------- | -------- | --------- | ------- |
| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4` / `mxfp4_dualscale` | Cascade experts resolve checkpoint metadata independently; explicit global runtime policy applies to both |
| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4` / `mxfp4_dualscale` | Per-expert configuration uses the same interface as T2V; validate I2V checkpoints separately |
| Wan2.2-TI2V-5B | Requires validation | Requires validation | This guide does not establish checkpoint compatibility or generation quality for TI2V-5B |
The choice between `mxfp4` and `mxfp4_dualscale` in **online mode** is about
quantization quality, not model compatibility — both work on cascade (A14B) and
single-transformer models alike, the same as `mxfp8` online:
Both methods can be integrated with single-transformer and cascade models.
This alone does not establish support or quality for every model variant:
- `mxfp4`: single-scale, lower overhead, simpler compute, online only
- `mxfp4_dualscale`: dual-scale + optional BF16 fallback, better accuracy, online **and** offline
- `mxfp4`: single-scale; online and pre-quantized weights, with optional W4A8 fallback on NPU
- `mxfp4_dualscale`: dual-scale; online and offline, with BF16 fallback
**Offline** checkpoints for A14B are always in `mxfp4_dualscale` format (produced
by the merge script); there is no offline `mxfp4` single-scale format.
The existing `merge_mxfp4_dualscale_checkpoint.py` produces **offline
`mxfp4_dualscale`** checkpoints. Single-scale weights must satisfy the separate
layout described below; changing a DualScale checkpoint's method name to
`mxfp4` does not convert its weights. Prepare single-scale checkpoints with
external tooling. Real exported checkpoint compatibility and model quality
require validation of that checkpoint; the synthetic tests do not establish
UOS export or video acceptance.
!!! note "Per-layer BF16 fallback in offline cascade models"
The A14B offline checkpoint uses `quant_method: mxfp4_dualscale`. Most
An A14B offline DualScale checkpoint uses `quant_method: mxfp4_dualscale`. Most
linear layers are stored as W4A4 MXFP4 DualScale; precision-sensitive layers
retain their original BF16 weights and are listed in `ignored_layers` inside
each transformer's `config.json`. The two transformers may have different
`ignored_layers` sets — the pipeline reads each transformer's own `config.json`
and rebuilds the config locally when they differ, so routing is always
per-transformer-accurate.
!!! warning "TI2V-5B not supported"
Wan2.2-TI2V-5B is excluded from W4A4 quantization. Its smaller parameter
count makes it significantly more sensitive to 4-bit quantization noise,
resulting in unacceptable accuracy loss. Use [MXFP8](mxfp8.md) for TI2V-5B.
and resolves the storage format and BF16 layer policy independently.
## Configuration
### W4A8 fallback at selected layers and denoising steps (Ascend NPU)
For single-scale `mxfp4`, set `w4a8_fallback_steps` to a list of zero-based
request denoising loop indices (not scheduler timestep values).
Set `w4a8_fallback_layers` to exact runtime Linear paths, for example
`["blocks.10.attn1.to_qkv", "blocks.12.ffn.net_2"]`. A selected layer uses W4A8
at every step; at a selected step, all other quantized layers also use W4A8.
The rules are combined with **OR**. W4A8 uses MXFP8 activations with MXFP4
weights. Omitting both lists or passing `[]` keeps the existing W4A4 behavior. Existing
`ignored_layers` take priority: **BF16 > (layer OR step) W4A8 > W4A4**.
DualScale's existing leading BF16 block rule is unchanged; it does not enable
W4A8.
```python
from vllm_omni import Omni
omni = Omni(
model="<Wan2.2-T2V-A14B-Diffusers>",
quantization_config={
"method": "mxfp4",
"w4a8_fallback_layers": ["blocks.10.attn1.to_qkv"],
"w4a8_fallback_steps": [0, 1, 38, 39],
},
)
```
Both policies also work with offline single-scale `mxfp4`. Any nonempty
W4A8 list with `mxfp4_dualscale`, online or offline, raises an explicit error
at configuration construction. Empty lists preserve its existing W4A4 path.
Each expert's checkpoint still determines its serialized
format and BF16 layer routing. A serialized checkpoint config is the complete
storage contract: `ignored_layers`, or its `modules_to_not_convert` alias,
selects the BF16 layers, and omitting both means no explicit BF16 layers. A
method-only checkpoint value such as `"mxfp4"` or
`{"quant_method": "mxfp4"}` carries no storage policy and leaves a caller's
online `ignored_layers` unchanged. An explicitly supplied quantization config
retains its W4A8 layer and step lists when the loader rebuilds either expert's
config from disk, including empty lists that disable checkpoint policies.
Within an explicit runtime config, an omitted W4A8 list defaults to `[]`;
individual omitted fields do not inherit saved policies. Without an explicit
runtime config, auto-detection uses the saved checkpoint policies.
Layer names are relative to each expert (`blocks.10...`, without a
`transformer.` or `transformer_2.` prefix). A global config applies the same
list to both experts; per-component `transformer`/`transformer_2` configs can
select different lists. Wan selects these expert names exactly: a `transformer`
entry does not match `transformer_2`; an omitted expert uses `default` if set.
A component mapped to `null` (or an unmatched component
without a default) stays unquantized and requires a BF16 checkpoint; loading a
quantized checkpoint for that disabled component raises an error. A top-level
`quantization_config=None` retains checkpoint auto-detection.
Names are exact, with no regex, wildcard, block expansion,
or checkpoint-name remapping. Wan validates them against its runtime Linear
modules and logs the selected paths/count and BF16 overrides per PP rank.
Use `attn1.to_qkv` for fused self-attention; `attn1.to_q`, `to_k`, or `to_v`
checkpoint names raise an error. Cross-attention `attn2.to_q/to_k/to_v` remains
separate. Layer-only fallback does not require a denoise ForwardContext.
The text-to-video example accepts the JSON directly:
```bash
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \
python examples/offline_inference/text_to_video/text_to_video.py \
--model /path/to/Wan2.2-T2V-A14B-Diffusers \
--quantization-config '{"method":"mxfp4","w4a8_fallback_steps":[0,2]}' \
--tensor-parallel-size 1 --ulysses-degree 2 --cfg-parallel-size 2 \
--enforce-eager --height 480 --width 832 --num-frames 81 \
--num-inference-steps 40 --seed 42 --output wan22-w4a8.mp4
```
Use either `--quantization` or `--quantization-config` in this example.
Serving accepts the same JSON through `--diffusion-quantization-config`.
The list uses one request's loop index across both high- and low-noise experts,
and both CFG branches see the same index. A new request starts at step zero.
Indices outside the request's step range are never selected. A pipeline must
publish `ForwardContext.denoise_step_idx`; requesting step fallback without
that context raises an error instead of silently ignoring the list.
Single-scale W4A4 and W4A8 reuse the same prepared `weight` and `weight_scale`.
Online loading quantizes the original BF16/FP16 weight once; offline loading
packs the numeric FP4 values and reshapes the saved E8M0 scale without
requantizing the weight. Only activation precision changes during forward.
There is no DualScale decoding, second W4 representation or `w4a8_weight` /
`w4a8_weight_scale` cache. No weight quantization runs inside the denoising loop.
The mixed FP8/FP4 GEMM uses BF16 output and a matching BF16 row bias for
compatibility with the target A5 runtime. An FP16 caller receives that result
converted back to FP16; its output shape and dtype are preserved.
The single-scale fallback validation target is eager Wan2.2 T2V A14B on A5.
Quality scores, optimal fallback lists, real UOS exports and additional
compile/offload combinations require separate evaluation. Existing W4A4-only
DualScale still needs complete 512-channel input groups; use TP=1 with
sequence/CFG parallelism for its initial multi-card run.
### `mxfp4` — Pre-Quantized Single-Scale Weights (Ascend NPU)
Set the following fields in each transformer's `config.json` when a
single-scale checkpoint uses calibrated Smooth scaling:
```json
{
"quantization_config": {
"quant_method": "mxfp4",
"is_checkpoint_mxfp4_serialized": true,
"require_smooth_scale": true,
"ignored_layers": []
}
}
```
Each quantized Linear must provide tensors under its model parameter name:
| Tensor | Checkpoint layout | Meaning |
| -------- | ------------------- | --------- |
| `weight` | `(N, K)` numeric FP4 values | Packed to FP4 after loading |
| `weight_scale` | `(N, ceil(K / 32))`, `uint8` | Raw E8M0 exponent bytes, required |
| `mul_scale` | `(K,)`, FP32 | Smooth pre-scale, required when declared above |
Both W4A4 and W4A8 quantize `x * mul_scale`. The Smooth tensor is converted
to the layer's activation dtype after loading. Row-parallel layers load the
corresponding input-channel shard. Pre-fused QKV weights must use one shared
Smooth tensor for the fused projection.
For backward compatibility, `require_smooth_scale` defaults to `false`:
an absent tensor uses identity scaling, while a present tensor is still loaded.
Without a declaration, an absent Smooth tensor cannot be distinguished from a
legacy unsmoothed checkpoint. Calibrated checkpoints should declare
`require_smooth_scale: true` so missing or misnamed Smooth tensors fail loading;
incorrect tensor shapes, non-floating tensors, NaN/Inf and nonpositive values
also fail. Values that overflow or underflow to zero in the activation dtype
are rejected during weight processing. The flag requires serialized single-scale
weights and is rejected for online quantization. It does not make DualScale
scales optional or change missing-parameter policies for other backends.
Either the checkpoint or an explicit caller can require Smooth: a `true`
declaration is retained when per-expert metadata rebuilds the configuration,
including when the other source omits the flag or sets it to `false`.
Prepare the canonical numeric FP4 checkpoint before inference using external
tooling. Runtime loading does not parse producer-specific export formats.
Retain the producer recipe to identify the weight-generation algorithm and
Smooth calibration; tensor storage alone does not establish UOS provenance.
### `mxfp4` — Single-Scale Block Scaling (OCP / UOS)
`mxfp4_scale_alg` selects the MXFP4 block-scaling method on NPU.
UOS is a scaling rule, not a separate activation dtype or an activation-only
algorithm. Both choices use FP4 E2M1 values with shared E8M0 scales.
| Mode | Weight preparation | A4 activation quantization | W4A8 fallback |
| --- | --- | --- | --- |
| Online | Selected scaling method, once at loading | Selected scaling method | Same W4; OCP MXFP8 A8 |
| Offline | Load existing W4 and scales unchanged | Selected scaling method | Same W4; OCP MXFP8 A8 |
The available settings are:
| Setting | MXFP4 scaling method | NPU parameters |
| --- | --- | --- |
| `0` (default) | OCP MX | `scale_alg=0` |
| `2` | UOS (`Qmax=7.25`) | `scale_alg=2, dst_type_max=7.25` |
Universal Optimal Scaling (UOS, `Qmax=7.25`) is derived in
[MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention](https://arxiv.org/abs/2607.24377),
Section 4.1. The value `7.25` is the boundary used to select the shared
power-of-two scale; the maximum finite E2M1 value remains `6`.
This option applies UOS to Linear quantization. It does not implement the
paper's full MXAttention pipeline or Pre-Normalization Quantization (PNQ).
Both use FP4 E2M1 with E8M0 scales, `axis=-1`, `block_size=32`,
`round_mode="rint"`. W4A8 always quantizes the activation to
`float8_e4m3fn` with `scale_alg=0` and the same grouping/rounding, without
UOS parameters. A4 and A8 share the prepared W4 and scale. UOS is currently
NPU-only; ROCm retains its existing AITER online algorithm.
An explicit UOS online configuration is
`{"method": "mxfp4", "mxfp4_scale_alg": 2}`. For offline checkpoints the
setting affects **only A4 activations**. Existing weights are never requantized:
a UOS checkpoint can run UOS W4 + UOS A4 or UOS W4 + OCP A4, and both fall back to
UOS W4 + OCP A8. Record the weight-generation algorithm separately from the
runtime activation setting. Smooth remains active with either setting. The
UOS configuration explicitly selects `2`; attention computation is unchanged.
### `mxfp4` — Single-Scale Online Mode
Online mode requires no pre-processing. vLLM-Omni quantizes BF16 weights to
MXFP4 at load time using `npu_dynamic_mx_quant`. A single block scale
(`float8_e8m0fnu`, one per 32 K elements) is computed on the fly; no
calibration `mul_scale` is available. Applies equally to single-transformer
and cascade (A14B) models — both transformers in a cascade receive the same
quantization config automatically.
Online mode quantizes BF16/FP16 weights once at loading and computes activation
scales during forward. Each block of 32 K elements shares one E8M0 scale;
no calibrated `mul_scale` is available. An explicit global quantization config
applies to both cascade experts. Per-component configs can select different
policies; checkpoint auto-detection resolves each expert independently.
```python
from vllm_omni import Omni
@@ -103,7 +289,7 @@ outputs = omni.generate(
```bash
# Single-transformer or cascade model — same command
python text_to_video.py --model <your-model> --quantization mxfp4
python examples/offline_inference/text_to_video/text_to_video.py --model <your-model> --quantization mxfp4
# Online serving
vllm serve <your-model> --omni --quantization mxfp4
@@ -113,19 +299,19 @@ vllm serve <your-model> --omni --quantization mxfp4
Online DualScale mode computes both fine and coarse scales on the fly from BF16
weights using `npu_dynamic_dual_level_mx_quant`. Applies equally to
single-transformer and cascade (A14B) models. Compared to `mxfp4` online,
DualScale provides better quantization accuracy at higher compute cost.
single-transformer and cascade (A14B) models with compatible Linear layers.
Compare quality and performance using the intended model and workload.
The default configuration keeps the leading 5 transformer blocks in BF16
(`num_bf16_fallback_layers=5`). Accuracy evaluation on Wan2.2-A14B shows this
is sufficient to meet quality requirements and is the recommended setting.
(`num_bf16_fallback_layers=5`). This is an implementation default; select and
validate the BF16 policy against your own quality target.
```python
omni = Omni(model="<your-model>", quantization="mxfp4_dualscale")
```
```bash
python text_to_video.py --model <your-model> --quantization mxfp4_dualscale
python examples/offline_inference/text_to_video/text_to_video.py --model <your-model> --quantization mxfp4_dualscale
```
If accuracy debugging identifies additional precision-sensitive layers, they
@@ -136,21 +322,22 @@ omni = Omni(
model="<your-model>",
quantization_config={
"method": "mxfp4_dualscale",
"ignored_layers": ["blocks.10.attn1.to_q"], # explicit per-layer override
"ignored_layers": ["blocks.10.attn1.to_qkv"], # explicit per-layer override
},
)
```
BF16 fallback routing in online mode applies two rules in priority order:
1. **`ignored_layers`** (explicit per-layer override): any layer whose prefix
matches is kept in BF16 regardless of block index.
1. **`ignored_layers`** (explicit per-layer override): a listed runtime Linear
module is kept in BF16 regardless of block index. Use exact module paths,
not arbitrary block prefixes or names ending in `.weight`.
2. **`num_bf16_fallback_layers`** (coarse leading-block rule): the first N
transformer blocks (`blocks.0` … `blocks.N-1`) fall back to BF16. Defaults
to `5` (recommended). Layers outside `blocks.N.*`
(e.g. `condition_embedder`) are always quantized.
to `5`. Layers outside `blocks.N.*` are quantized unless explicitly listed
in `ignored_layers`.
### `mxfp4_dualscale` — DualScale Offline Mode (Recommended)
### `mxfp4_dualscale` — DualScale Offline Mode
Offline mode loads a pre-quantized DualScale checkpoint from msModelSlim. A
preprocessing step converts the raw quantized output to the diffusers format
@@ -169,8 +356,8 @@ inside `config.json`. At runtime, each layer's prefix is matched against
Each quantized linear layer stores four tensors:
| Tensor | Shape | dtype | Description |
|--------|-------|-------|-------------|
| `weight` | `(N, K)` | float8_e4m3fn | FP4 packed (2 values per byte) |
| -------- | ------- | ------- | ------------- |
| `weight` | `(N, K)` | float8_e4m3fn | Numeric FP4 values; packed into two values per byte after loading |
| `weight_scale` | `(N, K//32)` | uint8 | Fine block scale (`float8_e8m0fnu` bit pattern) |
| `weight_dual_scale` | `(N, K//512, 1)` | float32 | Coarse block scale |
| `mul_scale` | `(K,)` | float32 | Per-input-channel smooth pre-scale (from calibration) |
@@ -223,7 +410,7 @@ python vllm_omni/quantization/tools/merge_mxfp4_dualscale_checkpoint.py \
```
| Argument | Description |
|----------|-------------|
| ---------- | ------------- |
| `--model-type` | Model variant: `Wan2.2-T2V-A14B` or `Wan2.2-I2V-A14B` |
| `--original-model` | Root directory of the original BF16 diffusers model |
| `--quant-path` | Root directory of the msModelSlim quantized output |
@@ -251,14 +438,14 @@ The `quantization_config` injected into `config.json` for each transformer:
```
`ignored_layers` lists every linear layer that retains its original BF16 weight,
using vllm-omni model parameter names (QKV-fused, FFN underscored, `to_out`
using vllm-omni runtime module paths (QKV-fused, FFN underscored, `to_out`
unindexed). The exact entries are determined by the quantization tool (msModelSlim)
and may differ between `transformer` and `transformer_2` in a cascade model.
#### Step 3 — Serve
```bash
python text_to_video.py --model /path/to/Wan2.2-T2V-A14B-MXFP4-DualScale
python examples/offline_inference/text_to_video/text_to_video.py --model /path/to/Wan2.2-T2V-A14B-MXFP4-DualScale
# Online serving
vllm serve /path/to/Wan2.2-T2V-A14B-MXFP4-DualScale --omni
@@ -275,25 +462,32 @@ omni = Omni(model="/path/to/Wan2.2-T2V-A14B-MXFP4-DualScale")
## Parameters
### `mxfp4` (single-scale, online only)
### `mxfp4` (single-scale, online + offline)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| ----------- | ------ | --------- | ------------- |
| `method` | str | — | `"mxfp4"` |
| `ignored_layers` | list[str] | `[]` | Layer prefixes to keep in BF16 |
| `mxfp4_scale_alg` | int | `0` | MXFP4 block scaling: `0` OCP MX, `2` UOS (`Qmax=7.25`, NPU only). Online: W4 and A4; offline: A4 only. A8 remains OCP MXFP8 |
| `is_checkpoint_mxfp4_serialized` | bool | `false` | Load pre-quantized single-scale weights and scales; declared in checkpoint `config.json` |
| `require_smooth_scale` | bool | `false` | Require calibrated Smooth tensors for offline single-scale weights |
| `ignored_layers` | list[str] | `[]` | Exact runtime Linear module paths to keep in BF16 |
| `w4a8_fallback_layers` | list[str] | `[]` | Exact runtime Linear paths that always use A8; NPU only |
| `w4a8_fallback_steps` | list[int] | `[]` | Zero-based denoise steps that use A8; NPU only |
### `mxfp4_dualscale` (dual-scale, online + offline)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| ----------- | ------ | --------- | ------------- |
| `method` | str | — | `"mxfp4_dualscale"` |
| `is_checkpoint_serialized` | bool | `False` | `True` for offline DualScale checkpoints; auto-set from `config.json` when using the preprocessing script |
| `ignored_layers` | list[str] | `[]` | Layer prefixes to keep in BF16. **Works in both modes**: offline — populated by the merge script for interleaved sensitive layers; online — user-supplied for explicit per-layer precision override |
| `num_bf16_fallback_layers` | int | `5` | **Online mode only**: leading N transformer blocks (`blocks.0` … `blocks.N-1`) kept in BF16. Applied after `ignored_layers`; ignored in offline mode. Default of `5` is the evaluated recommended value for Wan2.2-A14B |
| `ignored_layers` | list[str] | `[]` | Exact runtime Linear module paths to keep in BF16. **Works in both modes**: offline — populated by the merge script for interleaved sensitive layers; online — user-supplied for explicit per-layer precision override |
| `num_bf16_fallback_layers` | int | `5` | **Online mode only**: leading N transformer blocks (`blocks.0` … `blocks.N-1`) kept in BF16. Applied after `ignored_layers`; ignored in offline mode. Validate the default of `5` against the intended model and quality target |
| `w4a8_fallback_layers` | list[str] | `[]` | Must remain empty; DualScale does not support W4A8 fallback |
| `w4a8_fallback_steps` | list[int] | `[]` | Must remain empty; DualScale does not support W4A8 fallback |
#### BF16 fallback priority (online mode)
```
```text
for each linear layer:
if prefix in ignored_layers → BF16 (explicit override, highest priority)
elif block_idx < num_bf16_fallback_layers → BF16 (coarse leading-block rule)
@@ -307,8 +501,9 @@ unless they appear in `ignored_layers`.
1. **Online single-scale (`mxfp4`)** quantizes BF16 weights at load time using
`npu_dynamic_mx_quant` (single-scale). No calibration `mul_scale` is
available — all output partitions receive an identity pre-scale. No offline
checkpoint format exists for this method.
available — all output partitions receive an identity pre-scale. Offline
single-scale checkpoints use the canonical numeric FP4 layout
described above.
2. **Online dual-scale (`mxfp4_dualscale`, `is_checkpoint_serialized=False`)**
quantizes BF16 weights using `npu_dynamic_dual_level_mx_quant` (fine + coarse
@@ -316,7 +511,7 @@ unless they appear in `ignored_layers`.
explicit `ignored_layers` stay in BF16 for accuracy.
3. **Offline dual-scale (`mxfp4_dualscale`, `is_checkpoint_serialized=True`)** —
**recommended for production** — loads four tensors per quantized layer: FP4
loads four tensors per quantized layer: FP4
weight, fine scale (`uint8` reinterpreted as `float8_e8m0fnu`), coarse scale
(`float32`), and per-input-channel `mul_scale` (`float32`). BF16 fallback
layers have no quantization tensors and are routed via `ignored_layers`.
@@ -325,21 +520,21 @@ unless they appear in `ignored_layers`.
layout as `float8_e8m0fnu`) and reinterpreted at load time without a lossy
float32 round-trip.
5. **Cascade model config propagation**: in a cascade model (transformer +
transformer_2), vLLM-Omni reads each transformer's own `config.json` and
rebuilds the quant config locally when `ignored_layers` differs between
transformers, ensuring per-layer routing is accurate for each. The first
transformer's config is propagated to `od_config` so the second transformer
can reuse it as a starting point.
5. **Cascade model configuration**: checkpoint auto-detection resolves each
expert's own `config.json` independently; the first expert's detected config
is not reused as the second expert's policy. An explicit global runtime
config applies to both experts, while per-component configs select policies
by the exact `transformer` / `transformer_2` names. Serialized checkpoint
metadata still determines each expert's storage format and BF16 layers.
6. **Self-attention QKV fusion**: Q, K, V projection weights are fused into a
single `QKVParallelLinear` layer at runtime. `ignored_layers` entries use the
fused name (`attn1.to_qkv`), written automatically by the merge script.
7. W4A4 carries higher quantization noise than W8A8 (16 vs 256 levels). The
DualScale offline method mitigates this with calibrated `mul_scale` smooth
quantization. Use `ignored_layers` and `num_bf16_fallback_layers` to trade
off compression vs. accuracy for precision-sensitive layers.
7. FP4 has fewer representable values than FP8, so evaluate quantization
error and generation quality for the chosen recipe. Offline Smooth scaling
and BF16 fallback can help with sensitive layers.
`num_bf16_fallback_layers` applies only to online DualScale.
## Adapting MXFP4 for a New Model
@@ -350,27 +545,26 @@ a merge script for offline checkpoints.
### Step 1 — Discover runtime layer names
`ignored_layers` entries must match the **runtime parameter names** used inside
vllm-omni, which may differ from the names stored in the diffusers checkpoint.
The canonical source of truth is the model's own `named_parameters()`.
`ignored_layers` entries use exact **runtime Linear module paths**, without
`.weight`. Inspect the actual transformer instance in model initialization or
a developer harness; `Omni` does not expose an `omni.pipeline.transformer` API.
For a transformer instance named `transformer`:
```python
from vllm_omni import Omni
from vllm.model_executor.layers.linear import LinearBase
# Load the model without quantization to inspect parameter names.
omni = Omni(model="/path/to/your-model") # no --quantization flag
for name, _ in omni.pipeline.transformer.named_parameters():
if "weight" in name and "scale" not in name:
for name, module in transformer.named_modules():
if isinstance(module, LinearBase):
print(name)
```
Compare the printed names against the diffusers checkpoint keys
Compare these module paths against the corresponding diffusers checkpoint keys
(`safetensors.safe_open` or `torch.load`) to identify any renames your model
applies. Common patterns that differ in Wan2.2 (and may appear in other
models):
| Diffusers checkpoint name | vllm-omni runtime name | Reason |
|---------------------------|------------------------|--------|
| --------------------------- | ------------------------ | -------- |
| `attn1.to_q`, `attn1.to_k`, `attn1.to_v` | `attn1.to_qkv` | Self-attention Q/K/V fused into `QKVParallelLinear` |
| `ffn.net.0.proj` | `ffn.net_0.proj` | Dots in sub-module names replaced with underscores |
| `ffn.net.2` | `ffn.net_2` | Same underscore rule |
@@ -382,8 +576,9 @@ fused runtime parameters.
!!! warning "Partial QKV fallback is not allowed"
If your model fuses Q, K, V into a single layer, `ignored_layers` must
include **all three or none**. A partial fallback (e.g. `to_q` in BF16 but
`to_k`, `to_v` quantized) cannot be expressed at runtime because they share
name the fused runtime module (for Wan, `attn1.to_qkv`). The source
checkpoint must keep **all three or none** in BF16. A partial fallback
(e.g. `to_q` in BF16 but `to_k`, `to_v` quantized) cannot be expressed at runtime because they share
one `QKVParallelLinear`. The merge script enforces this and raises an error
if only some of the trio appear as non-quantized.
@@ -392,12 +587,13 @@ fused runtime parameters.
#### Online mode
Pass `ignored_layers` directly in the quantization config using the **runtime
names** discovered in Step 1. No code changes to the model are required.
names** discovered in Step 1. This assumes the model already passes its
quantization config and module prefixes to compatible vLLM Linear layers.
```python
omni = Omni(
model="/path/to/your-model",
quantization={
quantization_config={
"method": "mxfp4_dualscale",
"ignored_layers": [
"blocks.0.attn1.to_qkv", # runtime name, not diffusers name
@@ -409,9 +605,14 @@ omni = Omni(
```
```bash
# CLI does not support list-typed ignored_layers directly.
# Use the Python API or set ignored_layers in config.json (offline).
python your_script.py --model /path/to/your-model --quantization mxfp4_dualscale
# The text-to-video example accepts list-valued fields through JSON.
python examples/offline_inference/text_to_video/text_to_video.py \
--model /path/to/your-model \
--quantization-config '{"method":"mxfp4_dualscale","ignored_layers":["blocks.0.attn1.to_qkv"]}'
# Serving accepts the same configuration.
vllm serve /path/to/your-model --omni \
--diffusion-quantization-config '{"method":"mxfp4_dualscale","ignored_layers":["blocks.0.attn1.to_qkv"]}'
```
The `num_bf16_fallback_layers` coarse rule is an alternative to listing layers
@@ -436,9 +637,13 @@ the merge script is correct. The injected block:
}
```
To add a layer manually (e.g. to pin an additional layer to BF16 without
re-running the merge script), edit `config.json` inside the transformer
subfolder. Use runtime names, not diffusers checkpoint names.
Adding an offline BF16 fallback layer requires both the original BF16 weight
from the base model and a matching `ignored_layers` entry. Regenerate the
checkpoint with the revised storage policy, restoring the original weight
(and bias, if present) and removing that layer's quantization tensors.
Editing `config.json` alone does not reconstruct BF16 weights from stored FP4
values. For fused QKV, restore all three projections together. Use runtime
module paths in `ignored_layers`.
### Step 3 — Write a merge script for offline mode
@@ -474,6 +679,7 @@ renaming rules and collect the results.
!!! tip "Validate before serving"
After producing the offline checkpoint, load it without a `--quantization`
flag and verify that vLLM-Omni auto-detects the correct method. Check that
the layer count reported in the startup log matches expectations: quantized
layer count + `ignored_layers` count should equal total linear layer count.
Any mismatch indicates a name-mapping bug in the merge script.
each intended runtime Linear module resolves to the expected quantized or
BF16 method and that every BF16 entry has original BF16 weights. Account
for fused modules and, in online DualScale, the leading-block fallback rule;
the raw length of `ignored_layers` alone is not a reliable layer count.
+1 -1
View File
@@ -13,7 +13,7 @@ For the internal architecture and backend extension points, see the
| Mode | Guide | Description | Methods |
| ------ | ------- | ------------- | --------- |
| Online quantization | [Online Quantization](online.md) | vLLM-Omni computes quantized weights and scales while loading the model. | FP8 W8A8, Int8 W8A8, BitsAndBytes W4, MXFP8 W8A8, MXFP4 W4A4 |
| Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion Flash Attention tensors during inference. | FP8 FA |
| Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion attention tensors during inference. | FP8, MXFP8, MXFP4 |
| Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, TorchAO, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
## Hardware Support
@@ -2,41 +2,36 @@
## Overview
In DiT-based image and video generation, Flash Attention can take a large share
of denoising time, especially for high-resolution or long-frame workloads.
vLLM-Omni supports online FP8 quantization for eligible diffusion Flash
Attention (FA) to reduce FA latency while keeping model weights in their
original dtype.
In DiT-based image and video generation, attention can take a large share of
denoising time, especially for high-resolution or long-frame workloads.
vLLM-Omni supports online Q/K/V quantization for eligible diffusion attention
paths while keeping model weights in their original dtype.
This feature is configured through `diffusion_kv_cache_dtype` on
`OmniDiffusionConfig` (CLI: `--diffusion-kv-cache-dtype`). It is intentionally
**not** the same as vLLM's `--kv-cache-dtype`, which controls autoregressive
language-model KV cache storage and defaults to `"auto"`. Diffusion FA
quantization uses the dedicated diffusion flags so omni serve does not inherit
that default.
Runtime attention precision is configured through `diffusion_kv_cache_dtype`
on `OmniDiffusionConfig` (CLI: `--diffusion-kv-cache-dtype`). This setting is
separate from vLLM's `--kv-cache-dtype`, which controls autoregressive
language-model KV cache storage.
In vLLM-Omni diffusion pipelines, this is a runtime FA path: Q/K/V tensors are
dynamically quantized before the attention operator. It does not quantize model
weights and is separate from [FP8 W8A8](fp8.md), [Int8 W8A8](int8.md), or
pre-quantized checkpoint formats.
If `diffusion_kv_cache_dtype` is not set, behavior is unchanged and attention
runs in the native dtype.
If `diffusion_kv_cache_dtype` is not set, attention runs in the native dtype.
## Hardware Support
| Device | FP8 FA |
|--------|--------|
| Ascend NPU | ✅ |
| NVIDIA GPU | ❌ |
| AMD ROCm | ❌ |
| Intel XPU | ❌ |
| Device | Dense FP8 | Dense MXFP8/MXFP4 | BSA FP8/MXFP4 |
|--------|------------|-------------------|----------------|
| Ascend NPU | ✅ | ✅ | ✅ |
| NVIDIA GPU | ❌ | ❌ | ❌ |
| AMD ROCm | ❌ | ❌ | ❌ |
| Intel XPU | ❌ | ❌ | ❌ |
Legend: `✅` supported, `❌` unsupported.
FP8 FA is currently implemented only for the NPU Flash Attention backend. Other
backends do not support `diffusion_kv_cache_dtype="fp8"` for diffusion attention
and fall back to native dtype execution.
These runtime methods are implemented only by the NPU Flash Attention and BSA
backends. Other backends reject an incompatible explicit configuration.
## Model Type Support
@@ -44,7 +39,7 @@ and fall back to native dtype execution.
| Model | Scope | Status | Notes |
|-------|-------|--------|-------|
| Wan2.2 | Eligible DiT full-attention FA on Ascend NPU | Tested | Compare quality and latency against a BF16 baseline before production use |
| Wan2.2 | Self-attention on Ascend NPU | NPU interface and T2V smoke tested | Compare quality and latency against a BF16 baseline before production use |
| Other diffusion models | Eligible DiT full-attention FA on Ascend NPU | Not tested | You can try `diffusion_kv_cache_dtype="fp8"`; tune `diffusion_kv_cache_skip_steps` and `diffusion_kv_cache_skip_layers` when higher precision is needed |
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -105,12 +100,13 @@ The legacy keyword aliases `kv_cache_dtype`, `kv_cache_skip_steps`, and
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `diffusion_kv_cache_dtype` | str \| None | `None` | Set to `"fp8"` to enable dynamic FP8 FA on supported attention backends |
| `diffusion_kv_cache_dtype` | str \| None | `None` | Runtime attention method. Dense FA supports `fp8`, `mxfp8`, and `mxfp4`; BSA supports `fp8` and `mxfp4` |
| `diffusion_kv_cache_skip_steps` | str \| None | `None` | Denoising step selector to keep in native dtype, for example `"0,1,4-6"` |
| `diffusion_kv_cache_skip_layers` | str \| None | `None` | Transformer layer selector to keep in native dtype, for example `"0-2,10"` |
Selectors use comma-separated integers and inclusive ranges. Listed steps or
layers skip FP8 FA; all other eligible full-attention forwards use the FP8 path.
layers run floating-point attention through the selected Dense or BSA backend;
all other eligible forwards use `diffusion_kv_cache_dtype`.
## Validation and Notes
@@ -123,3 +119,60 @@ layers skip FP8 FA; all other eligible full-attention forwards use the FP8 path.
4. Report both latency and quality results when enabling this option for a new
model. For image or video models, include visual comparison and quantitative
metrics when available, such as PSNR or SSIM.
## Wan2.2 T2V quantized attention on Ascend
Wan2.2 T2V A14B supports quantized self-attention through MindIE-SD. Model
weights are unchanged, and model-level opt-out keeps cross-attention in its
native dtype.
| Attention path | self-attention `backend` | Supported `diffusion_kv_cache_dtype` |
| --- | --- | --- |
| Dense FA | `FLASH_ATTN` | `fp8`, `mxfp8`, `mxfp4` |
| BSA | `RAINFUSION_ATTN` | `fp8`, `mxfp4` |
Configure the method and fallback selectors once at the diffusion stage. The
per-role attention config only selects the self-attention backend:
```yaml
stages:
- stage_id: 0
diffusion_kv_cache_dtype: "mxfp8"
diffusion_kv_cache_skip_layers: "0,39"
diffusion_kv_cache_skip_steps: "0,1,38,39"
diffusion_attention_config:
per_role:
self:
backend: FLASH_ATTN # Use RAINFUSION_ATTN with fp8 or mxfp4 for BSA.
```
For 40 denoising steps, `diffusion_kv_cache_skip_steps: "0,1,38,39"` and
`diffusion_kv_cache_skip_layers: "0,39"` are recommended starting points. Steps
are zero-based across the complete request and do not reset when Wan switches
transformers; layer indices are zero-based and local to each transformer.
Selectors accept comma-separated indices and inclusive ranges such as
`"0,3-5"`. A selected forward uses floating-point attention while preserving
the Dense or BSA path.
Unsupported precision/input combinations raise an error and must be corrected
in the configuration; operator errors are not retried with another precision.
For BSA, change the backend to `RAINFUSION_ATTN` and set
`diffusion_kv_cache_dtype` to `fp8` or `mxfp4`.
See [RainFusion attention](../diffusion/attention_backends/rainfusion.md) for
sparse-path configuration and behavior.
### Requirement and interface test
Until these APIs are available in a MindIE-SD release, use revision
[`8637b5333b0225381b215390fd09a8732e671cc4`](https://gitcode.com/zqxu/MindIE-SD/commit/8637b5333b0225381b215390fd09a8732e671cc4).
Build the Python package, PyTorch plugin, and custom operators from that same
checkout. With CANN sourced, run the NPU interface tests from the Omni checkout:
```bash
python -m pytest tests/platforms/npu/quant/test_kv_quant_npu.py \
-k real_npu -vv -s -o addopts=''
```
A skipped NPU test is not a pass. Validate generated video quality against a
floating-point run with the same prompt, seed, resolution, and step count.