Merge remote-tracking branch 'origin/main' into qwen-image-2.1
# Conflicts: # vllm_omni/diffusion/executor/multiproc_executor.py # vllm_omni/diffusion/offloader/sequential_backend.py
This commit is contained in:
commit
9174554e40
1332 files changed
+160051
-60504
No files matched your search
@@ -71,6 +71,7 @@ nav:
|
||||
- Overview and Compatibility:
|
||||
- Overview: user_guide/diffusion_features.md
|
||||
- Feature Compatibility: user_guide/feature_compatibility.md
|
||||
- Device-side Video Postprocessing: user_guide/diffusion/device_side_video_postprocess.md
|
||||
- CPU Offloading:
|
||||
- Overview: user_guide/diffusion/cpu_offload.md
|
||||
- Model-Level: user_guide/diffusion/offloader/module_offload.md
|
||||
@@ -135,6 +136,7 @@ nav:
|
||||
- Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md
|
||||
- Mooncake Transfer Engine Connector: design/feature/omni_connectors/mooncake_transfer_engine_connector.md
|
||||
- Mori Transfer Engine Connector: design/feature/omni_connectors/mori_transfer_engine_connector.md
|
||||
- NIXL Connector: design/feature/omni_connectors/nixl_connector.md
|
||||
- Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
|
||||
- Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
|
||||
- Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
|
||||
@@ -159,6 +161,7 @@ nav:
|
||||
- Cache-DiT: design/feature/cache_dit.md
|
||||
- TeaCache: design/feature/teacache.md
|
||||
- Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
|
||||
- Mooncake KV Transfer: design/feature/mooncake-kv-transfer.md
|
||||
- Infrastructure and Performance:
|
||||
- Prometheus Metrics: design/metrics.md
|
||||
- Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
|
||||
|
||||
+1
-1
@@ -56,7 +56,7 @@ vLLM-Omni is flexible and easy to use with:
|
||||
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
|
||||
|
||||
- Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
|
||||
- TTS models (e.g. Qwen3-TTS, IndexTTS 2.5, dots.tts, CosyVoice3)
|
||||
- TTS models (e.g. Qwen3-TTS, IndexTTS 2.5, CosyVoice3)
|
||||
- Diffusion models — image, video, and audio generation (e.g. MiniMax H3, LTX-2.5, SANA-Video, Wan2.2)
|
||||
- Robot-policy and action models (e.g. π0, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
|
||||
|
||||
|
||||
+6
-1
@@ -5,6 +5,9 @@
|
||||
Main entry points for vLLM-Omni inference and serving.
|
||||
|
||||
- [vllm_omni.entrypoints.async_omni.AsyncOmni][]
|
||||
- [vllm_omni.entrypoints.async_omni_base.AsyncOmniBase][]
|
||||
- [vllm_omni.entrypoints.duplex_omni.DuplexOmni][]
|
||||
- [vllm_omni.entrypoints.duplex_omni.DuplexSessionHandle][]
|
||||
- [vllm_omni.engine.cfg_companion_tracker.CfgCompanionTracker][]
|
||||
- [vllm_omni.entrypoints.cli.benchmark.base.OmniBenchmarkSubcommandBase][]
|
||||
- [vllm_omni.entrypoints.cli.benchmark.main.OmniBenchmarkSubcommand][]
|
||||
@@ -17,12 +20,13 @@ Main entry points for vLLM-Omni inference and serving.
|
||||
|
||||
## Clients
|
||||
|
||||
Client libraries that talk to a running vLLM-Omni server (the full-duplex Realtime client and its per-model session presets).
|
||||
Client libraries for full-duplex sessions: the Realtime WebSocket client, the in-process client over `DuplexOmni`, and the per-model session presets.
|
||||
|
||||
- [vllm_omni.clients.duplex.AudioDelta][]
|
||||
- [vllm_omni.clients.duplex.AudioFormat][]
|
||||
- [vllm_omni.clients.duplex.ConnectionResumed][]
|
||||
- [vllm_omni.clients.duplex.DuplexClient][]
|
||||
- [vllm_omni.clients.duplex.DuplexClientBase][]
|
||||
- [vllm_omni.clients.duplex.DuplexClientError][]
|
||||
- [vllm_omni.clients.duplex.DuplexConnectionError][]
|
||||
- [vllm_omni.clients.duplex.DuplexEvent][]
|
||||
@@ -44,6 +48,7 @@ Client libraries that talk to a running vLLM-Omni server (the full-duplex Realti
|
||||
- [vllm_omni.clients.duplex.TextDelta][]
|
||||
- [vllm_omni.clients.duplex.TranscriptDelta][]
|
||||
- [vllm_omni.clients.duplex.WebSocketTransport][]
|
||||
- [vllm_omni.clients.inline_duplex.InlineDuplexClient][]
|
||||
|
||||
## Inputs
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 272 KiB After Width: | Height: | Size: 267 KiB |
+60
-3
@@ -23,7 +23,9 @@ You can use `vllm bench serve --omni --help=all` to get descriptions of all para
|
||||
The API endpoint exposed externally, to which clients send their requests.
|
||||
|
||||
- `--dataset-name`
|
||||
The name of the dataset used; random-mm indicates generating random multimodal inputs (images, videos, audio), while `omniinteract` replays official OmniInteract videos.
|
||||
The name of the dataset used; `random-mm` generates random multimodal inputs (images, videos, audio),
|
||||
`omniinteract` replays official OmniInteract videos, and `videomme` loads Video-MME MCQ videos
|
||||
(`lmms-eval/Video-MME`, MiniCPM OmniEvalKit packing).
|
||||
|
||||
- `--num-prompts`
|
||||
The total number of requests to send, an integer.
|
||||
@@ -316,8 +318,12 @@ arrives, checkpointing the response's history position so a later committed user
|
||||
outputs are ineligible and omitted from the official manifest; `audio_clipped_bytes` records output beyond the rounded video
|
||||
horizon.
|
||||
|
||||
TTFT, TTFP, and RTF start at client receipt of `response.created`. TPOT/ITL use engine stage-0 timing; ITL is emitted only when
|
||||
every token interval is present.
|
||||
Per-response TTFT and TTFP start when the server begins executing the native model-turn request that owns the response. RTF
|
||||
continues to use client receipt of `response.created` through the last audio packet, divided by emitted audio duration. Global
|
||||
TTFT, TTFP, and RTF cover the complete input-stream window. TPOT/ITL use engine stage-0 timing; ITL is emitted only when every
|
||||
token interval is present within a continuous generation segment. Model-unit pacing and gaps between generation segments are
|
||||
excluded from TPOT/ITL. Raw request metrics retain `response_created_to_first_text_ms` and
|
||||
`response_created_to_first_audio_ms` as client-envelope diagnostics.
|
||||
|
||||
The checked-in local performance configuration measures four deterministic cases from each OmniInteract subset (12 videos
|
||||
total), with no benchmark warmups and a maximum concurrency of two. Each subset also sends one readiness request before its
|
||||
@@ -339,6 +345,57 @@ transcript, event, and result artifacts without errors. A valid LISTEN-only case
|
||||
chunks. Official-manifest eligibility is reported separately because clipped or cancelled output is a benchmark-quality
|
||||
signal, not a transport failure. This local performance test does not score answer accuracy.
|
||||
|
||||
### Video-MME Benchmark
|
||||
|
||||
Video-MME (`--dataset-name videomme`) scores multiple-choice video QA. Default packing is
|
||||
OmniEvalKit MiniCPM `minicpm-frames` (up to 96 sampled frames as `image_url`). Pass a local
|
||||
mirror with `--dataset-path` / `--videomme-parquet` + `--videomme-video-dir`, or a Hugging
|
||||
Face dataset id (`lmms-eval/Video-MME` by default; any `org/name` is accepted when
|
||||
`--dataset-name videomme` is explicit). Relative `--videomme-video-dir` values are resolved
|
||||
to absolute `file://` URLs. For those URLs, start the server with
|
||||
`--allowed-local-media-path` covering the video root; otherwise use `--videomme-inline-local-video`.
|
||||
|
||||
For MiniCPM-o 4.5, the default deployment allows 64 images per request. To run the
|
||||
96-frame recipe, start the server with a stage 0 override (replace `/path/to/Video-MME`
|
||||
with the local video root shared by the benchmark client and server):
|
||||
|
||||
```bash
|
||||
vllm serve openbmb/MiniCPM-o-4_5 --omni \
|
||||
--allowed-local-media-path /path/to/Video-MME \
|
||||
--stage-overrides '{"0":{"limit_mm_per_prompt":{"image":96,"audio":64,"video":1}}}'
|
||||
```
|
||||
|
||||
The override applies to this server invocation only. Alternatively, use
|
||||
`--videomme-max-frames 64` in the benchmark to stay within the default image limit;
|
||||
this evaluates a different frame-sampling setting from the 96-frame recipe.
|
||||
|
||||
Run the benchmark against that server:
|
||||
|
||||
```bash
|
||||
vllm bench serve --omni \
|
||||
--backend openai-chat-omni \
|
||||
--dataset-name videomme \
|
||||
--dataset-path /path/to/Video-MME \
|
||||
--videomme-pack-mode minicpm-frames \
|
||||
--videomme-max-frames 96 \
|
||||
--model openbmb/MiniCPM-o-4_5 \
|
||||
--endpoint /v1/chat/completions \
|
||||
--num-prompts 8 \
|
||||
--max-concurrency 1 \
|
||||
--save-result
|
||||
```
|
||||
|
||||
Accuracy keys (`videomme_accuracy`, per-duration / domain / task breakdowns) are written into
|
||||
the saved JSON. `videomme_accuracy` excludes HTTP failures; `videomme_accuracy_incl_http_fail`
|
||||
counts them as wrong. `videomme_submitted` / `videomme_unique_question_ids` show when
|
||||
oversampling made the request count differ from unique-question coverage.
|
||||
`videomme_skipped_rows` counts rows skipped during sampling because of missing or unreadable
|
||||
media or invalid question fields. The opt-in accuracy runner rejects skipped rows, HTTP
|
||||
failures, duplicate question IDs, and missing gold answers. This checks the sampled subset;
|
||||
it does not establish that a supplied local mirror contains the entire official dataset.
|
||||
Use `--videomme-save-eval-items` (or `VIDEOMME_SAVE_EVAL_ITEMS=1`) for per-request
|
||||
`videomme_eval_items` rows.
|
||||
|
||||
### Multi-Modal Benchmark
|
||||
|
||||
<details class="admonition abstract" markdown="1">
|
||||
|
||||
@@ -1,5 +1,67 @@
|
||||
# vllm-omni serve
|
||||
|
||||
## Multiple API frontends
|
||||
|
||||
For local EngineCore pipelines with one or more stages, `--api-server-count` starts
|
||||
multiple API frontend processes that share one parent-owned set of stage
|
||||
engines:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--api-server-count 2 \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
This mode is intended for workloads whose request parsing, multimodal input
|
||||
processing, or response encoding can bottleneck a single frontend. It
|
||||
currently requires local multi-process EngineCore stages with one local
|
||||
process group per replica. A stage may still use multiple `num_replicas`.
|
||||
It cannot be combined with diffusion, headless or
|
||||
remote stages, intra-stage data parallelism, Ray, fault tolerance, elastic
|
||||
expert parallelism, sleep mode, or runtime LoRA updating. Runtime voice upload
|
||||
and deletion are also disabled because those mutations are process-local;
|
||||
built-in voices, inline reference audio, and voices restored at startup remain
|
||||
available. The `/v1/omni/sleep` and `/v1/omni/wakeup` control routes return HTTP
|
||||
409 for the same reason: their bookkeeping is process-local while stage engines
|
||||
are shared.
|
||||
|
||||
A single-stage model pipeline is supported through the normal launch command;
|
||||
this is distinct from the distributed `--stage-id` / `--headless` launch mode.
|
||||
Every stage in the pipeline must be an EngineCore stage. Mixed AR-to-DiT
|
||||
pipelines, including MiniMax-H3, remain unsupported even when their first
|
||||
stage is an EngineCore stage. Diffusion support requires per-frontend channels
|
||||
and shared-engine lifecycle handling in the diffusion backend.
|
||||
|
||||
### Workloads and performance validation
|
||||
|
||||
Each API process owns its frontend processor, Orchestrator, request state,
|
||||
and per-replica EngineCore channels. Requests return through the channel of
|
||||
the originating API process; model weights and GPU stage engines are shared.
|
||||
This distributes frontend and orchestration work across CPU processes.
|
||||
|
||||
| Workload | Potential benefit | Limitation |
|
||||
| ---------- | ------------------- | ------------ |
|
||||
| Concurrent cold image/video inputs on supported EngineCore pipelines | Parallel frontend processing when one API process is CPU-limited | A single cold request and GPU computation may be unchanged |
|
||||
| Many replicas producing frequent text/audio outputs | Less API/Orchestrator work and GIL contention per process | Routing remains serial within each Orchestrator |
|
||||
| Warm inputs or low-concurrency, GPU-limited generation | Additional frontend capacity | More API processes may add overhead without improving throughput |
|
||||
|
||||
Here, a cold input means media or preprocessing caches miss, not that model
|
||||
weights must be loaded or GPU kernels compiled. Each frontend has its own
|
||||
processor/cache state, so cache duplication, CPU thread contention, and memory
|
||||
usage also matter. Multi-API serving does not itself change the output queue,
|
||||
move response encoding to an executor, or change the model's numerical scheduler.
|
||||
|
||||
Compare `--api-server-count 1`, `2`, and higher counts with the same hardware,
|
||||
model/stage configuration, media, sampling parameters, output lengths, and
|
||||
offered load. Keep cold-cache and warmed-cache runs separate and report repeated
|
||||
measurements. Record CPU and request counts for every API worker, input and
|
||||
output queue delay, GPU utilization, first-output latency, and end-to-end latency.
|
||||
Confirm requests reach every worker: connection reuse can concentrate traffic
|
||||
on one process. Queue-only or single-API experiments do not establish a
|
||||
multi-API throughput gain. The multi-replica workload in
|
||||
[#4680](https://github.com/vllm-project/vllm-omni/issues/4680) remains a relevant
|
||||
validation target.
|
||||
|
||||
## Stage-based CLI quickstart
|
||||
|
||||
The stage-based CLI is designed for deployments that require launching each pipeline stage in an isolated process
|
||||
@@ -61,3 +123,10 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
## Arguments
|
||||
|
||||
--8<-- "docs/generated/argparse_omni/omni_serve.inc.md"
|
||||
|
||||
Async video jobs use process-local metadata and task handles. With multiple API
|
||||
workers, creating, listing, retrieving, downloading, and deleting these jobs
|
||||
returns HTTP 409. This restriction is independent of the current diffusion
|
||||
stage launch restriction: supporting multi-worker diffusion also requires a
|
||||
shared job lifecycle. Missing frontend topology state returns HTTP 503 for
|
||||
operations that require process-local state protection.
|
||||
@@ -70,6 +70,7 @@ The lifecycle labels used below are:
|
||||
| `OMNI_DIFFUSION_PROMPT_EMBED_CACHE_SIZE` | Positive integer; default `32` entries | Each diffusion runner; resolved during model setup | A valid environment value overrides the explicit cache size. A non-integer logs a warning; a non-positive value is ignored. | Experimental |
|
||||
| `OMNI_DIFFUSION_SESSION_STATE_MANAGER` | Boolean with the same accepted spellings as the prompt cache; default disabled | Experimental diffusion session manager; model setup | A recognized environment value overrides the explicit enable setting. An unrecognized value is ignored. | Experimental |
|
||||
| `OMNI_DIFFUSION_SESSION_STATE_MANAGER_MAX_SESSIONS` | Positive integer; default `64` | Experimental diffusion session manager; model setup | A valid environment value overrides the explicit maximum. A non-integer logs a warning; a non-positive value is ignored. | Experimental |
|
||||
| `VLLM_OMNI_AR_DIFFUSION_KV_GATHER` | `1` enables; default `0` (off) | AR-diffusion KV manager at allocation and attention at dispatch; set before worker startup | Only the exact value `1` enables contiguous KV gathering. History staging additionally requires `ARDiffusionKVConfig.reuse_history_staging`. | Experimental |
|
||||
|
||||
Backend names for `DIFFUSION_ATTENTION_BACKEND` are the members of
|
||||
`DiffusionAttentionBackendEnum`, such as `FLASH_ATTN`, `TORCH_SDPA`,
|
||||
@@ -82,6 +83,7 @@ depends on the installed kernels and model path.
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `SPEAKER_SAMPLES_DIR` | Filesystem path; default `~/.cache/vllm-omni/speakers` | Speech server; read when speaker storage initializes | Environment-only setting. The directory is created; filesystem errors propagate. | Stable |
|
||||
| `SPEAKER_MAX_UPLOADED` | Integer; default `1000` | Speech server; read when speaker storage initializes | Environment-only setting. A non-integer logs a warning and uses `1000`; range is not otherwise validated. | Stable |
|
||||
| `VLLM_OMNI_ABORT_TIMEOUT` | Float seconds; default `2` | Engine abort wait for Videos DELETE and `generate()` cancel/error cleanup; read when `async_omni` imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
|
||||
| `VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT` | Float seconds; default `600` | Diffusion engine async-output wait in `step_streaming`; resolved per call on the request path, not at import | Environment-only setting. A non-float or `<=0` value warns once and uses the default. | Experimental |
|
||||
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; default `0` (off) | Orchestration loop and the serving-side final-output drain; read once when the `Orchestrator` is constructed | Environment-only setting. Values are stripped and case-normalized; any unrecognized value leaves the legacy poll loop selected. | Experimental |
|
||||
| `VLLM_OMNI_INPUT_WAIT_TIMEOUT_S` | Float seconds; default `600`; `<=0` disables | Full-payload input coordinator, not async-chunk transfer; read when the scheduler module imports in each worker | Environment-only setting. A non-float logs a warning and uses `600`. | Stable operational control |
|
||||
@@ -90,6 +92,13 @@ depends on the installed kernels and model path.
|
||||
| `VLLM_VIDEO_ASYNC_CHUNK` | `on` or `off`; default `on` | Streaming video output; read on attribute access | Environment-only setting. Values are trimmed and case-normalized; an invalid value warns once and uses `on`. | Experimental |
|
||||
| `VLLM_VIDEO_AUDIO_DELTA_MODE` | `fast` or `slow`; default `fast` | Streaming video audio deltas; read on attribute access | Environment-only setting. Values are trimmed and case-normalized; an invalid value warns once and uses `fast`. | Experimental |
|
||||
|
||||
### NIXL stage transfer
|
||||
|
||||
| Name | Type and default | Applies to and read time | Precedence and invalid values | Lifecycle |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `VLLM_OMNI_NIXL_LEASE_S` | Float seconds; default `3600` | Unclaimed producer payload expiry; connector construction | Nonempty environment value overrides `lease_seconds`; empty falls back to config/default. Invalid floats raise `ValueError`. Claimed READ allocations never expire by time. | Experimental |
|
||||
| `VLLM_OMNI_NIXL_XFER_TIMEOUT_S` | Float seconds; default `300` | Receiver READ wait; connector construction | Nonempty environment value overrides `transfer_timeout_s`; empty falls back to config/default. Invalid floats raise `ValueError`. Timeout does not cancel DMA or release active allocations. | Experimental |
|
||||
|
||||
### Server storage
|
||||
|
||||
Storage names use Pydantic's nested-settings delimiter (`__`). They are read
|
||||
@@ -112,6 +121,7 @@ settings.
|
||||
|
||||
| Name | Type and default | Applies to and read time | Precedence and invalid values | Lifecycle |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `VLLM_OMNI_FUSED_QK_NORM_ROPE_MIN_TOKENS` | Non-negative integer; unset or empty means each consumer's own default (Boogu-Image: `2048`) | Token gate of the shared `fused_qk_norm_rope` op: a rotary table is packed for the fused kernel only when it spans at least this many positions (`B*S`); read at forward time | Environment-only. `0` = always fuse; a very large value disables the fused path. Any other value raises `ValueError` on the first forward. The crossover is host-dependent; re-benchmark before overriding. | Experimental performance control |
|
||||
| `VLLM_OMNI_SKIP_NVFP4_NAN_CLAMP` | Boolean truthy spellings: `1`, `true`, `yes`, `on`; default false | ModelOpt NVFP4 compatibility patch; read when `vllm_omni.patch` imports | Environment-only escape hatch. Any other value means false. Set only to diagnose the upstream NaN-scale issue. | Diagnostic and temporary |
|
||||
| `VLLM_OMNI_USE_QUACK_FP8` | Boolean truthy spellings: `1`, `true`, `yes`, `on`; unset means hardware auto-detection | FP8 scaled matrix multiplication; evaluated when quack capability is selected | A set value overrides auto-detection. Any non-truthy value forces quack off. If quack cannot load, vLLM-Omni warns and falls back to FlashInfer. | Experimental performance control |
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ Common `PipelineConfig` fields include:
|
||||
| ------- | ------------- |
|
||||
| `model_type` | Pipeline identifier used during model and config resolution. |
|
||||
| `default_deploy_config_name` | Bundled deploy YAML loaded when the user does not pass `deploy_config`. |
|
||||
| `duplex_plugin` | Dotted path of the model's `DuplexModelPlugin`. Set only for full-duplex models; it makes `vllm-omni serve` run the model through `DuplexOmni` (a server whose every surface runs on a duplex session) and the engine host a `DuplexOrchestrator` with the plugin loaded. |
|
||||
| `model_arch` | Default Hugging Face architecture for the pipeline. |
|
||||
| `hf_architectures` | Architecture names used to identify checkpoints whose `model_type` is shared. |
|
||||
| `hf_config_predicate` | Optional predicate used to select between pipelines with otherwise identical HF metadata. |
|
||||
@@ -38,7 +39,7 @@ Common `StagePipelineConfig` fields include:
|
||||
| `custom_process_input_func` | Processor applied to this stage's incoming payload. |
|
||||
| `custom_process_next_stage_input_func` | Processor used for full-payload handoff to the next stage. |
|
||||
| `async_chunk_process_next_stage_input_func` | Processor used for async chunk handoff. |
|
||||
| `sampling_constraints` | Model-owned sampling constraints that deploy defaults cannot override. |
|
||||
| `sampling_constraints` | Model-owned sampling constraints that deploy defaults cannot override. Scalar values replace deploy defaults; required `stop_token_ids` extend and deduplicate them. |
|
||||
|
||||
To add or change topology, define and register a new pipeline variant. Use
|
||||
deploy YAML only for runtime placement, resource sizing, connectors, and other
|
||||
@@ -52,7 +53,7 @@ The new deploy schema lives under `vllm_omni/deploy/` and is paired with a froze
|
||||
| ------- | ------ | ---------- | --------- | ------------- |
|
||||
| `base_config` | str (path) | optional | — | Overlay parent (relative or absolute). `stages:` / `platforms:` deep-merged by stage_id; other scalars overlay-wins. Intended for user-authored overlays; prod yamls stay flat. |
|
||||
| `async_chunk` | bool | optional | `true` | Enable chunked streaming between stages. Pin to `false` if the pipeline runs end-to-end. |
|
||||
| `session_mode` | str | optional | `"turn"` | Session behavior. MiniCPM-o 4.5 deploy YAMLs set `"duplex"` so `/v1/realtime` and chat share the same profile. |
|
||||
| `session_mode` | str | optional | `"turn"` | Session behavior. For pipelines with a `duplex_plugin`, explicitly select `"duplex"` for the duplex engine or `"turn"` for the ordinary online serving stack. The shipped MiniCPM-o 4.5 default remains `"duplex"`. See [Full Duplex](../serving/full_duplex_api.md#enable-full-duplex). |
|
||||
| `active_stream_window` | int | optional | `0` | Number of active downstream stream slots; `0` preserves all-stream cycling. |
|
||||
| `duplex_session` | dict | optional | runtime defaults | Full-duplex session lifecycle, buffering, replay, and capacity limits. |
|
||||
| `connectors` | dict | optional | `null` | Named connector specs (`{name, extra}`). Referenced by each stage's `input_connectors` / `output_connectors`. See [Connector schema](#connector-schema). |
|
||||
@@ -97,7 +98,7 @@ Each entry under `stages:` accepts any `StageDeployConfig` field directly (no ne
|
||||
| `devices` | str \| null | optional | `null` | Device list assigned to this stage. The number of device ids must equal this stage's local world size (`tensor_parallel_size` × local data-parallel size × `pipeline_parallel_size`, or `num_replicas` × that product for a replica pool); a mismatch fails early — see the note below. |
|
||||
| `output_connectors` | dict \| null | optional | `null` | Keyed by `to_stage_<n>`; values are names registered under top-level `connectors:`. |
|
||||
| `input_connectors` | dict \| null | optional | `null` | Keyed by `from_stage_<n>`; values are names registered under top-level `connectors:`. |
|
||||
| `default_sampling_params` | dict \| null | optional | `null` | Baseline sampling params. Deep-merged with pipeline `sampling_constraints` (pipeline wins). |
|
||||
| `default_sampling_params` | dict \| null | optional | `null` | Baseline sampling params. Merged with pipeline `sampling_constraints`; scalar constraints win, while required `stop_token_ids` are appended and deduplicated. |
|
||||
| `engine_extras` | dict | optional | `{}` | Catch-all for engine fields not listed above; deep-merged across overlays and forwarded to the stage engine. |
|
||||
|
||||
**Note:** a stage's `devices` count must equal its local world size (`tensor_parallel_size` × `data_parallel_size_local` × `pipeline_parallel_size`, falling back to global `data_parallel_size` when the local size is unset), or `num_replicas` × that product for a replica pool. A mismatch fails early and names the offending stage. A top-level `--tensor-parallel-size` is broadcast to every stage, so it can make a single-GPU stage violate this contract ([issue #5003](gh-issue:5003)); fix that case with `--stage-overrides` (set `tensor_parallel_size` and `devices` together per stage) or set TP only on the multi-GPU stage.
|
||||
@@ -289,3 +290,90 @@ vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/depl
|
||||
|
||||
!!! important
|
||||
We are actively iterating on the definition of deployment configurations, and we welcome feedback from users and developers.
|
||||
|
||||
## Qwen3-TTS with Model Runner V2
|
||||
|
||||
Qwen3-TTS can opt into the native CUDA Model Runner V2 pipeline on vLLM
|
||||
0.29.0. Select one of these deployment profiles:
|
||||
|
||||
| Profile | Runner | Code2Wav graph batches | Intended use |
|
||||
| --- | --- | --- | --- |
|
||||
| `qwen3_tts.yaml` | V1 | Existing defaults | Existing deployment / regression control |
|
||||
| `qwen3_tts_mrv2.yaml` | V2 | B1 | Native runner validation |
|
||||
| `qwen3_tts_high_concurrency_mrv2.yaml` | V2 | B1, B2 | Opt-in throughput tuning |
|
||||
| `qwen3_tts_high_concurrency_mrv2_b4.yaml` | V2 | B1, B2, B3, B4 | Experimental throughput / buffered playback |
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni \
|
||||
--deploy-config vllm_omni/deploy/qwen3_tts_mrv2.yaml
|
||||
```
|
||||
|
||||
The V2 profiles bound Talker prefill to 512 tokens per step and select the
|
||||
Talker AR runner and the Code2Wav generation
|
||||
runner together. Native inter-stage delivery carries codec payloads directly;
|
||||
request-owned snapshots preserve buffers through asynchronous completion and
|
||||
CUDA graph reuse. Terminal completion waits for upstream stage metrics before
|
||||
releasing request state. Platform sections retain V1 on NPU, XPU, ROCm and MUSA;
|
||||
this change does not qualify MRV2 on those backends or enable other model families.
|
||||
Selecting V2 for a stage without `supports_native_mrv2_data_plane` emits a warning
|
||||
when pipeline and deployment settings are merged. That stage retains the legacy
|
||||
transport path; the warning does not establish support for that combination.
|
||||
|
||||
MRV2 model hooks use capability declarations rather than architecture or stage
|
||||
names. Omni lifecycle flags select the model state, and `_returns_tuple` declares
|
||||
the capture output contract. The optional batched predictor hook is `mtp`, with
|
||||
an explicit `mtp_output_key` (a string or two-part payload key), optional
|
||||
`mtp_validity_key`, and `mtp_graph_safe`/`mtp_disable_graph` capture controls.
|
||||
Its inputs are token IDs, embeddings, previous hidden states and per-row
|
||||
conditioning; it returns updated embeddings and prediction codes. Models may
|
||||
supply `mtp_sampling_params` and `get_mtp_seed(sampling_params)` for model-local explicit seeds, and declare
|
||||
`mtp_accepts_per_row_generators`, `mtp_accepts_req_infos`, or `mtp_sample_uniforms`
|
||||
(with `mtp_sample_steps` and `mtp_sample_vocab_size`) as needed. Qwen3-TTS retains
|
||||
its existing `talker_mtp` entry point for V1.
|
||||
|
||||
### Included performance work
|
||||
|
||||
- Request snapshots have a fast path for immutable scalar leaves, including
|
||||
waveform lists. Mutable containers, aliases and cycles keep deep-copy
|
||||
semantics. This is not the historical shallow `dict(prompt)` experiment.
|
||||
- The API reference cache holds owned float32 arrays with a default capacity
|
||||
of 1024 entries and a 512 MiB waveform-payload budget. Cache hits return
|
||||
independent lists. Qwen3-TTS model artifact caches also default to 1024
|
||||
entries. These caches have different owners and eviction policies; equal
|
||||
entry limits do not make them a single coherent cache.
|
||||
- Talker state stays on GPU where its lifecycle allows it. BOS/EOS projections
|
||||
are cached in their projection dtype and invalidated on weight loading.
|
||||
- Code2Wav packs CPU codec inputs before device transfer. Optional B2/B4 graph
|
||||
buckets batch compatible requests without sharing their per-request state.
|
||||
- Native output materialization runs in a bounded worker and drains before
|
||||
closing the data plane. The existing control-only shortcut avoids launching
|
||||
model work for a step that has only lifecycle events.
|
||||
|
||||
MTP prefix re-prefill remains disabled in the high-concurrency V2 profile.
|
||||
Adding more graph shapes increases compilation cost and has not established a
|
||||
stable end-to-end gain for this extracted version.
|
||||
|
||||
### Decoder batches and playback
|
||||
|
||||
Changing `decode_cudagraph_batch_sizes` selects the captured batch buckets.
|
||||
Keep `decode_batch_max_size` consistent with the intended maximum too; the
|
||||
stateful graph path currently groups according to the captured buckets, while
|
||||
the stateless path also uses the explicit maximum.
|
||||
|
||||
B4 remains experimental. First-packet latency alone does not establish
|
||||
uninterrupted playback. Validate inter-chunk arrival times, buffering, WER and
|
||||
speaker similarity before adopting either batching preset for a production
|
||||
workload. Floating-point decoder outputs can differ across batch sizes; this PR
|
||||
does not claim bitwise or quality equivalence.
|
||||
|
||||
### Optional MPS deployment
|
||||
|
||||
NVIDIA MPS is an optional operator setting for colocated CUDA processes, not a
|
||||
YAML option or a library default. This PR does not establish a throughput or
|
||||
first-packet latency benefit from MPS. Measure the exact deployment with and
|
||||
without MPS before enabling it.
|
||||
|
||||
Use only assigned GPUs and an independent MPS pipe directory. A private MPS
|
||||
server does not provide exclusive GPU ownership or MIG isolation. For a
|
||||
single-GPU deployment, explicitly place both stages on that GPU; the supplied
|
||||
high-concurrency profile places its two stages on different GPUs by default.
|
||||
@@ -15,7 +15,8 @@ Canonical layout (prefer these paths for new changes):
|
||||
│ │ ├── skip_ci.py # skip-ci decision (docs / skip-mark / CI YAML paths)
|
||||
│ │ ├── upload_pipeline.py # Bootstrap + test-pipeline uploader (CUDA/NPU)
|
||||
│ │ └── resolve_skip_ci.sh # Shell helpers for AMD/Intel bootstrap
|
||||
│ └── ci_mirror_hardwares.yml # CUDA uploader presets (referenced by name only)
|
||||
│ ├── ci_mirror_hardwares.yml # CUDA/NPU hardware presets (referenced by name only)
|
||||
│ └── ci_source_file_dependencies.yml # L2–L5 path-prefix presets (referenced by name only)
|
||||
├── cuda/ # Primary NVIDIA CUDA CI
|
||||
│ ├── pipeline.yml # Bootstrap entry (hook upload)
|
||||
│ ├── bootstrap-upload-steps.yml # Bootstrap child steps (upload_pipeline --upload)
|
||||
@@ -34,6 +35,7 @@ Canonical layout (prefer these paths for new changes):
|
||||
├── amd/
|
||||
│ ├── test-amd-ready.yml # L2 job definitions (template input)
|
||||
│ ├── test-amd-merge.yml # L3 job definitions
|
||||
│ ├── test-amd-nightly.yml # Nightly full-model jobs (experimental)
|
||||
│ ├── test-template-amd-omni.j2 # Renders final pipeline.yaml
|
||||
│ └── scripts/
|
||||
│ ├── bootstrap-amd-omni.sh # Entry: skip-ci → Jinja → upload
|
||||
@@ -63,7 +65,7 @@ Canonical layout (prefer these paths for new changes):
|
||||
| -------- | ---------------- | -------------- | ---------------- | -------------------- |
|
||||
| **CUDA** | `cuda/pipeline.yml` | `test-ready.yml`, `test-merge.yml`, `test-nightly.yml`, `test-weekly.yml` | `upload_pipeline.py --upload` (expands uploader-only keys) | omit `mirror_hardwares` (from `-m`) / preset string + `MIRROR_HW` |
|
||||
| **NPU** | `npu/pipeline-npu.yml` | `test-npu-ready.yml`, `test-npu-nightly.yml` | `upload_pipeline.py --upload` | `mirror_hardwares: a2b3_npu_1` / `a2b3_npu_4` / `a3_npu_2` |
|
||||
| **AMD** | `amd/scripts/bootstrap-amd-omni.sh` | `test-amd-ready.yml`, `test-amd-merge.yml` | Jinja (`test-template-amd-omni.j2`) → `pipeline upload` | `agent_pool` + `mirror_hardwares: [amdproduction]` (array, template filter) |
|
||||
| **AMD** | `amd/scripts/bootstrap-amd-omni.sh` | `test-amd-ready.yml`, `test-amd-merge.yml`, `test-amd-nightly.yml` | Jinja (`test-template-amd-omni.j2`) → `pipeline upload` | `agent_pool` + `mirror_hardwares: [amdproduction]` (array, template filter) |
|
||||
| **Intel** | `intel/scripts/bootstrap-intel-omni.sh` | `intel/pipeline-intel.yml` (steps inline) | Direct `pipeline upload` | Inline `agents.queue` on each step |
|
||||
|
||||
## Platform configuration style
|
||||
@@ -108,19 +110,21 @@ Canonical layout (prefer these paths for new changes):
|
||||
|
||||
Inferred form: unset/`empty` `MIRROR_HW` matches `H100` or `L4` in `-m` (both present → H100); no match skips the step. `MIRROR_HW=b200` must appear in `-m` or the step is skipped. The uploader does **not** rewrite pytest `-m`; B200 collection comes from the YAML expression and from tests that declare `B200` in `hardware_test` / `hardware_marks` (for example `res={"cuda": ["H100", "B200"]}`). `MIRROR_HW` must be empty or `b200` (case-insensitive); unknown values (for example `b20o`) **fail the upload**. A CUDA preset string such as `mirror_hardwares: h100_4` is skipped when `MIRROR_HW=b200`. Unset `MIRROR_HW` keeps string presets. NPU presets ignore `MIRROR_HW`. A `mirror_hardwares` name that is not a key in `ci_mirror_hardwares.yml` **fails the upload**.
|
||||
|
||||
**Path filter in YAML:** `source_file_dependencies: <preset>` (string, or a list of preset names)—preset names in [`common/ci_source_file_dependencies.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/ci_source_file_dependencies.yml). Required on **E2E Test** leaf jobs in L2/L3, and on L4/L5 leaf jobs that should be diff-gated on PR labels; see [Step filtering](#step-filtering).
|
||||
|
||||
**Conventions**
|
||||
|
||||
- **`depends_on`:** leaf jobs depend on `upload-ready-pipeline`, `upload-merge-pipeline`, etc.
|
||||
- **`group` / `label`:** `:card_index_dividers:` groups; labels like `Diffusion · Qwen Image Test`.
|
||||
- **`commands`:** `timeout … pytest …` with markers and `--run-level` for the pipeline level.
|
||||
- **`source_file_dependencies`:** required on **E2E Test** leaf jobs in L2/L3; see [Step filtering](#step-filtering).
|
||||
- **`source_file_dependencies`:** required on **E2E Test** leaf jobs in L2/L3, and on L4/L5 leaf jobs that should be diff-gated on PR labels; name a job key from `ci_source_file_dependencies.yml` (see [Step filtering](#step-filtering)). The key must list every script the step runs; `commands` is not scanned.
|
||||
|
||||
**Adding a job**
|
||||
|
||||
1. Pick the level file (`test-ready.yml` / `test-merge.yml` / `test-nightly.yml` / `test-weekly.yml`).
|
||||
2. Add a step under the right **group** (usually **E2E Test** for model pytest).
|
||||
3. Set `label`, `commands`, `mirror_hardwares`, `depends_on: upload-ready-pipeline` (or `upload-merge-pipeline` / `upload-nightly-pipeline` / `upload-weekly-pipeline`).
|
||||
4. For L2/L3 E2E, add `source_file_dependencies` (pytest + model + deploy YAML prefixes).
|
||||
3. Set `label`, `commands`, `mirror_hardwares`, `depends_on: upload-<level>-pipeline`.
|
||||
4. For L2–L5 jobs that should be diff-gated, add a job key in `common/ci_source_file_dependencies.yml` and set `source_file_dependencies: <key>` on the step. YAML anchors hold model business-code paths only. The job key (`{model}_function` / `_perf` / `_accuracy` / `_reliability` / `_doc` / `_cov`) aliases that anchor and lists the pytest / example / benchmark scripts the step actually runs. The uploader does **not** parse `commands` for path filters—omitting a script from the key means editing that script will not select the job.
|
||||
5. Dry-run:
|
||||
|
||||
```bash
|
||||
@@ -144,6 +148,8 @@ Canonical layout (prefer these paths for new changes):
|
||||
- A5: `a5_npu_2`, `a5_npu_4`, `a5_npu_8`, `a5_npu_16`
|
||||
- 310P: `310p_npu_1`, `310p_npu_2`, `310p_npu_4`
|
||||
|
||||
**Path filter in YAML:** same `source_file_dependencies` presets as CUDA, on **L4** leaf jobs in `test-npu-nightly.yml`. Filtering applies on the PR `nightly-test` label; `main` + `NIGHTLY=1` keeps every job. If no job-key prefix matches, a change to the pipeline YAML being uploaded, or to any path under `source_filter_fallback`, keeps every job in that upload. See [Step filtering](#step-filtering).
|
||||
|
||||
**Conventions**
|
||||
|
||||
- **`depends_on: upload-ready-pipeline`** (or `upload-nightly-pipeline`) ties jobs to bootstrap upload keys.
|
||||
@@ -153,7 +159,7 @@ Canonical layout (prefer these paths for new changes):
|
||||
|
||||
1. Edit `test-npu-ready.yml` (L2) or `test-npu-nightly.yml` (L4).
|
||||
2. Add a step with `mirror_hardwares` (add a new preset in `ci_mirror_hardwares.yml` first if needed).
|
||||
3. Set `commands` to your pytest file and markers.
|
||||
3. Set `commands` to your pytest file and markers. For L4, set `source_file_dependencies` to a job key from `ci_source_file_dependencies.yml` that aliases the model anchor and lists that pytest file.
|
||||
4. Dry-run:
|
||||
|
||||
```bash
|
||||
@@ -166,9 +172,9 @@ Canonical layout (prefer these paths for new changes):
|
||||
|
||||
**Bootstrap:** [`amd/scripts/bootstrap-amd-omni.sh`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/scripts/bootstrap-amd-omni.sh)—skip-ci, diff filtering, Jinja render, then `buildkite-agent pipeline upload`.
|
||||
|
||||
**Test YAML (data):** `amd/test-amd-ready.yml` (L2 / `ready`), `test-amd-merge.yml` (L3 / `merge-test` and `main`). On a PR carrying both labels, AMD combines both suites behind one image build. PR builds without either tier label retain the legacy L2 fallback for compatibility with the `amd-test` trigger. `DEBUG_TEST_YAML` remains an explicit override. AMD has no L4 file yet, so `nightly-test` does not select an additional suite.
|
||||
**Test YAML (data):** `amd/test-amd-ready.yml` (L2 / `ready`), `test-amd-merge.yml` (L3 / `merge-test` and ordinary `main`), and experimental `test-amd-nightly.yml` (`full_model` / `nightly-test` or `main` with `NIGHTLY=1`). Multiple PR tier labels combine their suites behind one image build. PR builds that reach the bootstrap without a tier label retain the legacy ready-suite fallback. `DEBUG_TEST_YAML` accepts `ready`, `merge`, and `nightly` as an explicit override. During nightly burn-in every AMD nightly leaf is non-blocking.
|
||||
|
||||
**Trigger boundary:** AMD label handling has two layers. First, the external `vllm-omni-amd-ci` Buildkite pipeline condition decides whether a GitHub label event creates a build. Only after that build starts does this repository's bootstrap inspect all current PR labels and select L2, L3, or both. Repository-side selection therefore cannot make a `merge-test` event start AMD CI by itself. The external condition must admit both `ready` and `merge-test`, while preserving the legacy `amd-test` trigger and non-PR `main` / scheduled builds. For PR builds without a `DEBUG_TEST_YAML` override, an unreadable label set fails the bootstrap instead of silently selecting the wrong tier. Debug overrides skip label lookup and are validated directly; invalid values, including whitespace-only values, fail the bootstrap. Keep `nightly-test` out of the AMD condition until an AMD L4 suite exists; otherwise it would start a build without providing nightly coverage.
|
||||
**Trigger boundary:** AMD label handling has two layers. First, the external `vllm-omni-amd-ci` Buildkite pipeline condition decides whether a GitHub label event creates a build. Only after that build starts does this repository's bootstrap inspect all current PR labels and select the ready, merge, nightly, or combined suites. Repository-side selection therefore cannot start AMD CI by itself. AMD-only nightly validation uses both `amd-test` and `nightly-test`: the external condition should accept an event for either label only when the complete PR label set contains both. `amd-test` admits the AMD pipeline, while `nightly-test` makes the bootstrap select `NIGHTLY_TESTS:test-amd-nightly.yml`. Existing `ready` / `merge-test` behavior and non-PR `main` / scheduled builds remain separate. For PR builds without a `DEBUG_TEST_YAML` override, an unreadable label set fails the bootstrap instead of silently selecting the wrong tier. Debug overrides skip label lookup and are validated directly; invalid values, including whitespace-only values, fail the bootstrap. Until the paired-label condition is configured, use an authorized manual Buildkite build with `DEBUG_TEST_YAML=nightly` and verify that the bootstrap reports only the nightly suite.
|
||||
|
||||
**Rendering:** [`test-template-amd-omni.j2`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/test-template-amd-omni.j2) wraps data steps with `amd-build` image build and `amd_<agent_pool>` queues. Do **not** hand-edit generated `pipeline.yaml`.
|
||||
|
||||
@@ -184,7 +190,7 @@ Canonical layout (prefer these paths for new changes):
|
||||
|
||||
**Adding a job**
|
||||
|
||||
1. Edit `test-amd-ready.yml` or `test-amd-merge.yml`.
|
||||
1. Edit `test-amd-ready.yml`, `test-amd-merge.yml`, or `test-amd-nightly.yml`.
|
||||
2. Copy a neighboring block: `label`, `agent_pool`, `mirror_hardwares`, `commands`, optional `grade`.
|
||||
3. Regenerate via bootstrap / Jinja; update `skip_ci.py` if you add a new YAML path.
|
||||
|
||||
@@ -213,7 +219,7 @@ PR diffs drive **two independent skip layers**. Both read changed files from git
|
||||
| Layer | Script | When | What is skipped | Where you configure |
|
||||
| ----- | ------ | ---- | --------------- | ------------------- |
|
||||
| **Bootstrap** | [`skip_ci.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/skip_ci.py) + [`upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py) | Before child test pipelines upload (`cuda/pipeline.yml`, `npu/pipeline-npu.yml`, AMD/Intel bootstraps) | Entire default CI, or whole L2/L3 upload for a platform | Whitelists in `skip_ci.py`; bootstrap `if` injected by step `key` in `upload_pipeline.py` |
|
||||
| **Step filter** | [`upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py) | While uploading CUDA L2/L3 YAML | Individual Buildkite steps inside `test-ready.yml` / `test-merge.yml` | `source_file_dependencies` on each step or group |
|
||||
| **Step filter** | [`upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py) | While uploading CUDA L2–L5 YAML | Individual Buildkite steps inside `test-ready.yml` / `test-merge.yml` / `test-nightly.yml` / `test-weekly.yml` | `source_file_dependencies` key on each step (presets in `ci_source_file_dependencies.yml`) |
|
||||
|
||||
**Changed files** (both layers):
|
||||
|
||||
@@ -240,7 +246,7 @@ Label triggers (`ready`, `merge-test`) are unchanged—diff-aware logic only red
|
||||
| Non-qualifying skip-mark and no CI YAML | normal CI | all on |
|
||||
| Diff unavailable | normal CI | all on |
|
||||
|
||||
**`skip_all` exceptions (CUDA/NPU bootstrap only):** PR labels (`nightly-test`, `merge-test`, `weekly-test`, `ready`, …) do **not** revive jobs under `skip_all`. On `main`, scheduled `NIGHTLY=1` still builds the image and uploads L4; `WEEKLY=1` or `NON_CRITICAL=1` still builds the image and uploads L5 (CUDA). `WEEKLY=1` on `main` also uploads L2/L3 with `--e2e`.
|
||||
**`skip_all` exceptions:** For CUDA/NPU, PR labels (`nightly-test`, `merge-test`, `weekly-test`, `ready`, …) do **not** revive jobs under `skip_all`; on `main`, scheduled `NIGHTLY=1` still builds the image and uploads L4, while `WEEKLY=1` or `NON_CRITICAL=1` still builds the image and uploads L5 (CUDA). `WEEKLY=1` on `main` also uploads L2/L3 with `--e2e`. AMD applies `skip_ci.py gate` only to selected ready/merge specs, so an explicitly selected AMD nightly spec (`nightly-test`, `NIGHTLY=1`, or a nightly debug override) remains runnable even for a docs-only diff.
|
||||
|
||||
**Yaml-gated nightly/weekly:** L4/L5 upload steps keep their normal label / `NIGHTLY` / `WEEKLY` / `NON_CRITICAL` conditions (for example PR `nightly-test` still uploads nightly). Only L2/L3 upload steps are matrix-gated.
|
||||
|
||||
@@ -257,6 +263,7 @@ Register new files in `L2_YAML_FILES`, `L3_YAML_FILES`, or `L45_YAML_FILES` in `
|
||||
| L3 | `.buildkite/cuda/test-merge.yml` | cuda |
|
||||
| L3 | `.buildkite/amd/test-amd-merge.yml` | amd |
|
||||
| L4/L5 | `.buildkite/cuda/test-nightly.yml`, `test-weekly.yml` | cuda |
|
||||
| L4/L5 | `.buildkite/amd/test-amd-nightly.yml` | amd |
|
||||
| L4/L5 | `.buildkite/npu/test-npu-nightly.yml` | npu |
|
||||
|
||||
##### Yaml-gated category branches
|
||||
@@ -289,7 +296,7 @@ Docs/skip-mark mixed with any row below follows the same matrix.
|
||||
| CUDA ready + AMD merge | G4 | `cuda/l2` + `amd/l3` |
|
||||
| CUDA ready + AMD ready | G2 | `cuda/l2` + `amd/l2` |
|
||||
| CUDA merge + AMD merge | G3 | `cuda/l3` + `amd/l3` |
|
||||
| Only nightly / weekly / npu-nightly YAML | G1 | (none) |
|
||||
| Only nightly / weekly / amd-nightly / npu-nightly YAML | G1 | (none) |
|
||||
| CUDA ready + nightly | G5 | `cuda/l2` |
|
||||
| CUDA merge + nightly | G6 | `cuda/l3` |
|
||||
| CUDA ready + merge + nightly | G7 | `cuda/l2` + `cuda/l3` |
|
||||
@@ -306,40 +313,38 @@ Unit coverage: `tests/buildkite/test_skip_ci.py`.
|
||||
|
||||
#### Step filtering {#step-filtering}
|
||||
|
||||
CUDA **L2** (`.buildkite/cuda/test-ready.yml`) and **L3** (`.buildkite/cuda/test-merge.yml`) only. Bootstrap upload entry: `upload_pipeline.py --upload .buildkite/cuda/bootstrap-upload-steps.yml`.
|
||||
CUDA **L2** (`.buildkite/cuda/test-ready.yml`), **L3** (`.buildkite/cuda/test-merge.yml`), **L4** (`.buildkite/cuda/test-nightly.yml`), **L5** (`.buildkite/cuda/test-weekly.yml`), and NPU **L4** (`.buildkite/npu/test-npu-nightly.yml`). Bootstrap upload entry: `upload_pipeline.py --upload .buildkite/cuda/bootstrap-upload-steps.yml` (or `npu/bootstrap-upload-steps.yml`).
|
||||
|
||||
**When filtering applies:** **PR label** uploads (`ready`, `merge-test`, `nightly-test`, `weekly-test`) and **post-merge `main` L3** (`test-merge.yml`). Scheduled **`main` + env** uploads that pass `--all` (`NIGHTLY=1`, `WEEKLY=1` / `NON_CRITICAL=1`) or `--e2e` (`WEEKLY=1` on L2/L3) keep every selected job and still strip the key. If **no** job-key prefix matches the diff, a change to the pipeline YAML being uploaded, or any path under the `source_filter_fallback` key in `ci_source_file_dependencies.yml`, keeps every job in that upload. `source_filter_fallback` is not a job key — do not attach it to a step. When any job-key prefix already matches, normal filtering wins (the fallback does not expand the selection).
|
||||
|
||||
**Uploader-only keys** — removed before Buildkite sees the YAML; never used at runtime on agents:
|
||||
|
||||
| Key | Purpose |
|
||||
| --- | ------- |
|
||||
| `source_file_dependencies` | List of path **prefixes**. If any changed file equals a prefix or starts with `prefix/`, keep the step (or group); otherwise omit it. |
|
||||
| `source_file_dependencies` | Named preset (or list of presets) from [`ci_source_file_dependencies.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/ci_source_file_dependencies.yml). The uploader expands only that key; it does **not** scan `commands`. If any changed file equals a listed prefix or starts with `prefix/`, keep the step (or group); otherwise omit it. Inline path-prefix lists are still accepted. |
|
||||
| `mirror_hardwares` | Expand to `agents` + `plugins` (+ optional `image`) from `ci_mirror_hardwares.yml`. |
|
||||
|
||||
### Policy
|
||||
|
||||
- **Always uploaded** (no key): groups outside **E2E Test**—Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
|
||||
- **Diff-gated**: every **E2E Test** leaf job. List the smallest prefix set per step—pytest file(s), model code under `vllm_omni/model_executor/models/` or `vllm_omni/diffusion/models/`, plus `stage_input_processors/` and `vllm_omni/deploy/*.yaml` when applicable. A **group** may define the key instead; the whole group drops if no prefix matches.
|
||||
- **Always uploaded** (no key): L2/L3 groups outside **E2E Test**—Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
|
||||
- **Diff-gated**: every **E2E Test** leaf job in L2/L3, L4 nightly leaf jobs (CUDA and NPU), and L5 weekly jobs whose `if` includes the `weekly-test` label (Reliability / Perf). Jobs gated only by `WEEKLY=1` or `NON_CRITICAL=1` have no `source_file_dependencies`. Registry layout: YAML anchors are model business code only (`vllm_omni/model_executor/models/`, `vllm_omni/diffusion/models/`, plus `stage_input_processors/` and `vllm_omni/deploy/*.yaml` when applicable). Job keys alias the anchor and **also** list the scripts that job runs (pytest files, `run_cov_split.sh --offline` / `--online` targets, example README runners). Feature keys stay cross-cutting. A **group** may define the key instead; the whole group drops if no prefix matches.
|
||||
|
||||
### YAML examples
|
||||
|
||||
```yaml
|
||||
- label: "Diffusion · Qwen Image Test"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen_image.py
|
||||
- vllm_omni/diffusion/models/qwen_image/
|
||||
source_file_dependencies: diffusion_qwen_image_function
|
||||
# diffusion_qwen_image_function: *qwen_image + tests/e2e/online_serving/test_qwen_image.py
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model' ...
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "TTS · Qwen3-TTS CustomVoice Test"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen3_tts_customvoice.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
|
||||
- vllm_omni/deploy/qwen3_tts.yaml
|
||||
source_file_dependencies: tts_qwen3_tts_function
|
||||
# tts_qwen3_tts_function: *qwen3_tts + tests/e2e/online_serving/test_qwen3_tts_customvoice.py
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py ...
|
||||
mirror_hardwares: l4_4
|
||||
mirror_hardwares: l4_1
|
||||
```
|
||||
|
||||
### Local dry-run
|
||||
@@ -443,15 +448,16 @@ plugin, which provides the agent. `l4_*` jobs run on the EKS `l4-k8s` queue
|
||||
still needs `mount-buildkite-agent: true`.
|
||||
|
||||
List both `run_cov_split.sh` and `pyproject.toml` in every opted-in job's
|
||||
`source_file_dependencies` — both change what the job measures, so without them a
|
||||
`source_file_dependencies` preset — both change what the job measures, so without them a
|
||||
change there is filtered out of normal PR builds and only surfaces in a later
|
||||
nightly. `tests/buildkite/test_upload_pipeline.py` covers the filter behavior with
|
||||
a synthetic job (it does not pin real merge labels).
|
||||
Editing only the surrounding CI YAML still does not schedule them, so a PR that
|
||||
touches just the wiring needs a full E2E run (or the commands run on a GPU host)
|
||||
to produce artifacts. When checking a new model's
|
||||
artifacts, compare `lines-covered` between the online and offline XML rather than
|
||||
just confirming both files exist.
|
||||
If no job-key prefix matches, editing the pipeline YAML being uploaded, or any
|
||||
path under the `source_filter_fallback` registry key, keeps every job in that
|
||||
upload (including coverage jobs) so the wiring can be validated before merge.
|
||||
When a job-key prefix already matches, bypass does not expand the selection. A
|
||||
change to a different pipeline YAML still does not schedule them. When checking a new model's artifacts, compare `lines-covered`
|
||||
between the online and offline XML rather than just confirming both files exist.
|
||||
|
||||
### Validation checklist
|
||||
|
||||
|
||||
@@ -182,6 +182,8 @@ CUDA **L4** (`full_model`) jobs are defined once in [`.buildkite/cuda/test-night
|
||||
- **Nightly:** scheduled `main` builds with `NIGHTLY=1` (and PR labels such as `nightly-test` / `omni-test`). Default fleet is typically H100.
|
||||
- **Pre-release:** the same L4 suite runs again on extra GPU SKUs before a release (for example B200 via `MIRROR_HW=b200`), so `full_model` coverage is not limited to the nightly machine type.
|
||||
|
||||
AMD has an experimental, non-blocking `full_model` nightly vertical slice in [`.buildkite/amd/test-amd-nightly.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/test-amd-nightly.yml). It selects ROCm Qwen3-Omni function-expansion, accuracy, documentation-example, and AITER-on smoke coverage on the MI300 pool. AMD-only PR validation uses `amd-test` to admit the external AMD pipeline and `nightly-test` to select this suite; scheduled `main` builds select it with `NIGHTLY=1`. Keep it non-blocking while the lane accumulates stability and completion-time evidence.
|
||||
|
||||
How count-form `mirror_hardwares` and `MIRROR_HW` pick a SKU: [CI Settings](./ci_settings.md).
|
||||
|
||||
### L2 / L3 diff-aware CI (CUDA)
|
||||
|
||||
@@ -131,13 +131,13 @@ for the validated batching, attention, and quantization combinations.
|
||||
|
||||
### MiniMax-H3: conditioning, denoising, and decode
|
||||
|
||||
MiniMax-H3 illustrates logical component and phase boundaries between text
|
||||
encoding, task-specific denoising, and media decoding:
|
||||
`Text Encoder → DiT → VAE Decoder`. The current registered topology co-locates
|
||||
these components in one diffusion stage. These boundaries are useful for
|
||||
profiling and offload; independent placement would require a future
|
||||
multi-stage `PipelineConfig`. The stage mapping follows the implementation and
|
||||
serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3)
|
||||
MiniMax-H3 combines text and reference-media encoding, task-specific denoising,
|
||||
and media decoding. Standard serving co-locates these components in one diffusion
|
||||
stage. The opt-in
|
||||
[disaggregated topology](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3-Disaggregated.md)
|
||||
runs Qwen and the video/audio VAE encoders in Stage 0, then sends their conditioning
|
||||
to a Stage 1 containing the DiTs and VAE decoders. The stage mapping follows the
|
||||
implementation and serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3)
|
||||
notes that a detailed technical report is forthcoming.
|
||||
|
||||
#### Model architecture by component
|
||||
@@ -149,16 +149,15 @@ notes that a detailed technical report is forthcoming.
|
||||
| **VAE Decoder** | Video VAE and audio VAE decode the generated latents and assemble the synchronized output | Decoded video frames and stereo waveform, with FPS and audio sample rate metadata |
|
||||
|
||||
Both DiT partitions remain behind one task-selection point; `task` chooses
|
||||
which partition runs. In the current implementation, the three logical
|
||||
components execute inside one diffusion stage. This is different from
|
||||
Qwen3-Omni's three independently scheduled stages, even though both models
|
||||
produce synchronized audio and video-like outputs.
|
||||
which partition runs.
|
||||
|
||||
#### Representative component execution
|
||||
|
||||
The following table describes the standard single-stage deployment.
|
||||
|
||||
| Logical component | Batching | Attention and execution | Parallelism | Quantization |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Text Encoder** | The current co-located pipeline follows the diffusion request scheduler; independent prompt/reference batching would require a separate multi-stage topology | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks; independent placement is not part of the current topology | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder |
|
||||
| **Text Encoder** | Follows the diffusion request scheduler | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder |
|
||||
| **DiT** | The current H3 implementation executes one generation request per diffusion batch; use concurrency for service-level throughput | cuDNN attention for the validated two-GPU consumer profile; TRTLLM attention or FlashAttention-4 on supported Blackwell profiles | TP2 plus text-encoder TP and Ulysses/VAE parallel groups on multi-GPU profiles; DLO/CPU offload trade memory for transfer time | Online FP8 applies to eligible DiT linears and is incompatible with layerwise offload |
|
||||
| **VAE Decoder** | Decode follows each generation request; tile/patch work can be distributed even when denoising is not batched | VAE decode kernels rather than DiT attention | VAE patch parallelism and native tiled decode within the diffusion stage | BF16/FP32 baseline; the documented H3 FP8 path leaves both VAEs unchanged |
|
||||
|
||||
@@ -324,7 +323,7 @@ inputs, fields, and ownership rules are described below.
|
||||
flowchart TB
|
||||
layer1["Layer 1 · Authoring inputs<br/>PipelineConfig + DeployConfig"]
|
||||
layer2["Layer 2 · Production resolution boundary<br/>resolve_omni_config()<br/>(StageConfigFactory + VllmOmniConfig internally)"]
|
||||
layer3["Layer 3 · Startup hand-off<br/>OmniConfigResolution<br/>PipelineConfig + temporary OmegaConf stage bridge"]
|
||||
layer3["Layer 3 · Startup hand-off<br/>OmniConfigResolution<br/>PipelineConfig + typed stage configs"]
|
||||
layer4["Layer 4 · Runtime launch planning<br/>StageRuntime"]
|
||||
layer5["Layer 5 · Engine materialization<br/>VllmConfig / OmniDiffusionConfig"]
|
||||
|
||||
@@ -336,10 +335,10 @@ The production resolution boundary is `resolve_omni_config()` in
|
||||
It delegates typed construction to `StageConfigFactory.create_from_model()` and
|
||||
`VllmOmniConfig.from_pipeline_config()`, then returns an
|
||||
`OmniConfigResolution` consumed by both `AsyncOmniEngine` and headless startup.
|
||||
Until `StageRuntime` consumes typed stage configs directly, this envelope carries
|
||||
the effective `PipelineConfig` alongside OmegaConf-compatible `stage_configs` as
|
||||
a temporary runtime bridge; both views describe the same resolved topology,
|
||||
including injected stages.
|
||||
This envelope carries the effective `PipelineConfig` alongside typed
|
||||
`stage_configs`, which `StageRuntime` consumes directly. Both views describe the
|
||||
same resolved topology, including injected stages. Backend arguments are
|
||||
projected from these typed stages when each engine is initialized.
|
||||
|
||||
The legacy `stage_args` YAML path has been removed. Model topology now resolves
|
||||
through `PipelineConfig`, with runtime overrides supplied by `DeployConfig`.
|
||||
@@ -361,7 +360,7 @@ The important ownership rules are:
|
||||
3. CLI and Python overrides are applied at the resolution boundary, with
|
||||
per-stage overrides taking precedence over global values where supported.
|
||||
4. `OmniConfigResolution` is the sole startup hand-off. Its `pipeline_config`
|
||||
and temporary `stage_configs` compatibility view must describe the same
|
||||
and typed `stage_configs` must describe the same
|
||||
topology.
|
||||
5. `StageRuntime` owns launch planning and replica lifecycle. `ReplicaInitPlan`
|
||||
is runtime-private state, not a user configuration object.
|
||||
|
||||
@@ -76,6 +76,34 @@ preserve common `self` or `cross` policies. A model also decides whether its
|
||||
path is compatible with automatic TRTLLM selection, because only the model
|
||||
knows whether masking and packing satisfy the kernel contract.
|
||||
|
||||
### Model-owned implementation specializations
|
||||
|
||||
Use `Attention(..., impl_overrides={backend_name: implementation_class})`
|
||||
when a model needs different computation within an existing backend contract.
|
||||
The model defines the implementation in its own directory and passes it when
|
||||
constructing the layer:
|
||||
|
||||
```python
|
||||
# MyModelVSAImpl is a model-owned subclass of FastVideoVSAImpl.
|
||||
self.attention = Attention(
|
||||
num_heads=num_heads,
|
||||
head_size=head_size,
|
||||
softmax_scale=head_size**-0.5,
|
||||
causal=False,
|
||||
role="self",
|
||||
impl_overrides={"FASTVIDEO_VSA": MyModelVSAImpl},
|
||||
)
|
||||
```
|
||||
|
||||
The override is applied after role/platform selection, keyed by the selected
|
||||
backend's `get_name()`. It must subclass the **selected implementation**;
|
||||
incompatible platform implementations raise `TypeError`. Unselected overrides
|
||||
have no effect. Backend capabilities, constructor options, and shared
|
||||
parallel dispatch remain unchanged and must be honored by the specialization.
|
||||
|
||||
Use a new backend for a new provider/capability contract. `custom_attention`
|
||||
instead owns communication and requires `skip_sequence_parallel=True`.
|
||||
|
||||
## Adding or changing a backend
|
||||
|
||||
An implementation change is complete only when it:
|
||||
@@ -91,8 +119,9 @@ An implementation change is complete only when it:
|
||||
|
||||
## FastVideo VSA model metadata
|
||||
|
||||
FastVideo VSA support is scoped to `FastVideo/FastWan2.2-TI2V-5B-Diffusers`, whose text-to-video and image-to-video modes both use `Wan22Pipeline`. It operates on a flattened DiT sequence but partitions tokens in
|
||||
the original latent video grid. Wan integrations therefore attach the
|
||||
FastWan's text-to-video and image-to-video modes both use `Wan22Pipeline`.
|
||||
It operates on a flattened DiT sequence but partitions tokens in the original
|
||||
latent video grid. Wan integrations therefore attach the
|
||||
post-patch `(T, H, W)` shape as `vsa_dit_seq_shape` attention metadata. The
|
||||
backend validates that `T * H * W` equals the sequence length, derives the
|
||||
runtime block count, and selects the configured top-k key/value blocks for
|
||||
@@ -104,3 +133,8 @@ the checkpoint contains those weights. Checkpoints without the projection do
|
||||
not allocate or execute it. When top-k selects every block, native checkpoints
|
||||
route to SDPA, while FastVideo DMD checkpoints preserve the VSA all-block path
|
||||
to retain checkpoint semantics.
|
||||
|
||||
MiniMax-H3 supplies `MiniMaxH3VSAImpl` through `impl_overrides`.
|
||||
`models/minimax_h3/attention/fastvideo_h3.py` owns its prefix layout, video tiling, and
|
||||
learned compression gate. Pooling, block-map construction, and tile64 provider
|
||||
calls are shared through `attention/ops/block_sparse.py`.
|
||||
@@ -9,6 +9,7 @@ Backend-specific setup lives in separate docs:
|
||||
- [MooncakeStoreConnector](omni_connectors/mooncake_store_connector.md)
|
||||
- [MooncakeTransferEngineConnector](omni_connectors/mooncake_transfer_engine_connector.md)
|
||||
- [MoriTransferEngineConnector](omni_connectors/mori_transfer_engine_connector.md)
|
||||
- [NixlConnector](omni_connectors/nixl_connector.md)
|
||||
- [YuanrongConnector](omni_connectors/yuanrong_connector.md)
|
||||
- [YuanrongTransferEngineConnector](omni_connectors/yuanrong_transfer_engine_connector.md)
|
||||
|
||||
@@ -25,6 +26,7 @@ Current connectors operate in D2H2D (device to host to device) mode.
|
||||
| Multi node (Mooncake Store) | MooncakeStoreConnector | TCP-based, requires Mooncake Master + metadata server. |
|
||||
| Multi node (Mooncake RDMA) | MooncakeTransferEngineConnector | RDMA/TCP direct transfer with managed memory pool. Fastest. |
|
||||
| Multi node (Mori RDMA) | MoriTransferEngineConnector | RDMA direct transfer via Mori IOEngine. |
|
||||
| Multi node (NIXL) | NixlConnector | NIXL `READ` over UCX; the only backend covering Intel XPU. |
|
||||
| Multi node (Yuanrong) | YuanrongConnector | Requires Yuanrong Datasystem + etcd. |
|
||||
| Ascend NPU P2P (Yuanrong TE) | YuanrongTransferEngineConnector | Uses Yuanrong TransferEngine directly. Configure NPU device IPv4 and `memory_pool_device: "npu"`. |
|
||||
|
||||
|
||||
@@ -42,6 +42,11 @@ V1 does not include:
|
||||
|
||||
## Motivation and use cases
|
||||
|
||||
For investigating CPU memory retained by dependency tensor materialization,
|
||||
see the [standalone safetensors diagnostic](https://github.com/vllm-project/vllm-omni/blob/main/benchmarks/host_weight_runtime/README.md).
|
||||
It distinguishes repeated dependency calls from reuse of cached views and does
|
||||
not establish per-request HWR leakage.
|
||||
|
||||
Model loading can create the same final host representation repeatedly. This
|
||||
is especially expensive when loading performs checkpoint decoding, tensor
|
||||
renaming, TP slicing, quantization, packing, or scale construction before GPU
|
||||
@@ -313,8 +318,15 @@ One process per exact identity owns a build; other workers wait and then acquire
|
||||
leases for the published artifact. Publication is invisible until all payloads
|
||||
and metadata are validated, hashed, fsynced, and atomically renamed.
|
||||
|
||||
`coordination_timeout_seconds` bounds filesystem lock acquisition. It does not
|
||||
cancel synchronous validation, a producer that has already started, or atomic
|
||||
`coordination_timeout_seconds` bounds domain-initialization and lookup/build
|
||||
lock acquisition. Store construction and each later resolution or publication
|
||||
operation have separate budgets from the same wait policy, rather than one
|
||||
end-to-end startup deadline. A domain-init timeout follows the retryable domain
|
||||
failure policy below. After contention ends, a fresh runtime construction can
|
||||
retry initialization; the timed-out runtime retains its original failure.
|
||||
|
||||
The coordination budget does not cancel filesystem I/O, synchronous validation,
|
||||
a producer that has already started, or atomic
|
||||
publication. A hung in-process producer therefore blocks its owning process and
|
||||
must be handled by external process supervision. Enforceable producer
|
||||
cancellation requires a future process-isolated producer contract.
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
# Model-local KV caches
|
||||
|
||||
Some models keep their attention KV as HuggingFace `transformers` cache objects
|
||||
instead of using the engine's paged KV manager. Each such model declares what it
|
||||
holds by implementing `model_local_kv_specs()`, and
|
||||
`OmniGPUModelRunner.load_model` sums and logs the result after load.
|
||||
|
||||
## What is and is not already accounted for
|
||||
|
||||
This distinction decides what the protocol is for, and an earlier revision of
|
||||
this page got it wrong by claiming none of this memory was accounted for.
|
||||
|
||||
CUDA-graph capture happens *during* `load_model` -- MiMo in its constructor,
|
||||
Qwen3-TTS in `load_weights`. `determine_available_memory()` then samples
|
||||
per-process GPU memory through NVML and subtracts it from the requested budget
|
||||
(`vllm_omni/worker/base.py`). So the **graph-resident** portion is already
|
||||
charged. Subtracting the declared total from the KV budget would double-count
|
||||
it.
|
||||
|
||||
What is *not* charged is the **per-request** portion, which is allocated after
|
||||
profiling. That half is not small: Qwen3-TTS's codec decoder is 4.44 MiB per
|
||||
request, so a server at `max_num_seqs=256` reaches roughly 1.1 GiB that no
|
||||
per-stage number predicted.
|
||||
|
||||
The consumer therefore reports and does not subtract.
|
||||
|
||||
## What the four models declare
|
||||
|
||||
| Model | Cache | Bounded by | Per row | Rows |
|
||||
|---|---|---|---|---|
|
||||
| Qwen3-TTS codec decoder (async-chunk only) | sliding `DynamicCache` | `sliding_window - 1 = 71` | 4.44 MiB | `max_num_seqs`, x2 for the working copy |
|
||||
| Qwen3-TTS graph pool | retained captures | same | 4.44 MiB | fixed, from captured shapes |
|
||||
| MiMo-Audio local transformer | `DynamicCache` | `group_size + max(delay_pattern) = 11` | 704 KiB | `max_num_seqs` |
|
||||
| MiMo-Audio graph pool | captured | same | 704 KiB | fixed 261 (bucket sum) |
|
||||
| MiniCPM-o Whisper encoder (streaming only) | `EncoderDecoderCache` | `embed_positions` rows = 1500 | 140.62 MiB | fixed 1, per session |
|
||||
| ming_flash_omni talker | `StaticCache` | hardcoded `max_cache_len = 2048` | 24.00 MiB | fixed 1 |
|
||||
|
||||
**No cache is bounded by `max_model_len`.** Each has its own mechanism: a
|
||||
sliding window, a decode-loop trip count, an encoder frame limit, a constant.
|
||||
|
||||
**ming's geometry does not exist in this repo.** `self._llm_config` is a
|
||||
`Qwen2Config` from the checkpoint, so layers, kv-heads, head-dim and dtype are
|
||||
only knowable after load. That is what makes this a runtime query rather than a
|
||||
table.
|
||||
|
||||
## Lifetime is not multiplicity
|
||||
|
||||
`scope` records when memory appears and goes. It does **not** decide how wide
|
||||
the allocation is, and an earlier revision that derived scaling from it was
|
||||
wrong in two different directions at once:
|
||||
|
||||
- ming's talker is per-call but serialized -- `forward` takes
|
||||
`runtime_additional_information[0]` and runs the whole AR loop inline -- so
|
||||
scaling it by `max_num_seqs` over-reported it by that factor.
|
||||
- MiniCPM-o's encoder is per *session*, not per sequence, so `max_num_seqs` is
|
||||
simply the wrong number for it.
|
||||
|
||||
So width is declared explicitly, by naming the driver. `RowDriver` has two
|
||||
values, because two are all the known caches need:
|
||||
|
||||
- `FIXED` -- a count the model controls, with `rows_fixed` and a required
|
||||
`rows_reason`. `rows_fixed=1` on a cache that looks per-request is a claim a
|
||||
reviewer must be able to check.
|
||||
- `MAX_NUM_SEQS` -- one row per in-flight sequence; only the engine knows the
|
||||
value.
|
||||
|
||||
A third value belongs here when a model actually has a cache the engine widens
|
||||
by some other number. An earlier revision added one speculatively, for a single
|
||||
declarer, and paid for it with a class hierarchy, an engine-capacity object and
|
||||
a conditional-annotation field.
|
||||
|
||||
`rows` counts row-equivalents at peak and deliberately does not distinguish
|
||||
"one allocation N rows wide" from "N allocations of one row". Those cost the
|
||||
same, and a revision that tried to carry both got the object topology wrong in
|
||||
three of four declarers: MiniCPM-o keeps one cache object per session while
|
||||
asserting batch size 1, and both graph pools keep one distinct object per
|
||||
captured bucket. The layout belongs in `allocation_note`, where it cannot be
|
||||
mistaken for arithmetic.
|
||||
|
||||
## Declaring nothing is a valid declaration
|
||||
|
||||
A declaration must be inert when its path is. Qwen3-TTS holds KV only in
|
||||
async-chunk mode -- stateless decoding runs `_forward_exact`, which calls the
|
||||
transformer without `use_cache` -- so the stateless path declares nothing.
|
||||
Getting this wrong is easy in both directions, and both have happened here: one
|
||||
revision hung the declaration off the CUDA-graph wrapper and reported zero under
|
||||
`enforce_eager`, where the eager async-chunk path does allocate; the next
|
||||
declared unconditionally and invented roughly a gigabyte for stateless
|
||||
deployments.
|
||||
|
||||
Where the model cannot see the deciding setting, it declares the honest unit
|
||||
and says so in `allocation_note`. MiniCPM-o's encoder cache exists only on the
|
||||
streaming path and is one object per session; the session cap is not visible
|
||||
from the model, so it declares one session rather than scaling by a number it
|
||||
would have to guess at.
|
||||
|
||||
## Finding the declarers
|
||||
|
||||
`collect_model_local_kv_specs()` walks `named_modules()`. Two wrinkles:
|
||||
|
||||
- The runner's `self.model` may be a `CUDAGraphWrapper`, which is not an
|
||||
`nn.Module` but forwards attribute access to its runnable. The walk therefore
|
||||
works either way, but entering through both the wrapper and its runnable would
|
||||
count a root-level declarer twice. The collector calls the supported
|
||||
`unwrap()` accessor first.
|
||||
- An owner that is not a submodule at all (Qwen3-TTS's graph wrapper, ming's
|
||||
`MingAudioGenerator`) is reached by one explicit hop from the module that
|
||||
holds it.
|
||||
|
||||
## Adding a model
|
||||
|
||||
Return one entry per distinct allocation. Prefer `spec_from_hf_config()` and
|
||||
pass the same config object the cache is built from; it raises rather than
|
||||
guessing when an attribute is missing. `capacity_source` is diagnostic text --
|
||||
never branch on it.
|
||||
|
||||
## Known limits
|
||||
|
||||
Row counts taken from live state after warmup miss caches `transformers` has
|
||||
not yet materialized, which makes Qwen3-TTS's graph-pool figure a floor for the
|
||||
xvec path.
|
||||
|
||||
MiMo's graph-pool cache has no reachable Python object -- it is a local inside
|
||||
the captured frame -- so its width comes from the captured bucket list rather
|
||||
than from tensors. Its two entries are also mutually exclusive per call: a call
|
||||
either replays a bucket or runs eagerly. The sum is the right ceiling for the
|
||||
process and the wrong number for a single request.
|
||||
|
||||
MiMo captures every bucket gated only on `torch.cuda.is_available()`
|
||||
(`mimo_audio_llm.py:670`), so those 179.44 MiB stay resident even when the
|
||||
deploy config sets `enforce_eager: true`. Gating it is a one-line change that
|
||||
needs a model boot to verify, so it is noted rather than bundled here.
|
||||
@@ -0,0 +1,89 @@
|
||||
# Mooncake KV Transfer
|
||||
|
||||
## PR summary
|
||||
|
||||
This change adds an opt-in AR-to-DiT KV handoff using vLLM's native
|
||||
`KVTransferConfig`, Scheduler-owned diffusion KV pages, and
|
||||
`MooncakeConnector`. The existing `OmniKVTransferManager` path remains the
|
||||
fallback when `kv_transfer_config` is absent.
|
||||
|
||||
The orchestrator creates one transfer ticket per request and resolves the
|
||||
bound AR replica's Mooncake endpoint after generation. DiT allocates the
|
||||
destination pages, waits for every worker rank to finish loading, and releases
|
||||
the source and destination pages only after the connector lifecycle completes.
|
||||
Request registration failures are rolled back before Worker dispatch. Receive
|
||||
timeouts and cancellation terminate the request without recycling in-flight
|
||||
pages: the scheduler retains them until every CFG row has completed on every
|
||||
rank. Late completion is polled only while such requests remain. If a peer
|
||||
never completes, its pages remain reserved until shutdown; other requests can
|
||||
continue using the remaining capacity. Invalid pages, missing Worker ranks,
|
||||
and failures without a proven rollback still stop the replica before teardown.
|
||||
The example sets `transfer_timeout: 60.0` seconds explicitly. Cancellation
|
||||
queued during a synchronous receive is processed when that receive completes
|
||||
or times out; it does not interrupt the remote write.
|
||||
|
||||
## Reference configuration
|
||||
|
||||
| Stage | Replicas | Devices | Parallelism | KV cache |
|
||||
| --- | ---: | --- | --- | ---: |
|
||||
| AR | 2 | 0-3 | TP=2 per replica | 2 GiB per worker |
|
||||
| DiT | 2 | 4-7 | TP=2 per replica, EP enabled | 1 GiB per worker |
|
||||
|
||||
The acceptance workload uses 512x512 images, two denoising steps, guidance
|
||||
scale 2.0, and `max_num_seqs=1`.
|
||||
|
||||
For a four-GPU debug run, set both stages to one replica and assign AR TP=2 to
|
||||
devices `0,1` and DiT TP=2 to devices `2,3`.
|
||||
|
||||
## Experiment
|
||||
|
||||
The final comparison used one node with eight NVIDIA RTX PRO 6000 Blackwell
|
||||
Server Edition GPUs, TCP Mooncake transport, the same 50-prompt/seed T2I
|
||||
dataset, concurrency 8, and two independent 100-request runs per
|
||||
implementation. Each side completed 200/200 requests without an OOM or request
|
||||
failure.
|
||||
|
||||
| Implementation | Run 1 (req/s) | Run 2 (req/s) | Mean (req/s) |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| `OmniKVTransferManager` | 4.4103 | 4.4497 | 4.4300 |
|
||||
| Native `MooncakeConnector` | 4.4457 | 4.4699 | 4.4578 |
|
||||
|
||||
Native throughput is **0.63% higher** than the legacy mean. The absolute
|
||||
performance difference is below the 1% acceptance threshold.
|
||||
|
||||
For accuracy, a non-zero transferred prefix and the resulting attention output
|
||||
were compared bit-for-bit with the legacy manager path:
|
||||
|
||||
```text
|
||||
prefix_bits_equal True
|
||||
attention_bits_equal True
|
||||
max_abs_diff 0.0
|
||||
```
|
||||
|
||||
The one-AR/one-DiT TP=2 debug topology also completed 20/20 sequential
|
||||
requests. A separate real-TCP transport check transferred both CFG rows from a
|
||||
TP=1 producer to a strict-Ulysses SP=2 consumer with exact values on both
|
||||
target ranks.
|
||||
|
||||
## Scope and limitations
|
||||
|
||||
- The fully validated end-to-end topology is 2x AR TP=2 plus 2x DiT TP=2.
|
||||
- Heterogeneous TP=1 to SP=2 reshaping is transport-tested, but arbitrary mixed
|
||||
TP/CFG/SP end-to-end combinations have not been exhaustively validated.
|
||||
- CFG-parallel end-to-end deployment is not part of this acceptance run.
|
||||
- Native AR-to-DiT transfer currently requires `async_chunk: false`.
|
||||
- TCP is covered by the acceptance run. The supplementary single-node RDMA
|
||||
measurements use a separate short workload; they do not establish RDMA
|
||||
support for multi-node deployment, which remains out of scope.
|
||||
- The native path currently validates the two-stage `AR(0) -> DiT(1)` graph. Other stage graphs are rejected during startup until
|
||||
their producer/consumer edge mapping is implemented.
|
||||
- Sleep mode is disabled for the native paged path because registered KV pages
|
||||
must remain mapped for the connector lifetime.
|
||||
|
||||
- Ascend NPU adaptation and its performance validation are maintained in a
|
||||
separate follow-up PR; the GPU acceptance results above do not validate NPU.
|
||||
- The homogeneous piecewise CUDA batching optimization is retained. A native
|
||||
paged-attention CUDA graph capture/replay regression covers two CFG rows,
|
||||
strided QKV, mixed causal/image spans, changed inputs, and output stability.
|
||||
It must be run on CUDA hardware; CPU unit tests do not establish graph or
|
||||
end-to-end performance acceptance.
|
||||
@@ -41,7 +41,8 @@ Every backend implements `OffloadBackend`:
|
||||
- `is_enabled()` reports lifecycle state.
|
||||
|
||||
`disable()` does not promise to restore the pipeline's original device
|
||||
placement. The caller owns any subsequent rematerialization.
|
||||
placement. It must restore usable parameter and buffer storage before releasing
|
||||
backend-owned host weights. The caller owns subsequent device placement.
|
||||
|
||||
An enable failure must remove every partially installed hook. Rank-local
|
||||
backends restore ordinary tensors so the pipeline can be retried. A failed
|
||||
@@ -54,9 +55,19 @@ must use distinct hook names and remove only hooks they own.
|
||||
## Discovery and topology
|
||||
|
||||
`resolve_offload_plan(pipeline, config)` is the single topology entry point.
|
||||
Every backend calls it once when `enable()` starts and then consumes the
|
||||
resolved components, block stacks, resident heads, and staged-residency flags
|
||||
it returns. Backends do not re-read model declarations.
|
||||
The generic model-level and ordinary layerwise paths call it once when
|
||||
`enable()` starts and consume its component selection and block stacks.
|
||||
Ordinary layerwise offload also respects a resolved `skip_reason`, preserving
|
||||
the legacy missing-DiT no-op without placing auxiliary components. Non-block
|
||||
DiT and encoder state is placed by resolved block tensor identity, so the
|
||||
backend does not rediscover block containers by attribute name.
|
||||
|
||||
Two compatibility paths remain outside this completed generic cutover:
|
||||
model-level pipelines implementing `SupportsModelCpuOffload` still own their
|
||||
custom phase lifecycle (RFC #6648 J4), and distributed layerwise offload still
|
||||
reads declarations for nested-submodule staging and loader-owned mmap paths
|
||||
(J3). The custom model-level protocol is delegated to before generic plan
|
||||
resolution; it must not be subjected to the generic encoder/DiT swap checks.
|
||||
|
||||
Pipeline component discovery prefers `SupportsComponentDiscovery` declarations:
|
||||
|
||||
@@ -73,10 +84,22 @@ Block topology comes from the pipeline `OffloadPlan` first: `block_attrs` maps
|
||||
each DiT path to its ordered block containers, `encoder_block_attrs` declares
|
||||
streamable encoder stacks, `on_demand_component_paths` marks pipeline-managed
|
||||
residency, `resident_dit_paths` marks the DiTs that may hold resident layers,
|
||||
and `encoder_dlo_weight_replication` marks the encoders whose loader-produced
|
||||
weights are safe for AllGather. DiTs absent from the plan fall back to
|
||||
`_layerwise_offload_blocks_attrs` (including the deprecated singular-name
|
||||
compatibility path).
|
||||
`encoder_dlo_weight_replication` marks the encoders whose loader-produced
|
||||
weights are safe for AllGather, and `offload_submodules` maps a DiT submodule
|
||||
that owns its residency to its own block container. DiTs absent from the plan
|
||||
fall back to `_layerwise_offload_blocks_attrs` (including the deprecated
|
||||
singular-name compatibility path).
|
||||
|
||||
An undeclared DiT submodule joins the plan only when it is large enough that
|
||||
keeping it resident would defeat streaming; its block container is then found
|
||||
by scanning well-known attribute names, which warns once per submodule class.
|
||||
Size is computed from parameter shapes and dtypes, including meta parameters,
|
||||
so resolving before mmap loading preserves the same residency decision.
|
||||
Block ownership validation includes nested components: a block shared with a
|
||||
parent, sibling, or encoder is rejected before any hooks or storage are changed.
|
||||
A submodule with no block container must implement the `load_to_device` /
|
||||
`offload_to_cpu` lifecycle, and the resolver rejects it before any component of
|
||||
the pipeline is placed or hooked.
|
||||
|
||||
The resolver is pure: it moves no tensor, installs no hook, writes no module
|
||||
attribute, and reads no process group — multi-rank facts come from
|
||||
@@ -102,3 +125,17 @@ remains responsible for transfer, synchronization, and storage ownership.
|
||||
|
||||
The diffusion [Offloader module design](../../module/diffusion/offloader.md)
|
||||
describes how these feature contracts fit into the larger diffusion runtime.
|
||||
|
||||
## Backend acceptance tests
|
||||
|
||||
`tests/diffusion/offloader/test_backend_plan_contract.py` covers the two
|
||||
behaviors the generic cutover changed. A DiT attribute aliasing a streamed
|
||||
block must keep the ring's host residency, so placement cannot follow attribute
|
||||
names; the CUDA case fails against the previous name-based placement. Both
|
||||
backends must then run with their selector helpers and the model declaration
|
||||
made unreadable, proving execution consumes the resolved plan alone.
|
||||
|
||||
Selection errors, rollback, residency, and enable/disable cycles stay in
|
||||
`test_plan_resolver.py`, `test_layerwise_backend.py` and
|
||||
`test_sequential_backend.py`. Model-specific phases and executed tied aliases
|
||||
remain the J4 contract suite.
|
||||
@@ -318,18 +318,18 @@ The same behavior applies to the Base and VoiceDesign Qwen3-TTS checkpoints
|
||||
because they use the same Talker implementation.
|
||||
|
||||
!!! warning
|
||||
Do not pass `--no-async-chunk` or enable prefix caching when you want this
|
||||
optimization. Either change causes the runner to fall back to synchronous
|
||||
output construction. Enabling async chunk alone cannot activate the feature
|
||||
for a model that has not opted into async Omni output.
|
||||
Do not pass `--no-async-chunk` when you want this optimization. Disabling
|
||||
async chunk causes the runner to fall back to synchronous output
|
||||
construction. Enabling async chunk alone cannot activate the feature for a
|
||||
model that has not opted into async Omni output.
|
||||
|
||||
!!! note "Prefix cache compatibility"
|
||||
Async Omni output materialization and Omni prefix caching cannot currently
|
||||
run together. `_should_use_async_omni_output()` returns `False` whenever
|
||||
`self.omni_prefix_cache` is present, so prefix caching selects synchronous
|
||||
output materialization. Supporting both features requires snapshotting or
|
||||
otherwise synchronizing prefix-cache merge and update state before the
|
||||
background output path can consume it safely.
|
||||
Prefix cache and async Omni output can run together. `save_outputs` returns
|
||||
a `step_id`; the output builder later calls `materialize(sid, req_ids)` with
|
||||
the save-time request list, which may be a step late relative to the live
|
||||
`input_batch`. `_should_use_async_omni_output()` does not disable itself
|
||||
when the cache is present. See
|
||||
[Automatic Prefix Caching](prefix_caching.md#implementation).
|
||||
|
||||
## Compatibility and Fallbacks
|
||||
|
||||
@@ -341,10 +341,9 @@ runtime conditions hold:
|
||||
| AR async scheduling is enabled | The optimization relies on the scheduler advancing while the prior output is materialized |
|
||||
| `async_chunk` is enabled | The feature targets incremental downstream Omni payloads |
|
||||
| The model stage opts in with `use_async_omni_output` | Models must declare that their output lifecycle is safe to defer |
|
||||
| Omni prefix cache is disabled | Prefix-cache merge and update ordering currently requires synchronous materialization |
|
||||
| Speculative decoding is disabled | Speculative output state is not included in this deferred path |
|
||||
| Routed-expert output is disabled | Routed-expert extraction currently requires the synchronous path |
|
||||
| Postprocess is absent or explicitly runs eagerly | State needed by the next decode step must be updated before the runner returns |
|
||||
| Postprocess is absent or explicitly runs eagerly | State needed by the next decode step must be updated before the runner returns. Eager postprocess sees the scheduled slice (`None, None` combined tensors), not the prefix-cache-merged full prompt; it must only consume the tail |
|
||||
|
||||
These checks are evaluated per stage on runners based on `GPUARModelRunner`.
|
||||
An unsupported combination does not prevent serving; it only falls back to
|
||||
@@ -380,6 +379,8 @@ so Omni payload materialization remains synchronous.
|
||||
- `tests/worker/test_gpu_ar_model_runner.py`: Snapshot, guard, connector
|
||||
ordering, and background error-propagation tests.
|
||||
- [Async Chunk](async_chunk.md): Inter-stage chunking and scheduling design.
|
||||
- [Automatic Prefix Caching](prefix_caching.md#implementation): Prefix-cache
|
||||
`step_id` / `materialize` contract used by the deferred output builder.
|
||||
- [Qwen3-Omni optimization blog](https://vllm.ai/blog/2026-07-01-qwen3-omni-optimization):
|
||||
Optimization context and controlled performance results.
|
||||
- [PR #4476](https://github.com/vllm-project/vllm-omni/pull/4476): Feature
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
# NixlConnector
|
||||
|
||||
## When to Use
|
||||
|
||||
Multi-node or intra-node stage transfer over NIXL, which brokers RDMA/shared-memory
|
||||
transports through a single API. Useful when the deployment already standardises on
|
||||
NIXL for KV transfer, or when the target accelerator has no Mooncake/Mori backend --
|
||||
NIXL reaches Intel XPU through UCX's Level Zero support.
|
||||
|
||||
## Mechanism
|
||||
|
||||
Uses vLLM's `NixlWrapper` (`vllm.distributed.nixl_utils`) to register the producer's
|
||||
tensors and let the consumer pull them with a NIXL `READ`.
|
||||
|
||||
- Data Plane: NIXL agent-to-agent `READ`, GPU-to-GPU where the backend allows it.
|
||||
- Control Plane: a ZMQ ROUTER socket atomically claims the source allocation before
|
||||
returning descriptors. Forwarded `put()` metadata includes the generation and
|
||||
endpoint; the consumer claims that generation before submitting any READ.
|
||||
|
||||
Transfer metadata uses schema version 1. Tensor descriptors are grouped by NIXL
|
||||
memory type (`DRAM`, `VRAM`, and so on), while each descriptor retains its global
|
||||
tensor index so mixed CPU/accelerator payloads can be reconstructed in order.
|
||||
|
||||
Payloads are not restricted to tensors. A single tensor or a list of tensors is
|
||||
transferred as-is; a nested structure has its tensor leaves extracted and shipped
|
||||
alongside a msgpack-encoded skeleton; anything else is msgpack-encoded into one
|
||||
uint8 tensor. Mapping and sequence structure is preserved. `msgspec.Struct` values,
|
||||
including `OmniPayloadStruct`, are normalised to mappings so their tensor leaves use
|
||||
the direct NIXL path rather than CPU object serialisation.
|
||||
|
||||
Each NIXL agent uses strict thread synchronization. A consumer performs one short
|
||||
metadata query per scheduler poll (10ms by default), which prevents an unavailable
|
||||
producer from blocking unrelated requests. The regular transfer-completion handshake
|
||||
keeps the configured transfer timeout.
|
||||
|
||||
## Installation
|
||||
|
||||
CUDA hosts can install the published wheel:
|
||||
|
||||
```bash
|
||||
pip install nixl
|
||||
```
|
||||
|
||||
For Intel XPU, use `docker/Dockerfile.xpu`. Its upstream vLLM XPU base image
|
||||
provides a matching torch-xpu, UCX with Level Zero support, and NIXL runtime.
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
connectors:
|
||||
nixl_connector:
|
||||
name: NixlConnector
|
||||
extra:
|
||||
host: "auto"
|
||||
zmq_port: 50061
|
||||
backends: ["UCX"]
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: nixl_connector
|
||||
|
||||
- stage_id: 1
|
||||
input_connectors:
|
||||
from_stage_0: nixl_connector
|
||||
```
|
||||
|
||||
Parameters:
|
||||
|
||||
- `host`: address the producer binds its handshake socket to (`"auto"` to detect).
|
||||
- `zmq_port`: handshake port. If a producer omits it, an ephemeral port is opened
|
||||
for ownership claims and returned in `put()` metadata. The value in the deploy YAML is a
|
||||
base port. Purpose, replica, rank, and producer-stage offsets are added centrally,
|
||||
so colocated connectors and tensor-parallel workers receive distinct endpoints
|
||||
without one config entry per rank.
|
||||
- `sender_host` / `sender_zmq_port`: consumer-side override naming the producer's
|
||||
handshake endpoint. Only needed when the consumer cannot learn it from metadata.
|
||||
In replicated deployments, the producer endpoint is carried with each request and
|
||||
selected at `get()` time; it is not mutable connector-wide state.
|
||||
Explicit sender ports are preserved, not recomputed using the receiver replica.
|
||||
Intermediate NIXL stages retain their incoming receiver configuration and a
|
||||
separate outgoing bind configuration, so one worker can receive and publish.
|
||||
- `backends`: NIXL backends to register memory with. Defaults to `["UCX"]`.
|
||||
- `receive_device`: forces where received tensors land. By default the consumer
|
||||
keeps the producer's device *type* but uses its own current device of that type.
|
||||
- `memory_type`: overrides the NIXL memory type, otherwise `DRAM` for CPU tensors
|
||||
and `VRAM` for accelerator tensors.
|
||||
- `lease_seconds`: how long a `put()` payload stays registered while waiting to be
|
||||
read (default 3600). The consumer reports completion, so this only bounds payloads
|
||||
nobody has claimed. An internal reaper releases only unclaimed registrations.
|
||||
`VLLM_OMNI_NIXL_LEASE_S` overrides it.
|
||||
- `transfer_timeout_s`: how long a `get()` waits for its `READ` to complete
|
||||
(default 300). NIXL 1.3 has no transfer cancellation API, so a timed-out transfer's
|
||||
buffers and registrations remain owned by the connector until NIXL reports a
|
||||
terminal state; `close()` waits for that state before releasing them.
|
||||
`VLLM_OMNI_NIXL_XFER_TIMEOUT_S` overrides it.
|
||||
|
||||
### Source ownership and failure limits
|
||||
|
||||
Claims and expiry use the same lock. Completion identifies both a payload
|
||||
generation and a unique READ claim; duplicate or stale ACKs cannot release a
|
||||
different reader or replacement payload. Replacing a claimed key is rejected.
|
||||
Terminal errors also release claims; timeout, unknown status, and active sibling
|
||||
transfers retain the complete destination bundle and source claim until terminal.
|
||||
Zero-element tensors retain their shape/dtype/skeleton positions but never create
|
||||
DMA descriptors or registrations.
|
||||
|
||||
A lost metadata response, lost completion ACK, or abandoned consumer can retain a
|
||||
claim indefinitely. NIXL 1.3 cannot prove remote cancellation, so neither TTL nor
|
||||
`cleanup()` frees those allocations. Producer `close()` rejects new work and
|
||||
retains its agent, listener and claimed allocations; a later `close()` finishes
|
||||
teardown after claims drain. Permanently abandoned claims remain until process
|
||||
exit. This favors memory safety over bounded shutdown/memory usage. Only trusted
|
||||
peers may access this unauthenticated control plane. Both endpoints must use the
|
||||
claim-aware protocol; rolling compatibility with older GET_META clients is not
|
||||
provided. Legacy externally-owned metadata without a generation is accepted only
|
||||
under the caller's lifetime guarantee, not managed as a leased allocation here.
|
||||
|
||||
## Validation
|
||||
|
||||
The native CUDA smoke test requires NIXL and at least two visible CUDA devices:
|
||||
|
||||
```bash
|
||||
UCX_RCACHE_MAX_UNRELEASED=1024 pytest -o addopts='' -q -s \
|
||||
tests/distributed/omni_connectors/test_nixl_connector_native.py
|
||||
```
|
||||
|
||||
It starts independent producer and consumer processes, loads the real UCX backend,
|
||||
performs the ZMQ `GET_META` handshake and a native NIXL `READ` from GPU 0 to GPU 1,
|
||||
checks mixed CPU/GPU tensor leaves from an `OmniPayloadStruct`, sends `XFER_DONE`,
|
||||
and verifies that producer registrations are released. This validates same-host,
|
||||
cross-GPU operation. Inter-node UCX transport is supported by the design but was not
|
||||
validated by this test.
|
||||
|
||||
For more details, refer to the [NIXL repository](https://github.com/ai-dynamo/nixl).
|
||||
@@ -1,6 +1,5 @@
|
||||
# Automatic Prefix Caching in Omni Models
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
@@ -9,6 +8,8 @@
|
||||
- [High-Level Approach](#high-level-approach)
|
||||
- [Example](#example)
|
||||
- [What About Multimodal Inputs?](#what-about-multimodal-inputs)
|
||||
- [Implementation](#implementation)
|
||||
- [Related Files](#related-files)
|
||||
|
||||
---
|
||||
|
||||
@@ -27,6 +28,7 @@ vLLM implements automatic prefix caching for managing its kv-cache, which is bes
|
||||
This document describes vLLM-Omni's mechanism for caching tensor outputs that are meant to be passed between stages, when requests have common prefixes, similar to the way in which vLLM has prefix caching for the kv-cache. This works in conjunction with vLLM's multimodal encoder caching, but is distinct. See the final section for a concrete example for how they tie together in practice.
|
||||
|
||||
### High-Level Approach
|
||||
|
||||
!!! note "Note 2"
|
||||
Prior to reading this section, it's recommended to take a look at the design documents in vLLM for [Automatic Prefix Caching](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/), which will make some of the concepts more clear.
|
||||
|
||||
@@ -38,8 +40,10 @@ The main focus of vLLM-Omni's approach to prefix caching stage outputs is to bui
|
||||
|
||||
With this in mind, consider the set of blocks in a 2D layout, where the row represents the index of blocks being considered, and the columns represent the slots corresponding to tokens within each block. Since we know the `num_blocks` and `block_size` from our kv cache config, if we want to cache a tensor with feature size `D`, we can preallocate a CPU tensor of size `(num_blocks, block_size, D)`, and use the same block index and slot mapping to retrieve the corresponding feature vector.
|
||||
|
||||
Host footprint: each cached key costs `num_blocks × block_size × D × dtype_bytes` of **pinned** CPU memory (pinned so device→host can overlap compute), allocated on the first `save_outputs` that sees the key — the first real request pays the `cudaHostAlloc`. Measured on a Qwen3-Omni deployment: thinker `__hidden_states__` `[15092, 16, 2048]` bf16 ≈ 0.92 GiB (plus the same again for each `hidden_states.layer_*` key a model exposes), Qwen3-TTS talker `codes.audio` `[16180, 16, 16]` ≈ 33 MiB. Budget host RAM for the stage accordingly.
|
||||
|
||||
### Example
|
||||
|
||||
!!! note "Note 3"
|
||||
Prefix caching in vLLM-Omni currently is only supported on AutoRegressive stages with one kv-cache group. Configure it with the pipeline-wide `enable_prefix_caching` field in the deploy config.
|
||||
|
||||
@@ -56,7 +60,7 @@ The prefix cache flow is then outlined below.
|
||||
|
||||
2. Say we process the request `The quick brown fox was tired and slept beneath the shady tree`, which is 12 tokens and evenly divides into 3 blocks as shown below.
|
||||
|
||||
```
|
||||
```text
|
||||
[ The quick brown fox ] [ was tired and slept ] [beneath the shady tree ]
|
||||
Block 1: |<--- block tokens ---->|
|
||||
Block 2: |<------- prefix ------>| |<--- block tokens --->|
|
||||
@@ -65,10 +69,9 @@ Block 3: |<------------------ prefix -------------------->| |<--- block tokens -
|
||||
|
||||
When the request processes, we inspect the multimodal outputs and identify the `mm_feature` tensor, which will be of shape `(seq_len, feature_dim)`, i.e., `(12, 16)` in this example. We note that the first axis is dependent on the `seq_len` and add a new cache_tensor of shape `(num_blocks, block_size, feature_dim)` to our multimodal cache for tensors.
|
||||
|
||||
1. If we lay out the cache as a 2D tensor of shape (`num_blocks`, `block_size`), we'll have something like the following:
|
||||
|
||||
3. If we lay out the cache as a 2D tensor of shape (`num_blocks`, `block_size`), we'll have something like the following:
|
||||
|
||||
```
|
||||
```text
|
||||
0: [ The quick brown fox ]
|
||||
1: [ was tired and slept ]
|
||||
2: [beneath the shady tree ]
|
||||
@@ -78,7 +81,8 @@ When the request processes, we inspect the multimodal outputs and identify the `
|
||||
```
|
||||
|
||||
Or, if we flatten it down to 1D,
|
||||
```
|
||||
|
||||
```text
|
||||
0: The
|
||||
1: quick
|
||||
2: brown
|
||||
@@ -90,7 +94,8 @@ Or, if we flatten it down to 1D,
|
||||
```
|
||||
|
||||
which we can think of as row indices into the hidden states tensor if we view it as the 2D shape `(num_blocks x block_size, feature_dim)`. That is, the analogous flattened (from 3D -> 2D) mapping of the cache for hidden states becomes the following.
|
||||
```
|
||||
|
||||
```text
|
||||
0: <hidden states vector of len 2 corresponding to 'The'>
|
||||
1: <hidden states vector of len 2 corresponding to 'quick'>
|
||||
2: <hidden states vector of len 2 corresponding to 'brown'>
|
||||
@@ -103,10 +108,9 @@ which we can think of as row indices into the hidden states tensor if we view it
|
||||
|
||||
Similarly, for the multimodal outputs cache, the flattened coordinates are the same, but the `mm_feature` maps to vectors of length `16` instead of the hidden size of `2`. Note that in practice, we may have multiple multimodal output tensors per forward pass, which may have different names and different feature dimensions.
|
||||
|
||||
1. Now, say that we receive a new request `The quick brown fox jumped over the dog`.
|
||||
|
||||
4. Now, say that we receive a new request `The quick brown fox jumped over the dog`.
|
||||
|
||||
```
|
||||
```text
|
||||
[ The quick brown fox ] [ jumped over the dog ]
|
||||
Block 1: |<--- block tokens ---->|
|
||||
Block 2: |<------- prefix ------>| |<--- block tokens --->|
|
||||
@@ -116,7 +120,7 @@ Here, we will have a cache hit for `Block 1` which will be detected by vLLM base
|
||||
|
||||
Since we have the block indices / slot mappings from the kv cache manager, we can simply mirror the mappings and leverage the same indices for the cached hidden states and multimodal outputs. This allows us to look up the correct tensors from our externally maintained 3D caches.
|
||||
|
||||
```
|
||||
```text
|
||||
0: [ The quick brown fox ] < already in the cache
|
||||
1: [ was tired and slept ]
|
||||
2: [beneath the shady tree ]
|
||||
@@ -129,11 +133,11 @@ Since we have the block indices / slot mappings from the kv cache manager, we ca
|
||||
|
||||
Finally, to pass the full hidden states and multimodal outputs to the next stage, we simply concatenate the cached contents with the corresponding new tensors computed from the current forward call.
|
||||
|
||||
|
||||
### What About Multimodal Inputs?
|
||||
|
||||
It's also useful to consider the case about how Omni prefix caching is handled when we have multimodal inputs that don't cleanly end on block boundaries, as well as how this works with multimodal encoder caching in vLLM. For example:
|
||||
|
||||
```
|
||||
```text
|
||||
[ Im0 Im1 Im2 Im3 ] [ Im4 Im5 foo <empty> ]
|
||||
Block 1: |<--- block tokens ---->|
|
||||
Block 2: |<------- prefix ------>| |<--- block tokens --->|
|
||||
@@ -147,9 +151,9 @@ In reality, this isn't a big problem for correctness, because vLLM also maintain
|
||||
- The hash describing the image data starting at position 0 and with length 6
|
||||
- In vLLM's encoder cache, a mapping from the image hash above to the encoder output
|
||||
|
||||
|
||||
To understand what happens, say we get the following input as a second request:
|
||||
```
|
||||
|
||||
```text
|
||||
[ Im0 Im1 Im2 Im3 ] [ Im4 Im5 bar baz ]
|
||||
Block 1: |<--- block tokens ---->|
|
||||
Block 2: |<------- prefix ------>| |<--- block tokens --->|
|
||||
@@ -162,3 +166,162 @@ Because we have multimodal data in a scheduled span that isn't fully precomputed
|
||||
When we pass our multimodal tensors to the language model component in the same stage, we'll then expect the same outputs, because the prefix caching behaviors in vLLM-Omni / vLLM match, so the LLM will use vLLM's KV cache manager's prefix caching to correctly handle the attention information for `Block 1` while calculating the outputs for `Block 2`, giving us the correct results for processing `Block 2` with the context of `Block 1`.
|
||||
|
||||
Finally, we look up the output hidden states/multimodal tensors corresponding to the prefix cache hit `Block 1` and concatenate it with the forward pass result to get the final result, which is expected to be identical to the full hidden states when prefix caching is disabled.
|
||||
|
||||
### Implementation
|
||||
|
||||
The block/slot model is `vllm_omni/core/prefix_cache/`.
|
||||
`OmniPrefixCacheManager` owns slot occupancy, the request-task table,
|
||||
hit spans, and merge.
|
||||
`OmniPrefixCacheController` moves data: a reusable `StagingBufferPool` for
|
||||
this step's device→host copy, and writes into the durable `PrefixBlockPool`.
|
||||
The state lock covers those tables only.
|
||||
|
||||
Miss is not an error (this step's forward slice only). A hit span whose
|
||||
hidden rows are absent is fatal. For mm keys the rule is looser: a model may
|
||||
emit a key only for some requests, so a hit span with no rows behind an mm key
|
||||
reads zeros from the pool for those positions rather than raising. Abort still
|
||||
writes: once a hash entered this step's batch it must land in the cache.
|
||||
|
||||
`enable_prefix_caching` is refused on KV-consumer stages (`kv_role` of
|
||||
`kv_consumer` or `kv_both`). KV received from a producer is reported as
|
||||
`num_computed_tokens` too, and the manager cannot tell it from a local hit.
|
||||
Producer-only stages are unaffected. Pooling stages never save, so they get
|
||||
no cache. This gate is one function, `stage_prefix_cache_config`, called by
|
||||
both the GPU and the NPU model runner at kv-cache init.
|
||||
|
||||
Which stages may set `enable_prefix_caching: true`:
|
||||
|
||||
| Stage | `enable_prefix_caching` | Why |
|
||||
| --- | --- | --- |
|
||||
| AR stage with one full-attention kv group whose hidden states / per-token mm feed the next stage (Qwen3-Omni thinker and talker) | supported | The case the cache is built for: full-prompt hidden states are merged from the pool on a hit. |
|
||||
| AR stage that sets `requires_full_prefix_cached_hidden_states = False`, optionally with `deferred_prefix_cache_mm_keys` (Qwen3-TTS talker, Higgs v3 talker) | supported | Hidden is not cached; deferred codec rows are written once on finish. |
|
||||
| Pooling stage | ignored | Never saves; the gate returns no config. |
|
||||
| `kv_role: kv_consumer` / `kv_both` | refused at kv-cache init (`OmniPrefixCacheUnmatchError`) | Producer KV shows up as `num_computed_tokens` and is indistinguishable from a local hit. |
|
||||
| Speculative decoding on the stage | refused at kv-cache init | Under async scheduling vLLM keeps `num_computed_tokens_cpu` optimistic (all drafts accepted) during the forward and corrects it afterwards; `step_slots_cpu` would mirror rows at the wrong slots. Not verified; refused as a whole. |
|
||||
| `prefix_match_unit` smaller than `block_size` | refused at kv-cache init | Sub-block hits make `num_computed_tokens` unaligned; the hit registry only mirrors whole blocks. |
|
||||
| Hybrid / sliding-window / multi-group kv cache (e.g. a talker with `attention_type: sliding_recompute`) | refused at kv-cache init | The cache mirrors exactly one full-attention block table. |
|
||||
| Attention backend whose kernel block size differs from `--block-size` (FlashInfer / FlashMLA / CutlassMLA with a block size they do not list natively), or decode context parallel | refused at first step (`FullAttentionGroupView`) | `step_slots_cpu` computes `table[req, pos // block_size] * block_size + pos % block_size` over allocator block ids; hybrid kernel blocks and DCP token striping change that row layout. FlashAttention / Triton accept any multiple of 16 and never split blocks. |
|
||||
| Codec decoder / Code2Wav stages (Qwen3-Omni stage 2, Qwen3-TTS stage 1) | keep `false` | Nothing downstream consumes their hidden states; the cache would only add device→host copies. Not validated. |
|
||||
| Diffusion stages | n/a | No vLLM KV cache to mirror. |
|
||||
|
||||
Hit spans come from `scheduled_new_reqs` only, as in the pre-refactor cache:
|
||||
|
||||
- A new request with a (partial) prefix hit is the normal path: the hit
|
||||
blocks are read from the pool, the rest is this step's rows, and the
|
||||
divergent tail is written under its own blocks. The gather starts on the
|
||||
prefetch thread at `new_step_starts`; a same-step hit (vLLM hashes blocks
|
||||
at schedule time, so `b` can hit blocks `a` computes in the same forward)
|
||||
cannot plan until `a`'s write is registered and starts at `save_outputs`
|
||||
instead. Either way it runs before the next step can hand those blocks
|
||||
to a new tenant. vLLM frees blocks one step before `finished_req_ids`
|
||||
arrives, so a finished request whose last step materializes late may
|
||||
still find a hit slot reassigned. Each `(slot, key)` carries a write
|
||||
version, bumped whenever a new write claims it. A planned read captures
|
||||
that version and is registered in `_pending_reads`; a later write that
|
||||
reclaims those slots copy-on-writes the still-`COMMITTED` rows into the
|
||||
ref before it overwrites the pool, so a delayed fetch serves the original
|
||||
tenant. A version mismatch with no preserved copy raises for live and
|
||||
finished alike (the pool rows are a newer tenant's). Production defaults
|
||||
stay opt-in until preempt/resume hit spans are reconstructed.
|
||||
- `async_chunk` continuation: when the next upstream chunk arrives, the same
|
||||
request id re-enters `scheduled_new_reqs` with `num_computed_tokens` equal
|
||||
to what it already computed itself. Ids already in `live_reqs` are skipped
|
||||
for hit marking: those rows were delivered in earlier steps and re-emitting
|
||||
them would duplicate output. A `delivered_upto` span for this case is
|
||||
Phase 2.
|
||||
- Preemption + reschedule: vLLM resets `num_computed_tokens` to 0 on
|
||||
preemption and re-runs prefix matching on resume, so the resumed request
|
||||
can come back with a fresh hit. With the V1 model runner it arrives
|
||||
through `scheduled_cached_reqs` (id in `resumed_req_ids`, `new_block_ids`
|
||||
replaces the table); with the V2 runner it re-enters `scheduled_new_reqs`
|
||||
while still in `live_reqs`. Neither path marks an omni hit span: the
|
||||
resumed request gets only the rows it recomputes, and its still-open
|
||||
deferred write keeps appending (a slot written twice keeps the later
|
||||
chunk). Cache integrity holds either way — the hit blocks already have
|
||||
rows, from this request or the one it hit. Stages that need full prompt
|
||||
hidden states should be sized so preemption does not occur while prefix
|
||||
caching is on. Same as before this refactor; tracked for Phase 2.
|
||||
|
||||
Two write paths, split by `ModelCachePolicy.deferred_keys`:
|
||||
|
||||
- Immediate (`JOIN_NEXT_STEP`): save launches a whole-step device→host into
|
||||
a staging slot; the committer waits that event and writes the CPU pool.
|
||||
The next save waits `done` (the pool write): a reused slot must never
|
||||
leave a pending pool write behind, or a delayed hit read of the old rows
|
||||
would find nothing recoverable.
|
||||
- Deferred (`JOIN_ON_FINISH`): mm whose first dim is this step's token count
|
||||
stays on a per-request GPU clone; `_WriteChunk`s append across steps;
|
||||
finish/abort (or GPU-byte-budget pressure) forces the copy. One open
|
||||
WriteTask per request; a budget flush may close it mid-request, in which
|
||||
case the next save opens a new one (`write_n` + 1) and a hit reads both,
|
||||
so one long request cannot pin the whole budget.
|
||||
|
||||
A `WriteTask` moves through `TaskState` only via `transition()`, one step at
|
||||
a time along a strict chain:
|
||||
`PENDING` (registered, not queued) → `QUEUED` (on a copy queue) →
|
||||
`COPYING` (one thread owns the copy stage) → `HOST_READY` (host rows ready) →
|
||||
`WRITTEN` (in the CPU pool); `FAILED` is reachable from any non-terminal
|
||||
state. Skipping a step raises. The worker claims `COPYING` in the same `_wake`
|
||||
critical section as the queue pop, so `escalate` never re-queues a task it can
|
||||
see is already claimed. `host_ready` / `done` are wait primitives set by the
|
||||
transitions into `HOST_READY` / `WRITTEN` / `FAILED`. GPU-clone bytes are
|
||||
charged once per clone on a `_BudgetTicket` pinned by every task that views
|
||||
it, and uncharged when the last holder releases (idempotent per tid).
|
||||
|
||||
Mm whose first dim equals the unpadded scheduled length *or* the CUDA-graph
|
||||
padded length is registered on first sighting. Talker `codes.audio` is a
|
||||
cat of scheduled rows and stays unpadded while hidden is padded; both must
|
||||
open a pool key, whether immediate or deferred.
|
||||
Leftover mm (lists, `codes.ref`, any tensor whose first dim is not this
|
||||
step's token count) is copied to CPU at save without truncating that first
|
||||
dim. That leftover copy is the async-builder read replica for this step; it
|
||||
does not write the pool or carry abort/preempt occupancy.
|
||||
|
||||
```python
|
||||
cache.register_policy(ModelCachePolicy.from_model(model)) # load_model
|
||||
cache.new_step_starts(scheduler_output) # before _update_states
|
||||
sid = cache.save_outputs(hidden, mm_outputs, num_tokens_unpadded=n,
|
||||
num_tokens_padded=n_pad)
|
||||
outs = cache.materialize(sid, req_ids) # or discard_step(sid)
|
||||
```
|
||||
|
||||
Each step id is consumed exactly once. `req_ids` must be a subset of the save
|
||||
snapshot. At most `staging_depth` unused step ids may exist at once: every
|
||||
`save_outputs` claims one staging slot, including saves with only leftover
|
||||
mm that copy no device→host page. A slot is also held by each immediate
|
||||
write that views it, until the committer's pool write. A later save waits
|
||||
for `materialize`/`discard_step` or that pool write to free a slot;
|
||||
`staging_claim_timeout_s` then errors with the unused ids and the task count.
|
||||
`join`/`join_host_ready` use the same bound and raise with the stuck task's
|
||||
id and state instead of hanging the caller.
|
||||
`materialize` may run on the async output builder after the engine has
|
||||
entered the next step; leftover mm (not written to the pool) is copied
|
||||
to CPU at `save_outputs` so the builder never reads live CUDA-graph
|
||||
buffers. See
|
||||
[Async Omni Output Materialization](omni_async_output_materialization.md).
|
||||
|
||||
The cache is constructed only on the last pipeline-parallel rank
|
||||
(`_ensure_omni_prefix_cache`). Other ranks skip it: they never call
|
||||
`save_outputs`, so a hit table there would resolve to absent slots.
|
||||
|
||||
Threads, locks, and what each may block on:
|
||||
|
||||
| Thread | Role | May block on | Must not hold while blocked |
|
||||
| --- | --- | --- | --- |
|
||||
| Engine | `new_step_starts`, `save_outputs` | previous-step `join` (`done`) / finished-write `join_host_ready`; `reserve()` GPU-byte flush; staging-slot claim; `dispatch()` / finish-abort `escalate()` (eager mode: the copy + pool write run inline) | `_state_lock` |
|
||||
| Async output builder | `materialize` (may overlap the next engine step) | this step's `step_d2h_event`; `join` (`done`); deferred `fetch_host` | `_state_lock` |
|
||||
| Committer | `_worker_loop`: wait device→host / deferred copy / pool write | `_wake.wait`; `step_d2h_event` or copy-stream sync | never takes `_state_lock` |
|
||||
| Prefetch pool | hit-span gather during forward | `join` (`done`); deferred `fetch_host` | `_state_lock` |
|
||||
|
||||
| Lock | Covers | Does not cover |
|
||||
| --- | --- | --- |
|
||||
| manager `_state_lock` | occupancy tables, step contexts, hit spans, task registration, pool-key publish (`install_key`) | join, GPU-byte flush, copy, `step_d2h_event` wait, pool-key allocation, eager `dispatch()` / `escalate()` |
|
||||
| controller `_lock` / `_wake` | task registry, queues, GPU-clone byte budget | device→host / pool-write body (released before `synchronize`) |
|
||||
| `WriteTask.lock` | `state`, `reassigned`, `append_chunk` | waiting on `host_ready` / `done` (those are events) |
|
||||
|
||||
### Related Files
|
||||
|
||||
- `vllm_omni/core/prefix_cache/`
|
||||
- `vllm_omni/worker/gpu_model_runner.py` (`_ensure_omni_prefix_cache`)
|
||||
- `tests/core/test_prefix_cache.py`
|
||||
- [Async Omni Output Materialization](omni_async_output_materialization.md)
|
||||
@@ -143,8 +143,20 @@ fraction to select additional resident sessions up to the model-declared cap.
|
||||
All model-owned reservations are deducted before the paged self-attention pool
|
||||
is allocated.
|
||||
|
||||
LingBot reports its persistent full-horizon image-condition tensor. DreamZero
|
||||
reports the measured 603 MiB per-session Wan VAE causal-convolution state.
|
||||
LingBot advances a session-owned causal Wan VAE encoder for each condition
|
||||
block: the opening pixel frame is followed by four zero pixel frames per later
|
||||
latent frame. It retains one current condition block, with separate committed
|
||||
and in-flight encoder histories so a failed block cannot overwrite committed
|
||||
context. The state reservation includes both encoder histories, derived from
|
||||
the encoder's convolution input grids, and the streaming decoder cache. These
|
||||
caches are released through the same close/reset hooks. Condition encoding
|
||||
currently requires the unpatched, non-tiled Wan encoder path; the existing
|
||||
decoder's tiling fallback remains independent. Real-weight GPU parity with a
|
||||
full-horizon encode still requires validation because the pointwise posterior
|
||||
projection now runs per block.
|
||||
|
||||
DreamZero reports the measured 603 MiB per-session Wan VAE causal-convolution
|
||||
state.
|
||||
|
||||
## Concurrency and routing limits
|
||||
|
||||
@@ -178,6 +190,24 @@ For LingBot World v2:
|
||||
- `lingbot_world/pipeline.py` constructs conditioning, owns small non-KV session state, and
|
||||
produces one block plus the standard metadata envelope.
|
||||
|
||||
## Ulysses sequence parallelism
|
||||
|
||||
LingBot supports pure Ulysses sequence parallelism for both direct and
|
||||
AR-Diffusion execution. With Ulysses degree greater than one, hidden tokens,
|
||||
camera features, token-expanded timestep modulation, and RoPE tables are sharded
|
||||
together. Without SP, timestep modulation retains the frame-broadcast path.
|
||||
Self-attention performs the sequence-to-head all-to-all before reading or writing
|
||||
paged KV, while static text K/V uses the same local head shard. Text K/V shards
|
||||
own compact storage; cross-attention exchanges query/output layouts to use these
|
||||
shards. This retains the shared cache geometry and reduces text K/V storage at
|
||||
the cost of two all-to-all calls per layer.
|
||||
|
||||
The output head projects local tokens before gathering the flow values. For
|
||||
the 14B model this reduces the gathered width from 5120 to 64; frame modulation
|
||||
uses each shard's global token offset, including shards that split a frame. Only
|
||||
`ulysses_mode="strict"` is supported; `advanced_uaa`, Ring, and AllGather-KV modes
|
||||
remain unsupported for this model.
|
||||
|
||||
## Non-goals
|
||||
|
||||
This contract does not currently provide:
|
||||
|
||||
@@ -77,7 +77,9 @@ Use it when plain Ulysses-SP would otherwise fail because:
|
||||
|
||||
- `auto_pad=True` pads sequence tokens in `_sp_plan` and requires attention backends that support `attention_mask`.
|
||||
- `advanced_uaa` does not depend on mask-based token padding inside Ulysses attention. It is therefore a better fit for non-divisible head counts and uneven Ulysses shard sizes.
|
||||
- `auto_pad=True` remains incompatible with Ring attention because the ring backend does not consume `attention_mask`.
|
||||
- `auto_pad=True` remains incompatible with Ring attention because the ring backend does not consume arbitrary
|
||||
`attention_mask` values. A model with contiguous suffix padding may instead publish `valid_kv_length`; Ring trims
|
||||
each circulated K/V block against that global valid prefix.
|
||||
- `advanced_uaa` is still experimental and hybrid mode remains limited by Ring's equal-shape requirement.
|
||||
|
||||
---
|
||||
@@ -416,7 +418,7 @@ _sp_plan = {
|
||||
| Constraint | Description |
|
||||
|------------|-------------|
|
||||
| **Attention Backend Compatibility** | The attention backends must support `attention_mask`. Currently only `TORCH_SDPA` and `FLASH_ATTN` (default for diffusion models) are supported. |
|
||||
| **Ring Attention Limitation** | Ring attention does not support `attention_mask`. Therefore, when using `auto_pad=True`, the combination of Ulysses + Ring attention is not feasible. |
|
||||
| **Ring Attention Limitation** | Ring attention does not support arbitrary `attention_mask` values, so `auto_pad=True` remains incompatible. Contiguous suffix padding is supported when the model publishes the global `valid_kv_length` prefix. |
|
||||
|
||||
1. Enable `auto_pad=True` for all sequence-dimension inputs in `_sp_plan`:
|
||||
```python
|
||||
|
||||
@@ -20,7 +20,7 @@ The target is the engine-native path:
|
||||
-> PersonaPlex Talker
|
||||
-> streaming PersonaPlex Code2Wav
|
||||
-> PersonaPlex data-plane projector
|
||||
-> response.audio.delta + response.audio_transcript.delta
|
||||
-> response.output_audio.delta + response.output_audio_transcript.delta
|
||||
```
|
||||
|
||||
The standalone `/api/chat` and `/v1/audio/duplex` server that accompanied the
|
||||
@@ -197,7 +197,7 @@ model-neutral native results:
|
||||
|
||||
It owns per-request audio and text cursors so a cumulative output cannot replay
|
||||
old audio. The generic projector turns each native result into the Realtime
|
||||
`response.audio.delta` / `response.audio_transcript.delta` pair (and
|
||||
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair (and
|
||||
`response.output_text.delta` for text) under one `response_id`; the full
|
||||
mapping is the name map in the
|
||||
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide.
|
||||
@@ -271,3 +271,111 @@ The required evidence is:
|
||||
|
||||
Audio that is empty, all zero, non-finite, or only a protocol `listen` event is
|
||||
not a successful end-to-end result.
|
||||
|
||||
### Paced multi-session load driver
|
||||
|
||||
The existing driver keeps its two-session admission and slot-reuse validation
|
||||
when `--sessions` is omitted. Passing `--sessions N` selects a separate,
|
||||
bounded load mode; it does not replace the lifecycle test or change server
|
||||
admission capacity. Start an existing PersonaPlex server configured to admit
|
||||
the requested number of sessions, then run from a source checkout with the
|
||||
vLLM-Omni client dependencies installed:
|
||||
|
||||
```bash
|
||||
python -m tests.e2e.online_serving.personaplex_realtime_duplex \
|
||||
--model nvidia/personaplex-7b-v1 \
|
||||
--input-wav /path/to/input.wav \
|
||||
--sessions 2 --load-frames 1000 --drain-s 2 \
|
||||
--server-revision SERVER_COMMIT --server-hardware A100-80GB \
|
||||
--output-dir /tmp/personaplex-load-n2
|
||||
```
|
||||
|
||||
Use a new output directory for every run. The driver refuses to overwrite an
|
||||
existing `load-result.json`. The server labels are supplied by the operator,
|
||||
not detected or verified by the client. Record the server command, package
|
||||
versions, checkpoint revision and hardware alongside a published measurement.
|
||||
|
||||
All attempted sessions finish admission before admitted sessions share a
|
||||
common input start. A rejected session remains a failed row in the report;
|
||||
it is not removed from the denominator. Each session receives at most
|
||||
`--load-frames` input frames, without looping the WAV, and uses 80 ms pacing.
|
||||
A stalled send cannot cause a catch-up burst. The timeline records send
|
||||
schedule lateness so an overloaded client is visible rather than mistaken
|
||||
for a faster server. This mode measures steady concurrency, not join/leave
|
||||
interference; the original lifecycle mode still checks slot reuse.
|
||||
Load mode does not probe an additional N+1 session for `resource_exhausted`;
|
||||
the lifecycle mode owns that admission-limit check. A per-session
|
||||
`client_pacing_warning` and the summary's `client_pacing_warning_sessions`
|
||||
identify send-lateness p99 above one nominal frame (80 ms). This is a warning,
|
||||
not an extra acceptance gate: read it alongside server-facing measurements
|
||||
so a slow client is not mistaken for a slow server.
|
||||
|
||||
The fixed drain window is part of the measurement configuration. Audio
|
||||
arriving during cleanup is excluded, so closing cannot rescue a session that
|
||||
missed its observation deadline. Cleanup is attempted for admitted sessions
|
||||
even when audio validation fails. Protocol errors, missing/invalid audio,
|
||||
shared response IDs and cleanup failures make the report fail and the CLI
|
||||
exit nonzero. Cancellation closes transports and propagates to the caller.
|
||||
`--cleanup-timeout-s` (default 5 s) bounds session-close send and acknowledgement
|
||||
together, independently of `--timeout-s`. The subsequent WebSocket close
|
||||
handshake has the same separate bound; an unresponsive close acknowledgement
|
||||
cannot hold cancellation for the full inference timeout.
|
||||
|
||||
`load-result.json` retains per-session send and audio-packet timelines using
|
||||
the client's monotonic clock. It does not export handshake credentials or
|
||||
raw audio payloads. Metric definitions are explicit:
|
||||
|
||||
Both modes validate each nonempty audio packet's strict base64 encoding, PCM16
|
||||
sample alignment, 24 kHz sample rate and response identity. Valid aggregate
|
||||
audio cannot hide a malformed packet. In load mode, rejected packets make the session fail
|
||||
and appear in `invalid_audio_packets` with their event index, receive time and
|
||||
fixed diagnostic code; they do not contribute samples or timing intervals.
|
||||
Empty audio deltas remain legal and contribute no samples. This check does
|
||||
not change the public client's decoding. The lifecycle scenario and dispatch
|
||||
remain the same, but malformed packets that aggregate checks once tolerated
|
||||
now fail that mode as well.
|
||||
|
||||
The probe accepts both the current OpenAI Realtime audio event name
|
||||
`response.output_audio.delta` and the legacy `response.audio.delta`. Validation
|
||||
uses the wire payload directly rather than relying on a particular client
|
||||
library's alias table, so a server-side event-name migration cannot silently
|
||||
turn real audio into a zero-output measurement.
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `client_audio_packet_interval_ms` | Consecutive nonempty packet arrivals: median, linear p99, max, and zero-based index of the first maximal interval. Interval index 0 ends at packet 1. One packet can contain multiple codec frames; this is not server tick latency. |
|
||||
| `client_first_audio_after_stream_start_ms` | First audio receipt minus first input send start; admission is excluded. |
|
||||
| `client_stream_rtf` | First input send start to last audio receipt, divided by received audio duration. This includes real-time input pacing and is not an inference-only RTF. |
|
||||
| `client_send_lateness_ms` | Send start minus its original shared schedule, clipped at zero. |
|
||||
| `frame_deficit` | Integer input-frame minus output-frame count from the shared session validation; missing when valid whole-frame accounting is unavailable. |
|
||||
| `output_frames`, `voiced_frames`, `silent_frames`, `frame_coverage_ratio`, `audio_rms`, `audio_chunks` | Session-validation statistics retained on success and on acceptance failure once computed. |
|
||||
| `acceptance_check` | Failed local check, with numeric diagnostics in `error`; other exceptions retain phase/type only, without remote handshake or metadata text. |
|
||||
| `audio_underrun_s`, `audio_underrun_event_count`, `audio_continuity_ok` | Shared `compute_continuity_stats` on valid PCM16 packet arrivals, with its default 100 ms underrun threshold; absent audio gives missing values rather than a continuity pass. |
|
||||
|
||||
No output gives missing latency/RTF values, not zero. By default RTF is
|
||||
diagnostic; `--max-client-rtf` adds an explicit ceiling. The existing frame
|
||||
coverage, audible-frame and scheduler-data-plane checks still apply. A longer
|
||||
`--drain-s` can demonstrate eventual completion under backlog, but it must not
|
||||
be reported as real-time capacity when the fixed-window run failed or client
|
||||
RTF exceeded the stated target. Likewise, the first live session can pay
|
||||
one-time model/codec/JIT costs; either use the server's configured duplex
|
||||
warmup path or label that run as cold-start evidence before comparing steady
|
||||
capacity. Passing client-only metrics does not certify model quality or a
|
||||
real-time capacity target on another GPU. Begin with N=1 and N=2; higher N is a
|
||||
capacity experiment, not a promised property of an A100 or of this patch.
|
||||
Continuity is a client-arrival diagnostic, not proof of speaker playback and
|
||||
not an additional pass/fail gate. Packet-interval and send-lateness summaries
|
||||
retain their documented linear p99; the older client interval helper reports
|
||||
rounded nearest-rank p95 instead and is not substituted for that schema.
|
||||
|
||||
Deterministic driver and localhost WebSocket checks need no model or GPU:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/e2e/online_serving/test_personaplex_load_driver.py \
|
||||
-m 'core_model and cpu' --run-level core_model -q
|
||||
```
|
||||
|
||||
These checks test the driver against controlled protocol responses. Run the
|
||||
load-driver command against real weights separately before reporting model-serving
|
||||
performance. Server tick timestamps and generic TTFT/TPOT aggregation remain
|
||||
the responsibility of the existing duplex metrics paths, not this driver.
|
||||
+427
-1056
File diff suppressed because it is too large.
Load diff
@@ -26,6 +26,7 @@ implementation contract; it is not, by itself, a general support claim.
|
||||
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
|
||||
- [Runner-to-model Prefill/Decode Phase Contract](feature/preprocess_phase_contract.md)
|
||||
- [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
|
||||
- [Model-local KV Caches](feature/model_local_kv_caches.md)
|
||||
- [Realtime AR-Diffusion Sessions](feature/realtime_ar_diffusion.md)
|
||||
|
||||
### Communication
|
||||
@@ -35,6 +36,7 @@ implementation contract; it is not, by itself, a general support claim.
|
||||
- [Mooncake Store Connector](feature/omni_connectors/mooncake_store_connector.md)
|
||||
- [Mooncake Transfer Engine Connector](feature/omni_connectors/mooncake_transfer_engine_connector.md)
|
||||
- [Mori Transfer Engine Connector](feature/omni_connectors/mori_transfer_engine_connector.md)
|
||||
- [NIXL Connector](feature/omni_connectors/nixl_connector.md)
|
||||
- [Shared Memory Connector](feature/omni_connectors/shared_memory_connector.md)
|
||||
- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
|
||||
- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
|
||||
|
||||
+31
-14
@@ -60,7 +60,7 @@ Upstream per-engine metrics retain the `vllm:` prefix but are now registered by
|
||||
|
||||
There are five independent paths for metric collection.
|
||||
|
||||
**Path 1: Pipeline-level metrics (`vllm_omni:*`)**
|
||||
#### Path 1: Pipeline-level metrics (`vllm_omni:*`)
|
||||
|
||||
`OmniPrometheusMetrics` registers the Gauge / Counter / Histogram collectors at import time. It is instantiated once per entrypoint, labeled with the model name. The entrypoint calls its methods as requests progress:
|
||||
|
||||
@@ -69,7 +69,7 @@ There are five independent paths for metric collection.
|
||||
- `request_failed()` — recorded by the cleanup path when a request exits without natural completion. Internally maps to `finished_reason="abort"` on the existing completion Counter.
|
||||
- `inc_requests_failed(reason)` — increments the dedicated failure-attribution Counter once per request. Reasons are normalized to the bounded set `client_abort`, `client_disconnect`, `stage_error`, and `unknown`.
|
||||
|
||||
**Path 2: Image and diffusion metrics**
|
||||
#### Path 2: Image and diffusion metrics
|
||||
|
||||
Image and diffusion metrics use two cooperating collectors. `OmniPrometheusMetrics` owns pipeline/stage workload families, while `OmniModalityMetrics` owns per-replica diffusion breakdowns. Finished stage messages are consumed once in `OmniBase._process_single_result`, which prevents replayed terminal messages from incrementing Counters or Histograms twice.
|
||||
|
||||
@@ -81,15 +81,15 @@ Image and diffusion metrics use two cooperating collectors. `OmniPrometheusMetri
|
||||
|
||||
Zero and missing values have different meanings. Valid zero-duration queue observations are recorded, while unavailable optional measurements are omitted instead of being exported as synthetic zeroes. `peak_memory_mb`, `diffusion_forward_s`, `diffusion_kv_load_s`, `vae_decode_s`, and `kv_wait_s` therefore appear only when their data source is active.
|
||||
|
||||
**Path 3: Audio modality metrics (`vllm_omni:audio_*`)**
|
||||
#### Path 3: Audio modality metrics (`vllm_omni:audio_*`)
|
||||
|
||||
`OmniModalityMetrics` registers seven audio families with `{model_name, stage, replica}` (plus an extra `threshold_ms` / `reason` label on the two extra-cardinality Counters). Three observation sites:
|
||||
|
||||
- `observe_modality_at_finalize(...)` — called from `omni_base._process_single_result` inside the existing `e2e_done` finalize guard. For `output_type == "audio"` it emits `audio_frames_total`, `audio_duration_s`, `audio_rtf` (or `audio_skipped_requests_total{reason="no_audio_data"}` when no audio was produced). Sample rate is resolved from `engine_outputs.multimodal_output` via `definitions.resolve_audio_sample_rate(...)` (fallback chain mirrors `serving_chat.py`'s audio response path).
|
||||
- `observe_audio_first_packet(...)` — called from the OpenAI SSE audio branch in `serving_chat.py` on the first audio packet for a request. The once-per-request guard is held by `ClientRequestState.first_audio_ts`. The `request_arrival_ts` anchor is stored in `ClientRequestState` by `async_omni.generate()`, computed at request entry.
|
||||
- `observe_audio_streaming_finalize(...)` — called from `serving_chat.py` after the streaming chunk loop exhausts. It runs the per-chunk player simulation from `vllm_omni/benchmarks/audio_continuity.py` to compute the worst-case underrun and emits `audio_underrun_s` plus (when the request stayed below the threshold) `audio_continuity_ok_total{threshold_ms}`. Per-chunk PCM byte counts and arrival timestamps are recorded by the same audio branch that updates `first_audio_ts`.
|
||||
- `observe_audio_first_packet(...)` — called from the OpenAI SSE audio branch in `serving_chat.py` and the streaming TTS output paths in `serving_speech.py` on the first non-empty audio payload for a request. Streaming Speech observes the first PCM payload rather than a header-only WAV chunk. HTTP carries the middleware request timestamp through response generation; WebSocket uses the start of each sentence synthesis request. Non-streaming Speech does not emit TTFP and uses `e2e_request_latency_s` for overall latency instead.
|
||||
- `observe_audio_streaming_finalize(...)` — called from `serving_chat.py` and `serving_speech.py` after a streaming chunk loop exhausts normally. It runs the per-chunk player simulation from `vllm_omni/benchmarks/audio_continuity.py` to compute the worst-case underrun and emits `audio_underrun_s` plus (when the request stayed below the threshold) `audio_continuity_ok_total{threshold_ms}`. Per-chunk PCM byte counts and arrival timestamps are recorded by the same output paths that update TTFP. Speech records raw PCM payloads before raw/SSE/WebSocket encoding, excludes WAV headers and empty chunks, and does not emit continuity samples for failed, cancelled, or non-streaming requests.
|
||||
|
||||
**Path 4: Cross-stage transfer metrics (`vllm_omni:transfer_*`)**
|
||||
#### Path 4: Cross-stage transfer metrics (`vllm_omni:transfer_*`)
|
||||
|
||||
`OmniTransferMetrics` registers four Histogram families with `{model_name, from_stage, from_replica, to_stage, to_replica}` labels. Each observation corresponds to one physical transfer hop (one chunk between adjacent stages), not the per-request accumulated total — so the histograms track per-transfer distribution.
|
||||
|
||||
@@ -100,7 +100,7 @@ The hook lives in `OrchestratorAggregator.record_transfer_tx` and `record_transf
|
||||
|
||||
Defensive fail-safe: if `transfer_emitter` or `replica_resolver` is missing, or the resolver returns `None` for either side, the emit is skipped silently (the underlying `TransferEdgeStats` accumulation is unaffected).
|
||||
|
||||
**Path 5: Per-engine metrics (`vllm:*`, stage/replica wrap)**
|
||||
#### Path 5: Per-engine metrics (`vllm:*`, stage/replica wrap)
|
||||
|
||||
The Orchestrator instantiates `OmniPrometheusStatLogger` (a thin subclass of upstream `vllm.v1.metrics.loggers.PrometheusStatLogger`) and feeds it scheduler stats and iteration stats after processing each batch of engine outputs. This populates the standard ~37 vLLM metric families (TTFT, ITL, TPOT, KV cache usage, etc.) using the same upstream code path — but with the `engine` label reshaped into `stage` + `replica` so multi-replica deployments produce distinct series per replica. See the next section for the wrap mechanics.
|
||||
|
||||
@@ -174,7 +174,7 @@ second.
|
||||
### Pipeline (4)
|
||||
|
||||
| Metric | Type | Labels | Description |
|
||||
|--------|------|--------|-------------|
|
||||
| -------- | ------ | -------- | ------------- |
|
||||
| `vllm_omni:num_requests_running` | Gauge | `model_name` | Requests currently executing across all stages |
|
||||
| `vllm_omni:num_requests_waiting` | Gauge | `model_name` | Requests queued but not yet scheduled |
|
||||
| `vllm_omni:requests_success_total` | Counter | `model_name`, `finished_reason` | Total requests by completion reason ({stop, length, abort, ...}); aborts cover client-disconnect / cancellation paths in addition to upstream `FinishReason.ABORT` |
|
||||
@@ -183,7 +183,7 @@ second.
|
||||
### Image and diffusion service-level metrics
|
||||
|
||||
| Metric | Type | Labels | Description |
|
||||
|--------|------|--------|-------------|
|
||||
| -------- | ------ | -------- | ------------- |
|
||||
| `vllm_omni:stage_gen_time_s` | Histogram | `model_name`, `stage`, `stage_type` | Stage submit to finished output; includes in-stage queueing |
|
||||
| `vllm_omni:request_queue_wait_s` | Histogram | `model_name` | Orchestration-layer queue wait; a present zero is recorded |
|
||||
| `vllm_omni:stage_waiting_requests` | Gauge | `model_name`, `stage` | Sum of the latest waiting snapshots across live replicas in the stage |
|
||||
@@ -200,7 +200,7 @@ second.
|
||||
Labels: `{model_name, stage, replica}`.
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| -------- | ------ | ------------- |
|
||||
| `vllm_omni:diffusion_exec_s` | Histogram | Core diffusion-step execution time per request |
|
||||
| `vllm_omni:diffusion_exec_per_step_s` | Histogram | Core diffusion execution divided by inference-step count |
|
||||
| `vllm_omni:diffusion_preprocess_s` | Histogram | Diffusion input preprocessing time |
|
||||
@@ -218,7 +218,7 @@ The optional breakdown and memory families are sparse by design: when an engine
|
||||
Labels: `{model_name, stage, replica}` plus the listed extra label.
|
||||
|
||||
| Metric | Type | Extra label | Description |
|
||||
|--------|------|-------------|-------------|
|
||||
| -------- | ------ | ------------- | ------------- |
|
||||
| `vllm_omni:audio_ttfp_s` | Histogram | — | Time from request arrival to first audio packet/frame |
|
||||
| `vllm_omni:audio_duration_s` | Histogram | — | Audio content duration (`audio_frames / sample_rate`) |
|
||||
| `vllm_omni:audio_rtf` | Histogram | — | Real-time factor `stage_gen_time_s / audio_duration_s` (SLO `< 1`); uses `RTF_BUCKETS` |
|
||||
@@ -227,12 +227,29 @@ Labels: `{model_name, stage, replica}` plus the listed extra label.
|
||||
| `vllm_omni:audio_continuity_ok_total` | Counter | `threshold_ms` | Incremented when the request's worst underrun stayed below `threshold_ms` |
|
||||
| `vllm_omni:audio_skipped_requests_total` | Counter | `reason` | Silent-loss counter — code2wav rejected malformed codec input and returned `200 OK` with empty audio |
|
||||
|
||||
### Speech streaming (2)
|
||||
|
||||
Labels: `{model_name}` plus the listed extra label.
|
||||
|
||||
| Metric | Type | Extra label | Description |
|
||||
| -------- | ------ | ------------- | ------------- |
|
||||
| `vllm_omni:speech_stream_aborted_total` | Counter | `reason` | Interrupted audio generators, including before first PCM; reasons: `cancelled`, `closed`, `engine_dead`, `error` |
|
||||
| `vllm_omni:speech_stream_completed_total` | Counter | — | Normally completed audio generators; does not confirm client receipt |
|
||||
|
||||
Scope: started Speech audio generators (raw/SSE/WebSocket), counted once per
|
||||
generation; WebSocket counts each sentence. Chat, non-streaming Speech, and
|
||||
failures before generator execution are excluded. Interrupted streams do not
|
||||
emit continuity or underrun samples.
|
||||
|
||||
Interruption ratio: `aborted / (aborted + completed)`, using increments over the
|
||||
same time window and summing all reasons per model.
|
||||
|
||||
### Cross-stage transfer (4)
|
||||
|
||||
Labels: `{model_name, from_stage, from_replica, to_stage, to_replica}`.
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| -------- | ------ | ------------- |
|
||||
| `vllm_omni:transfer_size_bytes` | Histogram | Per-transfer payload size in bytes |
|
||||
| `vllm_omni:transfer_tx_s` | Histogram | Sender-side time (serialize + submit to connector) |
|
||||
| `vllm_omni:transfer_rx_s` | Histogram | Receiver-side time (recv + deserialize) |
|
||||
@@ -245,8 +262,8 @@ After the wrap, every upstream `vllm:*` family — TTFT, ITL, TPOT, e2e latency,
|
||||
## Naming Convention
|
||||
|
||||
- All time-bearing metrics use the `_s` suffix (values in seconds). Two bucket families are used:
|
||||
- `SECONDS_BUCKETS` (0.05 s – 300 s) for e2e / generation / TTFP style values.
|
||||
- `SECONDS_FAST_BUCKETS` (0.001 s – 60 s) for fine-grained cross-stage transfer and audio-underrun values that need millisecond-level resolution.
|
||||
- `SECONDS_BUCKETS` (0.05 s – 300 s) for e2e / generation / TTFP style values.
|
||||
- `SECONDS_FAST_BUCKETS` (0.001 s – 60 s) for fine-grained cross-stage transfer and audio-underrun values that need millisecond-level resolution.
|
||||
- Counters use the `_total` suffix (auto-appended by `prometheus_client`).
|
||||
- Sizes use the `_bytes` suffix.
|
||||
- All omni-specific families are prefixed `vllm_omni:`. The upstream `unregister_vllm_metrics()` function is monkey-patched to a scoped version that still strips upstream `vllm:*` collectors (so multi-engine init within one process does not crash on duplicate registration) but preserves anything prefixed `vllm_omni:` / `vllm_omni`.
|
||||
|
||||
@@ -16,6 +16,7 @@ required_reviewers:
|
||||
- "@yenuo26"
|
||||
- "@NickCao"
|
||||
primary_code_paths:
|
||||
- vllm_omni/engine/omni_engine_base.py
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/engine/async_engine_utils.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
@@ -28,8 +29,12 @@ related_code_paths:
|
||||
- vllm_omni/engine/stage_pool.py
|
||||
- vllm_omni/outputs/output_processor.py
|
||||
- vllm_omni/distributed/omni_coordinator/**
|
||||
# Duplex kernel design is owned by ../fullduplex.md; listed here because the
|
||||
# orchestrator hosts the duplex control plane.
|
||||
# The duplex siblings (DuplexOmniEngine / DuplexOrchestrator) and the
|
||||
# engine-resident session framework are owned by ../fullduplex.md; listed
|
||||
# here because they fill the template seams of OmniEngineBase and
|
||||
# OrchestratorBase.
|
||||
- vllm_omni/engine/duplex_omni_engine.py
|
||||
- vllm_omni/engine/duplex_orchestrator.py
|
||||
- vllm_omni/engine/duplex/**
|
||||
depends_on:
|
||||
- input_output_modality_contracts.md
|
||||
@@ -48,6 +53,8 @@ validation_paths:
|
||||
- tests/engine/test_cfg_companion_lifecycle.py
|
||||
- tests/engine/test_rpc_result_router.py
|
||||
- tests/engine/duplex/**
|
||||
- tests/engine/test_duplex_orchestrator.py
|
||||
- tests/engine/test_duplex_omni_engine.py
|
||||
- tests/engine/test_duplex_import_boundary.py
|
||||
upstream_refs:
|
||||
- vllm.v1.engine.EngineCoreRequest
|
||||
|
||||
@@ -19,7 +19,9 @@ required_reviewers:
|
||||
primary_code_paths:
|
||||
- vllm_omni/entrypoints/omni.py
|
||||
- vllm_omni/entrypoints/async_omni.py
|
||||
- vllm_omni/entrypoints/async_omni_base.py
|
||||
- vllm_omni/entrypoints/omni_base.py
|
||||
- vllm_omni/entrypoints/duplex_omni.py
|
||||
- vllm_omni/entrypoints/cli/**
|
||||
- vllm_omni/entrypoints/openai/**
|
||||
- vllm_omni/entrypoints/openpi/**
|
||||
@@ -27,14 +29,14 @@ primary_code_paths:
|
||||
- vllm_omni/entrypoints/stage_utils.py
|
||||
- vllm_omni/entrypoints/utils.py
|
||||
- vllm_omni/entrypoints/duplex/**
|
||||
- vllm_omni/entrypoints/duplex_request_client.py
|
||||
- vllm_omni/clients/**
|
||||
- vllm_omni/protocol/**
|
||||
primary_path_exceptions:
|
||||
- path: vllm_omni/entrypoints/openai/errors.py
|
||||
owner: error_contracts.md
|
||||
- path: vllm_omni/entrypoints/duplex/**
|
||||
owner: ../fullduplex.md
|
||||
- path: vllm_omni/entrypoints/duplex_request_client.py
|
||||
- path: vllm_omni/entrypoints/duplex_omni.py
|
||||
owner: ../fullduplex.md
|
||||
- path: vllm_omni/clients/**
|
||||
owner: ../fullduplex.md
|
||||
@@ -42,7 +44,9 @@ related_code_paths:
|
||||
- vllm_omni/errors.py
|
||||
- vllm_omni/inputs/**
|
||||
- vllm_omni/outputs/**
|
||||
- vllm_omni/engine/omni_engine_base.py
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/engine/duplex_omni_engine.py
|
||||
- vllm_omni/config/**
|
||||
- vllm_omni/deploy/**
|
||||
depends_on:
|
||||
@@ -50,10 +54,13 @@ depends_on:
|
||||
- error_contracts.md
|
||||
- engine_orchestration.md
|
||||
validation_paths:
|
||||
- tests/protocol/**
|
||||
- tests/entrypoints/test_omni_entrypoints.py
|
||||
- tests/entrypoints/test_async_omni.py
|
||||
- tests/entrypoints/test_async_omni_pause_sleep_routing.py
|
||||
- tests/entrypoints/test_async_omni_duplex.py
|
||||
- tests/entrypoints/test_duplex_omni.py
|
||||
- tests/entrypoints/duplex/**
|
||||
- tests/entrypoints/openai_api/test_duplex_api_server.py
|
||||
- tests/entrypoints/test_serve.py
|
||||
- tests/entrypoints/test_stream_finish_reason.py
|
||||
- tests/entrypoints/openai/**
|
||||
@@ -89,11 +96,43 @@ This document owns offline API semantics, CLI and serve composition,
|
||||
supported OpenAI-compatible routes, request validation and normalization,
|
||||
response conversion, streaming/session behavior, and engine handoff.
|
||||
|
||||
When `--api-server-count` is greater than one, the serve composition root
|
||||
starts the frontend process manager while a parent-owned `StageRuntime`
|
||||
launches the shared local stage engines. Each frontend receives only its own
|
||||
stage-client channel configuration and does not launch or retire backend
|
||||
processes. CLI validation rejects unsupported distributed, diffusion,
|
||||
fault-tolerant, elastic-EP, Ray, and runtime-LoRA combinations before startup.
|
||||
|
||||
It does not own configuration precedence, cross-stage routing, stage
|
||||
placement, payload implementation, or semantic error classification.
|
||||
`entrypoints/openai/errors.py` is an explicit primary-path exception owned by
|
||||
`error_contracts.md`.
|
||||
|
||||
It also owns `vllm_omni/protocol/**`, the shared wire codec promoted by RFC
|
||||
[#6592](https://github.com/vllm-project/vllm-omni/issues/6592) P0a. It follows the three tiers
|
||||
`docs/serving/realtime_duplex_api.md` already defines. `protocol/realtime/**`
|
||||
is Tier 1, identical to OpenAI: 30 server events, 10 client commands, their
|
||||
typed fields and pure wire rendering, audio-format negotiation,
|
||||
conversation-item rules and the error-envelope shape. `protocol/duplex/**` is
|
||||
Tier 2 (OpenAI names carrying vLLM-Omni extensions, each a subclass of its
|
||||
Tier 1 twin declaring only what it adds) and Tier 3 (ours alone, including our
|
||||
error-code vocabulary), kept separate so a GA-only consumer is not handed a
|
||||
vocabulary its clients never send. That package sits
|
||||
outside `entrypoints/` on purpose --- the engine depends on it, so it belongs
|
||||
to neither layer --- but deciding what a Realtime client may put on the wire is
|
||||
this document's subject, and the same reasoning already places
|
||||
`vllm_omni/errors.py`, `inputs/` and `outputs/` under contract documents rather
|
||||
than under a consumer.
|
||||
|
||||
It is deliberately **not** owned by `../fullduplex.md`, even though duplex is
|
||||
the codec's only consumer today. RFC #6592 asks for a named owner precisely so
|
||||
a second Realtime consumer does not need duplex review to change the codec
|
||||
(#6592 open question 8). Ownership here is review scope, not a dependency: the
|
||||
codec may not import the engine, the entrypoints or the model code, which
|
||||
`tests/protocol/realtime/test_protocol_import_boundary.py` enforces. When a
|
||||
second consumer lands (#6592 P0b) the package is expected to graduate to its
|
||||
own module document with owners drawn from both consumers.
|
||||
|
||||
## Candidate invariants
|
||||
|
||||
These identifiers are proposals while the document is `draft`.
|
||||
@@ -103,6 +142,22 @@ These identifiers are proposals while the document is `draft`.
|
||||
**Rule:** Entrypoints MUST NOT implement cross-stage routing or stage lifecycle
|
||||
policy.
|
||||
|
||||
### ENTRY-INV-002: The shared Realtime codec does not depend on its consumers
|
||||
|
||||
**Rule:** `vllm_omni/protocol/**` MUST NOT import `vllm_omni.engine`,
|
||||
`vllm_omni.entrypoints`, `vllm_omni.model_executor`, `vllm_omni.worker` or
|
||||
`vllm_omni.clients`; `protocol/duplex/**` MAY depend on `protocol/realtime/**`
|
||||
but never the reverse, and a Tier 1 class MUST NOT carry a vLLM-Omni extension
|
||||
field; and each protocol/codec behavior MUST have exactly one
|
||||
implementation that every consumer uses. A wire type MUST NOT carry a
|
||||
consumer's internal representation: the duplex mailbox rendering
|
||||
(`DuplexCommand.payload()`, whose channel differs from the client event for
|
||||
`session.update` and the `conversation.item.*` commands) stays engine-side.
|
||||
A duplex consumer (`engine/duplex/**`, `entrypoints/duplex/**`, the duplex
|
||||
clients) MUST import `vllm_omni.protocol.duplex` and MUST NOT import
|
||||
`vllm_omni.protocol.realtime` directly, so the tier boundary has exactly one
|
||||
extension point.
|
||||
|
||||
### ENTRY-INV-100: Public requests are normalized once
|
||||
|
||||
**Rule:** Public protocol values MUST be validated and converted to internal
|
||||
@@ -136,6 +191,9 @@ stage so `wake_up(stage_ids=[0])` does not skip a later `wake_up(stage_ids=[1])`
|
||||
Streaming input pumps take an admission slot immediately before each EngineCore
|
||||
ADD or update, not while waiting for the next client chunk. Frontend abort
|
||||
keeps `request_states` until `generate()` consumes the terminal output.
|
||||
Multi-API serving must preserve one client rank and one channel set per
|
||||
frontend, include every frontend in readiness/failure observation, and keep
|
||||
stage lifecycle ownership in the parent composition root.
|
||||
|
||||
## Promotion gate
|
||||
|
||||
|
||||
@@ -136,8 +136,16 @@ so only when it is marked retryable; unsupported capabilities, identity
|
||||
collisions, producer failures, and publication failures remain visible as
|
||||
startup failures.
|
||||
|
||||
`coordination_timeout_seconds` bounds acquisition of lookup and build locks; it
|
||||
is not a hard wall-clock deadline for in-process validation, production, or
|
||||
`coordination_timeout_seconds` bounds acquisition of the domain-initialization
|
||||
lock and lookup/build locks. Store construction and each later resolution or
|
||||
publication operation receive separate coordination budgets from the same
|
||||
`WaitPolicy`; they do not share an end-to-end startup deadline. A domain-init
|
||||
lock timeout is a retryable domain failure: preferred mode may fall back and
|
||||
required mode fails. Retrying initialization requires constructing a fresh
|
||||
runtime after contention ends; a waiter never cancels or unlocks the owner.
|
||||
|
||||
The coordination budget is not a hard wall-clock deadline for filesystem I/O,
|
||||
in-process validation, production, or
|
||||
atomic publication. Once this V1 implementation becomes the producer, the
|
||||
synchronous producer and publication run to completion. A hung producer blocks
|
||||
its owning process and requires external process supervision; enforceable
|
||||
|
||||
@@ -77,6 +77,22 @@ readiness, one pool per logical stage, replica identity and membership,
|
||||
selection and affinity, stage-client/process lifecycle, liveness, draining,
|
||||
and shutdown.
|
||||
|
||||
Local EngineCore initialization and multi-API serving use
|
||||
`StageRuntime.launch_stage_engines(num_api_servers=1)` as their common launch
|
||||
entry. It resolves plans when not supplied by the runtime's device-group
|
||||
scheduler, coordinates device locks and environment overlays, starts engines,
|
||||
and rolls back resources on startup or attachment failure. Single-API clients
|
||||
attach locally after readiness and take ownership of the engine resources;
|
||||
there is no additional API subprocess. Diffusion and remote attachment retain
|
||||
their backend-specific initialization.
|
||||
|
||||
For local multi-API serving, the parent runtime owns and launches each stage
|
||||
engine once. It supplies a distinct input/output channel for every API
|
||||
frontend and stage replica; frontend workers attach clients to those engines
|
||||
without acquiring stage-process ownership. Readiness observation and shutdown
|
||||
remain parent-owned. This topology currently excludes diffusion and
|
||||
remote/headless stages, intra-stage data parallelism, and Ray backends.
|
||||
|
||||
It does not own cross-stage request policy, model scheduler policy, connector
|
||||
transfer semantics, configuration precedence, or public error rendering.
|
||||
|
||||
@@ -114,6 +130,10 @@ affinity, membership changes, abort routing, process failure, draining, and
|
||||
repeated shutdown. Cross-stage policy changes belong in
|
||||
`engine_orchestration.md`.
|
||||
|
||||
For multi-API changes, additionally exercise per-frontend channel assignment,
|
||||
shared-engine startup handshakes, frontend failure observation, and parent
|
||||
cleanup without duplicate stage shutdown.
|
||||
|
||||
## Promotion gate
|
||||
|
||||
- Reconcile terminology and ownership after #5441 reaches a final state.
|
||||
|
||||
@@ -13,7 +13,6 @@ It can send model inference requests to either a locally running vLLM-Omni servi
|
||||
!!! tip
|
||||
If you run both ComfyUI and vLLM-Omni on the same device, you can create separate virtual environments and use different Python versions for them.
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
Copy the `apps/ComfyUI-vLLM-Omni` folder to the `custom_nodes` subfolder of your ComfyUI installation. Your directory should look like `ComfyUI/custom_nodes/ComfyUI-vLLM-Omni`.
|
||||
@@ -21,9 +20,10 @@ Copy the `apps/ComfyUI-vLLM-Omni` folder to the `custom_nodes` subfolder of your
|
||||
If you are running ComfyUI during copying, you should restart ComfyUI to load this extension.
|
||||
|
||||
!!! tip
|
||||
You can use utility websites such as https://download-directory.github.io/ to download a subdirectory of a repo. Also checkout community discussions (e.g., https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repository) for more info.
|
||||
You can use utility websites such as <https://download-directory.github.io/> to download a subdirectory of a repo. Also checkout community discussions (e.g., <https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repository>) for more info.
|
||||
|
||||
On the device and virtual environment you run ComfyUI, launch ComfyUI with
|
||||
|
||||
```bash
|
||||
cd ComfyUI
|
||||
|
||||
@@ -35,6 +35,7 @@ python main.py --cpu
|
||||
```
|
||||
|
||||
On the device and virtual environment you run vLLM-Omni, start a model service with
|
||||
|
||||
```bash
|
||||
vllm serve The_Model_ID_to_Serve --omni --port 8000
|
||||
```
|
||||
@@ -50,12 +51,27 @@ This extension offers the following nodes based on the output modalities:
|
||||
- **Generate Video** for text-to-video and image-to-video tasks
|
||||
- **Multimodality Understanding** for multimodality-to-text and multimodality-to-audio tasks
|
||||
- **TTS** and **TTS Voice Clone** for TTS tasks
|
||||
- **Generate Music** for text-to-music tasks
|
||||
|
||||
This extension also offers example workflows (at **ComfyUI sidebar -> Templates -> vLLM-Omni**)
|
||||
|
||||
!!! info
|
||||
The node UI and feature designs are intended to match vLLM-Omni online serving interfaces. It cannot offer more than what the interfaces support.
|
||||
|
||||
Every node carries the vLLM-Omni mark in its title bar and is tinted by what it outputs, so a graph is readable at a glance:
|
||||
|
||||
| Colour | Nodes | What they produce |
|
||||
| --- | --- | --- |
|
||||
| Blue | Generate Image, Generate Video, Multimodality Understanding, TTS, TTS Voice Clone | A generated image, video, audio, or text. These are the only nodes that reach a server. |
|
||||
| Amber | AR / Diffusion / Multi-Stage Sampling Params | Sampling parameters that apply to any model |
|
||||
| Purple | Qwen TTS Params, Wan Video Params, MiniMax-H3 Video Params | Parameters that only one model family accepts |
|
||||
| Red | LoRA, FastH3 Deployment | Which weights the server is expected to have loaded |
|
||||
| Teal | Video References | Reference media |
|
||||
|
||||
Recolouring a node by hand (right click -> Colors) overrides its tint, and the choice is kept.
|
||||
|
||||
**Generate Video** takes a clip length in seconds (`duration`), not a frame count. Frames stay the wire unit and are derived with the node's `fps`, so the length is always measured against the rate that is actually served; models that accept only certain frame counts still round to their own lattice server-side. Graphs saved before this widget existed stored `num_frames` in its place and are converted on load, using the fps recorded alongside it -- the browser console names every node it rewrites.
|
||||
|
||||
To build a simple workflow yourself,
|
||||
|
||||
- Drag a generation node onto the canvas.
|
||||
@@ -66,6 +82,18 @@ To build a simple workflow yourself,
|
||||
- For some multi-stage models like BAGEL, [only one stage's sampling parameters are exposed and tunable via vLLM-Omni's online serving API](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/examples/online_serving/bagel/). Thus, these models are treated as single-stage ones. Please check the vLLM-Omni documentation on how to correctly set each model's sampling parameters.
|
||||
- For multi-stage models where all stages are either autoregression or diffusion, you can also connect only a single Sampling Params node, indicating that this set of sampling parameters will be used for all stages.
|
||||
|
||||
## MiniMax H3 text-to-video workflow
|
||||
|
||||
The **MiniMax H3 Text to Video** template uses the existing remote Generate Video,
|
||||
H3 Params, sampling, and Remote LoRA nodes. It includes Base settings and an
|
||||
optional Turbo preset, with native resolution, 24 FPS, and `17k+5` frame counts.
|
||||
ComfyUI receives the generated video and audio from the server; it does not load
|
||||
H3 weights locally. Configure the server-side LoRA path before enabling Turbo.
|
||||
|
||||
See the [H3 workflow guide](https://github.com/vllm-project/vllm-omni/blob/main/apps/ComfyUI-vLLM-Omni/docs/minimax-h3-t2v.md)
|
||||
for server setup, template import, Turbo configuration, recorded validation,
|
||||
and saved-video/audio checks.
|
||||
|
||||
## Examples & Screenshots
|
||||
|
||||
Please read the [ComfyUI integration's README](https://github.com/vllm-project/vllm-omni/tree/main/apps/ComfyUI-vLLM-Omni) for more info.
|
||||
@@ -1,7 +1,7 @@
|
||||
# Custom Pipeline Extension Guide
|
||||
|
||||
Transformer already support Custom Pipeline via
|
||||
https://github.com/huggingface/diffusers/blob/main/docs/source/en/using-diffusers/custom_pipeline_overview.md
|
||||
<https://github.com/huggingface/diffusers/blob/main/docs/source/en/using-diffusers/custom_pipeline_overview.md>
|
||||
|
||||
This guide demonstrates how to use the newly added features for extending vLLM-Omni's diffusion pipeline with custom functionality.
|
||||
|
||||
@@ -20,8 +20,9 @@ Three main features enable custom pipeline extension:
|
||||
`WorkerWrapperBase` is a wrapper class that creates `DiffusionWorker` instances with optional extension support. It enables dynamic inheritance, allowing you to add custom methods and functionality to workers without modifying the base worker class.
|
||||
|
||||
**Key capabilities:**
|
||||
|
||||
- Dynamic worker class extension via `worker_extension_cls`
|
||||
- Support for custom pipeline initialization via `custom_pipeline_args`
|
||||
- Support for custom pipeline initialization via `custom_pipeline_args["pipeline_class"]`
|
||||
- Method delegation to underlying worker
|
||||
- Attribute access forwarding
|
||||
|
||||
@@ -42,6 +43,7 @@ The `diffusion_load_format` parameter controls the initial diffusion model load.
|
||||
`CustomPipelineWorkerExtension` is a mixin class that extends `DiffusionWorker` with the ability to re-initialize the pipeline with a custom implementation.
|
||||
|
||||
**Key method:**
|
||||
|
||||
- `re_init_pipeline(custom_pipeline_args)`: Re-initializes the pipeline with custom arguments, properly cleaning up the old pipeline first
|
||||
|
||||
**Location:** `vllm_omni/diffusion/worker/diffusion_worker.py`
|
||||
@@ -150,6 +152,7 @@ omni = Omni(
|
||||
},
|
||||
worker_extension_cls=MyCustomExtension,
|
||||
# Note: worker_extension_cls is an internal parameter
|
||||
# CustomPipelineWorkerExtension will automatically init pipeline when custom_pipeline_args is provided
|
||||
# CustomPipelineWorkerExtension will automatically init pipeline when
|
||||
# custom_pipeline_args includes "pipeline_class".
|
||||
)
|
||||
```
|
||||
@@ -1,11 +1,23 @@
|
||||
<!-- markdownlint-disable MD001 MD025 -->
|
||||
|
||||
# --8<-- [start:requirements]
|
||||
|
||||
- GPU: Validated on gfx942 (It should be supported on the AMD GPUs that are supported by vLLM.)
|
||||
|
||||
# --8<-- [end:requirements]
|
||||
|
||||
# --8<-- [start:set-up-using-python]
|
||||
|
||||
vLLM-Omni current recommends the steps in under setup through Docker Images.
|
||||
For ROCm, vLLM-Omni currently recommends the setup steps through Docker Images.
|
||||
|
||||
vLLM-Omni depends on the matching major/minor release of vLLM. The 0.29
|
||||
development line uses vLLM 0.29.x. Published 0.28.0 wheels and images use vLLM
|
||||
0.28.x.
|
||||
|
||||
The Dockerfile's `BASE_IMAGE` pin applies only to Docker builds. The
|
||||
`vllm-omni` package does not install vLLM as a dependency, so non-Docker source
|
||||
installs must install the matching ROCm vLLM release explicitly before
|
||||
installing vLLM-Omni, as shown below.
|
||||
|
||||
# --8<-- [start:pre-built-wheels]
|
||||
|
||||
@@ -14,6 +26,7 @@ vLLM-Omni current recommends the steps in under setup through Docker Images.
|
||||
These pre-built wheel instructions install the published vLLM-Omni 0.28.0 release. For the 0.29 development line, use the source-install instructions below.
|
||||
|
||||
vLLM-Omni is built based on vLLM. Please install it with command below.
|
||||
|
||||
```bash
|
||||
uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
|
||||
```
|
||||
@@ -36,6 +49,7 @@ uv pip install onnxruntime-rocm
|
||||
# --8<-- [start:build-wheel-from-source]
|
||||
|
||||
#### Installation of vLLM
|
||||
|
||||
If you do not need to modify source code of vLLM, you can directly install the stable 0.29.0 release version of the library
|
||||
|
||||
```bash
|
||||
@@ -45,7 +59,9 @@ uv pip install vllm==0.29.0+rocm723 --extra-index-url https://wheels.vllm.ai/roc
|
||||
The pre-built 0.29.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
|
||||
|
||||
#### Installation of vLLM-Omni
|
||||
|
||||
Since vllm-omni is rapidly evolving, it's recommended to install it from source
|
||||
|
||||
```bash
|
||||
git clone https://github.com/vllm-project/vllm-omni.git
|
||||
cd vllm-omni
|
||||
@@ -71,15 +87,31 @@ python3 setup.py develop
|
||||
|
||||
#### Build docker image
|
||||
|
||||
The source build defaults to the published upstream base image
|
||||
`vllm/vllm-openai-rocm:v0.29.0`, aligned with the vLLM release used by CI.
|
||||
Older or custom bases must provide the vLLM APIs checked by the Dockerfile's
|
||||
image-build canary. This upstream base is distinct from the prebuilt
|
||||
`vllm/vllm-omni-rocm` images discussed below; their published-tag availability
|
||||
is tracked in [#7405](https://github.com/vllm-project/vllm-omni/issues/7405).
|
||||
|
||||
```bash
|
||||
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm-omni-rocm .
|
||||
```
|
||||
|
||||
To select the upstream ROCm base explicitly, pass `BASE_IMAGE`:
|
||||
|
||||
```bash
|
||||
DOCKER_BUILDKIT=1 docker build \
|
||||
-f docker/Dockerfile.rocm \
|
||||
--build-arg BASE_IMAGE=vllm/vllm-openai-rocm:v0.29.0 \
|
||||
-t vllm-omni-rocm .
|
||||
```
|
||||
|
||||
#### Launch the docker image
|
||||
|
||||
##### Launch with OpenAI API Server
|
||||
|
||||
```
|
||||
```bash
|
||||
docker run --rm \
|
||||
--group-add=video \
|
||||
--ipc=host \
|
||||
@@ -97,7 +129,7 @@ vllm-omni-rocm \
|
||||
|
||||
##### Launch with interactive session for development
|
||||
|
||||
```
|
||||
```bash
|
||||
docker run --rm -it \
|
||||
--network=host \
|
||||
--group-add=video \
|
||||
@@ -119,7 +151,9 @@ vllm-omni-rocm
|
||||
vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as [vllm/vllm-omni-rocm](https://hub.docker.com/r/vllm/vllm-omni-rocm/tags). The version of vLLM-Omni indicates which release of vLLM it is based on.
|
||||
|
||||
#### Launch vLLM-Omni Server
|
||||
|
||||
Here's an example deployment command that has been verified on 2 x MI300's:
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
--group-add=video \
|
||||
@@ -136,8 +170,10 @@ docker run --rm \
|
||||
--model Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
|
||||
#### Launch an interactive terminal with prebuilt docker image.
|
||||
#### Launch an interactive terminal with prebuilt docker image
|
||||
|
||||
If you want to run in dev environment you can launch the docker image as follows:
|
||||
|
||||
```bash
|
||||
docker run --rm -it \
|
||||
--network=host \
|
||||
|
||||
@@ -10,25 +10,25 @@ The recommended way to use vLLM-Omni on NPU is through the vLLM-Ascend pre-built
|
||||
=== "A2"
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0
|
||||
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.29.0
|
||||
```
|
||||
|
||||
=== "A3"
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0-a3
|
||||
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.29.0-a3
|
||||
```
|
||||
|
||||
=== "A5"
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0-a5
|
||||
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.29.0-a5
|
||||
```
|
||||
|
||||
=== "310P (Experimental)"
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0-310p
|
||||
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.29.0-310p
|
||||
```
|
||||
|
||||
```bash
|
||||
@@ -53,7 +53,7 @@ docker run --rm \
|
||||
|
||||
# Inside the container, install vLLM-Omni from source
|
||||
cd /vllm-workspace
|
||||
git clone -b v0.28.0 https://github.com/vllm-project/vllm-omni.git
|
||||
git clone -b v0.29.0rc1 https://github.com/vllm-project/vllm-omni.git
|
||||
cd vllm-omni
|
||||
pip install -v -e . --no-build-isolation
|
||||
# or VLLM_OMNI_TARGET_DEVICE=npu pip install -v -e .
|
||||
@@ -71,10 +71,10 @@ We are keeping [issue #886](https://github.com/vllm-project/vllm-omni/issues/886
|
||||
|
||||
# --8<-- [start:installation-main]
|
||||
|
||||
You can also install vLLM-Omni from the latest main branch if you want to use the latest features or bug fixes. Use the aligned vLLM-Ascend v0.28.0 image described above, and check [issue #886](https://github.com/vllm-project/vllm-omni/issues/886) for the status of the latest vLLM-Omni commit on NPU.
|
||||
You can also install vLLM-Omni from the latest main branch if you want to use the latest features or bug fixes. Use the aligned vLLM-Ascend v0.29.0 image described above, and check [issue #886](https://github.com/vllm-project/vllm-omni/issues/886) for the status of the latest vLLM-Omni commit on NPU.
|
||||
|
||||
```bash
|
||||
# Inside the vLLM-Ascend v0.28.0 container, install vLLM-Omni from main
|
||||
# Inside the vLLM-Ascend v0.29.0 container, install vLLM-Omni from main
|
||||
cd /vllm-workspace
|
||||
git clone https://github.com/vllm-project/vllm-omni.git
|
||||
cd vllm-omni
|
||||
@@ -103,7 +103,7 @@ Here's an example deployment command that has been verified on 4 x NPUs:
|
||||
=== "A2"
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0
|
||||
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0
|
||||
docker run --rm \
|
||||
--name vllm-omni-a2 \
|
||||
--shm-size=64g \
|
||||
@@ -127,7 +127,7 @@ Here's an example deployment command that has been verified on 4 x NPUs:
|
||||
=== "A3"
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0-a3
|
||||
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0-a3
|
||||
docker run --rm \
|
||||
--name vllm-omni-a3 \
|
||||
--shm-size=64g \
|
||||
@@ -151,7 +151,7 @@ Here's an example deployment command that has been verified on 4 x NPUs:
|
||||
=== "A5"
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0-a5
|
||||
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0-a5
|
||||
docker run --rm \
|
||||
--name vllm-omni-a5 \
|
||||
--shm-size=64g \
|
||||
@@ -175,7 +175,7 @@ Here's an example deployment command that has been verified on 4 x NPUs:
|
||||
=== "310P (Experimental)"
|
||||
|
||||
```bash
|
||||
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0-310p
|
||||
export IMAGE=quay.io/ascend/vllm-omni:v0.29.0-310p
|
||||
docker run --rm \
|
||||
--name vllm-omni-310p \
|
||||
--shm-size=64g \
|
||||
|
||||
@@ -25,8 +25,8 @@ CATEGORIES = {
|
||||
"clients": {
|
||||
"name": "Clients",
|
||||
"description": (
|
||||
"Client libraries that talk to a running vLLM-Omni server "
|
||||
"(the full-duplex Realtime client and its per-model session presets)."
|
||||
"Client libraries for full-duplex sessions: the Realtime WebSocket client, "
|
||||
"the in-process client over `DuplexOmni`, and the per-model session presets."
|
||||
),
|
||||
},
|
||||
"inputs": {
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import ast
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
@@ -124,6 +125,7 @@ def extract_omni_serve_subparser_init():
|
||||
"make_arg_parser": lambda parser: parser, # no-op for doc
|
||||
"_ensure_vllm_platform": lambda: None, # no-op for doc
|
||||
"_nonneg_finite_float": float,
|
||||
"_json_object": json.loads,
|
||||
"_parse_stage_overrides": __import__("json").loads,
|
||||
"VLLM_SUBCMD_PARSER_EPILOG": "",
|
||||
"logger": logger,
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Reduce browser search-index memory for generated API reference pages."""
|
||||
|
||||
from material.plugins.search.plugin import SearchIndex
|
||||
|
||||
@@ -59,6 +59,7 @@ th {
|
||||
| `Magi2Pipeline` | MAGI-2 native Preview-stage T2VA and I2VA (272p/540p) | `sand-ai/MAGI-2-preview` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SandAI/MAGI-2-preview-L20X.md) |
|
||||
| `DreamZeroPipeline` | DreamZero-DROID | `GEAR-Dreams/DreamZero-DROID` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `Pi0Pipeline` | π0 (Pi-Zero) VLA | `lerobot/pi0_base` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/lerobot/Pi0.md) |
|
||||
| `Pi05Pipeline` | π0.5 VLA | `lerobot/pi05_base` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/lerobot/Pi05.md) |
|
||||
| `HeliosPipeline`, `HeliosPyramidPipeline` | Helios | `BestWishYsh/Helios-Base`, `BestWishYsh/Helios-Mid`, `BestWishYsh/Helios-Distilled` | ✅︎ | ✅︎ | ✅︎ | | — |
|
||||
| `OvisImagePipeline` | Ovis-Image | `OvisAI/Ovis-Image` | ✅︎ | ✅︎ | | ✅︎ | — |
|
||||
| `LongcatImagePipeline` | LongCat-Image | `meituan-longcat/LongCat-Image` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
@@ -69,9 +70,10 @@ th {
|
||||
| `StableDiffusionXLPipeline` | Stable-Diffusion-XL | `stabilityai/stable-diffusion-xl-base-1.0` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ | — |
|
||||
| `CosyVoice3Model` | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✅︎ | ✅︎ | | ✅︎ | — |
|
||||
| `AuKForConditionalGeneration` | AuK / AuK-Flash (instruction-driven speech generation and editing) | `tencent/AuK`, `tencent/AuK-Flash` (with `Qwen/Qwen2.5-Omni-3B`; assemble with `tools/prepare_auk_checkpoint.py`) | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/AuK-H100.md) |
|
||||
| `OmniVoiceModel` | OmniVoice | `k2-fsa/OmniVoice` | ✅︎ | | | ✅︎ | — |
|
||||
| `VoxCPM2TalkerForConditionalGeneration` | VoxCPM2 | `openbmb/VoxCPM2` | ✅︎ | | | ✅︎ | — |
|
||||
| `DotsTTSForConditionalGeneration` | dots.tts | `dots-studio/dots.tts-soar` | ✅︎ | | | | — |
|
||||
| `BreezeForConditionalGeneration` | Breeze-TTS-2 (TTS, voice clone, voice direction) | `BreezeBlue/Breeze-TTS-2` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md) |
|
||||
| `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | | — |
|
||||
| `MammothModa2ForConditionalGeneration` | MammothModa2-Dev (AR-only image understanding) | `bytedance-research/MammothModa2-Dev` | ✅︎ | | | | — |
|
||||
| `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
@@ -86,7 +88,7 @@ th {
|
||||
| `Qwen3TTSForConditionalGeneration` | Qwen3-TTS-12Hz-1.7B-Base | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `MingTTSForConditionalGeneration` | Ming-omni-tts dense 0.5B | `inclusionAI/Ming-omni-tts-0.5B` | ✅︎ | | | ✅︎ | — |
|
||||
| `GLMTTSForConditionalGeneration` | GLM-TTS | `zai-org/GLM-TTS` | ✅︎ | | | | — |
|
||||
| `MossTTSNanoForCausalLM` | MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | ✅︎ | | | | — |
|
||||
| `MossTTSNanoForCausalLM` | MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenMOSS/MOSS-TTS-Nano-NPU.md) |
|
||||
| `MossTTSDelayModel` | MOSS-TTS, MOSS-TTSD, MOSS-SoundEffect, MOSS-VoiceGenerator | `OpenMOSS-Team/MOSS-VoiceGenerator` | ✅︎ | | | | — |
|
||||
| `MossTTSRealtime` | MOSS-TTS-Realtime | `OpenMOSS-Team/MOSS-TTS-Realtime` | ✅︎ | | | | — |
|
||||
| `HiggsAudioV2ForConditionalGeneration` | Higgs-Audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✅︎ | | | | — |
|
||||
@@ -105,12 +107,12 @@ th {
|
||||
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | | — |
|
||||
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
|
||||
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
|
||||
| `DyninOmniForConditionalGeneration` | Dynin-Omni | `snu-aidas/Dynin-Omni` | ✅︎ | | | ✅︎ | — |
|
||||
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | — |
|
||||
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) |
|
||||
| `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — |
|
||||
| `HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
|
||||
| `HiDreamO1ImagePipeline` | HiDream-O1-Image | `HiDream-ai/HiDream-O1-Image` | ✅︎ | | | | — |
|
||||
| `AnimaPipeline` | Anima | `circlestone-labs/Anima` | ✅︎ | ✅︎ | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/circlestone-labs/Anima.md) |
|
||||
|
||||
The `Recipe` column links the corresponding deployment recipe. `Published` is
|
||||
preferred when a corresponding `recipes.vllm.ai` page exists; otherwise the
|
||||
|
||||
+12
-5
@@ -145,15 +145,22 @@ interchangeably.
|
||||
| Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) |
|
||||
| Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) |
|
||||
| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) |
|
||||
| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` or `WS /v1/duplex` | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) |
|
||||
| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`) | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) |
|
||||
| Generated video chunks | `WS /v1/realtime/video` | Start a diffusion request and receive fragmented MP4 | [Streaming Video Output](streaming_video_output_api.md) |
|
||||
| Robot policy inference | `WS /v1/realtime/robot/openpi` | Send MessagePack observations and receive action arrays | [OpenPI Robot Policy](openpi_api.md) |
|
||||
|
||||
All six routes are model- or configuration-dependent. In particular,
|
||||
`/v1/realtime` is not full duplex unless the client sets `duplex=1` and the
|
||||
deployment explicitly enables duplex sessions. Clients should also verify the
|
||||
duplex capability payload because the query-parameter form falls back to the
|
||||
ordinary realtime handler when duplex is unavailable.
|
||||
`/v1/realtime` is full duplex when the model is a duplex model (its pipeline
|
||||
declares a `duplex_plugin` and the deploy configuration sets
|
||||
`session_mode: duplex`); a stock Realtime client needs no vendor query
|
||||
parameter, and `duplex=0` is the explicit opt-out. Such a server serves the
|
||||
websocket route plus `POST /v1/chat/completions`, and no other turn-based
|
||||
HTTP route. On a duplex server that chat route is not the turn-based path:
|
||||
each request runs on a short-lived duplex session, so it holds one of
|
||||
`duplex_session.max_sessions` for its lifetime and answers at the model's
|
||||
real-time pace -- see [Full Duplex](full_duplex_api.md). Clients should
|
||||
also verify the duplex capability payload because the query-parameter form
|
||||
falls back to the ordinary realtime handler when duplex is unavailable.
|
||||
|
||||
## Related Endpoints
|
||||
|
||||
|
||||
@@ -15,40 +15,68 @@ capability gates, see the [Realtime Duplex API guide](realtime_duplex_api.md).
|
||||
|
||||
| Endpoint | Protocol | Recommended use |
|
||||
| --- | --- | --- |
|
||||
| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients |
|
||||
| `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing |
|
||||
|
||||
Both endpoints use the same duplex engine and require the same model-side
|
||||
adapter. Prefer the Realtime projection unless the native lifecycle events are
|
||||
specifically required.
|
||||
| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style events (the normative contract) | Applications and browser clients |
|
||||
| `WS /v1/duplex` | Alias of `/v1/realtime?duplex=1` (same protocol, same handler) | Clients that prefer a dedicated path |
|
||||
| Python: `DuplexOmni` / `InlineDuplexClient` | Typed commands and events in-process | Embedding the model without a server |
|
||||
|
||||
## Enable Full Duplex
|
||||
|
||||
The route becomes usable only when the deployment configuration explicitly
|
||||
sets:
|
||||
A model is served full duplex when its registered pipeline declares a
|
||||
`duplex_plugin` (the model's `DuplexModelPlugin`) and its deploy configuration
|
||||
sets `session_mode: duplex`. `vllm serve --omni` constructs `DuplexOmni`
|
||||
instead of `AsyncOmni`. It serves
|
||||
`/v1/realtime?duplex=1` (and its alias `/v1/duplex`),
|
||||
`POST /v1/chat/completions`, `/v1/models` and `/health`; every other
|
||||
turn-based HTTP route (speech, batch, embeddings, video, ...) answers "not
|
||||
available". Set `session_mode: turn` to select the ordinary online serving
|
||||
stack instead. The model's supported tasks and endpoint restrictions still apply.
|
||||
|
||||
`/v1/chat/completions` is the ordinary chat service running on the duplex
|
||||
engine: a request is a turn-based generation on the same stages, served
|
||||
alongside the live websocket sessions, with the request options the chat
|
||||
service supports. It is wired only when the model's plugin declares
|
||||
`DuplexCapabilities.supports_chat_completions`; on any other duplex model the
|
||||
route reports "not available". A model that should not serve the route at all
|
||||
lists it in the deploy configuration's `endpoint_restrictions`.
|
||||
|
||||
The deploy configuration of such a model must agree:
|
||||
|
||||
```yaml
|
||||
session_mode: duplex
|
||||
```
|
||||
|
||||
The selected model pipeline must also provide a duplex serving adapter.
|
||||
Model-native deployments configure the corresponding engine runtime extension
|
||||
and control plane as part of their registered pipeline.
|
||||
A duplex-capable model must explicitly set `session_mode` to `duplex` or `turn`
|
||||
in its deploy configuration, possibly through `base_config` inheritance.
|
||||
Missing or invalid values fail at startup. The default MiniCPM-o deployment
|
||||
continues to use duplex mode.
|
||||
|
||||
To run MiniCPM-o 4.5 with the turn-based engine:
|
||||
|
||||
```bash
|
||||
vllm serve openbmb/MiniCPM-o-4_5 --omni \
|
||||
--deploy-config vllm_omni/deploy/minicpmo_4_5_turn.yaml \
|
||||
--trust-remote-code \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
This profile inherits the default model and stage settings and overrides only
|
||||
`session_mode`. It supports ordinary HTTP requests without creating a duplex
|
||||
session handler. Mode selection happens at startup; a WebSocket query parameter
|
||||
does not switch engines. The Python `Omni` / `AsyncOmni` APIs are unchanged.
|
||||
|
||||
!!! warning
|
||||
|
||||
`WS /v1/duplex` fails with `Duplex API is not available` when duplex is not
|
||||
enabled. By contrast, `/v1/realtime?duplex=1` falls back to the ordinary
|
||||
turn-based realtime handler when no duplex handler exists. Confirm that
|
||||
On a deployment that is *not* duplex, `WS /v1/duplex` fails with
|
||||
`Duplex API is not available` and `/v1/realtime?duplex=1` falls back to the
|
||||
ordinary turn-based realtime handler. Confirm that
|
||||
`session.created.session.capabilities` is present before treating the
|
||||
connection as full duplex.
|
||||
|
||||
The current unified-runtime integrations are:
|
||||
|
||||
- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5.yaml`
|
||||
(`session_mode: duplex`);
|
||||
- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex;
|
||||
- Nemotron VoiceChat, via its registered duplex plugin package.
|
||||
**MiniCPM-o 4.5** (`vllm_omni/deploy/minicpmo_4_5.yaml`) is the only model
|
||||
served over this endpoint today. PersonaPlex and Nemotron VoiceChat still carry
|
||||
their pre-framework duplex code: their pipelines declare no `duplex_plugin`, so
|
||||
they run turn-based until the follow-up PRs port them to the plugin contract
|
||||
(RFC [vllm-omni#7181](https://github.com/vllm-project/vllm-omni/issues/7181)).
|
||||
|
||||
JoyVL is a separate HTTP interaction orchestrator and does not use these
|
||||
WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
|
||||
@@ -86,7 +114,7 @@ A typical `/v1/realtime?duplex=1` session follows this lifecycle:
|
||||
3. Send `input_audio_buffer.append` events while microphone audio arrives.
|
||||
4. Send `input_audio_buffer.commit` at a user-turn boundary when required by
|
||||
the model policy.
|
||||
5. Consume `response.created`, transcript deltas, `response.audio.delta`, and
|
||||
5. Consume `response.created`, transcript deltas, `response.output_audio.delta`, and
|
||||
`response.done` or `response.listen` events.
|
||||
6. Send `playback.ack` after audio has been played when the session advertises
|
||||
playback acknowledgement support.
|
||||
@@ -101,22 +129,23 @@ deferred, treated as a short acknowledgement, or used to interrupt output.
|
||||
The `session.created` payload includes capability fields such as
|
||||
`supports_barge_in`, `supports_playback_ack`, `supports_multi_session`,
|
||||
`supports_session_resume`, and `chunk_period_ms`. Treat this payload as the
|
||||
runtime contract.
|
||||
runtime contract and branch on the flags, never on the model name: a model
|
||||
that supports native overlapping speech may still advertise
|
||||
`supports_barge_in=false` when destructive output interruption and model-state
|
||||
rewind are not validated for it. Capacity and session-resume behavior also
|
||||
depend on the selected deployment configuration. The per-model table lives in
|
||||
the [Realtime Duplex API guide](realtime_duplex_api.md#capability-negotiation-by-model).
|
||||
|
||||
For example, PersonaPlex supports native overlapping speech but currently
|
||||
advertises `supports_barge_in=false`; destructive output interruption and
|
||||
model-state rewind have not been validated for that integration. Capacity and
|
||||
session-resume behavior also depend on the selected deployment configuration.
|
||||
## Python API
|
||||
|
||||
## Native Protocol
|
||||
`vllm_omni.entrypoints.duplex_omni.DuplexOmni` runs the same engine-resident
|
||||
sessions in-process: `open_session()` returns a `DuplexSessionHandle` whose
|
||||
`submit()` takes typed `DuplexCommand` objects and whose `events()` yields
|
||||
typed `DuplexEvent` objects (each with a `to_realtime()` wire rendering).
|
||||
`vllm_omni.clients.inline_duplex.InlineDuplexClient` exposes that handle
|
||||
behind the `DuplexClient` API. See the
|
||||
[Realtime Duplex API guide](realtime_duplex_api.md#using-the-python-api).
|
||||
|
||||
`WS /v1/duplex` exposes lower-level lifecycle names including
|
||||
`session.create`, `input_audio_buffer.append`, `turn.signal`, `playback.ack`,
|
||||
and `session.close`. It returns native session, input, response, overlap, and
|
||||
error events. This protocol is experimental and may evolve with the runtime;
|
||||
applications should use the provided Realtime client where possible.
|
||||
|
||||
See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo),
|
||||
[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex),
|
||||
See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo)
|
||||
and the [full-duplex runtime design](../design/fullduplex.md)
|
||||
for model-specific validation and architecture details.
|
||||
@@ -43,8 +43,8 @@ event is base64-encoded raw PCM16.
|
||||
| Client to server | `input_audio_buffer.commit` with `final: true` | Marks the end of input |
|
||||
| Server to client | `transcription.delta` | Carries incremental response text |
|
||||
| Server to client | `transcription.done` | Carries final text and token usage |
|
||||
| Server to client | `response.audio.delta` | Carries incremental PCM16 response audio |
|
||||
| Server to client | `response.audio.done` | Marks the end of response audio |
|
||||
| Server to client | `response.output_audio.delta` | Carries incremental PCM16 response audio |
|
||||
| Server to client | `response.output_audio.done` | Marks the end of response audio |
|
||||
| Server to client | `error` | Reports an invalid event, model, or audio payload |
|
||||
|
||||
A minimal client sends events in this order:
|
||||
@@ -63,7 +63,7 @@ engine is consuming the stream.
|
||||
## Audio Handling
|
||||
|
||||
- Input is mono PCM16 at 16 kHz for the Qwen3-Omni example.
|
||||
- `response.audio.delta.audio` contains base64-encoded PCM16 bytes.
|
||||
- `response.output_audio.delta.audio` contains base64-encoded PCM16 bytes.
|
||||
- Read `sample_rate_hz` from each audio event instead of assuming an output
|
||||
rate. Qwen3-Omni output is typically 24 kHz.
|
||||
- Concatenate audio deltas in receive order to construct the output waveform.
|
||||
|
||||
+295
-144
@@ -1,17 +1,47 @@
|
||||
# Realtime Duplex API
|
||||
|
||||
vLLM-Omni serves full-duplex speech models over a WebSocket endpoint,
|
||||
`/v1/realtime`, that speaks the OpenAI Realtime vocabulary plus a
|
||||
`/v1/realtime?duplex=1`, that speaks the OpenAI Realtime vocabulary plus a
|
||||
set of duplex extensions: the model decides when to listen and when to speak,
|
||||
the user can talk over the assistant, playback progress is reported back so
|
||||
history stays honest, and a dropped connection can resume the same session.
|
||||
This page covers how to run a duplex deployment, how to drive it from Python
|
||||
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
|
||||
|
||||
The handler is enabled by either a model-native `session_mode: duplex`
|
||||
deployment or an explicit `duplex_session` configuration for turn-based
|
||||
Server VAD. The runtime architecture is described in
|
||||
[Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
The endpoint is served for models that ship a duplex plugin — including
|
||||
MiniCPM-o 4.5 and Qwen3-Omni — and just for deploy configurations that declare
|
||||
`session_mode: duplex`. PersonaPlex and Nemotron VoiceChat are ported to the
|
||||
plugin contract in follow-up PRs and are not served over this endpoint yet. The runtime architecture
|
||||
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
|
||||
## Qwen3-Omni conversation history
|
||||
|
||||
Qwen3-Omni keeps incoming conversation items independently addressable by item
|
||||
ID. Before applying the model's chat template, it merges consecutive user
|
||||
items into one multimodal user message. An assistant message, including an
|
||||
empty message reserved for an interrupted or unheard answer, separates turns.
|
||||
The final user message places media before text instructions, following the
|
||||
[official Qwen3-Omni demo's `format_history()`](https://github.com/QwenLM/Qwen3-Omni/blob/e4235853125589c789f06a2dd83e9f4126df5e9d/web_demo.py#L84-L179).
|
||||
|
||||
A camera image or text item may arrive after an audio commit and before
|
||||
`response.create`. Those items belong to the same pending user turn: the
|
||||
committed audio remains associated with its history item even when that item
|
||||
is no longer last. Prompt preparation snapshots the selected inputs; later
|
||||
items do not modify an already prepared request.
|
||||
|
||||
The duplex adapter retains its existing budgets: at most four audio payloads
|
||||
(including the current input), an 8 MiB encoded-audio history budget (keeping
|
||||
at least the newest payload), and at most eight prompt images. The recent
|
||||
16-message window is rounded outwards to a complete turn. If any audio in an
|
||||
older turn is no longer retained, the entire turn's model context, including
|
||||
its images/text and assistant replies, is omitted. Images from that turn are
|
||||
therefore **not** permanent session-wide model context. These limits differ
|
||||
from the demo's one image-bearing turn and five audio-bearing turns; the
|
||||
shared rule is grouping before pruning, rather than copying its UI budgets.
|
||||
|
||||
Grouping and pruning affect only the model-input view. They do not merge or
|
||||
delete the source conversation items exposed to clients, and do not change
|
||||
playback acknowledgement or interruption handling.
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -26,11 +56,24 @@ vllm-omni serve openbmb/MiniCPM-o-4_5 \
|
||||
```
|
||||
|
||||
`vllm_omni/deploy/minicpmo_4_5.yaml` declares `session_mode: duplex` and
|
||||
`duplex_session.max_sessions: 4`, which mounts the WebSocket routes
|
||||
`ws://<host>:8099/v1/realtime` (this page) and `ws://<host>:8099/v1/duplex`
|
||||
(the native dialect for raw-protocol clients) next to the usual HTTP API.
|
||||
PersonaPlex and Nemotron VoiceChat use `vllm_omni/deploy/personaplex.yaml` and
|
||||
`vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml` in the same way.
|
||||
`duplex_session.max_sessions: 4`. Because the MiniCPM-o 4.5 pipeline declares a
|
||||
`duplex_plugin`, `vllm-omni serve` runs it through `DuplexOmni`: the server
|
||||
mounts `ws://<host>:8099/v1/realtime?duplex=1` (this page; `ws://<host>:8099/v1/duplex`
|
||||
is an alias of the same route), `POST /v1/chat/completions`, `/v1/models` and
|
||||
`/health`, and nothing else. The remaining turn-based HTTP routes (speech,
|
||||
batch, embeddings, video, ...) are not served in duplex mode.
|
||||
|
||||
`/v1/chat/completions` uses the ordinary chat service on the same engine
|
||||
when the plugin declares `supports_chat_completions`. HTTP chat requests do
|
||||
not create duplex sessions or consume session admission slots. Their options
|
||||
follow the [Chat Completions API](chat_completions_api.md) and the model's
|
||||
capabilities and endpoint restrictions.
|
||||
|
||||
For a turn-based online deployment, use `minicpmo_4_5_turn.yaml` or explicitly
|
||||
set `session_mode: turn` in a deploy configuration derived from the model's
|
||||
profile. This selects `AsyncOmni` and the ordinary API initialization instead
|
||||
of the duplex handler. See [Full Duplex](full_duplex_api.md#enable-full-duplex)
|
||||
for the command and configuration requirements.
|
||||
|
||||
### Run the Example Client
|
||||
|
||||
@@ -81,25 +124,24 @@ ships a preset:
|
||||
|
||||
| Preset | Input / output audio | What it sets |
|
||||
| --- | --- | --- |
|
||||
| `vllm_omni.clients.minicpmo_4_5.create_duplex_session_config(ref_audio=...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | `extra_body.native_duplex=True` (opt in to the model-native lane), `force_listen_count=0`, `overlap_policy="listen_only"`, `playback_commit_policy="ack_only"`; `ref_audio` is the assistant voice clip |
|
||||
| `vllm_omni.clients.minicpmo_4_5.create_duplex_session_config(ref_audio=...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | `force_listen_count=0`, `overlap_policy="listen_only"`, `playback_commit_policy="ack_only"`; `ref_audio` is the assistant voice clip |
|
||||
| `vllm_omni.clients.personaplex.create_duplex_session_config(voice="NATF2.pt", persona="")` | 24 kHz `pcm_f32le` / 24 kHz `pcm16` | bundled `.pt` voice prompt and the persona as `instructions` |
|
||||
| `SessionConfig(...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | model-neutral defaults; pass `extra_body`, `turn_detection`, `overlap_policy`, `playback_commit_policy`, `instructions`, `voice`, `temperature` yourself |
|
||||
|
||||
Keyword overrides on a preset replace the corresponding `SessionConfig`
|
||||
field, e.g. `create_duplex_session_config(ref_audio=..., temperature=0.6)`.
|
||||
|
||||
Constructor options: `session_id` (names the session the client creates; it
|
||||
does not resume or take over an existing session — entering the client always
|
||||
performs the `session.update` handshake), `reconnect` (a `ReconnectPolicy`,
|
||||
default five attempts with 0.25–4 s jittered backoff; `None` disables
|
||||
auto-resume), `heartbeat_interval_s` (default 30 s; `None` disables the lease
|
||||
heartbeat), `handshake_timeout_s`, and `connect` (a custom transport factory,
|
||||
used by tests and benchmarks). Resume is supported only as automatic reconnect
|
||||
within the same `DuplexClient` instance (see
|
||||
[Reconnect and resume](#reconnect-and-resume)); re-attaching from a new client
|
||||
or process requires the wire-level `session.resume` handshake
|
||||
(`resume_token`, `incarnation`, `last_received_server_event_seq`), which this
|
||||
client does not expose.
|
||||
Constructor options: `reconnect` (a `ReconnectPolicy`, default five
|
||||
attempts with 0.25–4 s jittered backoff; `None` disables auto-resume),
|
||||
`heartbeat_interval_s` (default 30 s; `None` disables the lease heartbeat),
|
||||
`handshake_timeout_s`, and `connect` (a custom transport factory, used by
|
||||
tests and benchmarks). The session id is allocated by the server and becomes
|
||||
`client.session_id` once `session.created` arrives; a client cannot choose
|
||||
it. Resume is supported only as automatic reconnect within the same
|
||||
`DuplexClient` instance (see [Reconnect and resume](#reconnect-and-resume));
|
||||
re-attaching from a new client or process requires the wire-level
|
||||
`session.resume` handshake (`resume_token`,
|
||||
`last_received_server_event_seq`), which this client does not expose.
|
||||
|
||||
### Stream audio and commit a turn
|
||||
|
||||
@@ -195,7 +237,7 @@ the stamped `event_id`.
|
||||
On a transport drop the client reconnects with the `ReconnectPolicy` backoff
|
||||
and sends `session.resume` from the last acknowledged `server_event_seq`.
|
||||
Replayed events are deduplicated, consumers see one `ConnectionResumed`
|
||||
event, and `client.incarnation` / `client.resume_token` are refreshed. The
|
||||
event, and `client.resume_token` is refreshed. The
|
||||
server keeps a detached session alive for `disconnect_grace_s` (30 s by
|
||||
default); after that a resume fails with `session_resume_expired`. Resume is
|
||||
gated by `capabilities.supports_session_resume` (MiniCPM-o 4.5 and Nemotron
|
||||
@@ -238,9 +280,67 @@ definitions out of the dependency-free client. `audio_bytes(response_id)`, `coun
|
||||
`openai-realtime-duplex` benchmark backend (`vllm-omni bench serve`) is built
|
||||
on the same helpers.
|
||||
|
||||
## Using the Python API
|
||||
|
||||
`vllm_omni.entrypoints.duplex_omni.DuplexOmni` is the in-process Python API
|
||||
of a duplex model: sessions run inside the engine, and the class only opens,
|
||||
resumes and closes them and pipes typed commands in and typed events out
|
||||
through a `DuplexSessionHandle`.
|
||||
|
||||
```python
|
||||
from vllm_omni.engine.duplex.commands import AppendAudio, Commit
|
||||
from vllm_omni.engine.duplex.events import AudioDelta, SessionClosed
|
||||
from vllm_omni.entrypoints.duplex_omni import DuplexOmni
|
||||
|
||||
omni = DuplexOmni(model="openbmb/MiniCPM-o-4_5", trust_remote_code=True)
|
||||
async with await omni.open_session({"modalities": ["audio", "text"], "ref_audio": ref_audio_data_url}) as session:
|
||||
async def consume():
|
||||
async for event in session.events(): # typed DuplexEvent objects
|
||||
if isinstance(event, AudioDelta):
|
||||
play(event.audio)
|
||||
elif isinstance(event, ResponseDone):
|
||||
break # one turn is enough here
|
||||
task = asyncio.create_task(consume())
|
||||
await session.submit(AppendAudio(audio=pcm16_bytes, format="pcm16", sample_rate_hz=16000))
|
||||
await session.submit(Commit())
|
||||
await task
|
||||
# leaving the block closes the session (the last event is SessionClosed)
|
||||
```
|
||||
|
||||
`open_session` takes a `DuplexSessionConfig` or the same Realtime `session`
|
||||
object the WebSocket handshake takes; the session id is always allocated by
|
||||
`DuplexOmni`. `DuplexSessionHandle` offers `submit(command)` plus one
|
||||
convenience wrapper per command (`append_audio`, `commit`, `create_response`,
|
||||
`cancel_response`, `barge_in`, `ack_playback`, `update`, `create_item`,
|
||||
`delete_item`, `truncate_item`, `heartbeat`, ...). `events()` is a
|
||||
single-consumer async iterator that ends after `session.closed` /
|
||||
`session.expired`; rejected commands come back as `ErrorEvent`s on it.
|
||||
Every event renders the wire JSON with `to_realtime()`, so anything written
|
||||
against the WebSocket protocol works unchanged on the typed stream.
|
||||
|
||||
`vllm_omni.clients.inline_duplex.InlineDuplexClient` wraps a `DuplexOmni`
|
||||
behind the `DuplexClient` API, so the same application code runs in-process
|
||||
(no server) or over the network:
|
||||
|
||||
```python
|
||||
from vllm_omni.clients.inline_duplex import InlineDuplexClient
|
||||
from vllm_omni.clients.minicpmo_4_5 import create_duplex_session_config
|
||||
|
||||
config = create_duplex_session_config(ref_audio=audio_data_url(Path("reference_voice.wav")))
|
||||
async with InlineDuplexClient(omni, model="openbmb/MiniCPM-o-4_5", config=config) as client:
|
||||
await client.stream_pcm(question)
|
||||
await client.commit()
|
||||
async for response in client.responses():
|
||||
...
|
||||
```
|
||||
|
||||
`examples/online_serving/barge_in_client.py --inline` drives the barge-in
|
||||
scenario this way. The inline client has no transport, so it has no
|
||||
reconnect, resume, heartbeat or event-acknowledgement machinery.
|
||||
|
||||
## API Reference
|
||||
|
||||
This section is the complete wire contract of `/v1/realtime`: how
|
||||
This section is the complete wire contract of `/v1/realtime?duplex=1`: how
|
||||
the dialect relates to the OpenAI Realtime protocol, which surfaces each
|
||||
model's capability flags gate, a catalogue of every client-to-server and
|
||||
server-to-client event, and a JSON example for each. `DuplexClient` speaks
|
||||
@@ -253,21 +353,26 @@ its protocol tests.
|
||||
|
||||
### Endpoint and transport
|
||||
|
||||
The endpoint is `ws(s)://<host>/v1/realtime`. Bare connections select the
|
||||
session handler when the deployment declares either `session_mode: duplex`
|
||||
or `duplex_session`. `?duplex=1` remains a compatibility alias, while
|
||||
`?duplex=0` selects the legacy Realtime handler. Optional query parameters
|
||||
are `model`, `session_id`,
|
||||
`autostart` (`0` means resume-only), `resume`, and `native_duplex`
|
||||
(`minicpmo45_native_duplex` is accepted as a deprecated alias and folded
|
||||
into the canonical name). Every message is one JSON object per WebSocket
|
||||
The endpoint is `ws(s)://<host>/v1/realtime`. On a deployment whose pipeline
|
||||
declares a `duplex_plugin` and whose deploy configuration declares
|
||||
`session_mode: duplex`, a bare connection is already a duplex session, so a
|
||||
stock Realtime client needs no vendor query parameter. `?duplex=1` (`1`,
|
||||
`true` or `on`) is the spelling this page uses and stays valid;
|
||||
`?duplex=0` (`0`, `false`, `off`) explicitly opts out and selects the
|
||||
turn-based Realtime handler, which a duplex server does not mount --
|
||||
that connection is answered with `Realtime API is not available`.
|
||||
`ws(s)://<host>/v1/duplex` is an alias that always selects the duplex
|
||||
handler. Optional query parameters are `model`, `autostart` (`0` means
|
||||
resume-only) and `resume`. Every message is one JSON object per WebSocket
|
||||
text frame, discriminated by `type`. Inbound events are applied in arrival
|
||||
order through one per-session mailbox; outbound events preserve that order
|
||||
and carry a monotonically increasing `server_event_seq`, which is the replay
|
||||
cursor for `session.resume`. Client events may carry an `event_id`, echoed in
|
||||
`error.error.event_id` when the event is rejected.
|
||||
|
||||
Session identity is `(session_id, incarnation)`; `attachment_generation`
|
||||
Session identity is the server-allocated `session_id` (`duplex-<uuid>`,
|
||||
announced in `session.created`; a `session_id` the client puts in the session
|
||||
object is ignored, and ids are never reused); `attachment_generation`
|
||||
names the current socket; `epoch` is the interruption fence that advances on
|
||||
every cancel, clear, or barge-in; `turn_id` counts committed model turns;
|
||||
`response_id` (item id `item_<response_id>`) names one assistant utterance.
|
||||
@@ -281,12 +386,13 @@ with no OpenAI counterpart.
|
||||
|
||||
The event vocabulary is uniform, but several surfaces are gated by the
|
||||
`capabilities` object the server returns in `session.created`; a client must
|
||||
branch on those flags rather than on the model name. The current model
|
||||
plugins advertise:
|
||||
branch on those flags rather than on the model name. MiniCPM-o 4.5 is the
|
||||
only model on the plugin contract today; the other two columns record what
|
||||
their integrations advertise once the follow-up PRs port them:
|
||||
|
||||
| Capability | MiniCPM-o 4.5 | PersonaPlex | Nemotron VoiceChat | Gated surface |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `implementation_level` | `model_native_duplex` (when `extra_body.native_duplex=true`) | `model_native_duplex` | `model_native_duplex` | model-owned `response.listen` / `response.speak`; the chat-fallback lane otherwise |
|
||||
| `implementation_level` | `model_native_duplex` | `model_native_duplex` | `model_native_duplex` | model-owned `response.listen` / `response.speak` (constant: every duplex model is model-native) |
|
||||
| `chunk_period_ms` | 1000 | 80 | 80 | the model unit that `response.listen` decisions and camera frames align to |
|
||||
| `supports_session_resume` | yes | no | yes | `session.resume`, `session.resumed`, `session.replaced`, replay after a transport drop |
|
||||
| `supports_barge_in` | yes | no | no | `barge_in`, `turn.signal{event:"barge_in"}`, `overlap_policy=barge_in_on_speech`, `turn_detection.server_vad` |
|
||||
@@ -311,13 +417,13 @@ work unmodified. The overlap falls into the three tiers used by the event catalo
|
||||
| Direction | Messages |
|
||||
| --- | --- |
|
||||
| Client → server | `session.update`, `input_audio_buffer.append` / `.commit` / `.clear`, `conversation.item.create` / `.retrieve` / `.truncate` / `.delete`, `response.create`, `response.cancel`, `output_audio_buffer.clear` (OpenAI: WebRTC-only; here also a WebSocket event) |
|
||||
| Server → client | `error` (same `{type, code, message, event_id, param}` envelope and the same classes `invalid_request_error` / `server_error` / `rate_limit_error`), `session.created`, `session.updated`, `input_audio_buffer.committed` / `.cleared` / `.speech_started` / `.speech_stopped`, `conversation.item.created` / `.added` / `.done` / `.deleted` / `.retrieved` / `.truncated`, `conversation.item.input_audio_transcription.completed`, `response.created`, `response.done`, `response.output_item.added` / `.done`, `response.content_part.added` / `.done`, `response.audio.delta` / `.done`, `response.audio_transcript.delta` / `.done`, `response.output_text.delta` / `.done`, `response.function_call_arguments.delta` / `.done`, `rate_limits.updated`, `output_audio_buffer.cleared` |
|
||||
| Server → client | `error` (same `{type, code, message, event_id, param}` envelope and the same classes `invalid_request_error` / `server_error` / `rate_limit_error`), `session.created`, `session.updated`, `input_audio_buffer.committed` / `.cleared` / `.speech_started` / `.speech_stopped`, `conversation.item.created` / `.added` / `.done` / `.deleted` / `.retrieved` / `.truncated`, `conversation.item.input_audio_transcription.completed`, `response.created`, `response.done`, `response.output_item.added` / `.done`, `response.content_part.added` / `.done`, `response.output_audio.delta` / `.done`, `response.output_audio_transcript.delta` / `.done`, `response.output_text.delta` / `.done`, `response.function_call_arguments.delta` / `.done`, `rate_limits.updated`, `output_audio_buffer.cleared` |
|
||||
| Session fields | `model`, `modalities` / `output_modalities`, `instructions`, `voice`, `input_audio_format` / `output_audio_format` (beta spelling) **and** `audio.input.format` / `audio.output.format` objects (GA spelling), `turn_detection.server_vad {threshold, prefix_padding_ms, silence_duration_ms}`, `input_audio_transcription`, `input_audio_noise_reduction`, `tools`, `tool_choice`, `temperature`, `max_response_output_tokens`, `speed`, `tracing` |
|
||||
| Object shapes | `realtime.session`, `realtime.item`, `realtime.response`; item content parts `input_text` / `input_audio` / `output_text` / `output_audio`; `function_call` and `function_call_output` items; `previous_item_id` chaining; `output_index` / `content_index` addressing |
|
||||
| Sequencing | `response.created → output_item.added → content_part.added → deltas → *.done → content_part.done → output_item.done → response.done → rate_limits.updated`; `speech_started → speech_stopped → committed` |
|
||||
|
||||
The server deliberately emits and accepts **both** the beta and the GA
|
||||
spellings at once (`response.audio.delta` alongside `output_modalities`,
|
||||
spellings at once (`response.output_audio.delta` alongside `output_modalities`,
|
||||
`conversation.item.added` alongside `conversation.item.created`) so either
|
||||
generation of OpenAI client parses the stream.
|
||||
|
||||
@@ -327,9 +433,9 @@ A stock client ignores the extra keys; the extensions are additive.
|
||||
|
||||
| Message | Extension |
|
||||
| --- | --- |
|
||||
| `session.created` / `session.updated` | top-level `incarnation`, `attachment_generation`, `resume_token`; inside `session`: `state`, `turn_state`, `epoch`, `turn_id`, `active_request_id`, `active_response_id`, `active_response_turn_id`, `overlap_policy`, `overlap_*_ms/rms`, `playback_commit_policy`, `playback`, `capabilities`, `ref_audio`, `extra_body`, `idle_timeout_s`, `response_format` |
|
||||
| `session.created` / `session.updated` | top-level `attachment_generation`, `resume_token`; inside `session`: `state`, `turn_state`, `epoch`, `turn_id`, `active_request_id`, `active_response_id`, `active_response_turn_id`, `overlap_policy`, `overlap_*_ms/rms`, `playback_commit_policy`, `playback`, `capabilities`, `ref_audio`, `extra_body`, `idle_timeout_s`, `response_format` |
|
||||
| `response.created` / `response.done` / `response.listen` | `response_id` at top level; the raw duplex event under `response.metadata` (`duplex_event`); `status_details.reason` uses vLLM reasons (`barge_in`, `client_cancelled`, `new_response`, …) |
|
||||
| `response.audio.delta` | `format`, `sample_rate_hz`, `metadata{session_id, epoch, model_speak, end_of_turn, audio_duration_ms, audio_text_marks, playback}` |
|
||||
| `response.output_audio.delta` | `format`, `sample_rate_hz`, `metadata{session_id, epoch, model_speak, end_of_turn, audio_duration_ms, audio_text_marks, playback}` |
|
||||
| `response.speak` (inserted before the first delta) | not an OpenAI event, but rides the OpenAI response envelope (`response_id`, `item_id`, `output_index`, `content_index`) |
|
||||
| `input_audio_buffer.append` | `is_speech`, `video_frames` (+ `max_slice_nums`), `duration_ms`, `audio_end_ms`, per-event `format` / `sample_rate_hz` |
|
||||
| `input_audio_buffer.commit` | `final`, `response_create`, `is_speech:false` (silence declaration) |
|
||||
@@ -343,14 +449,9 @@ Semantic divergences hidden behind shared names:
|
||||
before any commit; a commit may end in `response.listen` and no
|
||||
response, and the model may open a response with no commit at all.
|
||||
OpenAI: commit ⇒ item, `response.create` ⇒ exactly one response.
|
||||
- With `turn_detection={"type":"server_vad"}`, omitted `interrupt_response`
|
||||
defaults to `false` on turn-based models such as Qwen3-Omni, enabling
|
||||
endpointing and automatic responses without interruption. Explicit `true`
|
||||
is rejected there. Model-native duplex keeps the `true` default and rejects
|
||||
`false`; its VAD implies `overlap_policy=barge_in_on_speech`.
|
||||
This capability-dependent default differs from OpenAI's `true` default;
|
||||
the effective configuration is returned in `session.created` / `session.updated`.
|
||||
`semantic_vad` is unsupported.
|
||||
- `turn_detection.interrupt_response=false` is rejected;
|
||||
`create_response` is ignored; `semantic_vad` is unsupported; the VAD
|
||||
runs per session (Silero) and implies `overlap_policy=barge_in_on_speech`.
|
||||
- `rate_limits.updated` is always an empty list (compatibility only).
|
||||
- Cancellation never reuses a `response_id`; truncation is driven by
|
||||
`playback.ack` as well as `conversation.item.truncate`.
|
||||
@@ -359,11 +460,11 @@ Semantic divergences hidden behind shared names:
|
||||
|
||||
| Area | Messages / fields |
|
||||
| --- | --- |
|
||||
| Session lifetime | `session.heartbeat` / `session.heartbeat_ack`, `session.event_ack`, `session.close` / `session.closed`, `session.resume` / `session.resumed` / `session.replaced` / `session.expired` / `session.resync_required`; `server_event_seq`, `resume_token`, `incarnation`, `attachment_generation`; query params `?duplex=1`, `autostart`, `resume`, `native_duplex` |
|
||||
| Session lifetime | `session.heartbeat` / `session.heartbeat_ack`, `session.event_ack`, `session.close` / `session.closed`, `session.resume` / `session.resumed` / `session.replaced` / `session.expired` / `session.resync_required`; `server_event_seq`, `resume_token`, `attachment_generation`; query params `?duplex=1`, `autostart`, `resume` |
|
||||
| Turn-taking | `response.speak`, `response.listen`, `overlap.decision`, `overlap_policy`, `barge_in`, `turn.signal`, `input.cancel`, `input.text.append`, `epoch` / `turn_id` fencing, `force_listen` |
|
||||
| Playback truth | `playback.ack` / `playback.acknowledged`, `playback_commit_policy` (`ack_only` \| `commit_all_on_done`), `playback{generated_ms, sent_ms, played_ms, committed_ms}`, `history_committed`, `audio_text_marks` |
|
||||
| Model negotiation | `capabilities{implementation_level, supports_input_append, supports_barge_in, supports_session_resume, chunk_period_ms, input_modes, …}`, `ref_audio`, `extra_body` (`auto_response`, `native_duplex` — deprecated alias `minicpmo45_native_duplex` is folded into it, `force_listen_count`, `duplex_initial_user_text`) |
|
||||
| Diagnostics | `runtime.control`, `duplex.function_call.done`, the full error-code vocabulary (`REALTIME_ERROR_TYPES_BY_CODE` in `vllm_omni/entrypoints/duplex/realtime_state.py`) |
|
||||
| Model negotiation | `capabilities{implementation_level, supports_input_append, supports_barge_in, supports_session_resume, chunk_period_ms, input_modes, …}`, `ref_audio`, `extra_body` (`auto_response`, `force_listen_count`, `duplex_initial_user_text`) |
|
||||
| Diagnostics | `duplex.function_call.done`, the full error-code vocabulary (`REALTIME_ERROR_TYPES_BY_CODE` in `vllm_omni/engine/duplex/events.py`) |
|
||||
|
||||
#### Consequences for clients
|
||||
|
||||
@@ -381,14 +482,61 @@ Semantic divergences hidden behind shared names:
|
||||
the Tier 3 surface on top of the same Tier 1 events, so both kinds of
|
||||
client can share one deployment and one event log.
|
||||
|
||||
### Turn-based Chat audio metadata
|
||||
|
||||
For turn-based models served through Chat fallback, non-empty audio choices
|
||||
produced by the Omni Chat audio encoder include `audio_metadata`, in both
|
||||
streaming and complete Chat responses. Text-only choices do not require it.
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `format` | Encoding of the returned audio, such as `pcm` or `wav`; `pcm16` requests normalize to `pcm`. |
|
||||
| `sample_rate_hz` | Sample rate used by the encoder, in samples per second. |
|
||||
| `frame_count` | Samples per channel in this chunk, or in the complete waveform for a non-streaming choice. Not a byte count. |
|
||||
| `channels` | Number of audio channels. |
|
||||
|
||||
These values describe the waveform after audio transformations. Compressed
|
||||
formats may add codec padding that is not included in `frame_count`.
|
||||
The encoder uses a model-reported sample rate when available; Qwen3-Omni
|
||||
currently omits it and uses the existing 24 kHz default. This metadata does
|
||||
not independently determine a model's sample rate.
|
||||
|
||||
Chat fallback requests the session's output format explicitly. Its internal
|
||||
`response.output_audio.delta` carries `format`, `sample_rate_hz`, `channels`,
|
||||
and `audio_duration_ms`. The Realtime endpoint exposes output `format` and
|
||||
`sample_rate_hz` on `response.audio.delta`, with duration in
|
||||
`metadata.audio_duration_ms`; it does not currently forward `channels`.
|
||||
The duration is cumulative for the response:
|
||||
integer frames are summed before converting to milliseconds and rounding down.
|
||||
Non-empty audio without valid metadata, or a sample-rate change within one
|
||||
response, fails that fallback response with `response_error`.
|
||||
|
||||
In this Chat fallback path, `response.done.response.metadata.playback` reports:
|
||||
|
||||
- `generated_ms`: cumulative source-waveform duration.
|
||||
- `sent_ms`: cumulative audio accepted into the server output path, including
|
||||
queued or journaled audio. A journaled event remains accepted if the
|
||||
subsequent send fails; this is not proof of socket delivery.
|
||||
- `played_ms`: cumulative playback reported by the client.
|
||||
- `committed_ms`: playback position used for history reconciliation.
|
||||
|
||||
The native model path still records audio before sending; it does not yet
|
||||
share Chat fallback's output-acceptance timing. Only client playback reports
|
||||
establish what was played. The existing duration-proportional text truncation
|
||||
remains an approximation, not word-level audio alignment.
|
||||
|
||||
### Event catalogue
|
||||
|
||||
Every event type the Realtime route accepts or emits, with a worked example in the next section: 21
|
||||
client→server and 42 server→client. Aliases (`push_chunk`,
|
||||
`input.audio.append`, `input_text.append`, `push_text`, `signal_turn`,
|
||||
`audio.playback_ack`, `close_session`, `close`, `session_close`) share the
|
||||
payload of their canonical event and are not listed separately; the alias
|
||||
table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
client→server and 42 server→client. Aliases (`input_text.append`,
|
||||
`push_text`, `signal_turn`, `audio.playback_ack`, `close_session`, `close`)
|
||||
share the payload of their canonical event and are not listed
|
||||
separately; the alias table is `translate_realtime_command` in
|
||||
`vllm_omni/engine/duplex/realtime_commands.py`. The pre-Realtime WAV-append
|
||||
aliases `push_chunk` and `input.audio.append` are **not** accepted on this
|
||||
route: send `input_audio_buffer.append` with one of the supported input
|
||||
formats (`pcm16`, `pcm_s16le`, `s16le`, `pcm_f32le`, `g711_ulaw`,
|
||||
`g711_alaw`).
|
||||
|
||||
#### Client to server
|
||||
|
||||
@@ -398,18 +546,18 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
|
||||
| `session.heartbeat` | 3 | Keepalive; refreshes the engine session lease. |
|
||||
| `session.event_ack` | 3 | Acknowledges received server events by `server_event_seq`; trims the resume replay journal. |
|
||||
| `session.close` | 3 | Graceful close; answered with `session.closed`. |
|
||||
| `session.resume` | 3 | Re-attach a live session on a new socket (`resume_token`, `incarnation`, `last_received_server_event_seq`). |
|
||||
| `session.resume` | 3 | Re-attach a live session on a new socket (`session_id`, `resume_token`, `last_received_server_event_seq`). |
|
||||
| `input_audio_buffer.append` | 2 | Append one audio chunk. OpenAI name; extended with `is_speech`, `video_frames`, `duration_ms`, `audio_end_ms`, per-event `format`/`sample_rate_hz`. |
|
||||
| `input_audio_buffer.commit` | 2 | Seal the buffered utterance into a user item. OpenAI name; extended with `final`, `response_create`, `is_speech:false`. |
|
||||
| `input.commit` | 3 | Alias of `input_audio_buffer.commit` accepted by the runner. |
|
||||
| `input_audio_buffer.clear` | 1 | Drop un-committed input audio. |
|
||||
| `input.cancel` | 3 | Cancel pending input; advances `epoch`. |
|
||||
| `input.text.append` | 3 | Append text to the open input item (chat-fallback lane only). |
|
||||
| `input.text.append` | 3 | Append text input; model-native sessions reject it with `native_text_append_unsupported`. |
|
||||
| `conversation.item.create` | 1 | Add a completed user item (text, audio, or `function_call_output`) to history. |
|
||||
| `conversation.item.retrieve` | 1 | Fetch a stored item by id. |
|
||||
| `conversation.item.delete` | 1 | Delete a stored item by id. |
|
||||
| `conversation.item.truncate` | 1 | Truncate an assistant item's audio/transcript at `audio_end_ms`. |
|
||||
| `response.create` | 1 | Explicitly request a response with per-response overrides. |
|
||||
| `response.create` | 1 | Explicitly request a response with per-response overrides. Requires committed audio input; otherwise `response_create_without_input`. |
|
||||
| `response.cancel` | 1 | Cancel the active (or named) response; advances `epoch`. |
|
||||
| `output_audio_buffer.clear` | 1 | Discard queued output audio; advances `epoch` (OpenAI: WebRTC-only, here also WebSocket). |
|
||||
| `barge_in` | 3 | Explicit hard interrupt; requires `capabilities.supports_barge_in`. |
|
||||
@@ -420,10 +568,10 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
|
||||
|
||||
| Event type | Tier | Description |
|
||||
| --- | --- | --- |
|
||||
| `session.created` | 2 | Session opened. OpenAI name; adds `incarnation`, `attachment_generation`, `resume_token` and vLLM-Omni keys inside `session` (`epoch`, `turn_id`, `playback`, `capabilities`, ...). |
|
||||
| `session.created` | 2 | Session opened. OpenAI name; adds `attachment_generation`, `resume_token` and vLLM-Omni keys inside `session` (`id` is the server-allocated session id, `epoch`, `turn_id`, `playback`, `capabilities`, ...). |
|
||||
| `session.updated` | 2 | Echo of the effective session config after every `session.update` (same extended `session` object). |
|
||||
| `session.heartbeat_ack` | 3 | Reply to `session.heartbeat`. |
|
||||
| `session.closed` | 3 | Last event on the socket; `reason` ∈ `session_close`, `timeout`, `disconnect`, ... |
|
||||
| `session.closed` | 3 | Last event on the socket, emitted once the engine released the session; `reason` ∈ `client_close`, `disconnect`, `timeout`, `transport_error`, `shutdown`, ... |
|
||||
| `session.resumed` | 3 | Resume accepted; carries the new `attachment_generation` and rotated `resume_token`; journaled events are replayed after it. |
|
||||
| `session.replaced` | 3 | Sent to the superseded socket when another socket resumes the session. |
|
||||
| `session.expired` | 3 | Engine lease reaped (`disconnect_grace_expired`, `idle_ttl_expired`); socket closes after it. |
|
||||
@@ -443,12 +591,12 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
|
||||
| `response.output_item.added` | 1 | Assistant item (or `function_call` item) attached to the response. |
|
||||
| `response.content_part.added` | 1 | Audio/text content part opened on the assistant item. |
|
||||
| `response.speak` | 3 | Model chose to speak (native lane); at most once per response, before the first audio delta. |
|
||||
| `response.audio.delta` | 2 | One ordered audio chunk. OpenAI name; adds `format`, `sample_rate_hz`, `metadata{epoch, model_speak, end_of_turn, audio_text_marks, playback}`. |
|
||||
| `response.audio_transcript.delta` | 1 | Transcript text paired one-to-one with each audio delta. |
|
||||
| `response.output_audio.delta` | 2 | One ordered audio chunk. OpenAI name; adds `format`, `sample_rate_hz`, `metadata{epoch, model_speak, end_of_turn, audio_text_marks, playback}`. |
|
||||
| `response.output_audio_transcript.delta` | 1 | Transcript text paired one-to-one with each audio delta. |
|
||||
| `response.output_text.delta` | 1 | Text delta for text-modality responses. |
|
||||
| `response.output_text.done` | 1 | Final text of a text-modality response. |
|
||||
| `response.audio.done` | 1 | Audio stream closed for the response. |
|
||||
| `response.audio_transcript.done` | 1 | Full transcript (concatenation of the deltas). |
|
||||
| `response.output_audio.done` | 1 | Audio stream closed for the response. |
|
||||
| `response.output_audio_transcript.done` | 1 | Full transcript (concatenation of the deltas). |
|
||||
| `response.content_part.done` | 1 | Content part finalized with its transcript/text. |
|
||||
| `response.output_item.done` | 1 | Assistant item finalized. |
|
||||
| `response.done` | 2 | Terminal event; `status` ∈ `completed` \| `cancelled` \| `failed`. OpenAI name; vLLM-Omni `status_details.reason` values and raw duplex event under `metadata`. |
|
||||
@@ -459,8 +607,8 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
|
||||
| `duplex.function_call.done` | 3 | Raw fallback when a model tool call is malformed. |
|
||||
| `output_audio_buffer.cleared` | 1 | Reply to `output_audio_buffer.clear`. |
|
||||
| `playback.acknowledged` | 3 | Reply to `playback.ack`; returns the playback ledger and `history_committed`. |
|
||||
| `overlap.decision` | 3 | How an append that overlapped an active response was handled (`drop` \| `listen` \| `barge_in`). |
|
||||
| `runtime.control` | 3 | Diagnostic: an engine control signal was partially unsupported. |
|
||||
| `overlap.decision` | 3 | How an append that overlapped an active response was handled (`drop` \| `listen` \| `barge_in`); details under `metadata`. |
|
||||
| `turn.event` | 3 | A turn-taking signal the session accepted (`event`, resulting `turn_state`). |
|
||||
| `error` | 2 | Error envelope. OpenAI shape and classes; vLLM-Omni error codes. |
|
||||
|
||||
### Message examples
|
||||
@@ -479,7 +627,6 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
"event_id": "evt_0001",
|
||||
"session": {
|
||||
"model": "openbmb/MiniCPM-o-4_5",
|
||||
"session_id": "sess_demo_01",
|
||||
"modalities": ["audio", "text"],
|
||||
"instructions": "You are a concise voice assistant.",
|
||||
"input_audio_format": "pcm16",
|
||||
@@ -492,7 +639,6 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
"idle_timeout_s": 300,
|
||||
"extra_body": {
|
||||
"auto_response": true,
|
||||
"native_duplex": true,
|
||||
"force_listen_count": 0
|
||||
}
|
||||
}
|
||||
@@ -504,13 +650,12 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
```json
|
||||
{
|
||||
"type": "session.created",
|
||||
"incarnation": 0,
|
||||
"attachment_generation": 1,
|
||||
"resume_token": "rt_3f9c…",
|
||||
"session": {
|
||||
"object": "realtime.session",
|
||||
"type": "realtime",
|
||||
"id": "sess_demo_01",
|
||||
"id": "duplex-5f2a…",
|
||||
"model": "openbmb/MiniCPM-o-4_5",
|
||||
"state": "open",
|
||||
"turn_state": "idle",
|
||||
@@ -563,7 +708,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
`session.created` and every later `session.update`)
|
||||
|
||||
```json
|
||||
{"type": "session.updated", "session": {"object": "realtime.session", "id": "sess_demo_01", "…": "…"}}
|
||||
{"type": "session.updated", "session": {"object": "realtime.session", "id": "duplex-5f2a…", "…": "…"}}
|
||||
```
|
||||
|
||||
`C→S session.heartbeat` / `S→C session.heartbeat_ack`
|
||||
@@ -573,7 +718,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
```
|
||||
|
||||
```json
|
||||
{"type": "session.heartbeat_ack", "session_id": "sess_demo_01"}
|
||||
{"type": "session.heartbeat_ack", "session_id": "duplex-5f2a…"}
|
||||
```
|
||||
|
||||
`C→S session.event_ack`
|
||||
@@ -589,7 +734,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
```
|
||||
|
||||
```json
|
||||
{"type": "session.closed", "session_id": "sess_demo_01", "reason": "session_close"}
|
||||
{"type": "session.closed", "session_id": "duplex-5f2a…", "epoch": 3, "reason": "client_close", "event": {"type": "session.closed", "reason": "client_close"}}
|
||||
```
|
||||
|
||||
`C→S session.resume` (new socket, `?duplex=1&autostart=0`)
|
||||
@@ -597,8 +742,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
```json
|
||||
{
|
||||
"type": "session.resume",
|
||||
"session_id": "sess_demo_01",
|
||||
"incarnation": 0,
|
||||
"session_id": "duplex-5f2a…",
|
||||
"resume_token": "rt_3f9c…",
|
||||
"last_received_server_event_seq": 40
|
||||
}
|
||||
@@ -610,8 +754,8 @@ events with `server_event_seq > 40` are replayed in order)
|
||||
```json
|
||||
{
|
||||
"type": "session.resumed",
|
||||
"session_id": "sess_demo_01",
|
||||
"incarnation": 0,
|
||||
"session_id": "duplex-5f2a…",
|
||||
"session": {"id": "duplex-5f2a…", "state": "open", "epoch": 3, "turn_id": 2, "...": "the engine's current session object"},
|
||||
"attachment_generation": 2,
|
||||
"resume_token": "rt_a71e…"
|
||||
}
|
||||
@@ -620,19 +764,19 @@ events with `server_event_seq > 40` are replayed in order)
|
||||
`S→C session.replaced` (sent to the superseded socket, which is then closed)
|
||||
|
||||
```json
|
||||
{"type": "session.replaced", "session_id": "sess_demo_01", "attachment_generation": 1}
|
||||
{"type": "session.replaced", "session_id": "duplex-5f2a…", "attachment_generation": 1}
|
||||
```
|
||||
|
||||
`S→C session.expired`
|
||||
|
||||
```json
|
||||
{"type": "session.expired", "session_id": "sess_demo_01", "incarnation": 0, "reason": "disconnect_grace_expired"}
|
||||
{"type": "session.expired", "session_id": "duplex-5f2a…", "reason": "disconnect_grace_expired"}
|
||||
```
|
||||
|
||||
`S→C session.resync_required`
|
||||
|
||||
```json
|
||||
{"type": "session.resync_required", "session_id": "sess_demo_01", "reason": "journal_gap"}
|
||||
{"type": "session.resync_required", "session_id": "duplex-5f2a…", "reason": "journal_gap"}
|
||||
```
|
||||
|
||||
#### Input
|
||||
@@ -653,7 +797,10 @@ events with `server_event_seq > 40` are replayed in order)
|
||||
```
|
||||
|
||||
with camera frames (omni video; rides the append that closes a 1 s model
|
||||
unit; each entry is a **bare** base64 JPEG/PNG — no `data:` URL prefix)
|
||||
unit; each entry is a **bare** base64 JPEG/PNG — no `data:` URL prefix).
|
||||
Only a model whose Stage 0 interleaves a frame track at unit boundaries takes
|
||||
this field; a turn model has no such boundary and rejects it, expecting
|
||||
`conversation.item.create` with `input_image` content instead.
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -723,7 +870,7 @@ silence/noise declaration (no runtime append; answered with `response.listen`)
|
||||
"item_id": "item_9b2d…",
|
||||
"event": {
|
||||
"type": "input.committed",
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"turn_id": 1,
|
||||
"epoch": 0,
|
||||
"history_len": 1,
|
||||
@@ -753,7 +900,7 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"type": "input_audio_buffer.committed",
|
||||
"previous_item_id": "item_9b2d…",
|
||||
"item_id": "item_c044…",
|
||||
"event": {"type": "input.committed", "session_id": "sess_demo_01", "turn_id": 2, "epoch": 0, "history_len": 3, "response_create_deferred": true, "message": {"…": "…"}}
|
||||
"event": {"type": "input.committed", "session_id": "duplex-5f2a…", "turn_id": 2, "epoch": 0, "history_len": 3, "response_create_deferred": true, "message": {"…": "…"}}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -830,6 +977,10 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
{"type": "conversation.item.truncated", "item_id": "item_resp_01", "content_index": 0, "audio_end_ms": 1850, "event": {"…": "…"}}
|
||||
```
|
||||
|
||||
After this acknowledgement, `conversation.item.retrieve` returns the truncated
|
||||
transcript. Without text/audio alignment marks, its prefix is estimated from
|
||||
the requested position and audio duration; this is not exact word alignment.
|
||||
|
||||
#### Response lifecycle
|
||||
|
||||
`C→S response.create`
|
||||
@@ -862,7 +1013,7 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"status_details": null,
|
||||
"output": [],
|
||||
"modalities": ["audio", "text"],
|
||||
"metadata": {"duplex_event": {"type": "response.created", "session_id": "sess_demo_01", "response_id": "resp_01", "epoch": 0, "turn_id": 1}}
|
||||
"metadata": {"duplex_event": {"type": "response.created", "session_id": "duplex-5f2a…", "response_id": "resp_01", "epoch": 0, "turn_id": 1}}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -898,15 +1049,15 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"item_id": "item_resp_01",
|
||||
"output_index": 0,
|
||||
"content_index": 0,
|
||||
"metadata": {"session_id": "sess_demo_01", "epoch": 0, "model_speak": true}
|
||||
"metadata": {"session_id": "duplex-5f2a…", "epoch": 0, "model_speak": true}
|
||||
}
|
||||
```
|
||||
|
||||
`S→C response.audio.delta` (one ~1 s unit, 24 kHz PCM16)
|
||||
`S→C response.output_audio.delta` (one ~1 s unit, 24 kHz PCM16)
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "response.audio.delta",
|
||||
"type": "response.output_audio.delta",
|
||||
"response_id": "resp_01",
|
||||
"item_id": "item_resp_01",
|
||||
"output_index": 0,
|
||||
@@ -915,7 +1066,7 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"format": "pcm16",
|
||||
"sample_rate_hz": 24000,
|
||||
"metadata": {
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"epoch": 0,
|
||||
"model_speak": true,
|
||||
"end_of_turn": false,
|
||||
@@ -926,10 +1077,10 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
}
|
||||
```
|
||||
|
||||
`S→C response.audio_transcript.delta` (exactly one per audio delta)
|
||||
`S→C response.output_audio_transcript.delta` (exactly one per audio delta)
|
||||
|
||||
```json
|
||||
{"type": "response.audio_transcript.delta", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "delta": "It is sunny"}
|
||||
{"type": "response.output_audio_transcript.delta", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "delta": "It is sunny"}
|
||||
```
|
||||
|
||||
`S→C response.output_text.delta` / `response.output_text.done` (text modality)
|
||||
@@ -942,14 +1093,14 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
{"type": "response.output_text.done", "response_id": "resp_02", "item_id": "item_resp_02", "output_index": 0, "content_index": 0, "text": "Tomorrow looks cloudy."}
|
||||
```
|
||||
|
||||
`S→C response.audio.done` / `response.audio_transcript.done`
|
||||
`S→C response.output_audio.done` / `response.output_audio_transcript.done`
|
||||
|
||||
```json
|
||||
{"type": "response.audio.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0}
|
||||
{"type": "response.output_audio.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0}
|
||||
```
|
||||
|
||||
```json
|
||||
{"type": "response.audio_transcript.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "transcript": "It is sunny, 24 degrees."}
|
||||
{"type": "response.output_audio_transcript.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "transcript": "It is sunny, 24 degrees."}
|
||||
```
|
||||
|
||||
`S→C response.content_part.done`
|
||||
@@ -980,7 +1131,7 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"status": "completed",
|
||||
"status_details": null,
|
||||
"output": [{"id": "item_resp_01", "object": "realtime.item", "type": "message", "role": "assistant", "status": "completed", "content": [{"type": "output_audio", "transcript": "It is sunny, 24 degrees."}]}],
|
||||
"metadata": {"type": "response.done", "session_id": "sess_demo_01", "response_id": "resp_01", "epoch": 0, "committed": true, "playback": {"generated_ms": 2400, "sent_ms": 2400, "played_ms": 2400, "committed_ms": 2400}}
|
||||
"metadata": {"type": "response.done", "session_id": "duplex-5f2a…", "response_id": "resp_01", "epoch": 0, "committed": true, "playback": {"generated_ms": 2400, "sent_ms": 2400, "played_ms": 2400, "committed_ms": 2400}}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -1006,9 +1157,9 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
```json
|
||||
{
|
||||
"type": "response.listen",
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"epoch": 0,
|
||||
"response": {"object": "realtime.response", "status": "listening", "metadata": {"type": "response.listen", "session_id": "sess_demo_01", "epoch": 0, "reason": "model_listen"}}
|
||||
"response": {"object": "realtime.response", "status": "listening", "metadata": {"type": "response.listen", "session_id": "duplex-5f2a…", "epoch": 0, "reason": "model_listen"}}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -1055,7 +1206,7 @@ function-call response (Nemotron VoiceChat tools)
|
||||
raw fallback when a model tool call is malformed (`call_id`/`name` missing)
|
||||
|
||||
```json
|
||||
{"type": "duplex.function_call.done", "event": {"type": "function_call.done", "session_id": "sess_demo_01", "epoch": 0, "call_id": null, "name": null, "arguments": "{"}}
|
||||
{"type": "duplex.function_call.done", "event": {"type": "function_call.done", "session_id": "duplex-5f2a…", "epoch": 0, "call_id": null, "name": null, "arguments": "{"}}
|
||||
```
|
||||
|
||||
#### Playback
|
||||
@@ -1071,7 +1222,7 @@ raw fallback when a model tool call is malformed (`call_id`/`name` missing)
|
||||
"type": "playback.acknowledged",
|
||||
"event": {
|
||||
"type": "playback.acknowledged",
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"epoch": 0,
|
||||
"item_id": "item_resp_01",
|
||||
"played_ms": 1000,
|
||||
@@ -1096,20 +1247,17 @@ truncating ack after an interruption (what was actually heard)
|
||||
```json
|
||||
{
|
||||
"type": "overlap.decision",
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"epoch": 0,
|
||||
"policy": "listen_only",
|
||||
"action": "listen",
|
||||
"reason": "long_overlap_speech",
|
||||
"overlap_ms": 1400,
|
||||
"buffer_audio": true,
|
||||
"defer_runtime_append": true,
|
||||
"force_listen": true
|
||||
"metadata": {"overlap_ms": 1400, "buffer_audio": true, "defer_runtime_append": true, "force_listen": true}
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{"type": "overlap.decision", "session_id": "sess_demo_01", "epoch": 0, "policy": "barge_in_on_speech", "action": "barge_in", "reason": "server_vad_utterance_active"}
|
||||
{"type": "overlap.decision", "session_id": "duplex-5f2a…", "epoch": 0, "policy": "barge_in_on_speech", "action": "barge_in", "reason": "server_vad_utterance_active"}
|
||||
```
|
||||
|
||||
server-VAD session (`session.update` fragment) and the annotated append the
|
||||
@@ -1123,12 +1271,6 @@ translator forwards internally (shown for reference; clients never see it)
|
||||
{"type": "input_audio_buffer.append", "audio": "…", "format": "pcm16", "sample_rate_hz": 16000, "is_speech": true, "force_listen": true, "vad": {"backend": "silero", "is_speech": true, "speech_active": true, "speech_started": true, "speech_stopped": false, "speech_probability": 0.93}}
|
||||
```
|
||||
|
||||
`S→C runtime.control` (only when an engine signal was partially unsupported)
|
||||
|
||||
```json
|
||||
{"type": "runtime.control", "session_id": "sess_demo_01", "epoch": 1, "result": {"…": "redacted engine control result"}}
|
||||
```
|
||||
|
||||
#### Errors
|
||||
|
||||
`S→C error` — Realtime shape
|
||||
@@ -1154,71 +1296,80 @@ translator forwards internally (shown for reference; clients never see it)
|
||||
```
|
||||
|
||||
```json
|
||||
{"type": "error", "error": {"type": "rate_limit_error", "code": "duplex_session_busy", "message": "duplex session capacity reached (max_sessions=4)"}}
|
||||
{"type": "error", "error": {"type": "rate_limit_error", "code": "resource_exhausted", "message": "duplex_session_capacity_exhausted: limit=4"}}
|
||||
```
|
||||
|
||||
Handshake-stage errors (before the Realtime projector is bound) use the
|
||||
flat native shape:
|
||||
|
||||
```json
|
||||
{"type": "error", "error": "Unknown or expired duplex session: sess_demo_01", "code": "session_resume_expired"}
|
||||
{"type": "error", "error": "Unknown or expired duplex session: duplex-5f2a…", "code": "session_resume_expired"}
|
||||
```
|
||||
|
||||
#### Coverage
|
||||
|
||||
The examples above cover every event type the Realtime route emits or
|
||||
accepts (verified by diffing the `type` literals in
|
||||
`vllm_omni/entrypoints/duplex/` against this document). Not shown, because
|
||||
they never reach a `/v1/realtime?duplex=1` client unprojected: the native
|
||||
`/v1/duplex` dialect (`session.create` / `open_session` / `session.config`,
|
||||
`input.committed`, `input.cancelled`, `audio.cancelled`,
|
||||
`response.output_audio.delta`, `response.text.delta`, `response.message`,
|
||||
`function_call.done`) — their Realtime projections are the examples above
|
||||
(see the name map at the end of this section) — and the runner-internal markers `__timeout__`,
|
||||
`__disconnect__`, `__replaced_attachment__`. Input aliases (`push_chunk`,
|
||||
`input.audio.append`, `input_text.append`, `push_text`, `signal_turn`,
|
||||
`audio.playback_ack`, `close_session`, `close`, `session_close`) share the
|
||||
payload of their canonical event (see the alias table in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md)).
|
||||
`vllm_omni/engine/duplex/events.py` and `vllm_omni/engine/duplex/commands.py`
|
||||
against this document). Not shown, because they never reach a client
|
||||
unprojected: the session-internal events the runner produces
|
||||
(`input.committed`, `input.cancelled`, `audio.cancelled`,
|
||||
`response.output_audio.delta`, `response.text.delta`, `function_call.done`)
|
||||
— their Realtime projections are the examples above (see the name map at the
|
||||
end of this section). Every public event is a typed `DuplexEvent` whose
|
||||
`to_realtime()` is the wire object; every client event is parsed into a
|
||||
typed `DuplexCommand` by `command_from_realtime()`.
|
||||
|
||||
#### Native to Realtime name map
|
||||
#### Internal to Realtime name map
|
||||
|
||||
The native `/v1/duplex` dialect is what the session runner produces; the
|
||||
Realtime projector renames and fans out these events before they reach a
|
||||
`/v1/realtime?duplex=1` client.
|
||||
The session runner emits these session-internal events; the Realtime
|
||||
projector (`vllm_omni/engine/duplex/realtime_events.py`) renames and fans
|
||||
them out into typed events before they reach a client.
|
||||
|
||||
| Native / internal event | Realtime projection |
|
||||
| Internal event | Realtime projection |
|
||||
| --- | --- |
|
||||
| `session.create` (`open_session`, `session.config`) | first `session.update` |
|
||||
| `input.committed` | `conversation.item.added` / `conversation.item.created`, `input_audio_buffer.committed`, `conversation.item.input_audio_transcription.completed`, `conversation.item.done` |
|
||||
| `input.cancelled` | `input_audio_buffer.cleared` |
|
||||
| `response.output_audio.delta` (with `audio_transcript`) | optional `response.speak`, then `response.audio.delta` and `response.audio_transcript.delta` |
|
||||
| `response.output_audio.done` / `response.output_text.done` | `response.audio.done` / `response.output_text.done` |
|
||||
| `response.output_audio.delta` (with `audio_transcript`) | optional `response.speak`, then `response.output_audio.delta` and `response.output_audio_transcript.delta` |
|
||||
| `response.output_audio.done` / `response.output_text.done` | `response.output_audio.done` / `response.output_text.done` |
|
||||
| `response.text.delta` | `response.output_text.delta` |
|
||||
| `response.message` (chat-fallback raw chunk) | passed through |
|
||||
| `audio.cancelled` | optional `output_audio_buffer.cleared`, then the cancelled terminal events and `response.done` with `status: "cancelled"` |
|
||||
| `function_call.done` | `response.created`, `response.output_item.added` (`function_call` item), `response.function_call_arguments.delta` / `.done`, `response.output_item.done`, `response.done` |
|
||||
| `playback.acknowledged` | `playback.acknowledged` (wrapped verbatim under `event`) |
|
||||
| `runtime.control`, `session.resumed`, `session.replaced`, `session.expired`, `session.resync_required`, `session.heartbeat_ack` | passed through unchanged |
|
||||
| `session.resumed`, `session.replaced`, `session.expired`, `session.resync_required`, `session.heartbeat_ack` | passed through unchanged |
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- Only MiniCPM-o 4.5 is served over this endpoint today; PersonaPlex and
|
||||
Nemotron VoiceChat arrive with the follow-up PRs that port them to the
|
||||
plugin contract.
|
||||
- Several surfaces are capability-gated per model (see *Capability
|
||||
negotiation by model* above): PersonaPlex does not support session resume,
|
||||
barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
|
||||
or audio truncation; camera frames are consumed only by MiniCPM-o 4.5; tool
|
||||
calls are produced only by Nemotron VoiceChat.
|
||||
- `turn_detection` supports `server_vad` or `null`, not `semantic_vad`.
|
||||
Turn-based models support endpointing without interruption, including
|
||||
`create_response=false` for explicit response creation. Model-native duplex
|
||||
requires `interrupt_response=true` and `create_response=true`.
|
||||
- A response is generated from committed **audio**, so the OpenAI text-prompt
|
||||
shape does not drive one: `conversation.item.create` with an `input_text`
|
||||
part adds the item to history, but a following `response.create` with no
|
||||
committed audio is rejected with `response_create_without_input`, and
|
||||
`input.text.append` is rejected with `native_text_append_unsupported`.
|
||||
Text-to-speech has its own shape here — put the text in
|
||||
`extra_body.duplex_initial_user_text` on `session.update`, then stream audio
|
||||
units (silence is enough). The seeded turn is what the model answers.
|
||||
- `turn_detection` supports only `server_vad` with `interrupt_response=true`;
|
||||
`semantic_vad`, `interrupt_response=false`, and `create_response` are not
|
||||
supported.
|
||||
- `input_audio_transcription` and `input_audio_noise_reduction` are accepted
|
||||
and echoed but no separate transcription or noise-reduction stage runs;
|
||||
transcripts come from the model.
|
||||
- `rate_limits.updated` is emitted for compatibility only and always carries
|
||||
an empty list.
|
||||
- Session capacity is bounded by `duplex_session.max_sessions` in the deploy
|
||||
configuration; admission beyond it fails with `duplex_session_busy` or
|
||||
`resource_exhausted`.
|
||||
- The native `/v1/duplex` route speaks the internal dialect and does not
|
||||
support `session.resume`; use `/v1/realtime?duplex=1` for the contract on
|
||||
this page.
|
||||
configuration; admission beyond it fails with `resource_exhausted`
|
||||
(`rate_limit_error`, retryable).
|
||||
- A duplex deployment serves the websocket route and, when supported by the
|
||||
model, ordinary `/v1/chat/completions` requests on the same engine without a
|
||||
session per request (see [Start the Server](#start-the-server)). Other
|
||||
turn-based HTTP routes report "not available" in duplex mode. To use the
|
||||
ordinary online serving stack, select `session_mode: turn` as described in
|
||||
[Full Duplex](full_duplex_api.md#enable-full-duplex).
|
||||
@@ -6,8 +6,9 @@ vLLM-Omni provides an OpenAI-compatible API for text-to-speech (TTS) generation.
|
||||
- **Fish Speech S2 Pro** (`fishaudio/s2-pro`) -- Dual-AR TTS with DAC codec. Supports text-to-speech and voice cloning via reference audio. Output: 44.1 kHz.
|
||||
- **Voxtral TTS** (`mistralai/Voxtral-4B-TTS-2603`) -- AR + FlowMatching TTS with preset voices. Output: 24 kHz.
|
||||
- **CosyVoice3** (`FunAudioLLM/Fun-CosyVoice3-0.5B-2512`) -- 2-stage talker + flow-matching code2wav. Voice cloning via `ref_audio` + `ref_text` (no presets). Output: 24 kHz.
|
||||
- **Gepard-1.0** (`nineninesix/gepard-1.0`) -- single-stage native-AR TTS with a 22.05 kHz NanoCodec. Zero-shot default voice only.
|
||||
|
||||
See the [Supported Models](#supported-models) section below for the full list, including OmniVoice, VoxCPM2, and MOSS-TTS-Nano.
|
||||
See the [Supported Models](#supported-models) section below for the full list, including OmniVoice, VoxCPM2, MOSS-TTS-Nano, and Breeze-TTS-2.
|
||||
|
||||
!!! tip "Deployment recipes"
|
||||
TTS deployment recipes are published at
|
||||
@@ -40,11 +41,16 @@ vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091
|
||||
# CosyVoice3 (voice cloning only — supply ref_audio + ref_text per request)
|
||||
vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
|
||||
--omni --port 8091 --trust-remote-code
|
||||
|
||||
# Gepard-1.0 (zero-shot default voice; packaged gepard.yaml)
|
||||
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
|
||||
--stage-init-timeout 900 \
|
||||
--deploy-config vllm_omni/deploy/gepard.yaml
|
||||
```
|
||||
|
||||
### Generate Speech
|
||||
|
||||
**Using curl:**
|
||||
**Qwen3-TTS CustomVoice, using curl:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
@@ -56,7 +62,24 @@ curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
**Using Python:**
|
||||
**Gepard-1.0, using curl:**
|
||||
|
||||
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`,
|
||||
`stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields
|
||||
include `speed`, `language`, `instructions`, `task_type`, `ref_audio`,
|
||||
`ref_text`, `extra_params`, and `word_timestamps`.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is Gepard speaking.",
|
||||
"voice": "default",
|
||||
"seed": 7
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
**Qwen3-TTS CustomVoice, using Python:**
|
||||
|
||||
```python
|
||||
import httpx
|
||||
@@ -498,6 +521,15 @@ are cached in-process with a shared LRU so repeated requests with the same
|
||||
all TTS model types; deleting a voice invalidates every model-type slot at
|
||||
once.
|
||||
|
||||
Decoded reference waveforms use a separate LRU cache of owned, contiguous
|
||||
float32 arrays at the source sampling rate. Its byte budget counts numeric
|
||||
buffers (four bytes per mono sample), excluding cache metadata. Numeric storage
|
||||
avoids retaining a Python float object for every sample during garbage
|
||||
collection. MOSS reference encoding consumes these arrays directly; other
|
||||
list-based interfaces, including MOSS Nano, receive temporary lists that are
|
||||
not retained in the resolve cache. The MOSS reference encoder also releases
|
||||
completed batch inputs before waiting for more work.
|
||||
|
||||
### Precomputed Custom Voices
|
||||
|
||||
Qwen3-TTS Base and VoxCPM2 can load offline-precomputed voices at startup.
|
||||
@@ -761,6 +793,28 @@ Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` ne
|
||||
| ------- | ------------- |
|
||||
| `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | Voice cloning from `ref_audio` + `ref_text`. No built-in voice presets — upload a voice or pass `ref_audio`/`ref_text` per request. |
|
||||
|
||||
### Gepard-1.0
|
||||
|
||||
| Model | Description |
|
||||
| ----- | ----------- |
|
||||
| `nineninesix/gepard-1.0` | Zero-shot native-AR TTS. 22.05 kHz mono. `voice` must be omitted or `"default"`. |
|
||||
|
||||
Gepard request fields:
|
||||
|
||||
| Field | Behavior |
|
||||
| ----- | -------- |
|
||||
| `input` | Required. Empty/whitespace-only returns 400. |
|
||||
| `voice` | Omitted or `"default"` only. Other values 400. |
|
||||
| `response_format` | `wav` default. Non-streaming: `wav`/`pcm`/`flac`/`mp3`. `opus` returns 400 (22.05 kHz is not an Opus sample rate). Streaming: `pcm`/`wav` only. |
|
||||
| `stream` / `stream_format` | SSE (`speech.audio.*`) and raw `audio` byte streaming. |
|
||||
| `speed` | Must be `1.0`. Gepard has no native speed control. |
|
||||
| `max_new_tokens` | Frame budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms). Default 1000 from deploy YAML; adapter bounds 1..4096. |
|
||||
| `seed` | Optional. Reaches the in-model 32-head sampler. The packaged YAML currently pins `seed: 42`, so serving is deterministic by default until that pin is removed. |
|
||||
| `extra_params` | Any key returns 400, including `temperature`/`top_p`/`top_k`. |
|
||||
| Cloning / style fields | `ref_audio`, `ref_text`, `speaker_embedding`, `instructions`, `language`, `task_type`, `word_timestamps`, and similar declared-but-unsupported fields return 400. |
|
||||
|
||||
See the [Gepard section of the online TTS hub](../user_guide/examples/online_serving/text_to_speech.md#gepard-10) for launch commands. Native-AR recompute preemption is a known limitation under concurrency.
|
||||
|
||||
### OmniVoice
|
||||
|
||||
| Model | Description |
|
||||
@@ -777,12 +831,52 @@ hidden-state dtype before the output projection. Consequently, a float32 stage c
|
||||
| ------- | ------------- |
|
||||
| `openbmb/VoxCPM2` | TTS + voice cloning with built-in speaker presets and uploaded-voice support. Accepts `voice` (preset or uploaded) or `ref_audio` + optional `ref_text`. |
|
||||
|
||||
#### Startup LoRA adapter
|
||||
|
||||
To serve a fine-tuned VoxCPM2 voice, set
|
||||
`voxcpm2_runtime_config.startup_lora_path` in the stage's `hf_overrides`.
|
||||
Copy `vllm_omni/deploy/voxcpm2.yaml` to a local deploy config and add this key
|
||||
alongside its existing runtime settings:
|
||||
|
||||
```yaml
|
||||
# Under stages[0].engine_extras.hf_overrides.voxcpm2_runtime_config:
|
||||
startup_lora_path: /absolute/path/to/adapter
|
||||
```
|
||||
|
||||
Then launch with the modified config:
|
||||
|
||||
```bash
|
||||
vllm serve openbmb/VoxCPM2 --omni --deploy-config /absolute/path/to/voxcpm2-lora.yaml
|
||||
```
|
||||
|
||||
The directory must be accessible to the worker and contain the native VoxCPM2
|
||||
training export: `lora_config.json` (with a `lora_config` object containing
|
||||
`r`, `alpha`, enabled groups, and target module names) and
|
||||
`lora_weights.safetensors`. PEFT checkpoints and pickle checkpoints are not
|
||||
accepted. Missing, unexpected, non-finite, or incorrectly shaped adapter
|
||||
tensors fail model loading rather than silently loading a partial adapter.
|
||||
|
||||
The adapter is merged into the base LM, residual LM, LocDiT, and optional
|
||||
projection layers selected by its configuration, after base weight loading
|
||||
and before compilation or CUDA graph capture. All requests use that adapter;
|
||||
`voice` still selects a reference voice, not a LoRA adapter. Changing adapters
|
||||
requires restarting the server. Runtime loading/unloading, per-request
|
||||
multi-LoRA selection, and `load_format=dummy` are not supported by this path.
|
||||
Weight fusion rounds to the base model's dtype, so numerical and speech-quality
|
||||
parity should be checked against the upstream adapter on your deployment.
|
||||
|
||||
### MOSS-TTS-Nano
|
||||
|
||||
| Model | Description |
|
||||
| ------- | ------------- |
|
||||
| `OpenMOSS-Team/MOSS-TTS-Nano` | Voice cloning only. Requires `ref_audio` (or an uploaded `voice`); no built-in voice presets. `ref_text` is accepted but ignored — upstream's `voice_clone` mode does not consume a transcript. |
|
||||
|
||||
### Breeze-TTS-2
|
||||
|
||||
| Model | Description |
|
||||
| ------- | ------------- |
|
||||
| `BreezeBlue/Breeze-TTS-2` | Two-stage AR TTS (T5Gemma2 + Qwen3 talker with a depth decoder, bundled Qwen3-TTS codec) at 24 kHz. Four modes are selected from the request fields: plain (`input` + speaker tag `voice`, `S0`..`S9`), voice design (`instructions`), voice clone (`ref_audio` + `ref_text`, exactly one clip), and voice direction (reference + `instructions`). Greedy decoding only: `sample_rate` must be `24000`, `speed` must be `1.0`, and `guidance_scale`/`cfg_scale` other than `1.0`, `negative_prompt`, `temperature`/`top_p`/`top_k` overrides, `language`, and `speaker_embedding` are rejected. Streaming returns PCM `speech.audio.delta` events. See [`recipes/BreezeBlue/Breeze-TTS-2.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md). |
|
||||
|
||||
## Error Responses
|
||||
|
||||
### 400 Bad Request
|
||||
|
||||
@@ -41,7 +41,7 @@ bytes.
|
||||
| Server to client | `video.start` | Confirms the request ID, format, and accepted configuration |
|
||||
| Server to client | `video.chunk_metadata` | Describes the next binary media frame |
|
||||
| Server to client | Binary frame | Carries one fragmented MP4 chunk |
|
||||
| Client to server | `session.interaction` | Queues an optional prompt update during generation |
|
||||
| Client to server | `session.interaction` | Queues an optional interaction (e.g., prompt. camera action) during generation |
|
||||
| Server to client | `session.interaction.queued` | Confirms that an interaction was queued |
|
||||
| Client to server | `session.ping` | Refreshes the stall clock |
|
||||
| Server to client | `session.pong` | Acknowledges a ping |
|
||||
@@ -83,10 +83,39 @@ Models that implement interaction updates can change the active prompt:
|
||||
}
|
||||
```
|
||||
|
||||
Models that register the camera modality (for example LingBot-World) also
|
||||
accept structural SE3 payloads on `event.multi_modal_data.camera`. Coordinates
|
||||
follow Unity semantics (`+X` right, `+Y` up, `+Z` forward; same as LingBot's
|
||||
native camera frame). WASD key tokens are not admitted by the engine:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "session.interaction",
|
||||
"interaction": {
|
||||
"event_id": "forward-1",
|
||||
"event": {
|
||||
"multi_modal_data": {
|
||||
"camera": {
|
||||
"mode": "velocity",
|
||||
"data": {
|
||||
"translation": [0.0, 0.0, 0.05],
|
||||
"rotation": [0.0, 0.0, 0.0, 1.0]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`mode` is `target` (lerp to an absolute pose over `transition_chunks`) or
|
||||
`velocity` (hold a per-frame SE3 delta until replaced). Prompt and camera may
|
||||
be combined in one interaction event.
|
||||
|
||||
This feature is pipeline-dependent. An enabled endpoint does not imply that
|
||||
the loaded model supports midway prompt updates.
|
||||
the loaded model supports midway prompt or camera updates.
|
||||
|
||||
The default start timeout is 10 seconds and the server reports a stall after
|
||||
about 60 seconds without generation progress or a `session.ping`. See the
|
||||
[streaming video generation example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation)
|
||||
for prompt-update scheduling and browser playback.
|
||||
for prompt/camera-update scheduling and browser playback.
|
||||
@@ -45,11 +45,19 @@ WebSocket /v1/video/chat/stream
|
||||
| Server -> Client | `response.start` | none | Query generation started. |
|
||||
| Server -> Client | `response.text.delta` | `delta` | Incremental text output. |
|
||||
| Server -> Client | `response.text.done` | `text` | Final text output for the query. |
|
||||
| Server -> Client | `response.audio.delta` | `data`, `format` | Incremental generated audio, base64 WAV. |
|
||||
| Server -> Client | `response.audio.done` | none | Audio output finished. |
|
||||
| Server -> Client | `response.output_audio.delta` | `data`, `format` | Incremental generated audio, base64 WAV. |
|
||||
| Server -> Client | `response.output_audio.done` | none | Audio output finished. |
|
||||
| Server -> Client | `session.done` | none | Session closed. |
|
||||
| Server -> Client | `error` | `message` | Recoverable protocol or generation error. |
|
||||
|
||||
### Frame Consumption Reporting
|
||||
|
||||
When buffered frames include client-supplied `frame_id` values, the server emits `video.frames.consumed` after the first engine output. Its `frame_ids`, `frames`, and `latest_pts_ms` describe the images included in that query's prompt.
|
||||
|
||||
Queries stride-sample the buffered frames, keeping the last frame, then exclude frames already known to have failed decoding. Excluded frames are not replaced with other buffered frames. Frames whose background decoding has not finished remain eligible through the image URL path.
|
||||
|
||||
**Bugfix compatibility note:** `video.frames.consumed` now excludes known decode failures that were already excluded from the prompt. Older versions could report those frames and their timestamps as consumed. The event name and fields are unchanged; an empty selection reports empty lists and `latest_pts_ms: null`.
|
||||
|
||||
### `session.config` Fields
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
@@ -60,10 +68,24 @@ WebSocket /v1/video/chat/stream
|
||||
| `max_frames` | integer, 1-256 | `50` | Maximum retained frame buffer size. Oldest frames are evicted first. |
|
||||
| `system_prompt` | string or null | null | Optional custom system prompt. |
|
||||
| `use_audio_in_video` | bool | `true` | Include streamed audio chunks in multimodal video understanding when audio is present. |
|
||||
| `sampling_params_list` | list or null | null | Optional per-stage sampling parameter overrides. |
|
||||
| `sampling_params_list` | list or null | null | Optional per-stage parameter dictionaries. Each provided entry replaces that stage's deployment sampling settings. |
|
||||
| `enable_frame_filter` | bool | `true` | Enable EVS near-duplicate frame filtering. |
|
||||
| `frame_filter_threshold` | float, 0.0-1.0 | `0.95` | EVS similarity threshold. Higher keeps more frames; lower drops more near-duplicates. |
|
||||
|
||||
Sampling parameter bugfix: `sampling_params_list` is now forwarded to the engine;
|
||||
earlier versions accepted this field but silently used deployment defaults instead.
|
||||
Each provided entry constructs a fresh `SamplingParams` for that stage in pipeline
|
||||
order; it is not merged with that stage's deployment defaults. Fields absent from a
|
||||
provided entry use `SamplingParams` constructor defaults. Only omitted trailing
|
||||
stages keep their deployment defaults. An omitted, `null`, or empty list keeps the
|
||||
engine defaults. Invalid sampling parameters return an `error` when the query is
|
||||
submitted.
|
||||
|
||||
For example, `[{"temperature": 0.2, "max_tokens": 64}]` configures the thinker while
|
||||
the talker and code2wav retain their deployment defaults. With only
|
||||
`[{"temperature": 0.2}]`, the thinker's `max_tokens` and `top_p` use constructor
|
||||
defaults, not its YAML values.
|
||||
|
||||
### Legacy Aliases
|
||||
|
||||
The server accepts these legacy field names and rewrites them before validation. New clients should send the canonical names above.
|
||||
|
||||
@@ -73,6 +73,10 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
|
||||
| `image_reference` | string | null | JSON-encoded reference image payload; do not combine with `input_reference` or `video_reference` |
|
||||
| `video_reference` | string | null | JSON-encoded reference video payload; do not combine with `input_reference` or `image_reference` |
|
||||
| `audio_reference` | string | null | JSON-encoded audio reference for speech-to-video: `{"audio_url": "..."}` — supports HTTP(s) URLs or base64 data URLs |
|
||||
| `source_video` | file | null | MiniMax H3 latent-edit source video (`.mp4` or `.mov`, up to 512 MiB) |
|
||||
| `source_audio` | file | null | Optional MiniMax H3 latent-edit source audio (`.wav` or `.mp3`, up to 512 MiB) |
|
||||
| `video_noise_mask` | file | null | UTF-8 JSON MiniMax H3 video mask; `0` preserves and `1` regenerates a token |
|
||||
| `audio_noise_mask` | file | null | UTF-8 JSON MiniMax H3 audio mask; `0` preserves and `1` regenerates a token |
|
||||
| `width` | integer | model default | Output video width |
|
||||
| `height` | integer | model default | Output video height |
|
||||
| `num_frames` | integer | 1 | Number of generated frames |
|
||||
@@ -109,6 +113,16 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
|
||||
The final content is available from `/v1/videos/{video_id}/content` after the
|
||||
job status becomes `completed`.
|
||||
|
||||
`queued` means the request is still waiting for diffusion scheduler admission.
|
||||
The status changes to `in_progress` when the scheduler first selects the
|
||||
request for execution.
|
||||
|
||||
`DELETE /v1/videos/{video_id}` issues a bounded engine abort
|
||||
(`VLLM_OMNI_ABORT_TIMEOUT`, default 2s), then cancels the frontend
|
||||
task. Cancellation cleanup is also bounded and best-effort: it confirms
|
||||
the abort was queued, and the current request batch may still drain.
|
||||
The job is then re-read so a completed save is not orphaned.
|
||||
|
||||
### Synchronous Response
|
||||
|
||||
`POST /v1/videos/sync` blocks until generation finishes and returns raw video
|
||||
@@ -204,6 +218,49 @@ upstream vLLM's media URL allowlist protection. Deployments should therefore
|
||||
treat remote media URLs as untrusted and choose this setting as part of their
|
||||
URL access policy.
|
||||
|
||||
### MiniMax H3 Latent-Mask Editing
|
||||
|
||||
MiniMax H3 accepts request-scoped source media and video/audio noise masks.
|
||||
At least one mask is required. A nontrivial `video_noise_mask` requires
|
||||
`source_video`, while a nontrivial `audio_noise_mask` requires either
|
||||
`source_audio` or a `source_video` with an audio stream. Mask values are in
|
||||
`[0, 1]`: `0` preserves the source, `1` regenerates it, and fractional values
|
||||
blend the two behaviors. Exact all-one masks are no-ops and do not require a
|
||||
source. Source uploads without a mask are rejected. Masks may be a JSON scalar
|
||||
or arrays matching the H3 latent/token grid; pixel-resolution masks must be
|
||||
resized or pooled by the client before upload.
|
||||
|
||||
For an aligned output of `F` frames at `W x H`, the video latent grid is
|
||||
`[Tv, H/16, W/16]`, where `Tv = 2 + 5 * ((F - 5) / 17)`. The video mask may be
|
||||
a scalar, a flat token vector, `[Tv, H/32, W/32]`, or the full latent grid. For
|
||||
the model input, timestep, and velocity, a full-grid mask is max-pooled over
|
||||
each 2x2 spatial token and fractional values are rounded upward to 1/256
|
||||
levels. The final x0 restore uses the original, unquantized mask, so full-grid
|
||||
masks retain cell-level preservation inside a model token. The audio length is
|
||||
`Ta = round(F * 40 / 24)`, and its mask may be a scalar, `[Ta]`, `[2, Ta]`, or
|
||||
a flat `2 * Ta` vector. If source audio is short, its missing latent tail is
|
||||
forced to mask value `1` so H3 generates that portion. Each mask must be sent
|
||||
as a UTF-8 JSON file part and is limited to 8 MiB. A file may contain either a
|
||||
JSON scalar or an array.
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8091/v1/videos/sync \
|
||||
-F "prompt=Partially restyle the complete clip and soundtrack" \
|
||||
-F 'extra_params={"task":"t2va","duration":4.0,"aspect_ratio":"16:9"}' \
|
||||
-F "source_video=@source.mp4;type=video/mp4" \
|
||||
-F "source_audio=@source.wav;type=audio/wav" \
|
||||
-F "video_noise_mask=@video-mask.json;type=application/json" \
|
||||
-F "audio_noise_mask=@audio-mask.json;type=application/json" \
|
||||
-o edited.mp4
|
||||
```
|
||||
|
||||
For example, each mask file in the request above may contain the JSON scalar
|
||||
`0.5`.
|
||||
|
||||
The server streams source files to temporary request-scoped storage and removes
|
||||
them after synchronous or asynchronous generation finishes. Models that do not
|
||||
declare latent-mask editing support reject these fields.
|
||||
|
||||
### Speech-to-Video
|
||||
|
||||
For models that support audio-driven generation (e.g., Wan2.2-S2V), pass both
|
||||
@@ -234,6 +291,48 @@ curl -X POST http://localhost:8091/v1/videos/sync \
|
||||
-o output.mp4
|
||||
```
|
||||
|
||||
## Output Encoding
|
||||
|
||||
These `extra_params` control how the server turns decoded frames into MP4 bytes.
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `preencode_mp4` | boolean | false | Encode the MP4 on the worker while the VAE is still decoding, instead of after the full video is materialized |
|
||||
| `preencode_batch_frames` | positive integer | 17 (H3, Wan T2V/I2V); 1 (Wan S2V) | Minimum accumulated frames per worker transfer/encoding batch; used only with `preencode_mp4=true` |
|
||||
| `video_codec_options` | object | null | Encoder options passed through to the H.264 encoder, such as `{"preset": "ultrafast", "threads": "0"}` |
|
||||
|
||||
With `preencode_mp4` enabled, each committed VAE chunk leaves the accelerator and
|
||||
is encoded while later chunks are still decoding, so host transfer and CPU
|
||||
encoding overlap the remaining decode instead of following it. The response is
|
||||
unchanged: the same complete MP4, byte-for-byte equivalent frames.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/videos/sync \
|
||||
-F "prompt=A small robot walking through a neon city" \
|
||||
-F 'extra_params={"preencode_mp4": true, "preencode_batch_frames": 33, "video_codec_options": {"preset": "ultrafast"}}' \
|
||||
-o output.mp4
|
||||
```
|
||||
|
||||
Set `preencode_batch_frames` in `extra_params` (or `extra_args` for offline
|
||||
sampling) to tune batching. The worker accumulates complete VAE chunks until
|
||||
it has at least this many frames, then transfers and encodes them together.
|
||||
It always flushes the final partial batch. This is a threshold, not an exact
|
||||
chunk length: a value of 1 submits every native chunk immediately, and a value
|
||||
smaller than a native chunk does not split it. Larger values reduce transfers
|
||||
but retain more frames on the accelerator and delay encoding. The VAE decode
|
||||
window and output frame count stay unchanged. Wan S2V keeps its existing
|
||||
per-clip behavior by default. Zero, negative, fractional, boolean, string, and
|
||||
null values are rejected when pre-encoding is enabled.
|
||||
|
||||
`preencode_mp4` applies to the complete-MP4 response paths only. The
|
||||
`/v1/realtime/video` WebSocket endpoint rejects it, because that path already
|
||||
overlaps encoding through its own incremental fragmented-MP4 encoder. Wan also
|
||||
rejects it together with `enable_frame_interpolation`, which needs the decoded
|
||||
frames the pre-encoded path no longer materializes.
|
||||
|
||||
Support is per model: MiniMax-H3 and Wan 2.2 (T2V, I2V, and S2V) implement it,
|
||||
and other models ignore the flag and take the full-decode path.
|
||||
|
||||
## Storage
|
||||
|
||||
Set `VLLM_OMNI_SERVER_STORAGE__PATH` to control where asynchronous video outputs are
|
||||
|
||||
+17
-8
@@ -12,7 +12,7 @@ curl http://localhost:8000/metrics
|
||||
## Metric Namespaces
|
||||
|
||||
| Prefix | Source | Present when |
|
||||
|--------|--------|--------------|
|
||||
| -------- | -------- | -------------- |
|
||||
| `vllm_omni:` | vLLM-Omni orchestrator / audio modality / cross-stage transfer | Pipeline-dependent |
|
||||
| `vllm:` | Upstream vLLM engine, wrapped by `OmniPrometheusStatLogger` to expose `{stage, replica}` | Pipeline includes an LLM (AR) stage |
|
||||
| `http_` / `process_` | Uvicorn / Python runtime | Always |
|
||||
@@ -24,7 +24,7 @@ Defined in `vllm_omni/metrics/prometheus.py`. Track request lifecycle across the
|
||||
### Request counts
|
||||
|
||||
| Metric | Type | Labels | Description |
|
||||
|--------|------|--------|-------------|
|
||||
| -------- | ------ | -------- | ------------- |
|
||||
| `vllm_omni:num_requests_running` | Gauge | `model_name` | Pipeline-global in-flight requests (dispatched to engine, not yet finalized) |
|
||||
| `vllm_omni:num_requests_waiting` | Gauge | `model_name` | Requests waiting in the Orchestrator queue |
|
||||
| `vllm_omni:requests_success_total` | Counter | `model_name`, `finished_reason` | Total requests by completion reason. `finished_reason` ∈ {`stop`, `length`, `abort`, ...} mirroring upstream `vllm:request_success_total`; aborts cover client disconnect / cancellation paths in addition to upstream `FinishReason.ABORT` |
|
||||
@@ -32,15 +32,15 @@ Defined in `vllm_omni/metrics/prometheus.py`. Track request lifecycle across the
|
||||
### Latency
|
||||
|
||||
| Metric | Type | Labels | Description |
|
||||
|--------|------|--------|-------------|
|
||||
| -------- | ------ | -------- | ------------- |
|
||||
| `vllm_omni:e2e_request_latency_s` | Histogram | `model_name` | Pipeline-global end-to-end request latency in seconds |
|
||||
|
||||
## Audio Modality Metrics (`vllm_omni:`)
|
||||
|
||||
Emitted at request finalize, except for `audio_ttfp_s` (streaming-hook at the first audio packet) and `audio_underrun_s` / `audio_continuity_ok_total` (streaming finalize, after the chunk stream is exhausted). All carry `{model_name, stage, replica}` plus the listed extra label.
|
||||
Emitted at request finalize, except for `audio_ttfp_s` (the first audio packet/frame from streaming Chat or Speech APIs) and `audio_underrun_s` / `audio_continuity_ok_total` (Chat or Speech streaming finalize, after the chunk stream is exhausted). Non-streaming Speech latency is represented by `e2e_request_latency_s`, not TTFP. All audio metrics carry `{model_name, stage, replica}` plus the listed extra label.
|
||||
|
||||
| Metric | Type | Extra label | Description |
|
||||
|--------|------|-------------|-------------|
|
||||
| -------- | ------ | ------------- | ------------- |
|
||||
| `vllm_omni:audio_ttfp_s` | Histogram | — | Time from request arrival to first audio packet/frame |
|
||||
| `vllm_omni:audio_duration_s` | Histogram | — | Audio content duration (`audio_frames / sample_rate`) |
|
||||
| `vllm_omni:audio_rtf` | Histogram | — | Real-time factor (`stage_gen_time_s / audio_duration_s`); streaming TTS SLO red line `< 1`; uses `RTF_BUCKETS` |
|
||||
@@ -50,13 +50,22 @@ Emitted at request finalize, except for `audio_ttfp_s` (streaming-hook at the fi
|
||||
| `vllm_omni:audio_skipped_requests_total` | Counter | `reason` | Silent-loss counter — code2wav rejected malformed codec input and returned `200 OK` with empty audio |
|
||||
|
||||
The continuity math comes from `vllm_omni/benchmarks/audio_continuity.py::compute_continuity_stats` so the server-side observation aligns with the bench-side definition.
|
||||
For Speech, continuity applies to raw audio, SSE, and WebSocket streaming. WAV headers and empty chunks are excluded; non-streaming Speech requests do not emit continuity samples.
|
||||
|
||||
Continuity rate for the default 100 ms threshold can be derived from the successful-continuity counter and the underrun Histogram's request count:
|
||||
|
||||
```promql
|
||||
sum by (model_name, stage, replica) (rate(vllm_omni:audio_continuity_ok_total{threshold_ms="100"}[5m]))
|
||||
/
|
||||
sum by (model_name, stage, replica) (rate(vllm_omni:audio_underrun_s_count[5m]))
|
||||
```
|
||||
|
||||
## Diffusion Metrics (`vllm_omni:`)
|
||||
|
||||
Per-request timing breakdowns for diffusion (image/video) stages. Emitted at request finalize when `stage_metrics.diffusion_metrics` is present. All carry `{model_name, stage, replica}`.
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| -------- | ------ | ------------- |
|
||||
| `vllm_omni:diffusion_exec_s` | Histogram | DiT forward pass execution time per request in seconds |
|
||||
| `vllm_omni:diffusion_exec_per_step_s` | Histogram | DiT forward pass execution time per denoising step in seconds (`exec_s / num_inference_steps`) |
|
||||
| `vllm_omni:diffusion_preprocess_s` | Histogram | Diffusion input preprocessing time per request in seconds |
|
||||
@@ -80,7 +89,7 @@ rate(vllm_omni:diffusion_preprocess_s_sum[5m])
|
||||
Per-physical-transfer histograms tracking the data hop between adjacent stages. Labels `{model_name, from_stage, from_replica, to_stage, to_replica}` let dashboards attribute latency to specific replica edges. `from_replica` / `to_replica` are resolved from the orchestrator's sticky-routing binding (`stage_pool.get_bound_replica_id(request_id)`), so no extra plumbing through `TransferEdgeStats` is needed.
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| -------- | ------ | ------------- |
|
||||
| `vllm_omni:transfer_size_bytes` | Histogram | Per-transfer payload size in bytes |
|
||||
| `vllm_omni:transfer_tx_s` | Histogram | Sender-side time (serialize + submit to connector) |
|
||||
| `vllm_omni:transfer_rx_s` | Histogram | Receiver-side time (recv + deserialize) |
|
||||
@@ -106,7 +115,7 @@ For the full list of upstream metrics, see [the vLLM docs](https://github.com/vl
|
||||
## Metric Availability by Pipeline Type
|
||||
|
||||
| Metric group | Multi-stage LLM (Qwen3-Omni) |
|
||||
|---|---|
|
||||
| --- | --- |
|
||||
| `vllm_omni:` request tracking + latency | With `--log-stats` |
|
||||
| `vllm_omni:` audio modality | With `--log-stats`, if pipeline has a talker stage |
|
||||
| `vllm_omni:` transfer | With `--log-stats`, if pipeline has ≥ 2 stages |
|
||||
|
||||
@@ -1,24 +1,33 @@
|
||||
# FastVideo VSA
|
||||
|
||||
FastVideo Variable Sparse Attention (VSA) accelerates the `FastVideo/FastWan2.2-TI2V-5B-Diffusers` model's
|
||||
self-attention by partitioning the post-patch latent grid into spatiotemporal
|
||||
FastVideo Variable Sparse Attention (VSA) partitions the post-patch latent grid into spatiotemporal
|
||||
blocks. For every query block, VSA scores the key/value blocks and computes
|
||||
attention only against the selected top-k blocks.
|
||||
|
||||
The supported checkpoint provides both text-to-video and image-to-video modes through `Wan22Pipeline`; the separate Wan I2V-14B, S2V, and VACE pipelines are outside this backend's supported scope.
|
||||
## Supported models
|
||||
|
||||
MiniMax-H3 served with a FastH3 VSA adapter uses a second, H3-specific route
|
||||
through the same backend. See the
|
||||
[MiniMax-H3 recipe](https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3) for its geometry,
|
||||
supported topologies, and commands.
|
||||
| Model / checkpoint | Required adapter | Tasks | Sequence parallelism |
|
||||
| --- | --- | --- | --- |
|
||||
| `FastVideo/FastWan2.2-TI2V-5B-Diffusers` | None | T2V, I2V through `Wan22Pipeline` | Disabled |
|
||||
| `MiniMaxAI/MiniMax-H3` | [FastH3 VSA (recipe)](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter) | T2VA | Disabled or pure Ulysses |
|
||||
| Wan I2V-14B, S2V, VACE | — | Unsupported | — |
|
||||
|
||||
VSA is a CUDA-only, explicitly selected backend. It requires the
|
||||
`fastvideo-kernel` package and currently supports non-causal self-attention
|
||||
with equal query and key/value sequence lengths. Unsupported shapes, masks,
|
||||
dtypes, sequence-parallel execution, or kernel failures fall back to
|
||||
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
|
||||
sequence-parallel context is one of those fallbacks; the H3 route supports pure
|
||||
Ulysses and rejects ring or all-gather sequence parallelism at startup.
|
||||
## Installation
|
||||
|
||||
Install vLLM-Omni from this checkout with the `vsa` extra:
|
||||
|
||||
```bash
|
||||
uv pip install -e '.[vsa]'
|
||||
```
|
||||
|
||||
The extra installs the tested kernel dependency automatically. Prebuilt kernels require
|
||||
Linux, Python 3.12, and glibc 2.34 or newer (x86-64 or aarch64). The full
|
||||
FastVideo framework and provider environment variables are not required.
|
||||
|
||||
`FASTVIDEO_VSA` selects the attention algorithm. On SM120, the H3 integration
|
||||
currently executes FastVideo's 64-token Triton block-sparse kernel; it does not
|
||||
dispatch to FlashInfer. The `vsa` extra installs `fastvideo-kernel==0.3.4` for
|
||||
this execution path.
|
||||
|
||||
## Enable the backend
|
||||
|
||||
@@ -27,24 +36,18 @@ flag. Use `--fastvideo-vsa-topk` to set the number of key/value blocks retained
|
||||
for every query block:
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> \
|
||||
vllm serve <model> --omni \
|
||||
--diffusion-attention-backend FASTVIDEO_VSA \
|
||||
--fastvideo-vsa-topk 64
|
||||
```
|
||||
|
||||
The backwards-compatible environment variable selects the backend with the
|
||||
default `topk=64`:
|
||||
<details markdown="1">
|
||||
<summary>Alternative configuration formats</summary>
|
||||
|
||||
Equivalent structured configuration:
|
||||
|
||||
```bash
|
||||
export DIFFUSION_ATTENTION_BACKEND=FASTVIDEO_VSA
|
||||
vllm-omni serve <model>
|
||||
```
|
||||
|
||||
To tune top-k, pass the CLI backend and top-k flags together as shown above,
|
||||
or use the equivalent structured configuration:
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> \
|
||||
vllm serve <model> --omni \
|
||||
--diffusion-attention-config \
|
||||
'{"default":{"backend":"FASTVIDEO_VSA","fastvideo_vsa_topk":64}}'
|
||||
```
|
||||
@@ -73,8 +76,15 @@ stages:
|
||||
fastvideo_vsa_topk: 64
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Choose top-k
|
||||
|
||||
H3 uses 64-token video blocks and keeps
|
||||
all prefix blocks; see the [FastH3 VSA recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
|
||||
|
||||
### Wan top-k behavior
|
||||
|
||||
At runtime the backend logs the sequence shape and derived block count:
|
||||
|
||||
```text
|
||||
@@ -117,8 +127,16 @@ selecting VSA does not turn a native checkpoint into a distilled model.
|
||||
|
||||
## Verify routing and fallback
|
||||
|
||||
Check the startup and first-forward logs instead of assuming that selecting
|
||||
the backend guarantees sparse execution:
|
||||
VSA is a CUDA-only, explicitly selected backend. It requires the
|
||||
`fastvideo-kernel` package and currently supports non-causal self-attention
|
||||
with equal query and key/value sequence lengths. Unsupported shapes, masks,
|
||||
dtypes, sequence-parallel execution, or kernel failures fall back to
|
||||
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
|
||||
sequence-parallel context is one of those fallbacks; the H3 route supports pure
|
||||
Ulysses and rejects ring or all-gather sequence parallelism at startup.
|
||||
H3 accelerator faults propagate instead of attempting dense recovery.
|
||||
|
||||
On the Wan route, check the startup and first-forward logs:
|
||||
|
||||
- `route=VSA` means top-k block selection is active.
|
||||
- `route=VSA_ALL_BLOCKS` means the FastVideo DMD checkpoint retained all
|
||||
@@ -126,6 +144,10 @@ the backend guarantees sparse execution:
|
||||
- `route=SDPA` or `FASTVIDEO_VSA falling back to SDPA: ...` means dense SDPA
|
||||
executed; the warning includes the reason.
|
||||
|
||||
For H3, check `FastH3 adapter active` at startup and
|
||||
`FASTVIDEO_VSA H3 routing` during DiT execution, as described in the
|
||||
[model recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
|
||||
|
||||
The Wan route requires CUDA tensors in FP16 or BF16, 256-token blocks,
|
||||
standard `head_size**-0.5` scaling, equal Q/K/V head counts, no attention mask,
|
||||
and no active sequence-parallel context. The MiniMax-H3 route uses 64-token
|
||||
|
||||
@@ -78,3 +78,35 @@ Protected video tails require a compatible MindIE-SD release.
|
||||
For common configuration and selector behavior, see the
|
||||
[attention backend overview](../attention_backends.md) and the
|
||||
[backend selection design](../../../design/feature/attention_backend_selection.md).
|
||||
|
||||
## Wan2.2 T2V quantization on Ascend
|
||||
|
||||
`RAINFUSION_ATTN` accepts `diffusion_kv_cache_dtype: fp8` or `mxfp4` for
|
||||
Wan2.2 T2V A14B.
|
||||
MindIE-SD must expose `sparse_attention` with `rf_v3` and an explicit `precision`
|
||||
argument. Omni passes the selected precision directly to this API; the installed
|
||||
native operators must support that precision. Unsupported inputs or an
|
||||
incompatible Python API raise an actionable error; Omni does not change BSA
|
||||
precision automatically. Update `diffusion_kv_cache_dtype` to a precision
|
||||
supported by the installed MindIE-SD, or unset it. Native execution errors also
|
||||
propagate without retry.
|
||||
BSA FP8 and MXFP4 retain MindIE-SD's default rotation policy.
|
||||
|
||||
`diffusion_kv_cache_skip_layers`/`diffusion_kv_cache_skip_steps` explicitly
|
||||
select floating-point BSA and keep attention sparse. The existing `block_sparse`
|
||||
warmup/layer exclusions and short-sequence threshold instead select Dense
|
||||
attention, using the configured Dense precision. Leave `block_sparse.precision`
|
||||
at its default when using `diffusion_kv_cache_dtype` as the shared Dense/BSA
|
||||
precision setting.
|
||||
|
||||
Configure BSA quantization in the model's existing deployment YAML with
|
||||
`backend: RAINFUSION_ATTN` and stage-level `diffusion_kv_cache_dtype: fp8` or
|
||||
`mxfp4`. The precision and skip selectors are siblings of
|
||||
`diffusion_attention_config`; they are not per-role `quant` fields.
|
||||
For 40 denoising steps, `diffusion_kv_cache_skip_steps: "0,1,38,39"` and
|
||||
`diffusion_kv_cache_skip_layers: "0,39"` are recommended starting points. These
|
||||
fixed step indices do not automatically follow a changed step count. Selected
|
||||
forwards remain floating-point BSA; unsupported BSA precision is an error rather
|
||||
than an automatic fallback. See
|
||||
[quantized attention](../../quantization/quantized_kvcache.md#wan22-t2v-quantized-attention-on-ascend)
|
||||
for the shared configuration structure and index semantics.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Device-side video postprocessing
|
||||
|
||||
vLLM-Omni can convert decoded WAN2.2 video from normalized floating-point
|
||||
`[B, C, T, H, W]` tensors to contiguous `uint8` `[B, T, H, W, C]` frames before
|
||||
the worker-to-engine device-to-host transfer. This reduces the transported video
|
||||
payload by four times when the original IPC path widens bfloat16 to float32.
|
||||
|
||||
The optimization is disabled by default. Enable it with a structured diffusion
|
||||
configuration:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
engine = Omni(
|
||||
model="Wan-AI/Wan2.2-TI2V-5B-Diffusers",
|
||||
video_output_transport={"enable_device_postprocess": True},
|
||||
)
|
||||
```
|
||||
|
||||
For the server CLI, pass the same object as JSON:
|
||||
|
||||
```bash
|
||||
vllm serve Wan-AI/Wan2.2-TI2V-5B-Diffusers --omni \
|
||||
--video-output-transport '{"enable_device_postprocess": true}'
|
||||
```
|
||||
|
||||
For a deploy configuration, place it on the diffusion stage:
|
||||
|
||||
```yaml
|
||||
stages:
|
||||
- stage_id: 0
|
||||
video_output_transport:
|
||||
enable_device_postprocess: true
|
||||
```
|
||||
|
||||
## Runtime contract
|
||||
|
||||
WAN2.2 emits decoded video through the typed `DiffusionOutput.media` field. The
|
||||
model runner validates and splits the batch into request-local tensors before
|
||||
preparing each video for transport. The worker then performs the D2H copy and
|
||||
shared-memory IPC using the prepared representation. The engine uses the generic
|
||||
media finalizer instead of a model-specific postprocessor.
|
||||
|
||||
The runtime leaves the video in normalized floating-point form when:
|
||||
|
||||
- `enable_device_postprocess` is false;
|
||||
- the requested output type is not `np`; or
|
||||
- frame interpolation still requires the floating-point tensor.
|
||||
|
||||
These are policy fallbacks, not malformed-contract errors. If the device cannot
|
||||
allocate the temporary conversion buffer, the runner logs a warning and prepares
|
||||
the request-local float representation instead. Invalid tensor layout, encoding,
|
||||
value range, lifecycle state, and non-memory runtime failures still fail before
|
||||
IPC.
|
||||
|
||||
## Precision
|
||||
|
||||
Device preparation converts to float32 before denormalization and quantization.
|
||||
For bfloat16 WAN output this is more precise than the historical host path,
|
||||
which denormalizes in bfloat16. Relative to that path, values may differ by at
|
||||
most one level out of 255. Float32 inputs remain byte-identical after
|
||||
quantization.
|
||||
|
||||
## Scope
|
||||
|
||||
WAN2.2 is the reference migration for the first contract version. Other video
|
||||
pipelines continue using their existing output paths until they can declare the
|
||||
same typed media contract without bypassing model-specific float consumers such
|
||||
as safety checks or audio/video packaging.
|
||||
|
||||
See [RFC #6541](https://github.com/vllm-project/vllm-omni/issues/6541) for the
|
||||
contract, lifecycle, batching rules, and migration plan.
|
||||
@@ -28,7 +28,7 @@ and their optimal Triton tile configurations:
|
||||
|
||||
Config loading order (3-tier, see ops/mot_gemm.py):
|
||||
1. $VLLM_TUNED_CONFIG_FOLDER/<filename> (env override)
|
||||
2. This directory: vllm_omni/diffusion/layers/mot/configs/<filename>
|
||||
2. This directory: vllm_omni/diffusion/models/bagel/mot/configs/<filename>
|
||||
3. Conservative default config (compiles everywhere, sub-optimal perf)
|
||||
|
||||
If no config file matches the current device, a warning is printed with
|
||||
@@ -39,7 +39,7 @@ To generate configs for your hardware:
|
||||
python benchmarks/kernels/mot_linear_benchmarks.py \
|
||||
--model ByteDance-Seed/BAGEL-7B-MoT \
|
||||
--tp-size 1 --dtype w16a16 --tune \
|
||||
--save-dir vllm_omni/diffusion/layers/mot/configs/
|
||||
--save-dir vllm_omni/diffusion/models/bagel/mot/configs/
|
||||
|
||||
For multi-GPU tuning (uses Ray for parallel search):
|
||||
|
||||
|
||||
@@ -353,9 +353,14 @@ modes can resolve the same `None` value to different schedules.
|
||||
Cache-DiT on any selected component. Prompt-embedding cache is also
|
||||
incompatible when the text encoder uses `allgather`. Rank-local cache hits
|
||||
could otherwise make ranks enter different weight collectives.
|
||||
- Per-tensor online FP8 linears use the ordinary loader and can run with either
|
||||
DiT transfer path. With DiT `allgather`, every rank temporarily materializes
|
||||
the complete FP8 model in host memory before DLO retains only its shard.
|
||||
- Per-tensor online FP8, INT8, and MXFP8 linears use the ordinary loader and
|
||||
can run with either DiT transfer path. With DiT `allgather`, every rank
|
||||
temporarily materializes the complete quantized model in host memory before
|
||||
DLO retains only its shard. Layers too wide for the NPU quantization kernel
|
||||
(`npu_quant_matmul` rejects an output dimension past 65535) stay
|
||||
unquantized and load straight into host memory under offload-after-quant —
|
||||
their runtime layout is a plain contiguous bf16 weight, the same layout DLO
|
||||
shards on the ordinary path, so they are allowed under `allgather` too.
|
||||
Other online quantization methods require rank-local transfer for the
|
||||
affected component until their runtime layouts are validated.
|
||||
- Resident leading layers require DiT `rank-local` transfer and a model
|
||||
|
||||
@@ -19,6 +19,12 @@ CFG-Parallel accelerates diffusion models by distributing positive and negative
|
||||
|
||||
See supported models list in [Supported Models](../../diffusion_features.md#supported-models).
|
||||
|
||||
Distilled Cosmos3 checkpoints, including `nvidia/Cosmos3-Super-Image2Video-4Step`,
|
||||
run without CFG and reject `--cfg-parallel-size` greater than 1 at startup.
|
||||
Use `--ulysses-degree 2` for two-GPU inference. These checkpoints always use
|
||||
`guidance_scale=1.0`; other explicitly requested values produce a warning, and
|
||||
`negative_prompt` does not affect generation.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -34,5 +34,34 @@ that provide their own `setup_compile()` implementation manage their compilation
|
||||
policy independently. Compilation is lazy, so backend or graph errors can first
|
||||
surface on the initial request.
|
||||
|
||||
## Pinning one packed shape (MiniMax-H3)
|
||||
|
||||
Compiled regions are keyed by input shape. MiniMax-H3 packs a request into a
|
||||
row count that follows the request -- a longer prompt, more frames, a larger
|
||||
frame -- and rounds it up to the next 64-row boundary, so two requests can land
|
||||
on two different packed lengths.
|
||||
|
||||
Under the default `diffusion_compile_dynamic: true` a new packed length is
|
||||
absorbed by the dynamic shape, so it is not by itself a recompilation. It still
|
||||
hands the backend a shape it has not autotuned, which is the risk this knob
|
||||
exists to remove: a fixed-shape deployment (`--no-diffusion-compile-dynamic`)
|
||||
recompiles the transformer blocks per shape, and a first-seen large shape can
|
||||
push autotuning into its worst-case memory use.
|
||||
|
||||
A request pins the length with `extra_args["pad_seq_len"]`:
|
||||
|
||||
```python
|
||||
sampling_params = SamplingParams(extra_args={"pad_seq_len": 54080})
|
||||
```
|
||||
|
||||
The value must be a positive multiple of 64 and must cover the rows the request
|
||||
actually uses; the packed sequence is then padded to it instead of to the next
|
||||
64-row boundary. Pick a bucket that covers every request dimension the
|
||||
deployment accepts -- prompt length, frame count, frame size and reference
|
||||
blocks all change the used row count. The server logs the effective length as
|
||||
`MiniMax H3 packed sequence: ... pad_seq_len=... used=... seq_len=...` whenever
|
||||
a request pins it. The padding rows are masked out, so their cost is the
|
||||
attention and feed-forward work on those rows.
|
||||
|
||||
Use `--enforce-eager` to disable the model runner's generic compile setup.
|
||||
Pipelines that compile internally define their own eager-mode behavior.
|
||||
@@ -1,35 +0,0 @@
|
||||
# HunyuanImage-3.0 Image-to-Text Inference
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/hunyuan_image3>.
|
||||
|
||||
|
||||
This example demonstrates how to run HunyuanImage-3.0 Image-to-Text with the vLLM-Omni.
|
||||
|
||||
## Local CLI Usage
|
||||
|
||||
Download the example image:
|
||||
|
||||
```bash
|
||||
wget https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg
|
||||
```
|
||||
|
||||
Run example:
|
||||
|
||||
```bash
|
||||
python image_to_text.py \
|
||||
--image cherry_blossom.jpg \
|
||||
--prompt "<|startoftext|>You are an assistant that understands images and outputs text.<img>Describe the content of the picture."
|
||||
```
|
||||
|
||||
Key arguments:
|
||||
|
||||
- `--model`: Model used. Default is: tencent/HunyuanImage-3.0-Instruct (Optional).
|
||||
- `--image`: Path to input image (required).
|
||||
- `--prompt`: Text description used to guide image understanding (required).
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "image_to_text.py"
|
||||
``````py
|
||||
--8<-- "examples/offline_inference/hunyuan_image3/image_to_text.py"
|
||||
``````
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni>.
|
||||
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
|
||||
@@ -15,6 +14,19 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
|
||||
For a local deployment with multiple API frontend processes sharing one set
|
||||
of stage engines, add `--api-server-count`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--api-server-count 2 --port 8091
|
||||
```
|
||||
|
||||
This mode supports local EngineCore stages with one local process group per
|
||||
replica, including stages configured with multiple `num_replicas`. Headless,
|
||||
remote, diffusion, intra-replica data-parallel, and Ray stage deployments
|
||||
remain single-frontend.
|
||||
|
||||
The default deployment configuration situated at `vllm_omni/deploy/qwen3_omni_moe.yaml` is resolved and loaded
|
||||
automatically via the model registry, obviating the necessity for the `--deploy-config` flag in standard deployment topologies.
|
||||
The bundled Qwen3-Omni setup explicitly sets `moe_backend: triton` for the Thinker and Talker stages. This keeps
|
||||
@@ -22,7 +34,12 @@ both stages on vLLM's Triton unquantized MoE path and avoids the unstable FlashI
|
||||
some GPUs.
|
||||
Asynchronous chunk streaming is **enabled by default** within the bundled configuration.
|
||||
|
||||
For Thinker pipeline parallelism, configure `pipeline_parallel_size` and `devices`
|
||||
on the Thinker stage. Keep Talker and Code2Wav at `pipeline_parallel_size: 1`. The
|
||||
default asynchronous scheduling and audio chunk streaming can remain enabled.
|
||||
|
||||
To explicitly utilize a custom deployment YAML, specify the configuration path:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
|
||||
--deploy-config /path/to/deploy_config_file
|
||||
@@ -46,7 +63,7 @@ Adopt the stage-based CLI architecture to independently instantiate execution pr
|
||||
The example below pins Stage 0 to GPU 0 and Stage 1/2 to GPU 1 via
|
||||
`CUDA_VISIBLE_DEVICES`.
|
||||
|
||||
**1. Stage 0 (Thinker + API server)**
|
||||
#### 1. Stage 0 (Thinker + API server)
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
@@ -56,7 +73,7 @@ CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
**2. Stage 1 (Talker)**
|
||||
#### 2. Stage 1 (Talker)
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
@@ -66,7 +83,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
**3. Stage 2 (Code2Wav)**
|
||||
#### 3. Stage 2 (Code2Wav)
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
@@ -91,6 +108,9 @@ To experiment with another supported MoE backend, set `moe_backend` in the
|
||||
deployment YAML for both the Thinker and Talker stages. For the stage-based
|
||||
CLI, pass `--moe-backend` directly on each stage command.
|
||||
|
||||
For other stage settings, pass the option on that stage command. For example,
|
||||
to change its GPU memory utilization:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--stage-id 1 \
|
||||
@@ -103,11 +123,12 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
### Send Multi-modal Request
|
||||
|
||||
Get into the example folder
|
||||
|
||||
```bash
|
||||
cd examples/online_serving/qwen3_omni
|
||||
```
|
||||
|
||||
#### Send request via python
|
||||
#### Send request via python
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py --query-type use_image --port 8091 --host "localhost"
|
||||
@@ -122,7 +143,6 @@ The Python client supports the following command-line arguments:
|
||||
- `--audio-path` (or `-a`): Path to local audio file or URL. If not provided and query-type is `use_audio`, uses default audio URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common audio formats: MP3, WAV, OGG, FLAC, M4A. Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
|
||||
- `--prompt` (or `-p`): Custom text prompt/question. If not provided, uses default prompt for the selected query type. Example: `--prompt "What are the main activities shown in this video?"`
|
||||
|
||||
|
||||
For example, to use a local video file with custom prompt:
|
||||
|
||||
```bash
|
||||
@@ -132,19 +152,20 @@ python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--prompt "What are the main activities shown in this video?"
|
||||
```
|
||||
|
||||
#### Send request via curl
|
||||
#### Send request via curl
|
||||
|
||||
```bash
|
||||
bash run_curl_multimodal_generation.sh use_image
|
||||
```
|
||||
|
||||
## Modality control
|
||||
|
||||
You can control output modalities to specify which types of output the model should generate. This is useful when you only need text output and want to skip audio generation stages for better performance.
|
||||
|
||||
### Supported modalities
|
||||
|
||||
| Modalities | Output |
|
||||
|------------|--------|
|
||||
| ------------ | -------- |
|
||||
| `["text"]` | Text only |
|
||||
| `["audio"]` | Audio only |
|
||||
| `["text", "audio"]` | Text + Audio |
|
||||
@@ -222,7 +243,9 @@ print(response.choices[1].message.audio) # Audio response
|
||||
```
|
||||
|
||||
## Streaming Output
|
||||
|
||||
If you want to enable streaming output, please set the argument as below. The final output will be obtained just after generated by corresponding stage. We support both text streaming output and audio streaming output. Other modalities can output normally.
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type use_image \
|
||||
@@ -255,12 +278,14 @@ The convenience script launches both the vLLM server and Gradio demo together:
|
||||
```
|
||||
|
||||
This script will:
|
||||
|
||||
1. Start the vLLM server in the background
|
||||
2. Wait for the server to be ready
|
||||
3. Launch the Gradio demo
|
||||
4. Handle cleanup when you press Ctrl+C
|
||||
|
||||
The script supports the following arguments:
|
||||
|
||||
- `--model`: Model name/path (default: Qwen/Qwen3-Omni-30B-A3B-Instruct)
|
||||
- `--server-port`: Port for vLLM server (default: 8091)
|
||||
- `--gradio-port`: Port for Gradio demo (default: 7861)
|
||||
@@ -271,18 +296,19 @@ The script supports the following arguments:
|
||||
|
||||
#### Option 2: Manual Launch (Two-Step Process)
|
||||
|
||||
**Step 1: Launch the vLLM API server**
|
||||
##### Step 1: Launch the vLLM API server
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
|
||||
If you have a custom deploy config file:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --deploy-config /path/to/deploy_config_file
|
||||
```
|
||||
|
||||
**Step 2: Run the Gradio demo**
|
||||
##### Step 2: Run the Gradio demo
|
||||
|
||||
In a separate terminal:
|
||||
|
||||
@@ -295,7 +321,7 @@ Then open `http://localhost:7861/` on your local browser to interact with the we
|
||||
The gradio script supports the following arguments:
|
||||
|
||||
- `--model`: Model name/path (should match the server model)
|
||||
- `--api-base`: Base URL for the vLLM API server (default: http://localhost:8091/v1)
|
||||
- `--api-base`: Base URL for the vLLM API server (default: <http://localhost:8091/v1>)
|
||||
- `--ip`: Host/IP for Gradio server (default: 127.0.0.1)
|
||||
- `--port`: Port for Gradio server (default: 7861)
|
||||
- `--share`: Share the Gradio demo publicly (creates a public link)
|
||||
|
||||
@@ -18,6 +18,7 @@ For the full list of supported architectures across all modalities, see
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | — |
|
||||
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
|
||||
| Gepard-1.0 | `nineninesix/gepard-1.0` | — (zero-shot default voice) | ✓ (PCM / WAV stream) | `default` only | — |
|
||||
| higgs-audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✓ (`ref_audio`+`ref_text`) | ✓ (codec_streaming) | — | — |
|
||||
| GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
|
||||
| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
|
||||
@@ -92,7 +93,7 @@ curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
|
||||
```
|
||||
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others).
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for Gepard, 24 kHz for the others).
|
||||
|
||||
For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md).
|
||||
|
||||
@@ -120,6 +121,15 @@ vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remot
|
||||
|
||||
Streaming is on by default via `async_chunk: true` in `vllm_omni/deploy/cosyvoice3.yaml`. Pass `--no-async-chunk` (or `NO_ASYNC_CHUNK=1 ./cosyvoice3/run_server.sh`) for the legacy synchronous path.
|
||||
|
||||
Cross-request Stage-1 flow batching is opt-in. Enable it in the server environment when concurrent requests should share a flow-estimator call:
|
||||
|
||||
```bash
|
||||
export COSYVOICE3_BATCH_FLOW=1
|
||||
vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remote-code
|
||||
```
|
||||
|
||||
Batching preserves output lengths and streaming cache alignment, but the different GEMM shapes can produce small waveform differences compared with processing each request separately. Leave `COSYVOICE3_BATCH_FLOW` unset (or set it to `0`) when request-independent numerical behavior is required. Set `COSYVOICE3_BATCH_FLOW_DEBUG=1` to log the observed group-size distribution and enable detailed Stage-1 profiler scopes; diagnostics are disabled by default to avoid per-step profiling overhead.
|
||||
|
||||
### CLI client
|
||||
|
||||
The client defaults to the official upstream zero-shot prompt, so it runs without extra flags:
|
||||
@@ -203,6 +213,61 @@ python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server
|
||||
|
||||
---
|
||||
|
||||
## Gepard-1.0
|
||||
|
||||
Single-stage native AR TTS at 22.05 kHz mono. Zero-shot only: omit `voice` or pass `"default"`. Voice cloning from reference audio is not available yet.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Same NeMo NanoCodec install as the [offline Gepard section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#gepard-10). On a host whose CUDA toolkit cannot build kernels — no `nvcc`/`ninja`, or a consumer Blackwell (`sm_120`) card — also `export VLLM_USE_FLASHINFER_SAMPLER=0` before launch. That env is not a deploy-YAML field.
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
|
||||
--stage-init-timeout 900 \
|
||||
--deploy-config vllm_omni/deploy/gepard.yaml
|
||||
# or:
|
||||
./gepard/run_server.sh
|
||||
```
|
||||
|
||||
`--stage-init-timeout 900` matches the online e2e fixture and the offline example. Serve defaults to 300s, which is often too short for a cold download of the talker plus NanoCodec.
|
||||
|
||||
The packaged `vllm_omni/deploy/gepard.yaml` must be passed with `--deploy-config`. The checkpoint self-identifies as `qwen3_5_text`, so omitting the YAML launches a diffusion fallback instead of the Gepard pipeline. The YAML sets `async_chunk: false`, `max_num_seqs: 4`, and currently pins `seed: 42`, so serving is deterministic by default until that YAML seed is removed. Pass an explicit per-request `seed` in tests and clients rather than depending on either default.
|
||||
|
||||
### Sending requests
|
||||
|
||||
```bash
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking."
|
||||
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking." --seed 7 --stream --output output.pcm
|
||||
```
|
||||
|
||||
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`, `stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields include `speed`, `language`, `instructions`, `task_type`, `ref_audio`, `ref_text`, `extra_params`, and `word_timestamps`.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is Gepard speaking.",
|
||||
"voice": "default",
|
||||
"seed": 7
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 22.05 kHz mono. `max_new_tokens` is a **frame** budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms at 21.5 fps; adapter bounds 1..4096).
|
||||
- Supported request fields: `input` (required), `voice` (`default` only), `response_format` (`wav` default; `wav/pcm/flac/mp3` non-streaming; `opus` 400 because 22.05 kHz is not an Opus sample rate; streaming `pcm/wav` only), `stream` / `stream_format`, `max_new_tokens`, `seed`.
|
||||
- Unsupported: `speed`, `extra_params` (including `temperature`/`top_p`/`top_k`), `ref_audio`, `ref_text`, `speaker_embedding`, `task_type`, `instructions`, `language`, and `word_timestamps`.
|
||||
- Concurrent requests at `max_num_seqs: 4` are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below `max_num_seqs` and treat preemption as out of scope until the platform fix lands.
|
||||
- Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.
|
||||
- For the request-field table, see the [Speech API Gepard section](https://github.com/vllm-project/vllm-omni/blob/main/docs/serving/speech_api.md#gepard-10).
|
||||
|
||||
---
|
||||
|
||||
## higgs-audio v2
|
||||
|
||||
2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) that streams chunks back to the client.
|
||||
@@ -348,6 +413,22 @@ vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
|
||||
./qwen3_tts/run_server.sh Base
|
||||
```
|
||||
|
||||
For a local deployment with multiple API frontend processes sharing one set
|
||||
of TTS stage engines, add `--api-server-count`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||||
--omni --api-server-count 2 --port 8091
|
||||
```
|
||||
|
||||
This mode supports local EngineCore stages with one local process group per
|
||||
replica, including stages configured with multiple `num_replicas`. Headless,
|
||||
remote, intra-replica data-parallel, and Ray stage deployments are not
|
||||
supported. Runtime voice upload and deletion are disabled with multiple
|
||||
API frontends because their registries are process-local. Built-in voices,
|
||||
inline `ref_audio`, and voices restored from `custom_voice_dir` at startup are
|
||||
supported.
|
||||
|
||||
### Executor backend
|
||||
|
||||
Single-GPU serves now default to the uniproc executor (lower IPC overhead, the Base cloning use case from [#2603](https://github.com/vllm-project/vllm-omni/issues/2603) / [#2604](https://github.com/vllm-project/vllm-omni/pull/2604)). `vllm_omni/deploy/qwen3_tts.yaml` is the only Qwen3-TTS deploy config; pass `--deploy-config <path>` to override.
|
||||
@@ -413,6 +494,10 @@ checkpoint. When a request names an uploaded voice, the server infers
|
||||
`task_type="Base"`. Built-in presets such as `vivian` and `ryan` remain
|
||||
CustomVoice speakers and require a CustomVoice checkpoint.
|
||||
|
||||
The runtime upload and delete routes require a single API frontend; with
|
||||
`--api-server-count > 1`, use inline `ref_audio` or restore precomputed voices
|
||||
from `custom_voice_dir` at startup.
|
||||
|
||||
### Precomputed custom voices
|
||||
|
||||
For reused Base voice-cloning speakers, precompute the reference artifacts once
|
||||
|
||||
@@ -204,6 +204,91 @@ omni = Omni(
|
||||
| `--linear-backend cutlass` | str | auto | Select the validated CUTLASS linear backend for supported ModelOpt NVFP4 or mixed FP8/NVFP4 diffusion stages |
|
||||
| `--moe-backend cutlass` | str | auto | Select the validated CUTLASS MoE backend for supported ModelOpt mixed MoE checkpoints |
|
||||
|
||||
## Cosmos3 Mixed-Precision Schedule
|
||||
|
||||
The experimental Cosmos3 schedule can use the checkpoint-native quantized GEMM for middle denoising
|
||||
steps and dense 16-bit activation execution for selected first and last steps.
|
||||
The schedule supports serialized ModelOpt FP8 and NVFP4 checkpoints:
|
||||
|
||||
| Detected weight format | Native middle steps | A16 boundary steps |
|
||||
|------------------------|---------------------|--------------------|
|
||||
| FP8 | W8A8 | W8A16 |
|
||||
| NVFP4 | W4A4 | W4A16 |
|
||||
|
||||
The native and A16 paths share one quantized weight representation. The A16
|
||||
path dequantizes the live backend weight and calls `F.linear`; it does not keep
|
||||
a second checkpoint-weight snapshot. Scheduled FP8 currently requires
|
||||
serialized tensorwise scales and a backend that retains canonical FP8 weights.
|
||||
Scheduled NVFP4 requires a supported CUTLASS-compatible native or FlashInfer
|
||||
layout. SmoothQuant, Marlin-repacked FP8, and per-channel/per-token FP8
|
||||
schedules fail closed.
|
||||
|
||||
The reasoner uses dense A16 execution by default when its weights are FP8 or
|
||||
NVFP4. Set the nested `reasoner` field to `native` to retain the checkpoint-native
|
||||
W8A8/W4A4 path. A BF16 reasoner remains BF16 in either mode. The schedule
|
||||
currently requires tensor parallel size 1 and one active request per worker,
|
||||
does not support HSDP, and does not support block-scaled FP8. Its live-weight
|
||||
design is compatible with model-level and standard layer-wise offload, but
|
||||
those paths still require GPU end-to-end validation. Distributed layer-wise
|
||||
offload is rejected because its direct loader bypasses ModelOpt post-load
|
||||
transformations.
|
||||
|
||||
A checkpoint can opt in by adding a versioned policy to its authoritative
|
||||
`transformer/config.json`. The same policy schema is used for FP8 and NVFP4:
|
||||
|
||||
```json
|
||||
{
|
||||
"quantization_config": {
|
||||
"quant_method": "modelopt",
|
||||
"quant_algo": "FP8",
|
||||
"runtime": {
|
||||
"diffusion_step_policy": {
|
||||
"schema_version": 1,
|
||||
"type": "first_last_n",
|
||||
"index_space": "denoising_loop_iteration",
|
||||
"scope": ["transformer"],
|
||||
"default_mode": "native",
|
||||
"first_steps": {"count": 3, "mode": "a16"},
|
||||
"last_steps": {"count": 3, "mode": "a16"},
|
||||
"overlap": "a16",
|
||||
"reasoner": "a16"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The example preserves the checkpoint's existing `quant_method` and
|
||||
`quant_algo`; only `runtime` is added.
|
||||
|
||||
When present and valid, the checkpoint policy is the default. An explicit
|
||||
runtime object overrides it. To retain ordinary checkpoint-native execution
|
||||
for every step, disable only the schedule:
|
||||
|
||||
```bash
|
||||
vllm serve /path/to/Cosmos3-Nano-modelopt \
|
||||
--omni \
|
||||
--additional-config '{"cosmos3_mixed_precision":{"enabled":false}}'
|
||||
```
|
||||
|
||||
This does not disable checkpoint quantization; it prevents installation of the
|
||||
mixed-step runtime.
|
||||
|
||||
```bash
|
||||
vllm serve /path/to/Cosmos3-Nano-modelopt \
|
||||
--omni \
|
||||
--additional-config \
|
||||
'{"cosmos3_mixed_precision":{"first_steps":3,"last_steps":3,"reasoner":"a16"}}'
|
||||
```
|
||||
|
||||
The presence of `cosmos3_mixed_precision` supplies an explicit runtime
|
||||
override; an empty object uses the defaults shown above.
|
||||
The schedule uses vLLM's selected native backend and fails during loading if
|
||||
its live weight layout cannot support A16 dequantization.
|
||||
The runtime selects FP8 or NVFP4 from the checkpoint's ModelOpt configuration.
|
||||
BF16 linears are untouched. Mixed FP8/NVFP4 checkpoints are not supported by
|
||||
this schedule.
|
||||
|
||||
## Validation and Notes
|
||||
|
||||
1. Compare the ModelOpt checkpoint against the BF16 baseline with the same
|
||||
|
||||
@@ -6,22 +6,25 @@ W4A4 MXFP4 (Microscaling FP4) quantizes both weights and activations to FP4
|
||||
(`float4_e2m1fn_x2`, packed 2 values per byte) using the OCP MX format: groups
|
||||
of 32 K-dimension elements share a single `float8_e8m0fnu` exponent scale.
|
||||
|
||||
On Ascend, single-scale MXFP4 also supports Universal Optimal Scaling
|
||||
(UOS, `Qmax=7.25`), described in the single-scale scaling section below.
|
||||
|
||||
vLLM-Omni provides two quantization methods with different scale structures:
|
||||
|
||||
| Method | Scale structure | Mode | Use case |
|
||||
|--------|----------------|------|----------|
|
||||
| `mxfp4` | Single-scale (per-32 fine only) | Online only | Quick accuracy baseline; no checkpoint prep needed |
|
||||
| `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | Production; better accuracy; offline recommended |
|
||||
| -------- | ---------------- | ------ | ---------- |
|
||||
| `mxfp4` | Single-scale (per-32 fine only) | Online + pre-quantized single-scale weights | Online baseline, or an existing single-scale checkpoint in the layout below |
|
||||
| `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | W4A4 with dual-level scales; offline mode loads calibrated Smooth tensors |
|
||||
|
||||
!!! tip "Recommended: `mxfp4_dualscale` offline"
|
||||
For production deployments, use the `mxfp4_dualscale` offline mode with a
|
||||
!!! tip "Existing W4A4-only DualScale deployments"
|
||||
For W4A4-only deployments, the existing `mxfp4_dualscale` offline mode uses a
|
||||
pre-quantized checkpoint produced by msModelSlim. Offline checkpoints load
|
||||
calibrated `mul_scale` tensors from disk, providing measurably better accuracy
|
||||
than any online method. The one-time preprocessing cost amortises across all
|
||||
subsequent inference runs.
|
||||
calibrated `mul_scale` tensors from disk. Quality depends on the model,
|
||||
calibration recipe and BF16 fallback policy; evaluate the checkpoint against
|
||||
your quality target before deployment.
|
||||
|
||||
Use `mxfp4` online only for quick experimentation where preprocessing time
|
||||
is not acceptable and accuracy loss is tolerable.
|
||||
W4A8 layer/step fallback requires single-scale `mxfp4`, online or offline.
|
||||
DualScale supports W4A4 and BF16 routing only.
|
||||
|
||||
!!! warning "Online single-scale ≠ Offline dual-scale"
|
||||
`mxfp4_dualscale` offline mode uses `NPUMxfp4DualScaleLinearMethod`:
|
||||
@@ -34,61 +37,244 @@ vLLM-Omni provides two quantization methods with different scale structures:
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | Support |
|
||||
|--------|---------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ⭕ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ⭕ |
|
||||
| AMD ROCm (gfx950 / MI355X) | ✅ |
|
||||
| Intel XPU | ⭕ |
|
||||
| Ascend NPU (Atlas 950 A5) | ✅ |
|
||||
The following matrix describes these vLLM-Omni implementations, rather than
|
||||
hardware support for the MXFP4 format in general.
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
|
||||
| Device | Single-scale online OCP | Single-scale offline | UOS / W4A8 fallback | DualScale |
|
||||
| -------- | -------- | -------- | -------- | -------- |
|
||||
| Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ✅ | ✅ |
|
||||
| AMD ROCm (gfx950 / MI355X) | ✅ | ❌ | ❌ | ❌ |
|
||||
| Other AMD architectures | ❌ | ❌ | ❌ | ❌ |
|
||||
| NVIDIA GPU | ❌ | ❌ | ❌ | ❌ |
|
||||
| Intel XPU | ❌ | ❌ | ❌ | ❌ |
|
||||
|
||||
Legend: `✅` supported by the implementation, `❌` unsupported.
|
||||
ROCm uses the existing AITER online path and requires `mxfp4_scale_alg=0`.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
### Diffusion Model (Wan2.2)
|
||||
|
||||
| Model | Online | Offline | Notes |
|
||||
|-------|--------|---------|-------|
|
||||
| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade (`transformer` + `transformer_2`); both transformers quantized with the same config |
|
||||
| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade; same scheme as T2V-A14B |
|
||||
| Wan2.2-TI2V-5B | ❌ | ❌ | Parameter count too small; W4A4 causes unacceptable accuracy loss |
|
||||
| ------- | -------- | --------- | ------- |
|
||||
| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4` / `mxfp4_dualscale` | Cascade experts resolve checkpoint metadata independently; explicit global runtime policy applies to both |
|
||||
| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4` / `mxfp4_dualscale` | Per-expert configuration uses the same interface as T2V; validate I2V checkpoints separately |
|
||||
| Wan2.2-TI2V-5B | Requires validation | Requires validation | This guide does not establish checkpoint compatibility or generation quality for TI2V-5B |
|
||||
|
||||
The choice between `mxfp4` and `mxfp4_dualscale` in **online mode** is about
|
||||
quantization quality, not model compatibility — both work on cascade (A14B) and
|
||||
single-transformer models alike, the same as `mxfp8` online:
|
||||
Both methods can be integrated with single-transformer and cascade models.
|
||||
This alone does not establish support or quality for every model variant:
|
||||
|
||||
- `mxfp4`: single-scale, lower overhead, simpler compute, online only
|
||||
- `mxfp4_dualscale`: dual-scale + optional BF16 fallback, better accuracy, online **and** offline
|
||||
- `mxfp4`: single-scale; online and pre-quantized weights, with optional W4A8 fallback on NPU
|
||||
- `mxfp4_dualscale`: dual-scale; online and offline, with BF16 fallback
|
||||
|
||||
**Offline** checkpoints for A14B are always in `mxfp4_dualscale` format (produced
|
||||
by the merge script); there is no offline `mxfp4` single-scale format.
|
||||
The existing `merge_mxfp4_dualscale_checkpoint.py` produces **offline
|
||||
`mxfp4_dualscale`** checkpoints. Single-scale weights must satisfy the separate
|
||||
layout described below; changing a DualScale checkpoint's method name to
|
||||
`mxfp4` does not convert its weights. Prepare single-scale checkpoints with
|
||||
external tooling. Real exported checkpoint compatibility and model quality
|
||||
require validation of that checkpoint; the synthetic tests do not establish
|
||||
UOS export or video acceptance.
|
||||
|
||||
!!! note "Per-layer BF16 fallback in offline cascade models"
|
||||
The A14B offline checkpoint uses `quant_method: mxfp4_dualscale`. Most
|
||||
An A14B offline DualScale checkpoint uses `quant_method: mxfp4_dualscale`. Most
|
||||
linear layers are stored as W4A4 MXFP4 DualScale; precision-sensitive layers
|
||||
retain their original BF16 weights and are listed in `ignored_layers` inside
|
||||
each transformer's `config.json`. The two transformers may have different
|
||||
`ignored_layers` sets — the pipeline reads each transformer's own `config.json`
|
||||
and rebuilds the config locally when they differ, so routing is always
|
||||
per-transformer-accurate.
|
||||
|
||||
!!! warning "TI2V-5B not supported"
|
||||
Wan2.2-TI2V-5B is excluded from W4A4 quantization. Its smaller parameter
|
||||
count makes it significantly more sensitive to 4-bit quantization noise,
|
||||
resulting in unacceptable accuracy loss. Use [MXFP8](mxfp8.md) for TI2V-5B.
|
||||
and resolves the storage format and BF16 layer policy independently.
|
||||
|
||||
## Configuration
|
||||
|
||||
### W4A8 fallback at selected layers and denoising steps (Ascend NPU)
|
||||
|
||||
For single-scale `mxfp4`, set `w4a8_fallback_steps` to a list of zero-based
|
||||
request denoising loop indices (not scheduler timestep values).
|
||||
Set `w4a8_fallback_layers` to exact runtime Linear paths, for example
|
||||
`["blocks.10.attn1.to_qkv", "blocks.12.ffn.net_2"]`. A selected layer uses W4A8
|
||||
at every step; at a selected step, all other quantized layers also use W4A8.
|
||||
The rules are combined with **OR**. W4A8 uses MXFP8 activations with MXFP4
|
||||
weights. Omitting both lists or passing `[]` keeps the existing W4A4 behavior. Existing
|
||||
`ignored_layers` take priority: **BF16 > (layer OR step) W4A8 > W4A4**.
|
||||
DualScale's existing leading BF16 block rule is unchanged; it does not enable
|
||||
W4A8.
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
omni = Omni(
|
||||
model="<Wan2.2-T2V-A14B-Diffusers>",
|
||||
quantization_config={
|
||||
"method": "mxfp4",
|
||||
"w4a8_fallback_layers": ["blocks.10.attn1.to_qkv"],
|
||||
"w4a8_fallback_steps": [0, 1, 38, 39],
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Both policies also work with offline single-scale `mxfp4`. Any nonempty
|
||||
W4A8 list with `mxfp4_dualscale`, online or offline, raises an explicit error
|
||||
at configuration construction. Empty lists preserve its existing W4A4 path.
|
||||
Each expert's checkpoint still determines its serialized
|
||||
format and BF16 layer routing. A serialized checkpoint config is the complete
|
||||
storage contract: `ignored_layers`, or its `modules_to_not_convert` alias,
|
||||
selects the BF16 layers, and omitting both means no explicit BF16 layers. A
|
||||
method-only checkpoint value such as `"mxfp4"` or
|
||||
`{"quant_method": "mxfp4"}` carries no storage policy and leaves a caller's
|
||||
online `ignored_layers` unchanged. An explicitly supplied quantization config
|
||||
retains its W4A8 layer and step lists when the loader rebuilds either expert's
|
||||
config from disk, including empty lists that disable checkpoint policies.
|
||||
Within an explicit runtime config, an omitted W4A8 list defaults to `[]`;
|
||||
individual omitted fields do not inherit saved policies. Without an explicit
|
||||
runtime config, auto-detection uses the saved checkpoint policies.
|
||||
|
||||
Layer names are relative to each expert (`blocks.10...`, without a
|
||||
`transformer.` or `transformer_2.` prefix). A global config applies the same
|
||||
list to both experts; per-component `transformer`/`transformer_2` configs can
|
||||
select different lists. Wan selects these expert names exactly: a `transformer`
|
||||
entry does not match `transformer_2`; an omitted expert uses `default` if set.
|
||||
A component mapped to `null` (or an unmatched component
|
||||
without a default) stays unquantized and requires a BF16 checkpoint; loading a
|
||||
quantized checkpoint for that disabled component raises an error. A top-level
|
||||
`quantization_config=None` retains checkpoint auto-detection.
|
||||
Names are exact, with no regex, wildcard, block expansion,
|
||||
or checkpoint-name remapping. Wan validates them against its runtime Linear
|
||||
modules and logs the selected paths/count and BF16 overrides per PP rank.
|
||||
Use `attn1.to_qkv` for fused self-attention; `attn1.to_q`, `to_k`, or `to_v`
|
||||
checkpoint names raise an error. Cross-attention `attn2.to_q/to_k/to_v` remains
|
||||
separate. Layer-only fallback does not require a denoise ForwardContext.
|
||||
|
||||
The text-to-video example accepts the JSON directly:
|
||||
|
||||
```bash
|
||||
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \
|
||||
python examples/offline_inference/text_to_video/text_to_video.py \
|
||||
--model /path/to/Wan2.2-T2V-A14B-Diffusers \
|
||||
--quantization-config '{"method":"mxfp4","w4a8_fallback_steps":[0,2]}' \
|
||||
--tensor-parallel-size 1 --ulysses-degree 2 --cfg-parallel-size 2 \
|
||||
--enforce-eager --height 480 --width 832 --num-frames 81 \
|
||||
--num-inference-steps 40 --seed 42 --output wan22-w4a8.mp4
|
||||
```
|
||||
|
||||
Use either `--quantization` or `--quantization-config` in this example.
|
||||
Serving accepts the same JSON through `--diffusion-quantization-config`.
|
||||
|
||||
The list uses one request's loop index across both high- and low-noise experts,
|
||||
and both CFG branches see the same index. A new request starts at step zero.
|
||||
Indices outside the request's step range are never selected. A pipeline must
|
||||
publish `ForwardContext.denoise_step_idx`; requesting step fallback without
|
||||
that context raises an error instead of silently ignoring the list.
|
||||
|
||||
Single-scale W4A4 and W4A8 reuse the same prepared `weight` and `weight_scale`.
|
||||
Online loading quantizes the original BF16/FP16 weight once; offline loading
|
||||
packs the numeric FP4 values and reshapes the saved E8M0 scale without
|
||||
requantizing the weight. Only activation precision changes during forward.
|
||||
There is no DualScale decoding, second W4 representation or `w4a8_weight` /
|
||||
`w4a8_weight_scale` cache. No weight quantization runs inside the denoising loop.
|
||||
|
||||
The mixed FP8/FP4 GEMM uses BF16 output and a matching BF16 row bias for
|
||||
compatibility with the target A5 runtime. An FP16 caller receives that result
|
||||
converted back to FP16; its output shape and dtype are preserved.
|
||||
|
||||
The single-scale fallback validation target is eager Wan2.2 T2V A14B on A5.
|
||||
Quality scores, optimal fallback lists, real UOS exports and additional
|
||||
compile/offload combinations require separate evaluation. Existing W4A4-only
|
||||
DualScale still needs complete 512-channel input groups; use TP=1 with
|
||||
sequence/CFG parallelism for its initial multi-card run.
|
||||
|
||||
### `mxfp4` — Pre-Quantized Single-Scale Weights (Ascend NPU)
|
||||
|
||||
Set the following fields in each transformer's `config.json` when a
|
||||
single-scale checkpoint uses calibrated Smooth scaling:
|
||||
|
||||
```json
|
||||
{
|
||||
"quantization_config": {
|
||||
"quant_method": "mxfp4",
|
||||
"is_checkpoint_mxfp4_serialized": true,
|
||||
"require_smooth_scale": true,
|
||||
"ignored_layers": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each quantized Linear must provide tensors under its model parameter name:
|
||||
|
||||
| Tensor | Checkpoint layout | Meaning |
|
||||
| -------- | ------------------- | --------- |
|
||||
| `weight` | `(N, K)` numeric FP4 values | Packed to FP4 after loading |
|
||||
| `weight_scale` | `(N, ceil(K / 32))`, `uint8` | Raw E8M0 exponent bytes, required |
|
||||
| `mul_scale` | `(K,)`, FP32 | Smooth pre-scale, required when declared above |
|
||||
|
||||
Both W4A4 and W4A8 quantize `x * mul_scale`. The Smooth tensor is converted
|
||||
to the layer's activation dtype after loading. Row-parallel layers load the
|
||||
corresponding input-channel shard. Pre-fused QKV weights must use one shared
|
||||
Smooth tensor for the fused projection.
|
||||
|
||||
For backward compatibility, `require_smooth_scale` defaults to `false`:
|
||||
an absent tensor uses identity scaling, while a present tensor is still loaded.
|
||||
Without a declaration, an absent Smooth tensor cannot be distinguished from a
|
||||
legacy unsmoothed checkpoint. Calibrated checkpoints should declare
|
||||
`require_smooth_scale: true` so missing or misnamed Smooth tensors fail loading;
|
||||
incorrect tensor shapes, non-floating tensors, NaN/Inf and nonpositive values
|
||||
also fail. Values that overflow or underflow to zero in the activation dtype
|
||||
are rejected during weight processing. The flag requires serialized single-scale
|
||||
weights and is rejected for online quantization. It does not make DualScale
|
||||
scales optional or change missing-parameter policies for other backends.
|
||||
Either the checkpoint or an explicit caller can require Smooth: a `true`
|
||||
declaration is retained when per-expert metadata rebuilds the configuration,
|
||||
including when the other source omits the flag or sets it to `false`.
|
||||
|
||||
Prepare the canonical numeric FP4 checkpoint before inference using external
|
||||
tooling. Runtime loading does not parse producer-specific export formats.
|
||||
Retain the producer recipe to identify the weight-generation algorithm and
|
||||
Smooth calibration; tensor storage alone does not establish UOS provenance.
|
||||
|
||||
### `mxfp4` — Single-Scale Block Scaling (OCP / UOS)
|
||||
|
||||
`mxfp4_scale_alg` selects the MXFP4 block-scaling method on NPU.
|
||||
UOS is a scaling rule, not a separate activation dtype or an activation-only
|
||||
algorithm. Both choices use FP4 E2M1 values with shared E8M0 scales.
|
||||
|
||||
| Mode | Weight preparation | A4 activation quantization | W4A8 fallback |
|
||||
| --- | --- | --- | --- |
|
||||
| Online | Selected scaling method, once at loading | Selected scaling method | Same W4; OCP MXFP8 A8 |
|
||||
| Offline | Load existing W4 and scales unchanged | Selected scaling method | Same W4; OCP MXFP8 A8 |
|
||||
|
||||
The available settings are:
|
||||
|
||||
| Setting | MXFP4 scaling method | NPU parameters |
|
||||
| --- | --- | --- |
|
||||
| `0` (default) | OCP MX | `scale_alg=0` |
|
||||
| `2` | UOS (`Qmax=7.25`) | `scale_alg=2, dst_type_max=7.25` |
|
||||
|
||||
Universal Optimal Scaling (UOS, `Qmax=7.25`) is derived in
|
||||
[MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention](https://arxiv.org/abs/2607.24377),
|
||||
Section 4.1. The value `7.25` is the boundary used to select the shared
|
||||
power-of-two scale; the maximum finite E2M1 value remains `6`.
|
||||
This option applies UOS to Linear quantization. It does not implement the
|
||||
paper's full MXAttention pipeline or Pre-Normalization Quantization (PNQ).
|
||||
|
||||
Both use FP4 E2M1 with E8M0 scales, `axis=-1`, `block_size=32`,
|
||||
`round_mode="rint"`. W4A8 always quantizes the activation to
|
||||
`float8_e4m3fn` with `scale_alg=0` and the same grouping/rounding, without
|
||||
UOS parameters. A4 and A8 share the prepared W4 and scale. UOS is currently
|
||||
NPU-only; ROCm retains its existing AITER online algorithm.
|
||||
|
||||
An explicit UOS online configuration is
|
||||
`{"method": "mxfp4", "mxfp4_scale_alg": 2}`. For offline checkpoints the
|
||||
setting affects **only A4 activations**. Existing weights are never requantized:
|
||||
a UOS checkpoint can run UOS W4 + UOS A4 or UOS W4 + OCP A4, and both fall back to
|
||||
UOS W4 + OCP A8. Record the weight-generation algorithm separately from the
|
||||
runtime activation setting. Smooth remains active with either setting. The
|
||||
UOS configuration explicitly selects `2`; attention computation is unchanged.
|
||||
|
||||
### `mxfp4` — Single-Scale Online Mode
|
||||
|
||||
Online mode requires no pre-processing. vLLM-Omni quantizes BF16 weights to
|
||||
MXFP4 at load time using `npu_dynamic_mx_quant`. A single block scale
|
||||
(`float8_e8m0fnu`, one per 32 K elements) is computed on the fly; no
|
||||
calibration `mul_scale` is available. Applies equally to single-transformer
|
||||
and cascade (A14B) models — both transformers in a cascade receive the same
|
||||
quantization config automatically.
|
||||
Online mode quantizes BF16/FP16 weights once at loading and computes activation
|
||||
scales during forward. Each block of 32 K elements shares one E8M0 scale;
|
||||
no calibrated `mul_scale` is available. An explicit global quantization config
|
||||
applies to both cascade experts. Per-component configs can select different
|
||||
policies; checkpoint auto-detection resolves each expert independently.
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
@@ -103,7 +289,7 @@ outputs = omni.generate(
|
||||
|
||||
```bash
|
||||
# Single-transformer or cascade model — same command
|
||||
python text_to_video.py --model <your-model> --quantization mxfp4
|
||||
python examples/offline_inference/text_to_video/text_to_video.py --model <your-model> --quantization mxfp4
|
||||
|
||||
# Online serving
|
||||
vllm serve <your-model> --omni --quantization mxfp4
|
||||
@@ -113,19 +299,19 @@ vllm serve <your-model> --omni --quantization mxfp4
|
||||
|
||||
Online DualScale mode computes both fine and coarse scales on the fly from BF16
|
||||
weights using `npu_dynamic_dual_level_mx_quant`. Applies equally to
|
||||
single-transformer and cascade (A14B) models. Compared to `mxfp4` online,
|
||||
DualScale provides better quantization accuracy at higher compute cost.
|
||||
single-transformer and cascade (A14B) models with compatible Linear layers.
|
||||
Compare quality and performance using the intended model and workload.
|
||||
|
||||
The default configuration keeps the leading 5 transformer blocks in BF16
|
||||
(`num_bf16_fallback_layers=5`). Accuracy evaluation on Wan2.2-A14B shows this
|
||||
is sufficient to meet quality requirements and is the recommended setting.
|
||||
(`num_bf16_fallback_layers=5`). This is an implementation default; select and
|
||||
validate the BF16 policy against your own quality target.
|
||||
|
||||
```python
|
||||
omni = Omni(model="<your-model>", quantization="mxfp4_dualscale")
|
||||
```
|
||||
|
||||
```bash
|
||||
python text_to_video.py --model <your-model> --quantization mxfp4_dualscale
|
||||
python examples/offline_inference/text_to_video/text_to_video.py --model <your-model> --quantization mxfp4_dualscale
|
||||
```
|
||||
|
||||
If accuracy debugging identifies additional precision-sensitive layers, they
|
||||
@@ -136,21 +322,22 @@ omni = Omni(
|
||||
model="<your-model>",
|
||||
quantization_config={
|
||||
"method": "mxfp4_dualscale",
|
||||
"ignored_layers": ["blocks.10.attn1.to_q"], # explicit per-layer override
|
||||
"ignored_layers": ["blocks.10.attn1.to_qkv"], # explicit per-layer override
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
BF16 fallback routing in online mode applies two rules in priority order:
|
||||
|
||||
1. **`ignored_layers`** (explicit per-layer override): any layer whose prefix
|
||||
matches is kept in BF16 regardless of block index.
|
||||
1. **`ignored_layers`** (explicit per-layer override): a listed runtime Linear
|
||||
module is kept in BF16 regardless of block index. Use exact module paths,
|
||||
not arbitrary block prefixes or names ending in `.weight`.
|
||||
2. **`num_bf16_fallback_layers`** (coarse leading-block rule): the first N
|
||||
transformer blocks (`blocks.0` … `blocks.N-1`) fall back to BF16. Defaults
|
||||
to `5` (recommended). Layers outside `blocks.N.*`
|
||||
(e.g. `condition_embedder`) are always quantized.
|
||||
to `5`. Layers outside `blocks.N.*` are quantized unless explicitly listed
|
||||
in `ignored_layers`.
|
||||
|
||||
### `mxfp4_dualscale` — DualScale Offline Mode (Recommended)
|
||||
### `mxfp4_dualscale` — DualScale Offline Mode
|
||||
|
||||
Offline mode loads a pre-quantized DualScale checkpoint from msModelSlim. A
|
||||
preprocessing step converts the raw quantized output to the diffusers format
|
||||
@@ -169,8 +356,8 @@ inside `config.json`. At runtime, each layer's prefix is matched against
|
||||
Each quantized linear layer stores four tensors:
|
||||
|
||||
| Tensor | Shape | dtype | Description |
|
||||
|--------|-------|-------|-------------|
|
||||
| `weight` | `(N, K)` | float8_e4m3fn | FP4 packed (2 values per byte) |
|
||||
| -------- | ------- | ------- | ------------- |
|
||||
| `weight` | `(N, K)` | float8_e4m3fn | Numeric FP4 values; packed into two values per byte after loading |
|
||||
| `weight_scale` | `(N, K//32)` | uint8 | Fine block scale (`float8_e8m0fnu` bit pattern) |
|
||||
| `weight_dual_scale` | `(N, K//512, 1)` | float32 | Coarse block scale |
|
||||
| `mul_scale` | `(K,)` | float32 | Per-input-channel smooth pre-scale (from calibration) |
|
||||
@@ -223,7 +410,7 @@ python vllm_omni/quantization/tools/merge_mxfp4_dualscale_checkpoint.py \
|
||||
```
|
||||
|
||||
| Argument | Description |
|
||||
|----------|-------------|
|
||||
| ---------- | ------------- |
|
||||
| `--model-type` | Model variant: `Wan2.2-T2V-A14B` or `Wan2.2-I2V-A14B` |
|
||||
| `--original-model` | Root directory of the original BF16 diffusers model |
|
||||
| `--quant-path` | Root directory of the msModelSlim quantized output |
|
||||
@@ -251,14 +438,14 @@ The `quantization_config` injected into `config.json` for each transformer:
|
||||
```
|
||||
|
||||
`ignored_layers` lists every linear layer that retains its original BF16 weight,
|
||||
using vllm-omni model parameter names (QKV-fused, FFN underscored, `to_out`
|
||||
using vllm-omni runtime module paths (QKV-fused, FFN underscored, `to_out`
|
||||
unindexed). The exact entries are determined by the quantization tool (msModelSlim)
|
||||
and may differ between `transformer` and `transformer_2` in a cascade model.
|
||||
|
||||
#### Step 3 — Serve
|
||||
|
||||
```bash
|
||||
python text_to_video.py --model /path/to/Wan2.2-T2V-A14B-MXFP4-DualScale
|
||||
python examples/offline_inference/text_to_video/text_to_video.py --model /path/to/Wan2.2-T2V-A14B-MXFP4-DualScale
|
||||
|
||||
# Online serving
|
||||
vllm serve /path/to/Wan2.2-T2V-A14B-MXFP4-DualScale --omni
|
||||
@@ -275,25 +462,32 @@ omni = Omni(model="/path/to/Wan2.2-T2V-A14B-MXFP4-DualScale")
|
||||
|
||||
## Parameters
|
||||
|
||||
### `mxfp4` (single-scale, online only)
|
||||
### `mxfp4` (single-scale, online + offline)
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| ----------- | ------ | --------- | ------------- |
|
||||
| `method` | str | — | `"mxfp4"` |
|
||||
| `ignored_layers` | list[str] | `[]` | Layer prefixes to keep in BF16 |
|
||||
| `mxfp4_scale_alg` | int | `0` | MXFP4 block scaling: `0` OCP MX, `2` UOS (`Qmax=7.25`, NPU only). Online: W4 and A4; offline: A4 only. A8 remains OCP MXFP8 |
|
||||
| `is_checkpoint_mxfp4_serialized` | bool | `false` | Load pre-quantized single-scale weights and scales; declared in checkpoint `config.json` |
|
||||
| `require_smooth_scale` | bool | `false` | Require calibrated Smooth tensors for offline single-scale weights |
|
||||
| `ignored_layers` | list[str] | `[]` | Exact runtime Linear module paths to keep in BF16 |
|
||||
| `w4a8_fallback_layers` | list[str] | `[]` | Exact runtime Linear paths that always use A8; NPU only |
|
||||
| `w4a8_fallback_steps` | list[int] | `[]` | Zero-based denoise steps that use A8; NPU only |
|
||||
|
||||
### `mxfp4_dualscale` (dual-scale, online + offline)
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| ----------- | ------ | --------- | ------------- |
|
||||
| `method` | str | — | `"mxfp4_dualscale"` |
|
||||
| `is_checkpoint_serialized` | bool | `False` | `True` for offline DualScale checkpoints; auto-set from `config.json` when using the preprocessing script |
|
||||
| `ignored_layers` | list[str] | `[]` | Layer prefixes to keep in BF16. **Works in both modes**: offline — populated by the merge script for interleaved sensitive layers; online — user-supplied for explicit per-layer precision override |
|
||||
| `num_bf16_fallback_layers` | int | `5` | **Online mode only**: leading N transformer blocks (`blocks.0` … `blocks.N-1`) kept in BF16. Applied after `ignored_layers`; ignored in offline mode. Default of `5` is the evaluated recommended value for Wan2.2-A14B |
|
||||
| `ignored_layers` | list[str] | `[]` | Exact runtime Linear module paths to keep in BF16. **Works in both modes**: offline — populated by the merge script for interleaved sensitive layers; online — user-supplied for explicit per-layer precision override |
|
||||
| `num_bf16_fallback_layers` | int | `5` | **Online mode only**: leading N transformer blocks (`blocks.0` … `blocks.N-1`) kept in BF16. Applied after `ignored_layers`; ignored in offline mode. Validate the default of `5` against the intended model and quality target |
|
||||
| `w4a8_fallback_layers` | list[str] | `[]` | Must remain empty; DualScale does not support W4A8 fallback |
|
||||
| `w4a8_fallback_steps` | list[int] | `[]` | Must remain empty; DualScale does not support W4A8 fallback |
|
||||
|
||||
#### BF16 fallback priority (online mode)
|
||||
|
||||
```
|
||||
```text
|
||||
for each linear layer:
|
||||
if prefix in ignored_layers → BF16 (explicit override, highest priority)
|
||||
elif block_idx < num_bf16_fallback_layers → BF16 (coarse leading-block rule)
|
||||
@@ -307,8 +501,9 @@ unless they appear in `ignored_layers`.
|
||||
|
||||
1. **Online single-scale (`mxfp4`)** quantizes BF16 weights at load time using
|
||||
`npu_dynamic_mx_quant` (single-scale). No calibration `mul_scale` is
|
||||
available — all output partitions receive an identity pre-scale. No offline
|
||||
checkpoint format exists for this method.
|
||||
available — all output partitions receive an identity pre-scale. Offline
|
||||
single-scale checkpoints use the canonical numeric FP4 layout
|
||||
described above.
|
||||
|
||||
2. **Online dual-scale (`mxfp4_dualscale`, `is_checkpoint_serialized=False`)**
|
||||
quantizes BF16 weights using `npu_dynamic_dual_level_mx_quant` (fine + coarse
|
||||
@@ -316,7 +511,7 @@ unless they appear in `ignored_layers`.
|
||||
explicit `ignored_layers` stay in BF16 for accuracy.
|
||||
|
||||
3. **Offline dual-scale (`mxfp4_dualscale`, `is_checkpoint_serialized=True`)** —
|
||||
**recommended for production** — loads four tensors per quantized layer: FP4
|
||||
loads four tensors per quantized layer: FP4
|
||||
weight, fine scale (`uint8` reinterpreted as `float8_e8m0fnu`), coarse scale
|
||||
(`float32`), and per-input-channel `mul_scale` (`float32`). BF16 fallback
|
||||
layers have no quantization tensors and are routed via `ignored_layers`.
|
||||
@@ -325,21 +520,21 @@ unless they appear in `ignored_layers`.
|
||||
layout as `float8_e8m0fnu`) and reinterpreted at load time without a lossy
|
||||
float32 round-trip.
|
||||
|
||||
5. **Cascade model config propagation**: in a cascade model (transformer +
|
||||
transformer_2), vLLM-Omni reads each transformer's own `config.json` and
|
||||
rebuilds the quant config locally when `ignored_layers` differs between
|
||||
transformers, ensuring per-layer routing is accurate for each. The first
|
||||
transformer's config is propagated to `od_config` so the second transformer
|
||||
can reuse it as a starting point.
|
||||
5. **Cascade model configuration**: checkpoint auto-detection resolves each
|
||||
expert's own `config.json` independently; the first expert's detected config
|
||||
is not reused as the second expert's policy. An explicit global runtime
|
||||
config applies to both experts, while per-component configs select policies
|
||||
by the exact `transformer` / `transformer_2` names. Serialized checkpoint
|
||||
metadata still determines each expert's storage format and BF16 layers.
|
||||
|
||||
6. **Self-attention QKV fusion**: Q, K, V projection weights are fused into a
|
||||
single `QKVParallelLinear` layer at runtime. `ignored_layers` entries use the
|
||||
fused name (`attn1.to_qkv`), written automatically by the merge script.
|
||||
|
||||
7. W4A4 carries higher quantization noise than W8A8 (16 vs 256 levels). The
|
||||
DualScale offline method mitigates this with calibrated `mul_scale` smooth
|
||||
quantization. Use `ignored_layers` and `num_bf16_fallback_layers` to trade
|
||||
off compression vs. accuracy for precision-sensitive layers.
|
||||
7. FP4 has fewer representable values than FP8, so evaluate quantization
|
||||
error and generation quality for the chosen recipe. Offline Smooth scaling
|
||||
and BF16 fallback can help with sensitive layers.
|
||||
`num_bf16_fallback_layers` applies only to online DualScale.
|
||||
|
||||
## Adapting MXFP4 for a New Model
|
||||
|
||||
@@ -350,27 +545,26 @@ a merge script for offline checkpoints.
|
||||
|
||||
### Step 1 — Discover runtime layer names
|
||||
|
||||
`ignored_layers` entries must match the **runtime parameter names** used inside
|
||||
vllm-omni, which may differ from the names stored in the diffusers checkpoint.
|
||||
The canonical source of truth is the model's own `named_parameters()`.
|
||||
`ignored_layers` entries use exact **runtime Linear module paths**, without
|
||||
`.weight`. Inspect the actual transformer instance in model initialization or
|
||||
a developer harness; `Omni` does not expose an `omni.pipeline.transformer` API.
|
||||
For a transformer instance named `transformer`:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm.model_executor.layers.linear import LinearBase
|
||||
|
||||
# Load the model without quantization to inspect parameter names.
|
||||
omni = Omni(model="/path/to/your-model") # no --quantization flag
|
||||
for name, _ in omni.pipeline.transformer.named_parameters():
|
||||
if "weight" in name and "scale" not in name:
|
||||
for name, module in transformer.named_modules():
|
||||
if isinstance(module, LinearBase):
|
||||
print(name)
|
||||
```
|
||||
|
||||
Compare the printed names against the diffusers checkpoint keys
|
||||
Compare these module paths against the corresponding diffusers checkpoint keys
|
||||
(`safetensors.safe_open` or `torch.load`) to identify any renames your model
|
||||
applies. Common patterns that differ in Wan2.2 (and may appear in other
|
||||
models):
|
||||
|
||||
| Diffusers checkpoint name | vllm-omni runtime name | Reason |
|
||||
|---------------------------|------------------------|--------|
|
||||
| --------------------------- | ------------------------ | -------- |
|
||||
| `attn1.to_q`, `attn1.to_k`, `attn1.to_v` | `attn1.to_qkv` | Self-attention Q/K/V fused into `QKVParallelLinear` |
|
||||
| `ffn.net.0.proj` | `ffn.net_0.proj` | Dots in sub-module names replaced with underscores |
|
||||
| `ffn.net.2` | `ffn.net_2` | Same underscore rule |
|
||||
@@ -382,8 +576,9 @@ fused runtime parameters.
|
||||
|
||||
!!! warning "Partial QKV fallback is not allowed"
|
||||
If your model fuses Q, K, V into a single layer, `ignored_layers` must
|
||||
include **all three or none**. A partial fallback (e.g. `to_q` in BF16 but
|
||||
`to_k`, `to_v` quantized) cannot be expressed at runtime because they share
|
||||
name the fused runtime module (for Wan, `attn1.to_qkv`). The source
|
||||
checkpoint must keep **all three or none** in BF16. A partial fallback
|
||||
(e.g. `to_q` in BF16 but `to_k`, `to_v` quantized) cannot be expressed at runtime because they share
|
||||
one `QKVParallelLinear`. The merge script enforces this and raises an error
|
||||
if only some of the trio appear as non-quantized.
|
||||
|
||||
@@ -392,12 +587,13 @@ fused runtime parameters.
|
||||
#### Online mode
|
||||
|
||||
Pass `ignored_layers` directly in the quantization config using the **runtime
|
||||
names** discovered in Step 1. No code changes to the model are required.
|
||||
names** discovered in Step 1. This assumes the model already passes its
|
||||
quantization config and module prefixes to compatible vLLM Linear layers.
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="/path/to/your-model",
|
||||
quantization={
|
||||
quantization_config={
|
||||
"method": "mxfp4_dualscale",
|
||||
"ignored_layers": [
|
||||
"blocks.0.attn1.to_qkv", # runtime name, not diffusers name
|
||||
@@ -409,9 +605,14 @@ omni = Omni(
|
||||
```
|
||||
|
||||
```bash
|
||||
# CLI does not support list-typed ignored_layers directly.
|
||||
# Use the Python API or set ignored_layers in config.json (offline).
|
||||
python your_script.py --model /path/to/your-model --quantization mxfp4_dualscale
|
||||
# The text-to-video example accepts list-valued fields through JSON.
|
||||
python examples/offline_inference/text_to_video/text_to_video.py \
|
||||
--model /path/to/your-model \
|
||||
--quantization-config '{"method":"mxfp4_dualscale","ignored_layers":["blocks.0.attn1.to_qkv"]}'
|
||||
|
||||
# Serving accepts the same configuration.
|
||||
vllm serve /path/to/your-model --omni \
|
||||
--diffusion-quantization-config '{"method":"mxfp4_dualscale","ignored_layers":["blocks.0.attn1.to_qkv"]}'
|
||||
```
|
||||
|
||||
The `num_bf16_fallback_layers` coarse rule is an alternative to listing layers
|
||||
@@ -436,9 +637,13 @@ the merge script is correct. The injected block:
|
||||
}
|
||||
```
|
||||
|
||||
To add a layer manually (e.g. to pin an additional layer to BF16 without
|
||||
re-running the merge script), edit `config.json` inside the transformer
|
||||
subfolder. Use runtime names, not diffusers checkpoint names.
|
||||
Adding an offline BF16 fallback layer requires both the original BF16 weight
|
||||
from the base model and a matching `ignored_layers` entry. Regenerate the
|
||||
checkpoint with the revised storage policy, restoring the original weight
|
||||
(and bias, if present) and removing that layer's quantization tensors.
|
||||
Editing `config.json` alone does not reconstruct BF16 weights from stored FP4
|
||||
values. For fused QKV, restore all three projections together. Use runtime
|
||||
module paths in `ignored_layers`.
|
||||
|
||||
### Step 3 — Write a merge script for offline mode
|
||||
|
||||
@@ -474,6 +679,7 @@ renaming rules and collect the results.
|
||||
!!! tip "Validate before serving"
|
||||
After producing the offline checkpoint, load it without a `--quantization`
|
||||
flag and verify that vLLM-Omni auto-detects the correct method. Check that
|
||||
the layer count reported in the startup log matches expectations: quantized
|
||||
layer count + `ignored_layers` count should equal total linear layer count.
|
||||
Any mismatch indicates a name-mapping bug in the merge script.
|
||||
each intended runtime Linear module resolves to the expected quantized or
|
||||
BF16 method and that every BF16 entry has original BF16 weights. Account
|
||||
for fused modules and, in online DualScale, the leading-block fallback rule;
|
||||
the raw length of `ignored_layers` alone is not a reliable layer count.
|
||||
@@ -13,7 +13,7 @@ For the internal architecture and backend extension points, see the
|
||||
| Mode | Guide | Description | Methods |
|
||||
| ------ | ------- | ------------- | --------- |
|
||||
| Online quantization | [Online Quantization](online.md) | vLLM-Omni computes quantized weights and scales while loading the model. | FP8 W8A8, Int8 W8A8, BitsAndBytes W4, MXFP8 W8A8, MXFP4 W4A4 |
|
||||
| Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion Flash Attention tensors during inference. | FP8 FA |
|
||||
| Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion attention tensors during inference. | FP8, MXFP8, MXFP4 |
|
||||
| Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, TorchAO, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
|
||||
|
||||
## Hardware Support
|
||||
|
||||
@@ -2,41 +2,36 @@
|
||||
|
||||
## Overview
|
||||
|
||||
In DiT-based image and video generation, Flash Attention can take a large share
|
||||
of denoising time, especially for high-resolution or long-frame workloads.
|
||||
vLLM-Omni supports online FP8 quantization for eligible diffusion Flash
|
||||
Attention (FA) to reduce FA latency while keeping model weights in their
|
||||
original dtype.
|
||||
In DiT-based image and video generation, attention can take a large share of
|
||||
denoising time, especially for high-resolution or long-frame workloads.
|
||||
vLLM-Omni supports online Q/K/V quantization for eligible diffusion attention
|
||||
paths while keeping model weights in their original dtype.
|
||||
|
||||
This feature is configured through `diffusion_kv_cache_dtype` on
|
||||
`OmniDiffusionConfig` (CLI: `--diffusion-kv-cache-dtype`). It is intentionally
|
||||
**not** the same as vLLM's `--kv-cache-dtype`, which controls autoregressive
|
||||
language-model KV cache storage and defaults to `"auto"`. Diffusion FA
|
||||
quantization uses the dedicated diffusion flags so omni serve does not inherit
|
||||
that default.
|
||||
Runtime attention precision is configured through `diffusion_kv_cache_dtype`
|
||||
on `OmniDiffusionConfig` (CLI: `--diffusion-kv-cache-dtype`). This setting is
|
||||
separate from vLLM's `--kv-cache-dtype`, which controls autoregressive
|
||||
language-model KV cache storage.
|
||||
|
||||
In vLLM-Omni diffusion pipelines, this is a runtime FA path: Q/K/V tensors are
|
||||
dynamically quantized before the attention operator. It does not quantize model
|
||||
weights and is separate from [FP8 W8A8](fp8.md), [Int8 W8A8](int8.md), or
|
||||
pre-quantized checkpoint formats.
|
||||
|
||||
If `diffusion_kv_cache_dtype` is not set, behavior is unchanged and attention
|
||||
runs in the native dtype.
|
||||
If `diffusion_kv_cache_dtype` is not set, attention runs in the native dtype.
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | FP8 FA |
|
||||
|--------|--------|
|
||||
| Ascend NPU | ✅ |
|
||||
| NVIDIA GPU | ❌ |
|
||||
| AMD ROCm | ❌ |
|
||||
| Intel XPU | ❌ |
|
||||
| Device | Dense FP8 | Dense MXFP8/MXFP4 | BSA FP8/MXFP4 |
|
||||
|--------|------------|-------------------|----------------|
|
||||
| Ascend NPU | ✅ | ✅ | ✅ |
|
||||
| NVIDIA GPU | ❌ | ❌ | ❌ |
|
||||
| AMD ROCm | ❌ | ❌ | ❌ |
|
||||
| Intel XPU | ❌ | ❌ | ❌ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported.
|
||||
|
||||
FP8 FA is currently implemented only for the NPU Flash Attention backend. Other
|
||||
backends do not support `diffusion_kv_cache_dtype="fp8"` for diffusion attention
|
||||
and fall back to native dtype execution.
|
||||
These runtime methods are implemented only by the NPU Flash Attention and BSA
|
||||
backends. Other backends reject an incompatible explicit configuration.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
@@ -44,7 +39,7 @@ and fall back to native dtype execution.
|
||||
|
||||
| Model | Scope | Status | Notes |
|
||||
|-------|-------|--------|-------|
|
||||
| Wan2.2 | Eligible DiT full-attention FA on Ascend NPU | Tested | Compare quality and latency against a BF16 baseline before production use |
|
||||
| Wan2.2 | Self-attention on Ascend NPU | NPU interface and T2V smoke tested | Compare quality and latency against a BF16 baseline before production use |
|
||||
| Other diffusion models | Eligible DiT full-attention FA on Ascend NPU | Not tested | You can try `diffusion_kv_cache_dtype="fp8"`; tune `diffusion_kv_cache_skip_steps` and `diffusion_kv_cache_skip_layers` when higher precision is needed |
|
||||
|
||||
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
||||
@@ -105,12 +100,13 @@ The legacy keyword aliases `kv_cache_dtype`, `kv_cache_skip_steps`, and
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `diffusion_kv_cache_dtype` | str \| None | `None` | Set to `"fp8"` to enable dynamic FP8 FA on supported attention backends |
|
||||
| `diffusion_kv_cache_dtype` | str \| None | `None` | Runtime attention method. Dense FA supports `fp8`, `mxfp8`, and `mxfp4`; BSA supports `fp8` and `mxfp4` |
|
||||
| `diffusion_kv_cache_skip_steps` | str \| None | `None` | Denoising step selector to keep in native dtype, for example `"0,1,4-6"` |
|
||||
| `diffusion_kv_cache_skip_layers` | str \| None | `None` | Transformer layer selector to keep in native dtype, for example `"0-2,10"` |
|
||||
|
||||
Selectors use comma-separated integers and inclusive ranges. Listed steps or
|
||||
layers skip FP8 FA; all other eligible full-attention forwards use the FP8 path.
|
||||
layers run floating-point attention through the selected Dense or BSA backend;
|
||||
all other eligible forwards use `diffusion_kv_cache_dtype`.
|
||||
|
||||
## Validation and Notes
|
||||
|
||||
@@ -123,3 +119,60 @@ layers skip FP8 FA; all other eligible full-attention forwards use the FP8 path.
|
||||
4. Report both latency and quality results when enabling this option for a new
|
||||
model. For image or video models, include visual comparison and quantitative
|
||||
metrics when available, such as PSNR or SSIM.
|
||||
|
||||
## Wan2.2 T2V quantized attention on Ascend
|
||||
|
||||
Wan2.2 T2V A14B supports quantized self-attention through MindIE-SD. Model
|
||||
weights are unchanged, and model-level opt-out keeps cross-attention in its
|
||||
native dtype.
|
||||
|
||||
| Attention path | self-attention `backend` | Supported `diffusion_kv_cache_dtype` |
|
||||
| --- | --- | --- |
|
||||
| Dense FA | `FLASH_ATTN` | `fp8`, `mxfp8`, `mxfp4` |
|
||||
| BSA | `RAINFUSION_ATTN` | `fp8`, `mxfp4` |
|
||||
|
||||
Configure the method and fallback selectors once at the diffusion stage. The
|
||||
per-role attention config only selects the self-attention backend:
|
||||
|
||||
```yaml
|
||||
stages:
|
||||
- stage_id: 0
|
||||
diffusion_kv_cache_dtype: "mxfp8"
|
||||
diffusion_kv_cache_skip_layers: "0,39"
|
||||
diffusion_kv_cache_skip_steps: "0,1,38,39"
|
||||
diffusion_attention_config:
|
||||
per_role:
|
||||
self:
|
||||
backend: FLASH_ATTN # Use RAINFUSION_ATTN with fp8 or mxfp4 for BSA.
|
||||
```
|
||||
|
||||
For 40 denoising steps, `diffusion_kv_cache_skip_steps: "0,1,38,39"` and
|
||||
`diffusion_kv_cache_skip_layers: "0,39"` are recommended starting points. Steps
|
||||
are zero-based across the complete request and do not reset when Wan switches
|
||||
transformers; layer indices are zero-based and local to each transformer.
|
||||
Selectors accept comma-separated indices and inclusive ranges such as
|
||||
`"0,3-5"`. A selected forward uses floating-point attention while preserving
|
||||
the Dense or BSA path.
|
||||
|
||||
Unsupported precision/input combinations raise an error and must be corrected
|
||||
in the configuration; operator errors are not retried with another precision.
|
||||
|
||||
For BSA, change the backend to `RAINFUSION_ATTN` and set
|
||||
`diffusion_kv_cache_dtype` to `fp8` or `mxfp4`.
|
||||
See [RainFusion attention](../diffusion/attention_backends/rainfusion.md) for
|
||||
sparse-path configuration and behavior.
|
||||
|
||||
### Requirement and interface test
|
||||
|
||||
Until these APIs are available in a MindIE-SD release, use revision
|
||||
[`8637b5333b0225381b215390fd09a8732e671cc4`](https://gitcode.com/zqxu/MindIE-SD/commit/8637b5333b0225381b215390fd09a8732e671cc4).
|
||||
Build the Python package, PyTorch plugin, and custom operators from that same
|
||||
checkout. With CANN sourced, run the NPU interface tests from the Omni checkout:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/platforms/npu/quant/test_kv_quant_npu.py \
|
||||
-k real_npu -vv -s -o addopts=''
|
||||
```
|
||||
|
||||
A skipped NPU test is not a pass. Validate generated video quality against a
|
||||
floating-point run with the same prompt, seed, resolution, and step count.
|
||||
Reference in new issue
Block a user