Merge main and preserve event-driven defaults in shared engine bases
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
Sy03 committed
commit
d2bb944863
371 files changed
+43235
-38379
No files matched your search
+6
-1
@@ -5,6 +5,9 @@
|
||||
Main entry points for vLLM-Omni inference and serving.
|
||||
|
||||
- [vllm_omni.entrypoints.async_omni.AsyncOmni][]
|
||||
- [vllm_omni.entrypoints.async_omni_base.AsyncOmniBase][]
|
||||
- [vllm_omni.entrypoints.duplex_omni.DuplexOmni][]
|
||||
- [vllm_omni.entrypoints.duplex_omni.DuplexSessionHandle][]
|
||||
- [vllm_omni.engine.cfg_companion_tracker.CfgCompanionTracker][]
|
||||
- [vllm_omni.entrypoints.cli.benchmark.base.OmniBenchmarkSubcommandBase][]
|
||||
- [vllm_omni.entrypoints.cli.benchmark.main.OmniBenchmarkSubcommand][]
|
||||
@@ -17,12 +20,13 @@ Main entry points for vLLM-Omni inference and serving.
|
||||
|
||||
## Clients
|
||||
|
||||
Client libraries that talk to a running vLLM-Omni server (the full-duplex Realtime client and its per-model session presets).
|
||||
Client libraries for full-duplex sessions: the Realtime WebSocket client, the in-process client over `DuplexOmni`, and the per-model session presets.
|
||||
|
||||
- [vllm_omni.clients.duplex.AudioDelta][]
|
||||
- [vllm_omni.clients.duplex.AudioFormat][]
|
||||
- [vllm_omni.clients.duplex.ConnectionResumed][]
|
||||
- [vllm_omni.clients.duplex.DuplexClient][]
|
||||
- [vllm_omni.clients.duplex.DuplexClientBase][]
|
||||
- [vllm_omni.clients.duplex.DuplexClientError][]
|
||||
- [vllm_omni.clients.duplex.DuplexConnectionError][]
|
||||
- [vllm_omni.clients.duplex.DuplexEvent][]
|
||||
@@ -44,6 +48,7 @@ Client libraries that talk to a running vLLM-Omni server (the full-duplex Realti
|
||||
- [vllm_omni.clients.duplex.TextDelta][]
|
||||
- [vllm_omni.clients.duplex.TranscriptDelta][]
|
||||
- [vllm_omni.clients.duplex.WebSocketTransport][]
|
||||
- [vllm_omni.clients.inline_duplex.InlineDuplexClient][]
|
||||
|
||||
## Inputs
|
||||
|
||||
|
||||
@@ -82,6 +82,7 @@ depends on the installed kernels and model path.
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `SPEAKER_SAMPLES_DIR` | Filesystem path; default `~/.cache/vllm-omni/speakers` | Speech server; read when speaker storage initializes | Environment-only setting. The directory is created; filesystem errors propagate. | Stable |
|
||||
| `SPEAKER_MAX_UPLOADED` | Integer; default `1000` | Speech server; read when speaker storage initializes | Environment-only setting. A non-integer logs a warning and uses `1000`; range is not otherwise validated. | Stable |
|
||||
| `VLLM_OMNI_ABORT_TIMEOUT` | Float seconds; default `2` | Engine abort wait for Videos DELETE and `generate()` cancel/error cleanup; read when `async_omni` imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |
|
||||
| `VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT` | Float seconds; default `600` | Diffusion engine async-output wait in `step_streaming`; resolved per call on the request path, not at import | Environment-only setting. A non-float or `<=0` value warns once and uses the default. | Experimental |
|
||||
| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; default `0` (off), except Qwen3-TTS defaults to on | Orchestration loop and the serving-side final-output drain; read once when the `Orchestrator` is constructed | Environment-only setting. An explicit value always wins. Values are stripped and case-normalized; any unrecognized value leaves the legacy poll loop selected. | Experimental |
|
||||
| `VLLM_OMNI_INPUT_WAIT_TIMEOUT_S` | Float seconds; default `600`; `<=0` disables | Full-payload input coordinator, not async-chunk transfer; read when the scheduler module imports in each worker | Environment-only setting. A non-float logs a warning and uses `600`. | Stable operational control |
|
||||
|
||||
@@ -17,6 +17,7 @@ Common `PipelineConfig` fields include:
|
||||
| ------- | ------------- |
|
||||
| `model_type` | Pipeline identifier used during model and config resolution. |
|
||||
| `default_deploy_config_name` | Bundled deploy YAML loaded when the user does not pass `deploy_config`. |
|
||||
| `duplex_plugin` | Dotted path of the model's `DuplexModelPlugin`. Set only for full-duplex models; it makes `vllm-omni serve` run the model through `DuplexOmni` (a server whose every surface runs on a duplex session) and the engine host a `DuplexOrchestrator` with the plugin loaded. |
|
||||
| `model_arch` | Default Hugging Face architecture for the pipeline. |
|
||||
| `hf_architectures` | Architecture names used to identify checkpoints whose `model_type` is shared. |
|
||||
| `hf_config_predicate` | Optional predicate used to select between pipelines with otherwise identical HF metadata. |
|
||||
@@ -52,7 +53,7 @@ The new deploy schema lives under `vllm_omni/deploy/` and is paired with a froze
|
||||
| ------- | ------ | ---------- | --------- | ------------- |
|
||||
| `base_config` | str (path) | optional | — | Overlay parent (relative or absolute). `stages:` / `platforms:` deep-merged by stage_id; other scalars overlay-wins. Intended for user-authored overlays; prod yamls stay flat. |
|
||||
| `async_chunk` | bool | optional | `true` | Enable chunked streaming between stages. Pin to `false` if the pipeline runs end-to-end. |
|
||||
| `session_mode` | str | optional | `"turn"` | Session behavior. MiniCPM-o 4.5 deploy YAMLs set `"duplex"` so `/v1/realtime` and chat share the same profile. |
|
||||
| `session_mode` | str | optional | `"turn"` | Session behavior. Duplex models (pipelines with a `duplex_plugin`, e.g. MiniCPM-o 4.5) set `"duplex"`; `vllm-omni serve` then serves it over `/v1/realtime?duplex=1` and `/v1/chat/completions`, both on duplex sessions. |
|
||||
| `active_stream_window` | int | optional | `0` | Number of active downstream stream slots; `0` preserves all-stream cycling. |
|
||||
| `duplex_session` | dict | optional | runtime defaults | Full-duplex session lifecycle, buffering, replay, and capacity limits. |
|
||||
| `connectors` | dict | optional | `null` | Named connector specs (`{name, extra}`). Referenced by each stage's `input_connectors` / `output_connectors`. See [Connector schema](#connector-schema). |
|
||||
|
||||
@@ -131,13 +131,13 @@ for the validated batching, attention, and quantization combinations.
|
||||
|
||||
### MiniMax-H3: conditioning, denoising, and decode
|
||||
|
||||
MiniMax-H3 illustrates logical component and phase boundaries between text
|
||||
encoding, task-specific denoising, and media decoding:
|
||||
`Text Encoder → DiT → VAE Decoder`. The current registered topology co-locates
|
||||
these components in one diffusion stage. These boundaries are useful for
|
||||
profiling and offload; independent placement would require a future
|
||||
multi-stage `PipelineConfig`. The stage mapping follows the implementation and
|
||||
serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3)
|
||||
MiniMax-H3 combines text and reference-media encoding, task-specific denoising,
|
||||
and media decoding. Standard serving co-locates these components in one diffusion
|
||||
stage. The opt-in
|
||||
[disaggregated topology](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3-Disaggregated.md)
|
||||
runs Qwen and the video/audio VAE encoders in Stage 0, then sends their conditioning
|
||||
to a Stage 1 containing the DiTs and VAE decoders. The stage mapping follows the
|
||||
implementation and serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3)
|
||||
notes that a detailed technical report is forthcoming.
|
||||
|
||||
#### Model architecture by component
|
||||
@@ -149,16 +149,15 @@ notes that a detailed technical report is forthcoming.
|
||||
| **VAE Decoder** | Video VAE and audio VAE decode the generated latents and assemble the synchronized output | Decoded video frames and stereo waveform, with FPS and audio sample rate metadata |
|
||||
|
||||
Both DiT partitions remain behind one task-selection point; `task` chooses
|
||||
which partition runs. In the current implementation, the three logical
|
||||
components execute inside one diffusion stage. This is different from
|
||||
Qwen3-Omni's three independently scheduled stages, even though both models
|
||||
produce synchronized audio and video-like outputs.
|
||||
which partition runs.
|
||||
|
||||
#### Representative component execution
|
||||
|
||||
The following table describes the standard single-stage deployment.
|
||||
|
||||
| Logical component | Batching | Attention and execution | Parallelism | Quantization |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Text Encoder** | The current co-located pipeline follows the diffusion request scheduler; independent prompt/reference batching would require a separate multi-stage topology | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks; independent placement is not part of the current topology | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder |
|
||||
| **Text Encoder** | Follows the diffusion request scheduler | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder |
|
||||
| **DiT** | The current H3 implementation executes one generation request per diffusion batch; use concurrency for service-level throughput | cuDNN attention for the validated two-GPU consumer profile; TRTLLM attention or FlashAttention-4 on supported Blackwell profiles | TP2 plus text-encoder TP and Ulysses/VAE parallel groups on multi-GPU profiles; DLO/CPU offload trade memory for transfer time | Online FP8 applies to eligible DiT linears and is incompatible with layerwise offload |
|
||||
| **VAE Decoder** | Decode follows each generation request; tile/patch work can be distributed even when denoising is not batched | VAE decode kernels rather than DiT attention | VAE patch parallelism and native tiled decode within the diffusion stage | BF16/FP32 baseline; the documented H3 FP8 path leaves both VAEs unchanged |
|
||||
|
||||
|
||||
@@ -76,6 +76,34 @@ preserve common `self` or `cross` policies. A model also decides whether its
|
||||
path is compatible with automatic TRTLLM selection, because only the model
|
||||
knows whether masking and packing satisfy the kernel contract.
|
||||
|
||||
### Model-owned implementation specializations
|
||||
|
||||
Use `Attention(..., impl_overrides={backend_name: implementation_class})`
|
||||
when a model needs different computation within an existing backend contract.
|
||||
The model defines the implementation in its own directory and passes it when
|
||||
constructing the layer:
|
||||
|
||||
```python
|
||||
# MyModelVSAImpl is a model-owned subclass of FastVideoVSAImpl.
|
||||
self.attention = Attention(
|
||||
num_heads=num_heads,
|
||||
head_size=head_size,
|
||||
softmax_scale=head_size**-0.5,
|
||||
causal=False,
|
||||
role="self",
|
||||
impl_overrides={"FASTVIDEO_VSA": MyModelVSAImpl},
|
||||
)
|
||||
```
|
||||
|
||||
The override is applied after role/platform selection, keyed by the selected
|
||||
backend's `get_name()`. It must subclass the **selected implementation**;
|
||||
incompatible platform implementations raise `TypeError`. Unselected overrides
|
||||
have no effect. Backend capabilities, constructor options, and shared
|
||||
parallel dispatch remain unchanged and must be honored by the specialization.
|
||||
|
||||
Use a new backend for a new provider/capability contract. `custom_attention`
|
||||
instead owns communication and requires `skip_sequence_parallel=True`.
|
||||
|
||||
## Adding or changing a backend
|
||||
|
||||
An implementation change is complete only when it:
|
||||
@@ -91,8 +119,9 @@ An implementation change is complete only when it:
|
||||
|
||||
## FastVideo VSA model metadata
|
||||
|
||||
FastVideo VSA support is scoped to `FastVideo/FastWan2.2-TI2V-5B-Diffusers`, whose text-to-video and image-to-video modes both use `Wan22Pipeline`. It operates on a flattened DiT sequence but partitions tokens in
|
||||
the original latent video grid. Wan integrations therefore attach the
|
||||
FastWan's text-to-video and image-to-video modes both use `Wan22Pipeline`.
|
||||
It operates on a flattened DiT sequence but partitions tokens in the original
|
||||
latent video grid. Wan integrations therefore attach the
|
||||
post-patch `(T, H, W)` shape as `vsa_dit_seq_shape` attention metadata. The
|
||||
backend validates that `T * H * W` equals the sequence length, derives the
|
||||
runtime block count, and selects the configured top-k key/value blocks for
|
||||
@@ -104,3 +133,8 @@ the checkpoint contains those weights. Checkpoints without the projection do
|
||||
not allocate or execute it. When top-k selects every block, native checkpoints
|
||||
route to SDPA, while FastVideo DMD checkpoints preserve the VSA all-block path
|
||||
to retain checkpoint semantics.
|
||||
|
||||
MiniMax-H3 supplies `MiniMaxH3VSAImpl` through `impl_overrides`.
|
||||
`models/minimax_h3/attention/fastvideo_h3.py` owns its prefix layout, video tiling, and
|
||||
learned compression gate. Pooling, block-map construction, and tile64 provider
|
||||
calls are shared through `attention/ops/block_sparse.py`.
|
||||
+359
-1056
File diff suppressed because it is too large.
Load diff
@@ -16,6 +16,7 @@ required_reviewers:
|
||||
- "@yenuo26"
|
||||
- "@NickCao"
|
||||
primary_code_paths:
|
||||
- vllm_omni/engine/omni_engine_base.py
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/engine/async_engine_utils.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
@@ -28,8 +29,12 @@ related_code_paths:
|
||||
- vllm_omni/engine/stage_pool.py
|
||||
- vllm_omni/outputs/output_processor.py
|
||||
- vllm_omni/distributed/omni_coordinator/**
|
||||
# Duplex kernel design is owned by ../fullduplex.md; listed here because the
|
||||
# orchestrator hosts the duplex control plane.
|
||||
# The duplex siblings (DuplexOmniEngine / DuplexOrchestrator) and the
|
||||
# engine-resident session framework are owned by ../fullduplex.md; listed
|
||||
# here because they fill the template seams of OmniEngineBase and
|
||||
# OrchestratorBase.
|
||||
- vllm_omni/engine/duplex_omni_engine.py
|
||||
- vllm_omni/engine/duplex_orchestrator.py
|
||||
- vllm_omni/engine/duplex/**
|
||||
depends_on:
|
||||
- input_output_modality_contracts.md
|
||||
@@ -48,6 +53,8 @@ validation_paths:
|
||||
- tests/engine/test_cfg_companion_lifecycle.py
|
||||
- tests/engine/test_rpc_result_router.py
|
||||
- tests/engine/duplex/**
|
||||
- tests/engine/test_duplex_orchestrator.py
|
||||
- tests/engine/test_duplex_omni_engine.py
|
||||
- tests/engine/test_duplex_import_boundary.py
|
||||
upstream_refs:
|
||||
- vllm.v1.engine.EngineCoreRequest
|
||||
|
||||
@@ -19,7 +19,9 @@ required_reviewers:
|
||||
primary_code_paths:
|
||||
- vllm_omni/entrypoints/omni.py
|
||||
- vllm_omni/entrypoints/async_omni.py
|
||||
- vllm_omni/entrypoints/async_omni_base.py
|
||||
- vllm_omni/entrypoints/omni_base.py
|
||||
- vllm_omni/entrypoints/duplex_omni.py
|
||||
- vllm_omni/entrypoints/cli/**
|
||||
- vllm_omni/entrypoints/openai/**
|
||||
- vllm_omni/entrypoints/openpi/**
|
||||
@@ -27,14 +29,13 @@ primary_code_paths:
|
||||
- vllm_omni/entrypoints/stage_utils.py
|
||||
- vllm_omni/entrypoints/utils.py
|
||||
- vllm_omni/entrypoints/duplex/**
|
||||
- vllm_omni/entrypoints/duplex_request_client.py
|
||||
- vllm_omni/clients/**
|
||||
primary_path_exceptions:
|
||||
- path: vllm_omni/entrypoints/openai/errors.py
|
||||
owner: error_contracts.md
|
||||
- path: vllm_omni/entrypoints/duplex/**
|
||||
owner: ../fullduplex.md
|
||||
- path: vllm_omni/entrypoints/duplex_request_client.py
|
||||
- path: vllm_omni/entrypoints/duplex_omni.py
|
||||
owner: ../fullduplex.md
|
||||
- path: vllm_omni/clients/**
|
||||
owner: ../fullduplex.md
|
||||
@@ -42,7 +43,9 @@ related_code_paths:
|
||||
- vllm_omni/errors.py
|
||||
- vllm_omni/inputs/**
|
||||
- vllm_omni/outputs/**
|
||||
- vllm_omni/engine/omni_engine_base.py
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/engine/duplex_omni_engine.py
|
||||
- vllm_omni/config/**
|
||||
- vllm_omni/deploy/**
|
||||
depends_on:
|
||||
@@ -53,7 +56,9 @@ validation_paths:
|
||||
- tests/entrypoints/test_omni_entrypoints.py
|
||||
- tests/entrypoints/test_async_omni.py
|
||||
- tests/entrypoints/test_async_omni_pause_sleep_routing.py
|
||||
- tests/entrypoints/test_async_omni_duplex.py
|
||||
- tests/entrypoints/test_duplex_omni.py
|
||||
- tests/entrypoints/duplex/**
|
||||
- tests/entrypoints/openai_api/test_duplex_api_server.py
|
||||
- tests/entrypoints/test_serve.py
|
||||
- tests/entrypoints/test_stream_finish_reason.py
|
||||
- tests/entrypoints/openai/**
|
||||
|
||||
@@ -81,6 +81,18 @@ To build a simple workflow yourself,
|
||||
- For some multi-stage models like BAGEL, [only one stage's sampling parameters are exposed and tunable via vLLM-Omni's online serving API](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/examples/online_serving/bagel/). Thus, these models are treated as single-stage ones. Please check the vLLM-Omni documentation on how to correctly set each model's sampling parameters.
|
||||
- For multi-stage models where all stages are either autoregression or diffusion, you can also connect only a single Sampling Params node, indicating that this set of sampling parameters will be used for all stages.
|
||||
|
||||
## MiniMax H3 text-to-video workflow
|
||||
|
||||
The **MiniMax H3 Text to Video** template uses the existing remote Generate Video,
|
||||
H3 Params, sampling, and Remote LoRA nodes. It includes Base settings and an
|
||||
optional Turbo preset, with native resolution, 24 FPS, and `17k+5` frame counts.
|
||||
ComfyUI receives the generated video and audio from the server; it does not load
|
||||
H3 weights locally. Configure the server-side LoRA path before enabling Turbo.
|
||||
|
||||
See the [H3 workflow guide](https://github.com/vllm-project/vllm-omni/blob/main/apps/ComfyUI-vLLM-Omni/docs/minimax-h3-t2v.md)
|
||||
for server setup, template import, Turbo configuration, recorded validation,
|
||||
and saved-video/audio checks.
|
||||
|
||||
## Examples & Screenshots
|
||||
|
||||
Please read the [ComfyUI integration's README](https://github.com/vllm-project/vllm-omni/tree/main/apps/ComfyUI-vLLM-Omni) for more info.
|
||||
@@ -25,8 +25,8 @@ CATEGORIES = {
|
||||
"clients": {
|
||||
"name": "Clients",
|
||||
"description": (
|
||||
"Client libraries that talk to a running vLLM-Omni server "
|
||||
"(the full-duplex Realtime client and its per-model session presets)."
|
||||
"Client libraries for full-duplex sessions: the Realtime WebSocket client, "
|
||||
"the in-process client over `DuplexOmni`, and the per-model session presets."
|
||||
),
|
||||
},
|
||||
"inputs": {
|
||||
|
||||
@@ -107,7 +107,7 @@ th {
|
||||
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
|
||||
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
|
||||
| `DyninOmniForConditionalGeneration` | Dynin-Omni | `snu-aidas/Dynin-Omni` | ✅︎ | | | ✅︎ | — |
|
||||
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | — |
|
||||
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) |
|
||||
| `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — |
|
||||
| `HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
|
||||
|
||||
+12
-5
@@ -145,15 +145,22 @@ interchangeably.
|
||||
| Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) |
|
||||
| Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) |
|
||||
| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) |
|
||||
| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` or `WS /v1/duplex` | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) |
|
||||
| Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`) | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) |
|
||||
| Generated video chunks | `WS /v1/realtime/video` | Start a diffusion request and receive fragmented MP4 | [Streaming Video Output](streaming_video_output_api.md) |
|
||||
| Robot policy inference | `WS /v1/realtime/robot/openpi` | Send MessagePack observations and receive action arrays | [OpenPI Robot Policy](openpi_api.md) |
|
||||
|
||||
All six routes are model- or configuration-dependent. In particular,
|
||||
`/v1/realtime` is not full duplex unless the client sets `duplex=1` and the
|
||||
deployment explicitly enables duplex sessions. Clients should also verify the
|
||||
duplex capability payload because the query-parameter form falls back to the
|
||||
ordinary realtime handler when duplex is unavailable.
|
||||
`/v1/realtime` is full duplex when the model is a duplex model (its pipeline
|
||||
declares a `duplex_plugin` and the deploy configuration sets
|
||||
`session_mode: duplex`); a stock Realtime client needs no vendor query
|
||||
parameter, and `duplex=0` is the explicit opt-out. Such a server serves the
|
||||
websocket route plus `POST /v1/chat/completions`, and no other turn-based
|
||||
HTTP route. On a duplex server that chat route is not the turn-based path:
|
||||
each request runs on a short-lived duplex session, so it holds one of
|
||||
`duplex_session.max_sessions` for its lifetime and answers at the model's
|
||||
real-time pace -- see [Full Duplex](full_duplex_api.md). Clients should
|
||||
also verify the duplex capability payload because the query-parameter form
|
||||
falls back to the ordinary realtime handler when duplex is unavailable.
|
||||
|
||||
## Related Endpoints
|
||||
|
||||
|
||||
@@ -15,40 +15,52 @@ capability gates, see the [Realtime Duplex API guide](realtime_duplex_api.md).
|
||||
|
||||
| Endpoint | Protocol | Recommended use |
|
||||
| --- | --- | --- |
|
||||
| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients |
|
||||
| `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing |
|
||||
|
||||
Both endpoints use the same duplex engine and require the same model-side
|
||||
adapter. Prefer the Realtime projection unless the native lifecycle events are
|
||||
specifically required.
|
||||
| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style events (the normative contract) | Applications and browser clients |
|
||||
| `WS /v1/duplex` | Alias of `/v1/realtime?duplex=1` (same protocol, same handler) | Clients that prefer a dedicated path |
|
||||
| Python: `DuplexOmni` / `InlineDuplexClient` | Typed commands and events in-process | Embedding the model without a server |
|
||||
|
||||
## Enable Full Duplex
|
||||
|
||||
The route becomes usable only when the deployment configuration explicitly
|
||||
sets:
|
||||
A model is served full duplex when its registered pipeline declares a
|
||||
`duplex_plugin` (the model's `DuplexModelPlugin`). That declaration alone
|
||||
decides it: `vllm-omni serve` constructs `DuplexOmni` instead of `AsyncOmni`,
|
||||
and every surface the server exposes is backed by a duplex session. It serves
|
||||
`/v1/realtime?duplex=1` (and its alias `/v1/duplex`),
|
||||
`POST /v1/chat/completions`, `/v1/models` and `/health`; every other
|
||||
turn-based HTTP route (speech, batch, embeddings, video, ...) answers "not
|
||||
available". Turn-based use of such a model stays available offline through the
|
||||
Python API (`Omni` / `AsyncOmni`).
|
||||
|
||||
`/v1/chat/completions` is the ordinary chat service running on the duplex
|
||||
engine: a request is a turn-based generation on the same stages, served
|
||||
alongside the live websocket sessions, with the request options the chat
|
||||
service supports. It is wired only when the model's plugin declares
|
||||
`DuplexCapabilities.supports_chat_completions`; on any other duplex model the
|
||||
route reports "not available". A model that should not serve the route at all
|
||||
lists it in the deploy configuration's `endpoint_restrictions`.
|
||||
|
||||
The deploy configuration of such a model must agree:
|
||||
|
||||
```yaml
|
||||
session_mode: duplex
|
||||
```
|
||||
|
||||
The selected model pipeline must also provide a duplex serving adapter.
|
||||
Model-native deployments configure the corresponding engine runtime extension
|
||||
and control plane as part of their registered pipeline.
|
||||
A duplex model started with a deploy configuration that does not set it fails
|
||||
at startup rather than falling back to turn-based serving.
|
||||
|
||||
!!! warning
|
||||
|
||||
`WS /v1/duplex` fails with `Duplex API is not available` when duplex is not
|
||||
enabled. By contrast, `/v1/realtime?duplex=1` falls back to the ordinary
|
||||
turn-based realtime handler when no duplex handler exists. Confirm that
|
||||
On a deployment that is *not* duplex, `WS /v1/duplex` fails with
|
||||
`Duplex API is not available` and `/v1/realtime?duplex=1` falls back to the
|
||||
ordinary turn-based realtime handler. Confirm that
|
||||
`session.created.session.capabilities` is present before treating the
|
||||
connection as full duplex.
|
||||
|
||||
The current unified-runtime integrations are:
|
||||
|
||||
- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5.yaml`
|
||||
(`session_mode: duplex`);
|
||||
- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex;
|
||||
- Nemotron VoiceChat, via its registered duplex plugin package.
|
||||
**MiniCPM-o 4.5** (`vllm_omni/deploy/minicpmo_4_5.yaml`) is the only model
|
||||
served over this endpoint today. PersonaPlex and Nemotron VoiceChat still carry
|
||||
their pre-framework duplex code: their pipelines declare no `duplex_plugin`, so
|
||||
they run turn-based until the follow-up PRs port them to the plugin contract
|
||||
(RFC [vllm-omni#7181](https://github.com/vllm-project/vllm-omni/issues/7181)).
|
||||
|
||||
JoyVL is a separate HTTP interaction orchestrator and does not use these
|
||||
WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
|
||||
@@ -101,22 +113,23 @@ deferred, treated as a short acknowledgement, or used to interrupt output.
|
||||
The `session.created` payload includes capability fields such as
|
||||
`supports_barge_in`, `supports_playback_ack`, `supports_multi_session`,
|
||||
`supports_session_resume`, and `chunk_period_ms`. Treat this payload as the
|
||||
runtime contract.
|
||||
runtime contract and branch on the flags, never on the model name: a model
|
||||
that supports native overlapping speech may still advertise
|
||||
`supports_barge_in=false` when destructive output interruption and model-state
|
||||
rewind are not validated for it. Capacity and session-resume behavior also
|
||||
depend on the selected deployment configuration. The per-model table lives in
|
||||
the [Realtime Duplex API guide](realtime_duplex_api.md#capability-negotiation-by-model).
|
||||
|
||||
For example, PersonaPlex supports native overlapping speech but currently
|
||||
advertises `supports_barge_in=false`; destructive output interruption and
|
||||
model-state rewind have not been validated for that integration. Capacity and
|
||||
session-resume behavior also depend on the selected deployment configuration.
|
||||
## Python API
|
||||
|
||||
## Native Protocol
|
||||
`vllm_omni.entrypoints.duplex_omni.DuplexOmni` runs the same engine-resident
|
||||
sessions in-process: `open_session()` returns a `DuplexSessionHandle` whose
|
||||
`submit()` takes typed `DuplexCommand` objects and whose `events()` yields
|
||||
typed `DuplexEvent` objects (each with a `to_realtime()` wire rendering).
|
||||
`vllm_omni.clients.inline_duplex.InlineDuplexClient` exposes that handle
|
||||
behind the `DuplexClient` API. See the
|
||||
[Realtime Duplex API guide](realtime_duplex_api.md#using-the-python-api).
|
||||
|
||||
`WS /v1/duplex` exposes lower-level lifecycle names including
|
||||
`session.create`, `input_audio_buffer.append`, `turn.signal`, `playback.ack`,
|
||||
and `session.close`. It returns native session, input, response, overlap, and
|
||||
error events. This protocol is experimental and may evolve with the runtime;
|
||||
applications should use the provided Realtime client where possible.
|
||||
|
||||
See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo),
|
||||
[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex),
|
||||
See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo)
|
||||
and the [full-duplex runtime design](../design/fullduplex.md)
|
||||
for model-specific validation and architecture details.
|
||||
+214
-135
@@ -1,17 +1,18 @@
|
||||
# Realtime Duplex API
|
||||
|
||||
vLLM-Omni serves full-duplex speech models over a WebSocket endpoint,
|
||||
`/v1/realtime`, that speaks the OpenAI Realtime vocabulary plus a
|
||||
`/v1/realtime?duplex=1`, that speaks the OpenAI Realtime vocabulary plus a
|
||||
set of duplex extensions: the model decides when to listen and when to speak,
|
||||
the user can talk over the assistant, playback progress is reported back so
|
||||
history stays honest, and a dropped connection can resume the same session.
|
||||
This page covers how to run a duplex deployment, how to drive it from Python
|
||||
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
|
||||
|
||||
The handler is enabled by either a model-native `session_mode: duplex`
|
||||
deployment or an explicit `duplex_session` configuration for turn-based
|
||||
Server VAD. The runtime architecture is described in
|
||||
[Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
The endpoint is served for models that ship a duplex plugin — currently
|
||||
MiniCPM-o 4.5 only — and just for deploy configurations that declare
|
||||
`session_mode: duplex`. PersonaPlex and Nemotron VoiceChat are ported to the
|
||||
plugin contract in follow-up PRs and are not served over this endpoint yet. The runtime architecture
|
||||
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -26,11 +27,38 @@ vllm-omni serve openbmb/MiniCPM-o-4_5 \
|
||||
```
|
||||
|
||||
`vllm_omni/deploy/minicpmo_4_5.yaml` declares `session_mode: duplex` and
|
||||
`duplex_session.max_sessions: 4`, which mounts the WebSocket routes
|
||||
`ws://<host>:8099/v1/realtime` (this page) and `ws://<host>:8099/v1/duplex`
|
||||
(the native dialect for raw-protocol clients) next to the usual HTTP API.
|
||||
PersonaPlex and Nemotron VoiceChat use `vllm_omni/deploy/personaplex.yaml` and
|
||||
`vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml` in the same way.
|
||||
`duplex_session.max_sessions: 4`. Because the MiniCPM-o 4.5 pipeline declares a
|
||||
`duplex_plugin`, `vllm-omni serve` runs it through `DuplexOmni`: the server
|
||||
mounts `ws://<host>:8099/v1/realtime?duplex=1` (this page; `ws://<host>:8099/v1/duplex`
|
||||
is an alias of the same route), `POST /v1/chat/completions`, `/v1/models` and
|
||||
`/health`, and nothing else. The remaining turn-based HTTP routes (speech,
|
||||
batch, embeddings, video, ...) are not served by a duplex model.
|
||||
|
||||
`/v1/chat/completions` is not a turn-based path on a duplex server: each
|
||||
request opens a short-lived duplex session and reads the answer off it. Audio
|
||||
content becomes an ordinary committed turn; text is seeded as the session's
|
||||
opening turn, because a model-native model takes a turn only when it hears
|
||||
speech. A model that cannot be seeded that way declares so, and a text request
|
||||
to it is refused with 400. Two consequences are worth sizing for:
|
||||
|
||||
- **Every request holds an admission slot** for its lifetime, so
|
||||
`duplex_session.max_sessions` caps HTTP concurrency as well as websocket
|
||||
sessions. Beyond it a request is refused with HTTP 503.
|
||||
- **Latency follows the model's own clock.** A model-native session generates
|
||||
per audio unit, so even a short answer costs what that many units cost --
|
||||
noticeably more than the same model would take turn-based.
|
||||
|
||||
`n > 1`, `logprobs`, `tools`, and image or video content parts have no
|
||||
representation in one duplex turn and are refused with HTTP 400 -- a Realtime
|
||||
conversation item carries text and audio only. `temperature`,
|
||||
`max_completion_tokens`, `modalities` and
|
||||
`chat_template_kwargs.use_tts_template` apply; the other
|
||||
`chat_template_kwargs` keys are logged as ignored. `extra_body` is passed to
|
||||
the session unchanged, exactly as a websocket client's session `extra_body`
|
||||
is, which is how a request asking for `modalities: ["text", "audio"]` supplies
|
||||
the `ref_audio` MiniCPM-o requires for audio output. A model that should not
|
||||
serve the route lists it in the deploy config's `endpoint_restrictions`,
|
||||
exactly as a turn-based model does.
|
||||
|
||||
### Run the Example Client
|
||||
|
||||
@@ -81,25 +109,24 @@ ships a preset:
|
||||
|
||||
| Preset | Input / output audio | What it sets |
|
||||
| --- | --- | --- |
|
||||
| `vllm_omni.clients.minicpmo_4_5.create_duplex_session_config(ref_audio=...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | `extra_body.native_duplex=True` (opt in to the model-native lane), `force_listen_count=0`, `overlap_policy="listen_only"`, `playback_commit_policy="ack_only"`; `ref_audio` is the assistant voice clip |
|
||||
| `vllm_omni.clients.minicpmo_4_5.create_duplex_session_config(ref_audio=...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | `force_listen_count=0`, `overlap_policy="listen_only"`, `playback_commit_policy="ack_only"`; `ref_audio` is the assistant voice clip |
|
||||
| `vllm_omni.clients.personaplex.create_duplex_session_config(voice="NATF2.pt", persona="")` | 24 kHz `pcm_f32le` / 24 kHz `pcm16` | bundled `.pt` voice prompt and the persona as `instructions` |
|
||||
| `SessionConfig(...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | model-neutral defaults; pass `extra_body`, `turn_detection`, `overlap_policy`, `playback_commit_policy`, `instructions`, `voice`, `temperature` yourself |
|
||||
|
||||
Keyword overrides on a preset replace the corresponding `SessionConfig`
|
||||
field, e.g. `create_duplex_session_config(ref_audio=..., temperature=0.6)`.
|
||||
|
||||
Constructor options: `session_id` (names the session the client creates; it
|
||||
does not resume or take over an existing session — entering the client always
|
||||
performs the `session.update` handshake), `reconnect` (a `ReconnectPolicy`,
|
||||
default five attempts with 0.25–4 s jittered backoff; `None` disables
|
||||
auto-resume), `heartbeat_interval_s` (default 30 s; `None` disables the lease
|
||||
heartbeat), `handshake_timeout_s`, and `connect` (a custom transport factory,
|
||||
used by tests and benchmarks). Resume is supported only as automatic reconnect
|
||||
within the same `DuplexClient` instance (see
|
||||
[Reconnect and resume](#reconnect-and-resume)); re-attaching from a new client
|
||||
or process requires the wire-level `session.resume` handshake
|
||||
(`resume_token`, `incarnation`, `last_received_server_event_seq`), which this
|
||||
client does not expose.
|
||||
Constructor options: `reconnect` (a `ReconnectPolicy`, default five
|
||||
attempts with 0.25–4 s jittered backoff; `None` disables auto-resume),
|
||||
`heartbeat_interval_s` (default 30 s; `None` disables the lease heartbeat),
|
||||
`handshake_timeout_s`, and `connect` (a custom transport factory, used by
|
||||
tests and benchmarks). The session id is allocated by the server and becomes
|
||||
`client.session_id` once `session.created` arrives; a client cannot choose
|
||||
it. Resume is supported only as automatic reconnect within the same
|
||||
`DuplexClient` instance (see [Reconnect and resume](#reconnect-and-resume));
|
||||
re-attaching from a new client or process requires the wire-level
|
||||
`session.resume` handshake (`resume_token`,
|
||||
`last_received_server_event_seq`), which this client does not expose.
|
||||
|
||||
### Stream audio and commit a turn
|
||||
|
||||
@@ -195,7 +222,7 @@ the stamped `event_id`.
|
||||
On a transport drop the client reconnects with the `ReconnectPolicy` backoff
|
||||
and sends `session.resume` from the last acknowledged `server_event_seq`.
|
||||
Replayed events are deduplicated, consumers see one `ConnectionResumed`
|
||||
event, and `client.incarnation` / `client.resume_token` are refreshed. The
|
||||
event, and `client.resume_token` is refreshed. The
|
||||
server keeps a detached session alive for `disconnect_grace_s` (30 s by
|
||||
default); after that a resume fails with `session_resume_expired`. Resume is
|
||||
gated by `capabilities.supports_session_resume` (MiniCPM-o 4.5 and Nemotron
|
||||
@@ -238,9 +265,67 @@ definitions out of the dependency-free client. `audio_bytes(response_id)`, `coun
|
||||
`openai-realtime-duplex` benchmark backend (`vllm-omni bench serve`) is built
|
||||
on the same helpers.
|
||||
|
||||
## Using the Python API
|
||||
|
||||
`vllm_omni.entrypoints.duplex_omni.DuplexOmni` is the in-process Python API
|
||||
of a duplex model: sessions run inside the engine, and the class only opens,
|
||||
resumes and closes them and pipes typed commands in and typed events out
|
||||
through a `DuplexSessionHandle`.
|
||||
|
||||
```python
|
||||
from vllm_omni.engine.duplex.commands import AppendAudio, Commit
|
||||
from vllm_omni.engine.duplex.events import AudioDelta, SessionClosed
|
||||
from vllm_omni.entrypoints.duplex_omni import DuplexOmni
|
||||
|
||||
omni = DuplexOmni(model="openbmb/MiniCPM-o-4_5", trust_remote_code=True)
|
||||
async with await omni.open_session({"modalities": ["audio", "text"], "ref_audio": ref_audio_data_url}) as session:
|
||||
async def consume():
|
||||
async for event in session.events(): # typed DuplexEvent objects
|
||||
if isinstance(event, AudioDelta):
|
||||
play(event.audio)
|
||||
elif isinstance(event, ResponseDone):
|
||||
break # one turn is enough here
|
||||
task = asyncio.create_task(consume())
|
||||
await session.submit(AppendAudio(audio=pcm16_bytes, format="pcm16", sample_rate_hz=16000))
|
||||
await session.submit(Commit())
|
||||
await task
|
||||
# leaving the block closes the session (the last event is SessionClosed)
|
||||
```
|
||||
|
||||
`open_session` takes a `DuplexSessionConfig` or the same Realtime `session`
|
||||
object the WebSocket handshake takes; the session id is always allocated by
|
||||
`DuplexOmni`. `DuplexSessionHandle` offers `submit(command)` plus one
|
||||
convenience wrapper per command (`append_audio`, `commit`, `create_response`,
|
||||
`cancel_response`, `barge_in`, `ack_playback`, `update`, `create_item`,
|
||||
`delete_item`, `truncate_item`, `heartbeat`, ...). `events()` is a
|
||||
single-consumer async iterator that ends after `session.closed` /
|
||||
`session.expired`; rejected commands come back as `ErrorEvent`s on it.
|
||||
Every event renders the wire JSON with `to_realtime()`, so anything written
|
||||
against the WebSocket protocol works unchanged on the typed stream.
|
||||
|
||||
`vllm_omni.clients.inline_duplex.InlineDuplexClient` wraps a `DuplexOmni`
|
||||
behind the `DuplexClient` API, so the same application code runs in-process
|
||||
(no server) or over the network:
|
||||
|
||||
```python
|
||||
from vllm_omni.clients.inline_duplex import InlineDuplexClient
|
||||
from vllm_omni.clients.minicpmo_4_5 import create_duplex_session_config
|
||||
|
||||
config = create_duplex_session_config(ref_audio=audio_data_url(Path("reference_voice.wav")))
|
||||
async with InlineDuplexClient(omni, model="openbmb/MiniCPM-o-4_5", config=config) as client:
|
||||
await client.stream_pcm(question)
|
||||
await client.commit()
|
||||
async for response in client.responses():
|
||||
...
|
||||
```
|
||||
|
||||
`examples/online_serving/barge_in_client.py --inline` drives the barge-in
|
||||
scenario this way. The inline client has no transport, so it has no
|
||||
reconnect, resume, heartbeat or event-acknowledgement machinery.
|
||||
|
||||
## API Reference
|
||||
|
||||
This section is the complete wire contract of `/v1/realtime`: how
|
||||
This section is the complete wire contract of `/v1/realtime?duplex=1`: how
|
||||
the dialect relates to the OpenAI Realtime protocol, which surfaces each
|
||||
model's capability flags gate, a catalogue of every client-to-server and
|
||||
server-to-client event, and a JSON example for each. `DuplexClient` speaks
|
||||
@@ -253,21 +338,26 @@ its protocol tests.
|
||||
|
||||
### Endpoint and transport
|
||||
|
||||
The endpoint is `ws(s)://<host>/v1/realtime`. Bare connections select the
|
||||
session handler when the deployment declares either `session_mode: duplex`
|
||||
or `duplex_session`. `?duplex=1` remains a compatibility alias, while
|
||||
`?duplex=0` selects the legacy Realtime handler. Optional query parameters
|
||||
are `model`, `session_id`,
|
||||
`autostart` (`0` means resume-only), `resume`, and `native_duplex`
|
||||
(`minicpmo45_native_duplex` is accepted as a deprecated alias and folded
|
||||
into the canonical name). Every message is one JSON object per WebSocket
|
||||
The endpoint is `ws(s)://<host>/v1/realtime`. On a deployment whose pipeline
|
||||
declares a `duplex_plugin` and whose deploy configuration declares
|
||||
`session_mode: duplex`, a bare connection is already a duplex session, so a
|
||||
stock Realtime client needs no vendor query parameter. `?duplex=1` (`1`,
|
||||
`true` or `on`) is the spelling this page uses and stays valid;
|
||||
`?duplex=0` (`0`, `false`, `off`) explicitly opts out and selects the
|
||||
turn-based Realtime handler, which a duplex server does not mount --
|
||||
that connection is answered with `Realtime API is not available`.
|
||||
`ws(s)://<host>/v1/duplex` is an alias that always selects the duplex
|
||||
handler. Optional query parameters are `model`, `autostart` (`0` means
|
||||
resume-only) and `resume`. Every message is one JSON object per WebSocket
|
||||
text frame, discriminated by `type`. Inbound events are applied in arrival
|
||||
order through one per-session mailbox; outbound events preserve that order
|
||||
and carry a monotonically increasing `server_event_seq`, which is the replay
|
||||
cursor for `session.resume`. Client events may carry an `event_id`, echoed in
|
||||
`error.error.event_id` when the event is rejected.
|
||||
|
||||
Session identity is `(session_id, incarnation)`; `attachment_generation`
|
||||
Session identity is the server-allocated `session_id` (`duplex-<uuid>`,
|
||||
announced in `session.created`; a `session_id` the client puts in the session
|
||||
object is ignored, and ids are never reused); `attachment_generation`
|
||||
names the current socket; `epoch` is the interruption fence that advances on
|
||||
every cancel, clear, or barge-in; `turn_id` counts committed model turns;
|
||||
`response_id` (item id `item_<response_id>`) names one assistant utterance.
|
||||
@@ -281,12 +371,13 @@ with no OpenAI counterpart.
|
||||
|
||||
The event vocabulary is uniform, but several surfaces are gated by the
|
||||
`capabilities` object the server returns in `session.created`; a client must
|
||||
branch on those flags rather than on the model name. The current model
|
||||
plugins advertise:
|
||||
branch on those flags rather than on the model name. MiniCPM-o 4.5 is the
|
||||
only model on the plugin contract today; the other two columns record what
|
||||
their integrations advertise once the follow-up PRs port them:
|
||||
|
||||
| Capability | MiniCPM-o 4.5 | PersonaPlex | Nemotron VoiceChat | Gated surface |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `implementation_level` | `model_native_duplex` (when `extra_body.native_duplex=true`) | `model_native_duplex` | `model_native_duplex` | model-owned `response.listen` / `response.speak`; the chat-fallback lane otherwise |
|
||||
| `implementation_level` | `model_native_duplex` | `model_native_duplex` | `model_native_duplex` | model-owned `response.listen` / `response.speak` (constant: every duplex model is model-native) |
|
||||
| `chunk_period_ms` | 1000 | 80 | 80 | the model unit that `response.listen` decisions and camera frames align to |
|
||||
| `supports_session_resume` | yes | no | yes | `session.resume`, `session.resumed`, `session.replaced`, replay after a transport drop |
|
||||
| `supports_barge_in` | yes | no | no | `barge_in`, `turn.signal{event:"barge_in"}`, `overlap_policy=barge_in_on_speech`, `turn_detection.server_vad` |
|
||||
@@ -316,14 +407,8 @@ work unmodified. The overlap falls into the three tiers used by the event catalo
|
||||
| Object shapes | `realtime.session`, `realtime.item`, `realtime.response`; item content parts `input_text` / `input_audio` / `output_text` / `output_audio`; `function_call` and `function_call_output` items; `previous_item_id` chaining; `output_index` / `content_index` addressing |
|
||||
| Sequencing | `response.created → output_item.added → content_part.added → deltas → *.done → content_part.done → output_item.done → response.done → rate_limits.updated`; `speech_started → speech_stopped → committed` |
|
||||
|
||||
Audio/transcript output uses only the current (non-beta) OpenAI event names
|
||||
(`response.output_audio.delta`, not the deprecated `response.audio.delta`),
|
||||
matching `openai.types.realtime` in the official `openai` Python SDK and what
|
||||
clients built against it (e.g. `livekit-plugins-openai`'s non-Azure code path)
|
||||
expect; a client still on the old beta SDK types would not recognize these
|
||||
events. Session fields and `conversation.item.*` events, in contrast, are
|
||||
deliberately dual: the server emits and accepts **both** the beta and the GA
|
||||
spellings at once (`output_modalities` alongside `modalities`,
|
||||
The server deliberately emits and accepts **both** the beta and the GA
|
||||
spellings at once (`response.output_audio.delta` alongside `output_modalities`,
|
||||
`conversation.item.added` alongside `conversation.item.created`) so either
|
||||
generation of OpenAI client parses the stream.
|
||||
|
||||
@@ -333,7 +418,7 @@ A stock client ignores the extra keys; the extensions are additive.
|
||||
|
||||
| Message | Extension |
|
||||
| --- | --- |
|
||||
| `session.created` / `session.updated` | top-level `incarnation`, `attachment_generation`, `resume_token`; inside `session`: `state`, `turn_state`, `epoch`, `turn_id`, `active_request_id`, `active_response_id`, `active_response_turn_id`, `overlap_policy`, `overlap_*_ms/rms`, `playback_commit_policy`, `playback`, `capabilities`, `ref_audio`, `extra_body`, `idle_timeout_s`, `response_format` |
|
||||
| `session.created` / `session.updated` | top-level `attachment_generation`, `resume_token`; inside `session`: `state`, `turn_state`, `epoch`, `turn_id`, `active_request_id`, `active_response_id`, `active_response_turn_id`, `overlap_policy`, `overlap_*_ms/rms`, `playback_commit_policy`, `playback`, `capabilities`, `ref_audio`, `extra_body`, `idle_timeout_s`, `response_format` |
|
||||
| `response.created` / `response.done` / `response.listen` | `response_id` at top level; the raw duplex event under `response.metadata` (`duplex_event`); `status_details.reason` uses vLLM reasons (`barge_in`, `client_cancelled`, `new_response`, …) |
|
||||
| `response.output_audio.delta` | `format`, `sample_rate_hz`, `metadata{session_id, epoch, model_speak, end_of_turn, audio_duration_ms, audio_text_marks, playback}` |
|
||||
| `response.speak` (inserted before the first delta) | not an OpenAI event, but rides the OpenAI response envelope (`response_id`, `item_id`, `output_index`, `content_index`) |
|
||||
@@ -349,14 +434,9 @@ Semantic divergences hidden behind shared names:
|
||||
before any commit; a commit may end in `response.listen` and no
|
||||
response, and the model may open a response with no commit at all.
|
||||
OpenAI: commit ⇒ item, `response.create` ⇒ exactly one response.
|
||||
- With `turn_detection={"type":"server_vad"}`, omitted `interrupt_response`
|
||||
defaults to `false` on turn-based models such as Qwen3-Omni, enabling
|
||||
endpointing and automatic responses without interruption. Explicit `true`
|
||||
is rejected there. Model-native duplex keeps the `true` default and rejects
|
||||
`false`; its VAD implies `overlap_policy=barge_in_on_speech`.
|
||||
This capability-dependent default differs from OpenAI's `true` default;
|
||||
the effective configuration is returned in `session.created` / `session.updated`.
|
||||
`semantic_vad` is unsupported.
|
||||
- `turn_detection.interrupt_response=false` is rejected;
|
||||
`create_response` is ignored; `semantic_vad` is unsupported; the VAD
|
||||
runs per session (Silero) and implies `overlap_policy=barge_in_on_speech`.
|
||||
- `rate_limits.updated` is always an empty list (compatibility only).
|
||||
- Cancellation never reuses a `response_id`; truncation is driven by
|
||||
`playback.ack` as well as `conversation.item.truncate`.
|
||||
@@ -365,11 +445,11 @@ Semantic divergences hidden behind shared names:
|
||||
|
||||
| Area | Messages / fields |
|
||||
| --- | --- |
|
||||
| Session lifetime | `session.heartbeat` / `session.heartbeat_ack`, `session.event_ack`, `session.close` / `session.closed`, `session.resume` / `session.resumed` / `session.replaced` / `session.expired` / `session.resync_required`; `server_event_seq`, `resume_token`, `incarnation`, `attachment_generation`; query params `?duplex=1`, `autostart`, `resume`, `native_duplex` |
|
||||
| Session lifetime | `session.heartbeat` / `session.heartbeat_ack`, `session.event_ack`, `session.close` / `session.closed`, `session.resume` / `session.resumed` / `session.replaced` / `session.expired` / `session.resync_required`; `server_event_seq`, `resume_token`, `attachment_generation`; query params `?duplex=1`, `autostart`, `resume` |
|
||||
| Turn-taking | `response.speak`, `response.listen`, `overlap.decision`, `overlap_policy`, `barge_in`, `turn.signal`, `input.cancel`, `input.text.append`, `epoch` / `turn_id` fencing, `force_listen` |
|
||||
| Playback truth | `playback.ack` / `playback.acknowledged`, `playback_commit_policy` (`ack_only` \| `commit_all_on_done`), `playback{generated_ms, sent_ms, played_ms, committed_ms}`, `history_committed`, `audio_text_marks` |
|
||||
| Model negotiation | `capabilities{implementation_level, supports_input_append, supports_barge_in, supports_session_resume, chunk_period_ms, input_modes, …}`, `ref_audio`, `extra_body` (`auto_response`, `native_duplex` — deprecated alias `minicpmo45_native_duplex` is folded into it, `force_listen_count`, `duplex_initial_user_text`) |
|
||||
| Diagnostics | `runtime.control`, `duplex.function_call.done`, the full error-code vocabulary (`REALTIME_ERROR_TYPES_BY_CODE` in `vllm_omni/entrypoints/duplex/realtime_state.py`) |
|
||||
| Model negotiation | `capabilities{implementation_level, supports_input_append, supports_barge_in, supports_session_resume, chunk_period_ms, input_modes, …}`, `ref_audio`, `extra_body` (`auto_response`, `force_listen_count`, `duplex_initial_user_text`) |
|
||||
| Diagnostics | `duplex.function_call.done`, the full error-code vocabulary (`REALTIME_ERROR_TYPES_BY_CODE` in `vllm_omni/engine/duplex/events.py`) |
|
||||
|
||||
#### Consequences for clients
|
||||
|
||||
@@ -433,11 +513,15 @@ remains an approximation, not word-level audio alignment.
|
||||
### Event catalogue
|
||||
|
||||
Every event type the Realtime route accepts or emits, with a worked example in the next section: 21
|
||||
client→server and 42 server→client. Aliases (`push_chunk`,
|
||||
`input.audio.append`, `input_text.append`, `push_text`, `signal_turn`,
|
||||
`audio.playback_ack`, `close_session`, `close`, `session_close`) share the
|
||||
payload of their canonical event and are not listed separately; the alias
|
||||
table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
client→server and 42 server→client. Aliases (`input_text.append`,
|
||||
`push_text`, `signal_turn`, `audio.playback_ack`, `close_session`, `close`)
|
||||
share the payload of their canonical event and are not listed
|
||||
separately; the alias table is `translate_realtime_command` in
|
||||
`vllm_omni/engine/duplex/realtime_commands.py`. The pre-Realtime WAV-append
|
||||
aliases `push_chunk` and `input.audio.append` are **not** accepted on this
|
||||
route: send `input_audio_buffer.append` with one of the supported input
|
||||
formats (`pcm16`, `pcm_s16le`, `s16le`, `pcm_f32le`, `g711_ulaw`,
|
||||
`g711_alaw`).
|
||||
|
||||
#### Client to server
|
||||
|
||||
@@ -447,18 +531,18 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
|
||||
| `session.heartbeat` | 3 | Keepalive; refreshes the engine session lease. |
|
||||
| `session.event_ack` | 3 | Acknowledges received server events by `server_event_seq`; trims the resume replay journal. |
|
||||
| `session.close` | 3 | Graceful close; answered with `session.closed`. |
|
||||
| `session.resume` | 3 | Re-attach a live session on a new socket (`resume_token`, `incarnation`, `last_received_server_event_seq`). |
|
||||
| `session.resume` | 3 | Re-attach a live session on a new socket (`session_id`, `resume_token`, `last_received_server_event_seq`). |
|
||||
| `input_audio_buffer.append` | 2 | Append one audio chunk. OpenAI name; extended with `is_speech`, `video_frames`, `duration_ms`, `audio_end_ms`, per-event `format`/`sample_rate_hz`. |
|
||||
| `input_audio_buffer.commit` | 2 | Seal the buffered utterance into a user item. OpenAI name; extended with `final`, `response_create`, `is_speech:false`. |
|
||||
| `input.commit` | 3 | Alias of `input_audio_buffer.commit` accepted by the runner. |
|
||||
| `input_audio_buffer.clear` | 1 | Drop un-committed input audio. |
|
||||
| `input.cancel` | 3 | Cancel pending input; advances `epoch`. |
|
||||
| `input.text.append` | 3 | Append text to the open input item (chat-fallback lane only). |
|
||||
| `input.text.append` | 3 | Append text input; model-native sessions reject it with `native_text_append_unsupported`. |
|
||||
| `conversation.item.create` | 1 | Add a completed user item (text, audio, or `function_call_output`) to history. |
|
||||
| `conversation.item.retrieve` | 1 | Fetch a stored item by id. |
|
||||
| `conversation.item.delete` | 1 | Delete a stored item by id. |
|
||||
| `conversation.item.truncate` | 1 | Truncate an assistant item's audio/transcript at `audio_end_ms`. |
|
||||
| `response.create` | 1 | Explicitly request a response with per-response overrides. |
|
||||
| `response.create` | 1 | Explicitly request a response with per-response overrides. Requires committed audio input; otherwise `response_create_without_input`. |
|
||||
| `response.cancel` | 1 | Cancel the active (or named) response; advances `epoch`. |
|
||||
| `output_audio_buffer.clear` | 1 | Discard queued output audio; advances `epoch` (OpenAI: WebRTC-only, here also WebSocket). |
|
||||
| `barge_in` | 3 | Explicit hard interrupt; requires `capabilities.supports_barge_in`. |
|
||||
@@ -469,10 +553,10 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
|
||||
|
||||
| Event type | Tier | Description |
|
||||
| --- | --- | --- |
|
||||
| `session.created` | 2 | Session opened. OpenAI name; adds `incarnation`, `attachment_generation`, `resume_token` and vLLM-Omni keys inside `session` (`epoch`, `turn_id`, `playback`, `capabilities`, ...). |
|
||||
| `session.created` | 2 | Session opened. OpenAI name; adds `attachment_generation`, `resume_token` and vLLM-Omni keys inside `session` (`id` is the server-allocated session id, `epoch`, `turn_id`, `playback`, `capabilities`, ...). |
|
||||
| `session.updated` | 2 | Echo of the effective session config after every `session.update` (same extended `session` object). |
|
||||
| `session.heartbeat_ack` | 3 | Reply to `session.heartbeat`. |
|
||||
| `session.closed` | 3 | Last event on the socket; `reason` ∈ `session_close`, `timeout`, `disconnect`, ... |
|
||||
| `session.closed` | 3 | Last event on the socket, emitted once the engine released the session; `reason` ∈ `client_close`, `disconnect`, `timeout`, `transport_error`, `shutdown`, ... |
|
||||
| `session.resumed` | 3 | Resume accepted; carries the new `attachment_generation` and rotated `resume_token`; journaled events are replayed after it. |
|
||||
| `session.replaced` | 3 | Sent to the superseded socket when another socket resumes the session. |
|
||||
| `session.expired` | 3 | Engine lease reaped (`disconnect_grace_expired`, `idle_ttl_expired`); socket closes after it. |
|
||||
@@ -508,8 +592,8 @@ table lives in the normative contract section of [Full-Duplex Runtime (MiniCPM-o
|
||||
| `duplex.function_call.done` | 3 | Raw fallback when a model tool call is malformed. |
|
||||
| `output_audio_buffer.cleared` | 1 | Reply to `output_audio_buffer.clear`. |
|
||||
| `playback.acknowledged` | 3 | Reply to `playback.ack`; returns the playback ledger and `history_committed`. |
|
||||
| `overlap.decision` | 3 | How an append that overlapped an active response was handled (`drop` \| `listen` \| `barge_in`). |
|
||||
| `runtime.control` | 3 | Diagnostic: an engine control signal was partially unsupported. |
|
||||
| `overlap.decision` | 3 | How an append that overlapped an active response was handled (`drop` \| `listen` \| `barge_in`); details under `metadata`. |
|
||||
| `turn.event` | 3 | A turn-taking signal the session accepted (`event`, resulting `turn_state`). |
|
||||
| `error` | 2 | Error envelope. OpenAI shape and classes; vLLM-Omni error codes. |
|
||||
|
||||
### Message examples
|
||||
@@ -528,7 +612,6 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
"event_id": "evt_0001",
|
||||
"session": {
|
||||
"model": "openbmb/MiniCPM-o-4_5",
|
||||
"session_id": "sess_demo_01",
|
||||
"modalities": ["audio", "text"],
|
||||
"instructions": "You are a concise voice assistant.",
|
||||
"input_audio_format": "pcm16",
|
||||
@@ -541,7 +624,6 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
"idle_timeout_s": 300,
|
||||
"extra_body": {
|
||||
"auto_response": true,
|
||||
"native_duplex": true,
|
||||
"force_listen_count": 0
|
||||
}
|
||||
}
|
||||
@@ -553,13 +635,12 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
```json
|
||||
{
|
||||
"type": "session.created",
|
||||
"incarnation": 0,
|
||||
"attachment_generation": 1,
|
||||
"resume_token": "rt_3f9c…",
|
||||
"session": {
|
||||
"object": "realtime.session",
|
||||
"type": "realtime",
|
||||
"id": "sess_demo_01",
|
||||
"id": "duplex-5f2a…",
|
||||
"model": "openbmb/MiniCPM-o-4_5",
|
||||
"state": "open",
|
||||
"turn_state": "idle",
|
||||
@@ -612,7 +693,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
`session.created` and every later `session.update`)
|
||||
|
||||
```json
|
||||
{"type": "session.updated", "session": {"object": "realtime.session", "id": "sess_demo_01", "…": "…"}}
|
||||
{"type": "session.updated", "session": {"object": "realtime.session", "id": "duplex-5f2a…", "…": "…"}}
|
||||
```
|
||||
|
||||
`C→S session.heartbeat` / `S→C session.heartbeat_ack`
|
||||
@@ -622,7 +703,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
```
|
||||
|
||||
```json
|
||||
{"type": "session.heartbeat_ack", "session_id": "sess_demo_01"}
|
||||
{"type": "session.heartbeat_ack", "session_id": "duplex-5f2a…"}
|
||||
```
|
||||
|
||||
`C→S session.event_ack`
|
||||
@@ -638,7 +719,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
```
|
||||
|
||||
```json
|
||||
{"type": "session.closed", "session_id": "sess_demo_01", "reason": "session_close"}
|
||||
{"type": "session.closed", "session_id": "duplex-5f2a…", "epoch": 3, "reason": "client_close", "event": {"type": "session.closed", "reason": "client_close"}}
|
||||
```
|
||||
|
||||
`C→S session.resume` (new socket, `?duplex=1&autostart=0`)
|
||||
@@ -646,8 +727,7 @@ additionally carries `"server_event_seq": <int>` (omitted below for brevity).
|
||||
```json
|
||||
{
|
||||
"type": "session.resume",
|
||||
"session_id": "sess_demo_01",
|
||||
"incarnation": 0,
|
||||
"session_id": "duplex-5f2a…",
|
||||
"resume_token": "rt_3f9c…",
|
||||
"last_received_server_event_seq": 40
|
||||
}
|
||||
@@ -659,8 +739,8 @@ events with `server_event_seq > 40` are replayed in order)
|
||||
```json
|
||||
{
|
||||
"type": "session.resumed",
|
||||
"session_id": "sess_demo_01",
|
||||
"incarnation": 0,
|
||||
"session_id": "duplex-5f2a…",
|
||||
"session": {"id": "duplex-5f2a…", "state": "open", "epoch": 3, "turn_id": 2, "...": "the engine's current session object"},
|
||||
"attachment_generation": 2,
|
||||
"resume_token": "rt_a71e…"
|
||||
}
|
||||
@@ -669,19 +749,19 @@ events with `server_event_seq > 40` are replayed in order)
|
||||
`S→C session.replaced` (sent to the superseded socket, which is then closed)
|
||||
|
||||
```json
|
||||
{"type": "session.replaced", "session_id": "sess_demo_01", "attachment_generation": 1}
|
||||
{"type": "session.replaced", "session_id": "duplex-5f2a…", "attachment_generation": 1}
|
||||
```
|
||||
|
||||
`S→C session.expired`
|
||||
|
||||
```json
|
||||
{"type": "session.expired", "session_id": "sess_demo_01", "incarnation": 0, "reason": "disconnect_grace_expired"}
|
||||
{"type": "session.expired", "session_id": "duplex-5f2a…", "reason": "disconnect_grace_expired"}
|
||||
```
|
||||
|
||||
`S→C session.resync_required`
|
||||
|
||||
```json
|
||||
{"type": "session.resync_required", "session_id": "sess_demo_01", "reason": "journal_gap"}
|
||||
{"type": "session.resync_required", "session_id": "duplex-5f2a…", "reason": "journal_gap"}
|
||||
```
|
||||
|
||||
#### Input
|
||||
@@ -772,7 +852,7 @@ silence/noise declaration (no runtime append; answered with `response.listen`)
|
||||
"item_id": "item_9b2d…",
|
||||
"event": {
|
||||
"type": "input.committed",
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"turn_id": 1,
|
||||
"epoch": 0,
|
||||
"history_len": 1,
|
||||
@@ -802,7 +882,7 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"type": "input_audio_buffer.committed",
|
||||
"previous_item_id": "item_9b2d…",
|
||||
"item_id": "item_c044…",
|
||||
"event": {"type": "input.committed", "session_id": "sess_demo_01", "turn_id": 2, "epoch": 0, "history_len": 3, "response_create_deferred": true, "message": {"…": "…"}}
|
||||
"event": {"type": "input.committed", "session_id": "duplex-5f2a…", "turn_id": 2, "epoch": 0, "history_len": 3, "response_create_deferred": true, "message": {"…": "…"}}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -911,7 +991,7 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"status_details": null,
|
||||
"output": [],
|
||||
"modalities": ["audio", "text"],
|
||||
"metadata": {"duplex_event": {"type": "response.created", "session_id": "sess_demo_01", "response_id": "resp_01", "epoch": 0, "turn_id": 1}}
|
||||
"metadata": {"duplex_event": {"type": "response.created", "session_id": "duplex-5f2a…", "response_id": "resp_01", "epoch": 0, "turn_id": 1}}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -947,7 +1027,7 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"item_id": "item_resp_01",
|
||||
"output_index": 0,
|
||||
"content_index": 0,
|
||||
"metadata": {"session_id": "sess_demo_01", "epoch": 0, "model_speak": true}
|
||||
"metadata": {"session_id": "duplex-5f2a…", "epoch": 0, "model_speak": true}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -964,7 +1044,7 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"format": "pcm16",
|
||||
"sample_rate_hz": 24000,
|
||||
"metadata": {
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"epoch": 0,
|
||||
"model_speak": true,
|
||||
"end_of_turn": false,
|
||||
@@ -1029,7 +1109,7 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
"status": "completed",
|
||||
"status_details": null,
|
||||
"output": [{"id": "item_resp_01", "object": "realtime.item", "type": "message", "role": "assistant", "status": "completed", "content": [{"type": "output_audio", "transcript": "It is sunny, 24 degrees."}]}],
|
||||
"metadata": {"type": "response.done", "session_id": "sess_demo_01", "response_id": "resp_01", "epoch": 0, "committed": true, "playback": {"generated_ms": 2400, "sent_ms": 2400, "played_ms": 2400, "committed_ms": 2400}}
|
||||
"metadata": {"type": "response.done", "session_id": "duplex-5f2a…", "response_id": "resp_01", "epoch": 0, "committed": true, "playback": {"generated_ms": 2400, "sent_ms": 2400, "played_ms": 2400, "committed_ms": 2400}}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -1055,9 +1135,9 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
```json
|
||||
{
|
||||
"type": "response.listen",
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"epoch": 0,
|
||||
"response": {"object": "realtime.response", "status": "listening", "metadata": {"type": "response.listen", "session_id": "sess_demo_01", "epoch": 0, "reason": "model_listen"}}
|
||||
"response": {"object": "realtime.response", "status": "listening", "metadata": {"type": "response.listen", "session_id": "duplex-5f2a…", "epoch": 0, "reason": "model_listen"}}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -1104,7 +1184,7 @@ function-call response (Nemotron VoiceChat tools)
|
||||
raw fallback when a model tool call is malformed (`call_id`/`name` missing)
|
||||
|
||||
```json
|
||||
{"type": "duplex.function_call.done", "event": {"type": "function_call.done", "session_id": "sess_demo_01", "epoch": 0, "call_id": null, "name": null, "arguments": "{"}}
|
||||
{"type": "duplex.function_call.done", "event": {"type": "function_call.done", "session_id": "duplex-5f2a…", "epoch": 0, "call_id": null, "name": null, "arguments": "{"}}
|
||||
```
|
||||
|
||||
#### Playback
|
||||
@@ -1120,7 +1200,7 @@ raw fallback when a model tool call is malformed (`call_id`/`name` missing)
|
||||
"type": "playback.acknowledged",
|
||||
"event": {
|
||||
"type": "playback.acknowledged",
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"epoch": 0,
|
||||
"item_id": "item_resp_01",
|
||||
"played_ms": 1000,
|
||||
@@ -1145,20 +1225,17 @@ truncating ack after an interruption (what was actually heard)
|
||||
```json
|
||||
{
|
||||
"type": "overlap.decision",
|
||||
"session_id": "sess_demo_01",
|
||||
"session_id": "duplex-5f2a…",
|
||||
"epoch": 0,
|
||||
"policy": "listen_only",
|
||||
"action": "listen",
|
||||
"reason": "long_overlap_speech",
|
||||
"overlap_ms": 1400,
|
||||
"buffer_audio": true,
|
||||
"defer_runtime_append": true,
|
||||
"force_listen": true
|
||||
"metadata": {"overlap_ms": 1400, "buffer_audio": true, "defer_runtime_append": true, "force_listen": true}
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{"type": "overlap.decision", "session_id": "sess_demo_01", "epoch": 0, "policy": "barge_in_on_speech", "action": "barge_in", "reason": "server_vad_utterance_active"}
|
||||
{"type": "overlap.decision", "session_id": "duplex-5f2a…", "epoch": 0, "policy": "barge_in_on_speech", "action": "barge_in", "reason": "server_vad_utterance_active"}
|
||||
```
|
||||
|
||||
server-VAD session (`session.update` fragment) and the annotated append the
|
||||
@@ -1172,12 +1249,6 @@ translator forwards internally (shown for reference; clients never see it)
|
||||
{"type": "input_audio_buffer.append", "audio": "…", "format": "pcm16", "sample_rate_hz": 16000, "is_speech": true, "force_listen": true, "vad": {"backend": "silero", "is_speech": true, "speech_active": true, "speech_started": true, "speech_stopped": false, "speech_probability": 0.93}}
|
||||
```
|
||||
|
||||
`S→C runtime.control` (only when an engine signal was partially unsupported)
|
||||
|
||||
```json
|
||||
{"type": "runtime.control", "session_id": "sess_demo_01", "epoch": 1, "result": {"…": "redacted engine control result"}}
|
||||
```
|
||||
|
||||
#### Errors
|
||||
|
||||
`S→C error` — Realtime shape
|
||||
@@ -1203,71 +1274,79 @@ translator forwards internally (shown for reference; clients never see it)
|
||||
```
|
||||
|
||||
```json
|
||||
{"type": "error", "error": {"type": "rate_limit_error", "code": "duplex_session_busy", "message": "duplex session capacity reached (max_sessions=4)"}}
|
||||
{"type": "error", "error": {"type": "rate_limit_error", "code": "resource_exhausted", "message": "duplex_session_capacity_exhausted: limit=4"}}
|
||||
```
|
||||
|
||||
Handshake-stage errors (before the Realtime projector is bound) use the
|
||||
flat native shape:
|
||||
|
||||
```json
|
||||
{"type": "error", "error": "Unknown or expired duplex session: sess_demo_01", "code": "session_resume_expired"}
|
||||
{"type": "error", "error": "Unknown or expired duplex session: duplex-5f2a…", "code": "session_resume_expired"}
|
||||
```
|
||||
|
||||
#### Coverage
|
||||
|
||||
The examples above cover every event type the Realtime route emits or
|
||||
accepts (verified by diffing the `type` literals in
|
||||
`vllm_omni/entrypoints/duplex/` against this document). Not shown, because
|
||||
they never reach a `/v1/realtime?duplex=1` client unprojected: the native
|
||||
`/v1/duplex` dialect (`session.create` / `open_session` / `session.config`,
|
||||
`input.committed`, `input.cancelled`, `audio.cancelled`,
|
||||
`response.output_audio.delta`, `response.text.delta`, `response.message`,
|
||||
`function_call.done`) — their Realtime projections are the examples above
|
||||
(see the name map at the end of this section) — and the runner-internal markers `__timeout__`,
|
||||
`__disconnect__`, `__replaced_attachment__`. Input aliases (`push_chunk`,
|
||||
`input.audio.append`, `input_text.append`, `push_text`, `signal_turn`,
|
||||
`audio.playback_ack`, `close_session`, `close`, `session_close`) share the
|
||||
payload of their canonical event (see the alias table in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md)).
|
||||
`vllm_omni/engine/duplex/events.py` and `vllm_omni/engine/duplex/commands.py`
|
||||
against this document). Not shown, because they never reach a client
|
||||
unprojected: the session-internal events the runner produces
|
||||
(`input.committed`, `input.cancelled`, `audio.cancelled`,
|
||||
`response.output_audio.delta`, `response.text.delta`, `function_call.done`)
|
||||
— their Realtime projections are the examples above (see the name map at the
|
||||
end of this section). Every public event is a typed `DuplexEvent` whose
|
||||
`to_realtime()` is the wire object; every client event is parsed into a
|
||||
typed `DuplexCommand` by `command_from_realtime()`.
|
||||
|
||||
#### Native to Realtime name map
|
||||
#### Internal to Realtime name map
|
||||
|
||||
The native `/v1/duplex` dialect is what the session runner produces; the
|
||||
Realtime projector renames and fans out these events before they reach a
|
||||
`/v1/realtime?duplex=1` client.
|
||||
The session runner emits these session-internal events; the Realtime
|
||||
projector (`vllm_omni/engine/duplex/realtime_events.py`) renames and fans
|
||||
them out into typed events before they reach a client.
|
||||
|
||||
| Native / internal event | Realtime projection |
|
||||
| Internal event | Realtime projection |
|
||||
| --- | --- |
|
||||
| `session.create` (`open_session`, `session.config`) | first `session.update` |
|
||||
| `input.committed` | `conversation.item.added` / `conversation.item.created`, `input_audio_buffer.committed`, `conversation.item.input_audio_transcription.completed`, `conversation.item.done` |
|
||||
| `input.cancelled` | `input_audio_buffer.cleared` |
|
||||
| `response.output_audio.delta` (with `audio_transcript`) | optional `response.speak`, then `response.output_audio.delta` and `response.output_audio_transcript.delta` |
|
||||
| `response.output_audio.done` / `response.output_text.done` | `response.output_audio.done` / `response.output_text.done` |
|
||||
| `response.text.delta` | `response.output_text.delta` |
|
||||
| `response.message` (chat-fallback raw chunk) | passed through |
|
||||
| `audio.cancelled` | optional `output_audio_buffer.cleared`, then the cancelled terminal events and `response.done` with `status: "cancelled"` |
|
||||
| `function_call.done` | `response.created`, `response.output_item.added` (`function_call` item), `response.function_call_arguments.delta` / `.done`, `response.output_item.done`, `response.done` |
|
||||
| `playback.acknowledged` | `playback.acknowledged` (wrapped verbatim under `event`) |
|
||||
| `runtime.control`, `session.resumed`, `session.replaced`, `session.expired`, `session.resync_required`, `session.heartbeat_ack` | passed through unchanged |
|
||||
| `session.resumed`, `session.replaced`, `session.expired`, `session.resync_required`, `session.heartbeat_ack` | passed through unchanged |
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- Only MiniCPM-o 4.5 is served over this endpoint today; PersonaPlex and
|
||||
Nemotron VoiceChat arrive with the follow-up PRs that port them to the
|
||||
plugin contract.
|
||||
- Several surfaces are capability-gated per model (see *Capability
|
||||
negotiation by model* above): PersonaPlex does not support session resume,
|
||||
barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
|
||||
or audio truncation; camera frames are consumed only by MiniCPM-o 4.5; tool
|
||||
calls are produced only by Nemotron VoiceChat.
|
||||
- `turn_detection` supports `server_vad` or `null`, not `semantic_vad`.
|
||||
Turn-based models support endpointing without interruption, including
|
||||
`create_response=false` for explicit response creation. Model-native duplex
|
||||
requires `interrupt_response=true` and `create_response=true`.
|
||||
- A response is generated from committed **audio**, so the OpenAI text-prompt
|
||||
shape does not drive one: `conversation.item.create` with an `input_text`
|
||||
part adds the item to history, but a following `response.create` with no
|
||||
committed audio is rejected with `response_create_without_input`, and
|
||||
`input.text.append` is rejected with `native_text_append_unsupported`.
|
||||
Text-to-speech has its own shape here — put the text in
|
||||
`extra_body.duplex_initial_user_text` on `session.update`, then stream audio
|
||||
units (silence is enough). The seeded turn is what the model answers.
|
||||
- `turn_detection` supports only `server_vad` with `interrupt_response=true`;
|
||||
`semantic_vad`, `interrupt_response=false`, and `create_response` are not
|
||||
supported.
|
||||
- `input_audio_transcription` and `input_audio_noise_reduction` are accepted
|
||||
and echoed but no separate transcription or noise-reduction stage runs;
|
||||
transcripts come from the model.
|
||||
- `rate_limits.updated` is emitted for compatibility only and always carries
|
||||
an empty list.
|
||||
- Session capacity is bounded by `duplex_session.max_sessions` in the deploy
|
||||
configuration; admission beyond it fails with `duplex_session_busy` or
|
||||
`resource_exhausted`.
|
||||
- The native `/v1/duplex` route speaks the internal dialect and does not
|
||||
support `session.resume`; use `/v1/realtime?duplex=1` for the contract on
|
||||
this page.
|
||||
configuration; admission beyond it fails with `resource_exhausted`
|
||||
(`rate_limit_error`, retryable).
|
||||
- A duplex deployment serves the duplex session surfaces only: the websocket
|
||||
route and `/v1/chat/completions`, the latter on a session per request (see
|
||||
[Start the Server](#start-the-server)). The other turn-based HTTP routes
|
||||
report "not available" on a MiniCPM-o 4.5 server; turn-based use of the
|
||||
model stays available offline through `Omni` / `AsyncOmni`.
|
||||
@@ -6,6 +6,7 @@ vLLM-Omni provides an OpenAI-compatible API for text-to-speech (TTS) generation.
|
||||
- **Fish Speech S2 Pro** (`fishaudio/s2-pro`) -- Dual-AR TTS with DAC codec. Supports text-to-speech and voice cloning via reference audio. Output: 44.1 kHz.
|
||||
- **Voxtral TTS** (`mistralai/Voxtral-4B-TTS-2603`) -- AR + FlowMatching TTS with preset voices. Output: 24 kHz.
|
||||
- **CosyVoice3** (`FunAudioLLM/Fun-CosyVoice3-0.5B-2512`) -- 2-stage talker + flow-matching code2wav. Voice cloning via `ref_audio` + `ref_text` (no presets). Output: 24 kHz.
|
||||
- **Gepard-1.0** (`nineninesix/gepard-1.0`) -- single-stage native-AR TTS with a 22.05 kHz NanoCodec. Zero-shot default voice only.
|
||||
|
||||
See the [Supported Models](#supported-models) section below for the full list, including OmniVoice, VoxCPM2, MOSS-TTS-Nano, and Breeze-TTS-2.
|
||||
|
||||
@@ -40,11 +41,16 @@ vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091
|
||||
# CosyVoice3 (voice cloning only — supply ref_audio + ref_text per request)
|
||||
vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
|
||||
--omni --port 8091 --trust-remote-code
|
||||
|
||||
# Gepard-1.0 (zero-shot default voice; packaged gepard.yaml)
|
||||
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
|
||||
--stage-init-timeout 900 \
|
||||
--deploy-config vllm_omni/deploy/gepard.yaml
|
||||
```
|
||||
|
||||
### Generate Speech
|
||||
|
||||
**Using curl:**
|
||||
**Qwen3-TTS CustomVoice, using curl:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
@@ -56,7 +62,24 @@ curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
**Using Python:**
|
||||
**Gepard-1.0, using curl:**
|
||||
|
||||
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`,
|
||||
`stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields
|
||||
include `speed`, `language`, `instructions`, `task_type`, `ref_audio`,
|
||||
`ref_text`, `extra_params`, and `word_timestamps`.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is Gepard speaking.",
|
||||
"voice": "default",
|
||||
"seed": 7
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
**Qwen3-TTS CustomVoice, using Python:**
|
||||
|
||||
```python
|
||||
import httpx
|
||||
@@ -761,6 +784,28 @@ Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` ne
|
||||
| ------- | ------------- |
|
||||
| `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | Voice cloning from `ref_audio` + `ref_text`. No built-in voice presets — upload a voice or pass `ref_audio`/`ref_text` per request. |
|
||||
|
||||
### Gepard-1.0
|
||||
|
||||
| Model | Description |
|
||||
| ----- | ----------- |
|
||||
| `nineninesix/gepard-1.0` | Zero-shot native-AR TTS. 22.05 kHz mono. `voice` must be omitted or `"default"`. |
|
||||
|
||||
Gepard request fields:
|
||||
|
||||
| Field | Behavior |
|
||||
| ----- | -------- |
|
||||
| `input` | Required. Empty/whitespace-only returns 400. |
|
||||
| `voice` | Omitted or `"default"` only. Other values 400. |
|
||||
| `response_format` | `wav` default. Non-streaming: `wav`/`pcm`/`flac`/`mp3`. `opus` returns 400 (22.05 kHz is not an Opus sample rate). Streaming: `pcm`/`wav` only. |
|
||||
| `stream` / `stream_format` | SSE (`speech.audio.*`) and raw `audio` byte streaming. |
|
||||
| `speed` | Must be `1.0`. Gepard has no native speed control. |
|
||||
| `max_new_tokens` | Frame budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms). Default 1000 from deploy YAML; adapter bounds 1..4096. |
|
||||
| `seed` | Optional. Reaches the in-model 32-head sampler. The packaged YAML currently pins `seed: 42`, so serving is deterministic by default until that pin is removed. |
|
||||
| `extra_params` | Any key returns 400, including `temperature`/`top_p`/`top_k`. |
|
||||
| Cloning / style fields | `ref_audio`, `ref_text`, `speaker_embedding`, `instructions`, `language`, `task_type`, `word_timestamps`, and similar declared-but-unsupported fields return 400. |
|
||||
|
||||
See the [Gepard section of the online TTS hub](../user_guide/examples/online_serving/text_to_speech.md#gepard-10) for launch commands. Native-AR recompute preemption is a known limitation under concurrency.
|
||||
|
||||
### OmniVoice
|
||||
|
||||
| Model | Description |
|
||||
|
||||
@@ -50,6 +50,14 @@ WebSocket /v1/video/chat/stream
|
||||
| Server -> Client | `session.done` | none | Session closed. |
|
||||
| Server -> Client | `error` | `message` | Recoverable protocol or generation error. |
|
||||
|
||||
### Frame Consumption Reporting
|
||||
|
||||
When buffered frames include client-supplied `frame_id` values, the server emits `video.frames.consumed` after the first engine output. Its `frame_ids`, `frames`, and `latest_pts_ms` describe the images included in that query's prompt.
|
||||
|
||||
Queries stride-sample the buffered frames, keeping the last frame, then exclude frames already known to have failed decoding. Excluded frames are not replaced with other buffered frames. Frames whose background decoding has not finished remain eligible through the image URL path.
|
||||
|
||||
**Bugfix compatibility note:** `video.frames.consumed` now excludes known decode failures that were already excluded from the prompt. Older versions could report those frames and their timestamps as consumed. The event name and fields are unchanged; an empty selection reports empty lists and `latest_pts_ms: null`.
|
||||
|
||||
### `session.config` Fields
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|
||||
@@ -109,6 +109,16 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
|
||||
The final content is available from `/v1/videos/{video_id}/content` after the
|
||||
job status becomes `completed`.
|
||||
|
||||
`queued` means the request is still waiting for diffusion scheduler admission.
|
||||
The status changes to `in_progress` when the scheduler first selects the
|
||||
request for execution.
|
||||
|
||||
`DELETE /v1/videos/{video_id}` issues a bounded engine abort
|
||||
(`VLLM_OMNI_ABORT_TIMEOUT`, default 2s), then cancels the frontend
|
||||
task. Cancellation cleanup is also bounded and best-effort: it confirms
|
||||
the abort was queued, and the current request batch may still drain.
|
||||
The job is then re-read so a completed save is not orphaned.
|
||||
|
||||
### Synchronous Response
|
||||
|
||||
`POST /v1/videos/sync` blocks until generation finishes and returns raw video
|
||||
|
||||
@@ -1,24 +1,33 @@
|
||||
# FastVideo VSA
|
||||
|
||||
FastVideo Variable Sparse Attention (VSA) accelerates the `FastVideo/FastWan2.2-TI2V-5B-Diffusers` model's
|
||||
self-attention by partitioning the post-patch latent grid into spatiotemporal
|
||||
FastVideo Variable Sparse Attention (VSA) partitions the post-patch latent grid into spatiotemporal
|
||||
blocks. For every query block, VSA scores the key/value blocks and computes
|
||||
attention only against the selected top-k blocks.
|
||||
|
||||
The supported checkpoint provides both text-to-video and image-to-video modes through `Wan22Pipeline`; the separate Wan I2V-14B, S2V, and VACE pipelines are outside this backend's supported scope.
|
||||
## Supported models
|
||||
|
||||
MiniMax-H3 served with a FastH3 VSA adapter uses a second, H3-specific route
|
||||
through the same backend. See the
|
||||
[MiniMax-H3 recipe](https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3) for its geometry,
|
||||
supported topologies, and commands.
|
||||
| Model / checkpoint | Required adapter | Tasks | Sequence parallelism |
|
||||
| --- | --- | --- | --- |
|
||||
| `FastVideo/FastWan2.2-TI2V-5B-Diffusers` | None | T2V, I2V through `Wan22Pipeline` | Disabled |
|
||||
| `MiniMaxAI/MiniMax-H3` | [FastH3 VSA (recipe)](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter) | T2VA | Disabled or pure Ulysses |
|
||||
| Wan I2V-14B, S2V, VACE | — | Unsupported | — |
|
||||
|
||||
VSA is a CUDA-only, explicitly selected backend. It requires the
|
||||
`fastvideo-kernel` package and currently supports non-causal self-attention
|
||||
with equal query and key/value sequence lengths. Unsupported shapes, masks,
|
||||
dtypes, sequence-parallel execution, or kernel failures fall back to
|
||||
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
|
||||
sequence-parallel context is one of those fallbacks; the H3 route supports pure
|
||||
Ulysses and rejects ring or all-gather sequence parallelism at startup.
|
||||
## Installation
|
||||
|
||||
Install vLLM-Omni from this checkout with the `vsa` extra:
|
||||
|
||||
```bash
|
||||
uv pip install -e '.[vsa]'
|
||||
```
|
||||
|
||||
The extra installs the tested kernel dependency automatically. Prebuilt kernels require
|
||||
Linux, Python 3.12, and glibc 2.34 or newer (x86-64 or aarch64). The full
|
||||
FastVideo framework and provider environment variables are not required.
|
||||
|
||||
`FASTVIDEO_VSA` selects the attention algorithm. On SM120, the H3 integration
|
||||
currently executes FastVideo's 64-token Triton block-sparse kernel; it does not
|
||||
dispatch to FlashInfer. The `vsa` extra installs `fastvideo-kernel==0.3.4` for
|
||||
this execution path.
|
||||
|
||||
## Enable the backend
|
||||
|
||||
@@ -27,24 +36,18 @@ flag. Use `--fastvideo-vsa-topk` to set the number of key/value blocks retained
|
||||
for every query block:
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> \
|
||||
vllm serve <model> --omni \
|
||||
--diffusion-attention-backend FASTVIDEO_VSA \
|
||||
--fastvideo-vsa-topk 64
|
||||
```
|
||||
|
||||
The backwards-compatible environment variable selects the backend with the
|
||||
default `topk=64`:
|
||||
<details markdown="1">
|
||||
<summary>Alternative configuration formats</summary>
|
||||
|
||||
Equivalent structured configuration:
|
||||
|
||||
```bash
|
||||
export DIFFUSION_ATTENTION_BACKEND=FASTVIDEO_VSA
|
||||
vllm-omni serve <model>
|
||||
```
|
||||
|
||||
To tune top-k, pass the CLI backend and top-k flags together as shown above,
|
||||
or use the equivalent structured configuration:
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> \
|
||||
vllm serve <model> --omni \
|
||||
--diffusion-attention-config \
|
||||
'{"default":{"backend":"FASTVIDEO_VSA","fastvideo_vsa_topk":64}}'
|
||||
```
|
||||
@@ -73,8 +76,15 @@ stages:
|
||||
fastvideo_vsa_topk: 64
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Choose top-k
|
||||
|
||||
H3 uses 64-token video blocks and keeps
|
||||
all prefix blocks; see the [FastH3 VSA recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
|
||||
|
||||
### Wan top-k behavior
|
||||
|
||||
At runtime the backend logs the sequence shape and derived block count:
|
||||
|
||||
```text
|
||||
@@ -117,8 +127,16 @@ selecting VSA does not turn a native checkpoint into a distilled model.
|
||||
|
||||
## Verify routing and fallback
|
||||
|
||||
Check the startup and first-forward logs instead of assuming that selecting
|
||||
the backend guarantees sparse execution:
|
||||
VSA is a CUDA-only, explicitly selected backend. It requires the
|
||||
`fastvideo-kernel` package and currently supports non-causal self-attention
|
||||
with equal query and key/value sequence lengths. Unsupported shapes, masks,
|
||||
dtypes, sequence-parallel execution, or kernel failures fall back to
|
||||
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
|
||||
sequence-parallel context is one of those fallbacks; the H3 route supports pure
|
||||
Ulysses and rejects ring or all-gather sequence parallelism at startup.
|
||||
H3 accelerator faults propagate instead of attempting dense recovery.
|
||||
|
||||
On the Wan route, check the startup and first-forward logs:
|
||||
|
||||
- `route=VSA` means top-k block selection is active.
|
||||
- `route=VSA_ALL_BLOCKS` means the FastVideo DMD checkpoint retained all
|
||||
@@ -126,6 +144,10 @@ the backend guarantees sparse execution:
|
||||
- `route=SDPA` or `FASTVIDEO_VSA falling back to SDPA: ...` means dense SDPA
|
||||
executed; the warning includes the reason.
|
||||
|
||||
For H3, check `FastH3 adapter active` at startup and
|
||||
`FASTVIDEO_VSA H3 routing` during DiT execution, as described in the
|
||||
[model recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
|
||||
|
||||
The Wan route requires CUDA tensors in FP16 or BF16, 256-token blocks,
|
||||
standard `head_size**-0.5` scaling, equal Q/K/V head counts, no attention mask,
|
||||
and no active sequence-parallel context. The MiniMax-H3 route uses 64-token
|
||||
|
||||
@@ -18,6 +18,7 @@ For the full list of supported architectures across all modalities, see
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | — |
|
||||
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
|
||||
| Gepard-1.0 | `nineninesix/gepard-1.0` | — (zero-shot default voice) | ✓ (PCM / WAV stream) | `default` only | — |
|
||||
| higgs-audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✓ (`ref_audio`+`ref_text`) | ✓ (codec_streaming) | — | — |
|
||||
| GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
|
||||
| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
|
||||
@@ -92,7 +93,7 @@ curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
|
||||
```
|
||||
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others).
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for Gepard, 24 kHz for the others).
|
||||
|
||||
For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md).
|
||||
|
||||
@@ -212,6 +213,61 @@ python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server
|
||||
|
||||
---
|
||||
|
||||
## Gepard-1.0
|
||||
|
||||
Single-stage native AR TTS at 22.05 kHz mono. Zero-shot only: omit `voice` or pass `"default"`. Voice cloning from reference audio is not available yet.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Same NeMo NanoCodec install as the [offline Gepard section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#gepard-10). On a host whose CUDA toolkit cannot build kernels — no `nvcc`/`ninja`, or a consumer Blackwell (`sm_120`) card — also `export VLLM_USE_FLASHINFER_SAMPLER=0` before launch. That env is not a deploy-YAML field.
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
|
||||
--stage-init-timeout 900 \
|
||||
--deploy-config vllm_omni/deploy/gepard.yaml
|
||||
# or:
|
||||
./gepard/run_server.sh
|
||||
```
|
||||
|
||||
`--stage-init-timeout 900` matches the online e2e fixture and the offline example. Serve defaults to 300s, which is often too short for a cold download of the talker plus NanoCodec.
|
||||
|
||||
The packaged `vllm_omni/deploy/gepard.yaml` must be passed with `--deploy-config`. The checkpoint self-identifies as `qwen3_5_text`, so omitting the YAML launches a diffusion fallback instead of the Gepard pipeline. The YAML sets `async_chunk: false`, `max_num_seqs: 4`, and currently pins `seed: 42`, so serving is deterministic by default until that YAML seed is removed. Pass an explicit per-request `seed` in tests and clients rather than depending on either default.
|
||||
|
||||
### Sending requests
|
||||
|
||||
```bash
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking."
|
||||
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking." --seed 7 --stream --output output.pcm
|
||||
```
|
||||
|
||||
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`, `stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields include `speed`, `language`, `instructions`, `task_type`, `ref_audio`, `ref_text`, `extra_params`, and `word_timestamps`.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is Gepard speaking.",
|
||||
"voice": "default",
|
||||
"seed": 7
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 22.05 kHz mono. `max_new_tokens` is a **frame** budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms at 21.5 fps; adapter bounds 1..4096).
|
||||
- Supported request fields: `input` (required), `voice` (`default` only), `response_format` (`wav` default; `wav/pcm/flac/mp3` non-streaming; `opus` 400 because 22.05 kHz is not an Opus sample rate; streaming `pcm/wav` only), `stream` / `stream_format`, `max_new_tokens`, `seed`.
|
||||
- Unsupported: `speed`, `extra_params` (including `temperature`/`top_p`/`top_k`), `ref_audio`, `ref_text`, `speaker_embedding`, `task_type`, `instructions`, `language`, and `word_timestamps`.
|
||||
- Concurrent requests at `max_num_seqs: 4` are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below `max_num_seqs` and treat preemption as out of scope until the platform fix lands.
|
||||
- Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.
|
||||
- For the request-field table, see the [Speech API Gepard section](https://github.com/vllm-project/vllm-omni/blob/main/docs/serving/speech_api.md#gepard-10).
|
||||
|
||||
---
|
||||
|
||||
## higgs-audio v2
|
||||
|
||||
2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) that streams chunks back to the client.
|
||||
|
||||
Reference in new issue
Block a user