Merge main and preserve dispatch backlog across monitor windows
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
Sy03 committed
commit
ea7de7e3de
769 files changed
+73802
-22805
No files matched your search
@@ -22,14 +22,14 @@ capability gates, see the [Realtime Duplex API guide](realtime_duplex_api.md).
|
||||
## Enable Full Duplex
|
||||
|
||||
A model is served full duplex when its registered pipeline declares a
|
||||
`duplex_plugin` (the model's `DuplexModelPlugin`). That declaration alone
|
||||
decides it: `vllm-omni serve` constructs `DuplexOmni` instead of `AsyncOmni`,
|
||||
and every surface the server exposes is backed by a duplex session. It serves
|
||||
`duplex_plugin` (the model's `DuplexModelPlugin`) and its deploy configuration
|
||||
sets `session_mode: duplex`. `vllm serve --omni` constructs `DuplexOmni`
|
||||
instead of `AsyncOmni`. It serves
|
||||
`/v1/realtime?duplex=1` (and its alias `/v1/duplex`),
|
||||
`POST /v1/chat/completions`, `/v1/models` and `/health`; every other
|
||||
turn-based HTTP route (speech, batch, embeddings, video, ...) answers "not
|
||||
available". Turn-based use of such a model stays available offline through the
|
||||
Python API (`Omni` / `AsyncOmni`).
|
||||
available". Set `session_mode: turn` to select the ordinary online serving
|
||||
stack instead. The model's supported tasks and endpoint restrictions still apply.
|
||||
|
||||
`/v1/chat/completions` is the ordinary chat service running on the duplex
|
||||
engine: a request is a turn-based generation on the same stages, served
|
||||
@@ -45,8 +45,24 @@ The deploy configuration of such a model must agree:
|
||||
session_mode: duplex
|
||||
```
|
||||
|
||||
A duplex model started with a deploy configuration that does not set it fails
|
||||
at startup rather than falling back to turn-based serving.
|
||||
A duplex-capable model must explicitly set `session_mode` to `duplex` or `turn`
|
||||
in its deploy configuration, possibly through `base_config` inheritance.
|
||||
Missing or invalid values fail at startup. The default MiniCPM-o deployment
|
||||
continues to use duplex mode.
|
||||
|
||||
To run MiniCPM-o 4.5 with the turn-based engine:
|
||||
|
||||
```bash
|
||||
vllm serve openbmb/MiniCPM-o-4_5 --omni \
|
||||
--deploy-config vllm_omni/deploy/minicpmo_4_5_turn.yaml \
|
||||
--trust-remote-code \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
This profile inherits the default model and stage settings and overrides only
|
||||
`session_mode`. It supports ordinary HTTP requests without creating a duplex
|
||||
session handler. Mode selection happens at startup; a WebSocket query parameter
|
||||
does not switch engines. The Python `Omni` / `AsyncOmni` APIs are unchanged.
|
||||
|
||||
!!! warning
|
||||
|
||||
|
||||
@@ -8,12 +8,41 @@ history stays honest, and a dropped connection can resume the same session.
|
||||
This page covers how to run a duplex deployment, how to drive it from Python
|
||||
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
|
||||
|
||||
The endpoint is served for models that ship a duplex plugin — currently
|
||||
MiniCPM-o 4.5 only — and just for deploy configurations that declare
|
||||
The endpoint is served for models that ship a duplex plugin — including
|
||||
MiniCPM-o 4.5 and Qwen3-Omni — and just for deploy configurations that declare
|
||||
`session_mode: duplex`. PersonaPlex and Nemotron VoiceChat are ported to the
|
||||
plugin contract in follow-up PRs and are not served over this endpoint yet. The runtime architecture
|
||||
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
|
||||
## Qwen3-Omni conversation history
|
||||
|
||||
Qwen3-Omni keeps incoming conversation items independently addressable by item
|
||||
ID. Before applying the model's chat template, it merges consecutive user
|
||||
items into one multimodal user message. An assistant message, including an
|
||||
empty message reserved for an interrupted or unheard answer, separates turns.
|
||||
The final user message places media before text instructions, following the
|
||||
[official Qwen3-Omni demo's `format_history()`](https://github.com/QwenLM/Qwen3-Omni/blob/e4235853125589c789f06a2dd83e9f4126df5e9d/web_demo.py#L84-L179).
|
||||
|
||||
A camera image or text item may arrive after an audio commit and before
|
||||
`response.create`. Those items belong to the same pending user turn: the
|
||||
committed audio remains associated with its history item even when that item
|
||||
is no longer last. Prompt preparation snapshots the selected inputs; later
|
||||
items do not modify an already prepared request.
|
||||
|
||||
The duplex adapter retains its existing budgets: at most four audio payloads
|
||||
(including the current input), an 8 MiB encoded-audio history budget (keeping
|
||||
at least the newest payload), and at most eight prompt images. The recent
|
||||
16-message window is rounded outwards to a complete turn. If any audio in an
|
||||
older turn is no longer retained, the entire turn's model context, including
|
||||
its images/text and assistant replies, is omitted. Images from that turn are
|
||||
therefore **not** permanent session-wide model context. These limits differ
|
||||
from the demo's one image-bearing turn and five audio-bearing turns; the
|
||||
shared rule is grouping before pruning, rather than copying its UI budgets.
|
||||
|
||||
Grouping and pruning affect only the model-input view. They do not merge or
|
||||
delete the source conversation items exposed to clients, and do not change
|
||||
playback acknowledgement or interruption handling.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Start the Server
|
||||
@@ -32,33 +61,19 @@ vllm-omni serve openbmb/MiniCPM-o-4_5 \
|
||||
mounts `ws://<host>:8099/v1/realtime?duplex=1` (this page; `ws://<host>:8099/v1/duplex`
|
||||
is an alias of the same route), `POST /v1/chat/completions`, `/v1/models` and
|
||||
`/health`, and nothing else. The remaining turn-based HTTP routes (speech,
|
||||
batch, embeddings, video, ...) are not served by a duplex model.
|
||||
batch, embeddings, video, ...) are not served in duplex mode.
|
||||
|
||||
`/v1/chat/completions` is not a turn-based path on a duplex server: each
|
||||
request opens a short-lived duplex session and reads the answer off it. Audio
|
||||
content becomes an ordinary committed turn; text is seeded as the session's
|
||||
opening turn, because a model-native model takes a turn only when it hears
|
||||
speech. A model that cannot be seeded that way declares so, and a text request
|
||||
to it is refused with 400. Two consequences are worth sizing for:
|
||||
`/v1/chat/completions` uses the ordinary chat service on the same engine
|
||||
when the plugin declares `supports_chat_completions`. HTTP chat requests do
|
||||
not create duplex sessions or consume session admission slots. Their options
|
||||
follow the [Chat Completions API](chat_completions_api.md) and the model's
|
||||
capabilities and endpoint restrictions.
|
||||
|
||||
- **Every request holds an admission slot** for its lifetime, so
|
||||
`duplex_session.max_sessions` caps HTTP concurrency as well as websocket
|
||||
sessions. Beyond it a request is refused with HTTP 503.
|
||||
- **Latency follows the model's own clock.** A model-native session generates
|
||||
per audio unit, so even a short answer costs what that many units cost --
|
||||
noticeably more than the same model would take turn-based.
|
||||
|
||||
`n > 1`, `logprobs`, `tools`, and image or video content parts have no
|
||||
representation in one duplex turn and are refused with HTTP 400 -- a Realtime
|
||||
conversation item carries text and audio only. `temperature`,
|
||||
`max_completion_tokens`, `modalities` and
|
||||
`chat_template_kwargs.use_tts_template` apply; the other
|
||||
`chat_template_kwargs` keys are logged as ignored. `extra_body` is passed to
|
||||
the session unchanged, exactly as a websocket client's session `extra_body`
|
||||
is, which is how a request asking for `modalities: ["text", "audio"]` supplies
|
||||
the `ref_audio` MiniCPM-o requires for audio output. A model that should not
|
||||
serve the route lists it in the deploy config's `endpoint_restrictions`,
|
||||
exactly as a turn-based model does.
|
||||
For a turn-based online deployment, use `minicpmo_4_5_turn.yaml` or explicitly
|
||||
set `session_mode: turn` in a deploy configuration derived from the model's
|
||||
profile. This selects `AsyncOmni` and the ordinary API initialization instead
|
||||
of the duplex handler. See [Full Duplex](full_duplex_api.md#enable-full-duplex)
|
||||
for the command and configuration requirements.
|
||||
|
||||
### Run the Example Client
|
||||
|
||||
@@ -782,7 +797,10 @@ events with `server_event_seq > 40` are replayed in order)
|
||||
```
|
||||
|
||||
with camera frames (omni video; rides the append that closes a 1 s model
|
||||
unit; each entry is a **bare** base64 JPEG/PNG — no `data:` URL prefix)
|
||||
unit; each entry is a **bare** base64 JPEG/PNG — no `data:` URL prefix).
|
||||
Only a model whose Stage 0 interleaves a frame track at unit boundaries takes
|
||||
this field; a turn model has no such boundary and rejects it, expecting
|
||||
`conversation.item.create` with `input_image` content instead.
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -959,6 +977,10 @@ deferred commit during an active response (`event.response_create_deferred`)
|
||||
{"type": "conversation.item.truncated", "item_id": "item_resp_01", "content_index": 0, "audio_end_ms": 1850, "event": {"…": "…"}}
|
||||
```
|
||||
|
||||
After this acknowledgement, `conversation.item.retrieve` returns the truncated
|
||||
transcript. Without text/audio alignment marks, its prefix is estimated from
|
||||
the requested position and audio duration; this is not exact word alignment.
|
||||
|
||||
#### Response lifecycle
|
||||
|
||||
`C→S response.create`
|
||||
@@ -1345,8 +1367,9 @@ them out into typed events before they reach a client.
|
||||
- Session capacity is bounded by `duplex_session.max_sessions` in the deploy
|
||||
configuration; admission beyond it fails with `resource_exhausted`
|
||||
(`rate_limit_error`, retryable).
|
||||
- A duplex deployment serves the duplex session surfaces only: the websocket
|
||||
route and `/v1/chat/completions`, the latter on a session per request (see
|
||||
[Start the Server](#start-the-server)). The other turn-based HTTP routes
|
||||
report "not available" on a MiniCPM-o 4.5 server; turn-based use of the
|
||||
model stays available offline through `Omni` / `AsyncOmni`.
|
||||
- A duplex deployment serves the websocket route and, when supported by the
|
||||
model, ordinary `/v1/chat/completions` requests on the same engine without a
|
||||
session per request (see [Start the Server](#start-the-server)). Other
|
||||
turn-based HTTP routes report "not available" in duplex mode. To use the
|
||||
ordinary online serving stack, select `session_mode: turn` as described in
|
||||
[Full Duplex](full_duplex_api.md#enable-full-duplex).
|
||||
@@ -521,6 +521,15 @@ are cached in-process with a shared LRU so repeated requests with the same
|
||||
all TTS model types; deleting a voice invalidates every model-type slot at
|
||||
once.
|
||||
|
||||
Decoded reference waveforms use a separate LRU cache of owned, contiguous
|
||||
float32 arrays at the source sampling rate. Its byte budget counts numeric
|
||||
buffers (four bytes per mono sample), excluding cache metadata. Numeric storage
|
||||
avoids retaining a Python float object for every sample during garbage
|
||||
collection. MOSS reference encoding consumes these arrays directly; other
|
||||
list-based interfaces, including MOSS Nano, receive temporary lists that are
|
||||
not retained in the resolve cache. The MOSS reference encoder also releases
|
||||
completed batch inputs before waiting for more work.
|
||||
|
||||
### Precomputed Custom Voices
|
||||
|
||||
Qwen3-TTS Base and VoxCPM2 can load offline-precomputed voices at startup.
|
||||
|
||||
@@ -41,7 +41,7 @@ bytes.
|
||||
| Server to client | `video.start` | Confirms the request ID, format, and accepted configuration |
|
||||
| Server to client | `video.chunk_metadata` | Describes the next binary media frame |
|
||||
| Server to client | Binary frame | Carries one fragmented MP4 chunk |
|
||||
| Client to server | `session.interaction` | Queues an optional prompt update during generation |
|
||||
| Client to server | `session.interaction` | Queues an optional interaction (e.g., prompt. camera action) during generation |
|
||||
| Server to client | `session.interaction.queued` | Confirms that an interaction was queued |
|
||||
| Client to server | `session.ping` | Refreshes the stall clock |
|
||||
| Server to client | `session.pong` | Acknowledges a ping |
|
||||
@@ -83,10 +83,39 @@ Models that implement interaction updates can change the active prompt:
|
||||
}
|
||||
```
|
||||
|
||||
Models that register the camera modality (for example LingBot-World) also
|
||||
accept structural SE3 payloads on `event.multi_modal_data.camera`. Coordinates
|
||||
follow Unity semantics (`+X` right, `+Y` up, `+Z` forward; same as LingBot's
|
||||
native camera frame). WASD key tokens are not admitted by the engine:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "session.interaction",
|
||||
"interaction": {
|
||||
"event_id": "forward-1",
|
||||
"event": {
|
||||
"multi_modal_data": {
|
||||
"camera": {
|
||||
"mode": "velocity",
|
||||
"data": {
|
||||
"translation": [0.0, 0.0, 0.05],
|
||||
"rotation": [0.0, 0.0, 0.0, 1.0]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`mode` is `target` (lerp to an absolute pose over `transition_chunks`) or
|
||||
`velocity` (hold a per-frame SE3 delta until replaced). Prompt and camera may
|
||||
be combined in one interaction event.
|
||||
|
||||
This feature is pipeline-dependent. An enabled endpoint does not imply that
|
||||
the loaded model supports midway prompt updates.
|
||||
the loaded model supports midway prompt or camera updates.
|
||||
|
||||
The default start timeout is 10 seconds and the server reports a stall after
|
||||
about 60 seconds without generation progress or a `session.ping`. See the
|
||||
[streaming video generation example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/streaming_video_generation)
|
||||
for prompt-update scheduling and browser playback.
|
||||
for prompt/camera-update scheduling and browser playback.
|
||||
@@ -73,6 +73,10 @@ curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o output.mp4
|
||||
| `image_reference` | string | null | JSON-encoded reference image payload; do not combine with `input_reference` or `video_reference` |
|
||||
| `video_reference` | string | null | JSON-encoded reference video payload; do not combine with `input_reference` or `image_reference` |
|
||||
| `audio_reference` | string | null | JSON-encoded audio reference for speech-to-video: `{"audio_url": "..."}` — supports HTTP(s) URLs or base64 data URLs |
|
||||
| `source_video` | file | null | MiniMax H3 latent-edit source video (`.mp4` or `.mov`, up to 512 MiB) |
|
||||
| `source_audio` | file | null | Optional MiniMax H3 latent-edit source audio (`.wav` or `.mp3`, up to 512 MiB) |
|
||||
| `video_noise_mask` | file | null | UTF-8 JSON MiniMax H3 video mask; `0` preserves and `1` regenerates a token |
|
||||
| `audio_noise_mask` | file | null | UTF-8 JSON MiniMax H3 audio mask; `0` preserves and `1` regenerates a token |
|
||||
| `width` | integer | model default | Output video width |
|
||||
| `height` | integer | model default | Output video height |
|
||||
| `num_frames` | integer | 1 | Number of generated frames |
|
||||
@@ -214,6 +218,49 @@ upstream vLLM's media URL allowlist protection. Deployments should therefore
|
||||
treat remote media URLs as untrusted and choose this setting as part of their
|
||||
URL access policy.
|
||||
|
||||
### MiniMax H3 Latent-Mask Editing
|
||||
|
||||
MiniMax H3 accepts request-scoped source media and video/audio noise masks.
|
||||
At least one mask is required. A nontrivial `video_noise_mask` requires
|
||||
`source_video`, while a nontrivial `audio_noise_mask` requires either
|
||||
`source_audio` or a `source_video` with an audio stream. Mask values are in
|
||||
`[0, 1]`: `0` preserves the source, `1` regenerates it, and fractional values
|
||||
blend the two behaviors. Exact all-one masks are no-ops and do not require a
|
||||
source. Source uploads without a mask are rejected. Masks may be a JSON scalar
|
||||
or arrays matching the H3 latent/token grid; pixel-resolution masks must be
|
||||
resized or pooled by the client before upload.
|
||||
|
||||
For an aligned output of `F` frames at `W x H`, the video latent grid is
|
||||
`[Tv, H/16, W/16]`, where `Tv = 2 + 5 * ((F - 5) / 17)`. The video mask may be
|
||||
a scalar, a flat token vector, `[Tv, H/32, W/32]`, or the full latent grid. For
|
||||
the model input, timestep, and velocity, a full-grid mask is max-pooled over
|
||||
each 2x2 spatial token and fractional values are rounded upward to 1/256
|
||||
levels. The final x0 restore uses the original, unquantized mask, so full-grid
|
||||
masks retain cell-level preservation inside a model token. The audio length is
|
||||
`Ta = round(F * 40 / 24)`, and its mask may be a scalar, `[Ta]`, `[2, Ta]`, or
|
||||
a flat `2 * Ta` vector. If source audio is short, its missing latent tail is
|
||||
forced to mask value `1` so H3 generates that portion. Each mask must be sent
|
||||
as a UTF-8 JSON file part and is limited to 8 MiB. A file may contain either a
|
||||
JSON scalar or an array.
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8091/v1/videos/sync \
|
||||
-F "prompt=Partially restyle the complete clip and soundtrack" \
|
||||
-F 'extra_params={"task":"t2va","duration":4.0,"aspect_ratio":"16:9"}' \
|
||||
-F "source_video=@source.mp4;type=video/mp4" \
|
||||
-F "source_audio=@source.wav;type=audio/wav" \
|
||||
-F "video_noise_mask=@video-mask.json;type=application/json" \
|
||||
-F "audio_noise_mask=@audio-mask.json;type=application/json" \
|
||||
-o edited.mp4
|
||||
```
|
||||
|
||||
For example, each mask file in the request above may contain the JSON scalar
|
||||
`0.5`.
|
||||
|
||||
The server streams source files to temporary request-scoped storage and removes
|
||||
them after synchronous or asynchronous generation finishes. Models that do not
|
||||
declare latent-mask editing support reject these fields.
|
||||
|
||||
### Speech-to-Video
|
||||
|
||||
For models that support audio-driven generation (e.g., Wan2.2-S2V), pass both
|
||||
@@ -251,6 +298,7 @@ These `extra_params` control how the server turns decoded frames into MP4 bytes.
|
||||
| Field | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `preencode_mp4` | boolean | false | Encode the MP4 on the worker while the VAE is still decoding, instead of after the full video is materialized |
|
||||
| `preencode_batch_frames` | positive integer | 17 (H3, Wan T2V/I2V); 1 (Wan S2V) | Minimum accumulated frames per worker transfer/encoding batch; used only with `preencode_mp4=true` |
|
||||
| `video_codec_options` | object | null | Encoder options passed through to the H.264 encoder, such as `{"preset": "ultrafast", "threads": "0"}` |
|
||||
|
||||
With `preencode_mp4` enabled, each committed VAE chunk leaves the accelerator and
|
||||
@@ -261,16 +309,29 @@ unchanged: the same complete MP4, byte-for-byte equivalent frames.
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/videos/sync \
|
||||
-F "prompt=A small robot walking through a neon city" \
|
||||
-F 'extra_params={"preencode_mp4": true, "video_codec_options": {"preset": "ultrafast"}}' \
|
||||
-F 'extra_params={"preencode_mp4": true, "preencode_batch_frames": 33, "video_codec_options": {"preset": "ultrafast"}}' \
|
||||
-o output.mp4
|
||||
```
|
||||
|
||||
Set `preencode_batch_frames` in `extra_params` (or `extra_args` for offline
|
||||
sampling) to tune batching. The worker accumulates complete VAE chunks until
|
||||
it has at least this many frames, then transfers and encodes them together.
|
||||
It always flushes the final partial batch. This is a threshold, not an exact
|
||||
chunk length: a value of 1 submits every native chunk immediately, and a value
|
||||
smaller than a native chunk does not split it. Larger values reduce transfers
|
||||
but retain more frames on the accelerator and delay encoding. The VAE decode
|
||||
window and output frame count stay unchanged. Wan S2V keeps its existing
|
||||
per-clip behavior by default. Zero, negative, fractional, boolean, string, and
|
||||
null values are rejected when pre-encoding is enabled.
|
||||
|
||||
`preencode_mp4` applies to the complete-MP4 response paths only. The
|
||||
`/v1/realtime/video` WebSocket endpoint rejects it, because that path already
|
||||
overlaps encoding through its own incremental fragmented-MP4 encoder.
|
||||
overlaps encoding through its own incremental fragmented-MP4 encoder. Wan also
|
||||
rejects it together with `enable_frame_interpolation`, which needs the decoded
|
||||
frames the pre-encoded path no longer materializes.
|
||||
|
||||
Support is per model: MiniMax-H3 implements it, and other models ignore the flag
|
||||
and take the full-decode path.
|
||||
Support is per model: MiniMax-H3 and Wan 2.2 (T2V, I2V, and S2V) implement it,
|
||||
and other models ignore the flag and take the full-decode path.
|
||||
|
||||
## Storage
|
||||
|
||||
|
||||
Reference in new issue
Block a user