[Frontend] Allow explicit turn-based serving (#7675)
Signed-off-by: Linze-Shi <linzeshi0@gmail.com>
This commit is contained in:
1 parent
c32aaeb2da
commit
c8b3a17bd3
10 files changed
+205
-96
No files matched your search
@@ -22,14 +22,14 @@ capability gates, see the [Realtime Duplex API guide](realtime_duplex_api.md).
|
||||
## Enable Full Duplex
|
||||
|
||||
A model is served full duplex when its registered pipeline declares a
|
||||
`duplex_plugin` (the model's `DuplexModelPlugin`). That declaration alone
|
||||
decides it: `vllm-omni serve` constructs `DuplexOmni` instead of `AsyncOmni`,
|
||||
and every surface the server exposes is backed by a duplex session. It serves
|
||||
`duplex_plugin` (the model's `DuplexModelPlugin`) and its deploy configuration
|
||||
sets `session_mode: duplex`. `vllm serve --omni` constructs `DuplexOmni`
|
||||
instead of `AsyncOmni`. It serves
|
||||
`/v1/realtime?duplex=1` (and its alias `/v1/duplex`),
|
||||
`POST /v1/chat/completions`, `/v1/models` and `/health`; every other
|
||||
turn-based HTTP route (speech, batch, embeddings, video, ...) answers "not
|
||||
available". Turn-based use of such a model stays available offline through the
|
||||
Python API (`Omni` / `AsyncOmni`).
|
||||
available". Set `session_mode: turn` to select the ordinary online serving
|
||||
stack instead. The model's supported tasks and endpoint restrictions still apply.
|
||||
|
||||
`/v1/chat/completions` is the ordinary chat service running on the duplex
|
||||
engine: a request is a turn-based generation on the same stages, served
|
||||
@@ -45,8 +45,24 @@ The deploy configuration of such a model must agree:
|
||||
session_mode: duplex
|
||||
```
|
||||
|
||||
A duplex model started with a deploy configuration that does not set it fails
|
||||
at startup rather than falling back to turn-based serving.
|
||||
A duplex-capable model must explicitly set `session_mode` to `duplex` or `turn`
|
||||
in its deploy configuration, possibly through `base_config` inheritance.
|
||||
Missing or invalid values fail at startup. The default MiniCPM-o deployment
|
||||
continues to use duplex mode.
|
||||
|
||||
To run MiniCPM-o 4.5 with the turn-based engine:
|
||||
|
||||
```bash
|
||||
vllm serve openbmb/MiniCPM-o-4_5 --omni \
|
||||
--deploy-config vllm_omni/deploy/minicpmo_4_5_turn.yaml \
|
||||
--trust-remote-code \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
This profile inherits the default model and stage settings and overrides only
|
||||
`session_mode`. It supports ordinary HTTP requests without creating a duplex
|
||||
session handler. Mode selection happens at startup; a WebSocket query parameter
|
||||
does not switch engines. The Python `Omni` / `AsyncOmni` APIs are unchanged.
|
||||
|
||||
!!! warning
|
||||
|
||||
|
||||
Reference in new issue
Block a user