[Bugfix] Fix Nemotron VoiceChat duplex nightly handshake (#7884)
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
26 files changed
+1186
-934
No files matched your search
@@ -242,10 +242,12 @@ vllm_omni/
|
||||
│ └── overlap_policy.py / commit_policy.py / playback_ledger.py
|
||||
├── config/stage_config.py PipelineConfig.duplex_plugin; DuplexSessionRuntimeConfig
|
||||
├── model_executor/models/minicpmo_4_5/duplex/plugin.py MiniCPMO45DuplexPlugin (+ data_plane, input, policy, ...)
|
||||
├── model_executor/models/nemotron_voicechat/duplex/plugin.py NemotronVoiceChatDuplexPlugin (+ data_plane, input, capabilities)
|
||||
└── clients/
|
||||
├── duplex.py DuplexClientBase (ABC), DuplexClient (websocket), client-side events
|
||||
├── inline_duplex.py InlineDuplexClient (in-process, over DuplexOmni)
|
||||
├── minicpmo_4_5.py MiniCPM-o 4.5 session preset
|
||||
├── nemotron_voicechat.py Nemotron VoiceChat session preset
|
||||
└── personaplex.py PersonaPlex session preset
|
||||
```
|
||||
|
||||
@@ -388,7 +390,7 @@ policy that used to be two separately configured objects:
|
||||
| Half | Members |
|
||||
| --- | --- |
|
||||
| engine policy | `configure_sampling_params(runtime_config, defaults)`, `plan_append(...) -> DuplexAppendPlan` (the resumable Stage0 prompt for one unit), `decide_output(...) -> DuplexOutputDecision \| None` (e.g. the listen decision on a finished Stage0 segment) |
|
||||
| session policy | `capabilities(max_sessions)`, `validate_client_extra_body`, `prepare_runtime_config(config, model_config)` (server-owned runtime keys, reference audio resolution), `runtime_config_for_update`, `runtime_config_for_function_output`, `create_session_state() -> DuplexModelSessionState`, `data_plane: DuplexDataPlane` (projects raw stage outputs into internal events), `data_plane_context(...)` |
|
||||
| session policy | `capabilities(max_sessions)`, `validate_client_extra_body`, `prepare_runtime_config(config, model_config)` (server-owned runtime keys, reference audio resolution), `runtime_config_for_update`, `runtime_config_for_function_output`, `runtime_config_after_model_output` (consumption acknowledgement), `create_session_state() -> DuplexModelSessionState`, `data_plane: DuplexDataPlane` (projects raw stage outputs into internal events), `data_plane_context(...)` |
|
||||
|
||||
`DuplexOmniEngine._validate_deployment` loads the plugin before any stage
|
||||
starts; `DuplexSessionManager.__init__` validates it against the stage
|
||||
@@ -396,12 +398,12 @@ sampling defaults once the stage pools exist. Plugin hooks that may block
|
||||
(`prepare_runtime_config` fetching `ref_audio`) are awaited in `open()` and
|
||||
offloaded from the loop.
|
||||
|
||||
The MiniCPM-o 4.5 plugin (`model_executor/models/minicpmo_4_5/duplex/plugin.py`)
|
||||
is the only integration in this framework version. PersonaPlex and Nemotron
|
||||
VoiceChat still carry their pre-framework duplex code (runtime extension plus
|
||||
serving adapter) and are therefore not served over this framework yet: their
|
||||
pipelines declare no `duplex_plugin`, so they run turn-based until the
|
||||
follow-up PRs port them (RFC vllm-omni#7181, PR 2/3).
|
||||
Frame-based plugins use `engine/duplex/intermediate.py::build_duplex_append_prompt`
|
||||
for the shared request identity, sequencing and config snapshots. Token budgets,
|
||||
PCM framing and model-specific worker fields remain in the plugins.
|
||||
|
||||
See [supported models and deployments](../serving/full_duplex_api.md#enable-full-duplex)
|
||||
for the current plugin integrations and deployment configurations.
|
||||
|
||||
## Serving
|
||||
|
||||
|
||||
@@ -72,12 +72,21 @@ does not switch engines. The Python `Omni` / `AsyncOmni` APIs are unchanged.
|
||||
`session.created.session.capabilities` is present before treating the
|
||||
connection as full duplex.
|
||||
|
||||
**MiniCPM-o 4.5** (`vllm_omni/deploy/minicpmo_4_5.yaml`) is the only model
|
||||
served over this endpoint today. PersonaPlex and Nemotron VoiceChat still carry
|
||||
their pre-framework duplex code: their pipelines declare no `duplex_plugin`, so
|
||||
they run turn-based until the follow-up PRs port them to the plugin contract
|
||||
**MiniCPM-o 4.5** (`vllm_omni/deploy/minicpmo_4_5.yaml`), **Qwen3-Omni**
|
||||
(`vllm_omni/deploy/qwen3_omni_duplex.yaml`), and **Nemotron VoiceChat**
|
||||
(`vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml`) are served
|
||||
over this endpoint today. PersonaPlex still carries its pre-framework duplex
|
||||
code: its pipeline declares no `duplex_plugin`, so it runs turn-based until the
|
||||
follow-up PR ports it to the plugin contract
|
||||
(RFC [vllm-omni#7181](https://github.com/vllm-project/vllm-omni/issues/7181)).
|
||||
|
||||
For Nemotron VoiceChat Python clients, use
|
||||
`vllm_omni.clients.nemotron_voicechat.create_duplex_session_config()` with
|
||||
`DuplexClient`. The preset selects 16 kHz float32 input, 22.05 kHz PCM16 output,
|
||||
and automatic responses. Append 1,280-sample (80 ms) input frames; when using
|
||||
`stream_pcm`, set `chunk_ms=80`. Pass `instructions` and optional `tools` to
|
||||
the preset before opening the session.
|
||||
|
||||
JoyVL is a separate HTTP interaction orchestrator and does not use these
|
||||
WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
|
||||
|
||||
|
||||
@@ -8,11 +8,9 @@ history stays honest, and a dropped connection can resume the same session.
|
||||
This page covers how to run a duplex deployment, how to drive it from Python
|
||||
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
|
||||
|
||||
The endpoint is served for models that ship a duplex plugin — including
|
||||
MiniCPM-o 4.5 and Qwen3-Omni — and just for deploy configurations that declare
|
||||
`session_mode: duplex`. PersonaPlex and Nemotron VoiceChat are ported to the
|
||||
plugin contract in follow-up PRs and are not served over this endpoint yet. The runtime architecture
|
||||
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
The endpoint requires a duplex plugin and a deployment with `session_mode: duplex`.
|
||||
See [supported models and deployments](full_duplex_api.md#enable-full-duplex)
|
||||
and the [runtime architecture](../design/fullduplex.md).
|
||||
|
||||
## Qwen3-Omni conversation history
|
||||
|
||||
@@ -1340,9 +1338,6 @@ them out into typed events before they reach a client.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- Only MiniCPM-o 4.5 is served over this endpoint today; PersonaPlex and
|
||||
Nemotron VoiceChat arrive with the follow-up PRs that port them to the
|
||||
plugin contract.
|
||||
- Several surfaces are capability-gated per model (see *Capability
|
||||
negotiation by model* above): PersonaPlex does not support session resume,
|
||||
barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
|
||||
|
||||
Reference in new issue
Block a user