[Bugfix] Fix Nemotron VoiceChat duplex nightly handshake (#7884)
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
26 files changed
+1186
-934
No files matched your search
@@ -72,12 +72,21 @@ does not switch engines. The Python `Omni` / `AsyncOmni` APIs are unchanged.
|
||||
`session.created.session.capabilities` is present before treating the
|
||||
connection as full duplex.
|
||||
|
||||
**MiniCPM-o 4.5** (`vllm_omni/deploy/minicpmo_4_5.yaml`) is the only model
|
||||
served over this endpoint today. PersonaPlex and Nemotron VoiceChat still carry
|
||||
their pre-framework duplex code: their pipelines declare no `duplex_plugin`, so
|
||||
they run turn-based until the follow-up PRs port them to the plugin contract
|
||||
**MiniCPM-o 4.5** (`vllm_omni/deploy/minicpmo_4_5.yaml`), **Qwen3-Omni**
|
||||
(`vllm_omni/deploy/qwen3_omni_duplex.yaml`), and **Nemotron VoiceChat**
|
||||
(`vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml`) are served
|
||||
over this endpoint today. PersonaPlex still carries its pre-framework duplex
|
||||
code: its pipeline declares no `duplex_plugin`, so it runs turn-based until the
|
||||
follow-up PR ports it to the plugin contract
|
||||
(RFC [vllm-omni#7181](https://github.com/vllm-project/vllm-omni/issues/7181)).
|
||||
|
||||
For Nemotron VoiceChat Python clients, use
|
||||
`vllm_omni.clients.nemotron_voicechat.create_duplex_session_config()` with
|
||||
`DuplexClient`. The preset selects 16 kHz float32 input, 22.05 kHz PCM16 output,
|
||||
and automatic responses. Append 1,280-sample (80 ms) input frames; when using
|
||||
`stream_pcm`, set `chunk_ms=80`. Pass `instructions` and optional `tools` to
|
||||
the preset before opening the session.
|
||||
|
||||
JoyVL is a separate HTTP interaction orchestrator and does not use these
|
||||
WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
|
||||
|
||||
|
||||
@@ -8,11 +8,9 @@ history stays honest, and a dropped connection can resume the same session.
|
||||
This page covers how to run a duplex deployment, how to drive it from Python
|
||||
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
|
||||
|
||||
The endpoint is served for models that ship a duplex plugin — including
|
||||
MiniCPM-o 4.5 and Qwen3-Omni — and just for deploy configurations that declare
|
||||
`session_mode: duplex`. PersonaPlex and Nemotron VoiceChat are ported to the
|
||||
plugin contract in follow-up PRs and are not served over this endpoint yet. The runtime architecture
|
||||
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
The endpoint requires a duplex plugin and a deployment with `session_mode: duplex`.
|
||||
See [supported models and deployments](full_duplex_api.md#enable-full-duplex)
|
||||
and the [runtime architecture](../design/fullduplex.md).
|
||||
|
||||
## Qwen3-Omni conversation history
|
||||
|
||||
@@ -1340,9 +1338,6 @@ them out into typed events before they reach a client.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- Only MiniCPM-o 4.5 is served over this endpoint today; PersonaPlex and
|
||||
Nemotron VoiceChat arrive with the follow-up PRs that port them to the
|
||||
plugin contract.
|
||||
- Several surfaces are capability-gated per model (see *Capability
|
||||
negotiation by model* above): PersonaPlex does not support session resume,
|
||||
barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
|
||||
|
||||
Reference in new issue
Block a user