anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Bugfix] Fix Nemotron VoiceChat duplex nightly handshake (#7884)

Browse Source 
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
Sy03 authored and GitHub committed 2026-09-25 17:25:15 +00:00
1 parent 5d0963b3dc
commit d31a2a3ba8
26 files changed
+1186 -934
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/design/fullduplex.md
+9 -7
View File
@@ -242,10 +242,12 @@ vllm_omni/
│ └── overlap_policy.py / commit_policy.py / playback_ledger.py
├── config/stage_config.py PipelineConfig.duplex_plugin; DuplexSessionRuntimeConfig
├── model_executor/models/minicpmo_4_5/duplex/plugin.py MiniCPMO45DuplexPlugin (+ data_plane, input, policy, ...)
├── model_executor/models/nemotron_voicechat/duplex/plugin.py NemotronVoiceChatDuplexPlugin (+ data_plane, input, capabilities)
└── clients/
├── duplex.py DuplexClientBase (ABC), DuplexClient (websocket), client-side events
├── inline_duplex.py InlineDuplexClient (in-process, over DuplexOmni)
├── minicpmo_4_5.py MiniCPM-o 4.5 session preset
├── nemotron_voicechat.py Nemotron VoiceChat session preset
└── personaplex.py PersonaPlex session preset
```
@@ -388,7 +390,7 @@ policy that used to be two separately configured objects:
| Half | Members |
| --- | --- |
| engine policy | `configure_sampling_params(runtime_config, defaults)`, `plan_append(...) -> DuplexAppendPlan` (the resumable Stage0 prompt for one unit), `decide_output(...) -> DuplexOutputDecision \| None` (e.g. the listen decision on a finished Stage0 segment) |
| session policy | `capabilities(max_sessions)`, `validate_client_extra_body`, `prepare_runtime_config(config, model_config)` (server-owned runtime keys, reference audio resolution), `runtime_config_for_update`, `runtime_config_for_function_output`, `create_session_state() -> DuplexModelSessionState`, `data_plane: DuplexDataPlane` (projects raw stage outputs into internal events), `data_plane_context(...)` |
| session policy | `capabilities(max_sessions)`, `validate_client_extra_body`, `prepare_runtime_config(config, model_config)` (server-owned runtime keys, reference audio resolution), `runtime_config_for_update`, `runtime_config_for_function_output`, `runtime_config_after_model_output` (consumption acknowledgement), `create_session_state() -> DuplexModelSessionState`, `data_plane: DuplexDataPlane` (projects raw stage outputs into internal events), `data_plane_context(...)` |
`DuplexOmniEngine._validate_deployment` loads the plugin before any stage
starts; `DuplexSessionManager.__init__` validates it against the stage
@@ -396,12 +398,12 @@ sampling defaults once the stage pools exist. Plugin hooks that may block
(`prepare_runtime_config` fetching `ref_audio`) are awaited in `open()` and
offloaded from the loop.
The MiniCPM-o 4.5 plugin (`model_executor/models/minicpmo_4_5/duplex/plugin.py`)
is the only integration in this framework version. PersonaPlex and Nemotron
VoiceChat still carry their pre-framework duplex code (runtime extension plus
serving adapter) and are therefore not served over this framework yet: their
pipelines declare no `duplex_plugin`, so they run turn-based until the
follow-up PRs port them (RFC vllm-omni#7181, PR 2/3).
Frame-based plugins use `engine/duplex/intermediate.py::build_duplex_append_prompt`
for the shared request identity, sequencing and config snapshots. Token budgets,
PCM framing and model-specific worker fields remain in the plugins.
See [supported models and deployments](../serving/full_duplex_api.md#enable-full-duplex)
for the current plugin integrations and deployment configurations.
## Serving