chickeytonandClaude Opus 5
99ff4f3070
Signed-off-by: chickeyton <ngton2014@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
64 lines
3.4 KiB
Markdown
64 lines
3.4 KiB
Markdown
# Experimental Full-Duplex (JoyVL)
|
|
|
|
This package now contains the JoyVL framework and its example integration,
|
|
plus two client-side helpers kept for the benchmark and e2e drivers:
|
|
|
|
```text
|
|
core/ generic duplex scaffold used by the JoyVL adapter
|
|
joyvl/ JoyVL model-specific integration
|
|
client.py legacy Realtime probe client (RealtimeDuplexClient) used by
|
|
the omniinteract / omni-duplex-eval benchmarks and the
|
|
server-VAD and Nemotron e2e drivers; applications should use
|
|
vllm_omni.clients.duplex.DuplexClient instead
|
|
video_stacking.py camera-frame tiling for omni duplex video input
|
|
```
|
|
|
|
To run JoyVL, see
|
|
[`recipes/JD/JoyAI-VL-Interaction.md`](../../../recipes/JD/JoyAI-VL-Interaction.md).
|
|
|
|
The native full-duplex runtimes graduated out of this package. They now live
|
|
in the stable tree, where sessions are engine-resident (RFC
|
|
[vllm-omni#7181](https://github.com/vllm-project/vllm-omni/issues/7181)):
|
|
|
|
```text
|
|
vllm_omni/engine/duplex/ sessions, manager, runner, leases, typed contract
|
|
vllm_omni/engine/duplex_omni_engine.py DuplexOmniEngine (session message surface)
|
|
vllm_omni/engine/duplex_orchestrator.py DuplexOrchestrator (hosts the session manager)
|
|
vllm_omni/entrypoints/duplex_omni.py DuplexOmni + DuplexSessionHandle (Python API)
|
|
vllm_omni/entrypoints/duplex/ WebSocket transport (attachments, resume, replay)
|
|
vllm_omni/clients/duplex.py DuplexClient / DuplexClientBase
|
|
vllm_omni/model_executor/models/minicpmo_4_5/duplex/ MiniCPM-o 4.5 DuplexModelPlugin
|
|
vllm_omni/model_executor/models/personaplex/duplex/ PersonaPlex (pre-framework, not ported yet)
|
|
vllm_omni/model_executor/models/nemotron_voicechat/duplex/ Nemotron VoiceChat (pre-framework, not ported yet)
|
|
vllm_omni/model_executor/duplex_sampling.py AR-runner sampling hook helper
|
|
vllm_omni/outputs/duplex.py typed output decision envelope
|
|
```
|
|
|
|
For their architecture and validation scope, see
|
|
[`docs/design/fullduplex.md`](../../../docs/design/fullduplex.md) and
|
|
[`docs/design/fullduplex-personaplex.md`](../../../docs/design/fullduplex-personaplex.md).
|
|
PersonaPlex's single-process demo tier (browser client, standalone Moshi-web
|
|
server, `core/`-scaffold adapter) was demo-only and was removed rather than
|
|
graduated; the production path serves through the generic `/v1/duplex` stack.
|
|
|
|
## Adding a full-duplex model on the core contracts
|
|
|
|
The seam is `core.DuplexAdapter`. `core/` owns the session lifecycle,
|
|
epoch-based barge-in, playback cursor, and the event protocol; you implement
|
|
only model policy.
|
|
|
|
1. Create a sibling package `vllm_omni/experimental/fullduplex/<model>/`; keep
|
|
model-specific code there and do not touch `core/`.
|
|
2. Implement one `DuplexAdapter` (`capabilities` / `on_input` / `respond`; the
|
|
rest have defaults). Turn-based models run through `core.DuplexRuntime`
|
|
unchanged.
|
|
3. Promote a helper from a model package up into `core/` only once a second
|
|
model actually needs it.
|
|
|
|
For production serving, prefer the stable plugin seam instead: implement one
|
|
`vllm_omni.engine.duplex.plugin.DuplexModelPlugin` and name it in the model's
|
|
`pipeline.py` as `duplex_plugin`, as MiniCPM-o 4.5 does. PersonaPlex and
|
|
Nemotron VoiceChat still carry their pre-framework duplex code and are ported
|
|
to the plugin contract in follow-up PRs. The contract is documented in
|
|
[`docs/design/fullduplex.md`](../../../docs/design/fullduplex.md).
|