Yueqian LinandSy0307
e788ef6e76
Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Signed-off-by: Sy0307 <24841857+Sy0307@users.noreply.github.com> Co-authored-by: Sy0307 <24841857+Sy0307@users.noreply.github.com>
56 lines
2.6 KiB
Markdown
56 lines
2.6 KiB
Markdown
# Experimental Full-Duplex Runtime
|
|
|
|
This package contains three experimental integrations:
|
|
|
|
- the existing JoyVL framework and example integration;
|
|
- the MiniCPM-o 4.5 native audio path used by `/v1/duplex` and
|
|
`/v1/realtime?duplex=1`;
|
|
- the PersonaPlex lockstep speech-to-speech path (browser demo + batched
|
|
serving, see `recipes/NVIDIA/PersonaPlex.md`).
|
|
|
|
For the MiniCPM active runtime path, lifecycle invariants, capability boundary,
|
|
Realtime response contract, and validation scope, see [`DESIGN.md`](DESIGN.md).
|
|
|
|
To run JoyVL, see
|
|
[`recipes/JD/JoyAI-VL-Interaction.md`](../../../recipes/JD/JoyAI-VL-Interaction.md).
|
|
|
|
## Package boundaries
|
|
|
|
```text
|
|
core/ model-agnostic duplex contracts (adapter, session, turn runtime)
|
|
engine/ AsyncOmni/orchestrator scheduler data-plane adapter
|
|
openai/ WebSocket transport, Realtime projection, and audio codecs
|
|
minicpmo45/ MiniCPM input framing, policy, compatibility, and Stage0 state
|
|
joyvl/ JoyVL model-specific integration
|
|
personaplex/ PersonaPlex lockstep engine, model-owned runtime, and serving
|
|
```
|
|
|
|
MiniCPM does not run through the experimental `core.DuplexRuntime`
|
|
facade. Its active path uses the `openai` session controller, the experimental
|
|
engine contracts, the standard scheduler/model runners, and an injected
|
|
MiniCPM-specific runtime extension from `minicpmo45/runtime.py`.
|
|
|
|
`personaplex/` is a Moshi-class, pure-lockstep speech-to-speech model on the
|
|
`core/` contracts. It keeps `core/` untouched: the lockstep lifecycle (ONE
|
|
eternal, frame-clocked response that drains on close, instead of the turn-style
|
|
start/cancel-per-trigger one) is model policy and lives in the model package as
|
|
`PersonaPlexDuplexRuntime`, mirroring the model-owned runtime shape of the
|
|
MiniCPM-o duplex work. Its runnable serving path is `personaplex/serving/`
|
|
(single-session lease or `--batch-size` elastic slots) over
|
|
`personaplex/session.py` (lockstep driver).
|
|
|
|
## Adding a full-duplex model on the core contracts
|
|
|
|
The seam is `core.DuplexAdapter`. `core/` owns the session lifecycle,
|
|
epoch-based barge-in, playback cursor, and the event protocol; you implement
|
|
only model policy.
|
|
|
|
1. Create a sibling package `vllm_omni/experimental/fullduplex/<model>/`; keep
|
|
model-specific code there and do not touch `core/`.
|
|
2. Implement one `DuplexAdapter` (`capabilities` / `on_input` / `respond`; the
|
|
rest have defaults). Turn-based models run through `core.DuplexRuntime`
|
|
unchanged; a model needing a different lifecycle carries its own runtime in
|
|
its package (see `personaplex/adapter.py::PersonaPlexDuplexRuntime`).
|
|
3. Promote a helper from a model package up into `core/` only once a second
|
|
model actually needs it.
|