vllm-omni/vllm_omni/experimental/fullduplex

Experimental Full-Duplex Runtime

This package contains three experimental integrations:

• the existing JoyVL framework and example integration;
• the MiniCPM-o 4.5 native audio path used by /v1/duplex and /v1/realtime?duplex=1;
• the PersonaPlex lockstep speech-to-speech path (browser demo + batched serving, see recipes/NVIDIA/PersonaPlex.md).

For the MiniCPM active runtime path, lifecycle invariants, capability boundary, Realtime response contract, and validation scope, see DESIGN.md.

To run JoyVL, see recipes/JD/JoyAI-VL-Interaction.md.

Package boundaries

core/        model-agnostic duplex contracts (adapter, session, turn runtime)
engine/      AsyncOmni/orchestrator scheduler data-plane adapter
openai/      WebSocket transport, Realtime projection, and audio codecs
minicpmo45/  MiniCPM input framing, policy, compatibility, and Stage0 state
joyvl/       JoyVL model-specific integration
personaplex/ PersonaPlex lockstep engine, model-owned runtime, and serving

MiniCPM does not run through the experimental core.DuplexRuntime facade. Its active path uses the openai session controller, the experimental engine contracts, the standard scheduler/model runners, and an injected MiniCPM-specific runtime extension from minicpmo45/runtime.py.

personaplex/ is a Moshi-class, pure-lockstep speech-to-speech model on the core/ contracts. It keeps core/ untouched: the lockstep lifecycle (ONE eternal, frame-clocked response that drains on close, instead of the turn-style start/cancel-per-trigger one) is model policy and lives in the model package as PersonaPlexDuplexRuntime, mirroring the model-owned runtime shape of the MiniCPM-o duplex work. Its runnable serving path is personaplex/serving/ (single-session lease or --batch-size elastic slots) over personaplex/session.py (lockstep driver).

Adding a full-duplex model on the core contracts

The seam is core.DuplexAdapter. core/ owns the session lifecycle, epoch-based barge-in, playback cursor, and the event protocol; you implement only model policy.

1. Create a sibling package vllm_omni/experimental/fullduplex/<model>/; keep model-specific code there and do not touch core/.
2. Implement one DuplexAdapter (capabilities / on_input / respond; the rest have defaults). Turn-based models run through core.DuplexRuntime unchanged; a model needing a different lifecycle carries its own runtime in its package (see personaplex/adapter.py::PersonaPlexDuplexRuntime).
3. Promote a helper from a model package up into core/ only once a second model actually needs it.