vllm-omni/vllm_omni/experimental/fullduplex

Full-duplex interaction framework

A model-agnostic framework for real-time, full-duplex (streaming-in / streaming-out) model interaction, plus the JoyVL implementation built on it.

To run the JoyVL model, see the recipe: recipes/JD/JoyAI-VL-Interaction.md. This README covers the framework itself and how to add a new model.

Layout

vllm_omni/experimental/fullduplex/
  core/      generic full-duplex framework (model-agnostic): DuplexRuntime (event
             loop + epoch barge-in), DuplexSession, DuplexAdapter (ABC), protocol
  joyvl/     JoyVL implementation (model-specific):
             adapter.py            JoyVLDuplexAdapter (implements core.DuplexAdapter)
             decision/             policy + output_parser + prompts (speak/silence/delegate)
             memory/               InteractionBrain — 3-tier summary memory (async)
             serving/              OpenAI-compatible HTTP orchestrator
             bridges/              model backend + delegation

core/ is the only part shared across models; data planes differ by model and are intentionally not shared.

Scope

The runnable serving path is joyvl/serving/ driving decision/ + memory/ directly. joyvl/adapter.py + core/ are a demonstration of how a model plugs into the generic full-duplex framework (exercised by tests/fullduplex/), not the serving path the HTTP orchestrator currently uses — a fused-audio model (e.g. MiniCPM-o) is the case core/ is built for.

Adding a full-duplex model

The seam is core.DuplexAdapter. core/ owns the session lifecycle, epoch-based barge-in, playback cursor, and the event protocol — you implement only model policy.

1. Create a sibling package vllm_omni/experimental/fullduplex/<model>/ next to joyvl/; keep model-specific code there and do not touch core/.

2. Implement one DuplexAdapter (three required methods; the rest have defaults):

   from collections.abc import AsyncIterator
   from vllm_omni.experimental.fullduplex.core.adapter import DuplexAdapter, DuplexCapability, OutputChunk
   from vllm_omni.experimental.fullduplex.core.session import DuplexSession
   
   class MyModelAdapter(DuplexAdapter):
       def capabilities(self) -> DuplexCapability:
           return DuplexCapability(
               input_modalities=frozenset({"audio", "text"}),
               output_modalities=frozenset({"audio", "text"}),
               proactive=True,            # speak without being asked?
           )
   
       async def on_input(self, session: DuplexSession, modality: str, data) -> None:
           ...                            # buffer/route an incoming chunk
   
       async def respond(self, session: DuplexSession) -> AsyncIterator[OutputChunk]:
           async for piece in self._model_stream(session):
               yield OutputChunk(modality="audio", data=piece)   # runtime drops stale
               #                                                   chunks after a barge-in
   
       # optional: should_respond / on_barge_in / on_playback_ack
   

3. Run it through the shared runtime — no new control-plane code:

   from vllm_omni.experimental.fullduplex.core.runtime import DuplexRuntime
   from vllm_omni.experimental.fullduplex.core.session import DuplexSession, DuplexSessionConfig
   
   rt = DuplexRuntime(DuplexSession("sid", DuplexSessionConfig()), MyModelAdapter())
   await rt.run(input_events, emit)
   

joyvl/adapter.py is the worked demonstration (currently exercised by tests, not the HTTP serving path). Promote a helper from a model package up into core/ only once a second model actually needs it.