# Unified Full-Duplex Framework This document is the design record of the full-duplex serving framework: the engine-resident session model, the typed command/event contract, the single model plugin, and the serving/API surfaces built on top of them. It is the owner document for `vllm_omni/engine/duplex/**`, `vllm_omni/engine/duplex_omni_engine.py`, `vllm_omni/engine/duplex_orchestrator.py`, `vllm_omni/entrypoints/duplex_omni.py`, `vllm_omni/entrypoints/duplex/**`, `vllm_omni/clients/**` and the per-model `duplex/` packages. The public wire contract lives in [`docs/serving/realtime_duplex_api.md`](../serving/realtime_duplex_api.md); this page describes how it is produced. ## Purpose A full-duplex model listens and speaks at the same time: audio streams into the model while it produces audio, the model decides when to speak, and the user can interrupt. Serving such a model needs a long-lived session with its own identity (epoch, turn), ledgers (input, response, playback, history), a lease (idle TTL, disconnect grace, resume) and a binding to the resumable stage requests that carry the session's audio through the pipeline. The framework gives that session exactly one owner, keeps the generic turn-based stack free of duplex code, and exposes the same session to three consumers: the OpenAI Realtime WebSocket route, the in-process Python API and the model plugins. ## Architecture ```text Python users WebSocket clients ┌──────────────────────┐ ┌────────────────────────┐ │ InlineDuplexClient │ │ DuplexClient │ │ (DuplexClientBase) │ │ (DuplexClientBase) │ └──────────┬───────────┘ └───────────┬────────────┘ │ DuplexSessionHandle │ /v1/realtime?duplex=1 (alias /v1/duplex) │ ▼ │ ┌──────────────────────────────────────┐ │ │ OmniDuplexSessionHandler (thin) │ │ │ websocket I/O, command_from_realtime│ │ │ event.to_realtime(), attachment / │ │ │ resume tokens / replay journal │ │ └──────────────────┬───────────────────┘ ▼ ▼ DuplexSessionHandle ┌──────────────────────────────────────────────────────────────────────────────┐ │ DuplexOmni(AsyncOmni) [entrypoints/duplex_omni.py, thin] │ │ open_session / get_session / resume_session / detach_session / close_session│ │ DuplexSessionHandle: submit(DuplexCommand) / events() -> DuplexEvent │ └──────────────────────────────────────┬───────────────────────────────────────┘ │ engine.open/close/resume/touch_session_async (RPC) │ engine.submit_command_async (one-way) ▼ ┌──────────────────────────────────────────────────────────────────────────────┐ │ DuplexOmniEngine(AsyncOmniEngine) [engine/duplex_omni_engine.py, thin] │ │ loads the plugin, validates session_mode, builds queue messages │ │ └─ DuplexOrchestrator(Orchestrator) [engine/duplex_orchestrator.py]│ │ template seams + DuplexStagePort over the shared stage machinery │ │ ├─ DuplexSessionManager admission, backpressure, lease reaper, │ │ │ open/close/resume/touch, command dispatch │ │ └─ DuplexSessionRunner ×N ONE session state (DuplexEngineSession), │ │ ordered mailbox, append planning + stage │ │ submit, output projection, response / │ │ playback / history, VAD, typed events │ │ plugin: DuplexModelPlugin (PipelineConfig.duplex_plugin, one/model) │ └──────────────────────────────────────────────────────────────────────────────┘ ``` In each layer the duplex class extends the turn-based one, which extends the shared base. The dependency runs duplex -> turn-based and never the reverse, so the turn-based stack is unchanged and the duplex stack reuses its request machinery instead of duplicating it: ```text OmniBase OmniEngineBase OrchestratorBase ├─ Omni (sync, turn) └─ AsyncOmniEngine (turn requests) └─ Orchestrator (turn-based admission) └─ AsyncOmniBase └─ DuplexOmniEngine (session msgs) └─ DuplexOrchestrator (hosts DuplexSessionManager, └─ AsyncOmni (EngineClient) implements DuplexStagePort) └─ DuplexOmni AsyncOmni._create_engine -> AsyncOmniEngine AsyncOmniEngine._create_orchestrator -> Orchestrator DuplexOmni._create_engine -> DuplexOmniEngine DuplexOmniEngine._create_orchestrator -> DuplexOrchestrator ``` Each concrete class constructs its collaborator in an explicit factory method the base calls at the right moment: `OmniBase.__init__` calls `self._create_engine(...)`; `OmniEngineBase._bootstrap_orchestrator` calls `self._create_orchestrator(...)` on the orchestrator thread once the stages are initialized. `DuplexOmniEngine._create_orchestrator` passes the loaded plugin and the session runtime config to `DuplexOrchestrator` directly. ### Design rules 1. **One session authority.** The whole duplex session (config, epoch/turn, ledgers, lease, stage request resources, model state, projection state) is one object, `DuplexEngineSession`, owned by one `DuplexSessionRunner` on the orchestrator loop. Nothing above the engine keeps session state beyond a handle (session id, event queue, capabilities, public session object). 2. **Typed contract.** Commands into a session are `DuplexCommand` dataclasses (`engine/duplex/commands.py`); outputs are `DuplexEvent` dataclasses (`engine/duplex/events.py`). `command_from_realtime()` and `DuplexEvent.to_realtime()` derive the OpenAI Realtime JSON, so the wire format is never hand-built, and the websocket handler and the inline client share one conversion. 3. **Generic bases and the turn-based classes have zero duplex vocabulary.** `OmniBase`, `AsyncOmniBase`, `AsyncOmni`, `OmniEngineBase`, `AsyncOmniEngine`, `OrchestratorBase` and `Orchestrator` carry only template seams; `tests/engine/test_duplex_import_boundary.py` checks that importing the turn-based stack loads no duplex module. 4. **Every surface a duplex server exposes is backed by a session.** `vllm-omni serve` constructs `DuplexOmni` when the pipeline declares `duplex_plugin`; the server exposes `/v1/realtime?duplex=1` (alias `/v1/duplex`), `POST /v1/chat/completions`, `/v1/models` and `/health`, and every other turn-based route reports "not available". The chat route is served by `DuplexChatCompletionsAdapter`, a Realtime client that runs one short-lived session per request: speech becomes a committed turn, text is seeded as the session's opening turn because a model-native model takes a turn only when it hears speech. Whether a model can be reached with text is `DuplexCapabilities.supports_chat_completions`; one that cannot is refused at once rather than left to idle out. Turn-based use of the same model stays available offline through `Omni` / `AsyncOmni`. 5. **Serving is transport only.** The websocket handler does socket I/O, wire-envelope validation, command translation, event rendering and the attachment/resume/replay bookkeeping; it holds no session state. 6. **No `typing.Protocol`** in the duplex surfaces: the plugin, data plane, session state, PCM buffer, stage port and client transport seams are ABCs. ## Package layout ```text vllm_omni/ ├── entrypoints/ │ ├── omni_base.py OmniBase (+ abstract _create_engine) │ ├── async_omni_base.py AsyncOmniBase (output pump, _route_engine_message seam) │ ├── async_omni.py AsyncOmni (turn-based requests) │ ├── duplex_omni.py DuplexOmni, DuplexSessionHandle (thin pipe) │ ├── duplex/ SERVING (thin) │ │ ├── serving.py OmniDuplexSessionHandler │ │ ├── realtime_input.py RealtimeEnvelope (query rules, first message), parse_resume_request │ │ ├── session_attachment.py DuplexSessionAttachmentRegistry (resume tokens, replay journal) │ │ ├── audio_encoding.py encode_audio, injected into DuplexOmniEngine for the plugin's data plane │ │ ├── chat_completions.py DuplexChatCompletionsAdapter (/v1/chat/completions on a session per request) │ │ └── websocket.py websocket send/close/receive helpers │ └── openai/api_server.py builds DuplexOmni for duplex models; session-backed app state ├── engine/ │ ├── omni_engine_base.py OmniEngineBase (stage processes, orchestrator thread, queues, RPC) │ ├── async_omni_engine.py AsyncOmniEngine (turn-based request building) │ ├── duplex_omni_engine.py DuplexOmniEngine (session message surface, creates DuplexOrchestrator) │ ├── orchestrator.py OrchestratorBase + Orchestrator │ ├── duplex_orchestrator.py DuplexOrchestrator (+ DuplexOrchestratorRequestState; implements DuplexStagePort) │ └── duplex/ │ ├── commands.py DuplexCommand dataclasses, command_from_realtime │ ├── realtime_commands.py Realtime client event -> DuplexCommand translation │ ├── events.py DuplexEvent dataclasses (+ to_realtime), REALTIME_ERROR_TYPES_BY_CODE │ ├── realtime_events.py RealtimeProjectionState: internal event -> typed events │ ├── messages.py queue envelopes (Open/Close/Resume/Touch/Command/Result/Event), DuplexSessionError │ ├── config.py DuplexSessionConfig, DuplexCapabilities, ResponseCreateOptions │ ├── contracts.py DuplexFence (session_id, epoch, turn_id), stage request records, DuplexStagePort │ ├── plugin.py DuplexModelPlugin, DuplexModelSessionState, DuplexDataPlane, PcmAppendBuffer ABCs │ ├── turn_detection.py server-side VAD turn detector used by the session │ ├── audio.py / vad.py / intermediate.py │ └── session/ one engine-resident session and everything that runs it │ ├── engine_session.py DuplexEngineSession: ledgers, lease, fence, stage resources, append sequencing │ ├── runner.py DuplexSessionRunner (per-session mailbox on the orchestrator loop) │ ├── manager.py DuplexSessionManager (admission, backpressure, reaper, dispatch) │ ├── context.py DuplexSessionContext / DuplexRunState: what the runner shares with its components │ ├── emitter.py SessionEmitter: projection, epoch filter, domain effects of a terminal event │ ├── model_channel.py ModelChannel: submit an append, project stage output, continue a turn │ ├── control.py SessionControl: server VAD and the events that reconfigure it │ ├── append_task.py AppendAttempt: one append in flight and the rollback its failure owes │ ├── helpers.py pure reads and payload builders over a session │ ├── lease.py DuplexLeaseState (idle TTL, disconnect grace, resume generation) │ └── overlap_policy.py / commit_policy.py / playback_ledger.py ├── config/stage_config.py PipelineConfig.duplex_plugin; DuplexSessionRuntimeConfig ├── model_executor/models/minicpmo_4_5/duplex/plugin.py MiniCPMO45DuplexPlugin (+ data_plane, input, policy, ...) └── clients/ ├── duplex.py DuplexClientBase (ABC), DuplexClient (websocket), client-side events ├── inline_duplex.py InlineDuplexClient (in-process, over DuplexOmni) ├── minicpmo_4_5.py MiniCPM-o 4.5 session preset └── personaplex.py PersonaPlex session preset ``` ## Session identity - **The server allocates every session id.** `DuplexOmni.open_session` generates `duplex-`; a `session_id` / `id` key inside the Realtime session object is ignored (Realtime clients echo the session object back). The allocated id reaches the client in `session.created` and is the only handle for `session.resume`, `close` and every command. - **Ids are never reused** within an engine's lifetime, so the id alone identifies a session. There is no incarnation counter: stage request ids are `duplex-s..e..r.`, model-side per-session state is keyed by `session_id` alone, and a command, close, touch or resume for an id the manager no longer holds is answered with `unknown_session`. - **Staleness inside a live session is epoch-based.** `DuplexFence (session_id, epoch, turn_id)` is an engine-internal identity: every stage request is bound to the fence it was submitted under, cancels advance the epoch, and tracked tasks re-check the epoch after each `await`. - **Resume identity** is `(session_id, resume_token, expected_lease_generation)`: the token belongs to the attachment registry, the lease generation is the engine-side compare-and-swap. ## Session lifecycle ```text open DuplexOmni.open_session(config) -> engine.open_session_async(session_id, DuplexSessionConfig) [RPC] -> DuplexSessionManager.open: admission (max_sessions, closing sessions keep their slot), plugin.validate_client_extra_body / prepare_runtime_config, DuplexEngineSession, Stage0 request id reserved (ensure_stage_request), DuplexSessionRunner.start() -> first event: SessionCreated (announces the allocated id) command handle.submit(DuplexCommand) -> engine.submit_command_async -> DuplexSessionCommandMessage on the request queue [one-way] -> DuplexSessionManager.dispatch: unknown_session / input_backpressure checks, then runner mailbox output DuplexOrchestrator._intercept_stage_output -> runner.on_stage_output -> mailbox -> typed events -> output_sink (DuplexSessionEventMessage) -> DuplexOmni._route_engine_message -> handle.events() detach DuplexOmni.detach_session -> touch(DETACH): engine-owned disconnect grace; expiry -> SessionExpired resume DuplexOmni.resume_session(expected_lease_generation) -> lease CAS; the existing handle is re-entered close DuplexOmni.close_session -> close RPC; the manager tears the runner down, then the stage cleanup (abort submitted requests, release reserved ids), then SessionClosed, then the RPC result: seeing the event means the admission slot is free (it is held until the cleanup succeeded) reap DuplexSessionManager.reaper_loop: idle TTL / disconnect grace expiry, cleanup retries ``` ### Concurrency and ordering model of the runner Everything below runs on the orchestrator asyncio loop; there is no lock. - **Single writer per session.** `DuplexEngineSession` is mutated only by its runner. Inputs reach the runner through one mailbox in this order: client commands (from `_request_handler`, in request-queue order), stage outputs (`on_stage_output` enqueues an internal item; it does not mutate the session inline), timers (silence continuation, bounded auto-response) and internal items (deferred overlap promotion, VAD-synthesized commits). The mailbox worker processes items one at a time. - **Slow work is not awaited inline.** An `AppendAudio` handler reserves bytes, then schedules a tracked append task on the per-session append tail: the task awaits `_offload(decode/resample/VAD)` on a dedicated `ThreadPoolExecutor` owned by the manager, then `plugin.plan_append`, then `stage_port.submit`. The worker returns to the mailbox immediately, so a later `CancelResponse` is not blocked behind an in-flight append. - **Re-validation after every `await`.** A tracked task captures `(epoch, turn_id)` when it starts and re-checks them before the stage submit and before committing ledgers; a task that finds the epoch advanced rolls back its PCM reservation and exits. - **Cancel is atomic at the session.** `CancelResponse` / `BargeIn` advance the epoch, cancel tracked append tasks, abort the owned stage bindings through the stage port and emit the cancelled terminal events before yielding. An append task already past its last check is harmless: its stage request is bound to the old epoch's request id, which the abort covers, and the base never emits session-owned outputs to a client. - **Ordered event stream.** Because the epoch bump and every emission happen on one loop in program order, nothing can be emitted for an old epoch after its terminal event; `runner.emit()` still drops a terminal carrying a stale epoch and stamps `epoch` on every event for clients that filter defensively. This is a statement about order, not about latency: the engine output queue and each `DuplexSessionHandle` outbox are unbounded and FIFO, so a cancellation is delivered behind whatever audio was already emitted for the response it cancels. A client that must stop quickly cancels its own playback on `response.done` / `audio.cancelled` rather than waiting for the stream to drain. Bounding those buffers and letting an accepted invalidation skip undelivered media is left to the follow-up RFC. - **Backpressure before the mailbox.** `DuplexSessionManager.dispatch` checks `max_pending_input_bytes_per_session` and reserves a pending turn for `Commit` before the put; a rejected command is answered with `ErrorEvent(code="input_backpressure")`. - **Per-response stage metrics.** `_route_output` computes the per-segment `StageMetrics` before calling `_intercept_stage_output`; the runner accumulates the snapshot into the active response, so `ResponseDone.stage_metrics` and `metadata.vllm_omni.stage_metrics` on the wire keep their meaning. ## Orchestrator seams `OrchestratorBase` keeps the generic machinery (request handler skeleton, abort, collective RPC, output loops, stage error and dead-replica handling, `_route_output`, stage forwarding, prewarm, cleanup, shutdown) and offers these seams. `DuplexOrchestrator` extends `Orchestrator` rather than sitting beside it: a seam it does not override falls through to the turn-based implementation, and `_dispatch_message` hands anything that is not a session message to `super()`: | Seam | `Orchestrator` | `DuplexOrchestrator` | | --- | --- | --- | | `_dispatch_message(msg) -> bool` | `add_request`, `streaming_update`, `add_companion_request`, `interaction` | open/command/close/resume/touch messages -> `DuplexSessionManager` | | `_background_tasks()` | — | `[manager.reaper_loop()]` | | `_shutdown_extensions()` | — | `await manager.shutdown()` | | `_on_stage_submitted(stage_id, request_id, replica_id, req_state)` | — | bind the forwarded stage request to the owning session | | `_intercept_stage_output(...) -> bool` | — | hand every session-owned output (Stage0 decision, Stage1 audio) plus its metric snapshot to `runner.on_stage_output`; `True` = do not forward | | `_handle_forward_failure(...) -> bool` | — | fail the owning session's response and close the session, keep the loop alive | | `_cleanup_request_ids(ids, abort=, release_owners=)` | ignores `release_owners` | brackets `super()` with `close_sessions_for_request_ids` / `defer_request_cleanups` / `finalize_closed_sessions` | | `OrchestratorRequestState.session_owned` | never set | set by `ensure_request`: no synthetic terminal, no client emission, no terminal re-forward, no auto-cleanup on finish | `DuplexOrchestrator` also implements the `DuplexStagePort` ABC the manager and runner use (`stage_count`, `sampling_defaults`, `ensure_request`, `submit`, `cleanup`, `abort_requests`): the resumable Stage0 request is preregistered at open with `DuplexOrchestratorRequestState(session_owned=True, session_id, fence)`, submitted with `submit_initial` on the first unit and `submit_update` afterwards, and the request's `streaming.bridge_states["duplex"]` carries the session identity (`session_id`, `fence`, `epoch`, `turn_id`, `model_turn_id`), the public session config and the server-owned runtime config for the worker-side model hooks. ## Model plugin `DuplexModelPlugin` (ABC, `engine/duplex/plugin.py`) is the one class a model provides, selected by `PipelineConfig.duplex_plugin`; "is a duplex model" is `duplex_plugin is not None`. It merges the engine policy and the session policy that used to be two separately configured objects: | Half | Members | | --- | --- | | engine policy | `configure_sampling_params(runtime_config, defaults)`, `plan_append(...) -> DuplexAppendPlan` (the resumable Stage0 prompt for one unit), `decide_output(...) -> DuplexOutputDecision \| None` (e.g. the listen decision on a finished Stage0 segment) | | session policy | `capabilities(max_sessions)`, `validate_client_extra_body`, `prepare_runtime_config(config, model_config)` (server-owned runtime keys, reference audio resolution), `runtime_config_for_update`, `runtime_config_for_function_output`, `create_session_state() -> DuplexModelSessionState`, `data_plane: DuplexDataPlane` (projects raw stage outputs into internal events), `data_plane_context(...)` | `DuplexOmniEngine._validate_deployment` loads the plugin before any stage starts; `DuplexSessionManager.__init__` validates it against the stage sampling defaults once the stage pools exist. Plugin hooks that may block (`prepare_runtime_config` fetching `ref_audio`) are awaited in `open()` and offloaded from the loop. The MiniCPM-o 4.5 plugin (`model_executor/models/minicpmo_4_5/duplex/plugin.py`) is the only integration in this framework version. PersonaPlex and Nemotron VoiceChat still carry their pre-framework duplex code (runtime extension plus serving adapter) and are therefore not served over this framework yet: their pipelines declare no `duplex_plugin`, so they run turn-based until the follow-up PRs port them (RFC vllm-omni#7181, PR 2/3). ## Serving `OmniDuplexSessionHandler.handle_realtime_session(websocket)`: 1. accept; build the `RealtimeEnvelope` (autostart / default session from query params, `event_id` correlation, error rendering through the typed `ErrorEvent`); 2. handshake: first message `session.update` -> `omni.open_session(session)` (the id inside the payload is ignored; the allocated id is announced in `session.created` together with `resume_token` / `attachment_generation` when the model supports resume); `session.resume(session_id, resume_token, last_received_server_event_seq)` -> `attachment.authenticate_resume` -> `omni.resume_session`; 3. reader loop: JSON -> `RealtimeEnvelope.translate` (`translate_realtime_command` with the session's declared audio defaults) -> `handle.submit`; envelope-level errors (invalid JSON, oversize frame, unknown type, event acks) are answered locally; 4. writer pump (session-scoped, survives reconnects): `async for ev in handle.events(): attachment.send_event(ev.to_realtime())`, journaling for replay until the journal overflows (`session.resync_required`); 5. disconnect: a resumable session is detached (`attachment.detach` + `omni.detach_session`, engine-owned grace); a superseded socket's disconnect is a no-op; a non-resumable session is closed. Takeover sends `session.replaced` to the old socket and closes it. ## Validation - CPU suites: `tests/engine/duplex/**` (session, lease, manager, runner, plugin, events, commands), `tests/engine/test_duplex_orchestrator.py`, `tests/engine/test_duplex_omni_engine.py`, `tests/entrypoints/test_duplex_omni.py`, `tests/entrypoints/duplex/test_duplex_serving.py`, `tests/entrypoints/openai/test_duplex_session_attachment.py`, `tests/entrypoints/openai_api/test_duplex_api_server.py` (the duplex server: pipeline probe, app state, routes, warmup gate), `tests/entrypoints/duplex/test_chat_completions_adapter.py`, `tests/clients/**`, `tests/engine/test_duplex_import_boundary.py`, `tests/model_executor/models/minicpmo_4_5/duplex/**`, `tests/worker/test_native_duplex_hooks.py`. - GPU: `examples/online_serving/minicpmo/realtime_duplex_demo.py` (one turn over the websocket), `examples/online_serving/barge_in_client.py` (cancel during playback; `--inline` drives the same scenario through `InlineDuplexClient`), `tests/e2e/online_serving/run_minicpmo_realtime_duplex_multi_session.py` (admission, resume, takeover, expiry).