vllm-omni/docs/serving/full_duplex_api.md

Full-Duplex WebSocket API

vLLM-Omni provides a full-duplex runtime for models that can continue receiving speech while producing speech. It adds persistent session state, model-specific turn policy, overlap handling, playback acknowledgement, and optional session resume.

Full duplex is distinct from the turn-based Realtime Audio API.

This page is the endpoint overview. For the complete wire contract, the vllm_omni.clients.duplex.DuplexClient Python library, and the per-model capability gates, see the Realtime Duplex API guide.

Choose an Endpoint

Endpoint	Protocol	Recommended use
WS /v1/realtime?duplex=1	OpenAI Realtime-style events (the normative contract)	Applications and browser clients
WS /v1/duplex	Alias of /v1/realtime?duplex=1 (same protocol, same handler)	Clients that prefer a dedicated path
Python: DuplexOmni / InlineDuplexClient	Typed commands and events in-process	Embedding the model without a server

Enable Full Duplex

A model is served full duplex when its registered pipeline declares a duplex_plugin (the model's DuplexModelPlugin) and its deploy configuration sets session_mode: duplex. vllm serve --omni constructs DuplexOmni instead of AsyncOmni. It serves /v1/realtime?duplex=1 (and its alias /v1/duplex), POST /v1/chat/completions, /v1/models and /health; every other turn-based HTTP route (speech, batch, embeddings, video, ...) answers "not available". Set session_mode: turn to select the ordinary online serving stack instead. The model's supported tasks and endpoint restrictions still apply.

/v1/chat/completions is the ordinary chat service running on the duplex engine: a request is a turn-based generation on the same stages, served alongside the live websocket sessions, with the request options the chat service supports. It is wired only when the model's plugin declares DuplexCapabilities.supports_chat_completions; on any other duplex model the route reports "not available". A model that should not serve the route at all lists it in the deploy configuration's endpoint_restrictions.

The deploy configuration of such a model must agree:

session_mode: duplex

A duplex-capable model must explicitly set session_mode to duplex or turn in its deploy configuration, possibly through base_config inheritance. Missing or invalid values fail at startup. The default MiniCPM-o deployment continues to use duplex mode.

To run MiniCPM-o 4.5 with the turn-based engine:

vllm serve openbmb/MiniCPM-o-4_5 --omni \
  --deploy-config vllm_omni/deploy/minicpmo_4_5_turn.yaml \
  --trust-remote-code \
  --port 8091

This profile inherits the default model and stage settings and overrides only session_mode. It supports ordinary HTTP requests without creating a duplex session handler. Mode selection happens at startup; a WebSocket query parameter does not switch engines. The Python Omni / AsyncOmni APIs are unchanged.

!!! warning

On a deployment that is *not* duplex, `WS /v1/duplex` fails with
`Duplex API is not available` and `/v1/realtime?duplex=1` falls back to the
ordinary turn-based realtime handler. Confirm that
`session.created.session.capabilities` is present before treating the
connection as full duplex.

MiniCPM-o 4.5 (vllm_omni/deploy/minicpmo_4_5.yaml), Qwen3-Omni (vllm_omni/deploy/qwen3_omni_duplex.yaml), and Nemotron VoiceChat (vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml) are served over this endpoint today. PersonaPlex still carries its pre-framework duplex code: its pipeline declares no duplex_plugin, so it runs turn-based until the follow-up PR ports it to the plugin contract (RFC vllm-omni#7181).

For Nemotron VoiceChat Python clients, use vllm_omni.clients.nemotron_voicechat.create_duplex_session_config() with DuplexClient. The preset selects 16 kHz float32 input, 22.05 kHz PCM16 output, and automatic responses. Append 1,280-sample (80 ms) input frames; when using stream_pcm, set chunk_ms=80. Pass instructions and optional tools to the preset before opening the session.

JoyVL is a separate HTTP interaction orchestrator and does not use these WebSocket endpoints. See Standalone Experimental Servers.

MiniCPM-o Quick Start

Start the duplex deployment:

vllm serve openbmb/MiniCPM-o-4_5 --omni \
  --deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \
  --trust-remote-code \
  --port 8091

Stream a mono, PCM16, 16 kHz WAV file with the provided client:

python examples/online_serving/minicpmo/realtime_duplex_demo.py \
  --url 'ws://localhost:8091/v1/realtime?duplex=1' \
  --model openbmb/MiniCPM-o-4_5 \
  --input-wav input_16k_mono.wav \
  --ref-audio reference_voice.wav \
  --output-dir /tmp/minicpmo-duplex

Realtime Event Lifecycle

A typical /v1/realtime?duplex=1 session follows this lifecycle:

1. Send session.update with the model, modalities, audio formats, and session options.
2. Wait for session.created; inspect session.capabilities instead of assuming every model supports the same controls.
3. Send input_audio_buffer.append events while microphone audio arrives.
4. Send input_audio_buffer.commit at a user-turn boundary when required by the model policy.
5. Consume response.created, transcript deltas, response.output_audio.delta, and response.done or response.listen events.
6. Send playback.ack after audio has been played when the session advertises playback acknowledgement support.
7. Send session.close and wait for session.closed.

Unlike the turn-based realtime endpoint, input may continue while a response is active. The server can emit overlap.decision to describe whether input was deferred, treated as a short acknowledgement, or used to interrupt output.

Capabilities and Model Differences

The session.created payload includes capability fields such as supports_barge_in, supports_playback_ack, supports_multi_session, supports_session_resume, and chunk_period_ms. Treat this payload as the runtime contract and branch on the flags, never on the model name: a model that supports native overlapping speech may still advertise supports_barge_in=false when destructive output interruption and model-state rewind are not validated for it. Capacity and session-resume behavior also depend on the selected deployment configuration. The per-model table lives in the Realtime Duplex API guide.

Python API

vllm_omni.entrypoints.duplex_omni.DuplexOmni runs the same engine-resident sessions in-process: open_session() returns a DuplexSessionHandle whose submit() takes typed DuplexCommand objects and whose events() yields typed DuplexEvent objects (each with a to_realtime() wire rendering). vllm_omni.clients.inline_duplex.InlineDuplexClient exposes that handle behind the DuplexClient API. See the Realtime Duplex API guide.

See the MiniCPM-o example and the full-duplex runtime design for model-specific validation and architecture details.