vllm-omni/docs/serving/realtime_duplex_api.md

# Realtime Duplex API

vLLM-Omni serves full-duplex speech models over a WebSocket endpoint,
`/v1/realtime?duplex=1`, that speaks the OpenAI Realtime vocabulary plus a
set of duplex extensions: the model decides when to listen and when to speak,
the user can talk over the assistant, playback progress is reported back so
history stays honest, and a dropped connection can resume the same session.
This page covers how to run a duplex deployment, how to drive it from Python
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.

The endpoint requires a duplex plugin and a deployment with `session_mode: duplex`.
See [supported models and deployments](full_duplex_api.md#enable-full-duplex)
and the [runtime architecture](../design/fullduplex.md).

## Qwen3-Omni conversation history

Qwen3-Omni keeps incoming conversation items independently addressable by item
ID. Before applying the model's chat template, it merges consecutive user
items into one multimodal user message. An assistant message, including an
empty message reserved for an interrupted or unheard answer, separates turns.
The final user message places media before text instructions, following the
[official Qwen3-Omni demo's `format_history()`](https://github.com/QwenLM/Qwen3-Omni/blob/e4235853125589c789f06a2dd83e9f4126df5e9d/web_demo.py#L84-L179).

A camera image or text item may arrive after an audio commit and before
`response.create`. Those items belong to the same pending user turn: the
committed audio remains associated with its history item even when that item
is no longer last. Prompt preparation snapshots the selected inputs; later
items do not modify an already prepared request.

The duplex adapter retains its existing budgets: at most four audio payloads
(including the current input), an 8 MiB encoded-audio history budget (keeping
at least the newest payload), and at most eight prompt images. The recent
16-message window is rounded outwards to a complete turn. If any audio in an
older turn is no longer retained, the entire turn's model context, including
its images/text and assistant replies, is omitted. Images from that turn are
therefore **not** permanent session-wide model context. These limits differ
from the demo's one image-bearing turn and five audio-bearing turns; the
shared rule is grouping before pruning, rather than copying its UI budgets.

Grouping and pruning affect only the model-input view. They do not merge or
delete the source conversation items exposed to clients, and do not change
playback acknowledgement or interruption handling.

## Quick Start

### Start the Server

```bash
vllm-omni serve openbmb/MiniCPM-o-4_5 \
    --omni \
    --deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \
    --trust-remote-code \
    --host 0.0.0.0 --port 8099
```

`vllm_omni/deploy/minicpmo_4_5.yaml` declares `session_mode: duplex` and
`duplex_session.max_sessions: 16`. Because the MiniCPM-o 4.5 pipeline declares a
`duplex_plugin`, `vllm-omni serve` runs it through `DuplexOmni`: the server
mounts `ws://<host>:8099/v1/realtime?duplex=1` (this page; `ws://<host>:8099/v1/duplex`
is an alias of the same route), `POST /v1/chat/completions`, `/v1/models` and
`/health`, and nothing else. The remaining turn-based HTTP routes (speech,
batch, embeddings, video, ...) are not served in duplex mode.

`/v1/chat/completions` uses the ordinary chat service on the same engine
when the plugin declares `supports_chat_completions`. HTTP chat requests do
not create duplex sessions or consume session admission slots. Their options
follow the [Chat Completions API](chat_completions_api.md) and the model's
capabilities and endpoint restrictions.

For a turn-based online deployment, use `minicpmo_4_5_turn.yaml` or explicitly
set `session_mode: turn` in a deploy configuration derived from the model's
profile. This selects `AsyncOmni` and the ordinary API initialization instead
of the duplex handler. See [Full Duplex](full_duplex_api.md#enable-full-duplex)
for the command and configuration requirements.

### Run the Example Client

`examples/online_serving/barge_in_client.py` asks a question, interrupts the
answer mid-stream, and writes one WAV per response:

```bash
python examples/online_serving/barge_in_client.py \
    --url ws://127.0.0.1:8099/v1/realtime \
    --model openbmb/MiniCPM-o-4_5 \
    --ref-audio /path/to/reference_voice.wav \
    --question-wav question_16k.wav \
    --interrupt-wav follow_up_16k.wav \
    --output-dir ./duplex_out
```

Input WAVs must be mono 16 kHz PCM16. `--ref-audio` is required by the
MiniCPM-o preset (the session is otherwise rejected with
`ref_audio_required`). `--preset personaplex` switches the session shape to
the PersonaPlex preset. The flow is diagrammed in
`examples/online_serving/barge_in_client_flow.md`.

## Using DuplexClient

`vllm_omni.clients.duplex` is a client-side package: it depends only on the
standard library, `pybase64`, and the `websockets` package, and it is never
imported by server code. Full signatures are on the
[API reference page](../api/README.md#clients).

### Open a session

`DuplexClient` is an async context manager. Entering it connects, sends
`session.update`, and waits for `session.created`; exiting sends
`session.close` and waits for `session.closed`.

```python
from vllm_omni.clients.duplex import DuplexClient, audio_data_url
from vllm_omni.clients.minicpmo_4_5 import create_duplex_session_config

config = create_duplex_session_config(ref_audio=audio_data_url(Path("reference_voice.wav")))
async with DuplexClient("ws://127.0.0.1:8099/v1/realtime", model="openbmb/MiniCPM-o-4_5", config=config) as client:
    print(client.session_id, client.session_info["capabilities"])
```

The session shape comes from `SessionConfig`; model-specific knobs ride in
its `extra_body`, so the client itself stays model-neutral and each model
ships a preset:

| Preset | Input / output audio | What it sets |
| --- | --- | --- |
| `vllm_omni.clients.minicpmo_4_5.create_duplex_session_config(ref_audio=...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | `force_listen_count=0`, `overlap_policy="listen_only"`, `playback_commit_policy="ack_only"`; `ref_audio` is the assistant voice clip |
| `vllm_omni.clients.personaplex.create_duplex_session_config(voice="NATF2.pt", persona="")` | 24 kHz `pcm_f32le` / 24 kHz `pcm16` | bundled `.pt` voice prompt and the persona as `instructions` |
| `SessionConfig(...)` | 16 kHz `pcm16` / 24 kHz `pcm16` | model-neutral defaults; pass `extra_body`, `turn_detection`, `overlap_policy`, `playback_commit_policy`, `instructions`, `voice`, `temperature` yourself |

Keyword overrides on a preset replace the corresponding `SessionConfig`
field, e.g. `create_duplex_session_config(ref_audio=..., temperature=0.6)`.

Constructor options: `reconnect` (a `ReconnectPolicy`, default five
attempts with 0.25–4 s jittered backoff; `None` disables auto-resume),
`heartbeat_interval_s` (default 30 s; `None` disables the lease heartbeat),
`handshake_timeout_s`, and `connect` (a custom transport factory, used by
tests and benchmarks). The session id is allocated by the server and becomes
`client.session_id` once `session.created` arrives; a client cannot choose
it. Resume is supported only as automatic reconnect within the same
`DuplexClient` instance (see [Reconnect and resume](#reconnect-and-resume));
re-attaching from a new client or process requires the wire-level
`session.resume` handshake (`resume_token`,
`last_received_server_event_seq`), which this client does not expose.

### Stream audio and commit a turn

```python
from vllm_omni.clients.duplex import read_pcm16_wav

question = read_pcm16_wav(Path("question_16k.wav"))
await client.stream_pcm(question, chunk_ms=100)      # paced 100 ms appends
await client.commit(create_response=False)           # the model decides listen/speak
```

`stream_pcm` slices PCM into `chunk_ms` appends and paces them in real time
(`realtime=False` sends as fast as possible). `append_audio` sends one chunk
and accepts `is_speech` and `video_frames` (base64 JPEG/PNG strings riding
the append that closes a one-second model unit — data URLs are accepted and
their prefix stripped; `stream_pcm(video_frames=...,
stacked_video_frames=...)` interleaves a frame track automatically and
returns the number of frames sent). A turn is ended with `commit`, never by
a flag on the append.

`commit` seals the buffered utterance into a user item. With the default
`auto_response=True` a commit lets the model respond; pass
`create_response=False` to leave the listen/speak decision entirely to a
model-native session, or set `auto_response=False` in the config and call
`client.send({"type": "response.create"})` when you want an answer. In the
model-native lane the audio is already streaming into the model before the
commit, so the model may start answering — or emit a listen decision —
without any commit at all.

### Consume responses

`responses()` demultiplexes the event stream into one `ResponseHandle` per
assistant response. A model listen decision surfaces as an already-finished
handle with `decision == "listen"`.

```python
async for response in client.responses():
    if response.decision == "listen":
        continue                                   # the model kept listening
    async for chunk in response.audio():           # decoded PCM16 at 24 kHz
        play(chunk)
        await client.ack_playback(response.played_ms, response_id=response.response_id)
    await response.wait()
    print(response.transcript, response.done_event.raw["response"]["status"])
    break
```

`ResponseHandle` exposes `response_id`, `decision` (`"speak"` / `"listen"`),
the accumulated `transcript` and `text`, `played_ms`, `created_event`,
`done_event`, `finished`, `audio()` and `wait()`.

`ack_playback(played_ms, response_id=..., committed_ms=...)` reports the
cumulative playback position. Call it while audio plays: with the MiniCPM-o
preset's `playback_commit_policy="ack_only"` the assistant turn enters the
conversation history only as far as you acknowledge it, which is what keeps
history truthful after an interruption. `acknowledge_collected_playback(client,
collector)` acknowledges everything an `EventCollector` has received, which is
what the example and the benchmarks do at the end of a turn.

### Interrupt

```python
await client.cancel_response()        # cancel the active response (epoch advances)
await client.clear_input()            # drop un-committed input audio
```

In the model-native lane you usually do not need either: keep streaming
microphone audio while the assistant speaks and the server's overlap policy
decides — a short remark is acknowledged, meaningful overlap is deferred to
the turn end, and on models that advertise `supports_barge_in` (MiniCPM-o
4.5) the in-flight response is cancelled. Each decision is reported as an
`overlap.decision` event. `cancel_response` and `clear_input` compose a
client-forced barge-in for clients that run their own VAD.

### Work with raw events

`events()` (or `async for event in client`) yields every server event from
now on as a typed `DuplexEvent` (`SessionCreated`, `ResponseCreated`,
`AudioDelta`, `TranscriptDelta`, `TextDelta`, `ListenDecision`,
`SpeakDecision`, `ResponseDone`, `SessionClosed`, `SessionExpired`,
`ErrorEvent`, and `ConnectionResumed` after an automatic resume; unknown
types arrive as plain `DuplexEvent`). Common accessors are `type`,
`event_id`, `session_id`, `response_id`, `item_id`, `server_event_seq`,
`audio` (decoded bytes), `text`, and `raw` (the wire dict).
`wait_for("response.done", timeout_s=30)` blocks for the next event of the
given type(s), and `send({...})` is the escape hatch for any client event
the wrapper does not have a method for (`response.create`,
`conversation.item.create`, `conversation.item.truncate`, ...); it returns
the stamped `event_id`.

### Reconnect and resume

On a transport drop the client reconnects with the `ReconnectPolicy` backoff
and sends `session.resume` from the last acknowledged `server_event_seq`.
Replayed events are deduplicated, consumers see one `ConnectionResumed`
event, and `client.resume_token` is refreshed. The
server keeps a detached session alive for `disconnect_grace_s` (30 s by
default); after that a resume fails with `session_resume_expired`. Resume is
gated by `capabilities.supports_session_resume` (MiniCPM-o 4.5 and Nemotron
VoiceChat: yes; PersonaPlex: no). With `reconnect=None` the same drop surfaces
as `DuplexSessionClosedError`.

### Errors

Server `error` events are delivered as `ErrorEvent` (with `code`, `message`,
`related_event_id`) and raised as `DuplexProtocolError` from the awaiting
call; transport failures raise `DuplexConnectionError`; a session that ends
while you are awaiting it raises `DuplexSessionClosedError`. All three derive
from `DuplexClientError`.

### Measure latency

`EventCollector` accumulates events for assertions and metrics:

```python
from vllm_omni.clients.duplex import EventCollector

collector = EventCollector()
consume_task = asyncio.create_task(collector.consume(client))
commit_sent_at = time.monotonic()
await client.commit(create_response=False)
...
summary = collector.timing_summary(after_s=commit_sent_at, input_committed_at_s=commit_sent_at)
```

`timing_summary` reports client-observed first-text / first-audio latency,
audio cadence, and, when the deployment attaches engine stage metrics to the
response events (the benchmark requests them with
`extra_body.return_stage_metrics=True`), the Stage 0 token metrics
(`ttft_ms`, `tpot_ms`, `itls_ms`). It reports raw measurements only —
derived metrics such as the audio RTF are computed by the caller from
`audio_generation_ms` / `audio_duration_ms` (the benchmarks use
`vllm_omni.metrics.definitions.compute_audio_rtf`), keeping metric
definitions out of the dependency-free client. `audio_bytes(response_id)`, `count(type)`,
`errors()`, and `first_received_at(type)` cover the common assertions; the
`openai-realtime-duplex` benchmark backend (`vllm-omni bench serve`) is built
on the same helpers.

## Using the Python API

`vllm_omni.entrypoints.duplex_omni.DuplexOmni` is the in-process Python API
of a duplex model: sessions run inside the engine, and the class only opens,
resumes and closes them and pipes typed commands in and typed events out
through a `DuplexSessionHandle`.

```python
from vllm_omni.engine.duplex.commands import AppendAudio, Commit
from vllm_omni.engine.duplex.events import AudioDelta, SessionClosed
from vllm_omni.entrypoints.duplex_omni import DuplexOmni

omni = DuplexOmni(model="openbmb/MiniCPM-o-4_5", trust_remote_code=True)
async with await omni.open_session({"modalities": ["audio", "text"], "ref_audio": ref_audio_data_url}) as session:
    async def consume():
        async for event in session.events():          # typed DuplexEvent objects
            if isinstance(event, AudioDelta):
                play(event.audio)
            elif isinstance(event, ResponseDone):
                break                                  # one turn is enough here
    task = asyncio.create_task(consume())
    await session.submit(AppendAudio(audio=pcm16_bytes, format="pcm16", sample_rate_hz=16000))
    await session.submit(Commit())
    await task
# leaving the block closes the session (the last event is SessionClosed)
```

`open_session` takes a `DuplexSessionConfig` or the same Realtime `session`
object the WebSocket handshake takes; the session id is always allocated by
`DuplexOmni`. `DuplexSessionHandle` offers `submit(command)` plus one
convenience wrapper per command (`append_audio`, `commit`, `create_response`,
`cancel_response`, `barge_in`, `ack_playback`, `update`, `create_item`,
`delete_item`, `truncate_item`, `heartbeat`, ...). `events()` is a
single-consumer async iterator that ends after `session.closed` /
`session.expired`; rejected commands come back as `ErrorEvent`s on it.
Every event renders the wire JSON with `to_realtime()`, so anything written
against the WebSocket protocol works unchanged on the typed stream.

`vllm_omni.clients.inline_duplex.InlineDuplexClient` wraps a `DuplexOmni`
behind the `DuplexClient` API, so the same application code runs in-process
(no server) or over the network:

```python
from vllm_omni.clients.inline_duplex import InlineDuplexClient
from vllm_omni.clients.minicpmo_4_5 import create_duplex_session_config

config = create_duplex_session_config(ref_audio=audio_data_url(Path("reference_voice.wav")))
async with InlineDuplexClient(omni, model="openbmb/MiniCPM-o-4_5", config=config) as client:
    await client.stream_pcm(question)
    await client.commit()
    async for response in client.responses():
        ...
```

`examples/online_serving/barge_in_client.py --inline` drives the barge-in
scenario this way. The inline client has no transport, so it has no
reconnect, resume, heartbeat or event-acknowledgement machinery.

## API Reference

This section is the complete wire contract of `/v1/realtime?duplex=1`: how
the dialect relates to the OpenAI Realtime protocol, which surfaces each
model's capability flags gate, a catalogue of every client-to-server and
server-to-client event, and a JSON example for each. `DuplexClient` speaks
exactly this contract; raw WebSocket clients (any OpenAI Realtime client, or
your own) can use it directly. The ordering and cardinality rules behind it
are specified in the normative contract section of
[Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md); a
compatibility change must update this section together with that table and
its protocol tests.

### Endpoint and transport

The endpoint is `ws(s)://<host>/v1/realtime`. On a deployment whose pipeline
declares a `duplex_plugin` and whose deploy configuration declares
`session_mode: duplex`, a bare connection is already a duplex session, so a
stock Realtime client needs no vendor query parameter. `?duplex=1` (`1`,
`true` or `on`) is the spelling this page uses and stays valid;
`?duplex=0` (`0`, `false`, `off`) explicitly opts out and selects the
turn-based Realtime handler, which a duplex server does not mount --
that connection is answered with `Realtime API is not available`.
`ws(s)://<host>/v1/duplex` is an alias that always selects the duplex
handler. Optional query parameters are `model`, `autostart` (`0` means
resume-only) and `resume`. Every message is one JSON object per WebSocket
text frame, discriminated by `type`. Inbound events are applied in arrival
order through one per-session mailbox; outbound events preserve that order
and carry a monotonically increasing `server_event_seq`, which is the replay
cursor for `session.resume`. Client events may carry an `event_id`, echoed in
`error.error.event_id` when the event is rejected.

Session identity is the server-allocated `session_id` (`duplex-<uuid>`,
announced in `session.created`; a `session_id` the client puts in the session
object is ignored, and ids are never reused); `attachment_generation`
names the current socket; `epoch` is the interruption fence that advances on
every cancel, clear, or barge-in; `turn_id` counts committed model turns;
`response_id` (item id `item_<response_id>`) names one assistant utterance.

**Tier** in the tables below: **1** means identical to OpenAI Realtime in
name and semantics; **2** means an OpenAI event name carrying vLLM-Omni
extensions that a stock OpenAI client ignores; **3** means vLLM-Omni only,
with no OpenAI counterpart.

### Capability negotiation by model

The event vocabulary is uniform, but several surfaces are gated by the
`capabilities` object the server returns in `session.created`; a client must
branch on those flags rather than on the model name. MiniCPM-o 4.5 and
PersonaPlex are on the plugin contract today; the Nemotron VoiceChat column
records what its integration advertises once the follow-up PR ports it:

| Capability | MiniCPM-o 4.5 | PersonaPlex | Nemotron VoiceChat | Gated surface |
| --- | --- | --- | --- | --- |
| `implementation_level` | `model_native_duplex` | `model_native_duplex` | `model_native_duplex` | model-owned `response.listen` / `response.speak` (constant: every duplex model is model-native) |
| `chunk_period_ms` | 1000 | 80 | 80 | the model unit that `response.listen` decisions and camera frames align to |
| `supports_session_resume` | yes | no | yes | `session.resume`, `session.resumed`, `session.replaced`, replay after a transport drop |
| `supports_barge_in` | yes | no | no | `barge_in`, `turn.signal{event:"barge_in"}`, `overlap_policy=barge_in_on_speech`, `turn_detection.server_vad` |
| `supports_audio_truncate` | yes | no | no | `conversation.item.truncate` and truncating `playback.ack` adjusting the stored assistant item |
| video input (`video_frames` on append) | consumed by Stage 0 | ignored | ignored | omni camera track |
| tool calls | no | no | yes | `response.function_call_arguments.*`, `function_call` items |

Everything else in the catalogue — session lifecycle, heartbeat and event
acknowledgement, append/commit/clear, the response envelope, playback
acknowledgement, and the error envelope — behaves identically for every
model. Two PersonaPlex specifics follow from its capabilities rather than from
special-casing: a model with `supports_client_commit=false` auto-responds
without `extra_body.auto_response`, and `response.cancel` /
`output_audio_buffer.clear` restart its conversation context (a new Stage 0
request replays the voice/persona prefill).

### Compatibility with the OpenAI Realtime protocol

The Realtime dialect is, by design, an OpenAI-Realtime-compatible surface:
`RealtimeInputTranslator` / `RealtimeOutputProjector` exist to translate
the native duplex vocabulary onto OpenAI's schema so stock Realtime clients
work unmodified. The overlap falls into the three tiers used by the event catalogue below.

#### Tier 1: identical to OpenAI Realtime

| Direction | Messages |
| --- | --- |
| Client → server | `session.update`, `input_audio_buffer.append` / `.commit` / `.clear`, `conversation.item.create` / `.retrieve` / `.truncate` / `.delete`, `response.create`, `response.cancel`, `output_audio_buffer.clear` (OpenAI: WebRTC-only; here also a WebSocket event) |
| Server → client | `error` (same `{type, code, message, event_id, param}` envelope and the same classes `invalid_request_error` / `server_error` / `rate_limit_error`), `session.created`, `session.updated`, `input_audio_buffer.committed` / `.cleared` / `.speech_started` / `.speech_stopped`, `conversation.item.created` / `.added` / `.done` / `.deleted` / `.retrieved` / `.truncated`, `conversation.item.input_audio_transcription.completed`, `response.created`, `response.done`, `response.output_item.added` / `.done`, `response.content_part.added` / `.done`, `response.output_audio.delta` / `.done`, `response.output_audio_transcript.delta` / `.done`, `response.output_text.delta` / `.done`, `response.function_call_arguments.delta` / `.done`, `rate_limits.updated`, `output_audio_buffer.cleared` |
| Session fields | `model`, `modalities` / `output_modalities`, `instructions`, `voice`, `input_audio_format` / `output_audio_format` (beta spelling) **and** `audio.input.format` / `audio.output.format` objects (GA spelling), `turn_detection.server_vad {threshold, prefix_padding_ms, silence_duration_ms}`, `input_audio_transcription`, `input_audio_noise_reduction`, `tools`, `tool_choice`, `temperature`, `max_response_output_tokens`, `speed`, `tracing` |
| Object shapes | `realtime.session`, `realtime.item`, `realtime.response`; item content parts `input_text` / `input_audio` / `output_text` / `output_audio`; `function_call` and `function_call_output` items; `previous_item_id` chaining; `output_index` / `content_index` addressing |
| Sequencing | `response.created → output_item.added → content_part.added → deltas → *.done → content_part.done → output_item.done → response.done → rate_limits.updated`; `speech_started → speech_stopped → committed` |

The server deliberately emits and accepts **both** the beta and the GA
spellings at once (`response.output_audio.delta` alongside `output_modalities`,
`conversation.item.added` alongside `conversation.item.created`) so either
generation of OpenAI client parses the stream.

#### Tier 2: OpenAI names carrying vLLM-Omni extensions

A stock client ignores the extra keys; the extensions are additive.

| Message | Extension |
| --- | --- |
| `session.created` / `session.updated` | top-level `attachment_generation`, `resume_token`; inside `session`: `state`, `turn_state`, `epoch`, `turn_id`, `active_request_id`, `active_response_id`, `active_response_turn_id`, `overlap_policy`, `overlap_*_ms/rms`, `playback_commit_policy`, `playback`, `capabilities`, `ref_audio`, `extra_body`, `idle_timeout_s`, `response_format` |
| `response.created` / `response.done` / `response.listen` | `response_id` at top level; the raw duplex event under `response.metadata` (`duplex_event`); `status_details.reason` uses vLLM reasons (`barge_in`, `client_cancelled`, `new_response`, …) |
| `response.output_audio.delta` | `format`, `sample_rate_hz`, `metadata{session_id, epoch, model_speak, end_of_turn, audio_duration_ms, audio_text_marks, playback}` |
| `response.speak` (inserted before the first delta) | not an OpenAI event, but rides the OpenAI response envelope (`response_id`, `item_id`, `output_index`, `content_index`) |
| `input_audio_buffer.append` | `is_speech`, `video_frames` (+ `max_slice_nums`), `duration_ms`, `audio_end_ms`, per-event `format` / `sample_rate_hz` |
| `input_audio_buffer.commit` | `final`, `response_create`, `is_speech:false` (silence declaration) |
| `input_audio_buffer.committed` | the native `input.committed` event wrapped under `event` (`turn_id`, `epoch`, `history_len`, `message`, `response_create_deferred`) |
| `conversation.item.*` server events | echo of the originating event under `event` |
| `error` | the error **codes** are vLLM-Omni's (OpenAI standardises only the class); handshake-stage errors use the flat native shape `{type:"error", error:"…", code}` |

Semantic divergences hidden behind shared names:

- **Commit ≠ response.** In the native lane audio streams into the model
  before any commit; a commit may end in `response.listen` and no
  response, and the model may open a response with no commit at all.
  OpenAI: commit ⇒ item, `response.create` ⇒ exactly one response.
- `turn_detection.interrupt_response=false` is rejected;
  `create_response` is ignored; `semantic_vad` is unsupported; the VAD
  runs per session (Silero) and implies `overlap_policy=barge_in_on_speech`.
- `rate_limits.updated` is always an empty list (compatibility only).
- Cancellation never reuses a `response_id`; truncation is driven by
  `playback.ack` as well as `conversation.item.truncate`.

#### Tier 3: vLLM-Omni only

| Area | Messages / fields |
| --- | --- |
| Session lifetime | `session.heartbeat` / `session.heartbeat_ack`, `session.event_ack`, `session.close` / `session.closed`, `session.resume` / `session.resumed` / `session.replaced` / `session.expired` / `session.resync_required`; `server_event_seq`, `resume_token`, `attachment_generation`; query params `?duplex=1`, `autostart`, `resume` |
| Turn-taking | `response.speak`, `response.listen`, `overlap.decision`, `overlap_policy`, `barge_in`, `turn.signal`, `input.cancel`, `input.text.append`, `epoch` / `turn_id` fencing, `force_listen` |
| Playback truth | `playback.ack` / `playback.acknowledged`, `playback_commit_policy` (`ack_only` \| `commit_all_on_done`), `playback{generated_ms, sent_ms, played_ms, committed_ms}`, `history_committed`, `audio_text_marks` |
| Model negotiation | `capabilities{implementation_level, supports_input_append, supports_barge_in, supports_session_resume, chunk_period_ms, input_modes, …}`, `ref_audio`, `extra_body` (`auto_response`, `force_listen_count`, `duplex_initial_user_text`) |
| Diagnostics | `duplex.function_call.done`, the full error-code vocabulary (`REALTIME_ERROR_TYPES_BY_CODE` in `vllm_omni/engine/duplex/events.py`) |

#### Consequences for clients

- Event **names** are roughly 70 % OpenAI's; payloads are OpenAI's plus
  additive extensions; everything that makes the endpoint full-duplex —
  model-owned turn-taking, playback truth, interruption fencing, resumable
  sessions — lives in Tier 3.
- An **unmodified OpenAI Realtime client** can hold a conversation
  (commit-driven or `turn_detection: server_vad`), but it will never see
  `response.listen` / `response.speak`, cannot resume after a transport
  drop, will not receive replay after reconnect, and its server-side
  history is only as honest as `commit_all_on_done` allows because it never
  sends `playback.ack`.
- A **duplex-aware client** (`vllm_omni.clients.duplex.DuplexClient`) uses
  the Tier 3 surface on top of the same Tier 1 events, so both kinds of
  client can share one deployment and one event log.

### Turn-based Chat audio metadata

For turn-based models served through Chat fallback, non-empty audio choices
produced by the Omni Chat audio encoder include `audio_metadata`, in both
streaming and complete Chat responses. Text-only choices do not require it.

| Field | Meaning |
| --- | --- |
| `format` | Encoding of the returned audio, such as `pcm` or `wav`; `pcm16` requests normalize to `pcm`. |
| `sample_rate_hz` | Sample rate used by the encoder, in samples per second. |
| `frame_count` | Samples per channel in this chunk, or in the complete waveform for a non-streaming choice. Not a byte count. |
| `channels` | Number of audio channels. |

These values describe the waveform after audio transformations. Compressed
formats may add codec padding that is not included in `frame_count`.
The encoder uses a model-reported sample rate when available; Qwen3-Omni
currently omits it and uses the existing 24 kHz default. This metadata does
not independently determine a model's sample rate.

Chat fallback requests the session's output format explicitly. Its internal
`response.output_audio.delta` carries `format`, `sample_rate_hz`, `channels`,
and `audio_duration_ms`. The Realtime endpoint exposes output `format` and
`sample_rate_hz` on `response.audio.delta`, with duration in
`metadata.audio_duration_ms`; it does not currently forward `channels`.
The duration is cumulative for the response:
integer frames are summed before converting to milliseconds and rounding down.
Non-empty audio without valid metadata, or a sample-rate change within one
response, fails that fallback response with `response_error`.

In this Chat fallback path, `response.done.response.metadata.playback` reports:

- `generated_ms`: cumulative source-waveform duration.
- `sent_ms`: cumulative audio accepted into the server output path, including
  queued or journaled audio. A journaled event remains accepted if the
  subsequent send fails; this is not proof of socket delivery.
- `played_ms`: cumulative playback reported by the client.
- `committed_ms`: playback position used for history reconciliation.

The native model path still records audio before sending; it does not yet
share Chat fallback's output-acceptance timing. Only client playback reports
establish what was played. The existing duration-proportional text truncation
remains an approximation, not word-level audio alignment.

### Event catalogue

Every event type the Realtime route accepts or emits, with a worked example in the next section: 21
client→server and 42 server→client. Aliases (`input_text.append`,
`push_text`, `signal_turn`, `audio.playback_ack`, `close_session`, `close`)
share the payload of their canonical event and are not listed
separately; the alias table is `translate_realtime_command` in
`vllm_omni/engine/duplex/realtime_commands.py`. The pre-Realtime WAV-append
aliases `push_chunk` and `input.audio.append` are **not** accepted on this
route: send `input_audio_buffer.append` with one of the supported input
formats (`pcm16`, `pcm_s16le`, `s16le`, `pcm_f32le`, `g711_ulaw`,
`g711_alaw`).

#### Client to server

| Event type | Tier | Description |
| --- | --- | --- |
| `session.update` | 2 | Open handshake on first send (creates the session); later sends reconfigure it. OpenAI name; `session` object carries vLLM-Omni keys (`ref_audio`, `overlap_policy`, `playback_commit_policy`, `extra_body`, ...). |
| `session.heartbeat` | 3 | Keepalive; refreshes the engine session lease. |
| `session.event_ack` | 3 | Acknowledges received server events by `server_event_seq`; trims the resume replay journal. |
| `session.close` | 3 | Graceful close; answered with `session.closed`. |
| `session.resume` | 3 | Re-attach a live session on a new socket (`session_id`, `resume_token`, `last_received_server_event_seq`). |
| `input_audio_buffer.append` | 2 | Append one audio chunk. OpenAI name; extended with `is_speech`, `video_frames`, `duration_ms`, `audio_end_ms`, per-event `format`/`sample_rate_hz`. |
| `input_audio_buffer.commit` | 2 | Seal the buffered utterance into a user item. OpenAI name; extended with `final`, `response_create`, `is_speech:false`. |
| `input.commit` | 3 | Alias of `input_audio_buffer.commit` accepted by the runner. |
| `input_audio_buffer.clear` | 1 | Drop un-committed input audio. |
| `input.cancel` | 3 | Cancel pending input; advances `epoch`. |
| `input.text.append` | 3 | Append text input; model-native sessions reject it with `native_text_append_unsupported`. |
| `conversation.item.create` | 1 | Add a completed user item (text, audio, or `function_call_output`) to history. |
| `conversation.item.retrieve` | 1 | Fetch a stored item by id. |
| `conversation.item.delete` | 1 | Delete a stored item by id. |
| `conversation.item.truncate` | 1 | Truncate an assistant item's audio/transcript at `audio_end_ms`. |
| `response.create` | 1 | Explicitly request a response with per-response overrides. Requires committed audio input; otherwise `response_create_without_input`. |
| `response.cancel` | 1 | Cancel the active (or named) response; advances `epoch`. |
| `output_audio_buffer.clear` | 1 | Discard queued output audio; advances `epoch` (OpenAI: WebRTC-only, here also WebSocket). |
| `barge_in` | 3 | Explicit hard interrupt; requires `capabilities.supports_barge_in`. |
| `turn.signal` | 3 | External turn-taking signal (e.g. `event:"barge_in"`); same capability gate. |
| `playback.ack` | 3 | Report cumulative playback progress (`played_ms`, `committed_ms`, optional `truncate`). |

#### Server to client

| Event type | Tier | Description |
| --- | --- | --- |
| `session.created` | 2 | Session opened. OpenAI name; adds `attachment_generation`, `resume_token` and vLLM-Omni keys inside `session` (`id` is the server-allocated session id, `epoch`, `turn_id`, `playback`, `capabilities`, ...). |
| `session.updated` | 2 | Echo of the effective session config after every `session.update` (same extended `session` object). |
| `session.heartbeat_ack` | 3 | Reply to `session.heartbeat`. |
| `session.closed` | 3 | Last event on the socket, emitted once the engine released the session; `reason` ∈ `client_close`, `disconnect`, `timeout`, `transport_error`, `shutdown`, ... |
| `session.resumed` | 3 | Resume accepted; carries the new `attachment_generation` and rotated `resume_token`; journaled events are replayed after it. |
| `session.replaced` | 3 | Sent to the superseded socket when another socket resumes the session. |
| `session.expired` | 3 | Engine lease reaped (`disconnect_grace_expired`, `idle_ttl_expired`); socket closes after it. |
| `session.resync_required` | 3 | Replay impossible (`journal_gap` / `journal_overflow`); client must start a new session. |
| `input_audio_buffer.speech_started` | 1 | Speech onset detected (server VAD, client hint, or RMS heuristic). |
| `input_audio_buffer.speech_stopped` | 1 | Speech end detected. |
| `input_audio_buffer.committed` | 2 | Commit accepted. OpenAI name; wraps the native `input.committed` payload under `event` (`turn_id`, `epoch`, `response_create_deferred`, ...). |
| `input_audio_buffer.cleared` | 1 | Input buffer cleared (after `input_audio_buffer.clear` or `input.cancel`). |
| `conversation.item.added` | 2 | Item entered history (GA spelling). OpenAI name; carries `previous_item_id` and echoes the originating event. |
| `conversation.item.created` | 2 | Same as `conversation.item.added` in the beta spelling; both are emitted. |
| `conversation.item.done` | 2 | Item finalized (GA spelling). |
| `conversation.item.input_audio_transcription.completed` | 1 | Transcript of a committed user audio item (model-produced). |
| `conversation.item.deleted` | 2 | Reply to `conversation.item.delete` (echoes the request under `event`). |
| `conversation.item.retrieved` | 1 | Reply to `conversation.item.retrieve`. |
| `conversation.item.truncated` | 2 | Reply to `conversation.item.truncate` (echoes the request under `event`). |
| `response.created` | 2 | A visible assistant response starts. OpenAI name; top-level `response_id`, raw duplex event under `response.metadata`. |
| `response.output_item.added` | 1 | Assistant item (or `function_call` item) attached to the response. |
| `response.content_part.added` | 1 | Audio/text content part opened on the assistant item. |
| `response.speak` | 3 | Model chose to speak (native lane); at most once per response, before the first audio delta. |
| `response.output_audio.delta` | 2 | One ordered audio chunk. OpenAI name; adds `format`, `sample_rate_hz`, `metadata{epoch, model_speak, end_of_turn, audio_text_marks, playback}`. |
| `response.output_audio_transcript.delta` | 1 | Transcript text paired one-to-one with each audio delta. |
| `response.output_text.delta` | 1 | Text delta for text-modality responses. |
| `response.output_text.done` | 1 | Final text of a text-modality response. |
| `response.output_audio.done` | 1 | Audio stream closed for the response. |
| `response.output_audio_transcript.done` | 1 | Full transcript (concatenation of the deltas). |
| `response.content_part.done` | 1 | Content part finalized with its transcript/text. |
| `response.output_item.done` | 1 | Assistant item finalized. |
| `response.done` | 2 | Terminal event; `status` ∈ `completed` \| `cancelled` \| `failed`. OpenAI name; vLLM-Omni `status_details.reason` values and raw duplex event under `metadata`. |
| `rate_limits.updated` | 1 | Compatibility event after every `response.done`; always an empty list. |
| `response.listen` | 3 | Model chose to keep listening at a unit boundary or after a silence commit. Carries `response_id` when the decision terminates a precreated response (a `response.done` with the same id follows); otherwise no `response.created` precedes it. |
| `response.function_call_arguments.delta` | 1 | Tool-call arguments delta (Nemotron VoiceChat). |
| `response.function_call_arguments.done` | 1 | Tool-call arguments complete. |
| `duplex.function_call.done` | 3 | Raw fallback when a model tool call is malformed. |
| `output_audio_buffer.cleared` | 1 | Reply to `output_audio_buffer.clear`. |
| `playback.acknowledged` | 3 | Reply to `playback.ack`; returns the playback ledger and `history_committed`. |
| `overlap.decision` | 3 | How an append that overlapped an active response was handled (`drop` \| `listen` \| `barge_in`); details under `metadata`. |
| `turn.event` | 3 | A turn-taking signal the session accepted (`event`, resulting `turn_state`). |
| `error` | 2 | Error envelope. OpenAI shape and classes; vLLM-Omni error codes. |

### Message examples

Field values are illustrative; base64 payloads are shortened to `"…"`.
`C→S` = client sends, `S→C` = server sends. Every `S→C` event on the wire
additionally carries `"server_event_seq": <int>` (omitted below for brevity).

#### Session lifecycle

`C→S session.update` (first one = open handshake)

```json
{
  "type": "session.update",
  "event_id": "evt_0001",
  "session": {
    "model": "openbmb/MiniCPM-o-4_5",
    "modalities": ["audio", "text"],
    "instructions": "You are a concise voice assistant.",
    "input_audio_format": "pcm16",
    "output_audio_format": "pcm16",
    "ref_audio": "data:audio/wav;base64,…",
    "turn_detection": null,
    "overlap_policy": "listen_only",
    "playback_commit_policy": "ack_only",
    "temperature": 0.7,
    "idle_timeout_s": 300,
    "extra_body": {
      "auto_response": true,
      "force_listen_count": 0
    }
  }
}
```

`S→C session.created`

```json
{
  "type": "session.created",
  "attachment_generation": 1,
  "resume_token": "rt_3f9c…",
  "session": {
    "object": "realtime.session",
    "type": "realtime",
    "id": "duplex-5f2a…",
    "model": "openbmb/MiniCPM-o-4_5",
    "state": "open",
    "turn_state": "idle",
    "epoch": 0,
    "turn_id": 0,
    "active_request_id": null,
    "active_response_id": null,
    "active_response_turn_id": null,
    "modalities": ["audio", "text"],
    "output_modalities": ["audio", "text"],
    "instructions": "You are a concise voice assistant.",
    "voice": null,
    "response_format": "pcm16",
    "input_audio_format": "pcm16",
    "output_audio_format": "pcm16",
    "audio": {
      "input": {"format": {"type": "audio/pcm", "rate": 16000}, "sample_rate_hz": 16000},
      "output": {"format": {"type": "audio/pcm", "rate": 24000}}
    },
    "temperature": 0.7,
    "max_tokens": null,
    "speed": null,
    "idle_timeout_s": 300.0,
    "overlap_policy": "listen_only",
    "overlap_short_ack_ms": 700,
    "overlap_barge_in_ms": 1200,
    "overlap_silence_rms": 0.003,
    "playback_commit_policy": "ack_only",
    "turn_detection": null,
    "input_audio_transcription": null,
    "tracing": null,
    "playback": {"generated_ms": 0, "sent_ms": 0, "played_ms": 0, "committed_ms": 0},
    "capabilities": {
      "implementation_level": "model_native_duplex",
      "supports_input_append": true,
      "supports_barge_in": true,
      "supports_session_resume": true,
      "supports_session_lease": true,
      "supports_multi_session": true,
      "chunk_period_ms": 1000,
      "input_modes": ["append_audio_chunk"],
      "signal_sources": ["model_native", "client_event", "server_policy"],
      "target_barge_in_latency_ms": null
    }
  }
}
```

`S→C session.updated` (same `session` object; follows the first
`session.created` and every later `session.update`)

```json
{"type": "session.updated", "session": {"object": "realtime.session", "id": "duplex-5f2a…", "…": "…"}}
```

`C→S session.heartbeat` / `S→C session.heartbeat_ack`

```json
{"type": "session.heartbeat"}
```

```json
{"type": "session.heartbeat_ack", "session_id": "duplex-5f2a…"}
```

`C→S session.event_ack`

```json
{"type": "session.event_ack", "server_event_seq": 48}
```

`C→S session.close` / `S→C session.closed`

```json
{"type": "session.close"}
```

```json
{"type": "session.closed", "session_id": "duplex-5f2a…", "epoch": 3, "reason": "client_close", "event": {"type": "session.closed", "reason": "client_close"}}
```

`C→S session.resume` (new socket, `?duplex=1&autostart=0`)

```json
{
  "type": "session.resume",
  "session_id": "duplex-5f2a…",
  "resume_token": "rt_3f9c…",
  "last_received_server_event_seq": 40
}
```

`S→C session.resumed` (activation; the token is rotated, then journaled
events with `server_event_seq > 40` are replayed in order)

```json
{
  "type": "session.resumed",
  "session_id": "duplex-5f2a…",
  "session": {"id": "duplex-5f2a…", "state": "open", "epoch": 3, "turn_id": 2, "...": "the engine's current session object"},
  "attachment_generation": 2,
  "resume_token": "rt_a71e…"
}
```

`S→C session.replaced` (sent to the superseded socket, which is then closed)

```json
{"type": "session.replaced", "session_id": "duplex-5f2a…", "attachment_generation": 1}
```

`S→C session.expired`

```json
{"type": "session.expired", "session_id": "duplex-5f2a…", "reason": "disconnect_grace_expired"}
```

`S→C session.resync_required`

```json
{"type": "session.resync_required", "session_id": "duplex-5f2a…", "reason": "journal_gap"}
```

#### Input

`C→S input_audio_buffer.append` (200 ms of 16 kHz PCM16 = 6400 bytes)

```json
{
  "type": "input_audio_buffer.append",
  "event_id": "evt_0042",
  "audio": "…",
  "input_audio_format": "pcm16",
  "sample_rate_hz": 16000,
  "duration_ms": 200,
  "audio_end_ms": 1200,
  "is_speech": true
}
```

with camera frames (omni video; rides the append that closes a 1 s model
unit; each entry is a **bare** base64 JPEG/PNG — no `data:` URL prefix).
Only a model whose Stage 0 interleaves a frame track at unit boundaries takes
this field; a turn model has no such boundary and rejects it, expecting
`conversation.item.create` with `input_image` content instead.

```json
{
  "type": "input_audio_buffer.append",
  "audio": "…",
  "input_audio_format": "pcm16",
  "sample_rate_hz": 16000,
  "duration_ms": 200,
  "audio_end_ms": 1200,
  "video_frames": ["/9j/4AAQSkZJRg…", "/9j/4AAQSkZJRg…"]
}
```

`S→C input_audio_buffer.speech_started` / `speech_stopped`

```json
{"type": "input_audio_buffer.speech_started", "audio_start_ms": 1040, "item_id": "item_9b2d…"}
```

```json
{"type": "input_audio_buffer.speech_stopped", "audio_end_ms": 2860, "item_id": "item_9b2d…"}
```

`C→S input_audio_buffer.commit`

```json
{"type": "input_audio_buffer.commit", "event_id": "evt_0060", "final": true}
```

alias form accepted by the runner (`input.commit`)

```json
{"type": "input.commit", "final": true}
```

silence/noise declaration (no runtime append; answered with `response.listen`)

```json
{"type": "input_audio_buffer.commit", "final": true, "is_speech": false}
```

`S→C` commit fan-out (in this order)

```json
{
  "type": "conversation.item.added",
  "previous_item_id": null,
  "item": {
    "id": "item_9b2d…",
    "object": "realtime.item",
    "type": "message",
    "role": "user",
    "status": "completed",
    "content": [{"type": "input_audio", "transcript": "what is the weather today"}]
  }
}
```

```json
{"type": "conversation.item.created", "previous_item_id": null, "item": {"id": "item_9b2d…", "…": "…"}}
```

```json
{
  "type": "input_audio_buffer.committed",
  "previous_item_id": null,
  "item_id": "item_9b2d…",
  "event": {
    "type": "input.committed",
    "session_id": "duplex-5f2a…",
    "turn_id": 1,
    "epoch": 0,
    "history_len": 1,
    "message": {"role": "user", "content": [{"type": "audio_url", "audio_url": {"url": "native-duplex:input-audio"}}]},
    "realtime_item_id": "item_9b2d…"
  }
}
```

```json
{
  "type": "conversation.item.input_audio_transcription.completed",
  "item_id": "item_9b2d…",
  "content_index": 0,
  "transcript": "what is the weather today"
}
```

```json
{"type": "conversation.item.done", "previous_item_id": null, "item": {"id": "item_9b2d…", "status": "completed", "…": "…"}}
```

deferred commit during an active response (`event.response_create_deferred`)

```json
{
  "type": "input_audio_buffer.committed",
  "previous_item_id": "item_9b2d…",
  "item_id": "item_c044…",
  "event": {"type": "input.committed", "session_id": "duplex-5f2a…", "turn_id": 2, "epoch": 0, "history_len": 3, "response_create_deferred": true, "message": {"…": "…"}}
}
```

`C→S input_audio_buffer.clear` / `S→C input_audio_buffer.cleared`

```json
{"type": "input_audio_buffer.clear"}
```

```json
{"type": "input_audio_buffer.cleared"}
```

`C→S input.cancel` (advances `epoch`; also answered with `input_audio_buffer.cleared`)

```json
{"type": "input.cancel"}
```

`C→S input.text.append`

```json
{"type": "input.text.append", "text": "and tomorrow?"}
```

`C→S conversation.item.create` (text turn, or a tool result)

```json
{
  "type": "conversation.item.create",
  "previous_item_id": null,
  "item": {
    "type": "message",
    "role": "user",
    "content": [{"type": "input_text", "text": "Summarise what I said."}]
  }
}
```

```json
{
  "type": "conversation.item.create",
  "item": {"type": "function_call_output", "call_id": "call_7d1f", "output": "{\"temperature_c\": 24}"}
}
```

`C→S conversation.item.delete` / `S→C conversation.item.deleted`

```json
{"type": "conversation.item.delete", "item_id": "item_9b2d…"}
```

```json
{"type": "conversation.item.deleted", "item_id": "item_9b2d…", "event": {"type": "conversation.item.deleted", "item_id": "item_9b2d…"}}
```

`C→S conversation.item.retrieve` / `S→C conversation.item.retrieved`

```json
{"type": "conversation.item.retrieve", "item_id": "item_resp_01"}
```

```json
{"type": "conversation.item.retrieved", "item": {"id": "item_resp_01", "object": "realtime.item", "type": "message", "role": "assistant", "status": "completed", "content": [{"type": "output_audio", "transcript": "It is sunny, 24 degrees."}]}}
```

`C→S conversation.item.truncate` / `S→C conversation.item.truncated`

```json
{"type": "conversation.item.truncate", "item_id": "item_resp_01", "content_index": 0, "audio_end_ms": 1850}
```

```json
{"type": "conversation.item.truncated", "item_id": "item_resp_01", "content_index": 0, "audio_end_ms": 1850, "event": {"…": "…"}}
```

After this acknowledgement, `conversation.item.retrieve` returns the truncated
transcript. Without text/audio alignment marks, its prefix is estimated from
the requested position and audio duration; this is not exact word alignment.

#### Response lifecycle

`C→S response.create`

```json
{
  "type": "response.create",
  "event_id": "evt_0070",
  "response": {
    "modalities": ["audio", "text"],
    "instructions": "Answer in one sentence.",
    "output_audio_format": "pcm16",
    "temperature": 0.6,
    "max_output_tokens": 256,
    "metadata": {"trace": "turn-3"}
  }
}
```

`S→C response.created`

```json
{
  "type": "response.created",
  "response_id": "resp_01",
  "response": {
    "id": "resp_01",
    "object": "realtime.response",
    "status": "in_progress",
    "status_details": null,
    "output": [],
    "modalities": ["audio", "text"],
    "metadata": {"duplex_event": {"type": "response.created", "session_id": "duplex-5f2a…", "response_id": "resp_01", "epoch": 0, "turn_id": 1}}
  }
}
```

`S→C conversation.item.added` / `conversation.item.created` (assistant item, `status: in_progress`)

```json
{
  "type": "conversation.item.added",
  "previous_item_id": "item_9b2d…",
  "item": {"id": "item_resp_01", "object": "realtime.item", "type": "message", "role": "assistant", "status": "in_progress", "content": []}
}
```

`S→C response.output_item.added`

```json
{"type": "response.output_item.added", "response_id": "resp_01", "output_index": 0, "item": {"id": "item_resp_01", "object": "realtime.item", "type": "message", "role": "assistant", "status": "in_progress", "content": []}}
```

`S→C response.content_part.added`

```json
{"type": "response.content_part.added", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "part": {"type": "audio", "transcript": ""}}
```

`S→C response.speak`

```json
{
  "type": "response.speak",
  "response_id": "resp_01",
  "item_id": "item_resp_01",
  "output_index": 0,
  "content_index": 0,
  "metadata": {"session_id": "duplex-5f2a…", "epoch": 0, "model_speak": true}
}
```

`S→C response.output_audio.delta` (one ~1 s unit, 24 kHz PCM16)

```json
{
  "type": "response.output_audio.delta",
  "response_id": "resp_01",
  "item_id": "item_resp_01",
  "output_index": 0,
  "content_index": 0,
  "delta": "…",
  "format": "pcm16",
  "sample_rate_hz": 24000,
  "metadata": {
    "session_id": "duplex-5f2a…",
    "epoch": 0,
    "model_speak": true,
    "end_of_turn": false,
    "audio_duration_ms": 1000,
    "audio_text_marks": [{"text_chars": 11, "audio_end_ms": 1000}],
    "playback": {"generated_ms": 1000, "sent_ms": 1000, "played_ms": 0, "committed_ms": 0}
  }
}
```

`S→C response.output_audio_transcript.delta` (exactly one per audio delta)

```json
{"type": "response.output_audio_transcript.delta", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "delta": "It is sunny"}
```

`S→C response.output_text.delta` / `response.output_text.done` (text modality)

```json
{"type": "response.output_text.delta", "response_id": "resp_02", "item_id": "item_resp_02", "output_index": 0, "content_index": 0, "delta": "Tomorrow looks"}
```

```json
{"type": "response.output_text.done", "response_id": "resp_02", "item_id": "item_resp_02", "output_index": 0, "content_index": 0, "text": "Tomorrow looks cloudy."}
```

`S→C response.output_audio.done` / `response.output_audio_transcript.done`

```json
{"type": "response.output_audio.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0}
```

```json
{"type": "response.output_audio_transcript.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "transcript": "It is sunny, 24 degrees."}
```

`S→C response.content_part.done`

```json
{"type": "response.content_part.done", "response_id": "resp_01", "item_id": "item_resp_01", "output_index": 0, "content_index": 0, "part": {"type": "audio", "transcript": "It is sunny, 24 degrees."}}
```

`S→C response.output_item.done` / `conversation.item.done`

```json
{"type": "response.output_item.done", "response_id": "resp_01", "output_index": 0, "item": {"id": "item_resp_01", "object": "realtime.item", "type": "message", "role": "assistant", "status": "completed", "content": [{"type": "output_audio", "transcript": "It is sunny, 24 degrees."}]}}
```

```json
{"type": "conversation.item.done", "previous_item_id": "item_9b2d…", "item": {"id": "item_resp_01", "status": "completed", "…": "…"}}
```

`S→C response.done` (completed)

```json
{
  "type": "response.done",
  "response_id": "resp_01",
  "response": {
    "id": "resp_01",
    "object": "realtime.response",
    "status": "completed",
    "status_details": null,
    "output": [{"id": "item_resp_01", "object": "realtime.item", "type": "message", "role": "assistant", "status": "completed", "content": [{"type": "output_audio", "transcript": "It is sunny, 24 degrees."}]}],
    "metadata": {"type": "response.done", "session_id": "duplex-5f2a…", "response_id": "resp_01", "epoch": 0, "committed": true, "playback": {"generated_ms": 2400, "sent_ms": 2400, "played_ms": 2400, "committed_ms": 2400}}
  }
}
```

`S→C response.done` (cancelled by barge-in)

```json
{
  "type": "response.done",
  "response_id": "resp_01",
  "response": {"id": "resp_01", "object": "realtime.response", "status": "cancelled", "status_details": {"type": "cancelled", "reason": "barge_in"}, "output": [{"id": "item_resp_01", "status": "cancelled", "…": "…"}], "metadata": {"…": "…"}}
}
```

`S→C rate_limits.updated`

```json
{"type": "rate_limits.updated", "rate_limits": []}
```

`S→C response.listen`

```json
{
  "type": "response.listen",
  "session_id": "duplex-5f2a…",
  "epoch": 0,
  "response": {"object": "realtime.response", "status": "listening", "metadata": {"type": "response.listen", "session_id": "duplex-5f2a…", "epoch": 0, "reason": "model_listen"}}
}
```

`C→S response.cancel`

```json
{"type": "response.cancel", "response_id": "resp_01"}
```

`C→S output_audio_buffer.clear` / `S→C output_audio_buffer.cleared`

```json
{"type": "output_audio_buffer.clear"}
```

```json
{"type": "output_audio_buffer.cleared", "response_id": "resp_01"}
```

`C→S barge_in` / `turn.signal`

```json
{"type": "barge_in"}
```

```json
{"type": "turn.signal", "event": "barge_in", "payload": {"source": "client_vad"}}
```

function-call response (Nemotron VoiceChat tools)

```json
{"type": "response.output_item.added", "response_id": "resp_fc_01", "output_index": 0, "item": {"id": "item_fc_01", "object": "realtime.item", "type": "function_call", "status": "completed", "name": "get_weather", "call_id": "call_7d1f", "arguments": "{\"city\":\"Hong Kong\"}"}}
```

```json
{"type": "response.function_call_arguments.delta", "response_id": "resp_fc_01", "item_id": "item_fc_01", "output_index": 0, "call_id": "call_7d1f", "delta": "{\"city\":\"Hong Kong\"}"}
```

```json
{"type": "response.function_call_arguments.done", "response_id": "resp_fc_01", "item_id": "item_fc_01", "output_index": 0, "call_id": "call_7d1f", "arguments": "{\"city\":\"Hong Kong\"}"}
```

raw fallback when a model tool call is malformed (`call_id`/`name` missing)

```json
{"type": "duplex.function_call.done", "event": {"type": "function_call.done", "session_id": "duplex-5f2a…", "epoch": 0, "call_id": null, "name": null, "arguments": "{"}}
```

#### Playback

`C→S playback.ack` / `S→C playback.acknowledged`

```json
{"type": "playback.ack", "response_id": "resp_01", "item_id": "item_resp_01", "played_ms": 1000, "committed_ms": 1000}
```

```json
{
  "type": "playback.acknowledged",
  "event": {
    "type": "playback.acknowledged",
    "session_id": "duplex-5f2a…",
    "epoch": 0,
    "item_id": "item_resp_01",
    "played_ms": 1000,
    "committed_ms": 1000,
    "truncate": false,
    "playback": {"generated_ms": 2400, "sent_ms": 2400, "played_ms": 1000, "committed_ms": 1000},
    "history_committed": false
  }
}
```

truncating ack after an interruption (what was actually heard)

```json
{"type": "playback.ack", "response_id": "resp_01", "played_ms": 1850, "committed_ms": 1850, "truncate": true}
```

#### Overlap and barge-in

`S→C overlap.decision`

```json
{
  "type": "overlap.decision",
  "session_id": "duplex-5f2a…",
  "epoch": 0,
  "policy": "listen_only",
  "action": "listen",
  "reason": "long_overlap_speech",
  "metadata": {"overlap_ms": 1400, "buffer_audio": true, "defer_runtime_append": true, "force_listen": true}
}
```

```json
{"type": "overlap.decision", "session_id": "duplex-5f2a…", "epoch": 0, "policy": "barge_in_on_speech", "action": "barge_in", "reason": "server_vad_utterance_active"}
```

server-VAD session (`session.update` fragment) and the annotated append the
translator forwards internally (shown for reference; clients never see it)

```json
{"type": "session.update", "session": {"turn_detection": {"type": "server_vad", "threshold": 0.5, "prefix_padding_ms": 300, "silence_duration_ms": 500}, "overlap_policy": "barge_in_on_speech"}}
```

```json
{"type": "input_audio_buffer.append", "audio": "…", "format": "pcm16", "sample_rate_hz": 16000, "is_speech": true, "force_listen": true, "vad": {"backend": "silero", "is_speech": true, "speech_active": true, "speech_started": true, "speech_stopped": false, "speech_probability": 0.93}}
```

#### Errors

`S→C error` — Realtime shape

```json
{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "code": "input_audio_buffer_empty",
    "message": "input_audio_buffer.commit requires a non-empty input audio buffer.",
    "event_id": "evt_0060"
  }
}
```

```json
{"type": "error", "error": {"type": "invalid_request_error", "code": "unsupported_turn_detection", "message": "turn_detection.interrupt_response=false is unsupported; use turn_detection=null for model-owned listen/speak", "event_id": "evt_0001", "param": "turn_detection"}}
```

```json
{"type": "error", "error": {"type": "server_error", "code": "runtime_append_failed", "message": "stale epoch: expected 1, got 0"}}
```

```json
{"type": "error", "error": {"type": "rate_limit_error", "code": "resource_exhausted", "message": "duplex_session_capacity_exhausted: limit=4"}}
```

Handshake-stage errors (before the Realtime projector is bound) use the
flat native shape:

```json
{"type": "error", "error": "Unknown or expired duplex session: duplex-5f2a…", "code": "session_resume_expired"}
```

#### Coverage

The examples above cover every event type the Realtime route emits or
accepts (verified by diffing the `type` literals in
`vllm_omni/engine/duplex/events.py` and `vllm_omni/engine/duplex/commands.py`
against this document). Not shown, because they never reach a client
unprojected: the session-internal events the runner produces
(`input.committed`, `input.cancelled`, `audio.cancelled`,
`response.output_audio.delta`, `response.text.delta`, `function_call.done`)
— their Realtime projections are the examples above (see the name map at the
end of this section). Every public event is a typed `DuplexEvent` whose
`to_realtime()` is the wire object; every client event is parsed into a
typed `DuplexCommand` by `command_from_realtime()`.

#### Internal to Realtime name map

The session runner emits these session-internal events; the Realtime
projector (`vllm_omni/engine/duplex/realtime_events.py`) renames and fans
them out into typed events before they reach a client.

| Internal event | Realtime projection |
| --- | --- |
| `input.committed` | `conversation.item.added` / `conversation.item.created`, `input_audio_buffer.committed`, `conversation.item.input_audio_transcription.completed`, `conversation.item.done` |
| `input.cancelled` | `input_audio_buffer.cleared` |
| `response.output_audio.delta` (with `audio_transcript`) | optional `response.speak`, then `response.output_audio.delta` and `response.output_audio_transcript.delta` |
| `response.output_audio.done` / `response.output_text.done` | `response.output_audio.done` / `response.output_text.done` |
| `response.text.delta` | `response.output_text.delta` |
| `audio.cancelled` | optional `output_audio_buffer.cleared`, then the cancelled terminal events and `response.done` with `status: "cancelled"` |
| `function_call.done` | `response.created`, `response.output_item.added` (`function_call` item), `response.function_call_arguments.delta` / `.done`, `response.output_item.done`, `response.done` |
| `playback.acknowledged` | `playback.acknowledged` (wrapped verbatim under `event`) |
| `session.resumed`, `session.replaced`, `session.expired`, `session.resync_required`, `session.heartbeat_ack` | passed through unchanged |

## Known Limitations

- Several surfaces are capability-gated per model (see *Capability
  negotiation by model* above): PersonaPlex does not support session resume,
  barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
  or audio truncation; camera frames are consumed only by MiniCPM-o 4.5; tool
  calls are produced only by Nemotron VoiceChat.
- A response is generated from committed **audio**, so the OpenAI text-prompt
  shape does not drive one: `conversation.item.create` with an `input_text`
  part adds the item to history, but a following `response.create` with no
  committed audio is rejected with `response_create_without_input`, and
  `input.text.append` is rejected with `native_text_append_unsupported`.
  Text-to-speech has its own shape here — put the text in
  `extra_body.duplex_initial_user_text` on `session.update`, then stream audio
  units (silence is enough). The seeded turn is what the model answers.
- `turn_detection` supports only `server_vad` with `interrupt_response=true`;
  `semantic_vad`, `interrupt_response=false`, and `create_response` are not
  supported.
- `input_audio_transcription` and `input_audio_noise_reduction` are accepted
  and echoed but no separate transcription or noise-reduction stage runs;
  transcripts come from the model.
- `rate_limits.updated` is emitted for compatibility only and always carries
  an empty list.
- Session capacity is bounded by `duplex_session.max_sessions` in the deploy
  configuration; admission beyond it fails with `resource_exhausted`
  (`rate_limit_error`, retryable).
- A duplex deployment serves the websocket route and, when supported by the
  model, ordinary `/v1/chat/completions` requests on the same engine without a
  session per request (see [Start the Server](#start-the-server)). Other
  turn-based HTTP routes report "not available" in duplex mode. To use the
  ordinary online serving stack, select `session_mode: turn` as described in
  [Full Duplex](full_duplex_api.md#enable-full-duplex).