vllm-omni/docs/serving/realtime_api.md

Realtime Audio WebSocket API

Use WS /v1/realtime to stream PCM audio into a compatible omni model and receive incremental transcription and synthesized response audio. This is a turn-based streaming API: it starts one generation over an audio stream and finishes after the client closes that input stream.

For a model that listens while it is already speaking, use the experimental Full-Duplex API instead.

Quick Start

Start a compatible model. The currently documented example uses Qwen3-Omni:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

Run the provided client with a mono, 16-bit PCM, 16 kHz WAV file:

python examples/online_serving/qwen3_omni/openai_realtime_client.py \
  --url ws://localhost:8091/v1/realtime \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --input-wav input_16k_mono.wav \
  --output-wav response.wav \
  --output-text response.txt

The client dependency is websockets.

Protocol

Messages in both directions are JSON text frames. Audio carried inside an event is base64-encoded raw PCM16.

Direction	Event	Purpose
Server to client	session.created	Confirms that the WebSocket connection is ready
Client to server	session.update	Selects and validates the served model
Client to server	input_audio_buffer.commit with final: false	Starts generation over the incoming stream
Client to server	input_audio_buffer.append	Appends base64 PCM16 audio
Client to server	input_audio_buffer.commit with final: true	Marks the end of input
Server to client	transcription.delta	Carries incremental response text
Server to client	transcription.done	Carries final text and token usage
Server to client	response.audio.delta	Carries incremental PCM16 response audio
Server to client	response.audio.done	Marks the end of response audio
Server to client	error	Reports an invalid event, model, or audio payload

A minimal client sends events in this order:

{"type":"session.update","model":"Qwen/Qwen3-Omni-30B-A3B-Instruct"}
{"type":"input_audio_buffer.commit","final":false}
{"type":"input_audio_buffer.append","audio":"<base64-pcm16>"}
{"type":"input_audio_buffer.commit","final":true}

The initial non-final commit intentionally starts generation before all audio has arrived. Continue sending input_audio_buffer.append events while the engine is consuming the stream.

Audio Handling

• Input is mono PCM16 at 16 kHz for the Qwen3-Omni example.
• response.audio.delta.audio contains base64-encoded PCM16 bytes.
• Read sample_rate_hz from each audio event instead of assuming an output rate. Qwen3-Omni output is typically 24 kHz.
• Concatenate audio deltas in receive order to construct the output waveform.
• Set input_audio_buffer.commit.final to true after the last input chunk so the server can terminate the streaming request.

Availability and Limitations

The path is registered on the unified API server, but it is usable only when the loaded pipeline implements realtime audio input and produces compatible audio output. Unsupported deployments return an error event. The endpoint does not provide duplex session resume, playback acknowledgement, overlap policy, or barge-in controls.

See the Qwen3-Omni online serving example for concurrency options, chunk pacing, and per-delta audio debugging.