Nick CaoandCodex
d3493384cf
Signed-off-by: Nick Cao <ncao@redhat.com> Co-authored-by: Codex <noreply@openai.com>
83 lines
3.5 KiB
Markdown
83 lines
3.5 KiB
Markdown
# Realtime Audio WebSocket API
|
|
|
|
Use `WS /v1/realtime` to stream PCM audio into a compatible omni model and
|
|
receive incremental transcription and synthesized response audio. This is a
|
|
turn-based streaming API: it starts one generation over an audio stream and
|
|
finishes after the client closes that input stream.
|
|
|
|
For a model that listens while it is already speaking, use the
|
|
[Full-Duplex API](full_duplex_api.md) instead.
|
|
|
|
## Quick Start
|
|
|
|
Start a compatible model. The currently documented example uses Qwen3-Omni:
|
|
|
|
```bash
|
|
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
|
```
|
|
|
|
Run the provided client with a mono, 16-bit PCM, 16 kHz WAV file:
|
|
|
|
```bash
|
|
python examples/online_serving/qwen3_omni/openai_realtime_client.py \
|
|
--url ws://localhost:8091/v1/realtime \
|
|
--model Qwen/Qwen3-Omni-30B-A3B-Instruct \
|
|
--input-wav input_16k_mono.wav \
|
|
--output-wav response.wav \
|
|
--output-text response.txt
|
|
```
|
|
|
|
The client dependency is `websockets`.
|
|
|
|
## Protocol
|
|
|
|
Messages in both directions are JSON text frames. Audio carried inside an
|
|
event is base64-encoded raw PCM16.
|
|
|
|
| Direction | Event | Purpose |
|
|
| ----------- | ------- | --------- |
|
|
| Server to client | `session.created` | Confirms that the WebSocket connection is ready |
|
|
| Client to server | `session.update` | Selects and validates the served model |
|
|
| Client to server | `input_audio_buffer.commit` with `final: false` | Starts generation over the incoming stream |
|
|
| Client to server | `input_audio_buffer.append` | Appends base64 PCM16 audio |
|
|
| Client to server | `input_audio_buffer.commit` with `final: true` | Marks the end of input |
|
|
| Server to client | `transcription.delta` | Carries incremental response text |
|
|
| Server to client | `transcription.done` | Carries final text and token usage |
|
|
| Server to client | `response.output_audio.delta` | Carries incremental PCM16 response audio |
|
|
| Server to client | `response.output_audio.done` | Marks the end of response audio |
|
|
| Server to client | `error` | Reports an invalid event, model, or audio payload |
|
|
|
|
A minimal client sends events in this order:
|
|
|
|
```json
|
|
{"type":"session.update","model":"Qwen/Qwen3-Omni-30B-A3B-Instruct"}
|
|
{"type":"input_audio_buffer.commit","final":false}
|
|
{"type":"input_audio_buffer.append","audio":"<base64-pcm16>"}
|
|
{"type":"input_audio_buffer.commit","final":true}
|
|
```
|
|
|
|
The initial non-final commit intentionally starts generation before all audio
|
|
has arrived. Continue sending `input_audio_buffer.append` events while the
|
|
engine is consuming the stream.
|
|
|
|
## Audio Handling
|
|
|
|
- Input is mono PCM16 at 16 kHz for the Qwen3-Omni example.
|
|
- `response.output_audio.delta.audio` contains base64-encoded PCM16 bytes.
|
|
- Read `sample_rate_hz` from each audio event instead of assuming an output
|
|
rate. Qwen3-Omni output is typically 24 kHz.
|
|
- Concatenate audio deltas in receive order to construct the output waveform.
|
|
- Set `input_audio_buffer.commit.final` to `true` after the last input chunk so
|
|
the server can terminate the streaming request.
|
|
|
|
## Availability and Limitations
|
|
|
|
The path is registered on the unified API server, but it is usable only when
|
|
the loaded pipeline implements realtime audio input and produces compatible
|
|
audio output. Unsupported deployments return an `error` event. The endpoint
|
|
does not provide duplex session resume, playback acknowledgement, overlap
|
|
policy, or barge-in controls.
|
|
|
|
See the [Qwen3-Omni online serving example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni)
|
|
for concurrency options, chunk pacing, and per-delta audio debugging.
|