[Frontend] OpenAI-compliant fullduplex /v1/realtime with Qwen3-Omni support (#7285)

Signed-off-by: vraiti <vraiti@redhat.com>

docs/serving/README.md

@@ -144,7 +144,7 @@ interchangeably.
 | ---------- | ---------- | ------------------- | --------- |
 | Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) |
 | Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) |
-| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) |
+| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_openai_duplex_api.md) |
 | Continuous speech-to-speech interaction | `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`) | Listen and speak concurrently with session control | [Full Duplex](full_duplex_api.md) |
 | Generated video chunks | `WS /v1/realtime/video` | Start a diffusion request and receive fragmented MP4 | [Streaming Video Output](streaming_video_output_api.md) |
 | Robot policy inference | `WS /v1/realtime/robot/openpi` | Send MessagePack observations and receive action arrays | [OpenPI Robot Policy](openpi_api.md) |

docs/serving/full_duplex_api.md

@@ -5,7 +5,7 @@ continue receiving speech while producing speech. It adds persistent session
 state, model-specific turn policy, overlap handling, playback acknowledgement,
 and optional session resume.
 
-Full duplex is distinct from the turn-based [Realtime Audio API](realtime_api.md).
+Full duplex is distinct from the turn-based [Realtime Audio API](realtime_openai_duplex_api.md).
 
 This page is the endpoint overview. For the complete wire contract, the
 `vllm_omni.clients.duplex.DuplexClient` Python library, and the per-model

docs/serving/realtime_openai_duplex_api.md

@@ -1,33 +1,31 @@
 # Realtime Audio WebSocket API
 
-Use `WS /v1/realtime` to stream PCM audio into a compatible omni model and
-receive incremental transcription and synthesized response audio. This is a
-turn-based streaming API: it starts one generation over an audio stream and
-finishes after the client closes that input stream.
-
-For a model that listens while it is already speaking, use the
-[Full-Duplex API](full_duplex_api.md) instead.
+As a companion to the vLLM-Omni-proprietary [fullduplex `/v1/realtime`](full_duplex_api.md),
+supported models also provide a `/v1/realtime` handler supporting
+a substantial subset of the OpenAI `/v1/realtime` API supporting
+audio streaming, client-driven VAD, and tool calling
 
 ## Quick Start
 
-Start a compatible model. The currently documented example uses Qwen3-Omni:
+The endpoint is currently only supported for Qwen3-Omni. To start:
+
+Start vLLM-Omni with port `8000` bound to an address reachable from Docker:
 
 ```bash
-vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
+vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8000 --host 0.0.0.0
 ```
 
-Run the provided client with a mono, 16-bit PCM, 16 kHz WAV file:
+Then start the docker compose:
 
 ```bash
-python examples/online_serving/qwen3_omni/openai_realtime_client.py \
-  --url ws://localhost:8091/v1/realtime \
-  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
-  --input-wav input_16k_mono.wav \
-  --output-wav response.wav \
-  --output-text response.txt
+cd examples/online_serving/qwen3_omni/fullduplex-livekit-frontend
+VLLM_OMNI_HOST=$VLLM_OMNI_HOST docker compose up --build
 ```
 
-The client dependency is `websockets`.
+Where `VLLM_OMNI_HOST` is a network address. If vLLM-Omni is running on localhost, use
+`host.docker.internal`
+
+Open [http://localhost:3000](http://localhost:3000) and begin chatting.
 
 ## Protocol
 
@@ -37,10 +35,9 @@ event is base64-encoded raw PCM16.
 | Direction | Event | Purpose |
 | ----------- | ------- | --------- |
 | Server to client | `session.created` | Confirms that the WebSocket connection is ready |
-| Client to server | `session.update` | Selects and validates the served model |
-| Client to server | `input_audio_buffer.commit` with `final: false` | Starts generation over the incoming stream |
+| Client to server | `session.update` | Selects and validates the served model. Configures system prompt and tools |
 | Client to server | `input_audio_buffer.append` | Appends base64 PCM16 audio |
-| Client to server | `input_audio_buffer.commit` with `final: true` | Marks the end of input |
+| Client to server | `input_audio_buffer.commit` | Starts generation over the incoming stream |
 | Server to client | `transcription.delta` | Carries incremental response text |
 | Server to client | `transcription.done` | Carries final text and token usage |
 | Server to client | `response.output_audio.delta` | Carries incremental PCM16 response audio |
@@ -50,15 +47,15 @@ event is base64-encoded raw PCM16.
 A minimal client sends events in this order:
 
 ```json
-{"type":"session.update","model":"Qwen/Qwen3-Omni-30B-A3B-Instruct"}
-{"type":"input_audio_buffer.commit","final":false}
-{"type":"input_audio_buffer.append","audio":"<base64-pcm16>"}
-{"type":"input_audio_buffer.commit","final":true}
+{"type":"input_audio_buffer.append","audio":"<base64-pcm16>"},
+{"type":"input_audio_buffer.commit"}
 ```
 
-The initial non-final commit intentionally starts generation before all audio
-has arrived. Continue sending `input_audio_buffer.append` events while the
-engine is consuming the stream.
+This OpenAI-compatible streaming path is distinct from the vLLM-Omni-proprietary
+lane and as such is reached via different configuration. The vLLM-Omni-proprietary
+path must be specified explicitly via "duplex=1|true|on" as the WS query parameter
+or via an explicit configuration in the `deploy_config`. The OpenAI path simply replaces
+the default `/v1/realtime` handler in all supported models.
 
 ## Audio Handling
 
@@ -67,8 +64,6 @@ engine is consuming the stream.
 - Read `sample_rate_hz` from each audio event instead of assuming an output
   rate. Qwen3-Omni output is typically 24 kHz.
 - Concatenate audio deltas in receive order to construct the output waveform.
-- Set `input_audio_buffer.commit.final` to `true` after the last input chunk so
-  the server can terminate the streaming request.
 
 ## Availability and Limitations
 
@@ -76,7 +71,8 @@ The path is registered on the unified API server, but it is usable only when
 the loaded pipeline implements realtime audio input and produces compatible
 audio output. Unsupported deployments return an `error` event. The endpoint
 does not provide duplex session resume, playback acknowledgement, overlap
-policy, or barge-in controls.
+policy, or barge-in controls internally. Those must be driven by the client application
+via the relevant WS events (e.g. `response.cancel`, `conversation.item.truncate`)
 
 See the [Qwen3-Omni online serving example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni)
 for concurrency options, chunk pacing, and per-delta audio debugging.