vllm-omni/examples/online_serving/realtime_web/README.md

Shared Realtime Web UI

One browser shell provides microphone capture, speaker playback, a level meter, conversation text, and an event log. Model profiles own the wire protocol. This implements RFC #7222.

Profile	Turn control	Camera	Playback acknowledgement
minicpm-native	Model-controlled listen/speak, continuous audio input	Frames accompany audio	Yes
qwen3-turn --stt (default)	User presses Send turn	No	No
qwen3-turn --vad	Server detects trailing silence; speech interrupts replies	Sampled frames with each spoken turn	Yes
aura-ptt	Hold Hold to talk; release commits. Hold also stops local playback	Sticky frames with speech; vision-follow every 2 frames only while unlocked (locked from PTT release until text-final / listen)	Yes

Qwen3 VAD uses an engine-owned duplex plugin: microphone upload continues while replies stream, and speech can interrupt generation and playback. The checkpoint still generates committed turns; this is not native streaming-input KV decoding. STT remains explicit-turn and audio-only.

MiniCPM: existing command stays valid

Start your MiniCPM backend using the existing deployment instructions, then:

python -m examples.online_serving.minicpmo.realtime_web \
    --ws-backend ws://127.0.0.1:8099 \
    --ref-audio /path/to/ref_minicpm_signature.wav

The compatibility wrapper serves the shared assets with minicpm-native. The current native query, extra_body, reference voice, continuous microphone upload, camera frames and playback acknowledgements are retained.

AURA: push-to-talk duplex

AURA has no client VAD. Start the duplex serve, then the thin wrapper:

bash examples/online_serving/aura_omni/run_duplex_smoke_serve.sh

python -m examples.online_serving.aura_omni.realtime_web \
    --ws-backend ws://127.0.0.1:8099 --model aurateam/AURA --port 7862

Or select the profile on the shared host:

python -m examples.online_serving.realtime_web --profile aura-ptt \
    --ws-backend ws://127.0.0.1:8099 --model aurateam/AURA --port 7862

Open the UI, start a session, optionally enable Camera, then hold Hold to talk to stream is_speech=true PCM (with sticky camera frames). Release to input_audio_buffer.commit. Holding the button stops playback in the browser only; the in-flight response is not cancelled on the server. With the camera on, frames are still captured at 2 fps. While the button is up they are committed once per two frames (is_speech=false + silent PCM). Both frames of that commit are sent and packed as one video (<|video_pad|>). Turns still open from this client commit, not from a server-side auto trigger. but only after the prior spoken turn unlocks (text-final or response.listen). Release itself locks vision immediately so a follow-up frame cannot abort the just-committed speech turn on Stage0/1 (max_num_seqs=1). Audio still playing does not hold the next vision turn once text is done. MiniCPM / Qwen profiles do not set pushToTalk, so the PTT control stays hidden.

Qwen3: explicit-turn STT adapter

Start the backend, then the UI in a second terminal:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

python -m examples.online_serving.qwen3_omni.realtime_web \
    --backend ws://127.0.0.1:8091 --stt --port 7863

Open http://localhost:7863, start a session, speak, and press Send turn. Wait for the answer to finish playing before speaking again. The UI opens a fresh STT connection for each turn; the displayed conversation is a local log, and previous turns are not provided as model history. The STT endpoint accepts only model selection, so system-prompt controls are hidden in this mode.

The adapter follows the shipped qwen3_omni/openai_realtime_client.py and vllm_omni/entrypoints/openai/realtime_connection.py:

1. Explicit duplex=0 selects the legacy STT handler on a turn deployment. It does not enable STT on a duplex deployment.
2. Send {type: "session.update", model: ...} and commit(final=false).
3. Stream mono PCM16 at 16 kHz; Send turn flushes buffered audio, then sends commit(final=true).
4. response.output_audio.delta.audio contains PCM; response.output_audio.done terminates this adapter, including a response with no audio.
5. The local implementation emits model output on transcription.*, so the UI displays it as assistant text. This differs from the early RFC's suggested user-transcript mapping. The VAD adapter also supports response.output_text.* and audio-transcript events without duplicating the answer.

Qwen3 with Server VAD (automatic turns)

Run commands from the repository root with the vLLM-Omni environment activated. Prepare the pinned Silero v6.2 ONNX artifact as described in the Qwen3 Server VAD instructions. The backend requires ONNX Runtime and a compatible local artifact; setting --vad on the UI alone does not enable VAD on the backend.

Create an overlay named qwen3_vad.yaml, replacing both paths with absolute paths on the backend host:

base_config: /path/to/vllm-omni/vllm_omni/deploy/qwen3_omni_duplex.yaml
duplex_session:
  server_vad_model_path: /path/to/silero_vad.onnx

The base configuration enables duplex mode and sets the per-prompt audio and image limits.

Start the backend:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --deploy-config qwen3_vad.yaml

In a second terminal, wait until the health check succeeds, then start the UI:

curl --fail http://127.0.0.1:8091/health
python -m examples.online_serving.qwen3_omni.realtime_web \
    --backend ws://127.0.0.1:8091 --vad --port 7863

Open http://localhost:7863 and start a session. Speak and pause: the server commits the turn automatically after 500 ms of silence. No Send turn action is needed. Defaults are a speech threshold of 0.5 and 300 ms of prefix padding. Speak again to interrupt an answer. The engine owns input buffering, VAD, request cancellation, playback acknowledgements, and conversation history. Each committed utterance starts a normal Thinker → Talker → Code2Wav request; it does not append input to a live Qwen KV cache. The capability response reports implementation_level: turn_based_duplex and supports_core_resumable_request: false.

The plugin retains up to four audio inputs (including the current turn), pruning older audio at an 8 MiB base64 budget; one current utterance is bounded by the engine's pending-input limit. The UI reports playback progress both on normal drain and on interruption. Qwen does not provide text/audio alignment, so assistant text enters history only after the complete audio response has been generated and acknowledged as played. A partially played answer contributes no guessed text prefix. Image conversation items remain available until deleted. Model context limits still apply. The bundled duplex deploy allows four audio inputs and eight images per prompt. Use headphones to avoid speaker audio triggering VAD interruption. Reference voices and tool calls are not supported by this plugin.

Switching between VAD and manual turns

Mode	Backend	UI flag	Submit a turn
Without VAD	Default Qwen turn deployment	--stt (default)	Press Send turn
With VAD	Server VAD configuration and Silero artifact	--vad	Pause after speaking

Stop the existing UI process before starting another on the same port, then refresh the browser and reconnect. To use STT, start a backend with the default turn deployment and restart the UI with --stt. The duplex backend exposes Realtime duplex and ChatCompletion; it does not expose the legacy STT handler. To enable VAD on a default backend, first restart that backend with the overlay. Make --backend match the actual backend port; the UI and backend use separate ports. Connection refused means the target backend is unavailable: check its startup log and /health before connecting.

VAD connects with duplex=1 and sends nested session.audio.input configuration with create_response: true and interrupt_response: true. It sends playback ACKs when speaker audio drains. response.audio.done / response.output_audio.done only drain playback; response.done terminates a turn. Backpressure displays a notice without pausing the microphone. Failed handshakes time out after 15 seconds; Silero initialization failures display a specific error.

After updating the backend Python code, restart the backend, refresh the UI, and start a new session so the plugin and browser assets use the updated code. A successful /health check confirms readiness, not which deployment mode is active.

Qwen VAD camera input

After starting a session with --vad, click Camera and allow access. The UI sends one JPEG per second, resized to a longest side of 448 pixels, independently of microphone uploads (including while muted). Each frame is a user conversation item using the OpenAI Realtime image input shape:

{"type":"conversation.item.create","item":{"id":"camera_1","type":"message","role":"user","content":[{"type":"input_image","image_url":"data:image/jpeg;base64,..."}]}}

Images become conversation context immediately; they do not start a response. Speak and let VAD commit your audio, or send response.create after adding an image/text item. A manual input_audio_buffer.commit only commits audio; follow it with response.create to generate. Images persist across audio commits, interruptions, and audio-buffer clears. Remove them with conversation.item.delete. The browser retains eight camera items and deletes the oldest before adding a ninth. Camera-off stops capture but leaves those items in the conversation.

This deployment accepts JPEG/PNG Base64 data URLs with a total image-context budget of eight images and 4 MiB; external image URLs are not supported. Clients must delete old image items when reaching the limit. There is no precise video/audio timestamp alignment or native streaming-video KV update.

Protocol compatibility boundary

Qwen VAD uses session.update, 24 kHz PCM16 input_audio_buffer.append, conversation.item.create with input_image/input_text, and response.create. The engine resamples input to Qwen's 16 kHz processing rate. Qwen rejects the custom video_frames field on an append: it answers whole turns, so it has no audio unit for a frame track to align to. MiniCPM keeps that field for its native camera protocol, where Stage 0 interleaves frames at unit boundaries.

This is a supported subset, not full OpenAI Realtime API compatibility. duplex=1, overlap_policy, playback.ack, and session.close remain vLLM-Omni extensions. The legacy --stt mode is unchanged and is not the standard Realtime conversation API. This demo does not add WebRTC, semantic VAD, or tool calling.

Shared host options

The shared entry point can select a profile explicitly:

python -m examples.online_serving.realtime_web --profile qwen3-turn \
    --backend ws://127.0.0.1:8091 --stt --port 7863

• --backend / --ws-backend: backend WebSocket origin (not the full /v1/realtime URL).
• --model: override the wrapper's model name.
• --public-realtime-url: optional browser-visible WebSocket URL; otherwise the static host proxies /v1/realtime on the same origin.
• --host, --port: UI bind address and port (default port 7862).
• --ref-audio: required for MiniCPM; rejected for Qwen3 and AURA.

Microphone and camera access require localhost or HTTPS. For a remote backend, use an SSH tunnel to the UI host or an HTTPS reverse proxy with WebSocket support. An explicit wss:// public URL retains its scheme.

Validation

node --test tests/examples/test_realtime_web_profiles.cjs
pytest tests/examples/test_minicpmo_realtime_web_server.py \
       tests/examples/test_minicpmo_realtime_web_static.py \
       tests/examples/test_realtime_web_profiles.py
pytest -m cpu tests/model_executor/models/qwen3_omni/test_duplex_plugin.py \
       tests/engine/duplex/test_session_runner.py \
       tests/engine/test_duplex_orchestrator.py

These CPU tests cover profile messages, two STT turns, VAD terminal/drain ordering, backpressure, native capture during playback, server config injection, and shared worklets. Backend regressions cover current audio inclusion in history-based prompts, delayed playback ACK ordering, and no extra generation during silence after a completed VAD turn. Hardware acceptance still requires a MiniCPM call with camera/barge-in and two Qwen turns against each enabled backend, including speech interruption in VAD mode; simulated tests do not establish model quality or hardware latency.