vllm-omni/examples/online_serving/realtime_web

Shared Realtime Web UI

One browser shell provides microphone capture, speaker playback, a level meter, conversation text, and an event log. Model profiles own the wire protocol. This implements RFC #7222.

Profile	Turn control	Camera	Playback acknowledgement
minicpm-native	Model-controlled listen/speak, continuous audio input	Frames accompany audio	Yes
qwen3-turn --manual	Press Send turn; mic pauses while the reply plays	No	No
qwen3-turn --vad	Server detects trailing silence; speech interrupts replies	Sampled frames with each spoken turn	Yes
aura-ptt	Hold Hold to talk; release commits. Hold also stops local playback	Sticky frames with speech; vision-follow every 2 frames only while unlocked (locked from PTT release until text-final / listen)	Yes

Qwen3 VAD uses an engine-owned duplex plugin: microphone upload continues while replies stream, and speech can interrupt generation and playback. The checkpoint still generates committed turns; this is not native streaming-input KV decoding.

MiniCPM: existing command stays valid

Start your MiniCPM backend using the existing deployment instructions, then:

python -m examples.online_serving.minicpmo.realtime_web \
    --ws-backend ws://127.0.0.1:8099 \
    --ref-audio /path/to/ref_minicpm_signature.wav

The compatibility wrapper serves the shared assets with minicpm-native. The current native query, extra_body, reference voice, continuous microphone upload, camera frames and playback acknowledgements are retained.

AURA: push-to-talk duplex

AURA has no client VAD. Start the duplex serve, then the thin wrapper:

bash examples/online_serving/aura_omni/run_duplex_smoke_serve.sh

python -m examples.online_serving.aura_omni.realtime_web \
    --ws-backend ws://127.0.0.1:8099 --model aurateam/AURA --port 7862

Or select the profile on the shared host:

python -m examples.online_serving.realtime_web --profile aura-ptt \
    --ws-backend ws://127.0.0.1:8099 --model aurateam/AURA --port 7862

Open the UI, start a session, optionally enable Camera, then hold Hold to talk to stream is_speech=true PCM (with sticky camera frames). Release to input_audio_buffer.commit. Holding the button stops playback in the browser only; the in-flight response is not cancelled on the server. With the camera on, frames are still captured at 2 fps. While the button is up they are committed once per two frames (is_speech=false + silent PCM). Both frames of that commit are sent and packed as one video (<|video_pad|>). Turns still open from this client commit, not from a server-side auto trigger. but only after the prior spoken turn unlocks (text-final or response.listen). Release itself locks vision immediately so a follow-up frame cannot abort the just-committed speech turn on Stage0/1 (max_num_seqs=1). Audio still playing does not hold the next vision turn once text is done. MiniCPM / Qwen profiles do not set pushToTalk, so the PTT control stays hidden.

Qwen3: explicit-turn Realtime

The shared UI sends OpenAI Realtime events to /v1/realtime?duplex=0 and keeps conversation history on the same connection:

Start the turn-based backend:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

In another terminal, check readiness and start the UI:

curl --fail http://127.0.0.1:8091/health
python -m examples.online_serving.realtime_web --profile qwen3-turn \
    --backend ws://127.0.0.1:8091 --manual --port 7863

Speak, then press Send turn. The client commits the audio and requests a response; recording pauses while the reply plays.

Qwen3 with Server VAD (automatic turns)

Run commands from the repository root with the vLLM-Omni environment activated. The plugin uses the pinned Silero v6.2 ONNX artifact from istupakov/silero-vad-onnx and does not download it while serving. Make it available in the Hugging Face cache or set server_vad_model_path to a local copy in the overlay below. The backend requires ONNX Runtime; setting --vad on the UI alone does not enable VAD.

Create an overlay named qwen3_vad.yaml, replacing both paths with absolute paths on the backend host:

base_config: /path/to/vllm-omni/vllm_omni/deploy/qwen3_omni_duplex.yaml
duplex_session:
  server_vad_model_path: /path/to/silero_vad.onnx

The base configuration enables duplex mode and sets the per-prompt audio and image limits.

Start the backend:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --deploy-config qwen3_vad.yaml

In a second terminal, wait until the health check succeeds, then start the UI:

curl --fail http://127.0.0.1:8091/health
python -m examples.online_serving.realtime_web --profile qwen3-turn \
    --backend ws://127.0.0.1:8091 --vad --port 7863

Open http://localhost:7863 and start a session. Speak and pause: the server commits the turn automatically after 500 ms of silence. No Send turn action is needed. Defaults are a speech threshold of 0.5 and 300 ms of prefix padding. Speak again to interrupt an answer. The engine owns input buffering, VAD, request cancellation, playback acknowledgements, and conversation history. Each committed utterance starts a normal Thinker → Talker → Code2Wav request; it does not append input to a live Qwen KV cache. The capability response reports implementation_level: turn_based_duplex and supports_core_resumable_request: false.

The plugin retains up to four audio inputs (including the current turn), pruning older audio at an 8 MiB base64 budget; one current utterance is bounded by the engine's pending-input limit. The UI reports playback progress both on normal drain and on interruption. Qwen does not provide text/audio alignment, so assistant text enters history only after the complete audio response has been generated and acknowledged as played. A partially played answer contributes no guessed text prefix. Image conversation items remain available until deleted. Model context limits still apply. The bundled duplex deploy allows four audio inputs and eight images per prompt. Use headphones to avoid speaker audio triggering VAD interruption. Reference voices and tool calls are not supported by this plugin.

Using the Qwen3 web UI

The shared UI supports both explicit turns and Server VAD. Use the manual UI command above with the turn-based deployment, or start the VAD deployment above and launch the UI with --vad.

Stop the existing UI process before starting another on the same port, then refresh the browser and reconnect. To enable VAD on a default backend, restart that backend with the overlay. Make --backend match the actual backend port; the UI and backend use separate ports. Connection refused means the target backend is unavailable: check its startup log and /health before connecting.

VAD connects with duplex=1 and sends nested session.audio.input configuration with create_response: true and interrupt_response: true. It sends playback ACKs when speaker audio drains. response.audio.done / response.output_audio.done only drain playback; response.done terminates a turn. Backpressure displays a notice without pausing the microphone. Failed handshakes time out after 15 seconds; Silero initialization failures display a specific error.

After updating the backend Python code, restart the backend, refresh the UI, and start a new session so the plugin and browser assets use the updated code. A successful /health check confirms readiness, not which deployment mode is active.

Qwen VAD camera input

After starting a session with --vad, click Camera and allow access. The UI sends one JPEG per second, resized to a longest side of 448 pixels, independently of microphone uploads (including while muted). Each frame is a user conversation item using the OpenAI Realtime image input shape:

{"type":"conversation.item.create","item":{"id":"camera_1","type":"message","role":"user","content":[{"type":"input_image","image_url":"data:image/jpeg;base64,..."}]}}

Images become conversation context immediately; they do not start a response. Speak and let VAD commit your audio, or send response.create after adding an image/text item. A manual input_audio_buffer.commit only commits audio; follow it with response.create to generate. Images persist across audio commits, interruptions, and audio-buffer clears. Remove them with conversation.item.delete. The browser retains eight camera items and deletes the oldest before adding a ninth. Camera-off stops capture but leaves those items in the conversation.

This deployment accepts JPEG/PNG Base64 data URLs with a total image-context budget of eight images and 4 MiB; external image URLs are not supported. Clients must delete old image items when reaching the limit. There is no precise video/audio timestamp alignment or native streaming-video KV update.

Protocol compatibility boundary

Qwen VAD uses session.update, 24 kHz PCM16 input_audio_buffer.append, conversation.item.create with input_image/input_text, and response.create. The engine resamples input to Qwen's 16 kHz processing rate. Qwen rejects the custom video_frames field on an append: it answers whole turns, so it has no audio unit for a frame track to align to. MiniCPM keeps that field for its native camera protocol, where Stage 0 interleaves frames at unit boundaries.

This is a supported subset, not full OpenAI Realtime API compatibility. duplex=1, overlap_policy, playback.ack, and session.close remain vLLM-Omni extensions. This demo does not add WebRTC, semantic VAD, or tool calling.

Shared host options

The shared entry point can select a profile explicitly:

python -m examples.online_serving.realtime_web --profile qwen3-turn \
    --backend ws://127.0.0.1:8091 --vad --port 7863

This Qwen3 UI profile requires the server-VAD deployment above.

• --backend / --ws-backend: backend WebSocket origin (not the full /v1/realtime URL).
• --model: override the wrapper's model name.
• --manual / --vad: Qwen turn control; manual turns are the default.
• --public-realtime-url: optional browser-visible WebSocket URL; otherwise the static host proxies /v1/realtime on the same origin.
• --host, --port: UI bind address and port (default port 7862).
• --ref-audio: required for MiniCPM; rejected for Qwen3 and AURA.

Microphone and camera access require localhost or HTTPS. For a remote backend, use an SSH tunnel to the UI host or an HTTPS reverse proxy with WebSocket support. An explicit wss:// public URL retains its scheme.

Validation

node --test tests/examples/test_realtime_web_profiles.cjs
pytest tests/examples/test_minicpmo_realtime_web_server.py \
       tests/examples/test_minicpmo_realtime_web_static.py \
       tests/examples/test_realtime_web_profiles.py
pytest -m cpu tests/model_executor/models/qwen3_omni/test_duplex_plugin.py \
       tests/engine/duplex/test_session_runner.py \
       tests/engine/test_duplex_orchestrator.py

These CPU tests cover profile message serialization, VAD terminal/drain ordering, backpressure, native capture during playback, server config injection, and shared worklets. Backend regressions cover current audio inclusion in history-based prompts, delayed playback ACK ordering, and no extra generation during silence after a completed VAD turn. Hardware acceptance still requires a MiniCPM call with camera/barge-in and two Qwen turns against each enabled backend, including speech interruption in VAD mode; simulated tests do not establish model quality or hardware latency.