[Core] Generalized turn-based /v1/realtime implementation (#8339)

Signed-off-by: Nick Cao <ncao@redhat.com> Co-authored-by: Codex <noreply@openai.com>
2026-10-06 22:07:31 +08:00
1 parent 25fb257667
commit 65699fe94c
@@ -277,61 +277,6 @@ cd examples/online_serving/qwen3_omni
 python examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py --model Qwen/Qwen3-Omni-30B-A3B-Instruct --query-type use_image --port 8091 --host "localhost"
 ```

-#### Realtime WebSocket client (`openai_realtime_client.py`)
-
-[`openai_realtime_client.py`](./openai_realtime_client.py) connects to **`ws://<host>:<port>/v1/realtime`**, streams a local WAV as **PCM16 mono @ 16 kHz** in fixed-size chunks, and receives **`response.output_audio.*`**, **`response.output_text.*`**, and transcript events. By default it uses explicit `input_audio_buffer.commit`; pass **`--server-vad`** to let the server detect speech endpoints and commit the turn automatically. The client concatenates audio deltas and writes **`--output-wav`** (model output is typically **24 kHz**). Optional **`--delta-dump-dir`** saves each delta as `delta_000001.wav`, … for debugging.
-
-Streaming input works well for translation-style use cases; if the Thinker runs while input is still incomplete, consider limiting **`max_tokens`** in your session / server defaults to avoid over-generation.
-
-**Dependencies:**
-
-```bash
-pip install websockets
-```
-
-**From this directory** (`examples/online_serving/qwen3_omni`):
-
-```bash
-python openai_realtime_client.py \
-  --url ws://localhost:8091/v1/realtime \
-  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
-  --input-wav /path/to/input_16k_mono.wav \
-  --output-wav realtime_output.wav \
-  --server-vad \
-  --delta-dump-dir ./rt_delta_wavs
-```
-
-**Arguments:**
-
-| Flag | Default | Description |
-| ------ | --------- | ------------- |
-| `--url` | `ws://localhost:8091/v1/realtime` | Full WebSocket URL including path |
-| `--model` | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | Must match the served model (sent in `session.update`) |
-| `--input-wav` | *(required)* | Input WAV: mono, 16-bit PCM, **16 kHz** |
-| `--output-wav` | `realtime_output.wav` | Output path for concatenated reply audio |
-| `--output-text` | *(optional)* | If set, write final transcription text to this path |
-| `--chunk-ms` | `200` | Size of each uploaded audio chunk (milliseconds of audio) |
-| `--send-delay-ms` | `0` | Delay between chunk sends (simulate realtime upload) |
-| `--delta-dump-dir` | *(optional)* | Directory to write per-`response.output_audio.delta` WAV files |
-| `--server-vad` | disabled | Enable server-side Silero VAD instead of sending explicit commits |
-| `--num-requests` | `1` | Number of sequential sessions (see `--concurrency`) |
-| `--concurrency` | `1` | Max concurrent WebSocket sessions when `--num-requests` > 1 |
-
-Server VAD uses the pinned Silero v6.2 ONNX artifact from `istupakov/silero-vad-onnx` and never downloads model files while processing a session. Make the artifact available in the Hugging Face cache before startup, or configure a local artifact in the deployment YAML:
-
-```yaml
-base_config: /path/to/vllm-omni/vllm_omni/deploy/qwen3_omni_duplex.yaml
-duplex_session:
-  server_vad_model_path: /models/silero_vad.onnx
-```
-
-For the engine-owned plugin, use an overlay based on
-`vllm_omni/deploy/qwen3_omni_duplex.yaml`, with `duplex_session.server_vad_model_path`
-pointing to your local artifact. See the [complete duplex setup](../realtime_web/README.md#qwen3-with-server-vad-automatic-turns).
-The default `qwen3_omni_moe.yaml` remains a turn deployment. Duplex mode serves
-`/v1/realtime?duplex=1` and `/v1/chat/completions`; the legacy STT handler requires
-a turn deployment.
-
 The Python client supports the following command-line arguments:

 - `--query-type` (or `-q`): Query type (default: `use_video`). Options: `text`, `use_audio`, `use_image`, `use_video`
@@ -358,6 +303,13 @@ python examples/online_serving/openai_chat_completion_client_for_multimodal_gene
 bash run_curl_multimodal_generation.sh use_image
 ```

+### Realtime WebSocket
+
+The [shared Realtime UI](../realtime_web/README.md#qwen3-explicit-turn-realtime)
+supports explicit turns through `/v1/realtime?duplex=0`. The
+[Server VAD setup](../realtime_web/README.md#qwen3-with-server-vad-automatic-turns)
+uses the duplex endpoint.
+
 ### FAQ

 ## Modality control
@@ -582,26 +534,6 @@ The gradio script supports the following arguments:

 Run these commands from the repository root with the vLLM-Omni environment activated.

-### Without VAD: manually submit each turn
-
-Start the backend:
-
-```bash
-vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
-```
-
-In another terminal, check backend readiness and start the UI:
-
-```bash
-curl --fail http://127.0.0.1:8091/health
-python -m examples.online_serving.qwen3_omni.realtime_web \
-    --backend ws://127.0.0.1:8091 --stt --port 7863
-```
-
-Open `http://localhost:7863`, start a session, speak, and press **Send turn**.
-`--stt` is the default. Each turn uses a separate connection without shared model
-history; the conversation shown in the browser is a local log.
-
 ### With VAD: automatically submit after silence

 Start the backend with a Server VAD deployment overlay and a compatible Silero
@@ -610,7 +542,7 @@ ONNX artifact, following the
 After its health check succeeds, start the UI:

 ```bash
-python -m examples.online_serving.qwen3_omni.realtime_web \
+python -m examples.online_serving.realtime_web --profile qwen3-turn \
    --backend ws://127.0.0.1:8091 --vad --port 7863
 ```

@@ -621,9 +553,7 @@ VAD mode keeps uploading audio during replies and supports speech interruption.
 The engine-owned Qwen duplex plugin maintains conversation history and tracks
 played replies through playback acknowledgements. Each committed utterance starts
 a new generation request; the model does not use native streaming-input KV decoding.
-STT mode pauses microphone upload while the reply is generated and played.
 In VAD mode, click **Camera** to upload sampled frames with your spoken question;
 see [camera setup and limits](../realtime_web/README.md#qwen-vad-camera-input).
-STT remains audio-only. To switch back to manual turns, use the default turn
-deployment, restart the UI with `--stt`, refresh the page, and reconnect. Stop the existing UI before reusing port 7863.
-See the [shared UI guide](../realtime_web/README.md) for HTTPS access and testing.
+For explicit turns, use `--manual` with the shared UI as described in the
+[shared UI guide](../realtime_web/README.md#qwen3-explicit-turn-realtime).
@@ -7,14 +7,14 @@ This implements [RFC #7222](https://github.com/vllm-project/vllm-omni/issues/722
 | Profile | Turn control | Camera | Playback acknowledgement |
 | --- | --- | --- | --- |
 | `minicpm-native` | Model-controlled listen/speak, continuous audio input | Frames accompany audio | Yes |
-| `qwen3-turn --stt` (default) | User presses **Send turn** | No | No |
+| `qwen3-turn --manual` | Press **Send turn**; mic pauses while the reply plays | No | No |
 | `qwen3-turn --vad` | Server detects trailing silence; speech interrupts replies | Sampled frames with each spoken turn | Yes |
 | `aura-ptt` | Hold **Hold to talk**; release commits. Hold also stops local playback | Sticky frames with speech; vision-follow every 2 frames only while unlocked (locked from PTT release until text-final / listen) | Yes |

 Qwen3 VAD uses an engine-owned duplex plugin: microphone upload continues
 while replies stream, and speech can interrupt generation and playback. The
 checkpoint still generates committed turns; this is not native streaming-input
-KV decoding. STT remains explicit-turn and audio-only.
+KV decoding.

 ## MiniCPM: existing command stays valid

@@ -62,45 +62,36 @@ just-committed speech turn on Stage0/1 (`max_num_seqs=1`). Audio still playing
 does not hold the next vision turn once text is done.
 MiniCPM / Qwen profiles do not set `pushToTalk`, so the PTT control stays hidden.

-## Qwen3: explicit-turn STT adapter
+## Qwen3: explicit-turn Realtime

-Start the backend, then the UI in a second terminal:
+The shared UI sends OpenAI Realtime events to `/v1/realtime?duplex=0` and keeps
+conversation history on the same connection:
+
+Start the turn-based backend:

 ```bash
 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
-
-python -m examples.online_serving.qwen3_omni.realtime_web \
-    --backend ws://127.0.0.1:8091 --stt --port 7863
 ```

-Open `http://localhost:7863`, start a session, speak, and press **Send turn**.
-Wait for the answer to finish playing before speaking again. The UI opens a
-fresh STT connection for each turn; the displayed conversation is a local log,
-and **previous turns are not provided as model history**. The STT endpoint
-accepts only model selection, so system-prompt controls are hidden in this mode.
+In another terminal, check readiness and start the UI:

-The adapter follows the shipped `qwen3_omni/openai_realtime_client.py` and
-`vllm_omni/entrypoints/openai/realtime_connection.py`:
+```bash
+curl --fail http://127.0.0.1:8091/health
+python -m examples.online_serving.realtime_web --profile qwen3-turn \
+    --backend ws://127.0.0.1:8091 --manual --port 7863
+```

-
-1. Explicit `duplex=0` selects the legacy STT handler on a turn deployment.
-   It does not enable STT on a duplex deployment.
-2. Send `{type: "session.update", model: ...}` and `commit(final=false)`.
-3. Stream mono PCM16 at 16 kHz; **Send turn** flushes buffered audio, then sends `commit(final=true)`.
-4. `response.output_audio.delta.audio` contains PCM; `response.output_audio.done`
-   terminates this adapter, including a response with no audio.
-5. The local implementation emits **model output** on `transcription.*`, so the
-   UI displays it as assistant text. This differs from the early RFC's suggested
-   user-transcript mapping. The VAD adapter also supports `response.output_text.*`
-   and audio-transcript events without duplicating the answer.
+Speak, then press **Send turn**. The client commits the audio and requests a
+response; recording pauses while the reply plays.

 ## Qwen3 with Server VAD (automatic turns)

 Run commands from the repository root with the vLLM-Omni environment activated.
-Prepare the pinned Silero v6.2 ONNX artifact as described in the
-[Qwen3 Server VAD instructions](../qwen3_omni/README.md#realtime-websocket-client-openai_realtime_clientpy).
-The backend requires ONNX Runtime and a compatible local artifact; setting
-`--vad` on the UI alone does not enable VAD on the backend.
+The plugin uses the pinned Silero v6.2 ONNX artifact from
+`istupakov/silero-vad-onnx` and does not download it while serving. Make it
+available in the Hugging Face cache or set `server_vad_model_path` to a local
+copy in the overlay below. The backend requires ONNX Runtime; setting `--vad`
+on the UI alone does not enable VAD.

 Create an overlay named `qwen3_vad.yaml`, replacing both paths with absolute paths
 on the backend host:
@@ -124,7 +115,7 @@ In a second terminal, wait until the health check succeeds, then start the UI:

 ```bash
 curl --fail http://127.0.0.1:8091/health
-python -m examples.online_serving.qwen3_omni.realtime_web \
+python -m examples.online_serving.realtime_web --profile qwen3-turn \
    --backend ws://127.0.0.1:8091 --vad --port 7863
 ```

@@ -148,21 +139,17 @@ The bundled duplex deploy allows four audio inputs and eight images per prompt.
 Use headphones to avoid speaker audio triggering VAD interruption. Reference
 voices and tool calls are not supported by this plugin.

-### Switching between VAD and manual turns
+### Using the Qwen3 web UI

-| Mode | Backend | UI flag | Submit a turn |
-| --- | --- | --- | --- |
-| Without VAD | Default Qwen turn deployment | `--stt` (default) | Press **Send turn** |
-| With VAD | Server VAD configuration and Silero artifact | `--vad` | Pause after speaking |
+The shared UI supports both explicit turns and Server VAD. Use the manual UI
+command above with the turn-based deployment, or start the VAD deployment above
+and launch the UI with `--vad`.

 Stop the existing UI process before starting another on the same port, then
-refresh the browser and reconnect. To use STT, start a backend with the default
-turn deployment and restart the UI with `--stt`. The duplex backend exposes
-Realtime duplex and ChatCompletion; it does not expose the legacy STT handler.
-To enable VAD on a default backend, first restart that backend with the overlay.
-Make `--backend` match the actual backend port; the UI and backend use separate
-ports. `Connection refused` means the target backend is unavailable: check its
-startup log and `/health` before connecting.
+refresh the browser and reconnect. To enable VAD on a default backend, restart
+that backend with the overlay. Make `--backend` match the actual backend port;
+the UI and backend use separate ports. `Connection refused` means the target
+backend is unavailable: check its startup log and `/health` before connecting.

 VAD connects with `duplex=1` and sends nested `session.audio.input` configuration
 with `create_response: true` and `interrupt_response: true`. It sends playback
@@ -210,8 +197,7 @@ native camera protocol, where Stage 0 interleaves frames at unit boundaries.

 This is a supported subset, not full OpenAI Realtime API compatibility.
 `duplex=1`, `overlap_policy`, `playback.ack`, and `session.close` remain vLLM-Omni
-extensions. The legacy `--stt` mode is unchanged and is not the standard Realtime
-conversation API. This demo does not add WebRTC, semantic VAD, or tool calling.
+extensions. This demo does not add WebRTC, semantic VAD, or tool calling.

 ## Shared host options

@@ -219,11 +205,14 @@ The shared entry point can select a profile explicitly:

 ```bash
 python -m examples.online_serving.realtime_web --profile qwen3-turn \
-    --backend ws://127.0.0.1:8091 --stt --port 7863
+    --backend ws://127.0.0.1:8091 --vad --port 7863
 ```

+This Qwen3 UI profile requires the server-VAD deployment above.
+
 - `--backend` / `--ws-backend`: backend WebSocket origin (not the full `/v1/realtime` URL).
 - `--model`: override the wrapper's model name.
+- `--manual` / `--vad`: Qwen turn control; manual turns are the default.
 - `--public-realtime-url`: optional browser-visible WebSocket URL; otherwise the
  static host proxies `/v1/realtime` on the same origin.
 - `--host`, `--port`: UI bind address and port (default port 7862).
@@ -245,7 +234,7 @@ pytest -m cpu tests/model_executor/models/qwen3_omni/test_duplex_plugin.py \
       tests/engine/test_duplex_orchestrator.py
 ```

-These CPU tests cover profile messages, two STT turns, VAD terminal/drain ordering,
+These CPU tests cover profile message serialization, VAD terminal/drain ordering,
 backpressure, native capture during playback, server config injection, and shared
 worklets. Backend regressions cover current audio inclusion in history-based
 prompts, delayed playback ACK ordering, and no extra generation during silence
@@ -2,24 +2,24 @@
  'use strict';
  const profiles = root.OmniRealtimeProfiles;
  profiles['qwen3-turn'] = (config) => {
-    const vad = config.adapter === 'vad';
+    const vad = config.turnMode === 'vad';
    let interruptOnSpeech = vad;
    let imageSeq = 0;
    let imageItems = [];
    // The engine refuses a ninth image (or 4 MiB of them) for the whole
    // session, so the camera track retires its own oldest frame instead.
    const MAX_IMAGE_ITEMS = 8;
-    const hint = 'This Qwen deploy needs session_mode: duplex and Server VAD support; use --stt or enable Server VAD.';
+    const hint = 'This Qwen deployment needs session_mode: duplex with Server VAD enabled.';
    return {
-      title: 'Qwen3-Omni Voice', eyebrow: vad ? 'Engine-owned duplex call' : 'Turn-based realtime call',
+      title: 'Qwen3-Omni Voice', eyebrow: vad ? 'Engine-owned duplex call' : 'Turn-based Realtime call',
      policy: vad ? 'Server VAD • interruptible responses' : 'Send turn • one response at a time',
      description: vad
        ? 'Pause to send your turn; speak again to interrupt. Microphone upload continues during playback. Requires the Qwen duplex plugin and Silero. Optional camera frames accompany each spoken turn.'
-        : 'Speak, then press Send turn. Microphone upload pauses while the model answers. Each turn uses a fresh connection; conversation history is not carried between turns. No barge-in or camera input.',
+        : 'Speak, then press Send turn. Microphone upload pauses while the model answers. Conversation history stays on the Realtime session. No barge-in or camera input.',
      waiting: 'Waiting for you', camera: vad, cameraMaxDimension: 448, playbackAck: vad, clientCommit: !vad,
-      waitForResponseDone: true, halfDuplex: !vad, deduplicateTranscript: true, closeSession: vad, reconnectEachTurn: !vad,
-      inputSampleRate: vad ? 24000 : 16000,
-      readyEvent: vad ? 'session.updated' : 'session.created', instructions: vad, sendIntervalMs: 200,
+      waitForResponseDone: true, halfDuplex: !vad, deduplicateTranscript: true, closeSession: vad,
+      inputSampleRate: 24000,
+      readyEvent: 'session.updated', instructions: true, sendIntervalMs: 200,
      presets: { assistant: 'You are a helpful assistant. Answer clearly and concisely.' },
      url(config, location) {
        const url = profiles.url(config, location);
@@ -33,22 +33,17 @@
      initialMessages(config, instructions) {
        imageItems = [];
        interruptOnSpeech = vad;
-        if (!vad) return [
-          { type: 'session.update', model: config.model },
-          { type: 'input_audio_buffer.commit', final: false },
-        ];
-        const session = { model: config.model, overlap_policy: 'barge_in_on_speech', audio: { input: {
+        const session = { model: config.model, audio: { input: {
          format: { type: 'audio/pcm', rate: 24000 },
-          turn_detection: { type: 'server_vad', threshold: 0.5, prefix_padding_ms: 300,
-            silence_duration_ms: 500, create_response: true, interrupt_response: true },
+          turn_detection: vad ? { type: 'server_vad', threshold: 0.5, prefix_padding_ms: 300,
+            silence_duration_ms: 500, create_response: true, interrupt_response: true } : null,
        } } };
+        if (vad) session.overlap_policy = 'barge_in_on_speech';
+        else session.type = 'realtime';
        if (instructions) session.instructions = instructions;
        return [{ type: 'session.update', session }];
      },
-      append(audio) {
-        return vad ? { type: 'input_audio_buffer.append', audio }
-          : { type: 'input_audio_buffer.append', audio, format: 'pcm16', sample_rate_hz: 16000 };
-      },
+      append: (audio) => ({ type: 'input_audio_buffer.append', audio }),
      // Qwen is a turn model: a frame is a picture added to the conversation,
      // not a track interleaved into the audio. That is exactly the OpenAI
      // Realtime image interface, so the camera sends conversation items
@@ -68,7 +63,10 @@
        } });
        return messages;
      },
-      commitMessages: () => vad ? [] : [{ type: 'input_audio_buffer.commit', final: true }],
+      commitMessages: () => vad ? [] : [
+        { type: 'input_audio_buffer.commit' },
+        { type: 'response.create' },
+      ],
      ack: (responseId, playedMs) => vad ? { type: 'playback.ack', response_id: responseId,
        item_id: `item_${responseId}`, played_ms: playedMs, committed_ms: playedMs } : null,
      mapEvent(event) {
@@ -87,30 +85,28 @@
          return { kind: 'interrupt' };
        }
        if (event.type === 'playback.acknowledged') return { kind: 'ack', committedMs: (event.event || event).committed_ms || 0 };
-        if (event.type === 'response.output_text.delta' || event.type === 'transcription.delta') {
-          // On the shipped STT route transcription.* is model-generated text,
-          // not a separate ASR transcript of the user. See realtime_connection.py.
+        if (event.type === 'response.output_text.delta') {
          return { kind: 'text', role: 'assistant', channel: 'text', text: event.delta || '' };
        }
-        if (event.type === 'response.output_text.done' || event.type === 'transcription.done') {
+        if (event.type === 'response.output_text.done') {
          return { kind: 'text-final', role: 'assistant', channel: 'text', text: event.text || '' };
        }
        if (vad && (event.type === 'output_audio_buffer.cleared' ||
            (event.type === 'response.done' && event.response?.status === 'cancelled'))) {
          return { kind: 'interrupt', responseId: event.response_id || event.response?.id || null };
        }
-        if (event.type === 'input_audio_buffer.committed') return { kind: 'begin' };
+        if (vad && event.type === 'input_audio_buffer.committed') return { kind: 'begin' };
+        if (!vad && event.type === 'input_audio_buffer.cleared') return { kind: 'ignore' };
        if (event.type === 'input_audio_buffer.cleared') return { kind: 'backpressure', message: 'Input cleared. Model is answering; please wait.' };
-        if (!vad && event.type === 'response.output_audio.done') return { kind: 'done' };
        const action = profiles.event(event);
        if (action.kind === 'error') {
          if (action.code === 'input_backpressure') return { ...action, kind: 'backpressure', message: 'Input buffer is full; please repeat dropped speech after the response.' };
-          if (action.code === 'server_vad_initialization_failed') action.message = 'Server VAD could not load Silero. Configure the server artifact or restart this UI with --stt.';
+          if (action.code === 'server_vad_initialization_failed') action.message = 'Server VAD could not load Silero. Configure the server artifact and restart the UI.';
          else if (vad && action.code === 'unsupported') action.message += ` ${hint}`;
        }
        return action;
      },
-      connectionHint: vad ? hint : 'Check the Qwen /v1/realtime STT endpoint and model name.',
+      connectionHint: vad ? hint : 'Check the Qwen turn-based /v1/realtime deployment and model name.',
    };
  };
 })(globalThis);
@@ -74,12 +74,12 @@ def build_app(
    public_realtime_url: str | None = None,
    ref_audio: str | None = None,
    profile: str = "minicpm-native",
-    adapter: str = "stt",
+    turn_mode: str = "manual",
 ) -> FastAPI:
    if profile not in {"minicpm-native", "qwen3-turn", "aura-ptt"}:
        raise ValueError(f"Unknown profile: {profile}")
-    if adapter not in {"stt", "vad"}:
-        raise ValueError(f"Unknown adapter: {adapter}")
+    if turn_mode not in {"manual", "vad"}:
+        raise ValueError(f"Unknown turn mode: {turn_mode}")
    if profile == "qwen3-turn" and ref_audio:
        raise ValueError("Qwen3 profile does not accept --ref-audio")
    if profile == "aura-ptt" and ref_audio:
@@ -116,7 +116,7 @@ def build_app(
            {
                "model": model,
                "profile": profile,
-                "adapter": adapter,
+                "turnMode": turn_mode,
                "realtimePath": public_realtime_url or "v1/realtime",
                "refAudio": ref_audio_uri,
                "appVersion": app_version,
@@ -190,9 +190,9 @@ def main(default_profile: str = "minicpm-native") -> None:
    parser.add_argument("--profile", choices=("minicpm-native", "qwen3-turn", "aura-ptt"), default=default_profile)
    parser.add_argument("--ws-backend", "--backend", dest="ws_backend")
    mode = parser.add_mutually_exclusive_group()
-    mode.add_argument("--stt", dest="adapter", action="store_const", const="stt")
-    mode.add_argument("--vad", dest="adapter", action="store_const", const="vad")
-    parser.set_defaults(adapter="stt")
+    mode.add_argument("--manual", dest="turn_mode", action="store_const", const="manual")
+    mode.add_argument("--vad", dest="turn_mode", action="store_const", const="vad")
+    parser.set_defaults(turn_mode="manual")
    parser.add_argument(
        "--public-realtime-url",
        help="Browser-visible ws:// or wss:// Realtime URL; defaults to the same-origin proxy.",
@@ -225,7 +225,7 @@ def main(default_profile: str = "minicpm-native") -> None:
            public_realtime_url=args.public_realtime_url,
            ref_audio=args.ref_audio,
            profile=args.profile,
-            adapter=args.adapter,
+            turn_mode=args.turn_mode,
        ),
        host=args.host,
        port=args.port,