[Core] Generalized turn-based /v1/realtime implementation (#8339)

Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
Nick CaoandCodex authored and GitHub committed 2026-10-06 22:07:31 +08:00
1 parent 25fb257667
commit 65699fe94c
16 files changed
+716 -799

No files matched your search

+35 -46
View File
@@ -7,14 +7,14 @@ This implements [RFC #7222](https://github.com/vllm-project/vllm-omni/issues/722
| Profile | Turn control | Camera | Playback acknowledgement |
| --- | --- | --- | --- |
| `minicpm-native` | Model-controlled listen/speak, continuous audio input | Frames accompany audio | Yes |
| `qwen3-turn --stt` (default) | User presses **Send turn** | No | No |
| `qwen3-turn --manual` | Press **Send turn**; mic pauses while the reply plays | No | No |
| `qwen3-turn --vad` | Server detects trailing silence; speech interrupts replies | Sampled frames with each spoken turn | Yes |
| `aura-ptt` | Hold **Hold to talk**; release commits. Hold also stops local playback | Sticky frames with speech; vision-follow every 2 frames only while unlocked (locked from PTT release until text-final / listen) | Yes |
Qwen3 VAD uses an engine-owned duplex plugin: microphone upload continues
while replies stream, and speech can interrupt generation and playback. The
checkpoint still generates committed turns; this is not native streaming-input
KV decoding. STT remains explicit-turn and audio-only.
KV decoding.
## MiniCPM: existing command stays valid
@@ -62,45 +62,36 @@ just-committed speech turn on Stage0/1 (`max_num_seqs=1`). Audio still playing
does not hold the next vision turn once text is done.
MiniCPM / Qwen profiles do not set `pushToTalk`, so the PTT control stays hidden.
## Qwen3: explicit-turn STT adapter
## Qwen3: explicit-turn Realtime
Start the backend, then the UI in a second terminal:
The shared UI sends OpenAI Realtime events to `/v1/realtime?duplex=0` and keeps
conversation history on the same connection:
Start the turn-based backend:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
python -m examples.online_serving.qwen3_omni.realtime_web \
--backend ws://127.0.0.1:8091 --stt --port 7863
```
Open `http://localhost:7863`, start a session, speak, and press **Send turn**.
Wait for the answer to finish playing before speaking again. The UI opens a
fresh STT connection for each turn; the displayed conversation is a local log,
and **previous turns are not provided as model history**. The STT endpoint
accepts only model selection, so system-prompt controls are hidden in this mode.
In another terminal, check readiness and start the UI:
The adapter follows the shipped `qwen3_omni/openai_realtime_client.py` and
`vllm_omni/entrypoints/openai/realtime_connection.py`:
```bash
curl --fail http://127.0.0.1:8091/health
python -m examples.online_serving.realtime_web --profile qwen3-turn \
--backend ws://127.0.0.1:8091 --manual --port 7863
```
1. Explicit `duplex=0` selects the legacy STT handler on a turn deployment.
It does not enable STT on a duplex deployment.
2. Send `{type: "session.update", model: ...}` and `commit(final=false)`.
3. Stream mono PCM16 at 16 kHz; **Send turn** flushes buffered audio, then sends `commit(final=true)`.
4. `response.output_audio.delta.audio` contains PCM; `response.output_audio.done`
terminates this adapter, including a response with no audio.
5. The local implementation emits **model output** on `transcription.*`, so the
UI displays it as assistant text. This differs from the early RFC's suggested
user-transcript mapping. The VAD adapter also supports `response.output_text.*`
and audio-transcript events without duplicating the answer.
Speak, then press **Send turn**. The client commits the audio and requests a
response; recording pauses while the reply plays.
## Qwen3 with Server VAD (automatic turns)
Run commands from the repository root with the vLLM-Omni environment activated.
Prepare the pinned Silero v6.2 ONNX artifact as described in the
[Qwen3 Server VAD instructions](../qwen3_omni/README.md#realtime-websocket-client-openai_realtime_clientpy).
The backend requires ONNX Runtime and a compatible local artifact; setting
`--vad` on the UI alone does not enable VAD on the backend.
The plugin uses the pinned Silero v6.2 ONNX artifact from
`istupakov/silero-vad-onnx` and does not download it while serving. Make it
available in the Hugging Face cache or set `server_vad_model_path` to a local
copy in the overlay below. The backend requires ONNX Runtime; setting `--vad`
on the UI alone does not enable VAD.
Create an overlay named `qwen3_vad.yaml`, replacing both paths with absolute paths
on the backend host:
@@ -124,7 +115,7 @@ In a second terminal, wait until the health check succeeds, then start the UI:
```bash
curl --fail http://127.0.0.1:8091/health
python -m examples.online_serving.qwen3_omni.realtime_web \
python -m examples.online_serving.realtime_web --profile qwen3-turn \
--backend ws://127.0.0.1:8091 --vad --port 7863
```
@@ -148,21 +139,17 @@ The bundled duplex deploy allows four audio inputs and eight images per prompt.
Use headphones to avoid speaker audio triggering VAD interruption. Reference
voices and tool calls are not supported by this plugin.
### Switching between VAD and manual turns
### Using the Qwen3 web UI
| Mode | Backend | UI flag | Submit a turn |
| --- | --- | --- | --- |
| Without VAD | Default Qwen turn deployment | `--stt` (default) | Press **Send turn** |
| With VAD | Server VAD configuration and Silero artifact | `--vad` | Pause after speaking |
The shared UI supports both explicit turns and Server VAD. Use the manual UI
command above with the turn-based deployment, or start the VAD deployment above
and launch the UI with `--vad`.
Stop the existing UI process before starting another on the same port, then
refresh the browser and reconnect. To use STT, start a backend with the default
turn deployment and restart the UI with `--stt`. The duplex backend exposes
Realtime duplex and ChatCompletion; it does not expose the legacy STT handler.
To enable VAD on a default backend, first restart that backend with the overlay.
Make `--backend` match the actual backend port; the UI and backend use separate
ports. `Connection refused` means the target backend is unavailable: check its
startup log and `/health` before connecting.
refresh the browser and reconnect. To enable VAD on a default backend, restart
that backend with the overlay. Make `--backend` match the actual backend port;
the UI and backend use separate ports. `Connection refused` means the target
backend is unavailable: check its startup log and `/health` before connecting.
VAD connects with `duplex=1` and sends nested `session.audio.input` configuration
with `create_response: true` and `interrupt_response: true`. It sends playback
@@ -210,8 +197,7 @@ native camera protocol, where Stage 0 interleaves frames at unit boundaries.
This is a supported subset, not full OpenAI Realtime API compatibility.
`duplex=1`, `overlap_policy`, `playback.ack`, and `session.close` remain vLLM-Omni
extensions. The legacy `--stt` mode is unchanged and is not the standard Realtime
conversation API. This demo does not add WebRTC, semantic VAD, or tool calling.
extensions. This demo does not add WebRTC, semantic VAD, or tool calling.
## Shared host options
@@ -219,11 +205,14 @@ The shared entry point can select a profile explicitly:
```bash
python -m examples.online_serving.realtime_web --profile qwen3-turn \
--backend ws://127.0.0.1:8091 --stt --port 7863
--backend ws://127.0.0.1:8091 --vad --port 7863
```
This Qwen3 UI profile requires the server-VAD deployment above.
- `--backend` / `--ws-backend`: backend WebSocket origin (not the full `/v1/realtime` URL).
- `--model`: override the wrapper's model name.
- `--manual` / `--vad`: Qwen turn control; manual turns are the default.
- `--public-realtime-url`: optional browser-visible WebSocket URL; otherwise the
static host proxies `/v1/realtime` on the same origin.
- `--host`, `--port`: UI bind address and port (default port 7862).
@@ -245,7 +234,7 @@ pytest -m cpu tests/model_executor/models/qwen3_omni/test_duplex_plugin.py \
tests/engine/test_duplex_orchestrator.py
```
These CPU tests cover profile messages, two STT turns, VAD terminal/drain ordering,
These CPU tests cover profile message serialization, VAD terminal/drain ordering,
backpressure, native capture during playback, server config injection, and shared
worklets. Backend regressions cover current audio inclusion in history-based
prompts, delayed playback ACK ordering, and no extra generation during silence
@@ -2,24 +2,24 @@
'use strict';
const profiles = root.OmniRealtimeProfiles;
profiles['qwen3-turn'] = (config) => {
const vad = config.adapter === 'vad';
const vad = config.turnMode === 'vad';
let interruptOnSpeech = vad;
let imageSeq = 0;
let imageItems = [];
// The engine refuses a ninth image (or 4 MiB of them) for the whole
// session, so the camera track retires its own oldest frame instead.
const MAX_IMAGE_ITEMS = 8;
const hint = 'This Qwen deploy needs session_mode: duplex and Server VAD support; use --stt or enable Server VAD.';
const hint = 'This Qwen deployment needs session_mode: duplex with Server VAD enabled.';
return {
title: 'Qwen3-Omni Voice', eyebrow: vad ? 'Engine-owned duplex call' : 'Turn-based realtime call',
title: 'Qwen3-Omni Voice', eyebrow: vad ? 'Engine-owned duplex call' : 'Turn-based Realtime call',
policy: vad ? 'Server VAD • interruptible responses' : 'Send turn • one response at a time',
description: vad
? 'Pause to send your turn; speak again to interrupt. Microphone upload continues during playback. Requires the Qwen duplex plugin and Silero. Optional camera frames accompany each spoken turn.'
: 'Speak, then press Send turn. Microphone upload pauses while the model answers. Each turn uses a fresh connection; conversation history is not carried between turns. No barge-in or camera input.',
: 'Speak, then press Send turn. Microphone upload pauses while the model answers. Conversation history stays on the Realtime session. No barge-in or camera input.',
waiting: 'Waiting for you', camera: vad, cameraMaxDimension: 448, playbackAck: vad, clientCommit: !vad,
waitForResponseDone: true, halfDuplex: !vad, deduplicateTranscript: true, closeSession: vad, reconnectEachTurn: !vad,
inputSampleRate: vad ? 24000 : 16000,
readyEvent: vad ? 'session.updated' : 'session.created', instructions: vad, sendIntervalMs: 200,
waitForResponseDone: true, halfDuplex: !vad, deduplicateTranscript: true, closeSession: vad,
inputSampleRate: 24000,
readyEvent: 'session.updated', instructions: true, sendIntervalMs: 200,
presets: { assistant: 'You are a helpful assistant. Answer clearly and concisely.' },
url(config, location) {
const url = profiles.url(config, location);
@@ -33,22 +33,17 @@
initialMessages(config, instructions) {
imageItems = [];
interruptOnSpeech = vad;
if (!vad) return [
{ type: 'session.update', model: config.model },
{ type: 'input_audio_buffer.commit', final: false },
];
const session = { model: config.model, overlap_policy: 'barge_in_on_speech', audio: { input: {
const session = { model: config.model, audio: { input: {
format: { type: 'audio/pcm', rate: 24000 },
turn_detection: { type: 'server_vad', threshold: 0.5, prefix_padding_ms: 300,
silence_duration_ms: 500, create_response: true, interrupt_response: true },
turn_detection: vad ? { type: 'server_vad', threshold: 0.5, prefix_padding_ms: 300,
silence_duration_ms: 500, create_response: true, interrupt_response: true } : null,
} } };
if (vad) session.overlap_policy = 'barge_in_on_speech';
else session.type = 'realtime';
if (instructions) session.instructions = instructions;
return [{ type: 'session.update', session }];
},
append(audio) {
return vad ? { type: 'input_audio_buffer.append', audio }
: { type: 'input_audio_buffer.append', audio, format: 'pcm16', sample_rate_hz: 16000 };
},
append: (audio) => ({ type: 'input_audio_buffer.append', audio }),
// Qwen is a turn model: a frame is a picture added to the conversation,
// not a track interleaved into the audio. That is exactly the OpenAI
// Realtime image interface, so the camera sends conversation items
@@ -68,7 +63,10 @@
} });
return messages;
},
commitMessages: () => vad ? [] : [{ type: 'input_audio_buffer.commit', final: true }],
commitMessages: () => vad ? [] : [
{ type: 'input_audio_buffer.commit' },
{ type: 'response.create' },
],
ack: (responseId, playedMs) => vad ? { type: 'playback.ack', response_id: responseId,
item_id: `item_${responseId}`, played_ms: playedMs, committed_ms: playedMs } : null,
mapEvent(event) {
@@ -87,30 +85,28 @@
return { kind: 'interrupt' };
}
if (event.type === 'playback.acknowledged') return { kind: 'ack', committedMs: (event.event || event).committed_ms || 0 };
if (event.type === 'response.output_text.delta' || event.type === 'transcription.delta') {
// On the shipped STT route transcription.* is model-generated text,
// not a separate ASR transcript of the user. See realtime_connection.py.
if (event.type === 'response.output_text.delta') {
return { kind: 'text', role: 'assistant', channel: 'text', text: event.delta || '' };
}
if (event.type === 'response.output_text.done' || event.type === 'transcription.done') {
if (event.type === 'response.output_text.done') {
return { kind: 'text-final', role: 'assistant', channel: 'text', text: event.text || '' };
}
if (vad && (event.type === 'output_audio_buffer.cleared' ||
(event.type === 'response.done' && event.response?.status === 'cancelled'))) {
return { kind: 'interrupt', responseId: event.response_id || event.response?.id || null };
}
if (event.type === 'input_audio_buffer.committed') return { kind: 'begin' };
if (vad && event.type === 'input_audio_buffer.committed') return { kind: 'begin' };
if (!vad && event.type === 'input_audio_buffer.cleared') return { kind: 'ignore' };
if (event.type === 'input_audio_buffer.cleared') return { kind: 'backpressure', message: 'Input cleared. Model is answering; please wait.' };
if (!vad && event.type === 'response.output_audio.done') return { kind: 'done' };
const action = profiles.event(event);
if (action.kind === 'error') {
if (action.code === 'input_backpressure') return { ...action, kind: 'backpressure', message: 'Input buffer is full; please repeat dropped speech after the response.' };
if (action.code === 'server_vad_initialization_failed') action.message = 'Server VAD could not load Silero. Configure the server artifact or restart this UI with --stt.';
if (action.code === 'server_vad_initialization_failed') action.message = 'Server VAD could not load Silero. Configure the server artifact and restart the UI.';
else if (vad && action.code === 'unsupported') action.message += ` ${hint}`;
}
return action;
},
connectionHint: vad ? hint : 'Check the Qwen /v1/realtime STT endpoint and model name.',
connectionHint: vad ? hint : 'Check the Qwen turn-based /v1/realtime deployment and model name.',
};
};
})(globalThis);
@@ -74,12 +74,12 @@ def build_app(
public_realtime_url: str | None = None,
ref_audio: str | None = None,
profile: str = "minicpm-native",
adapter: str = "stt",
turn_mode: str = "manual",
) -> FastAPI:
if profile not in {"minicpm-native", "qwen3-turn", "aura-ptt"}:
raise ValueError(f"Unknown profile: {profile}")
if adapter not in {"stt", "vad"}:
raise ValueError(f"Unknown adapter: {adapter}")
if turn_mode not in {"manual", "vad"}:
raise ValueError(f"Unknown turn mode: {turn_mode}")
if profile == "qwen3-turn" and ref_audio:
raise ValueError("Qwen3 profile does not accept --ref-audio")
if profile == "aura-ptt" and ref_audio:
@@ -116,7 +116,7 @@ def build_app(
{
"model": model,
"profile": profile,
"adapter": adapter,
"turnMode": turn_mode,
"realtimePath": public_realtime_url or "v1/realtime",
"refAudio": ref_audio_uri,
"appVersion": app_version,
@@ -190,9 +190,9 @@ def main(default_profile: str = "minicpm-native") -> None:
parser.add_argument("--profile", choices=("minicpm-native", "qwen3-turn", "aura-ptt"), default=default_profile)
parser.add_argument("--ws-backend", "--backend", dest="ws_backend")
mode = parser.add_mutually_exclusive_group()
mode.add_argument("--stt", dest="adapter", action="store_const", const="stt")
mode.add_argument("--vad", dest="adapter", action="store_const", const="vad")
parser.set_defaults(adapter="stt")
mode.add_argument("--manual", dest="turn_mode", action="store_const", const="manual")
mode.add_argument("--vad", dest="turn_mode", action="store_const", const="vad")
parser.set_defaults(turn_mode="manual")
parser.add_argument(
"--public-realtime-url",
help="Browser-visible ws:// or wss:// Realtime URL; defaults to the same-origin proxy.",
@@ -225,7 +225,7 @@ def main(default_profile: str = "minicpm-native") -> None:
public_realtime_url=args.public_realtime_url,
ref_audio=args.ref_audio,
profile=args.profile,
adapter=args.adapter,
turn_mode=args.turn_mode,
),
host=args.host,
port=args.port,