diff --git a/docs/design/fullduplex.md b/docs/design/fullduplex.md index 2311ee1cf..87fb1f79f 100644 --- a/docs/design/fullduplex.md +++ b/docs/design/fullduplex.md @@ -66,7 +66,8 @@ The checkpoint keeps these verified contracts: - scheduler data-plane append over a resumable request; - stale epoch/turn/response fencing; - `/v1/duplex` and OpenAI Realtime projection; -- optional OpenAI `server_vad` turn detection backed by per-session Silero VAD; +- optional OpenAI `server_vad` turn detection backed by a process-shared Silero + model and per-session VAD state; - existing JoyVL behavior. The checkpoint does not claim: @@ -652,7 +653,7 @@ EOS decisions advance the model conversation. Explicit Realtime clients may still use `input_audio_buffer.commit` to create a conversation item. The translator validates that wire input before producing a -commit carrying `realtime_item_id`. Native auto-response may already have +commit carrying `item_id`. Native auto-response may already have streamed those PCM samples into the runtime, so the runner accepts a validated commit even when no runtime-side chunk remains. A truly empty explicit buffer continues to return `input_audio_buffer_empty`. @@ -677,8 +678,7 @@ multi-turn runtime evidence. During auto-response overlap, `preserve_realtime_input` distinguishes "do not append this silent chunk to the native buffer" from "clear the open Realtime -item". Silent overlap no longer discards earlier user PCM. A separate explicit -`server_vad` policy is documented in the Realtime contract above. +item". Silent overlap no longer discards earlier user PCM. The first chunk of one overlapping input item also reserves its target model turn. A later Realtime commit uses that reserved identity even if response EOS diff --git a/docs/serving/realtime_duplex_api.md b/docs/serving/realtime_duplex_api.md index 30021e107..d37d02291 100644 --- a/docs/serving/realtime_duplex_api.md +++ b/docs/serving/realtime_duplex_api.md @@ -1,17 +1,17 @@ # Realtime Duplex API vLLM-Omni serves full-duplex speech models over a WebSocket endpoint, -`/v1/realtime?duplex=1`, that speaks the OpenAI Realtime vocabulary plus a +`/v1/realtime`, that speaks the OpenAI Realtime vocabulary plus a set of duplex extensions: the model decides when to listen and when to speak, the user can talk over the assistant, playback progress is reported back so history stays honest, and a dropped connection can resume the same session. This page covers how to run a duplex deployment, how to drive it from Python with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract. -The endpoint is served for models that ship a duplex plugin — currently -MiniCPM-o 4.5, PersonaPlex, and Nemotron VoiceChat — and only for deploy -configurations that declare `session_mode: duplex`. The runtime architecture -is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md). +The handler is enabled by either a model-native `session_mode: duplex` +deployment or an explicit `duplex_session` configuration for turn-based +Server VAD. The runtime architecture is described in +[Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md). ## Quick Start @@ -27,7 +27,7 @@ vllm-omni serve openbmb/MiniCPM-o-4_5 \ `vllm_omni/deploy/minicpmo_4_5.yaml` declares `session_mode: duplex` and `duplex_session.max_sessions: 4`, which mounts the WebSocket routes -`ws://:8099/v1/realtime?duplex=1` (this page) and `ws://:8099/v1/duplex` +`ws://:8099/v1/realtime` (this page) and `ws://:8099/v1/duplex` (the native dialect for raw-protocol clients) next to the usual HTTP API. PersonaPlex and Nemotron VoiceChat use `vllm_omni/deploy/personaplex.yaml` and `vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml` in the same way. @@ -240,7 +240,7 @@ on the same helpers. ## API Reference -This section is the complete wire contract of `/v1/realtime?duplex=1`: how +This section is the complete wire contract of `/v1/realtime`: how the dialect relates to the OpenAI Realtime protocol, which surfaces each model's capability flags gate, a catalogue of every client-to-server and server-to-client event, and a JSON example for each. `DuplexClient` speaks @@ -253,10 +253,11 @@ its protocol tests. ### Endpoint and transport -The endpoint is `ws(s):///v1/realtime?duplex=1`; the `duplex` query -parameter (`1`, `true`, or `on`) selects the duplex session handler, and the -route is mounted only for deployments whose deploy configuration declares -`session_mode: duplex`. Optional query parameters are `model`, `session_id`, +The endpoint is `ws(s):///v1/realtime`. Bare connections select the +session handler when the deployment declares either `session_mode: duplex` +or `duplex_session`. `?duplex=1` remains a compatibility alias, while +`?duplex=0` selects the legacy Realtime handler. Optional query parameters +are `model`, `session_id`, `autostart` (`0` means resume-only), `resume`, and `native_duplex` (`minicpmo45_native_duplex` is accepted as a deprecated alias and folded into the canonical name). Every message is one JSON object per WebSocket @@ -342,9 +343,14 @@ Semantic divergences hidden behind shared names: before any commit; a commit may end in `response.listen` and no response, and the model may open a response with no commit at all. OpenAI: commit ⇒ item, `response.create` ⇒ exactly one response. -- `turn_detection.interrupt_response=false` is rejected; - `create_response` is ignored; `semantic_vad` is unsupported; the VAD - runs per session (Silero) and implies `overlap_policy=barge_in_on_speech`. +- With `turn_detection={"type":"server_vad"}`, omitted `interrupt_response` + defaults to `false` on turn-based models such as Qwen3-Omni, enabling + endpointing and automatic responses without interruption. Explicit `true` + is rejected there. Model-native duplex keeps the `true` default and rejects + `false`; its VAD implies `overlap_policy=barge_in_on_speech`. + This capability-dependent default differs from OpenAI's `true` default; + the effective configuration is returned in `session.created` / `session.updated`. + `semantic_vad` is unsupported. - `rate_limits.updated` is always an empty list (compatibility only). - Cancellation never reuses a `response_id`; truncation is driven by `playback.ack` as well as `conversation.item.truncate`. @@ -1201,9 +1207,10 @@ Realtime projector renames and fans out these events before they reach a barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in or audio truncation; camera frames are consumed only by MiniCPM-o 4.5; tool calls are produced only by Nemotron VoiceChat. -- `turn_detection` supports only `server_vad` with `interrupt_response=true`; - `semantic_vad`, `interrupt_response=false`, and `create_response` are not - supported. Server VAD requires a model-native session. +- `turn_detection` supports `server_vad` or `null`, not `semantic_vad`. + Turn-based models support endpointing without interruption, including + `create_response=false` for explicit response creation. Model-native duplex + requires `interrupt_response=true` and `create_response=true`. - `input_audio_transcription` and `input_audio_noise_reduction` are accepted and echoed but no separate transcription or noise-reduction stage runs; transcripts come from the model. diff --git a/examples/online_serving/qwen3_omni/README.md b/examples/online_serving/qwen3_omni/README.md index 926a58682..ef2367412 100644 --- a/examples/online_serving/qwen3_omni/README.md +++ b/examples/online_serving/qwen3_omni/README.md @@ -264,7 +264,7 @@ python examples/online_serving/openai_chat_completion_client_for_multimodal_gene #### Realtime WebSocket client (`openai_realtime_client.py`) -[`openai_realtime_client.py`](./openai_realtime_client.py) connects to **`ws://:/v1/realtime`**, streams a local WAV as **PCM16 mono @ 16 kHz** in fixed-size chunks (OpenAI-style `input_audio_buffer.append` / `commit`), and receives **`response.audio.delta`** (incremental PCM for the reply) plus **`transcription.*`** events. By default it concatenates audio deltas and writes **`--output-wav`** (model output is typically **24 kHz**). Optional **`--delta-dump-dir`** saves each delta as `delta_000001.wav`, … for debugging. +[`openai_realtime_client.py`](./openai_realtime_client.py) connects to **`ws://:/v1/realtime`**, streams a local WAV as **PCM16 mono @ 16 kHz** in fixed-size chunks, and receives **`response.audio.*`**, **`response.output_text.*`**, and transcript events. By default it uses explicit `input_audio_buffer.commit`; pass **`--server-vad`** to let the server detect speech endpoints and commit the turn automatically. The client concatenates audio deltas and writes **`--output-wav`** (model output is typically **24 kHz**). Optional **`--delta-dump-dir`** saves each delta as `delta_000001.wav`, … for debugging. Streaming input works well for translation-style use cases; if the Thinker runs while input is still incomplete, consider limiting **`max_tokens`** in your session / server defaults to avoid over-generation. @@ -282,6 +282,7 @@ python openai_realtime_client.py \ --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ --input-wav /path/to/input_16k_mono.wav \ --output-wav realtime_output.wav \ + --server-vad \ --delta-dump-dir ./rt_delta_wavs ``` @@ -297,15 +298,32 @@ python openai_realtime_client.py \ | `--chunk-ms` | `200` | Size of each uploaded audio chunk (milliseconds of audio) | | `--send-delay-ms` | `0` | Delay between chunk sends (simulate realtime upload) | | `--delta-dump-dir` | *(optional)* | Directory to write per-`response.audio.delta` WAV files | +| `--server-vad` | disabled | Enable server-side Silero VAD instead of sending explicit commits | | `--num-requests` | `1` | Number of sequential sessions (see `--concurrency`) | | `--concurrency` | `1` | Max concurrent WebSocket sessions when `--num-requests` > 1 | -Ensure the server is running, for example: +Server VAD uses the pinned Silero v6.2 ONNX artifact from `istupakov/silero-vad-onnx` and never downloads model files while processing a session. Make the artifact available in the Hugging Face cache before startup, or configure a local artifact in the deployment YAML: + +```yaml +duplex_session: + server_vad_model_path: /models/silero_vad.onnx +``` + +Start from the bundled Qwen3-Omni deployment YAML, add the fields above at the top level, and serve that complete configuration: ```bash -vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 +cp vllm_omni/deploy/qwen3_omni_moe.yaml /path/to/qwen3_omni_server_vad.yaml +# Edit /path/to/qwen3_omni_server_vad.yaml and add duplex_session. +vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \ + --omni \ + --port 8091 \ + --deploy-config /path/to/qwen3_omni_server_vad.yaml ``` +Keep Qwen's default `session_mode: turn`; `duplex_session` enables the Realtime handler without changing scheduler +semantics. Bare `/v1/realtime` selects that handler. The client uses `?duplex=0` only for the existing non-Server-VAD +wire flow; `?duplex=1` remains a supported compatibility alias. + The Python client supports the following command-line arguments: - `--query-type` (or `-q`): Query type (default: `use_video`). Options: `text`, `use_audio`, `use_image`, `use_video` diff --git a/examples/online_serving/qwen3_omni/openai_realtime_client.py b/examples/online_serving/qwen3_omni/openai_realtime_client.py index 79e30a3f5..8a6871580 100644 --- a/examples/online_serving/qwen3_omni/openai_realtime_client.py +++ b/examples/online_serving/qwen3_omni/openai_realtime_client.py @@ -1,9 +1,12 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + """Realtime client for vLLM-Omni /v1/realtime (audio + text events). This client: 1) Reads a local WAV file (must be mono, 16-bit PCM, 16kHz), 2) Streams PCM16 chunks to /v1/realtime with OpenAI-style events, -3) Receives response.audio.* and transcription.* events, +3) Receives response.audio.*, response.output_text.*, and transcript events, 4) Saves synthesized audio to an output WAV file and optional text file. By default each ``response.audio.delta`` is treated as an **incremental PCM** @@ -18,6 +21,7 @@ Usage: --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ --input-wav input_16k_mono.wav \ --output-wav realtime_output.wav \ + --server-vad \ --delta-dump-dir ./rt_delta_wavs Dependencies: @@ -32,6 +36,7 @@ import base64 import json import wave from pathlib import Path +from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit try: import websockets @@ -70,6 +75,14 @@ def _write_wav_pcm16(path: Path, pcm16_bytes: bytes, sample_rate_hz: int) -> Non wf.writeframes(pcm16_bytes) +def _with_realtime_route(url: str, *, server_vad: bool) -> str: + parts = urlsplit(url) + query = [(key, value) for key, value in parse_qsl(parts.query, keep_blank_values=True) if key != "duplex"] + if not server_vad: + query.append(("duplex", "0")) + return urlunsplit(parts._replace(query=urlencode(query))) + + async def run_client( url: str, model: str, @@ -79,6 +92,7 @@ async def run_client( chunk_ms: int, send_delay_ms: int, delta_dump_dir: Path | None, + server_vad: bool, request_idx: int = 1, total_requests: int = 1, ) -> None: @@ -92,23 +106,44 @@ async def run_client( delta_index = 0 text_chunks: list[str] = [] final_text: str = "" + final_audio_transcript: str = "" if delta_dump_dir is not None: delta_dump_dir.mkdir(parents=True, exist_ok=True) + # Keep the CLI mode authoritative even if the input URL already contains + # an explicit duplex query parameter. + url = _with_realtime_route(url, server_vad=server_vad) async with websockets.connect(url, max_size=64 * 1024 * 1024) as ws: # 1) Validate model. - await ws.send( - json.dumps( - { - "type": "session.update", + if server_vad: + session_update = { + "type": "session.update", + "session": { "model": model, - } - ) - ) + "audio": { + "input": { + "format": {"type": "audio/pcm", "rate": 16_000}, + "turn_detection": { + "type": "server_vad", + "threshold": 0.5, + "prefix_padding_ms": 300, + "silence_duration_ms": 500, + "create_response": True, + "interrupt_response": False, + }, + } + }, + }, + } + else: + # Preserve the existing non-duplex wire format. + session_update = {"type": "session.update", "model": model} + await ws.send(json.dumps(session_update)) - # 2) Start generation once (non-final commit). - await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": False})) + # 2) Preserve the legacy explicit-commit flow unless Server VAD owns endpointing. + if not server_vad: + await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": False})) # 3) Stream audio chunks. for i in range(0, len(pcm16), chunk_bytes): @@ -124,8 +159,20 @@ async def run_client( if send_delay_ms > 0: await asyncio.sleep(send_delay_ms / 1000.0) - # 4) Final commit closes input stream. - await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": True})) + # 4) Finalize the turn explicitly, or append silence for Server VAD endpointing. + if server_vad: + trailing_silence = bytes(16_000 * 2) + for i in range(0, len(trailing_silence), chunk_bytes): + await ws.send( + json.dumps( + { + "type": "input_audio_buffer.append", + "audio": base64.b64encode(trailing_silence[i : i + chunk_bytes]).decode("utf-8"), + } + ) + ) + else: + await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": True})) # 5) Receive server events until audio done. while True: @@ -144,7 +191,7 @@ async def run_client( sr = event.get("sample_rate_hz") if isinstance(sr, int) and sr > 0: output_sample_rate = sr - audio_b64 = event.get("audio", "") + audio_b64 = event.get("delta" if server_vad else "audio", "") if audio_b64: pcm_delta = base64.b64decode(audio_b64) incremental_pcm_parts.append(pcm_delta) @@ -158,14 +205,14 @@ async def run_client( ) continue - if event_type == "transcription.delta": + if event_type in {"response.output_text.delta", "transcription.delta"}: delta = event.get("delta", "") if delta: text_chunks.append(delta) print(delta, end="", flush=True) continue - if event_type == "transcription.done": + if event_type in {"response.output_text.done", "transcription.done"}: final_text = event.get("text", "") or "".join(text_chunks) usage = event.get("usage") final_text_with_tag = f"Final transcription: {final_text}" @@ -176,7 +223,16 @@ async def run_client( print(f"{log_prefix}text usage: {usage}") continue + if event_type == "response.audio_transcript.done": + final_audio_transcript = event.get("transcript", "") + continue + if event_type == "response.audio.done": + if not server_vad: + break + continue + + if event_type == "response.done" and server_vad: break if event_type == "error": @@ -191,7 +247,7 @@ async def run_client( print(f"{log_prefix}Saved realtime audio to: {output_wav} (incremental chunks joined)") if output_text is not None: - text_to_save = final_text if final_text else "".join(text_chunks) + text_to_save = final_text or "".join(text_chunks) or final_audio_transcript output_text.parent.mkdir(parents=True, exist_ok=True) output_text.write_text(text_to_save, encoding="utf-8") print(f"{log_prefix}Saved realtime text to: {output_text}") @@ -213,6 +269,7 @@ async def run_clients_concurrent( chunk_ms: int, send_delay_ms: int, delta_dump_dir: Path | None, + server_vad: bool, num_requests: int, concurrency: int, ) -> None: @@ -235,6 +292,7 @@ async def run_clients_concurrent( chunk_ms=chunk_ms, send_delay_ms=send_delay_ms, delta_dump_dir=per_delta_dir, + server_vad=server_vad, request_idx=index, total_requests=num_requests, ) @@ -283,6 +341,11 @@ def main() -> None: default=None, help="If set, each response.audio.delta is saved as delta_NNNNNN.wav under this directory", ) + parser.add_argument( + "--server-vad", + action="store_true", + help="Let the server detect speech endpoints instead of sending input_audio_buffer.commit", + ) parser.add_argument("--num-requests", type=int, default=1, help="Total number of requests to send") parser.add_argument( "--concurrency", @@ -309,6 +372,7 @@ def main() -> None: chunk_ms=args.chunk_ms, send_delay_ms=args.send_delay_ms, delta_dump_dir=args.delta_dump_dir, + server_vad=args.server_vad, ) ) else: @@ -322,6 +386,7 @@ def main() -> None: chunk_ms=args.chunk_ms, send_delay_ms=args.send_delay_ms, delta_dump_dir=args.delta_dump_dir, + server_vad=args.server_vad, num_requests=args.num_requests, concurrency=concurrency, ) diff --git a/pyproject.toml b/pyproject.toml index ce1eb9a13..3ba902977 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -77,7 +77,6 @@ dev = [ # Provides top-level cosyvoice2; do not co-install stepaudio2-minicpmo, # whose top-level stepaudio2 package conflicts with this distribution. "step-audio2==1.0.0", # Step-Audio2 / MiniCPM-o 4.5 Token2Wav (flashcosyvoice / hyperpyyaml) - "silero-vad==6.1", # Realtime server-VAD duplex E2E ] demo = [ @@ -87,9 +86,6 @@ demo = [ "cn2an>=0.5.22", # Chinese number normalization for TTS (e.g. "1次" -> "一次") ] -server-vad = [ - "silero-vad==6.1", -] forced-aligner = [ "qwen-asr>=0.0.6", # faithful Japanese/Korean word segmentation for the TTS forced aligner diff --git a/tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py b/tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py index 5b35c4b53..c3c6fd90a 100644 --- a/tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py +++ b/tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py @@ -30,10 +30,27 @@ from tests.e2e.online_serving.run_minicpmo_realtime_duplex_soft_interrupt import run_soft_interrupt, ) from tests.helpers.mark import hardware_test +from vllm_omni.entrypoints.duplex.server_vad import ( + SILERO_VAD_FILENAME, + SILERO_VAD_REPO_ID, + SILERO_VAD_REVISION, +) pytestmark = [pytest.mark.full_model, pytest.mark.omni] +@pytest.fixture(scope="module") +def _cached_server_vad_artifact() -> None: + """Cache the pinned ONNX artifact before the Realtime server starts.""" + from huggingface_hub import hf_hub_download + + hf_hub_download( + repo_id=SILERO_VAD_REPO_ID, + filename=SILERO_VAD_FILENAME, + revision=SILERO_VAD_REVISION, + ) + + @hardware_test(res={"cuda": "H100", "npu": "A3"}, num_cards=1) @pytest.mark.parametrize("omni_server", SERVER_PARAMS, indirect=True) def test_duplex_admission_and_expiry_reaper(omni_server, tmp_path: Path) -> None: @@ -97,7 +114,7 @@ def test_duplex_soft_interrupt(omni_server, tmp_path: Path) -> None: @hardware_test(res={"cuda": "H100"}, num_cards=1) @pytest.mark.parametrize("omni_server", SERVER_PARAMS, indirect=True) -def test_duplex_server_vad_hard_interrupt(omni_server) -> None: +def test_duplex_server_vad_hard_interrupt(_cached_server_vad_artifact, omni_server) -> None: result = asyncio.run( run_server_vad_interrupt( SimpleNamespace( diff --git a/tests/engine/duplex/test_duplex_deploy_config.py b/tests/engine/duplex/test_duplex_deploy_config.py index 800070efb..e9168729c 100644 --- a/tests/engine/duplex/test_duplex_deploy_config.py +++ b/tests/engine/duplex/test_duplex_deploy_config.py @@ -33,6 +33,7 @@ def test_duplex_session_runtime_defaults_are_typed_and_immutable(tmp_path) -> No assert deploy.duplex_session.max_pending_turns_per_session == 4 assert deploy.duplex_session.max_sessions == 1 assert deploy.duplex_session.completed_append_cache_size == 256 + assert deploy.duplex_session.server_vad_model_path is None with pytest.raises(FrozenInstanceError): deploy.duplex_session.idle_ttl_s = 1.0 # type: ignore[misc] @@ -46,6 +47,29 @@ def test_duplex_session_runtime_accepts_disabled_idle_expiry(tmp_path) -> None: assert deploy.duplex_session.idle_ttl_s is None +def test_duplex_session_runtime_accepts_local_server_vad_model(tmp_path) -> None: + deploy_path = tmp_path / "duplex.yaml" + deploy_path.write_text( + "duplex_session:\n server_vad_model_path: /models/silero_vad.onnx\nstages: []\n", + encoding="utf-8", + ) + + deploy = load_deploy_config(deploy_path) + + assert deploy.duplex_session.server_vad_model_path == "/models/silero_vad.onnx" + + +def test_duplex_session_runtime_rejects_empty_server_vad_model_path(tmp_path) -> None: + deploy_path = tmp_path / "duplex.yaml" + deploy_path.write_text( + "duplex_session:\n server_vad_model_path: ''\nstages: []\n", + encoding="utf-8", + ) + + with pytest.raises(ValueError, match="server_vad_model_path"): + load_deploy_config(deploy_path) + + @pytest.mark.parametrize( ("name", "value"), [ diff --git a/tests/engine/test_single_stage_mode.py b/tests/engine/test_single_stage_mode.py index cacda1f60..675a0650b 100644 --- a/tests/engine/test_single_stage_mode.py +++ b/tests/engine/test_single_stage_mode.py @@ -376,14 +376,26 @@ class TestSingleStageModeDetection: mocker: MockerFixture, *, stage_cfgs: list[Any] | None = None, + resolved_config_path: str = "/fake/path", + patch_deploy_config: bool = True, **kwargs: Any, ) -> AsyncOmniEngine: mock_stage_configs = stage_cfgs or [_make_stage_cfg(0)] + if patch_deploy_config: + mocker.patch.object( + StageConfigFactory, + "get_pipeline_config", + return_value=None, + ) + mocker.patch( + "vllm_omni.engine.async_omni_engine.load_deploy_config", + return_value=SimpleNamespace(duplex_session=DuplexSessionRuntimeConfig()), + ) mocker.patch.object( AsyncOmniEngine, "_resolve_stage_configs", - return_value=("/fake/path", mock_stage_configs), + return_value=(resolved_config_path, mock_stage_configs), ) mocker.patch.object(AsyncOmniEngine, "_bootstrap_orchestrator") mock_thread_cls = mocker.patch("threading.Thread") @@ -445,6 +457,8 @@ class TestSingleStageModeDetection: engine = self._make_engine_no_thread( mocker, deploy_config="/fake/duplex.yaml", + resolved_config_path="/fake/duplex.yaml", + patch_deploy_config=False, ) get_pipeline_config.assert_called_once_with( @@ -455,6 +469,30 @@ class TestSingleStageModeDetection: load_deploy_config.assert_called_once_with("/fake/duplex.yaml") assert engine.duplex_session_config is duplex_session + def test_auto_discovered_deploy_loads_duplex_runtime_config( + self, + mocker: MockerFixture, + ) -> None: + deploy_path = "/resolved/qwen3_omni_moe.yaml" + duplex_session = DuplexSessionRuntimeConfig(server_vad_model_path="/models/silero_vad.onnx") + mocker.patch( + "vllm_omni.engine.async_omni_engine.StageConfigFactory.get_pipeline_config", + return_value=None, + ) + load_deploy_config = mocker.patch( + "vllm_omni.engine.async_omni_engine.load_deploy_config", + return_value=SimpleNamespace(duplex_session=duplex_session), + ) + + engine = self._make_engine_no_thread( + mocker, + resolved_config_path=deploy_path, + patch_deploy_config=False, + ) + + load_deploy_config.assert_called_once_with(deploy_path) + assert engine.duplex_session_config is duplex_session + def test_single_stage_mode_without_stage_id_has_no_filter(self, mocker: MockerFixture): engine = self._make_engine_no_thread( mocker, @@ -518,6 +556,10 @@ class TestEndpointRestrictionsTrustRemoteCode: StageConfigFactory.try_infer_model_type.cache_clear() def _make_engine_no_thread(self, mocker: MockerFixture, **kwargs: Any) -> AsyncOmniEngine: + mocker.patch( + "vllm_omni.engine.async_omni_engine.load_deploy_config", + return_value=SimpleNamespace(duplex_session=DuplexSessionRuntimeConfig()), + ) mocker.patch.object( AsyncOmniEngine, "_resolve_stage_configs", diff --git a/tests/entrypoints/duplex/test_runtime_adapter_boundary.py b/tests/entrypoints/duplex/test_runtime_adapter_boundary.py index dc7fffccd..cb90e4bd5 100644 --- a/tests/entrypoints/duplex/test_runtime_adapter_boundary.py +++ b/tests/entrypoints/duplex/test_runtime_adapter_boundary.py @@ -61,20 +61,19 @@ if loaded: assert result.returncode == 0, result.stderr -def test_generic_handler_requires_explicit_serving_runtime_adapter() -> None: +def test_generic_handler_supports_adapterless_chat_fallback() -> None: repo_root = Path(__file__).resolve().parents[3] script = """ from types import SimpleNamespace from vllm_omni.entrypoints.duplex.serving import OmniDuplexSessionHandler -chat_service = SimpleNamespace(engine_client=SimpleNamespace()) -try: - OmniDuplexSessionHandler(chat_service=chat_service) -except ValueError as exc: - if "serving runtime adapter" not in str(exc).lower(): - raise -else: +chat_service = SimpleNamespace( + engine_client=SimpleNamespace(), + model_config=SimpleNamespace(model="test-model"), +) +handler = OmniDuplexSessionHandler(chat_service=chat_service) +if handler._serving_runtime_adapter is not None: raise SystemExit("generic handler silently selected a model serving adapter") """ diff --git a/tests/entrypoints/openai_api/test_api_server_guards.py b/tests/entrypoints/openai_api/test_api_server_guards.py index 707b4b961..71f93f6dc 100644 --- a/tests/entrypoints/openai_api/test_api_server_guards.py +++ b/tests/entrypoints/openai_api/test_api_server_guards.py @@ -568,6 +568,48 @@ def test_websocket_routes_emit_stable_unavailable_frames_and_close(path: str, pa websocket.receive_text() +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("duplex_query", "expected_handler"), + [ + (None, "duplex"), + ("1", "duplex"), + ("true", "duplex"), + ("on", "duplex"), + ("0", "legacy"), + ("false", "legacy"), + ], +) +async def test_realtime_route_defaults_to_configured_duplex_handler( + monkeypatch, duplex_query: str | None, expected_handler: str +) -> None: + calls: list[str] = [] + + class _DuplexHandler: + async def handle_realtime_session(self, _websocket) -> None: + calls.append("duplex") + + class _LegacyConnection: + async def handle_connection(self) -> None: + calls.append("legacy") + + monkeypatch.setattr(api_server, "RealtimeConnection", lambda _websocket, _serving: _LegacyConnection()) + query_params = {} if duplex_query is None else {"duplex": duplex_query} + websocket = SimpleNamespace( + app=SimpleNamespace( + state=SimpleNamespace( + openai_serving_duplex=_DuplexHandler(), + openai_serving_realtime=object(), + ) + ), + query_params=query_params, + ) + + await api_server.realtime_websocket(websocket) + + assert calls == [expected_handler] + + def test_health_without_engine_returns_stable_unhealthy_response() -> None: """Lock ``/health`` with no engine initialized. diff --git a/tests/entrypoints/openai_api/test_audio_format.py b/tests/entrypoints/openai_api/test_audio_format.py index e47885d9b..a9bf76319 100644 --- a/tests/entrypoints/openai_api/test_audio_format.py +++ b/tests/entrypoints/openai_api/test_audio_format.py @@ -163,6 +163,8 @@ class TestStreamingAudioResampler: chunked = StreamingAudioResampler(24000, 8000) pieces = [chunked.process(chunk) for chunk in np.split(waveform, [137, 2048, 9001, 17003])] + # Bounded FIR history must own its data, not retain the last large chunk. + assert chunked._history.base is None pieces.append(chunked.process(np.empty(0), final=True)) actual = np.concatenate(pieces) @@ -193,9 +195,15 @@ class TestStreamingAudioResampler: output_rms = np.sqrt(np.mean(output[100:-100] ** 2)) assert output_rms < input_rms * 0.01 - def test_rejects_non_integer_downsampling_ratio(self): - with pytest.raises(ValueError, match="integer downsampling ratio"): - StreamingAudioResampler(24000, 16000) + def test_supports_rational_downsampling_ratio(self): + waveform = np.sin(np.linspace(0, 200 * np.pi, 24000, endpoint=False)).astype(np.float32) + resampler = StreamingAudioResampler(24000, 16000) + + output = resampler.process(waveform, final=True) + reference = torchaudio.functional.resample(torch.from_numpy(waveform), 24000, 16000).numpy() + + assert output.shape == (16000,) + np.testing.assert_allclose(output[100:-100], reference[100:-100], atol=2e-3, rtol=1e-3) def test_rejects_multichannel_audio(self): resampler = StreamingAudioResampler(24000, 8000) diff --git a/tests/entrypoints/openai_api/test_duplex_handler.py b/tests/entrypoints/openai_api/test_duplex_handler.py index c69bc5c47..d3bbc5d01 100644 --- a/tests/entrypoints/openai_api/test_duplex_handler.py +++ b/tests/entrypoints/openai_api/test_duplex_handler.py @@ -5,8 +5,10 @@ from __future__ import annotations import asyncio import base64 +import io import json import struct +import wave from types import SimpleNamespace from typing import Any @@ -30,8 +32,8 @@ from vllm_omni.engine.duplex.control_client import DuplexControlRequestError from vllm_omni.engine.duplex.lease import DuplexLeaseActivity from vllm_omni.engine.duplex.messages import DuplexFence, DuplexSessionLifecycleMessage from vllm_omni.engine.duplex.runtime import duplex_resource_request_id -from vllm_omni.entrypoints.duplex import vad as realtime_vad from vllm_omni.entrypoints.duplex.protocol import ( + DuplexCapabilities, DuplexOverlapPolicy, DuplexPlaybackCommitPolicy, DuplexSession, @@ -69,6 +71,11 @@ from vllm_omni.outputs import OmniRequestOutput from vllm_omni.outputs.duplex import attach_duplex_output_decision pytestmark = [pytest.mark.core_model, pytest.mark.cpu] +INVALID_VALIDATED_SAMPLE_RATES = ( + pytest.param(16_000.5, id="fractional"), + pytest.param(float("inf"), id="infinite"), + pytest.param(float("nan"), id="nan"), +) def test_native_input_append_supports_explicit_session_opt_out(): @@ -327,6 +334,46 @@ class FakeChatService: return SimpleNamespace(audio_data=f"wav-{int(audio_obj.audio_tensor.shape[0])}") +class TurnBasedFakeChatService(FakeChatService): + """Chat-fallback service without a model-native duplex adapter.""" + + duplex_serving_adapter_path = None + + def __init__(self, engine_client: FakeEngineClient) -> None: + super().__init__(engine_client) + self.requests: list[Any] = [] + + async def create_chat_completion(self, request, raw_request=None): + del raw_request + self.requests.append(request) + self.seen_request_ids.append(request.request_id) + + async def _gen(): + yield "data: [DONE]\n\n" + + return _gen() + + +class GateChatService(FakeChatService): + def __init__(self, engine_client: FakeEngineClient) -> None: + super().__init__(engine_client) + self.first_response_gate = asyncio.Event() + self.response_count = 0 + + async def create_chat_completion(self, request, raw_request=None): + del raw_request + self.seen_request_ids.append(request.request_id) + self.response_count += 1 + response_number = self.response_count + + async def _gen(): + if response_number == 1: + await self.first_response_gate.wait() + yield "data: [DONE]\n\n" + + return _gen() + + def _test_data_plane() -> MiniCPMO45DataPlaneSession: def encode_audio( audio: object, @@ -405,6 +452,105 @@ class TimedWebSocket: return [m.get("type", "") for m in self.sent] +class FakeServerVADBackend: + frame_samples = 160 + + def __init__(self, probabilities: list[float]) -> None: + self.probabilities = probabilities + self.calls = 0 + + def new_state(self) -> int: + return 0 + + def infer(self, frame: np.ndarray, state: object) -> tuple[float, object]: + del frame + self.calls += 1 + index = int(state) + probability = self.probabilities[index] if index < len(self.probabilities) else 0.0 + return probability, index + 1 + + +class FakeServerVADProvider: + def __init__(self, backend: FakeServerVADBackend) -> None: + self.backend = backend + + def get(self) -> FakeServerVADBackend: + return self.backend + + +class FailingServerVADProvider: + def __init__(self) -> None: + self.calls = 0 + + def get(self) -> FakeServerVADBackend: + self.calls += 1 + raise RuntimeError("server VAD backend unavailable") + + +class FailOnceServerVADBackend(FakeServerVADBackend): + def __init__(self, probabilities: list[float]) -> None: + super().__init__(probabilities) + self._failed = False + + def infer(self, frame: np.ndarray, state: object) -> tuple[float, object]: + if not self._failed and int(state) == 2: + self._failed = True + raise RuntimeError("server VAD inference failed") + return super().infer(frame, state) + + +def _server_vad_turn_detection(*, create_response: bool = True) -> dict[str, Any]: + return { + "type": "server_vad", + "prefix_padding_ms": 10, + "silence_duration_ms": 20, + "create_response": create_response, + "interrupt_response": False, + } + + +def _server_vad_session_update( + turn_detection: dict[str, Any] | None, + *, + event_id: str | None = None, + sample_rate_hz: int = 16_000, +) -> dict[str, Any]: + event: dict[str, Any] = { + "type": "session.update", + "session": { + "model": "test-model", + "audio": { + "input": { + "format": {"type": "audio/pcm", "rate": sample_rate_hz}, + "turn_detection": turn_detection, + } + }, + }, + } + if event_id is not None: + event["event_id"] = event_id + return event + + +def _server_vad_audio_append( + frame_count: int = 5, + *, + event_id: str | None = None, + is_speech: bool | None = None, + sample_rate_hz: int = 16_000, +) -> dict[str, Any]: + sample_count = frame_count * 160 * sample_rate_hz // 16_000 + event: dict[str, Any] = { + "type": "input_audio_buffer.append", + "audio": base64.b64encode(np.zeros(sample_count, dtype=" None: + if data.get("type") == "response.done": + ws.put({"type": "session.close"}) + + ws = TimedWebSocket(on_send=on_send) + if not at_creation: + ws.put(_server_vad_session_update(_server_vad_turn_detection())) + update = _server_vad_session_update(_server_vad_turn_detection(), event_id="bad-vad-audio-format") + update["session"]["audio"]["input"].update(input_update) + ws.put(update) + if not at_creation: + ws.put(_server_vad_audio_append()) + handler = OmniDuplexSessionHandler( + chat_service=TurnBasedFakeChatService(FakeEngineClient()), + server_vad_backend_provider=FakeServerVADProvider(FakeServerVADBackend([0.0, 0.9, 0.8, 0.0, 0.0])), + ) + + await handler.handle_realtime_session(ws) # type: ignore[arg-type] + + (error,) = [event["error"] for event in ws.sent if event.get("type") == "error"] + assert error["code"] == "unsupported_turn_detection" + assert error["event_id"] == "bad-vad-audio-format" + assert "mono PCM16" in error["message"] + assert ("session.created" in ws.sent_types()) is not at_creation + assert ("response.done" in ws.sent_types()) is not at_creation + + +@pytest.mark.asyncio +@pytest.mark.parametrize("sample_rate_hz", [16_000, 24_000]) +@pytest.mark.parametrize("create_response", [False, True], ids=["commit-only", "minimal-config"]) +@pytest.mark.parametrize("at_creation", [True, False], ids=["create", "update"]) +async def test_realtime_server_vad_commits_without_client_commit( + sample_rate_hz: int, create_response: bool, at_creation: bool +): + def on_send(ws: TimedWebSocket, data: dict[str, Any]) -> None: + terminal_event = "response.done" if create_response else "input_audio_buffer.committed" + if data.get("type") == terminal_event: + ws.put({"type": "session.close"}) + + ws = TimedWebSocket(on_send=on_send) + if not at_creation: + ws.put(_server_vad_session_update(None, sample_rate_hz=sample_rate_hz)) + ws.put( + _server_vad_session_update( + {"type": "server_vad"} if create_response else _server_vad_turn_detection(create_response=False), + sample_rate_hz=sample_rate_hz, + ) + ) + # Cover the default 500 ms silence timeout, plus streaming FIR right context. + ws.put(_server_vad_audio_append(54, sample_rate_hz=sample_rate_hz)) + engine = FakeEngineClient() + chat_service = TurnBasedFakeChatService(engine) + handler = OmniDuplexSessionHandler( + chat_service=chat_service, + server_vad_backend_provider=FakeServerVADProvider(FakeServerVADBackend([0.0, 0.9, 0.8, 0.0, 0.0])), + ) + + await handler.handle_realtime_session(ws) # type: ignore[arg-type] + + event_types = ws.sent_types() + assert "error" not in event_types + effective_session = [ + event["session"] for event in ws.sent if event["type"] in {"session.created", "session.updated"} + ][-1] + assert effective_session["turn_detection"]["interrupt_response"] is False + assert effective_session["audio"]["input"]["turn_detection"] == effective_session["turn_detection"] + assert effective_session["overlap_policy"] == "listen_only" + assert event_types.index("input_audio_buffer.speech_started") < event_types.index( + "input_audio_buffer.speech_stopped" + ) + assert event_types.index("input_audio_buffer.speech_stopped") < event_types.index("input_audio_buffer.committed") + assert event_types.index("input_audio_buffer.committed") < event_types.index("conversation.item.added") + assert event_types.index("conversation.item.added") < event_types.index("conversation.item.done") + assert "conversation.item.created" not in event_types + committed = next(event for event in ws.sent if event.get("type") == "input_audio_buffer.committed") + added = next(event for event in ws.sent if event.get("type") == "conversation.item.added") + done = next(event for event in ws.sent if event.get("type") == "conversation.item.done") + assert committed["item_id"] == added["item"]["id"] == done["item"]["id"] + assert event_types.count("response.created") == int(create_response) + assert len(chat_service.seen_request_ids) == int(create_response) + if create_response: + assert event_types.index("input_audio_buffer.committed") < event_types.index("response.created") + assert event_types.index("response.created") < event_types.index("response.done") + assert engine.opened == [] + assert engine.signals == [] + assert engine.closed == [] + + +@pytest.mark.asyncio +async def test_realtime_server_vad_create_response_false_accepts_response_create(): + def on_send(ws: TimedWebSocket, data: dict[str, Any]) -> None: + if data.get("type") == "input_audio_buffer.committed": + ws.put({"type": "response.create"}) + if data.get("type") == "response.done": + ws.put({"type": "session.close"}) + + ws = TimedWebSocket(on_send=on_send) + ws.put(_server_vad_session_update(_server_vad_turn_detection(create_response=False))) + ws.put(_server_vad_audio_append()) + engine = FakeEngineClient() + chat_service = TurnBasedFakeChatService(engine) + handler = OmniDuplexSessionHandler( + chat_service=chat_service, + server_vad_backend_provider=FakeServerVADProvider(FakeServerVADBackend([0.0, 0.9, 0.8, 0.0, 0.0])), + ) + + await handler.handle_realtime_session(ws) # type: ignore[arg-type] + + event_types = ws.sent_types() + assert event_types.index("input_audio_buffer.committed") < event_types.index("response.created") + assert event_types.index("response.created") < event_types.index("response.done") + assert event_types.count("response.created") == 1 + assert len(chat_service.seen_request_ids) == 1 + assert engine.opened == engine.signals == engine.closed == [] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("sample_rate_hz", [16_000, 24_000]) +async def test_realtime_server_vad_complete_audio_item_bypasses_vad(sample_rate_hz: int): + def on_send(ws: TimedWebSocket, data: dict[str, Any]) -> None: + item = data.get("item") + if ( + data.get("type") == "conversation.item.done" + and isinstance(item, dict) + and item.get("id") == "item-complete-audio" + ): + ws.put({"type": "response.create"}) + if data.get("type") == "response.done": + ws.put({"type": "session.close"}) + + ws = TimedWebSocket(on_send=on_send) + ws.put(_server_vad_session_update(_server_vad_turn_detection(), sample_rate_hz=sample_rate_hz)) + samples = np.arange(sample_rate_hz // 10, dtype=np.float32) + pcm16 = np.rint(0.5 * np.sin(2 * np.pi * 1_000 * samples / sample_rate_hz) * 32_767).astype(" None: + if data.get("type") == "response.done": + ws.put({"type": "session.close"}) + + ws = TimedWebSocket(on_send=on_send) + ws.put(_server_vad_session_update(_server_vad_turn_detection())) + first_append = _server_vad_audio_append(frame_count=2) + first_append["audio"] = base64.b64encode(np.full(320, 12_000, dtype=" None: + nonlocal committed_count, response_count + if data.get("type") == "input_audio_buffer.committed": + committed_count += 1 + if data.get("type") == "error" and data.get("error", {}).get("code") == "input_backpressure": + chat_service.first_response_gate.set() + if data.get("type") == "response.created": + response_count += 1 + if response_count == 2: + ws.put({"type": "session.close"}) + + ws = TimedWebSocket(on_send=on_send) + ws.put(_server_vad_session_update(_server_vad_turn_detection())) + for _ in range(3): + ws.put(_server_vad_audio_append()) + handler = OmniDuplexSessionHandler( + chat_service=chat_service, + server_vad_backend_provider=FakeServerVADProvider(FakeServerVADBackend([0.0, 0.9, 0.8, 0.0, 0.0] * 3)), + ) + + await handler.handle_realtime_session(ws) # type: ignore[arg-type] + + assert committed_count == 2 + assert response_count == 2 + event_types = ws.sent_types() + committed_indices = [index for index, event in enumerate(event_types) if event == "input_audio_buffer.committed"] + response_created_indices = [index for index, event in enumerate(event_types) if event == "response.created"] + assert committed_indices[1] < event_types.index("response.done") < response_created_indices[1] + backpressure = [ + event + for event in ws.sent + if event.get("type") == "error" and event.get("error", {}).get("code") == "input_backpressure" + ] + assert len(backpressure) == 1 + + @pytest.mark.asyncio async def test_native_session_update_rejects_client_runtime_config(): engine = FakeEngineClient() @@ -1110,6 +1967,7 @@ async def test_minicpmo_native_session_update_requires_ref_audio_before_enabling def test_personaplex_candidate_update_does_not_leak_minicpmo_ref_audio_check(): handler = OmniDuplexSessionHandler( chat_service=SimpleNamespace(engine_client=SimpleNamespace()), + served_model_name="test-model", serving_runtime_adapter=PersonaPlexServingRuntimeAdapter(lambda *_: None), ) session = DuplexSession( @@ -1124,6 +1982,51 @@ def test_personaplex_candidate_update_does_not_leak_minicpmo_ref_audio_check(): assert error is None +@pytest.mark.parametrize( + ("native", "turn_detection", "expected_code"), + [ + (False, {"type": "server_vad", "interrupt_response": False}, None), + (False, {"type": "server_vad"}, None), + (False, {"type": "server_vad", "interrupt_response": True}, "server_vad_requires_native_duplex"), + (True, {"type": "server_vad", "interrupt_response": False}, "unsupported_turn_detection"), + (True, {"type": "server_vad"}, None), + ], + ids=[ + "turn-based-endpointing", + "turn-based-default-endpointing", + "turn-based-explicit-interruption", + "native-non-interrupting", + "native-default-interruption", + ], +) +def test_server_vad_runtime_capability_matrix( + native: bool, + turn_detection: dict[str, object], + expected_code: str | None, +): + handler = OmniDuplexSessionHandler( + chat_service=TurnBasedFakeChatService(FakeEngineClient()), + ) + capabilities = minicpmo45_native_capabilities() if native else DuplexCapabilities() + session = DuplexSession( + "sid-server-vad-capability", + DuplexSessionConfig(), + capabilities=capabilities, + ) + candidate = DuplexSessionConfig.from_event({"session": {"turn_detection": turn_detection}}) + handler._resolve_server_vad_defaults(session, candidate, {"turn_detection": turn_detection}) + + error = handler._runtime_session_candidate_update_error(session, candidate) + + assert (error or {}).get("code") == expected_code + if expected_code is None: + assert candidate.server_vad is not None + assert candidate.server_vad.interrupt_response is native + assert candidate.overlap_policy == ("barge_in_on_speech" if native else "listen_only") + elif expected_code == "server_vad_requires_native_duplex": + assert "set interrupt_response=false" in error["error"] + + @pytest.mark.asyncio async def test_minicpmo_native_session_update_rejects_instructions_after_native_context_locked(): engine = FakeEngineClient() @@ -1550,6 +2453,23 @@ def test_duplex_session_playback_commit_uses_multi_delta_audio_text_marks(): assert session.history == (committed,) +def test_duplex_session_preserves_response_history_order_when_user_item_arrives_during_generation(): + session = DuplexSession(session_id="sid-history-order", config=DuplexSessionConfig()) + first_user = {"role": "user", "content": "first turn"} + next_user = { + "role": "user", + "content": [{"type": "audio_url", "audio_url": {"url": "data:audio/wav;base64,AAAA"}}], + } + session.append_history_message(first_user) + session.begin_response() + session.append_assistant_text("first answer") + session.append_history_message(next_user) + + assistant = session.end_response(commit_text=True) + + assert session.history == (first_user, assistant, next_user) + + @pytest.mark.asyncio async def test_duplex_session_actor_preserves_wire_order_before_control(): ws = TimedWebSocket() @@ -1616,7 +2536,7 @@ def _native_realtime_session_update( } -def _server_vad_update(event_id: str, **session: object) -> dict[str, object]: +def _native_server_vad_update(event_id: str, **session: object) -> dict[str, object]: return { "type": "session.update", "event_id": event_id, @@ -2356,7 +3276,7 @@ def test_auto_response_force_barge_in_has_no_waiting_turn_variant(): ) -def test_duplex_endpoint_requires_explicit_session_mode_duplex(): +def test_duplex_endpoint_requires_explicit_realtime_configuration(): assert should_enable_duplex_endpoint(None) is False assert should_enable_duplex_endpoint([]) is False assert should_enable_duplex_endpoint([SimpleNamespace(session_mode="turn")]) is False @@ -2388,6 +3308,22 @@ stage_args: assert should_enable_duplex_endpoint([], config_path=str(config_path)) is True +def test_duplex_endpoint_supports_inherited_duplex_session(tmp_path): + base_path = tmp_path / "base.yaml" + base_path.write_text( + """ +session_mode: turn +duplex_session: {} +stages: [] +""", + encoding="utf-8", + ) + config_path = tmp_path / "deploy.yaml" + config_path.write_text("base_config: base.yaml\n", encoding="utf-8") + + assert should_enable_duplex_endpoint([], config_path=str(config_path)) is True + + def test_duplex_handler_splits_data_plane_audio_list_into_deltas(): import torch @@ -4254,7 +5190,7 @@ async def test_duplex_handler_runtime_open_failure_is_reported_to_client(): chat_service = FakeChatService(engine) handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1) ws = TimedWebSocket() - ws.put(_session_create("sid-open-fail")) + ws.put(_native_session_create("sid-open-fail")) ws.put({"type": "session.close"}) await handler.handle_session(ws) @@ -4289,7 +5225,7 @@ async def test_duplex_handler_preserves_typed_admission_error(): idle_timeout_s=1, ) ws = TimedWebSocket() - ws.put(_session_create("sid-admission-rejected")) + ws.put(_native_session_create("sid-admission-rejected")) await handler.handle_session(ws) @@ -4903,13 +5839,37 @@ async def test_cancel_active_native_data_plane_request_aborts_stage_request_id() assert "audio.cancelled" in ws.sent_types() +@pytest.mark.asyncio +async def test_cancel_chat_fallback_response_aborts_request_and_task(): + engine = FakeEngineClient() + handler = OmniDuplexSessionHandler(chat_service=TurnBasedFakeChatService(engine)) + session = DuplexSession("sid-chat-cancel", DuplexSessionConfig()) + response_id = session.begin_response() + session.bind_request("chatcmpl-cancel") + ws = TimedWebSocket() + task = asyncio.create_task(asyncio.sleep(60)) + try: + cancelled = await handler._cancel_active_response(session, task, ws.send_json, reason="response.cancel") + + assert cancelled is True + assert task.cancelled() + assert engine.internal_abort_batches == [["chatcmpl-cancel"]] + assert "error" not in ws.sent_types() + (event,) = [event for event in ws.sent if event["type"] == "audio.cancelled"] + assert event["response_id"] == response_id + assert event["reason"] == "response.cancel" + finally: + task.cancel() + await asyncio.gather(task, return_exceptions=True) + + @pytest.mark.asyncio async def test_duplex_handler_explicit_close_closes_runtime_once_with_client_reason(): engine = FakeEngineClient() chat_service = FakeChatService(engine) handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1) ws = TimedWebSocket() - ws.put(_session_create("sid-close")) + ws.put(_native_session_create("sid-close")) ws.put({"type": "session.close"}) await handler.handle_session(ws) @@ -5600,7 +6560,7 @@ async def test_duplex_handler_idle_timeout_close_does_not_emit_runtime_control() chat_service = FakeChatService(engine) handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=0.1) ws = TimedWebSocket() - ws.put(_session_create("sid-disconnect")) + ws.put(_native_session_create("sid-disconnect")) await handler.handle_session(ws) @@ -5633,7 +6593,7 @@ async def test_duplex_handler_runtime_close_failure_is_reported_without_closed_a chat_service = FakeChatService(engine) handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1) ws = TimedWebSocket() - ws.put(_session_create("sid-close-fail")) + ws.put(_native_session_create("sid-close-fail")) ws.put({"type": "session.close"}) await handler.handle_session(ws) @@ -5662,7 +6622,7 @@ async def test_duplex_handler_control_close_failure_is_reported_without_closed_a chat_service = FakeChatService(engine) handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1) ws = TimedWebSocket() - ws.put(_session_create("sid-control-close-fail")) + ws.put(_native_session_create("sid-control-close-fail")) ws.put({"type": "session.close"}) await handler.handle_session(ws) @@ -5701,7 +6661,7 @@ async def test_turn_signal_input_cancel_uses_epoch_fence_transition(): idle_timeout_s=1, ) ws = TimedWebSocket() - ws.put(_session_create("sid-turn-signal-cancel")) + ws.put(_native_session_create("sid-turn-signal-cancel")) ws.put({"type": "input.text.append", "text": "discard me"}) ws.put({"type": "turn.signal", "event": "input.cancel"}) ws.put({"type": "session.close"}) @@ -5731,59 +6691,6 @@ async def test_duplex_handler_local_turn_signal_does_not_round_trip_runtime(): assert "runtime_signal_failed" not in {m.get("code") for m in ws.sent} -@pytest.mark.asyncio -async def test_duplex_barge_in_aborts_active_response_when_runtime_signal_fails(): - engine = FakeEngineClient(fail_signal_events={"barge_in"}) - chat_service = FakeChatService(engine) - handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1) - - def on_send(ws: TimedWebSocket, data: dict[str, Any]) -> None: - if data.get("type") == "response.created": - ws.put({"type": "input.cancel", "reason": "test_barge_in"}) - - ws = TimedWebSocket(on_send=on_send) - ws.put(_session_create("sid-barge-signal-fail")) - ws.put({"type": "input.text.append", "text": "hello"}) - ws.put({"type": "input.commit"}) - - await handler.handle_session(ws) - - assert "audio.cancelled" in ws.sent_types() - assert engine.aborted == ["chatcmpl-duplex-sid-barge-signal-fail-0-1"] - assert engine.internal_abort_batches == [["chatcmpl-duplex-sid-barge-signal-fail-0-1"]] - error = next(m for m in ws.sent if m.get("type") == "error") - assert error["code"] == "runtime_signal_failed" - - -@pytest.mark.asyncio -async def test_duplex_handler_surfaces_stage_unsupported_result_to_client(): - control_result = { - "operation": "open", - "session_id": "sid-unsupported", - "ok": True, - "unsupported_count": 1, - "error_count": 0, - "stage_results": [ - { - "stage_id": 0, - "replica_id": 0, - "result": {"supported": False, "reason": "not implemented"}, - } - ], - } - engine = FakeEngineClient(open_result=control_result) - chat_service = FakeChatService(engine) - handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1) - ws = TimedWebSocket() - ws.put(_session_create("sid-unsupported")) - ws.put({"type": "session.close"}) - - await handler.handle_session(ws) - - created = next(m for m in ws.sent if m.get("type") == "session.created") - assert created["runtime_control"]["unsupported_count"] == 1 - - @pytest.mark.asyncio async def test_minicpmo_native_duplex_append_without_format_defaults_to_pcm16(): engine = FakeEngineClient() @@ -5842,6 +6749,38 @@ async def test_minicpmo_native_duplex_rejects_invalid_sample_rate_without_append assert error["code"] == "bad_event" +@pytest.mark.asyncio +@pytest.mark.parametrize("sample_rate_hz", INVALID_VALIDATED_SAMPLE_RATES) +async def test_turn_based_server_vad_rejects_invalid_direct_sample_rate_without_closing_session( + sample_rate_hz: object, +): + backend = FakeServerVADBackend([0.9]) + handler = OmniDuplexSessionHandler( + chat_service=TurnBasedFakeChatService(FakeEngineClient()), + config_timeout_s=0.1, + idle_timeout_s=1, + server_vad_backend_provider=FakeServerVADProvider(backend), + ) + ws = TimedWebSocket() + session_create = _session_create("sid-server-vad-invalid-rate") + session_create["session"]["turn_detection"] = _server_vad_turn_detection(create_response=False) + ws.put(session_create) + invalid_append = _server_vad_audio_append(frame_count=1) + invalid_append["sample_rate_hz"] = sample_rate_hz + ws.put(invalid_append) + ws.put(_server_vad_audio_append(frame_count=1)) + ws.put({"type": "session.close"}) + + await handler.handle_session(ws) + + errors = [event for event in ws.sent if event.get("type") == "error"] + assert [error["code"] for error in errors] == ["bad_audio"] + assert "internal_error" not in {error["code"] for error in errors} + assert "session.created" in ws.sent_types() + assert "session.closed" in ws.sent_types() + assert backend.calls == 1 + + @pytest.mark.asyncio async def test_native_input_commit_does_not_send_noop_runtime_signal(): engine = FakeEngineClient() @@ -6023,12 +6962,12 @@ async def test_duplex_handler_session_update_control_failure_rolls_back_config() chat_service = FakeChatService(engine) handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1) ws = TimedWebSocket() - ws.put(_session_create("sid-control-signal-fail")) + ws.put(_native_session_create("sid-control-signal-fail")) ws.put( { "type": "turn.signal", "event": "session.update", - "payload": {"instructions": "must roll back"}, + "payload": {"temperature": 0.2}, } ) ws.put({"type": "session.close"}) @@ -6041,7 +6980,7 @@ async def test_duplex_handler_session_update_control_failure_rolls_back_config() assert error["runtime_control"]["error_count"] == 1 signaled_config = engine.signal_session_configs[-1] assert signaled_config is not None - assert signaled_config["instructions"] == "must roll back" + assert signaled_config["temperature"] == 0.2 @pytest.mark.asyncio @@ -6059,19 +6998,24 @@ async def test_native_session_update_commits_config_only_after_runtime_ack(): return await super().signal_duplex_turn_async(session_id, **kwargs) engine = BlockingUpdateEngine() - handler = OmniDuplexSessionHandler(chat_service=FakeChatService(engine)) + handler = OmniDuplexSessionHandler( + chat_service=FakeChatService(engine), + config_timeout_s=0.1, + idle_timeout_s=1, + server_vad_backend_provider=FakeServerVADProvider(FakeServerVADBackend([])), + ) ws = TimedWebSocket() protocol = NativeRealtimeSessionProtocol(ws) # type: ignore[arg-type] ws.put(_native_realtime_session_update("sid-update-two-phase")) - ws.put(_server_vad_update("update-two-phase")) + ws.put(_native_server_vad_update("update-two-phase")) handler_task = asyncio.create_task(handler.handle_session(ws, realtime_protocol=protocol)) - await asyncio.wait_for(engine.update_started.wait(), timeout=1) + await asyncio.wait_for(engine.update_started.wait(), timeout=10) live_session = handler._registry.get("sid-update-two-phase") assert live_session is not None assert live_session.config.overlap_policy == DuplexOverlapPolicy.LISTEN_ONLY.value assert protocol._pending_turn_detection_update is not None - assert protocol._server_vad is None + assert protocol._turn_detection is None protocol.encode_outbound_event({"type": "error", "code": "unrelated", "error": "unrelated"}) assert protocol._pending_turn_detection_update is not None @@ -6082,7 +7026,7 @@ async def test_native_session_update_commits_config_only_after_runtime_ack(): updated = next(message for message in reversed(ws.sent) if message.get("type") == "session.updated") assert updated["session"]["overlap_policy"] == DuplexOverlapPolicy.BARGE_IN_ON_SPEECH.value - assert isinstance(protocol._server_vad, realtime_vad.SileroStreamingVAD) + assert protocol._turn_detection is not None @pytest.mark.parametrize( @@ -6090,9 +7034,14 @@ async def test_native_session_update_commits_config_only_after_runtime_ack(): [(True, "session_update_aborted"), (False, "instructions_update_unsupported")], ) @pytest.mark.asyncio -async def test_realtime_vad_update_rejection_does_not_stall(append_fails, expected_code): +async def test_native_server_vad_update_rejection_does_not_stall(append_fails, expected_code): append_result = {"operation": "append", "ok": False, "error_count": 1} if append_fails else None - handler = OmniDuplexSessionHandler(chat_service=FakeChatService(FakeEngineClient(append_result=append_result))) + handler = OmniDuplexSessionHandler( + chat_service=FakeChatService(FakeEngineClient(append_result=append_result)), + config_timeout_s=0.1, + idle_timeout_s=1, + server_vad_backend_provider=FakeServerVADProvider(FakeServerVADBackend([])), + ) ws = TimedWebSocket() protocol = NativeRealtimeSessionProtocol(ws) # type: ignore[arg-type] create = _native_realtime_session_update("sid-rejected-vad-update") @@ -6100,18 +7049,18 @@ async def test_realtime_vad_update_rejection_does_not_stall(append_fails, expect ws.put(create) if append_fails: ws.put({"type": "input_audio_buffer.append", "audio": _pcm_f32_b64(16_000), "format": "pcm_f32le"}) - update = _server_vad_update("rejected-vad-update") + update = _native_server_vad_update("rejected-vad-update") if not append_fails: update["session"]["instructions"] = "You are now a pirate." ws.put(update) ws.put({"type": "session.close"}) - await asyncio.wait_for(handler.handle_session(ws, realtime_protocol=protocol), timeout=2) + + await asyncio.wait_for(handler.handle_session(ws, realtime_protocol=protocol), timeout=10) errors = [message["error"] for message in ws.sent if isinstance(message.get("error"), dict)] error = next(error for error in errors if error.get("code") == expected_code) assert error["event_id"] == "rejected-vad-update" assert protocol._pending_turn_detection_update is None - assert protocol._server_vad is None @pytest.mark.asyncio @@ -6318,7 +7267,7 @@ async def test_duplex_handler_signal_unsupported_workers_with_data_plane_ack_is_ chat_service = FakeChatService(engine) handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1) ws = TimedWebSocket() - ws.put(_session_create("sid-control-signal-data-plane")) + ws.put(_native_session_create("sid-control-signal-data-plane")) ws.put({"type": "turn.signal", "event": "barge_in"}) ws.put({"type": "session.close"}) @@ -6645,6 +7594,91 @@ def test_minicpmo_native_duplex_explicit_barge_in_request_interrupts(): } +@pytest.mark.asyncio +async def test_native_server_vad_complete_audio_item_reaches_runtime(): + engine = FakeEngineClient() + backend = FakeServerVADBackend([0.9] * 10) + ws = TimedWebSocket() + create = _native_realtime_session_update("sid-native-complete-audio") + create["session"]["turn_detection"] = {"type": "server_vad"} + create["session"]["input_audio_format"] = "pcm16" + ws.put(create) + ws.put( + { + "type": "conversation.item.create", + "item": { + "id": "item-native-audio", + "type": "message", + "role": "user", + "content": [{"type": "input_audio", "audio": _pcm16_b64(2400)}], + }, + } + ) + ws.put({"type": "response.create"}) + ws.put({"type": "session.close"}) + handler = OmniDuplexSessionHandler( + chat_service=FakeChatService(engine), + server_vad_backend_provider=FakeServerVADProvider(backend), + ) + + await handler.handle_realtime_session(ws) # type: ignore[arg-type] + + assert any(final for _, _, _, final in engine.appended) + assert "error" not in ws.sent_types() + + +@pytest.mark.asyncio +async def test_native_server_vad_uses_shared_pipeline_and_emits_boundaries(): + backend = FakeServerVADBackend([0.9] * 10 + [0.0] * 50) + engine = FakeEngineClient() + + def commit_after_speech_stops(ws: TimedWebSocket, event: dict[str, Any]) -> None: + if event.get("type") == "input_audio_buffer.speech_stopped": + ws.put({"type": "input_audio_buffer.commit", "final": True}) + ws.put({"type": "response.create"}) + ws.put({"type": "session.close"}) + + handler = OmniDuplexSessionHandler( + chat_service=FakeChatService(engine), + config_timeout_s=0.1, + idle_timeout_s=1, + server_vad_backend_provider=FakeServerVADProvider(backend), + ) + ws = TimedWebSocket(on_send=commit_after_speech_stops) + create = _native_realtime_session_update("sid-native-shared-server-vad") + create["session"]["turn_detection"] = {"type": "server_vad"} + ws.put(create) + ws.put( + { + "type": "input_audio_buffer.append", + "audio": _pcm_f32_b64(60 * backend.frame_samples, value=0.0), + "format": "pcm_f32le", + "sample_rate_hz": 16_000, + } + ) + await asyncio.wait_for( + handler.handle_session(ws, realtime_protocol=NativeRealtimeSessionProtocol({})), + timeout=10, + ) + + boundary_events = [ + event + for event in ws.sent + if event.get("type") in {"input_audio_buffer.speech_started", "input_audio_buffer.speech_stopped"} + ] + assert [event["type"] for event in boundary_events] == [ + "input_audio_buffer.speech_started", + "input_audio_buffer.speech_stopped", + ] + assert boundary_events[0]["item_id"] == boundary_events[1]["item_id"] + created_session = next(event["session"] for event in ws.sent if event["type"] == "session.created") + assert created_session["audio"]["input"]["turn_detection"]["interrupt_response"] is True + assert created_session["overlap_policy"] == "barge_in_on_speech" + assert backend.calls == 60 + assert any(final for _, _, _, final in engine.appended) + assert not any(event.get("error", {}).get("code") == "server_vad_manual_commit_unsupported" for event in ws.sent) + + @pytest.mark.asyncio @pytest.mark.parametrize( "barge_in_event", @@ -7096,7 +8130,7 @@ async def test_minicpmo_native_auto_response_accepts_realtime_commit_after_strea { "type": "input_audio_buffer.commit", "final": True, - "realtime_item_id": "item-consumed-stream", + "item_id": "item-consumed-stream", } ) ws.put({"type": "session.close"}) @@ -7105,7 +8139,7 @@ async def test_minicpmo_native_auto_response_accepts_realtime_commit_after_strea assert not any(message.get("code") == "input_audio_buffer_empty" for message in ws.sent) committed = next(message for message in ws.sent if message.get("type") == "input.committed") - assert committed["realtime_item_id"] == "item-consumed-stream" + assert committed["item_id"] == "item-consumed-stream" assert committed["native_audio"] is True diff --git a/tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py b/tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py index 7664b2402..199bea0ed 100644 --- a/tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py +++ b/tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py @@ -3,10 +3,11 @@ """ E2E online tests for Qwen3-Omni /v1/realtime WebSocket (streaming PCM in, audio out). -Three scenarios: +Four scenarios: - Ready CI: async_chunk on, smoke only (no send delay, no accuracy check). - Merge CI: async_chunk on + send delay, full accuracy check. - Merge CI: async_chunk off, no send delay, full accuracy check. +- Server VAD: two turns without client commits. """ from __future__ import annotations @@ -28,11 +29,18 @@ from tests.helpers.media import ( generate_synthetic_audio, ) from tests.helpers.runtime import OmniServerParams -from tests.helpers.stage_config import get_deploy_config_path +from tests.helpers.stage_config import get_deploy_config_path, modify_stage_config +from vllm_omni.config.stage_config import load_deploy_config +from vllm_omni.entrypoints.duplex.server_vad import ( + SILERO_VAD_FILENAME, + SILERO_VAD_REPO_ID, + SILERO_VAD_REVISION, +) os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn" -MODEL = "Qwen/Qwen3-Omni-30B-A3B-Instruct" +MODEL = os.environ.get("VLLM_OMNI_TEST_QWEN3_OMNI_MODEL", "Qwen/Qwen3-Omni-30B-A3B-Instruct") +SERVER_VAD_MODEL_PATH = os.environ.get("VLLM_OMNI_TEST_SILERO_VAD_MODEL_PATH") # Synthetic input for realtime E2E (``generate_synthetic_audio``); distinct cache file per phrase. REALTIME_SYNTH_PHRASE_TEXT = ( @@ -48,6 +56,15 @@ SEND_DELAY_MS = 200 # CI overlay bakes in async_chunk: False and covers CUDA/ROCm/XPU via ``platforms:``. default_stage_config = get_deploy_config_path("ci/qwen3_omni_moe.yaml") +server_vad_stage_config = modify_stage_config( + default_stage_config, + { + "async_chunk": True, + "duplex_session": ( + {"server_vad_model_path": SERVER_VAD_MODEL_PATH} if SERVER_VAD_MODEL_PATH is not None else {} + ), + }, +) realtime_sync_server_params = [ pytest.param( @@ -72,6 +89,17 @@ realtime_async_chunk_server_params = [ ), ] +realtime_server_vad_server_params = [ + pytest.param( + OmniServerParams( + model=MODEL, + stage_config_path=server_vad_stage_config, + use_stage_cli=True, + ), + id="server_vad", + ), +] + def _pcm16_mono_16k_from_wav_bytes(wav_bytes: bytes) -> bytes: with wave.open(io.BytesIO(wav_bytes), "rb") as wf: @@ -183,6 +211,86 @@ async def _run_realtime_audio_roundtrip( } +async def _append_pcm16_chunks(ws, pcm16: bytes, chunk_bytes: int) -> None: + for offset in range(0, len(pcm16), chunk_bytes): + await ws.send( + json.dumps( + { + "type": "input_audio_buffer.append", + "audio": base64.b64encode(pcm16[offset : offset + chunk_bytes]).decode("utf-8"), + } + ) + ) + + +async def _receive_server_vad_turn(ws) -> list[dict]: + events: list[dict] = [] + while True: + message = await asyncio.wait_for(ws.recv(), timeout=600) + if isinstance(message, bytes): + continue + event = json.loads(message) + events.append(event) + if event.get("type") == "error": + raise AssertionError(f"WebSocket error: {event}") + if event.get("type") == "response.done": + return events + + +async def _run_server_vad_audio_roundtrips( + host: str, + port: int, + model: str, + pcm16: bytes, + *, + chunk_ms: int = 100, + turns: int = 2, +) -> list[list[dict]]: + chunk_bytes = max(16_000 * 2 // 1000 * chunk_ms, 2) + turn_events: list[list[dict]] = [] + + async with websockets.connect(f"ws://{host}:{port}/v1/realtime", max_size=64 * 1024 * 1024) as ws: + await ws.send( + json.dumps( + { + "type": "session.update", + "session": { + "model": model, + "audio": { + "input": { + "format": {"type": "audio/pcm", "rate": 16_000}, + "turn_detection": {"type": "server_vad"}, + } + }, + }, + } + ) + ) + + silence = bytes(16_000 * 2 * 3 // 2) + for _ in range(turns): + await _append_pcm16_chunks(ws, pcm16, chunk_bytes) + await _append_pcm16_chunks(ws, silence, chunk_bytes) + turn_events.append(await _receive_server_vad_turn(ws)) + + return turn_events + + +@pytest.fixture(scope="class") +def cached_silero_vad_artifact() -> str: + """Prepare the pinned artifact before the serving subprocess starts.""" + if SERVER_VAD_MODEL_PATH is not None: + return SERVER_VAD_MODEL_PATH + + from huggingface_hub import hf_hub_download + + return hf_hub_download( + repo_id=SILERO_VAD_REPO_ID, + filename=SILERO_VAD_FILENAME, + revision=SILERO_VAD_REVISION, + ) + + def _synthetic_pcm16_input( *, phrase_text: str = REALTIME_SYNTH_PHRASE_TEXT, @@ -296,6 +404,66 @@ class TestQwen3OmniRealtimeWebSocket: f"Expected an issue-like audio response longer than 5 seconds, got {output_duration_s:.2f}s" ) + @pytest.mark.advanced_model + @pytest.mark.omni + @hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=2) + @pytest.mark.parametrize("omni_server", realtime_server_vad_server_params, indirect=True) + def test_server_vad_multi_turn_without_client_commit( + self, + cached_silero_vad_artifact: str, + omni_server, + ) -> None: + """Two Qwen turns are endpointed without client commits.""" + assert cached_silero_vad_artifact + deploy = load_deploy_config(server_vad_stage_config) + assert deploy.session_mode == "turn" + assert deploy.async_chunk is True + assert deploy.duplex_session.server_vad_model_path == SERVER_VAD_MODEL_PATH + pcm16 = _synthetic_pcm16_input() + + turns = asyncio.run( + _run_server_vad_audio_roundtrips( + omni_server.host, + omni_server.port, + omni_server.model, + pcm16, + chunk_ms=100, + turns=2, + ) + ) + + assert len(turns) == 2 + required_sequence = [ + "input_audio_buffer.speech_started", + "input_audio_buffer.speech_stopped", + "input_audio_buffer.committed", + "response.created", + "response.audio.delta", + "response.audio.done", + "response.done", + ] + input_item_ids: list[str] = [] + response_ids: list[str] = [] + for events in turns: + event_types = [event["type"] for event in events] + positions = [event_types.index(event_type) for event_type in required_sequence] + assert positions == sorted(positions) + + started = next(event for event in events if event["type"] == "input_audio_buffer.speech_started") + stopped = next(event for event in events if event["type"] == "input_audio_buffer.speech_stopped") + committed = next(event for event in events if event["type"] == "input_audio_buffer.committed") + created = next(event for event in events if event["type"] == "response.created")["response"] + done = next(event for event in events if event["type"] == "response.done")["response"] + + assert started["item_id"] == stopped["item_id"] == committed["item_id"] + assert created["id"] == done["id"] + assert done["status"] == "completed" + input_item_ids.append(committed["item_id"]) + response_ids.append(created["id"]) + + assert len(set(input_item_ids)) == 2 + assert len(set(response_ids)) == 2 + @pytest.mark.advanced_model @pytest.mark.omni @hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=2) diff --git a/tests/entrypoints/openai_api/test_server_vad.py b/tests/entrypoints/openai_api/test_server_vad.py new file mode 100644 index 000000000..67476be73 --- /dev/null +++ b/tests/entrypoints/openai_api/test_server_vad.py @@ -0,0 +1,429 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +from __future__ import annotations + +import asyncio +import base64 +import io +import sys +import threading +import wave +import weakref +from concurrent.futures import ThreadPoolExecutor +from types import SimpleNamespace + +import numpy as np +import pytest + +from vllm_omni.entrypoints.duplex.protocol import ( + DuplexSession, + DuplexSessionConfig, +) +from vllm_omni.entrypoints.duplex.server_vad import ( + ServerVADConfig, + ServerVADFrame, + ServerVADPipeline, + SileroVADBackend, + SileroVADBackendProvider, + SpeechEndpointDecision, + ThresholdEndpointPolicy, + parse_session_turn_detection, +) + +pytestmark = [pytest.mark.core_model, pytest.mark.cpu] + + +class SequenceDetector: + frame_samples = 160 + + def __init__(self, probabilities: list[float]) -> None: + self.probabilities = probabilities + + def new_state(self) -> int: + return 0 + + def infer(self, frame: np.ndarray, state: object) -> tuple[float, object]: + index = int(state) + probability = self.probabilities[index] if index < len(self.probabilities) else 0.0 + return probability, index + 1 + + +def test_server_vad_config_defaults_and_validation(): + config = ServerVADConfig.from_value({"type": "server_vad"}) + + assert config.type == "server_vad" + assert config.threshold == 0.5 + assert config.prefix_padding_ms == 300 + assert config.silence_duration_ms == 500 + assert config.create_response is True + assert config.interrupt_response is None + assert "interrupt_response" not in config.as_dict() + assert config.min_speech_duration_ms is None + assert config.as_dict()["type"] == config.type + + interrupting = ServerVADConfig.from_value( + { + "type": "server_vad", + "interrupt_response": True, + "min_speech_duration_ms": 96, + } + ) + assert interrupting.interrupt_response is True + assert interrupting.min_speech_duration_ms == 96 + for value in (None, 0, "true"): + with pytest.raises(ValueError, match="interrupt_response must be a boolean"): + ServerVADConfig.from_value({"type": "server_vad", "interrupt_response": value}) + for enabled in (False, True): + _, aliased = parse_session_turn_detection( + { + "turn_detection": {"type": "server_vad"}, + "audio": {"input": {"turn_detection": {"type": "server_vad", "interrupt_response": enabled}}}, + } + ) + assert aliased is not None and aliased.interrupt_response is enabled + with pytest.raises(ValueError, match="Unknown"): + ServerVADConfig.from_value({"type": "server_vad", "semantic_eagerness": "high"}) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("prefix_padding_ms", [0, 10, 30]) +async def test_server_vad_pipeline_handles_arbitrary_chunk_boundaries(prefix_padding_ms: int): + detector = SequenceDetector([0.0, 0.9, 0.8, 0.0, 0.0]) + pipeline = ServerVADPipeline( + detector, + ServerVADConfig(prefix_padding_ms=prefix_padding_ms, silence_duration_ms=20), + ) + samples = np.zeros(detector.frame_samples * 5, dtype=np.float32) + + batches = [ + await pipeline.push(samples[:73]), + await pipeline.push(samples[73:431]), + await pipeline.push(samples[431:]), + ] + frames = [frame for batch in batches for frame in batch.frames] + + assert len(frames) == 5 + assert [index for index, frame in enumerate(frames) if frame.decision.speech_started] == [1] + assert [index for index, frame in enumerate(frames) if frame.decision.speech_stopped] == [4] + assert frames[1].decision.audio_start_ms == max(0, 10 - prefix_padding_ms) + assert frames[4].decision.audio_end_ms == 50 + + pipeline.reset() + reset_batch = await pipeline.push(samples) + assert [index for index, frame in enumerate(reset_batch.frames) if frame.decision.speech_started] == [1] + assert [index for index, frame in enumerate(reset_batch.frames) if frame.decision.speech_stopped] == [4] + # Keep session-relative timestamps, but never reference prefix audio + # discarded by reset at 50 ms. + assert reset_batch.frames[1].decision.audio_start_ms == max(50, 60 - prefix_padding_ms) + assert reset_batch.frames[4].decision.audio_end_ms == 100 + + +@pytest.mark.asyncio +async def test_server_vad_retained_audio_does_not_keep_input_chunk(): + pipeline = ServerVADPipeline(SequenceDetector([]), ServerVADConfig()) + session = DuplexSession("server-vad-memory", DuplexSessionConfig()) + samples = np.zeros(160 * 100 + 1, dtype=np.float32) + source_ref = weakref.ref(samples) + batch = await pipeline.push(samples) + session.append_server_vad_frame( + batch.frames[-1].samples, + speech_started=False, + speech_stopped=False, + prefix_samples=80, + ) + session.append_server_vad_frame( + batch.frames[-1].samples, + speech_started=True, + speech_stopped=False, + prefix_samples=80, + ) + del batch, samples + # Allow the completed to_thread callback to release its result, too. + await asyncio.sleep(0) + + assert source_ref() is None + assert pipeline.scratch_bytes == 4 + assert session.server_vad_utterance_bytes == (80 + 160) * 4 + + +@pytest.mark.asyncio +async def test_server_vad_pipeline_pcm16_resampling_is_split_invariant(): + sample_index = np.arange(2_400, dtype=np.int32) + source = ((sample_index * 7919) % 65_536 - 32_768).astype(" tuple[np.ndarray, list[SpeechEndpointDecision]]: + pipeline = ServerVADPipeline( + SequenceDetector([0.0] * 10), + ServerVADConfig(), + ) + frames: list[ServerVADFrame] = [] + offset = 0 + for chunk_size in chunk_sizes: + chunk = continuous_source[offset : offset + chunk_size] + batch = await pipeline.push_pcm16(chunk.tobytes(), source_sample_rate_hz=24_000) + frames.extend(batch.frames) + offset += chunk_size + assert offset == continuous_source.size + return np.concatenate([frame.samples for frame in frames]), [frame.decision for frame in frames] + + whole_samples, whole_decisions = await process([continuous_source.size]) + split_samples, split_decisions = await process([7, 13, 511, 512, 513, continuous_source.size - 1_556]) + + np.testing.assert_array_equal(split_samples, whole_samples) + assert whole_samples.size == 1_600 + assert split_decisions == whole_decisions + + +@pytest.mark.asyncio +async def test_server_vad_pipeline_reset_clears_resampler_and_source_rate_lock(): + pipeline = ServerVADPipeline( + SequenceDetector([0.0]), + ServerVADConfig(), + ) + + batch = await pipeline.push_pcm16( + np.asarray([30_000, -30_000], dtype=" None: + self.providers = providers + self.sess_options = sess_options + self.calls: list[dict[str, np.ndarray]] = [] + self.run_barrier: threading.Barrier | None = None + sessions.append(self) + + def get_inputs(self) -> list[SimpleNamespace]: + return [SimpleNamespace(name=name) for name in ("input", "state", "sr")] + + def run(self, _outputs: object, inputs: dict[str, np.ndarray]) -> list[np.ndarray]: + self.calls.append({name: np.array(value, copy=True) for name, value in inputs.items()}) + if self.run_barrier is not None: + self.run_barrier.wait(timeout=5) + return [ + np.asarray([[0.75]], dtype=np.float32), + np.full((2, 1, 128), len(self.calls), dtype=np.float32), + ] + + sessions: list[FakeInferenceSession] = [] + monkeypatch.setitem( + sys.modules, + "onnxruntime", + SimpleNamespace( + SessionOptions=FakeSessionOptions, + InferenceSession=FakeInferenceSession, + ), + ) + + model_path = tmp_path / "silero_vad.onnx" + model_path.write_bytes(b"fake-model") + backend = SileroVADBackend(model_path) + session = sessions[0] + + assert session.providers == ["CPUExecutionProvider"] + assert session.sess_options.inter_op_num_threads == 1 + assert session.sess_options.intra_op_num_threads == 1 + assert session.calls[0]["input"].shape == (1, 576) + assert session.calls[0]["state"].shape == (2, 1, 128) + np.testing.assert_array_equal(session.calls[0]["state"], np.zeros((2, 1, 128), dtype=np.float32)) + assert session.calls[0]["sr"].shape == () + assert session.calls[0]["sr"].item() == 16_000 + + state = backend.new_state() + first_frame = np.arange(backend.frame_samples, dtype=np.float32) + probability, state = backend.infer(first_frame, state) + first_call = session.calls[1] + + assert probability == pytest.approx(0.75) + np.testing.assert_array_equal( + first_call["input"][:, : backend.context_samples], + np.zeros((1, backend.context_samples), dtype=np.float32), + ) + np.testing.assert_array_equal( + first_call["input"][:, backend.context_samples :], + first_frame[None, :], + ) + np.testing.assert_array_equal(first_call["state"], np.zeros((2, 1, 128), dtype=np.float32)) + + second_frame = -first_frame + backend.infer(second_frame, state) + second_call = session.calls[2] + np.testing.assert_array_equal( + second_call["input"][:, : backend.context_samples], + first_frame[None, -backend.context_samples :], + ) + np.testing.assert_array_equal( + second_call["input"][:, backend.context_samples :], + second_frame[None, :], + ) + np.testing.assert_array_equal( + second_call["state"], + np.full((2, 1, 128), 2, dtype=np.float32), + ) + + # Independent pipeline states may enter the shared ORT session concurrently. + session.run_barrier = threading.Barrier(2) + with ThreadPoolExecutor(max_workers=2) as executor: + futures = [ + executor.submit(backend.infer, first_frame, backend.new_state()), + executor.submit(backend.infer, second_frame, backend.new_state()), + ] + assert [future.result()[0] for future in futures] == pytest.approx([0.75, 0.75]) + + +def test_duplex_session_owns_server_vad_prefix_and_utterance_audio(): + session = DuplexSession("server-vad-session", DuplexSessionConfig()) + frame = np.zeros(160, dtype=np.float32) + + session.reserve_input_bytes(frame.nbytes * 3, limit=frame.nbytes * 4) + released = session.append_server_vad_frame( + frame, + speech_started=False, + speech_stopped=False, + prefix_samples=160, + ) + assert released == 0 + released = session.append_server_vad_frame( + frame, + speech_started=False, + speech_stopped=False, + prefix_samples=160, + ) + assert released == frame.nbytes + session.release_input_bytes(released) + session.append_server_vad_frame( + frame, + speech_started=True, + speech_stopped=False, + prefix_samples=160, + ) + + assert session.server_vad_utterance_bytes == frame.nbytes * 2 + assert session.stage_server_vad_audio_for_commit() is True + committed = session.commit_user_input() + assert committed is not None + assert committed.message["role"] == "user" + audio_url = committed.message["content"][0]["audio_url"]["url"] + assert audio_url.startswith("data:audio/wav;base64,") + wav_bytes = base64.b64decode(audio_url.partition(",")[2]) + with wave.open(io.BytesIO(wav_bytes), "rb") as wav_file: + assert wav_file.getnchannels() == 1 + assert wav_file.getsampwidth() == 2 + assert wav_file.getframerate() == 16_000 + assert wav_file.getnframes() == frame.size * 2 + + +def test_silero_provider_rejects_missing_or_invalid_local_artifact(tmp_path): + missing = SileroVADBackendProvider(model_path=str(tmp_path / "missing.onnx")) + with pytest.raises(RuntimeError, match="does not exist"): + missing.get() + + invalid_path = tmp_path / "silero_vad.onnx" + invalid_path.write_bytes(b"not-the-pinned-model") + invalid = SileroVADBackendProvider(model_path=str(invalid_path)) + with pytest.raises(RuntimeError, match="checksum mismatch"): + invalid.get() diff --git a/tests/entrypoints/test_duplex_fence_propagation.py b/tests/entrypoints/test_duplex_fence_propagation.py index 2f1292ec7..550723ec8 100644 --- a/tests/entrypoints/test_duplex_fence_propagation.py +++ b/tests/entrypoints/test_duplex_fence_propagation.py @@ -74,7 +74,7 @@ async def test_openai_handler_passes_current_session_fence_for_runtime_controls( handler = OmniDuplexSessionHandler(chat_service=FakeChatService(engine)) session = DuplexSession( session_id="sid", - config=DuplexSessionConfig(), + config=DuplexSessionConfig(extra_body={"native_duplex": True}), epoch=1, turn_id=2, ) @@ -92,7 +92,7 @@ async def test_openai_handler_passes_current_session_fence_for_runtime_controls( async def test_runtime_signal_uses_fence_captured_before_session_turn_advances(): engine = FenceRecordingEngine() handler = OmniDuplexSessionHandler(chat_service=FakeChatService(engine)) - session = DuplexSession(session_id="sid", config=DuplexSessionConfig()) + session = DuplexSession(session_id="sid", config=DuplexSessionConfig(extra_body={"native_duplex": True})) captured_fence = DuplexFence("sid", epoch=0, turn_id=0) session.turn_id = 1 @@ -111,7 +111,7 @@ async def test_openai_handler_forwards_cancel_fence_through_async_omni_facade(): app = object.__new__(AsyncOmni) app.engine = engine handler = OmniDuplexSessionHandler(chat_service=FakeChatService(app)) - session = DuplexSession(session_id="sid-cancel", config=DuplexSessionConfig()) + session = DuplexSession(session_id="sid-cancel", config=DuplexSessionConfig(extra_body={"native_duplex": True})) cancelled_fence = DuplexFence("sid-cancel", epoch=2, turn_id=3) next_fence = DuplexFence("sid-cancel", epoch=3, turn_id=3) diff --git a/tests/entrypoints/test_utils.py b/tests/entrypoints/test_utils.py index fa6bf8dbe..680bb110c 100644 --- a/tests/entrypoints/test_utils.py +++ b/tests/entrypoints/test_utils.py @@ -28,6 +28,7 @@ from vllm_omni.entrypoints.utils import ( coerce_param_message_types, filter_dataclass_kwargs, ) +from vllm_omni.model_executor.models.qwen3_omni.pipeline import QWEN3_OMNI_PIPELINE pytestmark = [pytest.mark.core_model, pytest.mark.cpu] @@ -295,6 +296,31 @@ class TestFilterDataclassKwargs: class TestResolveOmniConfig: + def test_bare_deploy_name_returns_packaged_resolved_path(self, tmp_path, mocker: MockerFixture): + deploy_name = "qwen3_omni_moe.yaml" + deploy_path = tmp_path / deploy_name + deploy_path.write_text( + "duplex_session:\n server_vad_model_path: /models/silero_vad.onnx\nstages: []\n", + encoding="utf-8", + ) + mocker.patch("vllm_omni.config.config_factory._DEPLOY_DIR", tmp_path) + mocker.patch("vllm_omni.config.omni_config._DEPLOY_DIR", tmp_path) + mocker.patch( + "vllm_omni.config.config_factory.StageConfigFactory.get_pipeline_config", + return_value=QWEN3_OMNI_PIPELINE, + ) + + resolved = resolve_omni_config( + "dummy-model", + trust_remote_code=False, + deploy_config_path=deploy_name, + cli_overrides={}, + stage_overrides=None, + strategy_config_path=None, + ) + + assert resolved.config_path == str(deploy_path) + def test_stage_lookup_error_lists_resolved_ids(self): resolved = OmniConfigResolution( config_path=None, diff --git a/vllm_omni/config/stage_config.py b/vllm_omni/config/stage_config.py index 06840e55a..2ac1f99dd 100644 --- a/vllm_omni/config/stage_config.py +++ b/vllm_omni/config/stage_config.py @@ -509,6 +509,7 @@ class DuplexSessionRuntimeConfig: max_pending_turns_per_session: int = 4 max_sessions: int = 1 completed_append_cache_size: int = 256 + server_vad_model_path: str | None = None # Startup warmup: run this many silent 80 ms-style frames through a # throwaway realtime session before real clients are admitted, so # one-time costs (kernel JIT, first prefill/decode paths, codec caches) @@ -528,6 +529,10 @@ class DuplexSessionRuntimeConfig: } if self.idle_ttl_s is not None and self.idle_ttl_s <= 0: raise ValueError("duplex_session.idle_ttl_s must be positive or null") + if self.server_vad_model_path is not None and ( + not isinstance(self.server_vad_model_path, str) or not self.server_vad_model_path.strip() + ): + raise ValueError("duplex_session.server_vad_model_path must be a non-empty string or null") for name, value in positive.items(): if value <= 0: raise ValueError(f"duplex_session.{name} must be positive") diff --git a/vllm_omni/engine/async_omni_engine.py b/vllm_omni/engine/async_omni_engine.py index 4c34f5898..6850e6b29 100644 --- a/vllm_omni/engine/async_omni_engine.py +++ b/vllm_omni/engine/async_omni_engine.py @@ -224,7 +224,7 @@ class AsyncOmniEngine: trust_remote_code=bool(trust_remote_code), deploy_config_path=deploy_config_path, ) - self._set_pipeline_runtime_config(pipeline_config, deploy_config_path) + self._set_pipeline_runtime_config(pipeline_config, self.config_path) else: self._set_pipeline_runtime_config( self._config_resolution.pipeline_config, diff --git a/vllm_omni/entrypoints/duplex/audio.py b/vllm_omni/entrypoints/duplex/audio.py index 644a7a297..747293a1a 100644 --- a/vllm_omni/entrypoints/duplex/audio.py +++ b/vllm_omni/entrypoints/duplex/audio.py @@ -21,8 +21,12 @@ MIN_INPUT_SAMPLE_RATE_HZ = 8_000 MAX_INPUT_SAMPLE_RATE_HZ = 192_000 -def validate_input_sample_rate_hz(sample_rate_hz: int | float) -> int: - if isinstance(sample_rate_hz, bool) or (isinstance(sample_rate_hz, float) and not math.isfinite(sample_rate_hz)): +def validate_input_sample_rate_hz(sample_rate_hz: object) -> int: + if ( + isinstance(sample_rate_hz, bool) + or not isinstance(sample_rate_hz, int | float) + or (isinstance(sample_rate_hz, float) and not math.isfinite(sample_rate_hz)) + ): raise ValueError("sample_rate_hz must be a finite integer") rate = int(sample_rate_hz) if rate != sample_rate_hz: @@ -35,14 +39,27 @@ def validate_input_sample_rate_hz(sample_rate_hz: int | float) -> int: def resample_pcm16_mono(raw: bytes, *, source_rate_hz: int, target_rate_hz: int) -> bytes: if source_rate_hz <= 0 or target_rate_hz <= 0 or source_rate_hz == target_rate_hz: return raw - samples = np.frombuffer(raw, dtype=" StreamingAudioResampler._max_polyphase_factor: + # The FIR size scales with the reduced ratio. Keep unusual client + # rates bounded while common audio rates use the high-quality path. + if samples.size <= 1: + return raw + target_size = max(1, int(round(samples.size * target_rate_hz / source_rate_hz))) + source_x = np.linspace(0.0, 1.0, num=samples.size, endpoint=True) + target_x = np.linspace(0.0, 1.0, num=target_size, endpoint=True) + resampled = np.interp(target_x, source_x, samples) + return np.clip(resampled, -32_768, 32_767).astype(" bytes: @@ -116,6 +133,24 @@ def wav_payload_to_pcm16(raw: bytes) -> tuple[bytes | None, int | None]: return None, None +def encode_float32_mono_wav_base64( + samples: np.ndarray, + *, + sample_rate_hz: int, +) -> str: + """Package normalized mono float32 samples as a PCM16 WAV data payload.""" + rate = validate_input_sample_rate_hz(sample_rate_hz) + normalized = np.ascontiguousarray(samples, dtype=np.float32).reshape(-1) + pcm16 = np.clip(np.rint(normalized * 32768.0), -32768, 32767).astype(" bool: - """Enable duplex routes only for deployments that explicitly opt in.""" + """Enable the realtime session handler for explicitly configured deployments.""" if stage_configs: for stage in stage_configs: session_mode = ( @@ -21,14 +21,13 @@ def should_enable_duplex_endpoint( return True if config_path: try: - from omegaconf import OmegaConf + from vllm_omni.config.stage_config import resolve_deploy_yaml - raw_config = OmegaConf.load(config_path) - session_mode = raw_config.get("session_mode") if hasattr(raw_config, "get") else None - if session_mode == "duplex": + raw_config = resolve_deploy_yaml(config_path) + if raw_config.get("session_mode") == "duplex" or isinstance(raw_config.get("duplex_session"), dict): return True except Exception as exc: - logger.warning("Failed to inspect duplex session_mode from %s: %s", config_path, exc) + logger.warning("Failed to inspect realtime session configuration from %s: %s", config_path, exc) return False diff --git a/vllm_omni/entrypoints/duplex/chat_fallback.py b/vllm_omni/entrypoints/duplex/chat_fallback.py index 31df63833..6e4d0a0e0 100644 --- a/vllm_omni/entrypoints/duplex/chat_fallback.py +++ b/vllm_omni/entrypoints/duplex/chat_fallback.py @@ -38,7 +38,14 @@ class ChatFallbackProjectorMixin: request = self._build_chat_request(session, request_id) result = await self._chat_service.create_chat_completion(request, raw_request=None) if isinstance(result, ErrorResponse): - await send_json({"type": "error", "error": result.message, "code": result.type or "chat_error"}) + error = result.error + await send_json( + { + "type": "error", + "error": error.message if error else "Chat request failed", + "code": error.type if error else "chat_error", + } + ) session.end_response(commit_text=False) return if hasattr(result, "__aiter__"): diff --git a/vllm_omni/entrypoints/duplex/protocol.py b/vllm_omni/entrypoints/duplex/protocol.py index 9fd3b0ef5..ff91e0db5 100644 --- a/vllm_omni/entrypoints/duplex/protocol.py +++ b/vllm_omni/entrypoints/duplex/protocol.py @@ -10,6 +10,16 @@ from enum import Enum from types import MappingProxyType from uuid import uuid4 +import numpy as np + +from vllm_omni.entrypoints.duplex.audio import ( + encode_float32_mono_wav_base64, +) +from vllm_omni.entrypoints.duplex.server_vad import ( + ServerVADConfig, + parse_session_turn_detection, +) + class DuplexOverlapPolicy(str, Enum): LISTEN_ONLY = "listen_only" @@ -231,10 +241,12 @@ class DuplexSessionConfig: overlap_barge_in_ms: int = 1200 overlap_silence_rms: float = 0.003 playback_commit_policy: str = DuplexPlaybackCommitPolicy.COMMIT_ALL_ON_DONE.value + turn_detection_configured: bool = False + server_vad: ServerVADConfig | None = None extra_body: dict[str, object] = field(default_factory=dict) def as_dict(self) -> dict[str, object]: - return { + payload: dict[str, object] = { "model": self.model, "modalities": list(self.modalities), "instructions": self.instructions, @@ -253,6 +265,9 @@ class DuplexSessionConfig: "playback_commit_policy": self.playback_commit_policy, "extra_body": dict(self.extra_body), } + if self.turn_detection_configured: + payload["turn_detection"] = self.server_vad.as_dict() if self.server_vad is not None else None + return payload @classmethod def from_event(cls, event: dict[str, object]) -> DuplexSessionConfig: @@ -295,9 +310,13 @@ class DuplexSessionConfig: config.playback_commit_policy = cls._normalize_playback_commit_policy(source["playback_commit_policy"]) if isinstance(source.get("modalities"), list) and all(isinstance(x, str) for x in source["modalities"]): config.modalities = list(source["modalities"]) + turn_detection_configured, server_vad = parse_session_turn_detection(source) if isinstance(source.get("extra_body"), dict): config.extra_body = normalize_native_duplex_key(dict(source["extra_body"])) extra = config.extra_body + raw_realtime_session = extra.get("realtime_session_payload") + if not turn_detection_configured and isinstance(raw_realtime_session, dict): + turn_detection_configured, server_vad = parse_session_turn_detection(raw_realtime_session) if isinstance(extra.get("overlap_policy"), str): config.overlap_policy = cls._normalize_overlap_policy(extra["overlap_policy"]) if isinstance(extra.get("overlap_short_ack_ms"), int | float): @@ -308,6 +327,9 @@ class DuplexSessionConfig: config.overlap_silence_rms = max(0.0, float(extra["overlap_silence_rms"])) if isinstance(extra.get("playback_commit_policy"), str): config.playback_commit_policy = cls._normalize_playback_commit_policy(extra["playback_commit_policy"]) + if turn_detection_configured: + config.turn_detection_configured = True + config.server_vad = server_vad return config @staticmethod @@ -354,6 +376,13 @@ class DuplexCommittedInput: input_commit_seq: int +@dataclass +class PendingServerVADTurn: + committed: DuplexCommittedInput + create_response: bool + item_id: str + + @dataclass class DuplexAssistantAudioTextMark: text_chars: int @@ -368,6 +397,13 @@ class InputBufferState: overlap_speech_ms: int = 0 reserved_input_bytes: int = 0 pending_turns: int = 0 + turn_detection_config_locked: bool = False + vad_prefix_audio: list[np.ndarray] = field(default_factory=list) + vad_prefix_samples: int = 0 + vad_utterance_audio: list[np.ndarray] = field(default_factory=list) + vad_speech_active: bool = False + active_server_vad_item_id: str | None = None + pending_server_vad_turn: PendingServerVADTurn | None = None @dataclass @@ -627,6 +663,9 @@ class DuplexSession: return active_turn_id is None or int(turn_id) == active_turn_id def append_history_message(self, message: dict[str, object]) -> None: + response_id = self.active_response_id + if response_id is not None: + self.reserve_history_item(f"item_{response_id}") self._conversation.messages.append(message) def stage_pending_history_item(self, item_id: str, message: dict[str, object]) -> None: @@ -653,6 +692,13 @@ class DuplexSession: self._input.reserved_input_bytes += size return True + @property + def turn_detection_config_locked(self) -> bool: + return self._input.turn_detection_config_locked + + def lock_turn_detection_config(self) -> None: + self._input.turn_detection_config_locked = True + def release_input_bytes(self, size: int) -> None: self._input.reserved_input_bytes = max(0, self._input.reserved_input_bytes - max(0, int(size))) @@ -677,6 +723,92 @@ class DuplexSession: self._input.pending_audio.append(DuplexAudioChunk(data=data, format=fmt, sample_rate_hz=sample_rate_hz)) self.turn_state = DuplexTurnState.USER_SPEAKING + def append_server_vad_frame( + self, + samples: np.ndarray, + *, + speech_started: bool, + speech_stopped: bool, + prefix_samples: int, + ) -> int: + """Store one detector frame while keeping commit-eligible audio session-owned. + + Returns the number of raw float32 bytes released when the rolling + pre-speech prefix is trimmed. + """ + # Session-owned audio must not retain the batch backing a detector frame. + frame = np.array(samples, dtype=np.float32, copy=True).reshape(-1) + released_samples = 0 + if not self._input.vad_speech_active: + if speech_started: + self._input.vad_utterance_audio = [*self._input.vad_prefix_audio, frame] + self._input.vad_prefix_audio = [] + self._input.vad_prefix_samples = 0 + self._input.vad_speech_active = True + else: + self._input.vad_prefix_audio.append(frame) + self._input.vad_prefix_samples += frame.size + while self._input.vad_prefix_samples > prefix_samples and self._input.vad_prefix_audio: + overflow = self._input.vad_prefix_samples - prefix_samples + first = self._input.vad_prefix_audio[0] + if first.size <= overflow: + self._input.vad_prefix_audio.pop(0) + self._input.vad_prefix_samples -= first.size + released_samples += first.size + else: + self._input.vad_prefix_audio[0] = first[overflow:].copy() + self._input.vad_prefix_samples -= overflow + released_samples += overflow + else: + self._input.vad_utterance_audio.append(frame) + + if speech_stopped: + self._input.vad_speech_active = False + if self._input.vad_speech_active or speech_started: + self.turn_state = DuplexTurnState.USER_SPEAKING + return released_samples * np.dtype(np.float32).itemsize + + def stage_server_vad_audio_for_commit(self) -> bool: + if not self._input.vad_utterance_audio: + return False + samples = np.concatenate(self._input.vad_utterance_audio) + encoded = encode_float32_mono_wav_base64(samples, sample_rate_hz=16_000) + self._input.pending_audio.append(DuplexAudioChunk(data=encoded, format="wav", sample_rate_hz=16_000)) + self._input.vad_utterance_audio.clear() + self._input.vad_speech_active = False + return True + + def clear_server_vad_audio(self) -> None: + self._input.vad_prefix_audio.clear() + self._input.vad_prefix_samples = 0 + self._input.vad_utterance_audio.clear() + self._input.vad_speech_active = False + self._input.active_server_vad_item_id = None + + @property + def active_server_vad_item_id(self) -> str | None: + return self._input.active_server_vad_item_id + + @property + def server_vad_utterance_bytes(self) -> int: + return sum(frame.nbytes for frame in self._input.vad_utterance_audio) + + def discard_uncommitted_server_vad_utterance(self) -> int: + released = self.server_vad_utterance_bytes + self._input.vad_utterance_audio.clear() + self._input.vad_speech_active = False + self._input.active_server_vad_item_id = None + self._input.pending_text.clear() + return released + + def begin_server_vad_speech(self, item_id: str) -> None: + self._input.active_server_vad_item_id = item_id + + def finish_server_vad_speech(self) -> str | None: + item_id = self._input.active_server_vad_item_id + self._input.active_server_vad_item_id = None + return item_id + def mark_user_input_activity(self) -> None: self.turn_state = DuplexTurnState.USER_SPEAKING @@ -692,10 +824,16 @@ class DuplexSession: self._input.pending_audio.clear() self._input.reserved_input_bytes = 0 self._input.pending_turns = 0 + self.clear_server_vad_audio() self.turn_state = DuplexTurnState.IDLE return cancelled - def commit_user_input(self) -> DuplexCommittedInput | None: + def commit_user_input( + self, + *, + append_history: bool = True, + retain_input_bytes: int = 0, + ) -> DuplexCommittedInput | None: text = "".join(self._input.pending_text).strip() audio_chunks = list(self._input.pending_audio) if not text and not audio_chunks: @@ -722,10 +860,11 @@ class DuplexSession: self._bind_active_response_to_input_commit(self._input.commit_seq + 1) self._input.commit_seq += 1 message = {"role": "user", "content": content} - self._conversation.messages.append(message) + if append_history: + self._conversation.messages.append(message) self._input.pending_text.clear() self._input.pending_audio.clear() - self._input.reserved_input_bytes = 0 + self._input.reserved_input_bytes = max(0, int(retain_input_bytes)) self.turn_state = DuplexTurnState.USER_COMMITTED return DuplexCommittedInput( message=message, @@ -734,6 +873,40 @@ class DuplexSession: input_commit_seq=self._input.commit_seq, ) + @property + def pending_server_vad_turn(self) -> PendingServerVADTurn | None: + return self._input.pending_server_vad_turn + + def stage_server_vad_turn( + self, + committed: DuplexCommittedInput, + *, + create_response: bool, + item_id: str, + ) -> None: + if self._input.pending_server_vad_turn is not None: + raise RuntimeError("A Server VAD turn is already pending") + response_id = self.active_response_id + if response_id is not None: + # A pending user turn advances input_commit_seq before the active + # response completes. Reserve its current history position first. + self.reserve_history_item(f"item_{response_id}") + self._input.pending_server_vad_turn = PendingServerVADTurn( + committed=committed, + create_response=create_response, + item_id=item_id, + ) + + def activate_server_vad_turn(self) -> PendingServerVADTurn | None: + pending = self._input.pending_server_vad_turn + if pending is None: + return None + self._input.pending_server_vad_turn = None + self._conversation.messages.append(pending.committed.message) + self.register_history_item(pending.item_id, pending.committed.message) + self.turn_state = DuplexTurnState.USER_COMMITTED + return pending + def commit_native_audio_input( self, *, @@ -1020,7 +1193,14 @@ class DuplexSession: response_input_commit_seq = self._response.active_response_input_commit_seq if response_input_commit_seq is None: response_input_commit_seq = self.input_commit_seq - response_history_is_late = self.input_commit_seq > response_input_commit_seq + response_item_id = f"item_{response_id}" if response_id is not None else None + response_history_is_reserved = response_item_id is not None and ( + response_item_id in self._conversation.item_ids + or response_item_id in self._conversation.history_item_placeholders + ) + response_history_is_late = ( + self.input_commit_seq > response_input_commit_seq and not response_history_is_reserved + ) assistant_text = "".join(self._response.assistant_text_buffer).strip() message = None if assistant_text: @@ -1041,9 +1221,8 @@ class DuplexSession: committed_text = "" if commit_text and committed_text: message = {"role": "assistant", "content": committed_text} - item_id = f"item_{response_id}" if response_id is not None else None - if item_id is not None and item_id in self._conversation.history_item_placeholders: - self._store_history_item_message(item_id, message) + if response_item_id is not None and response_history_is_reserved: + self._store_history_item_message(response_item_id, message) else: self._conversation.messages.append(message) effective_playback_policy = playback_commit_policy or self.config.playback_commit_policy @@ -1054,7 +1233,7 @@ class DuplexSession: and effective_playback_policy == DuplexPlaybackCommitPolicy.ACK_ONLY.value ): self.register_history_item(f"item_{response_id}", None) - elif response_id is not None and not assistant_text: + elif response_id is not None and (not assistant_text or not commit_text): self._discard_history_item_placeholder(f"item_{response_id}") self._response.assistant_text_buffer.clear() if not preserve_request: @@ -1113,6 +1292,11 @@ class DuplexSession: ) def delete_history_item(self, item_id: str) -> bool: + removed_pending_turn = False + pending_server_vad_turn = self._input.pending_server_vad_turn + if pending_server_vad_turn is not None and pending_server_vad_turn.item_id == item_id: + self._input.pending_server_vad_turn = None + removed_pending_turn = True response_id = item_id.removeprefix("item_") if item_id.startswith("item_") else None message = self._conversation.item_ids.pop(item_id, None) self._conversation.item_audio_text_marks.pop(item_id, None) @@ -1125,7 +1309,7 @@ class DuplexSession: if response_id is not None: self._conversation.assistant_response_snapshots.pop(response_id, None) if message is None: - return pending is not None or removed_placeholder + return removed_pending_turn or pending is not None or removed_placeholder try: self._conversation.messages.remove(message) except ValueError: @@ -1390,6 +1574,8 @@ class DuplexSession: self._response.active_response_turn_id = None self._response.active_response_input_commit_seq = None self._response.active_response_awaits_input_commit = False + self.clear_server_vad_audio() + self._input.pending_server_vad_turn = None self._clear_response_metrics() self._restore_response_config() @@ -1420,6 +1606,8 @@ class DuplexSession: "playback": self.playback.as_dict(), "capabilities": self.capabilities.as_dict(), } + if self.config.turn_detection_configured: + payload["turn_detection"] = self.config.server_vad.as_dict() if self.config.server_vad is not None else None if isinstance(self.config.extra_body.get("realtime_tools"), list): payload["tools"] = self.config.extra_body["realtime_tools"] if isinstance(self.config.extra_body.get("realtime_tool_choice"), str | dict): diff --git a/vllm_omni/entrypoints/duplex/realtime_input.py b/vllm_omni/entrypoints/duplex/realtime_input.py index 03710b43b..7fdadbcb2 100644 --- a/vllm_omni/entrypoints/duplex/realtime_input.py +++ b/vllm_omni/entrypoints/duplex/realtime_input.py @@ -7,27 +7,23 @@ import asyncio import base64 import binascii from collections.abc import Callable -from typing import Any from uuid import uuid4 import numpy as np from vllm_omni.entrypoints.duplex.audio import ( convert_input_audio_with_rate, + encode_float32_mono_wav_base64, + validate_input_sample_rate_hz, ) from vllm_omni.entrypoints.duplex.protocol import normalize_native_duplex_key from vllm_omni.entrypoints.duplex.realtime_state import ( REALTIME_INPUT_AUDIO_FORMATS, REALTIME_OUTPUT_AUDIO_FORMATS, RealtimeStateOwner, + realtime_default_sample_rate_hz, ) -from vllm_omni.entrypoints.duplex.vad import ( - SILERO_VAD_MIN_THRESHOLD, - ServerVADUnavailableError, - SileroStreamingVAD, - SileroVADConfig, - StreamingVADResult, -) +from vllm_omni.entrypoints.duplex.server_vad import ServerVADConfig, parse_session_turn_detection class RealtimeInputTranslator(RealtimeStateOwner): @@ -49,7 +45,6 @@ class RealtimeInputTranslator(RealtimeStateOwner): _conversation_items: dict[str, dict[str, object]] _item_truncation_cursors: dict[str, tuple[int, int]] _pending_outbound: asyncio.Queue[dict[str, object]] - _pending_commit_item_ids: asyncio.Queue[str] async def _send_realtime_payload(self, payload: dict[str, object]) -> None: raise NotImplementedError @@ -66,6 +61,17 @@ class RealtimeInputTranslator(RealtimeStateOwner): _response_is_done: Callable[[object], bool] + @property + def _turn_based_server_vad(self) -> bool: + return self._turn_detection is not None and not self._native_input_append + + def _reset_realtime_input_buffer_state(self) -> None: + self._input_speech_started = False + self._active_input_item_id = None + self._input_audio_buffer_has_audio = False + self._input_audio_buffer_had_non_speech = False + self._input_audio_buffer_transcript_parts.clear() + async def discard_pending_input_audio( self, *, @@ -88,11 +94,7 @@ class RealtimeInputTranslator(RealtimeStateOwner): "item_id": self._active_input_item_id, } ) - self._input_speech_started = False - self._active_input_item_id = None - self._input_audio_buffer_has_audio = False - self._input_audio_buffer_had_non_speech = False - self._input_audio_buffer_transcript_parts.clear() + self._reset_realtime_input_buffer_state() async def _send_realtime_input_ack(self, event: dict[str, object]) -> None: if event.get("type") != "conversation.item.create": @@ -100,7 +102,6 @@ class RealtimeInputTranslator(RealtimeStateOwner): item = event.get("item") if not isinstance(item, dict): return - item = self._normalize_conversation_item(item) previous_item_id = event.get("previous_item_id") if isinstance(previous_item_id, str): item["_previous_item_id"] = previous_item_id @@ -115,18 +116,40 @@ class RealtimeInputTranslator(RealtimeStateOwner): if event_type == "session.update": session = event.get("session") session_payload: dict[str, object] = session if isinstance(session, dict) else event - format_error = self._validate_realtime_session_audio_formats(session_payload) - if format_error is not None: + try: + turn_detection_configured, server_vad = parse_session_turn_detection(session_payload) + except ValueError as exc: await self._send_realtime_payload( self._realtime_error_payload( - "unsupported_audio_format", - format_error, + "unsupported_turn_detection", + str(exc), event_id=event.get("event_id"), + param="turn_detection", ) ) return None - turn_detection_error = self._validate_realtime_turn_detection(session_payload) + normalized_turn_detection = server_vad.as_dict() if server_vad is not None else None + previous_protocol_config = self._realtime_protocol_config() + try: + self._apply_realtime_session_defaults(session_payload) + except ValueError as exc: + self._set_realtime_protocol_config(previous_protocol_config) + await self._send_realtime_payload( + self._realtime_error_payload( + "unsupported_audio_format", + str(exc), + event_id=event.get("event_id"), + param="audio.input.format", + ) + ) + return None + turn_detection_error = self._validate_realtime_turn_detection( + session_payload, + configured=turn_detection_configured, + normalized=normalized_turn_detection, + ) if turn_detection_error is not None: + self._set_realtime_protocol_config(previous_protocol_config) await self._send_realtime_payload( self._realtime_error_payload( "unsupported_turn_detection", @@ -136,31 +159,38 @@ class RealtimeInputTranslator(RealtimeStateOwner): ) ) return None - self._prepare_realtime_turn_detection_update(session_payload) - self._apply_realtime_session_defaults(session_payload) - session_payload.update(self._realtime_overlap_fields(session_payload)) - if not self._opened: - self._opened = True - self._initial_session_update = True - return self._session_create_from_realtime(session_payload) - return { - "type": "turn.signal", - "event": "session.update", - "payload": session_payload, - "realtime_event_id": event.get("event_id"), - } - if event_type == "conversation.item.create": - item = event.get("item") - format_error = self._validate_conversation_item_audio_formats(item) - if format_error is not None: - await self._send_realtime_payload( - self._realtime_error_payload( - "unsupported_audio_format", - format_error, - event_id=event.get("event_id"), - ) + turn_detection_candidate = self._prepare_realtime_turn_detection_update( + session_payload, + configured=turn_detection_configured, + normalized=normalized_turn_detection, + ) + try: + session_payload.update(self._realtime_overlap_fields(session_payload)) + if not self._opened: + self._opened = True + self._initial_session_update = True + translated = self._session_create_from_realtime(session_payload) + event_id = event.get("event_id") + if isinstance(event_id, str) and event_id: + translated["_realtime_event_id"] = event_id + else: + translated = { + "type": "turn.signal", + "event": "session.update", + "payload": session_payload, + "realtime_event_id": event.get("event_id"), + } + candidate_protocol_config = self._realtime_protocol_config() + finally: + self._set_realtime_protocol_config(previous_protocol_config) + if turn_detection_candidate[0] or candidate_protocol_config != previous_protocol_config: + self._pending_turn_detection_update = ( + *turn_detection_candidate, + asyncio.Event(), + candidate_protocol_config, ) - return None + return translated + if event_type == "conversation.item.create": return await self._conversation_item_to_duplex(event) if event_type == "conversation.item.delete": item_id = event.get("item_id") @@ -293,27 +323,20 @@ class RealtimeInputTranslator(RealtimeStateOwner): return None if event_type == "input_audio_buffer.append": audio = event.get("audio") or event.get("delta") - fmt, format_rate = self._parse_realtime_audio_format( - event.get("format") or event.get("input_audio_format") or self._input_audio_format - ) - sample_rate_hz = ( - event.get("sample_rate_hz") or event.get("sample_rate") or format_rate or self._input_sample_rate_hz - ) - if not self._is_supported_realtime_input_format(fmt): - await self._send_realtime_payload( - self._realtime_error_payload( - "unsupported_audio_format", - f"Unsupported input_audio_format: {fmt}", - event_id=event.get("event_id"), - ) - ) - return None + event_format = event.get("format") or event.get("input_audio_format") + native_server_vad = self._turn_detection is not None and self._native_input_append + turn_based_server_vad = self._turn_based_server_vad try: - audio, fmt, sample_rate_hz = convert_input_audio_with_rate( - audio, - fmt, - sample_rate_hz=sample_rate_hz if isinstance(sample_rate_hz, int | float) else None, - ) + fmt, format_rate = self._parse_realtime_audio_format(event_format or self._input_audio_format) + sample_rate_hz = event.get("sample_rate_hz") or event.get("sample_rate") or format_rate + if sample_rate_hz is None: + sample_rate_hz = ( + realtime_default_sample_rate_hz(fmt) + if event_format is not None and fmt != self._input_audio_format + else self._input_sample_rate_hz + ) + if turn_based_server_vad: + sample_rate_hz = validate_input_sample_rate_hz(sample_rate_hz) except ValueError as exc: await self._send_realtime_payload( self._realtime_error_payload( @@ -324,63 +347,65 @@ class RealtimeInputTranslator(RealtimeStateOwner): ) ) return None - detector = self._server_vad - vad_result: StreamingVADResult | None = None - if isinstance(detector, SileroStreamingVAD): + channels = event.get("channels", 1) + if not self._is_supported_realtime_input_format(fmt): + await self._send_realtime_payload( + self._realtime_error_payload( + "unsupported_audio_format", + f"Unsupported input_audio_format: {fmt}", + event_id=event.get("event_id"), + ) + ) + return None + if turn_based_server_vad and (fmt != "pcm16" or channels != 1 or sample_rate_hz not in {16_000, 24_000}): + await self._send_realtime_payload( + self._realtime_error_payload( + "unsupported_audio_format", + "server_vad requires mono PCM16 input at 16 kHz or 24 kHz", + event_id=event.get("event_id"), + param="audio.input.format", + ) + ) + return None + looks_like_speech: bool | None = None + if not turn_based_server_vad: try: - vad_result = detector.process_base64(audio, fmt=fmt, sample_rate_hz=sample_rate_hz) - except (ServerVADUnavailableError, ValueError) as exc: - unavailable = isinstance(exc, ServerVADUnavailableError) + audio, fmt, sample_rate_hz = convert_input_audio_with_rate( + audio, + fmt, + sample_rate_hz=sample_rate_hz if isinstance(sample_rate_hz, int | float) else None, + ) + except ValueError as exc: await self._send_realtime_payload( self._realtime_error_payload( - "server_vad_unavailable" if unavailable else "bad_audio", + "bad_event", str(exc), event_id=event.get("event_id"), - param="turn_detection" if unavailable else "audio", + param="sample_rate_hz", ) ) return None - looks_like_speech = ( - vad_result.is_speech - if vad_result is not None - else self._input_looks_like_speech(event, audio=audio, fmt=fmt) - ) - self._input_audio_buffer_has_audio = self._input_audio_buffer_has_audio or ( - looks_like_speech and isinstance(audio, str) and bool(audio) - ) - self._input_audio_buffer_had_non_speech = self._input_audio_buffer_had_non_speech or ( - not looks_like_speech and isinstance(audio, str) and bool(audio) - ) - stopped_event: dict[str, object] | None = None - if vad_result is not None and vad_result.speech_stopped: - stopped_event = dict(event) - if vad_result.speech_end_ms is not None: - stopped_event["audio_end_ms"] = vad_result.speech_end_ms - if stopped_event is not None and vad_result is not None and vad_result.speech_active: - await self._emit_input_speech_stopped( - stopped_event, - item_id=self._active_input_item_id or f"item_{uuid4().hex}", + if not turn_based_server_vad and not native_server_vad: + looks_like_speech = self._input_looks_like_speech(event, audio=audio, fmt=fmt) + has_audio_payload = isinstance(audio, str) and bool(audio) + self._input_audio_buffer_has_audio = self._input_audio_buffer_has_audio or ( + looks_like_speech and has_audio_payload ) - if vad_result is not None and vad_result.speech_started: - started_event = dict(event) - if vad_result.speech_start_ms is not None: - started_event["audio_start_ms"] = vad_result.speech_start_ms - await self._emit_input_speech_started(started_event) - elif vad_result is None and looks_like_speech: - await self._emit_input_speech_started(event) - if stopped_event is not None and vad_result is not None and not vad_result.speech_active: - await self._emit_input_speech_stopped( - stopped_event, - item_id=self._active_input_item_id or f"item_{uuid4().hex}", + self._input_audio_buffer_had_non_speech = self._input_audio_buffer_had_non_speech or ( + not looks_like_speech and has_audio_payload ) - if looks_like_speech: - self._remember_input_transcript_hint(event) + if looks_like_speech: + await self._emit_input_speech_started(event) + self._remember_input_transcript_hint(event) payload = { "type": "input_audio_buffer.append", "audio": audio, "format": fmt, "sample_rate_hz": sample_rate_hz, } + event_id = event.get("event_id") + if self._turn_detection is not None and isinstance(event_id, str) and event_id: + payload["_realtime_event_id"] = event_id video_frames = event.get("video_frames") if video_frames is not None: frames_error = self._validate_realtime_video_frames(video_frames, event.get("max_slice_nums")) @@ -397,25 +422,27 @@ class RealtimeInputTranslator(RealtimeStateOwner): if isinstance(video_frames, list): payload["video_frames"] = [frame for frame in video_frames if isinstance(frame, str) and frame] self._copy_realtime_input_hints(event, payload) - payload["is_speech"] = looks_like_speech - if vad_result is not None: - payload["vad"] = { - "backend": "silero", - "is_speech": vad_result.is_speech, - "speech_active": vad_result.speech_active, - "speech_started": vad_result.speech_started, - "speech_stopped": vad_result.speech_stopped, - "speech_probability": vad_result.speech_probability, - } - if vad_result.speech_active: - payload["force_listen"] = True + if looks_like_speech is not None: + payload["is_speech"] = looks_like_speech return payload if event_type == "input_audio_buffer.commit": + if self._turn_based_server_vad: + await self._send_realtime_payload( + self._realtime_error_payload( + "server_vad_manual_commit_unsupported", + ( + "input_audio_buffer.commit is not supported while server_vad is enabled; " + "wait for server endpointing, or start a session with turn_detection=null " + "to use manual commit" + ), + event_id=event.get("event_id"), + ) + ) + return None if not self._input_audio_buffer_has_audio: if self._input_audio_buffer_had_non_speech: self._input_audio_buffer_had_non_speech = False self._active_input_item_id = None - self._reset_server_vad() return { "type": "input_audio_buffer.commit", "final": event.get("final", True), @@ -431,29 +458,22 @@ class RealtimeInputTranslator(RealtimeStateOwner): ) return None item_id = self._active_input_item_id or f"item_{uuid4().hex}" - await self._pending_commit_item_ids.put(item_id) await self._emit_input_speech_stopped(event, item_id=item_id) transcript = self._consume_input_transcript_hint() self._active_input_item_id = None self._input_audio_buffer_has_audio = False self._input_audio_buffer_had_non_speech = False - self._reset_server_vad() payload = { "type": "input_audio_buffer.commit", "final": event.get("final", True), - "realtime_item_id": item_id, + "item_id": item_id, "response_create": bool(event.get("response_create", False)), } if transcript: payload["transcript"] = transcript return payload if event_type == "input_audio_buffer.clear": - self._input_speech_started = False - self._active_input_item_id = None - self._input_audio_buffer_has_audio = False - self._input_audio_buffer_had_non_speech = False - self._input_audio_buffer_transcript_parts.clear() - self._reset_server_vad() + self._reset_realtime_input_buffer_state() return {"type": "input_audio_buffer.clear", "reason": event_type} if event_type == "output_audio_buffer.clear": payload = {"type": "output_audio_buffer.clear", "reason": event_type} @@ -643,19 +663,19 @@ class RealtimeInputTranslator(RealtimeStateOwner): "previous_item_id": previous_item_id, "item": item, }, - { - "type": "conversation.item.created", - "previous_item_id": previous_item_id, - "item": item, - }, ] def _conversation_item_done_event(self, item: dict[str, object]) -> dict[str, object]: item_id = item.get("id") previous_item_id = self._previous_item_id(item_id) if isinstance(item_id, str) else None if isinstance(item_id, str) and item_id: + already_known = item_id in self._conversation_items self._conversation_items[item_id] = item - self._last_conversation_item_id = item_id + # Completing an item that was already added must not move it to the + # end of the conversation. A later user item may have been added + # while this response was still in progress. + if not already_known: + self._last_conversation_item_id = item_id return { "type": "conversation.item.done", "previous_item_id": previous_item_id, @@ -737,17 +757,14 @@ class RealtimeInputTranslator(RealtimeStateOwner): self, *, item_id: str, - event: dict[str, Any], ) -> dict[str, object]: return { "type": "input_audio_buffer.committed", "previous_item_id": self._previous_item_id(item_id), "item_id": item_id, - "event": event, } def _session_create_from_realtime(self, session_payload: dict[str, object]) -> dict[str, object]: - self._apply_realtime_session_defaults(session_payload) model = session_payload.get("model") audio_config = session_payload.get("audio") audio_input = audio_config.get("input") if isinstance(audio_config, dict) else None @@ -825,8 +842,13 @@ class RealtimeInputTranslator(RealtimeStateOwner): audio_input = audio_config.get("input") if isinstance(audio_input, dict): input_format = audio_input.get("format") + input_format_configured = input_format is not None input_format, input_rate = self._parse_realtime_audio_format(input_format) - if isinstance(input_format, str) and input_format.lower() in REALTIME_INPUT_AUDIO_FORMATS: + if input_format is not None and not ( + isinstance(input_format, str) and input_format.lower() in REALTIME_INPUT_AUDIO_FORMATS + ): + raise ValueError(f"Unsupported input_audio_format: {input_format}") + if isinstance(input_format, str): self._input_audio_format = input_format output_format: object = session_payload.get("output_audio_format") or session_payload.get("response_format") output_rate_raw: object | None = None @@ -836,9 +858,13 @@ class RealtimeInputTranslator(RealtimeStateOwner): output_format = audio_output.get("format") output_rate_raw = audio_output.get("sample_rate_hz") or audio_output.get("sample_rate") output_format, output_rate = self._parse_realtime_audio_format(output_format) - if output_rate is None and isinstance(output_rate_raw, int | float) and output_rate_raw > 0: - output_rate = int(output_rate_raw) - if isinstance(output_format, str) and output_format.lower() in REALTIME_OUTPUT_AUDIO_FORMATS: + if output_rate is None and output_rate_raw is not None: + output_rate = validate_input_sample_rate_hz(output_rate_raw) + if output_format is not None and not ( + isinstance(output_format, str) and output_format.lower() in REALTIME_OUTPUT_AUDIO_FORMATS + ): + raise ValueError(f"Unsupported output_audio_format: {output_format}") + if isinstance(output_format, str): self._output_audio_format = self._realtime_output_format(output_format) sample_rate = session_payload.get("sample_rate_hz") or session_payload.get("sample_rate") if sample_rate is None and isinstance(audio_config, dict): @@ -847,41 +873,19 @@ class RealtimeInputTranslator(RealtimeStateOwner): sample_rate = audio_input.get("sample_rate_hz") or audio_input.get("sample_rate") if sample_rate is None: sample_rate = input_rate - if isinstance(sample_rate, int | float) and sample_rate > 0: - self._input_sample_rate_hz = int(sample_rate) - if isinstance(output_rate, int | float) and output_rate > 0: - self._output_sample_rate_hz = int(output_rate) + if sample_rate is not None: + self._input_sample_rate_hz = validate_input_sample_rate_hz(sample_rate) + elif input_format_configured: + default_input_rate = realtime_default_sample_rate_hz(input_format) + if default_input_rate is not None: + self._input_sample_rate_hz = default_input_rate + if output_rate is not None: + self._output_sample_rate_hz = output_rate overlap_fields = self._realtime_overlap_fields(session_payload) overlap_silence_rms = overlap_fields.get("overlap_silence_rms") if isinstance(overlap_silence_rms, int | float): self._overlap_silence_rms = max(0.0, float(overlap_silence_rms)) - @classmethod - def _validate_realtime_session_audio_formats(cls, session_payload: dict[str, object]) -> str | None: - audio_config = session_payload.get("audio") - input_format: object = session_payload.get("input_audio_format") - if input_format is None and isinstance(audio_config, dict): - audio_input = audio_config.get("input") - if isinstance(audio_input, dict): - input_format = audio_input.get("format") - parsed_input, _ = cls._parse_realtime_audio_format(input_format) - if input_format is not None and not ( - isinstance(parsed_input, str) and parsed_input.lower() in REALTIME_INPUT_AUDIO_FORMATS - ): - return f"Unsupported input_audio_format: {input_format}" - - output_format: object = session_payload.get("output_audio_format") or session_payload.get("response_format") - if output_format is None and isinstance(audio_config, dict): - audio_output = audio_config.get("output") - if isinstance(audio_output, dict): - output_format = audio_output.get("format") - parsed_output, _ = cls._parse_realtime_audio_format(output_format) - if output_format is not None and not ( - isinstance(parsed_output, str) and parsed_output.lower() in REALTIME_OUTPUT_AUDIO_FORMATS - ): - return f"Unsupported output_audio_format: {output_format}" - return None - @classmethod def _validate_realtime_response_audio_formats(cls, response_payload: dict[str, object]) -> str | None: output_format: object = response_payload.get("output_audio_format") or response_payload.get("response_format") @@ -890,33 +894,16 @@ class RealtimeInputTranslator(RealtimeStateOwner): audio_output = audio_config.get("output") if isinstance(audio_output, dict): output_format = audio_output.get("format") - parsed_output, _ = cls._parse_realtime_audio_format(output_format) + try: + parsed_output, _ = cls._parse_realtime_audio_format(output_format) + except ValueError as exc: + return str(exc) if output_format is not None and not ( isinstance(parsed_output, str) and parsed_output.lower() in REALTIME_OUTPUT_AUDIO_FORMATS ): return f"Unsupported output_audio_format: {output_format}" return None - @classmethod - def _validate_conversation_item_audio_formats(cls, item: object) -> str | None: - if not isinstance(item, dict): - return None - content = item.get("content") - if not isinstance(content, list): - return None - for part in content: - if not isinstance(part, dict): - continue - if part.get("type") not in {"input_audio", "audio"}: - continue - raw_format = part.get("format") - parsed_format, _ = cls._parse_realtime_audio_format(raw_format) - if raw_format is not None and not ( - isinstance(parsed_format, str) and parsed_format.lower() in REALTIME_INPUT_AUDIO_FORMATS - ): - return f"Unsupported input_audio format in conversation.item.create: {raw_format}" - return None - @staticmethod def _json_safe_realtime_payload(payload: dict[str, object]) -> dict[str, object]: clean: dict[str, object] = {} @@ -958,7 +945,7 @@ class RealtimeInputTranslator(RealtimeStateOwner): if not isinstance(raw_format, dict): return raw_format, None rate = raw_format.get("rate") or raw_format.get("sample_rate_hz") or raw_format.get("sample_rate") - sample_rate_hz = int(rate) if isinstance(rate, int | float) and rate > 0 else None + sample_rate_hz = validate_input_sample_rate_hz(rate) if rate is not None else None fmt = raw_format.get("type") or raw_format.get("format") if not isinstance(fmt, str): return raw_format, sample_rate_hz @@ -1107,124 +1094,122 @@ class RealtimeInputTranslator(RealtimeStateOwner): return "video_frames entries must be JPEG or PNG images" return None - @staticmethod - def _configured_realtime_turn_detection(session_payload: dict[str, object]) -> tuple[str | None, object]: - field = "turn_detection" if "turn_detection" in session_payload else None - selected = session_payload.get("turn_detection") - audio_config = session_payload.get("audio") - if isinstance(audio_config, dict): - audio_input = audio_config.get("input") - if isinstance(audio_input, dict) and "turn_detection" in audio_input: - field, selected = "audio.input.turn_detection", audio_input["turn_detection"] - return field, selected - - @classmethod - def _validate_realtime_turn_detection(cls, session_payload: dict[str, object]) -> str | None: - field, turn_detection = cls._configured_realtime_turn_detection(session_payload) - if field is None: + def _validate_realtime_turn_detection( + self, + session_payload: dict[str, object], + *, + configured: bool, + normalized: dict[str, object] | None, + ) -> str | None: + effective_turn_detection = normalized if configured else self._turn_detection + if effective_turn_detection is None: if session_payload.get("overlap_policy") == "barge_in_on_speech": - return ( - "overlap_policy='barge_in_on_speech' requires " - "turn_detection.type='server_vad' on the Realtime endpoint" - ) + return "overlap_policy='barge_in_on_speech' requires turn_detection.type='server_vad'" return None - if turn_detection is not None and not isinstance(turn_detection, dict): - return f"{field} must be null or an object with type='server_vad'" - if isinstance(turn_detection, dict): - if turn_detection.get("type") != "server_vad": - return f"{field}.type must be 'server_vad'" - threshold = turn_detection.get("threshold", 0.5) - if not cls._valid_vad_number(threshold, minimum=SILERO_VAD_MIN_THRESHOLD, strict=True, maximum=1): - return f"{field}.threshold must be greater than {SILERO_VAD_MIN_THRESHOLD} and at most 1" - for name in ("prefix_padding_ms", "silence_duration_ms", "min_speech_duration_ms"): - value = turn_detection.get(name) - if value is not None and not cls._valid_vad_number(value, minimum=0): - return f"{field}.{name} must be a non-negative number" - if turn_detection.get("interrupt_response", True) is not True: - return ( - f"{field}.interrupt_response=false is unsupported; " - "use turn_detection=null for model-owned listen/speak" - ) - desired_policy = "barge_in_on_speech" if turn_detection is not None else "listen_only" + + interrupt_response = effective_turn_detection.get("interrupt_response") + if interrupt_response is None: + # Serving resolves the default and validates overlap policy once + # the session's runtime capabilities are known. + return None + desired_policy = "barge_in_on_speech" if interrupt_response else "listen_only" overlap_policy = session_payload.get("overlap_policy") if isinstance(overlap_policy, str) and overlap_policy != desired_policy: return f"overlap_policy={overlap_policy!r} conflicts with turn_detection; expected {desired_policy!r}" return None - @staticmethod - def _valid_vad_number(value: object, *, minimum: float, maximum: float | None = None, strict: bool = False) -> bool: + def validate_server_vad_input_audio(self, session_payload: dict[str, object]) -> str | None: + """Validate turn-based VAD audio after serving resolves runtime capabilities.""" + candidate = self._pending_turn_detection_update + input_format, input_rate, *_ = candidate[3] if candidate is not None else self._realtime_protocol_config() + audio_config = session_payload.get("audio") + audio_input = audio_config.get("input") if isinstance(audio_config, dict) else None + channels = audio_input.get("channels") if isinstance(audio_input, dict) else None + raw_format = session_payload.get("input_audio_format") + if raw_format is None and isinstance(audio_input, dict): + raw_format = audio_input.get("format") + if channels is None and isinstance(raw_format, dict): + channels = raw_format.get("channels") + if channels is None: + channels = 1 + if input_format != "pcm16" or channels != 1 or input_rate not in {16_000, 24_000}: + return "server_vad requires mono PCM16 input at 16 kHz or 24 kHz" + return None + + def _prepare_realtime_turn_detection_update( + self, + session_payload: dict[str, object], + *, + configured: bool, + normalized: dict[str, object] | None, + ) -> tuple[bool, dict[str, object] | None]: + if self._pending_turn_detection_update is not None: + raise RuntimeError("A Realtime session update is already pending") + + turn_detection = dict(normalized) if configured and normalized is not None else None + if configured: + if turn_detection is None: + session_payload.setdefault("overlap_policy", "listen_only") + elif "interrupt_response" in turn_detection: + session_payload.setdefault( + "overlap_policy", "barge_in_on_speech" if turn_detection["interrupt_response"] else "listen_only" + ) + session_payload["turn_detection"] = turn_detection + return configured, turn_detection + + def _realtime_protocol_config(self) -> tuple[str, int, str, int | None, float]: return ( - isinstance(value, int | float) - and not isinstance(value, bool) - and np.isfinite(float(value)) - and (float(value) > minimum if strict else float(value) >= minimum) - and (maximum is None or float(value) <= maximum) + self._input_audio_format, + self._input_sample_rate_hz, + self._output_audio_format, + self._output_sample_rate_hz, + self._overlap_silence_rms, ) - def _prepare_realtime_turn_detection_update(self, session_payload: dict[str, object]) -> None: - field, configured = self._configured_realtime_turn_detection(session_payload) - if field is None: - return - if self._pending_turn_detection_update is not None: - raise RuntimeError("A Realtime turn-detection update is already pending") + def _set_realtime_protocol_config( + self, + config: tuple[str, int, str, int | None, float], + ) -> None: + ( + self._input_audio_format, + self._input_sample_rate_hz, + self._output_audio_format, + self._output_sample_rate_hz, + self._overlap_silence_rms, + ) = config - turn_detection: dict[str, object] | None = None - detector: SileroStreamingVAD | None = None - if configured is None: - session_payload["overlap_policy"] = "listen_only" - else: - assert isinstance(configured, dict) - turn_detection = { - "type": "server_vad", - "interrupt_response": True, - "threshold": 0.5, - "prefix_padding_ms": 300, - "silence_duration_ms": 500, - "min_speech_duration_ms": 96, - **configured, - } - detector = SileroStreamingVAD( - SileroVADConfig( - threshold=float(turn_detection["threshold"]), - prefix_padding_ms=int(turn_detection["prefix_padding_ms"]), - silence_duration_ms=int(turn_detection["silence_duration_ms"]), - min_speech_duration_ms=max(32, int(turn_detection["min_speech_duration_ms"])), - ) - ) - session_payload["turn_detection"] = dict(turn_detection) - session_payload["overlap_policy"] = "barge_in_on_speech" - - self._pending_turn_detection_update = (turn_detection, detector, asyncio.Event()) + def pending_realtime_input_audio_config_changed(self) -> bool: + candidate = self._pending_turn_detection_update + if candidate is None: + return False + candidate_protocol_config = candidate[3] + return candidate_protocol_config[:2] != self._realtime_protocol_config()[:2] async def wait_for_realtime_turn_detection_update(self) -> None: candidate = self._pending_turn_detection_update if candidate is not None: await candidate[2].wait() - def commit_realtime_turn_detection_update(self) -> None: - self._resolve_realtime_turn_detection_update(commit=True) + def commit_realtime_turn_detection_update(self, server_vad: ServerVADConfig | None = None) -> None: + self._resolve_realtime_turn_detection_update(commit=True, server_vad=server_vad) def reject_realtime_turn_detection_update(self) -> None: self._resolve_realtime_turn_detection_update(commit=False) - def _resolve_realtime_turn_detection_update(self, *, commit: bool) -> None: + def _resolve_realtime_turn_detection_update( + self, *, commit: bool, server_vad: ServerVADConfig | None = None + ) -> None: candidate = self._pending_turn_detection_update if candidate is None: return - turn_detection, detector, resolved = candidate - discarded = self._server_vad if commit else detector + configured, turn_detection, resolved, protocol_config = candidate if commit: - self._turn_detection, self._server_vad = turn_detection, detector + self._set_realtime_protocol_config(protocol_config) + if configured: + self._turn_detection = server_vad.as_dict() if server_vad is not None else turn_detection + self._turn_detection_configured = True self._pending_turn_detection_update = None resolved.set() - reset = getattr(discarded, "reset", None) - if callable(reset): - reset() - - def _reset_server_vad(self) -> None: - reset = getattr(self._server_vad, "reset", None) - if callable(reset): - reset() @staticmethod def _input_audio_transcription_config(session_payload: dict[str, object]) -> dict[str, object] | None: @@ -1257,19 +1242,19 @@ class RealtimeInputTranslator(RealtimeStateOwner): return None async def _conversation_item_to_duplex(self, event: dict[str, object]) -> dict[str, object] | None: - item = event.get("item") - if not isinstance(item, dict): + raw_item = event.get("item") + if not isinstance(raw_item, dict): return None - item = self._normalize_conversation_item(item) + public_item = self._normalize_conversation_item(raw_item) previous_item_id = event.get("previous_item_id") if isinstance(previous_item_id, str): - item["_previous_item_id"] = previous_item_id - item_id = str(item["id"]) - item_type = item.get("type") - role = item.get("role") + public_item["_previous_item_id"] = previous_item_id + item_id = str(public_item["id"]) + item_type = public_item.get("type") + role = public_item.get("role") if item_type == "function_call_output": - call_id = item.get("call_id") - output = item.get("output") + call_id = public_item.get("call_id") + output = public_item.get("output") matching_call = any( known.get("type") == "function_call" and known.get("call_id") == call_id for known in self._conversation_items.values() @@ -1307,30 +1292,110 @@ class RealtimeInputTranslator(RealtimeStateOwner): # ``conversation.item.created`` event records it in the shared # input/output protocol state; on failure the client may retry the # same call_id instead of being rejected by provisional history. + content = public_item.get("content") + if not isinstance(content, list): + return None + parsed_audio_formats: dict[int, tuple[object, int | None]] = {} + for index, part in enumerate(content): + if not isinstance(part, dict) or part.get("type") not in {"input_audio", "audio"}: + continue + raw_format = part.get("format") + try: + parsed_format = self._parse_realtime_audio_format(raw_format or self._input_audio_format) + except ValueError as exc: + await self._send_realtime_payload( + self._realtime_error_payload( + "unsupported_audio_format", + str(exc), + event_id=event.get("event_id"), + param="item.content.audio", + ) + ) + return None + if not self._is_supported_realtime_input_format(parsed_format[0]): + await self._send_realtime_payload( + self._realtime_error_payload( + "unsupported_audio_format", + f"Unsupported input_audio format in conversation.item.create: {raw_format}", + event_id=event.get("event_id"), + param="item.content.audio", + ) + ) + return None + parsed_audio_formats[index] = parsed_format if item_type != "message" or role in {"assistant", "system"}: + event["item"] = public_item return { "type": "turn.signal", "event": "conversation.item.create", - "payload": {"item": item}, + "payload": {"item": public_item}, } - self._conversation_items[item_id] = item - content = item.get("content") - if not isinstance(content, list): - return None + server_vad_full_audio_item = self._turn_based_server_vad text_chunks: list[str] = [] audio_events: list[dict[str, object]] = [] - for part in content: + normalized_content: list[object] = [] + normalized_full_audio = False + for index, part in enumerate(content): if not isinstance(part, dict): + normalized_content.append(part) continue if part.get("type") in {"input_text", "text"} and isinstance(part.get("text"), str): text_chunks.append(str(part["text"])) if part.get("type") in {"input_audio", "audio"}: audio = part.get("audio") or part.get("data") - fmt, format_rate = self._parse_realtime_audio_format(part.get("format") or self._input_audio_format) - sample_rate_hz = ( - part.get("sample_rate_hz") or part.get("sample_rate") or format_rate or self._input_sample_rate_hz - ) - if not self._is_supported_realtime_input_format(fmt): + if not isinstance(audio, str) or not audio: + await self._send_realtime_payload( + self._realtime_error_payload( + "bad_audio", + "conversation.item.create input_audio must not be empty", + event_id=event.get("event_id"), + param="item.content.audio", + ) + ) + return None + raw_part_format = part.get("format") + fmt, format_rate = parsed_audio_formats[index] + explicit_sample_rate_hz = part.get("sample_rate_hz") + if explicit_sample_rate_hz is None: + explicit_sample_rate_hz = part.get("sample_rate") + if explicit_sample_rate_hz is None and isinstance(raw_part_format, dict): + explicit_sample_rate_hz = ( + raw_part_format.get("rate") + if "rate" in raw_part_format + else raw_part_format.get("sample_rate_hz", raw_part_format.get("sample_rate")) + ) + sample_rate_hz = explicit_sample_rate_hz or format_rate or self._input_sample_rate_hz + if server_vad_full_audio_item and ( + fmt != self._input_audio_format + or (explicit_sample_rate_hz is not None and explicit_sample_rate_hz != self._input_sample_rate_hz) + ): + await self._send_realtime_payload( + self._realtime_error_payload( + "bad_audio", + "conversation.item.create input_audio must use the Session input format and sample rate", + event_id=event.get("event_id"), + param="item.content.audio", + ) + ) + return None + if server_vad_full_audio_item: + try: + normalized_audio = self._normalize_complete_audio_item_part(audio, self._input_sample_rate_hz) + except ValueError as exc: + await self._send_realtime_payload( + self._realtime_error_payload( + "bad_audio", + str(exc), + event_id=event.get("event_id"), + param="item.content.audio", + ) + ) + return None + normalized_part = dict(part) + normalized_part.pop("data", None) + normalized_part.update(audio=normalized_audio, format="wav", sample_rate_hz=16_000) + normalized_content.append(normalized_part) + normalized_full_audio = True continue try: audio, fmt, sample_rate_hz = convert_input_audio_with_rate( @@ -1341,15 +1406,13 @@ class RealtimeInputTranslator(RealtimeStateOwner): except ValueError as exc: await self._send_realtime_payload( self._realtime_error_payload( - "bad_event", + "bad_audio", str(exc), event_id=event.get("event_id"), - param="sample_rate_hz", + param="item.content.audio", ) ) return None - if not isinstance(audio, str) or not audio: - continue speech_hints = dict(event) speech_hints.update(part) if not self._input_looks_like_speech(speech_hints, audio=audio, fmt=fmt): @@ -1364,31 +1427,69 @@ class RealtimeInputTranslator(RealtimeStateOwner): self._copy_realtime_input_hints(part, payload) self._copy_realtime_input_hints(event, payload) audio_events.append(payload) + continue + normalized_content.append(part) + if server_vad_full_audio_item and normalized_full_audio: + history_item = dict(public_item) + history_item["content"] = normalized_content + history_item["status"] = "completed" + event["item"] = public_item + return { + "type": "turn.signal", + "event": "conversation.item.create", + "payload": { + "item": public_item, + "history_item": history_item, + }, + } if audio_events: - self._input_audio_buffer_has_audio = True - transcript = self._input_transcript_from_item(item) + transcript = self._input_transcript_from_item(public_item) for extra_event in audio_events[1:]: self._pending_outbound.put_nowait(extra_event) commit_payload: dict[str, object] = { "type": "input_audio_buffer.commit", "final": True, - "realtime_item_id": item_id, + "item_id": item_id, "response_create": False, } if transcript: commit_payload["transcript"] = transcript self._pending_outbound.put_nowait(commit_payload) + # This complete item is already committed locally; a later ack + # must not be needed to start accepting the next input turn. + self._reset_realtime_input_buffer_state() + event["item"] = public_item return audio_events[0] if not text_chunks: return None - text_item = dict(item) + text_item = dict(public_item) text_item["status"] = "completed" + event["item"] = text_item return { "type": "turn.signal", "event": "conversation.item.create", "payload": {"item": text_item}, } + @staticmethod + def _normalize_complete_audio_item_part( + audio: str, + sample_rate_hz: int, + ) -> str: + """Package a complete Realtime audio item for the chat fallback path.""" + try: + raw = base64.b64decode(audio.strip(), validate=True) + except (binascii.Error, ValueError) as exc: + raise ValueError("conversation.item.create input_audio is not valid Base64") from exc + if not raw or len(raw) % np.dtype(" None: transcript = event.get("transcript") if not isinstance(transcript, str): @@ -1454,12 +1555,6 @@ class RealtimeInputTranslator(RealtimeStateOwner): } return None - def _pop_pending_commit_item_id(self) -> str: - try: - return self._pending_commit_item_ids.get_nowait() - except asyncio.QueueEmpty: - return f"item_{uuid4().hex}" - async def _emit_input_speech_started(self, event: dict[str, object]) -> None: if self._input_speech_started: return diff --git a/vllm_omni/entrypoints/duplex/realtime_output.py b/vllm_omni/entrypoints/duplex/realtime_output.py index e8c99f218..82ce9761e 100644 --- a/vllm_omni/entrypoints/duplex/realtime_output.py +++ b/vllm_omni/entrypoints/duplex/realtime_output.py @@ -39,6 +39,28 @@ class RealtimeOutputProjector: if event_type == "session.updated": session = self._realtime_session_payload(event.get("session")) return [{"type": "session.updated", "session": session}] + if event_type == "input_audio_buffer.speech_started": + item_id = event.get("item_id") + if isinstance(item_id, str): + self._active_input_item_id = item_id + self._input_speech_started = True + self._input_audio_buffer_has_audio = True + return [ + { + "type": "input_audio_buffer.speech_started", + "audio_start_ms": max(0, int(event.get("audio_start_ms", 0) or 0)), + "item_id": item_id, + } + ] + if event_type == "input_audio_buffer.speech_stopped": + self._input_speech_started = False + return [ + { + "type": "input_audio_buffer.speech_stopped", + "audio_end_ms": max(0, int(event.get("audio_end_ms", 0) or 0)), + "item_id": event.get("item_id") or self._active_input_item_id, + } + ] if event_type in { "session.resumed", "session.heartbeat_ack", @@ -201,14 +223,15 @@ class RealtimeOutputProjector: ), ] if event_type == "input.committed": - event_item_id = event.get("realtime_item_id") - item_id = ( - event_item_id - if isinstance(event_item_id, str) and event_item_id - else self._pop_pending_commit_item_id() - ) + event_item_id = event.get("item_id") + # Acknowledgements carry their own identity; untracked silence + # commits must not consume another turn's item ID. + item_id = event_item_id if isinstance(event_item_id, str) and event_item_id else f"item_{uuid4().hex}" + # The reader may already have accepted audio for the next turn. + if self._active_input_item_id == item_id: + self._reset_realtime_input_buffer_state() item = self._conversation_items.get(item_id) - created_payload: list[dict[str, object]] = [] + payloads = [self._input_audio_buffer_committed_event(item_id=item_id)] if item is None: message = event.get("message") no_response = event.get("no_response") is True @@ -226,19 +249,18 @@ class RealtimeOutputProjector: ), } self._conversation_items[item_id] = item - created_payload.extend(self._conversation_item_added_events(item)) + payloads.extend(self._conversation_item_added_events(item)) item["status"] = "completed" - payloads = created_payload + [ - self._input_audio_buffer_committed_event(item_id=item_id, event=event), - ] transcription_event = self._input_audio_transcription_completed_event(item_id, item) if transcription_event is not None: payloads.append(transcription_event) payloads.append(self._conversation_item_done_event(item)) return payloads if event_type == "input.cancelled": + self._reset_realtime_input_buffer_state() return [{"type": "input_audio_buffer.cleared"}] if event_type == "input_audio_buffer.cleared": + self._reset_realtime_input_buffer_state() return [{"type": "input_audio_buffer.cleared"}] if event_type == "audio.cancelled": response_id = event.get("response_id") @@ -416,7 +438,14 @@ class RealtimeOutputProjector: return [event] message = str(raw_error or event.get("message") or "Duplex runtime error") code = str(event.get("code") or "duplex_error") - return [self._realtime_error_payload(code, message, event_id=event.get("realtime_event_id"))] + return [ + self._realtime_error_payload( + code, + message, + event_id=event.get("realtime_event_id") or event.get("event_id"), + param=event.get("param"), + ) + ] if event_type == "session.closed": return [{"type": "session.closed", "event": event}] return [{"type": f"duplex.{event_type}", "event": event}] @@ -490,7 +519,14 @@ class RealtimeOutputProjector: }, }, ) - payload.setdefault("turn_detection", payload.get("turn_detection")) + if self._turn_detection_configured or "turn_detection" in payload: + turn_detection = payload.get("turn_detection", self._turn_detection) + audio = dict(payload["audio"]) if isinstance(payload.get("audio"), dict) else {} + audio_input = dict(audio["input"]) if isinstance(audio.get("input"), dict) else {} + audio_input["turn_detection"] = turn_detection + audio["input"] = audio_input + payload["audio"] = audio + payload["turn_detection"] = turn_detection payload.setdefault("input_audio_transcription", payload.get("input_audio_transcription")) payload.setdefault("tracing", payload.get("tracing")) return payload diff --git a/vllm_omni/entrypoints/duplex/realtime_session.py b/vllm_omni/entrypoints/duplex/realtime_session.py index c4604ff39..122a7ab52 100644 --- a/vllm_omni/entrypoints/duplex/realtime_session.py +++ b/vllm_omni/entrypoints/duplex/realtime_session.py @@ -42,6 +42,10 @@ class NativeRealtimeSessionProtocol( def bind_sender(self, send_realtime_json) -> None: self._send_realtime_json = send_realtime_json + def bind_native_input_append(self, enabled: bool) -> None: + """Use the serving-selected input path, independent of interruption policy.""" + self._native_input_append = enabled + def _default_session_payload(self) -> dict[str, object]: payload: dict[str, object] = { "model": self._default_model, diff --git a/vllm_omni/entrypoints/duplex/realtime_state.py b/vllm_omni/entrypoints/duplex/realtime_state.py index 4a1e7bdba..babb470a5 100644 --- a/vllm_omni/entrypoints/duplex/realtime_state.py +++ b/vllm_omni/entrypoints/duplex/realtime_state.py @@ -24,6 +24,32 @@ REALTIME_OUTPUT_AUDIO_FORMATS = { "g711_ulaw", "g711_alaw", } +REALTIME_PCM_DEFAULT_SAMPLE_RATE_HZ = 24_000 +REALTIME_PCM_F32_DEFAULT_SAMPLE_RATE_HZ = 16_000 +REALTIME_G711_DEFAULT_SAMPLE_RATE_HZ = 8_000 + +_RealtimeProtocolConfig = tuple[str, int, str, int | None, float] +_PendingTurnDetectionUpdate = tuple[ + bool, + dict[str, object] | None, + asyncio.Event, + _RealtimeProtocolConfig, +] + + +def realtime_default_sample_rate_hz(fmt: object) -> int | None: + if not isinstance(fmt, str): + return None + normalized = fmt.lower() + if normalized in {"pcm16", "pcm_s16le", "s16le", "pcm", "audio/pcm"}: + return REALTIME_PCM_DEFAULT_SAMPLE_RATE_HZ + if normalized in {"pcm_f32le", "audio/pcm_f32le"}: + return REALTIME_PCM_F32_DEFAULT_SAMPLE_RATE_HZ + if normalized in {"g711_ulaw", "g711_alaw", "audio/g711_ulaw", "audio/g711_alaw"}: + return REALTIME_G711_DEFAULT_SAMPLE_RATE_HZ + return None + + REALTIME_ERROR_TYPES_BY_CODE = { "bad_event": "invalid_request_error", "bad_audio": "invalid_request_error", @@ -43,12 +69,16 @@ REALTIME_ERROR_TYPES_BY_CODE = { "runtime_data_plane_text_without_audio": "server_error", "response_error": "server_error", "chat_error": "server_error", + "server_vad_initialization_failed": "server_error", + "server_vad_inference_failed": "server_error", "duplex_session_busy": "rate_limit_error", "resource_exhausted": "rate_limit_error", + "input_backpressure": "rate_limit_error", "response_already_active": "invalid_request_error", "response_not_active": "invalid_request_error", "response_create_without_input": "invalid_request_error", "input_audio_buffer_empty": "invalid_request_error", + "server_vad_manual_commit_unsupported": "invalid_request_error", "missing_item_id": "invalid_request_error", "item_not_found": "invalid_request_error", "playback_item_mismatch": "invalid_request_error", @@ -63,7 +93,6 @@ REALTIME_ERROR_TYPES_BY_CODE = { "native_text_append_unsupported": "invalid_request_error", "runtime_native_stage_role_required": "server_error", "runtime_native_runner_kv_required": "server_error", - "server_vad_unavailable": "server_error", } @@ -109,12 +138,12 @@ class RealtimeSessionState: _default_session_id: object | None = None _default_extra_body: dict[str, object] = field(default_factory=dict) _input_audio_format: str = "pcm16" - _input_sample_rate_hz: int = 16000 + _input_sample_rate_hz: int = REALTIME_PCM_DEFAULT_SAMPLE_RATE_HZ _output_audio_format: str = "pcm16" _overlap_silence_rms: float = 0.003 _turn_detection: dict[str, object] | None = None - _server_vad: object | None = None - _pending_turn_detection_update: tuple[dict[str, object] | None, object | None, asyncio.Event] | None = None + _native_input_append: bool = False + _pending_turn_detection_update: _PendingTurnDetectionUpdate | None = None _send_realtime_json: Any = None _initial_session_update: bool = False _input_speech_started: bool = False @@ -123,13 +152,13 @@ class RealtimeSessionState: _active_response_id: str | None = None _last_response_id: str | None = None _conversation_items: dict[str, dict[str, object]] = field(default_factory=dict) - _pending_commit_item_ids: asyncio.Queue[str] = field(default_factory=asyncio.Queue) _last_conversation_item_id: str | None = None _output_sample_rate_hz: int | None = None _active_input_item_id: str | None = None _input_audio_buffer_has_audio: bool = False _input_audio_buffer_had_non_speech: bool = False _input_audio_buffer_transcript_parts: list[str] = field(default_factory=list) + _turn_detection_configured: bool = False @classmethod def from_query_params(cls, query_params: Any) -> RealtimeSessionState: @@ -197,10 +226,8 @@ class RealtimeStateOwner: _output_audio_format: str = _RealtimeStateField() _overlap_silence_rms: float = _RealtimeStateField() _turn_detection: dict[str, object] | None = _RealtimeStateField() - _server_vad: object | None = _RealtimeStateField() - _pending_turn_detection_update: tuple[dict[str, object] | None, object | None, asyncio.Event] | None = ( - _RealtimeStateField() - ) + _native_input_append: bool = _RealtimeStateField() + _pending_turn_detection_update: _PendingTurnDetectionUpdate | None = _RealtimeStateField() _send_realtime_json: Any = _RealtimeStateField() _initial_session_update: bool = _RealtimeStateField() _input_speech_started: bool = _RealtimeStateField() @@ -209,10 +236,10 @@ class RealtimeStateOwner: _active_response_id: str | None = _RealtimeStateField() _last_response_id: str | None = _RealtimeStateField() _conversation_items: dict[str, dict[str, object]] = _RealtimeStateField() - _pending_commit_item_ids: asyncio.Queue[str] = _RealtimeStateField() _last_conversation_item_id: str | None = _RealtimeStateField() _output_sample_rate_hz: int | None = _RealtimeStateField() _active_input_item_id: str | None = _RealtimeStateField() _input_audio_buffer_has_audio: bool = _RealtimeStateField() _input_audio_buffer_had_non_speech: bool = _RealtimeStateField() _input_audio_buffer_transcript_parts: list[str] = _RealtimeStateField() + _turn_detection_configured: bool = _RealtimeStateField() diff --git a/vllm_omni/entrypoints/duplex/runtime_adapter.py b/vllm_omni/entrypoints/duplex/runtime_adapter.py index 8bf23e1ed..a873f33a8 100644 --- a/vllm_omni/entrypoints/duplex/runtime_adapter.py +++ b/vllm_omni/entrypoints/duplex/runtime_adapter.py @@ -5,6 +5,7 @@ from __future__ import annotations import asyncio from collections.abc import Awaitable, Callable, Iterable, Mapping, MutableMapping +from dataclasses import dataclass, field from importlib import import_module from typing import Any, Protocol @@ -106,6 +107,104 @@ class ServingRuntimeSessionState(Protocol): def clear_continuation(self) -> None: ... +class _TurnBasedPcmAppendBuffer: + """Empty native-audio buffer for chat-fallback sessions. + + Turn-based sessions keep input audio in ``DuplexSession`` and never use + the model-native append path. The runner still shares lifecycle and + cancellation code with native sessions, so it needs an inert buffer for + those common operations. + """ + + @property + def pending_byte_count(self) -> int: + return 0 + + def clear(self) -> None: + return None + + def clear_force_listen(self) -> None: + return None + + def has_pending(self) -> bool: + return False + + def has_reserved(self) -> bool: + return False + + def prepare_append( + self, + payload: dict[str, object], + *, + operation_id: str, + chunk_period_ms: int, + allow_emit: bool, + ) -> PcmAppendReservation | None: + del payload, operation_id, chunk_period_ms, allow_emit + raise RuntimeError("Model-native audio append is not enabled for this session") + + def prepare_commit( + self, + *, + operation_id: str, + chunk_period_ms: int, + ) -> PcmAppendReservation: + del operation_id, chunk_period_ms + raise RuntimeError("Model-native audio append is not enabled for this session") + + def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None: + del chunk_period_ms + return None + + +@dataclass(slots=True) +class TurnBasedServingSessionState: + """Runner-local state for sessions served through chat fallback.""" + + audio_buffer: PcmAppendBuffer = field(default_factory=_TurnBasedPcmAppendBuffer) + input_since_commit: bool = False + speech_since_commit: bool = False + native_context_locked: bool = False + committed_audio_payload: dict[str, object] | None = None + committed_audio_operation_id: str | None = None + committed_audio_reserved_bytes: int = 0 + deferred_response_create: bool = False + deferred_precreate_response: bool = False + data_plane_task: asyncio.Task[None] | None = None + data_plane_restart_requested: bool = False + continuation_owner_id: str | None = None + continuation_units: int = 0 + pending_silence_task: asyncio.Task[bool] | None = None + pending_silence_owner_id: str | None = None + silence_continuation_scheduler: Callable[..., Awaitable[bool]] | None = None + + def retain_committed_audio( + self, + payload: dict[str, object], + *, + operation_id: str | None, + reserved_bytes: int = 0, + ) -> None: + self.committed_audio_payload = payload + self.committed_audio_operation_id = operation_id + self.committed_audio_reserved_bytes += max(0, int(reserved_bytes)) + + def clear_committed_audio(self) -> int: + reserved_bytes = self.committed_audio_reserved_bytes + self.committed_audio_payload = None + self.committed_audio_operation_id = None + self.committed_audio_reserved_bytes = 0 + self.deferred_response_create = False + self.deferred_precreate_response = False + return reserved_bytes + + def clear_continuation(self) -> None: + self.continuation_owner_id = None + self.continuation_units = 0 + self.pending_silence_task = None + self.pending_silence_owner_id = None + + class RuntimeDataPlane(Protocol): def begin_request(self, request_id: str) -> None: ... diff --git a/vllm_omni/entrypoints/duplex/runtime_bridge.py b/vllm_omni/entrypoints/duplex/runtime_bridge.py index 6db5be236..bf839523c 100644 --- a/vllm_omni/entrypoints/duplex/runtime_bridge.py +++ b/vllm_omni/entrypoints/duplex/runtime_bridge.py @@ -39,6 +39,11 @@ class NativeRuntimeBridgeMixin: } async def _open_runtime_session(self, session: DuplexSession, send_json) -> dict[str, object] | bool: + # Turn-based sessions use the chat-fallback path and do not own a + # model-native duplex runtime. Do not send control messages merely + # because the shared engine client exposes the duplex RPC methods. + if not self._uses_serving_runtime_adapter(session.config): + return True contract_error = self._native_runtime_contract_error(session) if contract_error is not None: await send_json( @@ -535,6 +540,8 @@ class NativeRuntimeBridgeMixin: session_config: dict[str, object] | None = None, runtime_config: dict[str, object] | None = None, ) -> bool: + if not self._uses_serving_runtime_adapter(session.config): + return True signal_turn = getattr(self._chat_service.engine_client, "signal_duplex_turn_async", None) if not callable(signal_turn): return True @@ -582,6 +589,8 @@ class NativeRuntimeBridgeMixin: return True async def _close_runtime_session(self, session: DuplexSession, *, reason: str, send_json=None) -> bool: + if not self._uses_serving_runtime_adapter(session.config): + return True close_session = getattr(self._chat_service.engine_client, "close_duplex_session_async", None) if not callable(close_session): return True @@ -1301,11 +1310,19 @@ class NativeRuntimeBridgeMixin: def _cleanup_duplex_session_state(self, session: DuplexSession) -> None: session_id = session.session_id - self._serving_runtime_adapter.remove_session_state(session_id) - self._serving_runtime_adapter.data_plane.close_session( - session_id, - active_request_id=session.active_request_id, - ) + pipeline = self._server_vad_pipelines.pop(session_id, None) + if pipeline is not None: + pipeline.reset() + self._realtime_vad_metrics.session_finished() + adapter = self._serving_runtime_adapter + if adapter is None: + self._serving_session_states.pop(session_id, None) + else: + adapter.remove_session_state(session_id) + adapter.data_plane.close_session( + session_id, + active_request_id=session.active_request_id, + ) def _encode_native_data_plane_audio( self, diff --git a/vllm_omni/entrypoints/duplex/server_vad.py b/vllm_omni/entrypoints/duplex/server_vad.py new file mode 100644 index 000000000..610e51c8c --- /dev/null +++ b/vllm_omni/entrypoints/duplex/server_vad.py @@ -0,0 +1,528 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +from __future__ import annotations + +import asyncio +import hashlib +import threading +import time +from collections.abc import Mapping +from dataclasses import dataclass, replace +from pathlib import Path +from typing import TYPE_CHECKING, Literal, Protocol + +import numpy as np +from vllm.transformers_utils.repo_utils import try_get_local_file + +if TYPE_CHECKING: + from vllm_omni.entrypoints.openai.audio_utils_mixin import StreamingAudioResampler + +SILERO_VAD_REPO_ID = "istupakov/silero-vad-onnx" +SILERO_VAD_REVISION = "8b14476858ef240c50b3884bb38cc67290c1cc70" +SILERO_VAD_FILENAME = "silero_vad.onnx" +SILERO_VAD_SHA256 = "1a153a22f4509e292a94e67d6f9b85e8deb25b4988682b7e174c65279d8788e3" +SILERO_VAD_MIN_THRESHOLD = 0.15 +SILERO_VAD_DEFAULT_MIN_SPEECH_DURATION_MS = 96 + +_SERVER_VAD_FIELDS = { + "type", + "threshold", + "prefix_padding_ms", + "silence_duration_ms", + "create_response", + "interrupt_response", + "min_speech_duration_ms", +} + + +@dataclass(frozen=True, slots=True) +class ServerVADConfig: + """Validated OpenAI-compatible ``server_vad`` session configuration.""" + + type: Literal["server_vad"] = "server_vad" + threshold: float = 0.5 + prefix_padding_ms: int = 300 + silence_duration_ms: int = 500 + create_response: bool = True + # Omission is resolved from runtime capabilities before the session is opened. + interrupt_response: bool | None = None + min_speech_duration_ms: int | None = None + + @classmethod + def from_value(cls, value: object) -> ServerVADConfig: + if not isinstance(value, dict): + raise ValueError("turn_detection must be null or an object") + unknown = sorted(set(value) - _SERVER_VAD_FIELDS) + if unknown: + raise ValueError(f"Unknown server_vad field(s): {', '.join(unknown)}") + vad_type = value.get("type") + if vad_type != "server_vad": + raise ValueError("turn_detection.type must be 'server_vad'") + + threshold = value.get("threshold", 0.5) + if isinstance(threshold, bool) or not isinstance(threshold, int | float) or not 0 <= threshold <= 1: + raise ValueError("server_vad.threshold must be a number between 0 and 1") + + prefix_padding_ms = value.get("prefix_padding_ms", 300) + if isinstance(prefix_padding_ms, bool) or not isinstance(prefix_padding_ms, int) or prefix_padding_ms < 0: + raise ValueError("server_vad.prefix_padding_ms must be a non-negative integer") + + silence_duration_ms = value.get("silence_duration_ms", 500) + if ( + isinstance(silence_duration_ms, bool) + or not isinstance(silence_duration_ms, int) + or silence_duration_ms <= 0 + ): + raise ValueError("server_vad.silence_duration_ms must be a positive integer") + + create_response = value.get("create_response", True) + if not isinstance(create_response, bool): + raise ValueError("server_vad.create_response must be a boolean") + + interrupt_response = value.get("interrupt_response") + if "interrupt_response" in value and not isinstance(interrupt_response, bool): + raise ValueError("server_vad.interrupt_response must be a boolean") + + min_speech_duration_ms = value.get("min_speech_duration_ms") + if min_speech_duration_ms is not None and ( + isinstance(min_speech_duration_ms, bool) + or not isinstance(min_speech_duration_ms, int | float) + or not np.isfinite(float(min_speech_duration_ms)) + or min_speech_duration_ms < 0 + ): + raise ValueError("server_vad.min_speech_duration_ms must be a non-negative number") + + return cls( + type=vad_type, + threshold=float(threshold), + prefix_padding_ms=prefix_padding_ms, + silence_duration_ms=silence_duration_ms, + create_response=create_response, + interrupt_response=interrupt_response, + min_speech_duration_ms=(int(min_speech_duration_ms) if min_speech_duration_ms is not None else None), + ) + + def as_dict(self) -> dict[str, object]: + result: dict[str, object] = { + "type": self.type, + "threshold": self.threshold, + "prefix_padding_ms": self.prefix_padding_ms, + "silence_duration_ms": self.silence_duration_ms, + "create_response": self.create_response, + } + if self.interrupt_response is not None: + result["interrupt_response"] = self.interrupt_response + if self.min_speech_duration_ms is not None: + result["min_speech_duration_ms"] = self.min_speech_duration_ms + return result + + +def parse_session_turn_detection( + payload: Mapping[str, object], +) -> tuple[bool, ServerVADConfig | None]: + """Extract and validate OpenAI-compatible turn detection aliases.""" + configured_values: list[tuple[str, object]] = [] + if "turn_detection" in payload: + configured_values.append(("turn_detection", payload["turn_detection"])) + audio = payload.get("audio") + if isinstance(audio, Mapping): + audio_input = audio.get("input") + if isinstance(audio_input, Mapping) and "turn_detection" in audio_input: + configured_values.append(("audio.input.turn_detection", audio_input["turn_detection"])) + if not configured_values: + return False, None + + first_path, first_value = configured_values[0] + first_config = None if first_value is None else ServerVADConfig.from_value(first_value) + for field_path, value in configured_values[1:]: + config = None if value is None else ServerVADConfig.from_value(value) + if first_config is not None and config is not None: + # An omitted option must not conflict with an explicit alias value. + if first_config.interrupt_response is None: + first_config = replace(first_config, interrupt_response=config.interrupt_response) + if config.interrupt_response is None: + config = replace(config, interrupt_response=first_config.interrupt_response) + if config != first_config: + raise ValueError(f"{first_path} and {field_path} must not specify conflicting values") + return True, first_config + + +class SpeechDetectorBackend(Protocol): + """Detector contract; per-stream state is owned by the pipeline.""" + + frame_samples: int + + def new_state(self) -> object: ... + + def infer(self, frame: np.ndarray, state: object) -> tuple[float, object]: ... + + +class SpeechDetectorBackendProvider(Protocol): + def get(self) -> SpeechDetectorBackend: ... + + +@dataclass(frozen=True, slots=True) +class SpeechEndpointDecision: + speech_started: bool = False + speech_stopped: bool = False + audio_start_ms: int | None = None + audio_end_ms: int | None = None + endpoint_delay_ms: int | None = None + + +class SpeechEndpointPolicy(Protocol): + @property + def speech_active(self) -> bool: ... + + def update( + self, + probability: float, + *, + frame_start_sample: int, + frame_samples: int, + ) -> SpeechEndpointDecision: ... + + def reset(self) -> None: ... + + +class ThresholdEndpointPolicy: + """Apply threshold and trailing-silence endpoint rules.""" + + def __init__(self, config: ServerVADConfig, *, sample_rate_hz: int) -> None: + self.config = config + self.sample_rate_hz = sample_rate_hz + self._speech_active = False + self._candidate_samples = 0 + self._candidate_start_sample: int | None = None + self._silence_samples = 0 + + @property + def speech_active(self) -> bool: + return self._speech_active + + def update( + self, + probability: float, + *, + frame_start_sample: int, + frame_samples: int, + ) -> SpeechEndpointDecision: + if probability >= self.config.threshold: + self._silence_samples = 0 + if not self._speech_active: + if self._candidate_samples == 0: + self._candidate_start_sample = frame_start_sample + self._candidate_samples += frame_samples + min_speech_duration_ms = self.config.min_speech_duration_ms or 0 + min_speech_samples = max(1, round(min_speech_duration_ms * self.sample_rate_hz / 1000)) + if self._candidate_samples < min_speech_samples: + return SpeechEndpointDecision() + self._speech_active = True + detected_start_sample = ( + self._candidate_start_sample if self._candidate_start_sample is not None else frame_start_sample + ) + detected_start_ms = round(detected_start_sample * 1000 / self.sample_rate_hz) + self._candidate_samples = 0 + self._candidate_start_sample = None + return SpeechEndpointDecision( + speech_started=True, + audio_start_ms=max(0, detected_start_ms - self.config.prefix_padding_ms), + ) + return SpeechEndpointDecision() + + if not self._speech_active: + self._candidate_samples = 0 + self._candidate_start_sample = None + return SpeechEndpointDecision() + + # Match Silero v6.2's streaming VAD hysteresis: activation uses the + # configured threshold, while potential speech end starts only below + # ``max(threshold - 0.15, 0.01)``. Once a silence candidate exists, + # intermediate frames keep elapsed time moving but cannot themselves + # close the turn. + negative_threshold = max(self.config.threshold - SILERO_VAD_MIN_THRESHOLD, 0.01) + below_negative_threshold = probability < negative_threshold + if not below_negative_threshold and self._silence_samples == 0: + return SpeechEndpointDecision() + + self._silence_samples += frame_samples + if not below_negative_threshold: + return SpeechEndpointDecision() + silence_limit = max(1, round(self.config.silence_duration_ms * self.sample_rate_hz / 1000)) + if self._silence_samples < silence_limit: + return SpeechEndpointDecision() + + # OpenAI defines ``audio_end_ms`` as the end of the audio sent to the + # model, including the trailing silence used for endpoint detection. + audio_end_sample = frame_start_sample + frame_samples + endpoint_delay_ms = round(self._silence_samples * 1000 / self.sample_rate_hz) + self.reset() + return SpeechEndpointDecision( + speech_stopped=True, + audio_end_ms=max(0, round(audio_end_sample * 1000 / self.sample_rate_hz)), + endpoint_delay_ms=endpoint_delay_ms, + ) + + def reset(self) -> None: + self._speech_active = False + self._candidate_samples = 0 + self._candidate_start_sample = None + self._silence_samples = 0 + + +@dataclass(frozen=True, slots=True) +class ServerVADFrame: + samples: np.ndarray + decision: SpeechEndpointDecision + + +@dataclass(frozen=True, slots=True) +class ServerVADBatch: + frames: tuple[ServerVADFrame, ...] + inference_ms: float + + +class ServerVADPipeline: + """Frame continuous 16 kHz audio and apply acoustic endpointing. + + The pipeline retains only input-resampling residuals, a partial frame, + detector state, and timing counters. Commit-eligible prefix and utterance + audio remain session-owned. + """ + + sample_rate_hz = 16_000 + + def __init__( + self, + backend: SpeechDetectorBackend, + config: ServerVADConfig, + *, + endpoint_policy: SpeechEndpointPolicy | None = None, + ) -> None: + self.backend = backend + self.config = config + self.endpoint_policy = endpoint_policy or ThresholdEndpointPolicy( + config, + sample_rate_hz=self.sample_rate_hz, + ) + self._input_resampler: StreamingAudioResampler | None = None + self._source_sample_rate_hz: int | None = None + self._scratch = np.empty(0, dtype=np.float32) + self._detector_state = backend.new_state() + self._processed_samples = 0 + self._stream_start_ms = 0 + + @property + def speech_active(self) -> bool: + return self.endpoint_policy.speech_active + + @property + def source_sample_rate_hz(self) -> int | None: + return self._source_sample_rate_hz + + @property + def scratch_bytes(self) -> int: + resampler_bytes = self._input_resampler.scratch_bytes if self._input_resampler is not None else 0 + return int(self._scratch.nbytes) + resampler_bytes + + async def push_pcm16( + self, + samples: bytes, + *, + source_sample_rate_hz: int, + ) -> ServerVADBatch: + """Normalize chunked mono PCM16 input and run endpoint detection.""" + if isinstance(source_sample_rate_hz, bool) or source_sample_rate_hz not in {16_000, 24_000}: + raise ValueError("server_vad PCM16 input sample rate must be 16000 or 24000 Hz") + + if len(samples) % np.dtype(" ServerVADBatch: + normalized = np.ascontiguousarray(samples, dtype=np.float32).reshape(-1) + return await asyncio.to_thread(self._push_sync, normalized) + + def _push_sync(self, samples: np.ndarray) -> ServerVADBatch: + started_at = time.perf_counter() + if self._scratch.size: + samples = np.concatenate((self._scratch, samples)) + frame_samples = int(self.backend.frame_samples) + complete_samples = samples.size - samples.size % frame_samples + # Retained state must not keep the whole input chunk alive through a view. + self._scratch = samples[complete_samples:].copy() + + results: list[ServerVADFrame] = [] + for offset in range(0, complete_samples, frame_samples): + frame = np.ascontiguousarray(samples[offset : offset + frame_samples], dtype=np.float32) + frame_start = self._processed_samples + probability, self._detector_state = self.backend.infer(frame, self._detector_state) + probability = min(1.0, max(0.0, float(probability))) + decision = self.endpoint_policy.update( + probability, + frame_start_sample=frame_start, + frame_samples=frame_samples, + ) + if decision.speech_started and decision.audio_start_ms is not None: + # Prefix audio before the most recent reset is no longer available. + decision = replace(decision, audio_start_ms=max(self._stream_start_ms, decision.audio_start_ms)) + + self._processed_samples += frame_samples + results.append( + ServerVADFrame( + samples=frame, + decision=decision, + ) + ) + return ServerVADBatch( + frames=tuple(results), + inference_ms=(time.perf_counter() - started_at) * 1000, + ) + + def reset(self) -> None: + """Discard stream state while preserving the session-relative audio clock.""" + self._stream_start_ms = round(self._processed_samples * 1000 / self.sample_rate_hz) + self._input_resampler = None + self._source_sample_rate_hz = None + self._scratch = np.empty(0, dtype=np.float32) + self._detector_state = self.backend.new_state() + self.endpoint_policy.reset() + + +@dataclass(frozen=True, slots=True) +class _SileroVADState: + model_state: np.ndarray + context: np.ndarray + + +class SileroVADBackend: + """Shared ONNX Runtime Silero v6.2 detector running on CPU.""" + + sample_rate_hz = 16_000 + frame_samples = 512 + context_samples = 64 + model_state_shape = (2, 1, 128) + + def __init__(self, model_path: str | Path) -> None: + try: + import onnxruntime as ort + except ImportError as exc: # pragma: no cover - platform packaging supplies ORT. + raise RuntimeError("server_vad requires ONNX Runtime") from exc + + session_options = ort.SessionOptions() + session_options.inter_op_num_threads = 1 + session_options.intra_op_num_threads = 1 + + self.model_path = Path(model_path) + self._session = ort.InferenceSession( + str(self.model_path), + providers=["CPUExecutionProvider"], + sess_options=session_options, + ) + input_names = {item.name for item in self._session.get_inputs()} + if "input" not in input_names or "sr" not in input_names or "state" not in input_names: + raise RuntimeError(f"Unsupported Silero ONNX input contract: {sorted(input_names)}") + self._warm_up() + + def _warm_up(self) -> None: + self.infer(np.zeros(self.frame_samples, dtype=np.float32), self.new_state()) + + def new_state(self) -> _SileroVADState: + return _SileroVADState( + model_state=np.zeros(self.model_state_shape, dtype=np.float32), + context=np.zeros((1, self.context_samples), dtype=np.float32), + ) + + def infer(self, frame: np.ndarray, state: object) -> tuple[float, object]: + if not isinstance(state, _SileroVADState): + raise TypeError("Silero detector state must be created by SileroVADBackend.new_state()") + model_state = np.ascontiguousarray(state.model_state, dtype=np.float32) + context = np.ascontiguousarray(state.context, dtype=np.float32) + if model_state.shape != self.model_state_shape: + raise ValueError(f"Silero model state must have shape {self.model_state_shape}, got {model_state.shape}") + expected_context_shape = (1, self.context_samples) + if context.shape != expected_context_shape: + raise ValueError(f"Silero context must have shape {expected_context_shape}, got {context.shape}") + + audio = np.ascontiguousarray(frame, dtype=np.float32).reshape(1, -1) + if audio.shape[1] != self.frame_samples: + raise ValueError( + f"Silero detector frame must contain exactly {self.frame_samples} samples, got {audio.shape[1]}" + ) + model_input = np.concatenate((context, audio), axis=1) + inputs = { + "input": model_input, + "state": model_state, + "sr": np.asarray(self.sample_rate_hz, dtype=np.int64), + } + # ONNX Runtime permits concurrent Run calls on one CPU session. Stream + # state is explicit, so independent pipelines can safely share the session. + output = self._session.run(None, inputs) + if len(output) < 2: + raise RuntimeError("Silero ONNX model did not return probability and model state") + probability = float(np.asarray(output[0]).reshape(-1)[0]) + next_state = _SileroVADState( + model_state=np.ascontiguousarray(output[1], dtype=np.float32), + context=np.ascontiguousarray(model_input[:, -self.context_samples :], dtype=np.float32), + ) + return probability, next_state + + +class SileroVADBackendProvider: + """Resolve, verify, and load one Silero model instance per process.""" + + def __init__(self, *, model_path: str | None = None) -> None: + self.model_path = model_path + self._backend: SileroVADBackend | None = None + self._lock = threading.Lock() + + def get(self) -> SileroVADBackend: + if self._backend is not None: + return self._backend + with self._lock: + if self._backend is None: + path = self._resolve_local_artifact() + self._verify_checksum(path) + self._backend = SileroVADBackend(path) + return self._backend + + def _resolve_local_artifact(self) -> Path: + if self.model_path: + path = Path(self.model_path).expanduser() + if path.is_file(): + return path + raise RuntimeError(f"Configured Silero VAD model does not exist: {path}") + cached = try_get_local_file( + model=SILERO_VAD_REPO_ID, + file_name=SILERO_VAD_FILENAME, + revision=SILERO_VAD_REVISION, + ) + if isinstance(cached, Path) and cached.is_file(): + return cached + raise RuntimeError( + "Silero VAD artifact is not available locally. Pre-download the pinned artifact " + "or configure duplex_session.server_vad_model_path." + ) + + def _verify_checksum(self, path: Path) -> None: + digest = hashlib.sha256(path.read_bytes()).hexdigest() + if digest != SILERO_VAD_SHA256: + raise RuntimeError(f"Silero VAD checksum mismatch for {path}: expected {SILERO_VAD_SHA256}, got {digest}") diff --git a/vllm_omni/entrypoints/duplex/serving.py b/vllm_omni/entrypoints/duplex/serving.py index d1a028bd4..7bd690841 100644 --- a/vllm_omni/entrypoints/duplex/serving.py +++ b/vllm_omni/entrypoints/duplex/serving.py @@ -8,9 +8,9 @@ import base64 import binascii import inspect import json -from collections.abc import Mapping +from collections.abc import Mapping, MutableMapping from contextlib import suppress -from dataclasses import dataclass +from dataclasses import dataclass, replace from typing import TYPE_CHECKING import numpy as np @@ -49,12 +49,21 @@ from vllm_omni.entrypoints.duplex.runtime_adapter import ( ServingRuntimeAdapter, ServingRuntimeConfigError, ServingRuntimeSessionState, + TurnBasedServingSessionState, load_serving_runtime_adapter, validate_serving_runtime_adapter, ) from vllm_omni.entrypoints.duplex.runtime_bridge import ( NativeRuntimeBridgeMixin, ) +from vllm_omni.entrypoints.duplex.server_vad import ( + SILERO_VAD_DEFAULT_MIN_SPEECH_DURATION_MS, + SILERO_VAD_MIN_THRESHOLD, + ServerVADPipeline, + SileroVADBackendProvider, + SpeechDetectorBackendProvider, + parse_session_turn_detection, +) from vllm_omni.entrypoints.duplex.session_attachment import ( DuplexJournalGapError, DuplexSessionAttachmentRegistry, @@ -68,6 +77,7 @@ from vllm_omni.entrypoints.duplex.websocket import ( DuplexSessionTasks, DuplexWebSocketActor, ) +from vllm_omni.metrics.realtime import RealtimeVADMetrics if TYPE_CHECKING: from vllm_omni.entrypoints.openai.serving_chat import OmniOpenAIServingChat @@ -85,6 +95,7 @@ class _DuplexSessionHandshake: session: DuplexSession resumed: bool = False attachment_generation: int | None = None + event_id: str | None = None class OmniDuplexSessionHandler( @@ -105,16 +116,27 @@ class OmniDuplexSessionHandler( self, *, chat_service: OmniOpenAIServingChat, + served_model_name: str | None = None, config_timeout_s: float = _DEFAULT_CONFIG_TIMEOUT_S, idle_timeout_s: float = _DEFAULT_IDLE_TIMEOUT_S, + log_stats: bool = True, duplex_session_config: DuplexSessionRuntimeConfig | None = None, serving_runtime_adapter: ServingRuntimeAdapter | None = None, serving_runtime_adapter_path: str | None = None, + server_vad_backend_provider: SpeechDetectorBackendProvider | None = None, ) -> None: self._chat_service = chat_service self._config_timeout_s = config_timeout_s self._idle_timeout_s = idle_timeout_s self._duplex_session_config = duplex_session_config or DuplexSessionRuntimeConfig() + self._server_vad_backend_provider = server_vad_backend_provider or SileroVADBackendProvider( + model_path=self._duplex_session_config.server_vad_model_path, + ) + self._server_vad_pipelines: dict[str, ServerVADPipeline] = {} + self._realtime_vad_metrics = RealtimeVADMetrics( + served_model_name or chat_service.model_config.model, + log_stats=log_stats, + ) self._registry = DuplexSessionRegistry( DuplexCapabilities( supports_model_native_turn_policy=False, @@ -131,6 +153,7 @@ class OmniDuplexSessionHandler( "duplex_serving_adapter_path", None, ) + self._serving_runtime_adapter: ServingRuntimeAdapter | None if serving_runtime_adapter is not None: self._serving_runtime_adapter = validate_serving_runtime_adapter(serving_runtime_adapter) elif isinstance(adapter_path, str) and adapter_path: @@ -139,7 +162,10 @@ class OmniDuplexSessionHandler( self._encode_native_data_plane_audio, ) else: - raise ValueError("A duplex serving runtime adapter must be explicitly configured") + self._serving_runtime_adapter = None + self._serving_session_states: MutableMapping[str, ServingRuntimeSessionState] = ( + self._serving_runtime_adapter.session_states if self._serving_runtime_adapter is not None else {} + ) self._session_tasks: dict[str, DuplexSessionTasks] = {} self._realtime_protocols: dict[str, NativeRealtimeSessionProtocol] = {} self._lease_generations: dict[str, int] = {} @@ -223,7 +249,7 @@ class OmniDuplexSessionHandler( active_response_task.cancel() with suppress(asyncio.CancelledError): await active_response_task - native = self._serving_runtime_adapter.session_states.get(session.session_id) + native = self._serving_session_states.get(session.session_id) if native is not None and native.data_plane_task is not None: data_plane_task = native.data_plane_task native.data_plane_task = None @@ -794,8 +820,27 @@ class OmniDuplexSessionHandler( duration_ms = cls._input_audio_duration_ms(event, payload) return 0 < duration_ms <= session.config.overlap_short_ack_ms + def _create_runtime_session_state(self) -> ServingRuntimeSessionState: + adapter = self._serving_runtime_adapter + if adapter is not None: + return adapter.create_session_state() + return TurnBasedServingSessionState() + def _runtime_session_state(self, session: DuplexSession) -> ServingRuntimeSessionState: - return self._serving_runtime_adapter.session_state(session.session_id) + adapter = self._serving_runtime_adapter + if adapter is not None: + return adapter.session_state(session.session_id) + state = self._serving_session_states.get(session.session_id) + if state is None: + state = TurnBasedServingSessionState() + self._serving_session_states[session.session_id] = state + return state + + def _require_serving_runtime_adapter(self) -> ServingRuntimeAdapter: + adapter = self._serving_runtime_adapter + if adapter is None: + raise RuntimeError("Model-native duplex runtime is not configured for this session") + return adapter def _should_force_listen_for_auto_response_overlap( self, @@ -823,21 +868,25 @@ class OmniDuplexSessionHandler( *, session: DuplexSession, ) -> bool: - for key in ("is_speech", "speech"): - value = event.get(key) - if isinstance(value, bool): - return value - vad = event.get("vad") - if isinstance(vad, dict): - value = vad.get("is_speech") - if isinstance(value, bool): - return value - probability = vad.get("speech_probability", vad.get("probability")) + # Prefer serving-derived fields in ``payload`` over optional client + # hints retained in ``event``. In server-VAD mode the pipeline, rather + # than the client or the RMS fallback below, owns speech classification. + for source in (payload, event): + for key in ("is_speech", "speech"): + value = source.get(key) + if isinstance(value, bool): + return value + vad = source.get("vad") + if isinstance(vad, dict): + value = vad.get("is_speech") + if isinstance(value, bool): + return value + probability = vad.get("speech_probability", vad.get("probability")) + if isinstance(probability, int | float): + return float(probability) >= 0.5 + probability = source.get("speech_probability") if isinstance(probability, int | float): return float(probability) >= 0.5 - probability = event.get("speech_probability") - if isinstance(probability, int | float): - return float(probability) >= 0.5 fmt = payload.get("format") audio = payload.get("audio") @@ -925,14 +974,26 @@ class OmniDuplexSessionHandler( ) return None - config = DuplexSessionConfig.from_event(event) + try: + config = DuplexSessionConfig.from_event(event) + except ValueError as exc: + error: dict[str, object] = { + "type": "error", + "error": str(exc), + "code": "invalid_request_error", + } + event_id = event.get("_realtime_event_id") + if isinstance(event_id, str) and event_id: + error["event_id"] = event_id + await send_json(error) + return None if config.idle_timeout_s == _DEFAULT_IDLE_TIMEOUT_S: config.idle_timeout_s = self._idle_timeout_s - use_native_runtime = self._uses_serving_runtime_adapter(config) + runtime_adapter = self._serving_runtime_adapter if self._uses_serving_runtime_adapter(config) else None runtime_config: dict[str, object] = {} - if use_native_runtime: + if runtime_adapter is not None: try: - runtime_config = await self._serving_runtime_adapter.prepare_runtime_config( + runtime_config = await runtime_adapter.prepare_runtime_config( config, model_config=getattr(self._chat_service, "model_config", None), ) @@ -944,14 +1005,20 @@ class OmniDuplexSessionHandler( return None session_id = event.get("session_id") if isinstance(event.get("session_id"), str) else None session = self._registry.create(config=config, session_id=session_id) - if use_native_runtime: + if runtime_adapter is not None: session.replace_capabilities( - self._serving_runtime_adapter.capabilities( - max_sessions=self._duplex_session_config.max_sessions, - ) + runtime_adapter.capabilities(max_sessions=self._duplex_session_config.max_sessions) ) session.replace_runtime_config(runtime_config) - return _DuplexSessionHandshake(session=session) + session_payload = event.get("session") + self._resolve_server_vad_defaults( + session, config, session_payload if isinstance(session_payload, dict) else event + ) + event_id = event.get("_realtime_event_id") + return _DuplexSessionHandshake( + session=session, + event_id=event_id if isinstance(event_id, str) and event_id else None, + ) async def _resume_session_handshake( self, @@ -1151,7 +1218,8 @@ class OmniDuplexSessionHandler( return None def _uses_serving_runtime_adapter(self, config: DuplexSessionConfig) -> bool: - return self._serving_runtime_adapter.is_enabled(config) + adapter = self._serving_runtime_adapter + return adapter is not None and adapter.is_enabled(config) def _runtime_session_update_error( self, @@ -1161,7 +1229,7 @@ class OmniDuplexSessionHandler( if not self._uses_native_input_append(session): return None try: - self._serving_runtime_adapter.validate_client_extra_body(payload.get("extra_body")) + self._require_serving_runtime_adapter().validate_client_extra_body(payload.get("extra_body")) except ServingRuntimeConfigError as exc: return { "type": "error", @@ -1178,24 +1246,109 @@ class OmniDuplexSessionHandler( ) -> dict[str, object]: if not self._uses_native_input_append(session): return dict(session.runtime_config) - return self._serving_runtime_adapter.runtime_config_for_update( + return self._require_serving_runtime_adapter().runtime_config_for_update( candidate_config, dict(session.runtime_config), ) + def _resolve_server_vad_defaults( + self, + session: DuplexSession, + config: DuplexSessionConfig, + payload: dict[str, object], + ) -> None: + """Resolve omitted options without overriding explicit client policies.""" + if config.server_vad is None: + return + if config.server_vad.interrupt_response is None: + config.server_vad = replace(config.server_vad, interrupt_response=self._uses_native_input_append(session)) + extra = payload.get("extra_body") + overlap_policy = extra.get("overlap_policy") if isinstance(extra, dict) else None + if not isinstance(overlap_policy, str): + overlap_policy = payload.get("overlap_policy") + config.overlap_policy = ( + overlap_policy + if isinstance(overlap_policy, str) + else "barge_in_on_speech" + if config.server_vad.interrupt_response + else "listen_only" + ) + def _runtime_session_candidate_update_error( self, session: DuplexSession, candidate_config: DuplexSessionConfig, + *, + realtime_protocol: NativeRealtimeSessionProtocol | None = None, ) -> dict[str, object] | None: + if candidate_config.server_vad is not None: + desired_policy = "barge_in_on_speech" if candidate_config.server_vad.interrupt_response else "listen_only" + if candidate_config.overlap_policy != desired_policy: + return { + "type": "error", + "code": "unsupported_turn_detection", + "error": f"overlap_policy conflicts with turn_detection; expected {desired_policy!r}", + "param": "overlap_policy", + } if not self._uses_native_input_append(session): - if candidate_config.overlap_policy != DuplexOverlapPolicy.BARGE_IN_ON_SPEECH.value: - return None - return { - "type": "error", - "code": "server_vad_requires_native_duplex", - "error": "Realtime server_vad requires a model-native duplex runtime", - } + if ( + candidate_config.server_vad is not None + and candidate_config.server_vad.min_speech_duration_ms is not None + ): + return { + "type": "error", + "code": "unsupported_turn_detection", + "error": "server_vad.min_speech_duration_ms is only supported by model-native duplex runtimes", + } + requests_interruption = ( + candidate_config.server_vad is not None and candidate_config.server_vad.interrupt_response + ) or candidate_config.overlap_policy == DuplexOverlapPolicy.BARGE_IN_ON_SPEECH.value + if requests_interruption: + return { + "type": "error", + "code": "server_vad_requires_native_duplex", + "error": ( + "Realtime server_vad interruption requires a model-native duplex runtime; " + "set interrupt_response=false for turn-based models" + ), + } + if candidate_config.server_vad is not None and realtime_protocol is not None: + session_payload = candidate_config.extra_body.get("realtime_session_payload") + audio_error = realtime_protocol.validate_server_vad_input_audio( + session_payload if isinstance(session_payload, dict) else {} + ) + if audio_error is not None: + return { + "type": "error", + "code": "unsupported_turn_detection", + "error": audio_error, + "param": "audio.input.format", + } + return None + if candidate_config.server_vad is not None: + if not candidate_config.server_vad.interrupt_response: + return { + "type": "error", + "session_id": session.session_id, + "code": "unsupported_turn_detection", + "error": "Model-native duplex server_vad requires interrupt_response=true", + } + if candidate_config.server_vad.threshold <= SILERO_VAD_MIN_THRESHOLD: + return { + "type": "error", + "session_id": session.session_id, + "code": "unsupported_turn_detection", + "error": ( + f"Model-native duplex server_vad.threshold must be greater than {SILERO_VAD_MIN_THRESHOLD}" + ), + } + if not candidate_config.server_vad.create_response: + return { + "type": "error", + "session_id": session.session_id, + "code": "unsupported_turn_detection", + "error": "Model-native duplex server_vad requires create_response=true", + } if ( candidate_config.instructions != session.config.instructions and self._runtime_session_state(session).native_context_locked @@ -1280,7 +1433,7 @@ class OmniDuplexSessionHandler( session: DuplexSession, committed: DuplexCommittedInput, *, - realtime_item_id: object | None = None, + item_id: object | None = None, ) -> dict[str, object]: payload: dict[str, object] = { "type": "input.committed", @@ -1290,15 +1443,15 @@ class OmniDuplexSessionHandler( "history_len": len(session.history), "message": committed.message, } - if isinstance(realtime_item_id, str) and realtime_item_id: - payload["realtime_item_id"] = realtime_item_id + if isinstance(item_id, str) and item_id: + payload["item_id"] = item_id return payload @staticmethod def _commit_native_audio_input( session: DuplexSession, *, - realtime_item_id: object | None = None, + item_id: object | None = None, transcript: object | None = None, turn_id: int | None = None, ) -> DuplexCommittedInput: @@ -1307,8 +1460,8 @@ class OmniDuplexSessionHandler( transcript=clean_transcript or None, turn_id=turn_id, ) - if isinstance(realtime_item_id, str) and realtime_item_id: - session.register_history_item(realtime_item_id, committed.message) + if isinstance(item_id, str) and item_id: + session.register_history_item(item_id, committed.message) return committed @staticmethod @@ -1316,7 +1469,7 @@ class OmniDuplexSessionHandler( session: DuplexSession, *, committed: DuplexCommittedInput | None = None, - realtime_item_id: object | None = None, + item_id: object | None = None, transcript: object | None = None, ) -> dict[str, object]: message = committed.message if committed is not None else None @@ -1342,12 +1495,15 @@ class OmniDuplexSessionHandler( } if isinstance(transcript, str) and transcript: payload["transcript"] = transcript - if isinstance(realtime_item_id, str) and realtime_item_id: - payload["realtime_item_id"] = realtime_item_id + if isinstance(item_id, str) and item_id: + payload["item_id"] = item_id return payload - @staticmethod - def _apply_session_update(session: DuplexSession, payload: dict[str, object]) -> dict[str, object] | None: + def _apply_session_update( + self, + session: DuplexSession, + payload: dict[str, object], + ) -> dict[str, object] | None: model = payload.get("model") audio_config = payload.get("audio") audio_input = audio_config.get("input") if isinstance(audio_config, dict) else None @@ -1385,6 +1541,14 @@ class OmniDuplexSessionHandler( extra_body_payload = normalize_native_duplex_key(dict(extra_body_payload)) payload = {**payload, "extra_body": extra_body_payload} if isinstance(extra_body_payload, dict) and NATIVE_DUPLEX_KEY in extra_body_payload: + requested_native_duplex = extra_body_payload[NATIVE_DUPLEX_KEY] + if not isinstance(requested_native_duplex, bool): + return { + "type": "error", + "session_id": session.session_id, + "code": "invalid_request_error", + "error": "extra_body.native_duplex must be a boolean", + } current_native_duplex = native_duplex_opt_in(session.config.extra_body) if current_native_duplex is not None and extra_body_payload[NATIVE_DUPLEX_KEY] != current_native_duplex: return { @@ -1498,8 +1662,69 @@ class OmniDuplexSessionHandler( session.config.extra_body["realtime_session_payload"] = ( NativeRealtimeSessionProtocol._json_safe_realtime_payload(payload) ) + try: + turn_detection_configured, server_vad = parse_session_turn_detection(payload) + if turn_detection_configured: + vad_config = replace(session.config, server_vad=server_vad) + self._resolve_server_vad_defaults(session, vad_config, payload) + server_vad = vad_config.server_vad + if ( + not self._uses_native_input_append(session) + and session.turn_detection_config_locked + and (not session.config.turn_detection_configured or server_vad != session.config.server_vad) + ): + return { + "type": "error", + "session_id": session.session_id, + "code": "unsupported_turn_detection", + "error": "turn_detection cannot be changed after the first audio append", + } + session.config.turn_detection_configured = True + session.config.server_vad = server_vad + session.config.overlap_policy = vad_config.overlap_policy + except ValueError as exc: + return { + "type": "error", + "session_id": session.session_id, + "code": "invalid_request_error", + "error": str(exc), + } return None + async def _configure_server_vad(self, session: DuplexSession) -> None: + pipeline = await self._prepare_server_vad_pipeline(session, session.config) + self._install_server_vad_pipeline(session, pipeline) + + def _install_server_vad_pipeline( + self, + session: DuplexSession, + pipeline: ServerVADPipeline | None, + ) -> None: + existing = self._server_vad_pipelines.pop(session.session_id, None) + if existing is not None: + existing.reset() + self._realtime_vad_metrics.session_finished() + if pipeline is None: + return + self._server_vad_pipelines[session.session_id] = pipeline + self._realtime_vad_metrics.session_started() + + async def _prepare_server_vad_pipeline( + self, + session: DuplexSession, + config: DuplexSessionConfig, + ) -> ServerVADPipeline | None: + if config.server_vad is None: + return None + backend = await asyncio.to_thread(self._server_vad_backend_provider.get) + pipeline_config = config.server_vad + if self._uses_native_input_append(session) and pipeline_config.min_speech_duration_ms is None: + pipeline_config = replace( + pipeline_config, + min_speech_duration_ms=SILERO_VAD_DEFAULT_MIN_SPEECH_DURATION_MS, + ) + return ServerVADPipeline(backend, pipeline_config) + def _apply_response_create_options( self, session: DuplexSession, @@ -1580,7 +1805,7 @@ class OmniDuplexSessionHandler( response_extra.update( (key, value) for key, value in extra_body.items() - if key not in self._serving_runtime_adapter.private_runtime_config_keys + if key not in self._require_serving_runtime_adapter().private_runtime_config_keys ) else: response_extra.update(extra_body) @@ -1841,7 +2066,8 @@ class OmniDuplexSessionHandler( # Epoch-scoped request ids prevent state reuse, but explicitly # release projector/parser cursors so cancelled epochs do not # accumulate until the whole session closes. - self._serving_runtime_adapter.data_plane.close_stream(old_request_id) + if self._serving_runtime_adapter is not None: + self._serving_runtime_adapter.data_plane.close_stream(old_request_id) await self._abort_request_background( session, old_request_id, diff --git a/vllm_omni/entrypoints/duplex/session_runner.py b/vllm_omni/entrypoints/duplex/session_runner.py index 821e2c9c7..ed18d8591 100644 --- a/vllm_omni/entrypoints/duplex/session_runner.py +++ b/vllm_omni/entrypoints/duplex/session_runner.py @@ -4,17 +4,23 @@ from __future__ import annotations import asyncio +import binascii import json import uuid from contextlib import suppress from copy import deepcopy +import numpy as np +import pybase64 as base64 from fastapi import WebSocket, WebSocketDisconnect from vllm.logger import init_logger from vllm_omni.engine.duplex.lease import DuplexLeaseActivity from vllm_omni.engine.duplex.messages import DuplexFence -from vllm_omni.entrypoints.duplex.audio import convert_input_audio_with_rate +from vllm_omni.entrypoints.duplex.audio import ( + convert_input_audio_with_rate, + validate_input_sample_rate_hz, +) from vllm_omni.entrypoints.duplex.commit_policy import ( CommitAction, CommitSnapshot, @@ -36,6 +42,7 @@ from vllm_omni.entrypoints.duplex.runtime_adapter import ( ServingRuntimeSessionState, payload_turn_id, ) +from vllm_omni.entrypoints.duplex.server_vad import ServerVADPipeline from vllm_omni.entrypoints.duplex.session_attachment import ( DuplexJournalOverflowError, ) @@ -138,7 +145,7 @@ class DuplexSessionRunnerMixin: await actor.send_json(raw_payload) realtime_protocol.bind_sender(send_realtime_raw) - native: ServingRuntimeSessionState = self._serving_runtime_adapter.create_session_state() + native: ServingRuntimeSessionState = self._create_runtime_session_state() def begin_close(reason: str) -> None: actor.closing = True @@ -291,6 +298,120 @@ class DuplexSessionRunnerMixin: return True return False + def start_chat_response() -> None: + assert session is not None + + async def run_and_notify() -> None: + task = asyncio.current_task() + try: + await self._run_response(session, emit_event) + finally: + await actor.enqueue_event( + { + "type": "__chat_response_finished__", + "_response_task": task, + } + ) + + actor.active_response_task = asyncio.create_task(run_and_notify()) + + def activate_waiting_server_vad_turn() -> None: + assert session is not None + pending_turn = session.activate_server_vad_turn() + if pending_turn is not None and pending_turn.create_response: + start_chat_response() + + def correlate_realtime_error( + payload: dict[str, object], + event_id: object, + ) -> dict[str, object]: + if isinstance(event_id, str) and event_id: + payload["event_id"] = event_id + return payload + + async def reject_and_clear_server_vad_input( + pipeline: ServerVADPipeline, + *, + code: str, + error: str, + metric_reason: str, + event_id: object, + ) -> None: + assert session is not None + session.cancel_pending_input() + pipeline.reset() + self._realtime_vad_metrics.error(metric_reason) + await emit_event( + correlate_realtime_error( + { + "type": "error", + "session_id": session.session_id, + "code": code, + "error": error, + }, + event_id, + ) + ) + await emit_event( + { + "type": "input_audio_buffer.cleared", + "session_id": session.session_id, + } + ) + + async def commit_server_vad_turn( + *, + item_id: str, + create_response: bool, + retain_input_bytes: int, + event_id: object = None, + ) -> None: + assert session is not None + # Retain ownership until the actor consumes the terminal event so + # queued audio cannot overtake pending-turn activation. + active_response = actor.active_response_task is not None + if active_response and session.pending_server_vad_turn is not None: + released = session.discard_uncommitted_server_vad_utterance() + session.release_input_bytes(released) + self._realtime_vad_metrics.error("pending_turn_backpressure") + await emit_event( + correlate_realtime_error( + { + "type": "error", + "session_id": session.session_id, + "code": "input_backpressure", + "error": "Only one committed Server VAD turn may wait for the active response", + }, + event_id, + ) + ) + return + if not session.stage_server_vad_audio_for_commit(): + return + committed = session.commit_user_input( + append_history=not active_response, + retain_input_bytes=retain_input_bytes, + ) + if committed is None: + return + if active_response: + session.stage_server_vad_turn( + committed, + create_response=create_response, + item_id=item_id, + ) + else: + session.register_history_item(item_id, committed.message) + await emit_event( + self._input_committed_payload( + session, + committed, + item_id=item_id, + ) + ) + if create_response and not active_response: + start_chat_response() + def clear_completed_pending_silence() -> None: task = native.pending_silence_task if task is not None and task.done(): @@ -679,11 +800,18 @@ class DuplexSessionRunnerMixin: if handshake is None: return session = handshake.session - if initial_update_error := self._runtime_session_candidate_update_error(session, session.config): - await emit_event(initial_update_error) + if initial_update_error := self._runtime_session_candidate_update_error( + session, session.config, realtime_protocol=realtime_protocol + ): + await emit_event( + correlate_realtime_error( + initial_update_error, + handshake.event_id, + ) + ) return if handshake.resumed: - native = self._serving_runtime_adapter.session_states[session.session_id] + native = self._serving_session_states[session.session_id] actor.tasks = self._session_tasks[session.session_id] persisted_protocol = self._realtime_protocols.get(session.session_id) if persisted_protocol is None: @@ -697,18 +825,36 @@ class DuplexSessionRunnerMixin: self._ensure_lifecycle_listener() reader_task = asyncio.create_task(read_event_loop(), name="duplex-session-reader") else: - self._serving_runtime_adapter.session_states[session.session_id] = native + self._serving_session_states[session.session_id] = native self._session_tasks[session.session_id] = actor.tasks if realtime_protocol is not None: self._realtime_protocols[session.session_id] = realtime_protocol native.silence_continuation_scheduler = schedule_native_silence_continuation if realtime_protocol is not None: + realtime_protocol.bind_native_input_append(self._uses_native_input_append(session)) session.config.playback_commit_policy = DuplexPlaybackCommitPolicy.ACK_ONLY.value if not handshake.resumed: open_result = await self._open_runtime_session(session, emit_event) if open_result is False: return runtime_opened = True + try: + await self._configure_server_vad(session) + except Exception as exc: + logger.exception("Server VAD initialization failed: %s", exc) + self._realtime_vad_metrics.error("initialization") + await emit_event( + correlate_realtime_error( + { + "type": "error", + "session_id": session.session_id, + "code": "server_vad_initialization_failed", + "error": str(exc), + }, + handshake.event_id, + ) + ) + return created_attachment = await self._attachment_registry.create( session.session_id, incarnation=session.incarnation, @@ -734,7 +880,7 @@ class DuplexSessionRunnerMixin: if isinstance(open_result, dict): created_payload["runtime_control"] = self._redact_runtime_control_result(open_result) if realtime_protocol is not None: - realtime_protocol.commit_realtime_turn_detection_update() + realtime_protocol.commit_realtime_turn_detection_update(session.config.server_vad) await emit_event(created_payload) resume_credential_delivered = session.capabilities.supports_session_resume reader_task = asyncio.create_task(read_event_loop(), name="duplex-session-reader") @@ -746,6 +892,14 @@ class DuplexSessionRunnerMixin: if event_type == "__replaced_attachment__": return + if event_type == "__chat_response_finished__": + completed_task = event.get("_response_task") + if completed_task is not actor.active_response_task: + continue + actor.active_response_task = None + activate_waiting_server_vad_turn() + continue + if event_type == "__timeout__": begin_close("timeout") native.audio_buffer.clear() @@ -896,6 +1050,9 @@ class DuplexSessionRunnerMixin: if event_type == "input_audio_buffer.clear": native.audio_buffer.clear() session.release_all_input_bytes() + server_vad_pipeline = self._server_vad_pipelines.get(session.session_id) + if server_vad_pipeline is not None: + server_vad_pipeline.reset() native.input_since_commit = False native.speech_since_commit = False native.clear_committed_audio() @@ -932,10 +1089,8 @@ class DuplexSessionRunnerMixin: if event_type == "response.cancel": requested_response_id = event.get("response_id") has_active_response_work = native_response_in_progress() - if ( - isinstance(requested_response_id, str) - and session.active_response_id is not None - and requested_response_id != session.active_response_id + if isinstance(requested_response_id, str) and ( + session.active_response_id is None or requested_response_id != session.active_response_id ): await emit_event( { @@ -946,6 +1101,7 @@ class DuplexSessionRunnerMixin: } ) continue + if event_type == "response.cancel": if not has_active_response_work: if realtime_protocol is not None and isinstance(requested_response_id, str): continue @@ -1056,6 +1212,7 @@ class DuplexSessionRunnerMixin: } ) actor.active_response_task = None + activate_waiting_server_vad_turn() continue if not cancelled: await self._cancel_pending_input(session, emit_event, reason="barge_in") @@ -1077,6 +1234,7 @@ class DuplexSessionRunnerMixin: ): continue actor.active_response_task = None + activate_waiting_server_vad_turn() continue if event_type == "turn.signal": @@ -1119,6 +1277,27 @@ class DuplexSessionRunnerMixin: ) reject_update() continue + if ( + realtime_protocol is not None + and not self._uses_native_input_append(session) + and session.config.server_vad is not None + and session.turn_detection_config_locked + and realtime_protocol.pending_realtime_input_audio_config_changed() + ): + await emit_update_event( + { + "type": "error", + "session_id": session.session_id, + "code": "unsupported_audio_format", + "error": ( + "session.update cannot change Server VAD input format or sample rate " + "after the first audio append" + ), + "param": "audio.input.format", + } + ) + reject_update() + continue runtime_update_error = self._runtime_session_update_error(session, payload) if runtime_update_error is not None: await emit_update_event(runtime_update_error) @@ -1131,6 +1310,7 @@ class DuplexSessionRunnerMixin: update_error = self._apply_session_update(session, payload) finally: session.replace_config(previous_config) + server_vad_changed = candidate_config.server_vad != previous_config.server_vad if update_error is not None: await emit_update_event(update_error) reject_update() @@ -1138,6 +1318,7 @@ class DuplexSessionRunnerMixin: runtime_update_error = self._runtime_session_candidate_update_error( session, candidate_config, + realtime_protocol=realtime_protocol, ) if runtime_update_error is not None: await emit_update_event(runtime_update_error) @@ -1159,19 +1340,46 @@ class DuplexSessionRunnerMixin: ) reject_update() continue - if not await self._signal_runtime_session( + candidate_server_vad_pipeline = None + if server_vad_changed: + try: + candidate_server_vad_pipeline = await self._prepare_server_vad_pipeline( + session, + candidate_config, + ) + except Exception as exc: + logger.exception("Server VAD session update failed: %s", exc) + self._realtime_vad_metrics.error("initialization") + await emit_update_event( + { + "type": "error", + "session_id": session.session_id, + "code": "server_vad_initialization_failed", + "error": str(exc), + } + ) + reject_update() + continue + runtime_updated = await self._signal_runtime_session( session, turn_event, emit_update_event, session_config=candidate_config.as_dict(), runtime_config=candidate_runtime_config, - ): + ) + if not runtime_updated: reject_update() continue session.replace_config(candidate_config) session.replace_runtime_config(candidate_runtime_config) + if server_vad_changed: + self._install_server_vad_pipeline( + session, + candidate_server_vad_pipeline, + ) if realtime_protocol is not None: - realtime_protocol.commit_realtime_turn_detection_update() + realtime_protocol.bind_native_input_append(self._uses_native_input_append(session)) + realtime_protocol.commit_realtime_turn_detection_update(session.config.server_vad) await emit_update_event( { "type": "session.updated", @@ -1229,7 +1437,10 @@ class DuplexSessionRunnerMixin: expected_model_turn_id=session.turn_id, ) continue - message = self._realtime_item_to_history_message(item) + history_item = payload.get("history_item") if isinstance(payload, dict) else None + if not isinstance(history_item, dict): + history_item = item + message = self._realtime_item_to_history_message(history_item) item_id = item.get("id") if isinstance(item, dict) else None if message is not None: session.append_history_message(message) @@ -1346,24 +1557,30 @@ class DuplexSessionRunnerMixin: default_sample_rate_hz = 16000 sr_raw = event.get("sample_rate_hz") or event.get("sample_rate") sample_rate_hz = sr_raw if isinstance(sr_raw, int | float) else default_sample_rate_hz - try: - audio, fmt, sample_rate_hz = convert_input_audio_with_rate( - audio, - fmt, - sample_rate_hz=sample_rate_hz, - ) - except ValueError as exc: - await emit_event({"type": "error", "error": str(exc), "code": "bad_event"}) - continue - if isinstance(fmt, str) and fmt.lower() in {"pcm16", "pcm_s16le", "s16le"}: - await emit_event( - { - "type": "error", - "error": "input_audio_buffer.append pcm16 audio could not be decoded", - "code": "bad_audio", - } - ) - continue + native_input = self._uses_native_input_append(session) + server_vad_config = session.config.server_vad + turn_based_server_vad = server_vad_config is not None and not native_input + if not turn_based_server_vad: + try: + audio, fmt, sample_rate_hz = convert_input_audio_with_rate( + audio, + fmt, + sample_rate_hz=sample_rate_hz, + ) + except ValueError as exc: + await emit_event({"type": "error", "error": str(exc), "code": "bad_event"}) + continue + if isinstance(fmt, str) and fmt.lower() in {"pcm16", "pcm_s16le", "s16le"}: + await emit_event( + { + "type": "error", + "error": "input_audio_buffer.append pcm16 audio could not be decoded", + "code": "bad_audio", + } + ) + continue + if server_vad_config is None: + session.lock_turn_detection_config() force_listen = bool(event.get("force_listen", False)) payload = { "type": "audio", @@ -1377,11 +1594,186 @@ class DuplexSessionRunnerMixin: frames = [frame for frame in video_frames if isinstance(frame, str) and frame] if frames: payload["video_frames"] = frames + if server_vad_config is not None: + realtime_event_id = event.get("_realtime_event_id") + pipeline = self._server_vad_pipelines.get(session.session_id) + if pipeline is None: + await emit_event( + correlate_realtime_error( + { + "type": "error", + "session_id": session.session_id, + "code": "internal_error", + "error": "server_vad is configured but its detector pipeline is unavailable", + }, + realtime_event_id, + ) + ) + continue + speech_was_active = pipeline.speech_active + reserved_bytes = 0 + previous_scratch_bytes = pipeline.scratch_bytes + try: + if native_input: + if fmt != "pcm_f32le" or sample_rate_hz != pipeline.sample_rate_hz: + raise ValueError("native server_vad input must be mono PCM float32 at 16000 Hz") + raw_audio = base64.b64decode(audio, validate=True) + if len(raw_audio) % np.dtype(" 0 + else: + if fmt != "pcm16": + raise ValueError("server_vad input must be mono PCM16") + source_sample_rate_hz = validate_input_sample_rate_hz(sample_rate_hz) + if source_sample_rate_hz not in {16_000, 24_000}: + raise ValueError("server_vad PCM16 input sample rate must be 16000 or 24000 Hz") + if ( + pipeline.source_sample_rate_hz is not None + and source_sample_rate_hz != pipeline.source_sample_rate_hz + ): + raise ValueError( + "server_vad input sample rate cannot change within a continuous audio stream" + ) + try: + raw_audio = base64.b64decode(audio, validate=True) + except (binascii.Error, ValueError) as exc: + raise ValueError( + "server_vad input audio must be valid base64-encoded PCM16" + ) from exc + if len(raw_audio) % np.dtype(" 0 + except (binascii.Error, ValueError) as exc: + await emit_event( + correlate_realtime_error( + { + "type": "error", + "session_id": session.session_id, + "code": "bad_audio", + "error": str(exc), + "param": "audio", + }, + realtime_event_id, + ) + ) + continue + if not native_input: + normalized_sample_upper_bound = ( + source_samples + if source_sample_rate_hz == pipeline.sample_rate_hz + else ((source_samples + 2) // 3) * 2 + ) + reserved_bytes = normalized_sample_upper_bound * np.dtype(np.float32).itemsize + if not session.reserve_input_bytes( + reserved_bytes, + limit=self._duplex_session_config.max_pending_input_bytes_per_session, + ): + await reject_and_clear_server_vad_input( + pipeline, + code="input_backpressure", + error="Server VAD input exceeds the per-session input limit", + metric_reason="input_backpressure", + event_id=realtime_event_id, + ) + continue + try: + vad_batch = ( + await pipeline.push(normalized_audio) + if native_input + else await pipeline.push_pcm16( + raw_audio, + source_sample_rate_hz=source_sample_rate_hz, + ) + ) + except Exception as exc: + logger.exception("Server VAD inference failed: %s", exc) + await reject_and_clear_server_vad_input( + pipeline, + code="server_vad_inference_failed", + error=str(exc), + metric_reason="inference", + event_id=realtime_event_id, + ) + continue + if not native_input: + retained_delta = ( + sum(frame.samples.nbytes for frame in vad_batch.frames) + + pipeline.scratch_bytes + - previous_scratch_bytes + ) + session.release_input_bytes(max(0, reserved_bytes - retained_delta)) + self._realtime_vad_metrics.observe_inference(vad_batch.inference_ms) + prefix_samples = round(server_vad_config.prefix_padding_ms * pipeline.sample_rate_hz / 1000) + speech_started = False + speech_stopped = False + for frame_index, vad_frame in enumerate(vad_batch.frames): + decision = vad_frame.decision + if not native_input: + released_bytes = session.append_server_vad_frame( + vad_frame.samples, + speech_started=decision.speech_started, + speech_stopped=decision.speech_stopped, + prefix_samples=prefix_samples, + ) + session.release_input_bytes(released_bytes) + if decision.speech_started: + speech_started = True + item_id = f"item_{uuid.uuid4().hex}" + session.begin_server_vad_speech(item_id) + await emit_event( + { + "type": "input_audio_buffer.speech_started", + "session_id": session.session_id, + "item_id": item_id, + "audio_start_ms": decision.audio_start_ms or 0, + } + ) + if decision.speech_stopped: + speech_stopped = True + item_id = session.finish_server_vad_speech() or f"item_{uuid.uuid4().hex}" + await emit_event( + { + "type": "input_audio_buffer.speech_stopped", + "session_id": session.session_id, + "item_id": item_id, + "audio_end_ms": decision.audio_end_ms or 0, + } + ) + if decision.endpoint_delay_ms is not None: + self._realtime_vad_metrics.observe_endpoint_delay(decision.endpoint_delay_ms) + if not native_input: + remaining_frame_bytes = sum( + frame.samples.nbytes for frame in vad_batch.frames[frame_index + 1 :] + ) + await commit_server_vad_turn( + item_id=item_id, + create_response=server_vad_config.create_response, + retain_input_bytes=remaining_frame_bytes + pipeline.scratch_bytes, + event_id=realtime_event_id, + ) + if accepted_audio: + session.lock_turn_detection_config() + if not native_input: + continue + speech_active = pipeline.speech_active + contained_speech = speech_was_active or speech_started or speech_stopped or speech_active + payload["is_speech"] = contained_speech + payload["vad"] = { + "backend": "silero", + "is_speech": contained_speech, + "speech_active": speech_active, + "speech_started": speech_started, + "speech_stopped": speech_stopped, + } + if speech_active: + payload["force_listen"] = True # Speech/silence tag for the Stage0 turn-ended latch. payload["is_speech"] = self._input_looks_like_speech(event, payload, session=session) defer_native_append = False buffer_overlap_audio = True - if self._uses_native_input_append(session): + if native_input: mark_pending_silence_superseded() overlap_active = native_response_in_progress() and ( not self._session_auto_responds(session) @@ -1570,7 +1962,7 @@ class DuplexSessionRunnerMixin: payload = pcm_reservation.payload else: session.append_audio(audio, fmt=fmt, sample_rate_hz=sample_rate_hz) - if self._uses_native_input_append(session): + if native_input: await start_native_append( payload, final=False, @@ -1582,11 +1974,23 @@ class DuplexSessionRunnerMixin: continue if event_type in {"input.commit", "input_audio_buffer.commit", "response.create"}: - realtime_item_id = event.get("realtime_item_id") + if ( + event_type == "response.create" + and session.config.server_vad is not None + and actor.active_response_task is not None + ): + await emit_event( + { + "type": "error", + "session_id": session.session_id, + "code": "response_already_active", + "error": "response.create must wait for the active response to complete", + } + ) + continue + item_id = event.get("item_id") realtime_validated_audio_commit = ( - event_type == "input_audio_buffer.commit" - and isinstance(realtime_item_id, str) - and bool(realtime_item_id) + event_type == "input_audio_buffer.commit" and isinstance(item_id, str) and bool(item_id) ) if ( self._uses_native_input_append(session) @@ -1604,6 +2008,7 @@ class DuplexSessionRunnerMixin: { "type": "input.committed", "session_id": session.session_id, + "item_id": item_id, "turn_id": session.turn_id, "epoch": session.epoch, "empty": True, @@ -1699,6 +2104,7 @@ class DuplexSessionRunnerMixin: { "type": "input.committed", "session_id": session.session_id, + "item_id": item_id, "turn_id": session.turn_id, "epoch": session.epoch, "empty": True, @@ -1736,13 +2142,13 @@ class DuplexSessionRunnerMixin: native.speech_since_commit = False committed = self._commit_native_audio_input( session, - realtime_item_id=event.get("realtime_item_id"), + item_id=event.get("item_id"), transcript=event.get("transcript"), ) committed_payload = self._native_audio_committed_payload( session, committed=committed, - realtime_item_id=event.get("realtime_item_id"), + item_id=event.get("item_id"), transcript=event.get("transcript"), ) committed_payload["overlap_deferred"] = True @@ -1777,7 +2183,7 @@ class DuplexSessionRunnerMixin: data_plane_turn_id = session.turn_id committed = self._commit_native_audio_input( session, - realtime_item_id=event.get("realtime_item_id"), + item_id=event.get("item_id"), transcript=event.get("transcript"), turn_id=data_plane_turn_id, ) @@ -1785,7 +2191,7 @@ class DuplexSessionRunnerMixin: self._native_audio_committed_payload( session, committed=committed, - realtime_item_id=event.get("realtime_item_id"), + item_id=event.get("item_id"), transcript=event.get("transcript"), ) ) @@ -1883,14 +2289,14 @@ class DuplexSessionRunnerMixin: native.input_since_commit = False committed = self._commit_native_audio_input( session, - realtime_item_id=event.get("realtime_item_id"), + item_id=event.get("item_id"), transcript=event.get("transcript"), ) await emit_event( self._native_audio_committed_payload( session, committed=committed, - realtime_item_id=event.get("realtime_item_id"), + item_id=event.get("item_id"), transcript=event.get("transcript"), ) ) @@ -1949,7 +2355,7 @@ class DuplexSessionRunnerMixin: committed = ( self._commit_native_audio_input( session, - realtime_item_id=event.get("realtime_item_id"), + item_id=event.get("item_id"), transcript=event.get("transcript"), ) if native_had_uncommitted_audio @@ -1959,7 +2365,7 @@ class DuplexSessionRunnerMixin: self._native_audio_committed_payload( session, committed=committed, - realtime_item_id=event.get("realtime_item_id"), + item_id=event.get("item_id"), transcript=event.get("transcript"), ) ) @@ -1968,19 +2374,20 @@ class DuplexSessionRunnerMixin: { "type": "input.committed", "session_id": session.session_id, + "item_id": item_id, "empty": True, } ) continue if committed is not None: - realtime_item_id = event.get("realtime_item_id") - if isinstance(realtime_item_id, str): - session.register_history_item(realtime_item_id, committed.message) + item_id = event.get("item_id") + if isinstance(item_id, str): + session.register_history_item(item_id, committed.message) await emit_event( self._input_committed_payload( session, committed, - realtime_item_id=realtime_item_id, + item_id=item_id, ) ) if not should_create_response: @@ -1992,7 +2399,10 @@ class DuplexSessionRunnerMixin: emit_event, reason="new_response", ) - actor.active_response_task = asyncio.create_task(self._run_response(session, emit_event)) + if session.config.server_vad is not None and not self._uses_native_input_append(session): + start_chat_response() + else: + actor.active_response_task = asyncio.create_task(self._run_response(session, emit_event)) continue await emit_event( diff --git a/vllm_omni/entrypoints/duplex/vad.py b/vllm_omni/entrypoints/duplex/vad.py deleted file mode 100644 index 362e500ee..000000000 --- a/vllm_omni/entrypoints/duplex/vad.py +++ /dev/null @@ -1,181 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project - -from __future__ import annotations - -import binascii -from collections.abc import Callable -from dataclasses import dataclass - -import numpy as np -import pybase64 as base64 - -SILERO_VAD_MIN_THRESHOLD = 0.15 - - -class ServerVADUnavailableError(RuntimeError): - pass - - -@dataclass(frozen=True, slots=True) -class SileroVADConfig: - threshold: float = 0.5 - prefix_padding_ms: int = 300 - silence_duration_ms: int = 500 - min_speech_duration_ms: int = 96 - - -@dataclass(frozen=True, slots=True) -class StreamingVADResult: - is_speech: bool - speech_active: bool - speech_started: bool = False - speech_stopped: bool = False - speech_probability: float = 0.0 - speech_start_ms: int | None = None - speech_end_ms: int | None = None - - -class SileroStreamingVAD: - _SAMPLE_RATE_HZ = 16_000 - _WINDOW_SAMPLES = 512 - - def __init__( - self, - config: SileroVADConfig, - *, - frame_scorer: Callable[[np.ndarray], float] | None = None, - ) -> None: - self.config = config - self._frame_scorer = frame_scorer - self._model: object | None = None - self._clear_stream_state() - - def _clear_stream_state(self) -> None: - self._pending = np.empty(0, dtype=np.float32) - self._speech_active = False - self._candidate_samples = 0 - self._candidate_start_sample: int | None = None - self._silence_samples = 0 - self._processed_samples = 0 - self._resample_rate_hz: int | None = None - self._resample_remainder = 0 - - def reset(self) -> None: - self._clear_stream_state() - reset_states = getattr(self._model, "reset_states", None) - if callable(reset_states): - reset_states() - - def process_base64( - self, - audio: object, - *, - fmt: object, - sample_rate_hz: object, - ) -> StreamingVADResult: - if fmt != "pcm_f32le" or not isinstance(audio, str): - raise ValueError("Silero server VAD requires decoded pcm_f32le audio") - try: - raw = base64.b64decode(audio, validate=True) - except (binascii.Error, ValueError) as exc: - raise ValueError("Silero server VAD received invalid base64 audio") from exc - if len(raw) % 4: - raise ValueError("Silero server VAD received an incomplete pcm_f32le frame") - samples = np.frombuffer(raw, dtype=" StreamingVADResult: - samples = np.asarray(samples, dtype=np.float32).reshape(-1) - if samples.size: - self._pending = np.concatenate((self._pending, samples)) - - started = False - stopped = False - start_ms: int | None = None - end_ms: int | None = None - max_probability = 0.0 - contained_speech = self._speech_active - min_speech_samples = self.config.min_speech_duration_ms * 16 - min_silence_samples = self.config.silence_duration_ms * 16 - negative_threshold = max(0.0, self.config.threshold - SILERO_VAD_MIN_THRESHOLD) - - while self._pending.size >= self._WINDOW_SAMPLES: - frame = self._pending[: self._WINDOW_SAMPLES] - self._pending = self._pending[self._WINDOW_SAMPLES :] - frame_start = self._processed_samples - self._processed_samples += self._WINDOW_SAMPLES - probability = min(1.0, max(0.0, float(self._score_frame(frame)))) - max_probability = max(max_probability, probability) - - if self._speech_active: - contained_speech = True - if probability < negative_threshold: - self._silence_samples += self._WINDOW_SAMPLES - if self._silence_samples >= min_silence_samples: - speech_end_sample = self._processed_samples - self._silence_samples - self._speech_active = False - self._silence_samples = 0 - stopped = True - end_ms = max(0, int(round(speech_end_sample / 16))) - else: - self._silence_samples = 0 - continue - - if probability >= self.config.threshold: - if self._candidate_samples == 0: - self._candidate_start_sample = frame_start - self._candidate_samples += self._WINDOW_SAMPLES - if self._candidate_samples >= min_speech_samples: - candidate_start = self._candidate_start_sample or 0 - prefix_samples = self.config.prefix_padding_ms * 16 - self._speech_active = True - self._candidate_samples = self._silence_samples = 0 - self._candidate_start_sample = None - contained_speech = True - started = True - start_ms = max(0, int(round((candidate_start - prefix_samples) / 16))) - else: - self._candidate_samples = 0 - self._candidate_start_sample = None - - return StreamingVADResult( - contained_speech, self._speech_active, started, stopped, max_probability, start_ms, end_ms - ) - - def _score_frame(self, frame: np.ndarray) -> float: - if self._frame_scorer is not None: - return float(self._frame_scorer(frame)) - if self._model is None: - try: - import torch - from silero_vad import load_silero_vad - except ImportError as exc: - raise ServerVADUnavailableError( - "server_vad requires the optional 'silero-vad' package; install vllm-omni[server-vad]" - ) from exc - self._model = load_silero_vad(onnx=False) - self._frame_scorer = lambda value: float( - self._model(torch.from_numpy(np.ascontiguousarray(value)), self._SAMPLE_RATE_HZ).item() - ) - return float(self._frame_scorer(frame)) diff --git a/vllm_omni/entrypoints/openai/api_server.py b/vllm_omni/entrypoints/openai/api_server.py index d49d95175..f10edfc7f 100644 --- a/vllm_omni/entrypoints/openai/api_server.py +++ b/vllm_omni/entrypoints/openai/api_server.py @@ -1290,10 +1290,12 @@ async def omni_init_app_state( state.openai_serving_duplex = None if state.openai_serving_chat is not None and should_enable_duplex_endpoint( state.stage_configs, - config_path=getattr(args, "deploy_config", None), + config_path=getattr(engine_client, "config_path", None) or getattr(args, "deploy_config", None), ): state.openai_serving_duplex = OmniDuplexSessionHandler( chat_service=state.openai_serving_chat, + served_model_name=model_name, + log_stats=state.log_stats, duplex_session_config=getattr(engine_client, "duplex_session_config", None), serving_runtime_adapter_path=getattr(engine_client, "duplex_serving_adapter_path", None), ) @@ -1850,8 +1852,8 @@ async def realtime_websocket(websocket: WebSocket): logger.warning("Duplex warmup still running after 120 s; admitting the client anyway.") duplex_handler = getattr(websocket.app.state, "openai_serving_duplex", None) duplex_query = websocket.query_params.get("duplex") - use_duplex_realtime = ( - duplex_handler is not None and isinstance(duplex_query, str) and duplex_query.lower() in {"1", "true", "on"} + use_duplex_realtime = duplex_handler is not None and ( + duplex_query is None or (isinstance(duplex_query, str) and duplex_query.lower() in {"1", "true", "on"}) ) if use_duplex_realtime and duplex_handler is not None: await duplex_handler.handle_realtime_session(websocket) diff --git a/vllm_omni/entrypoints/openai/audio_utils_mixin.py b/vllm_omni/entrypoints/openai/audio_utils_mixin.py index 3cfcfc31d..e3bce1a67 100644 --- a/vllm_omni/entrypoints/openai/audio_utils_mixin.py +++ b/vllm_omni/entrypoints/openai/audio_utils_mixin.py @@ -1,6 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +import math from io import BytesIO import numpy as np @@ -19,85 +20,152 @@ logger = init_logger(__name__) class StreamingAudioResampler: - """Stateful resampler for streaming mono audio. + """Stateful polyphase resampler for streaming mono float audio. - Only integer downsampling ratios are supported so every input chunk can - produce output without buffering the complete stream. + Retains filter state so output is invariant to input chunk boundaries. """ + _half_filter_width = 10 + _kaiser_beta = 5.0 + _max_polyphase_factor = 2_048 + def __init__(self, source_rate: int, target_rate: int): if source_rate <= 0 or target_rate <= 0: raise ValueError("Audio sample rates must be positive") self.source_rate = source_rate self.target_rate = target_rate - if source_rate < target_rate or source_rate % target_rate != 0: - raise ValueError( - "Streaming audio resampling requires an integer downsampling ratio, " - f"got {source_rate} Hz to {target_rate} Hz" + rate_gcd = math.gcd(source_rate, target_rate) + self._up = target_rate // rate_gcd + self._down = source_rate // rate_gcd + if max(self._up, self._down) > self._max_polyphase_factor: + raise ValueError("source and target sample rates have an unsupported resampling ratio") + self._half_len, self._phase_kernels = self._design_polyphase_filter() + self._history_samples = self._phase_kernels.shape[1] - 1 + self.reset() + + def _design_polyphase_filter(self) -> tuple[int, np.ndarray]: + if self._up == self._down: + return 0, np.ones((1, 1), dtype=np.float32) + + max_rate = max(self._up, self._down) + half_len = self._half_filter_width * max_rate + offsets = np.arange(-half_len, half_len + 1, dtype=np.float64) + cutoff = 1.0 / max_rate + taps = cutoff * np.sinc(cutoff * offsets) + taps *= np.kaiser(taps.size, self._kaiser_beta) + taps /= np.sum(taps) + taps *= self._up + taps = np.ascontiguousarray(taps, dtype=np.float32) + + phases = [taps[phase :: self._up] for phase in range(self._up)] + phase_width = max(phase.size for phase in phases) + kernels = np.zeros((self._up, phase_width), dtype=np.float32) + for phase_index, phase in enumerate(phases): + kernels[phase_index, -phase.size :] = phase[::-1] + return half_len, kernels + + @property + def scratch_bytes(self) -> int: + pending_samples = self._ceil_div(self._input_samples * self._up, self._down) - self._output_samples + return max(0, pending_samples) * np.dtype(np.float32).itemsize + + @staticmethod + def _ceil_div(numerator: int, denominator: int) -> int: + return -(-numerator // denominator) + + def _render_outputs( + self, + combined: np.ndarray, + *, + first_output: int, + output_count: int, + input_start: int, + ) -> np.ndarray: + if output_count <= 0: + return np.empty(0, dtype=np.float32) + + output_indexes = np.arange(first_output, first_output + output_count, dtype=np.int64) + filter_indexes = output_indexes * self._down + self._half_len + source_indexes = filter_indexes // self._up + phases = filter_indexes % self._up + windows = np.lib.stride_tricks.sliding_window_view( + combined, + self._phase_kernels.shape[1], + ) + selected_windows = windows[source_indexes - input_start] + return np.sum( + selected_windows * self._phase_kernels[phases], + axis=1, + dtype=np.float32, + ) + + def _push(self, chunk: np.ndarray) -> np.ndarray: + if chunk.size == 0: + return np.empty(0, dtype=np.float32) + if self._flushed: + raise RuntimeError("cannot process audio after the resampler has been finalized; call reset first") + + combined = np.concatenate((self._history, chunk)) + next_input_samples = self._input_samples + chunk.size + stable_output_samples = max( + 0, + self._ceil_div(next_input_samples * self._up - self._half_len, self._down), + ) + output = self._render_outputs( + combined, + first_output=self._output_samples, + output_count=stable_output_samples - self._output_samples, + input_start=self._input_samples, + ) + if self._history_samples: + self._history = combined[-self._history_samples :].copy() + self._input_samples = next_input_samples + self._output_samples = stable_output_samples + return output + + def _flush(self) -> np.ndarray: + if self._flushed: + return np.empty(0, dtype=np.float32) + + total_output_samples = self._ceil_div(self._input_samples * self._up, self._down) + output_count = total_output_samples - self._output_samples + if output_count: + last_output = total_output_samples - 1 + last_source_index = (last_output * self._down + self._half_len) // self._up + right_padding = max(0, last_source_index - self._input_samples + 1) + combined = np.concatenate((self._history, np.zeros(right_padding, dtype=np.float32))) + output = self._render_outputs( + combined, + first_output=self._output_samples, + output_count=output_count, + input_start=self._input_samples, ) - - self._ratio = source_rate // target_rate - self._buffer = np.empty((0,), dtype=np.float32) - self._buffer_start = 0 - self._total_samples = 0 - self._next_center = 0 - - if self._ratio == 1: - self._taps = np.ones((1,), dtype=np.float32) else: - # Use a Kaiser-windowed sinc and leave a small transition band - # below the target Nyquist frequency. - num_taps = 20 * self._ratio + 1 - half = num_taps // 2 - offsets = np.arange(num_taps, dtype=np.float64) - half - cutoff = 0.95 / self._ratio - taps = cutoff * np.sinc(cutoff * offsets) * np.kaiser(num_taps, 5.0) - self._taps = (taps / taps.sum()).astype(np.float32) + output = np.empty(0, dtype=np.float32) + self._output_samples = total_output_samples + self._flushed = True + return output def process(self, audio: np.ndarray, *, final: bool = False) -> np.ndarray: chunk = np.asarray(audio, dtype=np.float32) if chunk.ndim != 1: raise ValueError(f"Streaming audio resampling only supports mono audio, got shape {chunk.shape}") + chunk = np.ascontiguousarray(chunk, dtype=np.float32) + output = self._push(chunk) + if not final: + return output + tail = self._flush() + if not output.size: + return tail + if not tail.size: + return output + return np.concatenate((output, tail)) - if self._ratio == 1: - return chunk - - if chunk.size: - self._buffer = np.concatenate((self._buffer, chunk)) - self._total_samples += int(chunk.size) - - half = self._taps.size // 2 - max_center = self._total_samples - 1 if final else self._total_samples - 1 - half - if self._next_center > max_center: - return np.empty((0,), dtype=np.float32) - - centers = np.arange(self._next_center, max_center + 1, self._ratio, dtype=np.int64) - first_center = int(centers[0]) - last_center = int(centers[-1]) - window_start = first_center - half - window_end = last_center + half - segment = np.zeros((window_end - window_start + 1,), dtype=np.float32) - - copy_start = max(window_start, self._buffer_start, 0) - copy_end = min(window_end + 1, self._buffer_start + self._buffer.size) - if copy_end > copy_start: - dst_start = copy_start - window_start - src_start = copy_start - self._buffer_start - segment[dst_start : dst_start + copy_end - copy_start] = self._buffer[ - src_start : src_start + copy_end - copy_start - ] - - windows = np.lib.stride_tricks.sliding_window_view(segment, self._taps.size)[:: self._ratio] - output = windows @ self._taps[::-1] - self._next_center = last_center + self._ratio - - keep_from = max(0, self._next_center - half) - drop = min(max(keep_from - self._buffer_start, 0), self._buffer.size) - if drop: - self._buffer = self._buffer[drop:] - self._buffer_start += int(drop) - - return np.asarray(output, dtype=np.float32) + def reset(self) -> None: + self._history = np.zeros(self._history_samples, dtype=np.float32) + self._input_samples = 0 + self._output_samples = 0 + self._flushed = False class AudioMixin: diff --git a/vllm_omni/metrics/definitions.py b/vllm_omni/metrics/definitions.py index 4b0043bad..6cc2825c1 100644 --- a/vllm_omni/metrics/definitions.py +++ b/vllm_omni/metrics/definitions.py @@ -161,6 +161,12 @@ AUDIO_UNDERRUN_S = METRIC_PREFIX + AUDIO_UNDERRUN + "_s" AUDIO_CONTINUITY_OK_METRIC = METRIC_PREFIX + AUDIO_CONTINUITY_OK AUDIO_SKIPPED_REQUESTS_METRIC = METRIC_PREFIX + AUDIO_SKIPPED_REQUESTS +# Realtime Server VAD serving metrics. +REALTIME_VAD_ACTIVE_SESSIONS = METRIC_PREFIX + "realtime_vad_active_sessions" +REALTIME_VAD_INFERENCE_LATENCY_S = METRIC_PREFIX + "realtime_vad_inference_latency_s" +REALTIME_VAD_ENDPOINT_DELAY_S = METRIC_PREFIX + "realtime_vad_endpoint_delay_s" +REALTIME_VAD_ERRORS = METRIC_PREFIX + "realtime_vad_errors" + # ============================================================================ # Diffusion family (per-stage + per-replica diffusion timing breakdowns) diff --git a/vllm_omni/metrics/realtime.py b/vllm_omni/metrics/realtime.py new file mode 100644 index 000000000..a82bb5b17 --- /dev/null +++ b/vllm_omni/metrics/realtime.py @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +from __future__ import annotations + +from prometheus_client import Counter, Gauge, Histogram + +from vllm_omni.metrics import definitions as defs + +_active_sessions = Gauge( + defs.REALTIME_VAD_ACTIVE_SESSIONS, + "Number of active Realtime sessions using server-side VAD.", + labelnames=list(defs.PIPELINE_LABELS), +) +_inference_latency = Histogram( + defs.REALTIME_VAD_INFERENCE_LATENCY_S, + "Server VAD inference time for one appended audio batch, in seconds.", + labelnames=list(defs.PIPELINE_LABELS), + buckets=defs.SECONDS_FAST_BUCKETS, +) +_endpoint_delay = Histogram( + defs.REALTIME_VAD_ENDPOINT_DELAY_S, + "Detected silence between the last speech frame and the committed endpoint, in seconds.", + labelnames=list(defs.PIPELINE_LABELS), + buckets=defs.SECONDS_FAST_BUCKETS, +) +_errors = Counter( + defs.REALTIME_VAD_ERRORS, + "Server VAD errors and input-overflow rejections by reason.", + labelnames=list(defs.FAILED_LABELS), +) + + +class RealtimeVADMetrics: + def __init__(self, model_name: str, log_stats: bool = True) -> None: + self._model_name = model_name + self._log_stats = log_stats + self._active_sessions = _active_sessions.labels(model_name=model_name) + self._inference_latency = _inference_latency.labels(model_name=model_name) + self._endpoint_delay = _endpoint_delay.labels(model_name=model_name) + + def session_started(self) -> None: + if self._log_stats: + self._active_sessions.inc() + + def session_finished(self) -> None: + if self._log_stats: + self._active_sessions.dec() + + def observe_inference(self, latency_ms: float) -> None: + if self._log_stats: + self._inference_latency.observe(latency_ms / 1000) + + def observe_endpoint_delay(self, delay_ms: int) -> None: + if self._log_stats: + self._endpoint_delay.observe(delay_ms / 1000) + + def error(self, reason: str) -> None: + if self._log_stats: + _errors.labels(model_name=self._model_name, reason=reason).inc()