[Frontend] Add server-side VAD for turn-based omni models (#6618)

Signed-off-by: LHXuuu <xulianhao.xlh@antgroup.com>

docs/design/fullduplex.md

@@ -66,7 +66,8 @@ The checkpoint keeps these verified contracts:
 - scheduler data-plane append over a resumable request;
 - stale epoch/turn/response fencing;
 - `/v1/duplex` and OpenAI Realtime projection;
-- optional OpenAI `server_vad` turn detection backed by per-session Silero VAD;
+- optional OpenAI `server_vad` turn detection backed by a process-shared Silero
+  model and per-session VAD state;
 - existing JoyVL behavior.
 
 The checkpoint does not claim:
@@ -652,7 +653,7 @@ EOS decisions advance the model conversation.
 
 Explicit Realtime clients may still use `input_audio_buffer.commit` to create a
 conversation item. The translator validates that wire input before producing a
-commit carrying `realtime_item_id`. Native auto-response may already have
+commit carrying `item_id`. Native auto-response may already have
 streamed those PCM samples into the runtime, so the runner accepts a validated
 commit even when no runtime-side chunk remains. A truly empty explicit buffer
 continues to return `input_audio_buffer_empty`.
@@ -677,8 +678,7 @@ multi-turn runtime evidence.
 
 During auto-response overlap, `preserve_realtime_input` distinguishes "do not
 append this silent chunk to the native buffer" from "clear the open Realtime
-item". Silent overlap no longer discards earlier user PCM. A separate explicit
-`server_vad` policy is documented in the Realtime contract above.
+item". Silent overlap no longer discards earlier user PCM.
 
 The first chunk of one overlapping input item also reserves its target model
 turn. A later Realtime commit uses that reserved identity even if response EOS

docs/serving/realtime_duplex_api.md

@@ -1,17 +1,17 @@
 # Realtime Duplex API
 
 vLLM-Omni serves full-duplex speech models over a WebSocket endpoint,
-`/v1/realtime?duplex=1`, that speaks the OpenAI Realtime vocabulary plus a
+`/v1/realtime`, that speaks the OpenAI Realtime vocabulary plus a
 set of duplex extensions: the model decides when to listen and when to speak,
 the user can talk over the assistant, playback progress is reported back so
 history stays honest, and a dropped connection can resume the same session.
 This page covers how to run a duplex deployment, how to drive it from Python
 with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
 
-The endpoint is served for models that ship a duplex plugin — currently
-MiniCPM-o 4.5, PersonaPlex, and Nemotron VoiceChat — and only for deploy
-configurations that declare `session_mode: duplex`. The runtime architecture
-is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
+The handler is enabled by either a model-native `session_mode: duplex`
+deployment or an explicit `duplex_session` configuration for turn-based
+Server VAD. The runtime architecture is described in
+[Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
 
 ## Quick Start
 
@@ -27,7 +27,7 @@ vllm-omni serve openbmb/MiniCPM-o-4_5 \
 
 `vllm_omni/deploy/minicpmo_4_5.yaml` declares `session_mode: duplex` and
 `duplex_session.max_sessions: 4`, which mounts the WebSocket routes
-`ws://<host>:8099/v1/realtime?duplex=1` (this page) and `ws://<host>:8099/v1/duplex`
+`ws://<host>:8099/v1/realtime` (this page) and `ws://<host>:8099/v1/duplex`
 (the native dialect for raw-protocol clients) next to the usual HTTP API.
 PersonaPlex and Nemotron VoiceChat use `vllm_omni/deploy/personaplex.yaml` and
 `vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml` in the same way.
@@ -240,7 +240,7 @@ on the same helpers.
 
 ## API Reference
 
-This section is the complete wire contract of `/v1/realtime?duplex=1`: how
+This section is the complete wire contract of `/v1/realtime`: how
 the dialect relates to the OpenAI Realtime protocol, which surfaces each
 model's capability flags gate, a catalogue of every client-to-server and
 server-to-client event, and a JSON example for each. `DuplexClient` speaks
@@ -253,10 +253,11 @@ its protocol tests.
 
 ### Endpoint and transport
 
-The endpoint is `ws(s)://<host>/v1/realtime?duplex=1`; the `duplex` query
-parameter (`1`, `true`, or `on`) selects the duplex session handler, and the
-route is mounted only for deployments whose deploy configuration declares
-`session_mode: duplex`. Optional query parameters are `model`, `session_id`,
+The endpoint is `ws(s)://<host>/v1/realtime`. Bare connections select the
+session handler when the deployment declares either `session_mode: duplex`
+or `duplex_session`. `?duplex=1` remains a compatibility alias, while
+`?duplex=0` selects the legacy Realtime handler. Optional query parameters
+are `model`, `session_id`,
 `autostart` (`0` means resume-only), `resume`, and `native_duplex`
 (`minicpmo45_native_duplex` is accepted as a deprecated alias and folded
 into the canonical name). Every message is one JSON object per WebSocket
@@ -342,9 +343,14 @@ Semantic divergences hidden behind shared names:
   before any commit; a commit may end in `response.listen` and no
   response, and the model may open a response with no commit at all.
   OpenAI: commit ⇒ item, `response.create` ⇒ exactly one response.
-- `turn_detection.interrupt_response=false` is rejected;
-  `create_response` is ignored; `semantic_vad` is unsupported; the VAD
-  runs per session (Silero) and implies `overlap_policy=barge_in_on_speech`.
+- With `turn_detection={"type":"server_vad"}`, omitted `interrupt_response`
+  defaults to `false` on turn-based models such as Qwen3-Omni, enabling
+  endpointing and automatic responses without interruption. Explicit `true`
+  is rejected there. Model-native duplex keeps the `true` default and rejects
+  `false`; its VAD implies `overlap_policy=barge_in_on_speech`.
+  This capability-dependent default differs from OpenAI's `true` default;
+  the effective configuration is returned in `session.created` / `session.updated`.
+  `semantic_vad` is unsupported.
 - `rate_limits.updated` is always an empty list (compatibility only).
 - Cancellation never reuses a `response_id`; truncation is driven by
   `playback.ack` as well as `conversation.item.truncate`.
@@ -1201,9 +1207,10 @@ Realtime projector renames and fans out these events before they reach a
   barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
   or audio truncation; camera frames are consumed only by MiniCPM-o 4.5; tool
   calls are produced only by Nemotron VoiceChat.
-- `turn_detection` supports only `server_vad` with `interrupt_response=true`;
-  `semantic_vad`, `interrupt_response=false`, and `create_response` are not
-  supported. Server VAD requires a model-native session.
+- `turn_detection` supports `server_vad` or `null`, not `semantic_vad`.
+  Turn-based models support endpointing without interruption, including
+  `create_response=false` for explicit response creation. Model-native duplex
+  requires `interrupt_response=true` and `create_response=true`.
 - `input_audio_transcription` and `input_audio_noise_reduction` are accepted
   and echoed but no separate transcription or noise-reduction stage runs;
   transcripts come from the model.

examples/online_serving/qwen3_omni/README.md

@@ -264,7 +264,7 @@ python examples/online_serving/openai_chat_completion_client_for_multimodal_gene
 
 #### Realtime WebSocket client (`openai_realtime_client.py`)
 
-[`openai_realtime_client.py`](./openai_realtime_client.py) connects to **`ws://<host>:<port>/v1/realtime`**, streams a local WAV as **PCM16 mono @ 16 kHz** in fixed-size chunks (OpenAI-style `input_audio_buffer.append` / `commit`), and receives **`response.audio.delta`** (incremental PCM for the reply) plus **`transcription.*`** events. By default it concatenates audio deltas and writes **`--output-wav`** (model output is typically **24 kHz**). Optional **`--delta-dump-dir`** saves each delta as `delta_000001.wav`, … for debugging.
+[`openai_realtime_client.py`](./openai_realtime_client.py) connects to **`ws://<host>:<port>/v1/realtime`**, streams a local WAV as **PCM16 mono @ 16 kHz** in fixed-size chunks, and receives **`response.audio.*`**, **`response.output_text.*`**, and transcript events. By default it uses explicit `input_audio_buffer.commit`; pass **`--server-vad`** to let the server detect speech endpoints and commit the turn automatically. The client concatenates audio deltas and writes **`--output-wav`** (model output is typically **24 kHz**). Optional **`--delta-dump-dir`** saves each delta as `delta_000001.wav`, … for debugging.
 
 Streaming input works well for translation-style use cases; if the Thinker runs while input is still incomplete, consider limiting **`max_tokens`** in your session / server defaults to avoid over-generation.
 
@@ -282,6 +282,7 @@ python openai_realtime_client.py \
   --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
   --input-wav /path/to/input_16k_mono.wav \
   --output-wav realtime_output.wav \
+  --server-vad \
   --delta-dump-dir ./rt_delta_wavs
 ```
 
@@ -297,15 +298,32 @@ python openai_realtime_client.py \
 | `--chunk-ms` | `200` | Size of each uploaded audio chunk (milliseconds of audio) |
 | `--send-delay-ms` | `0` | Delay between chunk sends (simulate realtime upload) |
 | `--delta-dump-dir` | *(optional)* | Directory to write per-`response.audio.delta` WAV files |
+| `--server-vad` | disabled | Enable server-side Silero VAD instead of sending explicit commits |
 | `--num-requests` | `1` | Number of sequential sessions (see `--concurrency`) |
 | `--concurrency` | `1` | Max concurrent WebSocket sessions when `--num-requests` > 1 |
 
-Ensure the server is running, for example:
+Server VAD uses the pinned Silero v6.2 ONNX artifact from `istupakov/silero-vad-onnx` and never downloads model files while processing a session. Make the artifact available in the Hugging Face cache before startup, or configure a local artifact in the deployment YAML:
+
+```yaml
+duplex_session:
+  server_vad_model_path: /models/silero_vad.onnx
+```
+
+Start from the bundled Qwen3-Omni deployment YAML, add the fields above at the top level, and serve that complete configuration:
 
 ```bash
-vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
+cp vllm_omni/deploy/qwen3_omni_moe.yaml /path/to/qwen3_omni_server_vad.yaml
+# Edit /path/to/qwen3_omni_server_vad.yaml and add duplex_session.
+vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
+  --omni \
+  --port 8091 \
+  --deploy-config /path/to/qwen3_omni_server_vad.yaml
 ```
 
+Keep Qwen's default `session_mode: turn`; `duplex_session` enables the Realtime handler without changing scheduler
+semantics. Bare `/v1/realtime` selects that handler. The client uses `?duplex=0` only for the existing non-Server-VAD
+wire flow; `?duplex=1` remains a supported compatibility alias.
+
 The Python client supports the following command-line arguments:
 
 - `--query-type` (or `-q`): Query type (default: `use_video`). Options: `text`, `use_audio`, `use_image`, `use_video`

examples/online_serving/qwen3_omni/openai_realtime_client.py

@@ -1,9 +1,12 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 """Realtime client for vLLM-Omni /v1/realtime (audio + text events).
 
 This client:
 1) Reads a local WAV file (must be mono, 16-bit PCM, 16kHz),
 2) Streams PCM16 chunks to /v1/realtime with OpenAI-style events,
-3) Receives response.audio.* and transcription.* events,
+3) Receives response.audio.*, response.output_text.*, and transcript events,
 4) Saves synthesized audio to an output WAV file and optional text file.
 
 By default each ``response.audio.delta`` is treated as an **incremental PCM**
@@ -18,6 +21,7 @@ Usage:
       --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
       --input-wav input_16k_mono.wav \
       --output-wav realtime_output.wav \
+      --server-vad \
       --delta-dump-dir ./rt_delta_wavs
 
 Dependencies:
@@ -32,6 +36,7 @@ import base64
 import json
 import wave
 from pathlib import Path
+from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
 
 try:
     import websockets
@@ -70,6 +75,14 @@ def _write_wav_pcm16(path: Path, pcm16_bytes: bytes, sample_rate_hz: int) -> Non
         wf.writeframes(pcm16_bytes)
 
 
+def _with_realtime_route(url: str, *, server_vad: bool) -> str:
+    parts = urlsplit(url)
+    query = [(key, value) for key, value in parse_qsl(parts.query, keep_blank_values=True) if key != "duplex"]
+    if not server_vad:
+        query.append(("duplex", "0"))
+    return urlunsplit(parts._replace(query=urlencode(query)))
+
+
 async def run_client(
     url: str,
     model: str,
@@ -79,6 +92,7 @@ async def run_client(
     chunk_ms: int,
     send_delay_ms: int,
     delta_dump_dir: Path | None,
+    server_vad: bool,
     request_idx: int = 1,
     total_requests: int = 1,
 ) -> None:
@@ -92,23 +106,44 @@ async def run_client(
     delta_index = 0
     text_chunks: list[str] = []
     final_text: str = ""
+    final_audio_transcript: str = ""
 
     if delta_dump_dir is not None:
         delta_dump_dir.mkdir(parents=True, exist_ok=True)
 
+    # Keep the CLI mode authoritative even if the input URL already contains
+    # an explicit duplex query parameter.
+    url = _with_realtime_route(url, server_vad=server_vad)
     async with websockets.connect(url, max_size=64 * 1024 * 1024) as ws:
         # 1) Validate model.
-        await ws.send(
-            json.dumps(
-                {
-                    "type": "session.update",
+        if server_vad:
+            session_update = {
+                "type": "session.update",
+                "session": {
                     "model": model,
-                }
-            )
-        )
+                    "audio": {
+                        "input": {
+                            "format": {"type": "audio/pcm", "rate": 16_000},
+                            "turn_detection": {
+                                "type": "server_vad",
+                                "threshold": 0.5,
+                                "prefix_padding_ms": 300,
+                                "silence_duration_ms": 500,
+                                "create_response": True,
+                                "interrupt_response": False,
+                            },
+                        }
+                    },
+                },
+            }
+        else:
+            # Preserve the existing non-duplex wire format.
+            session_update = {"type": "session.update", "model": model}
+        await ws.send(json.dumps(session_update))
 
-        # 2) Start generation once (non-final commit).
-        await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": False}))
+        # 2) Preserve the legacy explicit-commit flow unless Server VAD owns endpointing.
+        if not server_vad:
+            await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": False}))
 
         # 3) Stream audio chunks.
         for i in range(0, len(pcm16), chunk_bytes):
@@ -124,8 +159,20 @@ async def run_client(
             if send_delay_ms > 0:
                 await asyncio.sleep(send_delay_ms / 1000.0)
 
-        # 4) Final commit closes input stream.
-        await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": True}))
+        # 4) Finalize the turn explicitly, or append silence for Server VAD endpointing.
+        if server_vad:
+            trailing_silence = bytes(16_000 * 2)
+            for i in range(0, len(trailing_silence), chunk_bytes):
+                await ws.send(
+                    json.dumps(
+                        {
+                            "type": "input_audio_buffer.append",
+                            "audio": base64.b64encode(trailing_silence[i : i + chunk_bytes]).decode("utf-8"),
+                        }
+                    )
+                )
+        else:
+            await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": True}))
 
         # 5) Receive server events until audio done.
         while True:
@@ -144,7 +191,7 @@ async def run_client(
                 sr = event.get("sample_rate_hz")
                 if isinstance(sr, int) and sr > 0:
                     output_sample_rate = sr
-                audio_b64 = event.get("audio", "")
+                audio_b64 = event.get("delta" if server_vad else "audio", "")
                 if audio_b64:
                     pcm_delta = base64.b64decode(audio_b64)
                     incremental_pcm_parts.append(pcm_delta)
@@ -158,14 +205,14 @@ async def run_client(
                         )
                 continue
 
-            if event_type == "transcription.delta":
+            if event_type in {"response.output_text.delta", "transcription.delta"}:
                 delta = event.get("delta", "")
                 if delta:
                     text_chunks.append(delta)
                     print(delta, end="", flush=True)
                 continue
 
-            if event_type == "transcription.done":
+            if event_type in {"response.output_text.done", "transcription.done"}:
                 final_text = event.get("text", "") or "".join(text_chunks)
                 usage = event.get("usage")
                 final_text_with_tag = f"Final transcription: {final_text}"
@@ -176,7 +223,16 @@ async def run_client(
                     print(f"{log_prefix}text usage: {usage}")
                 continue
 
+            if event_type == "response.audio_transcript.done":
+                final_audio_transcript = event.get("transcript", "")
+                continue
+
             if event_type == "response.audio.done":
+                if not server_vad:
+                    break
+                continue
+
+            if event_type == "response.done" and server_vad:
                 break
 
             if event_type == "error":
@@ -191,7 +247,7 @@ async def run_client(
         print(f"{log_prefix}Saved realtime audio to: {output_wav} (incremental chunks joined)")
 
         if output_text is not None:
-            text_to_save = final_text if final_text else "".join(text_chunks)
+            text_to_save = final_text or "".join(text_chunks) or final_audio_transcript
             output_text.parent.mkdir(parents=True, exist_ok=True)
             output_text.write_text(text_to_save, encoding="utf-8")
             print(f"{log_prefix}Saved realtime text to: {output_text}")
@@ -213,6 +269,7 @@ async def run_clients_concurrent(
     chunk_ms: int,
     send_delay_ms: int,
     delta_dump_dir: Path | None,
+    server_vad: bool,
     num_requests: int,
     concurrency: int,
 ) -> None:
@@ -235,6 +292,7 @@ async def run_clients_concurrent(
                     chunk_ms=chunk_ms,
                     send_delay_ms=send_delay_ms,
                     delta_dump_dir=per_delta_dir,
+                    server_vad=server_vad,
                     request_idx=index,
                     total_requests=num_requests,
                 )
@@ -283,6 +341,11 @@ def main() -> None:
         default=None,
         help="If set, each response.audio.delta is saved as delta_NNNNNN.wav under this directory",
     )
+    parser.add_argument(
+        "--server-vad",
+        action="store_true",
+        help="Let the server detect speech endpoints instead of sending input_audio_buffer.commit",
+    )
     parser.add_argument("--num-requests", type=int, default=1, help="Total number of requests to send")
     parser.add_argument(
         "--concurrency",
@@ -309,6 +372,7 @@ def main() -> None:
                 chunk_ms=args.chunk_ms,
                 send_delay_ms=args.send_delay_ms,
                 delta_dump_dir=args.delta_dump_dir,
+                server_vad=args.server_vad,
             )
         )
     else:
@@ -322,6 +386,7 @@ def main() -> None:
                 chunk_ms=args.chunk_ms,
                 send_delay_ms=args.send_delay_ms,
                 delta_dump_dir=args.delta_dump_dir,
+                server_vad=args.server_vad,
                 num_requests=args.num_requests,
                 concurrency=concurrency,
             )

pyproject.toml

@@ -77,7 +77,6 @@ dev = [
     # Provides top-level cosyvoice2; do not co-install stepaudio2-minicpmo,
     # whose top-level stepaudio2 package conflicts with this distribution.
     "step-audio2==1.0.0", # Step-Audio2 / MiniCPM-o 4.5 Token2Wav (flashcosyvoice / hyperpyyaml)
-    "silero-vad==6.1", # Realtime server-VAD duplex E2E
 ]
 
 demo = [
@@ -87,9 +86,6 @@ demo = [
     "cn2an>=0.5.22",  # Chinese number normalization for TTS (e.g. "1次" -> "一次")
 ]
 
-server-vad = [
-    "silero-vad==6.1",
-]
 
 forced-aligner = [
     "qwen-asr>=0.0.6",  # faithful Japanese/Korean word segmentation for the TTS forced aligner

tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py

@@ -30,10 +30,27 @@ from tests.e2e.online_serving.run_minicpmo_realtime_duplex_soft_interrupt import
     run_soft_interrupt,
 )
 from tests.helpers.mark import hardware_test
+from vllm_omni.entrypoints.duplex.server_vad import (
+    SILERO_VAD_FILENAME,
+    SILERO_VAD_REPO_ID,
+    SILERO_VAD_REVISION,
+)
 
 pytestmark = [pytest.mark.full_model, pytest.mark.omni]
 
 
+@pytest.fixture(scope="module")
+def _cached_server_vad_artifact() -> None:
+    """Cache the pinned ONNX artifact before the Realtime server starts."""
+    from huggingface_hub import hf_hub_download
+
+    hf_hub_download(
+        repo_id=SILERO_VAD_REPO_ID,
+        filename=SILERO_VAD_FILENAME,
+        revision=SILERO_VAD_REVISION,
+    )
+
+
 @hardware_test(res={"cuda": "H100", "npu": "A3"}, num_cards=1)
 @pytest.mark.parametrize("omni_server", SERVER_PARAMS, indirect=True)
 def test_duplex_admission_and_expiry_reaper(omni_server, tmp_path: Path) -> None:
@@ -97,7 +114,7 @@ def test_duplex_soft_interrupt(omni_server, tmp_path: Path) -> None:
 
 @hardware_test(res={"cuda": "H100"}, num_cards=1)
 @pytest.mark.parametrize("omni_server", SERVER_PARAMS, indirect=True)
-def test_duplex_server_vad_hard_interrupt(omni_server) -> None:
+def test_duplex_server_vad_hard_interrupt(_cached_server_vad_artifact, omni_server) -> None:
     result = asyncio.run(
         run_server_vad_interrupt(
             SimpleNamespace(

tests/engine/duplex/test_duplex_deploy_config.py

@@ -33,6 +33,7 @@ def test_duplex_session_runtime_defaults_are_typed_and_immutable(tmp_path) -> No
     assert deploy.duplex_session.max_pending_turns_per_session == 4
     assert deploy.duplex_session.max_sessions == 1
     assert deploy.duplex_session.completed_append_cache_size == 256
+    assert deploy.duplex_session.server_vad_model_path is None
     with pytest.raises(FrozenInstanceError):
         deploy.duplex_session.idle_ttl_s = 1.0  # type: ignore[misc]
 
@@ -46,6 +47,29 @@ def test_duplex_session_runtime_accepts_disabled_idle_expiry(tmp_path) -> None:
     assert deploy.duplex_session.idle_ttl_s is None
 
 
+def test_duplex_session_runtime_accepts_local_server_vad_model(tmp_path) -> None:
+    deploy_path = tmp_path / "duplex.yaml"
+    deploy_path.write_text(
+        "duplex_session:\n  server_vad_model_path: /models/silero_vad.onnx\nstages: []\n",
+        encoding="utf-8",
+    )
+
+    deploy = load_deploy_config(deploy_path)
+
+    assert deploy.duplex_session.server_vad_model_path == "/models/silero_vad.onnx"
+
+
+def test_duplex_session_runtime_rejects_empty_server_vad_model_path(tmp_path) -> None:
+    deploy_path = tmp_path / "duplex.yaml"
+    deploy_path.write_text(
+        "duplex_session:\n  server_vad_model_path: ''\nstages: []\n",
+        encoding="utf-8",
+    )
+
+    with pytest.raises(ValueError, match="server_vad_model_path"):
+        load_deploy_config(deploy_path)
+
+
 @pytest.mark.parametrize(
     ("name", "value"),
     [

tests/engine/test_single_stage_mode.py

@@ -376,14 +376,26 @@ class TestSingleStageModeDetection:
         mocker: MockerFixture,
         *,
         stage_cfgs: list[Any] | None = None,
+        resolved_config_path: str = "/fake/path",
+        patch_deploy_config: bool = True,
         **kwargs: Any,
     ) -> AsyncOmniEngine:
         mock_stage_configs = stage_cfgs or [_make_stage_cfg(0)]
 
+        if patch_deploy_config:
+            mocker.patch.object(
+                StageConfigFactory,
+                "get_pipeline_config",
+                return_value=None,
+            )
+            mocker.patch(
+                "vllm_omni.engine.async_omni_engine.load_deploy_config",
+                return_value=SimpleNamespace(duplex_session=DuplexSessionRuntimeConfig()),
+            )
         mocker.patch.object(
             AsyncOmniEngine,
             "_resolve_stage_configs",
-            return_value=("/fake/path", mock_stage_configs),
+            return_value=(resolved_config_path, mock_stage_configs),
         )
         mocker.patch.object(AsyncOmniEngine, "_bootstrap_orchestrator")
         mock_thread_cls = mocker.patch("threading.Thread")
@@ -445,6 +457,8 @@ class TestSingleStageModeDetection:
         engine = self._make_engine_no_thread(
             mocker,
             deploy_config="/fake/duplex.yaml",
+            resolved_config_path="/fake/duplex.yaml",
+            patch_deploy_config=False,
         )
 
         get_pipeline_config.assert_called_once_with(
@@ -455,6 +469,30 @@ class TestSingleStageModeDetection:
         load_deploy_config.assert_called_once_with("/fake/duplex.yaml")
         assert engine.duplex_session_config is duplex_session
 
+    def test_auto_discovered_deploy_loads_duplex_runtime_config(
+        self,
+        mocker: MockerFixture,
+    ) -> None:
+        deploy_path = "/resolved/qwen3_omni_moe.yaml"
+        duplex_session = DuplexSessionRuntimeConfig(server_vad_model_path="/models/silero_vad.onnx")
+        mocker.patch(
+            "vllm_omni.engine.async_omni_engine.StageConfigFactory.get_pipeline_config",
+            return_value=None,
+        )
+        load_deploy_config = mocker.patch(
+            "vllm_omni.engine.async_omni_engine.load_deploy_config",
+            return_value=SimpleNamespace(duplex_session=duplex_session),
+        )
+
+        engine = self._make_engine_no_thread(
+            mocker,
+            resolved_config_path=deploy_path,
+            patch_deploy_config=False,
+        )
+
+        load_deploy_config.assert_called_once_with(deploy_path)
+        assert engine.duplex_session_config is duplex_session
+
     def test_single_stage_mode_without_stage_id_has_no_filter(self, mocker: MockerFixture):
         engine = self._make_engine_no_thread(
             mocker,
@@ -518,6 +556,10 @@ class TestEndpointRestrictionsTrustRemoteCode:
         StageConfigFactory.try_infer_model_type.cache_clear()
 
     def _make_engine_no_thread(self, mocker: MockerFixture, **kwargs: Any) -> AsyncOmniEngine:
+        mocker.patch(
+            "vllm_omni.engine.async_omni_engine.load_deploy_config",
+            return_value=SimpleNamespace(duplex_session=DuplexSessionRuntimeConfig()),
+        )
         mocker.patch.object(
             AsyncOmniEngine,
             "_resolve_stage_configs",

tests/entrypoints/duplex/test_runtime_adapter_boundary.py

@@ -61,20 +61,19 @@ if loaded:
     assert result.returncode == 0, result.stderr
 
 
-def test_generic_handler_requires_explicit_serving_runtime_adapter() -> None:
+def test_generic_handler_supports_adapterless_chat_fallback() -> None:
     repo_root = Path(__file__).resolve().parents[3]
     script = """
 from types import SimpleNamespace
 
 from vllm_omni.entrypoints.duplex.serving import OmniDuplexSessionHandler
 
-chat_service = SimpleNamespace(engine_client=SimpleNamespace())
-try:
-    OmniDuplexSessionHandler(chat_service=chat_service)
-except ValueError as exc:
-    if "serving runtime adapter" not in str(exc).lower():
-        raise
-else:
+chat_service = SimpleNamespace(
+    engine_client=SimpleNamespace(),
+    model_config=SimpleNamespace(model="test-model"),
+)
+handler = OmniDuplexSessionHandler(chat_service=chat_service)
+if handler._serving_runtime_adapter is not None:
     raise SystemExit("generic handler silently selected a model serving adapter")
 """
 

tests/entrypoints/openai_api/test_api_server_guards.py

@@ -568,6 +568,48 @@ def test_websocket_routes_emit_stable_unavailable_frames_and_close(path: str, pa
                 websocket.receive_text()
 
 
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    ("duplex_query", "expected_handler"),
+    [
+        (None, "duplex"),
+        ("1", "duplex"),
+        ("true", "duplex"),
+        ("on", "duplex"),
+        ("0", "legacy"),
+        ("false", "legacy"),
+    ],
+)
+async def test_realtime_route_defaults_to_configured_duplex_handler(
+    monkeypatch, duplex_query: str | None, expected_handler: str
+) -> None:
+    calls: list[str] = []
+
+    class _DuplexHandler:
+        async def handle_realtime_session(self, _websocket) -> None:
+            calls.append("duplex")
+
+    class _LegacyConnection:
+        async def handle_connection(self) -> None:
+            calls.append("legacy")
+
+    monkeypatch.setattr(api_server, "RealtimeConnection", lambda _websocket, _serving: _LegacyConnection())
+    query_params = {} if duplex_query is None else {"duplex": duplex_query}
+    websocket = SimpleNamespace(
+        app=SimpleNamespace(
+            state=SimpleNamespace(
+                openai_serving_duplex=_DuplexHandler(),
+                openai_serving_realtime=object(),
+            )
+        ),
+        query_params=query_params,
+    )
+
+    await api_server.realtime_websocket(websocket)
+
+    assert calls == [expected_handler]
+
+
 def test_health_without_engine_returns_stable_unhealthy_response() -> None:
     """Lock ``/health`` with no engine initialized.
 

tests/entrypoints/openai_api/test_audio_format.py

@@ -163,6 +163,8 @@ class TestStreamingAudioResampler:
 
         chunked = StreamingAudioResampler(24000, 8000)
         pieces = [chunked.process(chunk) for chunk in np.split(waveform, [137, 2048, 9001, 17003])]
+        # Bounded FIR history must own its data, not retain the last large chunk.
+        assert chunked._history.base is None
         pieces.append(chunked.process(np.empty(0), final=True))
         actual = np.concatenate(pieces)
 
@@ -193,9 +195,15 @@ class TestStreamingAudioResampler:
         output_rms = np.sqrt(np.mean(output[100:-100] ** 2))
         assert output_rms < input_rms * 0.01
 
-    def test_rejects_non_integer_downsampling_ratio(self):
-        with pytest.raises(ValueError, match="integer downsampling ratio"):
-            StreamingAudioResampler(24000, 16000)
+    def test_supports_rational_downsampling_ratio(self):
+        waveform = np.sin(np.linspace(0, 200 * np.pi, 24000, endpoint=False)).astype(np.float32)
+        resampler = StreamingAudioResampler(24000, 16000)
+
+        output = resampler.process(waveform, final=True)
+        reference = torchaudio.functional.resample(torch.from_numpy(waveform), 24000, 16000).numpy()
+
+        assert output.shape == (16000,)
+        np.testing.assert_allclose(output[100:-100], reference[100:-100], atol=2e-3, rtol=1e-3)
 
     def test_rejects_multichannel_audio(self):
         resampler = StreamingAudioResampler(24000, 8000)

tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py

@@ -3,10 +3,11 @@
 """
 E2E online tests for Qwen3-Omni /v1/realtime WebSocket (streaming PCM in, audio out).
 
-Three scenarios:
+Four scenarios:
 - Ready CI: async_chunk on, smoke only (no send delay, no accuracy check).
 - Merge CI: async_chunk on + send delay, full accuracy check.
 - Merge CI: async_chunk off, no send delay, full accuracy check.
+- Server VAD: two turns without client commits.
 """
 
 from __future__ import annotations
@@ -28,11 +29,18 @@ from tests.helpers.media import (
     generate_synthetic_audio,
 )
 from tests.helpers.runtime import OmniServerParams
-from tests.helpers.stage_config import get_deploy_config_path
+from tests.helpers.stage_config import get_deploy_config_path, modify_stage_config
+from vllm_omni.config.stage_config import load_deploy_config
+from vllm_omni.entrypoints.duplex.server_vad import (
+    SILERO_VAD_FILENAME,
+    SILERO_VAD_REPO_ID,
+    SILERO_VAD_REVISION,
+)
 
 os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
 
-MODEL = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
+MODEL = os.environ.get("VLLM_OMNI_TEST_QWEN3_OMNI_MODEL", "Qwen/Qwen3-Omni-30B-A3B-Instruct")
+SERVER_VAD_MODEL_PATH = os.environ.get("VLLM_OMNI_TEST_SILERO_VAD_MODEL_PATH")
 
 # Synthetic input for realtime E2E (``generate_synthetic_audio``); distinct cache file per phrase.
 REALTIME_SYNTH_PHRASE_TEXT = (
@@ -48,6 +56,15 @@ SEND_DELAY_MS = 200
 
 # CI overlay bakes in async_chunk: False and covers CUDA/ROCm/XPU via ``platforms:``.
 default_stage_config = get_deploy_config_path("ci/qwen3_omni_moe.yaml")
+server_vad_stage_config = modify_stage_config(
+    default_stage_config,
+    {
+        "async_chunk": True,
+        "duplex_session": (
+            {"server_vad_model_path": SERVER_VAD_MODEL_PATH} if SERVER_VAD_MODEL_PATH is not None else {}
+        ),
+    },
+)
 
 realtime_sync_server_params = [
     pytest.param(
@@ -72,6 +89,17 @@ realtime_async_chunk_server_params = [
     ),
 ]
 
+realtime_server_vad_server_params = [
+    pytest.param(
+        OmniServerParams(
+            model=MODEL,
+            stage_config_path=server_vad_stage_config,
+            use_stage_cli=True,
+        ),
+        id="server_vad",
+    ),
+]
+
 
 def _pcm16_mono_16k_from_wav_bytes(wav_bytes: bytes) -> bytes:
     with wave.open(io.BytesIO(wav_bytes), "rb") as wf:
@@ -183,6 +211,86 @@ async def _run_realtime_audio_roundtrip(
     }
 
 
+async def _append_pcm16_chunks(ws, pcm16: bytes, chunk_bytes: int) -> None:
+    for offset in range(0, len(pcm16), chunk_bytes):
+        await ws.send(
+            json.dumps(
+                {
+                    "type": "input_audio_buffer.append",
+                    "audio": base64.b64encode(pcm16[offset : offset + chunk_bytes]).decode("utf-8"),
+                }
+            )
+        )
+
+
+async def _receive_server_vad_turn(ws) -> list[dict]:
+    events: list[dict] = []
+    while True:
+        message = await asyncio.wait_for(ws.recv(), timeout=600)
+        if isinstance(message, bytes):
+            continue
+        event = json.loads(message)
+        events.append(event)
+        if event.get("type") == "error":
+            raise AssertionError(f"WebSocket error: {event}")
+        if event.get("type") == "response.done":
+            return events
+
+
+async def _run_server_vad_audio_roundtrips(
+    host: str,
+    port: int,
+    model: str,
+    pcm16: bytes,
+    *,
+    chunk_ms: int = 100,
+    turns: int = 2,
+) -> list[list[dict]]:
+    chunk_bytes = max(16_000 * 2 // 1000 * chunk_ms, 2)
+    turn_events: list[list[dict]] = []
+
+    async with websockets.connect(f"ws://{host}:{port}/v1/realtime", max_size=64 * 1024 * 1024) as ws:
+        await ws.send(
+            json.dumps(
+                {
+                    "type": "session.update",
+                    "session": {
+                        "model": model,
+                        "audio": {
+                            "input": {
+                                "format": {"type": "audio/pcm", "rate": 16_000},
+                                "turn_detection": {"type": "server_vad"},
+                            }
+                        },
+                    },
+                }
+            )
+        )
+
+        silence = bytes(16_000 * 2 * 3 // 2)
+        for _ in range(turns):
+            await _append_pcm16_chunks(ws, pcm16, chunk_bytes)
+            await _append_pcm16_chunks(ws, silence, chunk_bytes)
+            turn_events.append(await _receive_server_vad_turn(ws))
+
+    return turn_events
+
+
+@pytest.fixture(scope="class")
+def cached_silero_vad_artifact() -> str:
+    """Prepare the pinned artifact before the serving subprocess starts."""
+    if SERVER_VAD_MODEL_PATH is not None:
+        return SERVER_VAD_MODEL_PATH
+
+    from huggingface_hub import hf_hub_download
+
+    return hf_hub_download(
+        repo_id=SILERO_VAD_REPO_ID,
+        filename=SILERO_VAD_FILENAME,
+        revision=SILERO_VAD_REVISION,
+    )
+
+
 def _synthetic_pcm16_input(
     *,
     phrase_text: str = REALTIME_SYNTH_PHRASE_TEXT,
@@ -296,6 +404,66 @@ class TestQwen3OmniRealtimeWebSocket:
             f"Expected an issue-like audio response longer than 5 seconds, got {output_duration_s:.2f}s"
         )
 
+    @pytest.mark.advanced_model
+    @pytest.mark.omni
+    @hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=2)
+    @pytest.mark.parametrize("omni_server", realtime_server_vad_server_params, indirect=True)
+    def test_server_vad_multi_turn_without_client_commit(
+        self,
+        cached_silero_vad_artifact: str,
+        omni_server,
+    ) -> None:
+        """Two Qwen turns are endpointed without client commits."""
+        assert cached_silero_vad_artifact
+        deploy = load_deploy_config(server_vad_stage_config)
+        assert deploy.session_mode == "turn"
+        assert deploy.async_chunk is True
+        assert deploy.duplex_session.server_vad_model_path == SERVER_VAD_MODEL_PATH
+        pcm16 = _synthetic_pcm16_input()
+
+        turns = asyncio.run(
+            _run_server_vad_audio_roundtrips(
+                omni_server.host,
+                omni_server.port,
+                omni_server.model,
+                pcm16,
+                chunk_ms=100,
+                turns=2,
+            )
+        )
+
+        assert len(turns) == 2
+        required_sequence = [
+            "input_audio_buffer.speech_started",
+            "input_audio_buffer.speech_stopped",
+            "input_audio_buffer.committed",
+            "response.created",
+            "response.audio.delta",
+            "response.audio.done",
+            "response.done",
+        ]
+        input_item_ids: list[str] = []
+        response_ids: list[str] = []
+        for events in turns:
+            event_types = [event["type"] for event in events]
+            positions = [event_types.index(event_type) for event_type in required_sequence]
+            assert positions == sorted(positions)
+
+            started = next(event for event in events if event["type"] == "input_audio_buffer.speech_started")
+            stopped = next(event for event in events if event["type"] == "input_audio_buffer.speech_stopped")
+            committed = next(event for event in events if event["type"] == "input_audio_buffer.committed")
+            created = next(event for event in events if event["type"] == "response.created")["response"]
+            done = next(event for event in events if event["type"] == "response.done")["response"]
+
+            assert started["item_id"] == stopped["item_id"] == committed["item_id"]
+            assert created["id"] == done["id"]
+            assert done["status"] == "completed"
+            input_item_ids.append(committed["item_id"])
+            response_ids.append(created["id"])
+
+        assert len(set(input_item_ids)) == 2
+        assert len(set(response_ids)) == 2
+
     @pytest.mark.advanced_model
     @pytest.mark.omni
     @hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=2)

tests/entrypoints/openai_api/test_server_vad.py

@@ -0,0 +1,429 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import io
+import sys
+import threading
+import wave
+import weakref
+from concurrent.futures import ThreadPoolExecutor
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+
+from vllm_omni.entrypoints.duplex.protocol import (
+    DuplexSession,
+    DuplexSessionConfig,
+)
+from vllm_omni.entrypoints.duplex.server_vad import (
+    ServerVADConfig,
+    ServerVADFrame,
+    ServerVADPipeline,
+    SileroVADBackend,
+    SileroVADBackendProvider,
+    SpeechEndpointDecision,
+    ThresholdEndpointPolicy,
+    parse_session_turn_detection,
+)
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+class SequenceDetector:
+    frame_samples = 160
+
+    def __init__(self, probabilities: list[float]) -> None:
+        self.probabilities = probabilities
+
+    def new_state(self) -> int:
+        return 0
+
+    def infer(self, frame: np.ndarray, state: object) -> tuple[float, object]:
+        index = int(state)
+        probability = self.probabilities[index] if index < len(self.probabilities) else 0.0
+        return probability, index + 1
+
+
+def test_server_vad_config_defaults_and_validation():
+    config = ServerVADConfig.from_value({"type": "server_vad"})
+
+    assert config.type == "server_vad"
+    assert config.threshold == 0.5
+    assert config.prefix_padding_ms == 300
+    assert config.silence_duration_ms == 500
+    assert config.create_response is True
+    assert config.interrupt_response is None
+    assert "interrupt_response" not in config.as_dict()
+    assert config.min_speech_duration_ms is None
+    assert config.as_dict()["type"] == config.type
+
+    interrupting = ServerVADConfig.from_value(
+        {
+            "type": "server_vad",
+            "interrupt_response": True,
+            "min_speech_duration_ms": 96,
+        }
+    )
+    assert interrupting.interrupt_response is True
+    assert interrupting.min_speech_duration_ms == 96
+    for value in (None, 0, "true"):
+        with pytest.raises(ValueError, match="interrupt_response must be a boolean"):
+            ServerVADConfig.from_value({"type": "server_vad", "interrupt_response": value})
+    for enabled in (False, True):
+        _, aliased = parse_session_turn_detection(
+            {
+                "turn_detection": {"type": "server_vad"},
+                "audio": {"input": {"turn_detection": {"type": "server_vad", "interrupt_response": enabled}}},
+            }
+        )
+        assert aliased is not None and aliased.interrupt_response is enabled
+    with pytest.raises(ValueError, match="Unknown"):
+        ServerVADConfig.from_value({"type": "server_vad", "semantic_eagerness": "high"})
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("prefix_padding_ms", [0, 10, 30])
+async def test_server_vad_pipeline_handles_arbitrary_chunk_boundaries(prefix_padding_ms: int):
+    detector = SequenceDetector([0.0, 0.9, 0.8, 0.0, 0.0])
+    pipeline = ServerVADPipeline(
+        detector,
+        ServerVADConfig(prefix_padding_ms=prefix_padding_ms, silence_duration_ms=20),
+    )
+    samples = np.zeros(detector.frame_samples * 5, dtype=np.float32)
+
+    batches = [
+        await pipeline.push(samples[:73]),
+        await pipeline.push(samples[73:431]),
+        await pipeline.push(samples[431:]),
+    ]
+    frames = [frame for batch in batches for frame in batch.frames]
+
+    assert len(frames) == 5
+    assert [index for index, frame in enumerate(frames) if frame.decision.speech_started] == [1]
+    assert [index for index, frame in enumerate(frames) if frame.decision.speech_stopped] == [4]
+    assert frames[1].decision.audio_start_ms == max(0, 10 - prefix_padding_ms)
+    assert frames[4].decision.audio_end_ms == 50
+
+    pipeline.reset()
+    reset_batch = await pipeline.push(samples)
+    assert [index for index, frame in enumerate(reset_batch.frames) if frame.decision.speech_started] == [1]
+    assert [index for index, frame in enumerate(reset_batch.frames) if frame.decision.speech_stopped] == [4]
+    # Keep session-relative timestamps, but never reference prefix audio
+    # discarded by reset at 50 ms.
+    assert reset_batch.frames[1].decision.audio_start_ms == max(50, 60 - prefix_padding_ms)
+    assert reset_batch.frames[4].decision.audio_end_ms == 100
+
+
+@pytest.mark.asyncio
+async def test_server_vad_retained_audio_does_not_keep_input_chunk():
+    pipeline = ServerVADPipeline(SequenceDetector([]), ServerVADConfig())
+    session = DuplexSession("server-vad-memory", DuplexSessionConfig())
+    samples = np.zeros(160 * 100 + 1, dtype=np.float32)
+    source_ref = weakref.ref(samples)
+    batch = await pipeline.push(samples)
+    session.append_server_vad_frame(
+        batch.frames[-1].samples,
+        speech_started=False,
+        speech_stopped=False,
+        prefix_samples=80,
+    )
+    session.append_server_vad_frame(
+        batch.frames[-1].samples,
+        speech_started=True,
+        speech_stopped=False,
+        prefix_samples=80,
+    )
+    del batch, samples
+    # Allow the completed to_thread callback to release its result, too.
+    await asyncio.sleep(0)
+
+    assert source_ref() is None
+    assert pipeline.scratch_bytes == 4
+    assert session.server_vad_utterance_bytes == (80 + 160) * 4
+
+
+@pytest.mark.asyncio
+async def test_server_vad_pipeline_pcm16_resampling_is_split_invariant():
+    sample_index = np.arange(2_400, dtype=np.int32)
+    source = ((sample_index * 7919) % 65_536 - 32_768).astype("<i2")
+    # Supply ample silence after the signal so the streaming resampler can
+    # emit a complete set of VAD frames without depending on its FIR width.
+    right_context = np.zeros(240, dtype="<i2")
+    continuous_source = np.concatenate((source, right_context))
+
+    async def process(chunk_sizes: list[int]) -> tuple[np.ndarray, list[SpeechEndpointDecision]]:
+        pipeline = ServerVADPipeline(
+            SequenceDetector([0.0] * 10),
+            ServerVADConfig(),
+        )
+        frames: list[ServerVADFrame] = []
+        offset = 0
+        for chunk_size in chunk_sizes:
+            chunk = continuous_source[offset : offset + chunk_size]
+            batch = await pipeline.push_pcm16(chunk.tobytes(), source_sample_rate_hz=24_000)
+            frames.extend(batch.frames)
+            offset += chunk_size
+        assert offset == continuous_source.size
+        return np.concatenate([frame.samples for frame in frames]), [frame.decision for frame in frames]
+
+    whole_samples, whole_decisions = await process([continuous_source.size])
+    split_samples, split_decisions = await process([7, 13, 511, 512, 513, continuous_source.size - 1_556])
+
+    np.testing.assert_array_equal(split_samples, whole_samples)
+    assert whole_samples.size == 1_600
+    assert split_decisions == whole_decisions
+
+
+@pytest.mark.asyncio
+async def test_server_vad_pipeline_reset_clears_resampler_and_source_rate_lock():
+    pipeline = ServerVADPipeline(
+        SequenceDetector([0.0]),
+        ServerVADConfig(),
+    )
+
+    batch = await pipeline.push_pcm16(
+        np.asarray([30_000, -30_000], dtype="<i2").tobytes(),
+        source_sample_rate_hz=24_000,
+    )
+    assert not batch.frames
+
+    pipeline.reset()
+
+    source = np.zeros(160, dtype="<i2")
+    source[:3] = [-32768, 0, 32767]
+    batch = await pipeline.push_pcm16(source.tobytes(), source_sample_rate_hz=16_000)
+
+    assert len(batch.frames) == 1
+    np.testing.assert_array_equal(
+        batch.frames[0].samples[:3],
+        np.asarray([-1.0, 0.0, 32767 / 32768], dtype=np.float32),
+    )
+
+
+def test_threshold_endpoint_policy_uses_silero_v62_exit_hysteresis():
+    policy = ThresholdEndpointPolicy(
+        ServerVADConfig(threshold=0.5, silence_duration_ms=20),
+        sample_rate_hz=16_000,
+    )
+    frame_samples = 160
+
+    started = policy.update(0.8, frame_start_sample=0, frame_samples=frame_samples)
+    assert started.speech_started is True
+
+    # Values below the activation threshold but above threshold - 0.15 must
+    # not begin a silence candidate or close an active speech segment.
+    for frame_index in range(1, 5):
+        decision = policy.update(
+            0.4,
+            frame_start_sample=frame_index * frame_samples,
+            frame_samples=frame_samples,
+        )
+        assert decision.speech_stopped is False
+
+    first_silence = policy.update(
+        0.2,
+        frame_start_sample=5 * frame_samples,
+        frame_samples=frame_samples,
+    )
+    stopped = policy.update(
+        0.2,
+        frame_start_sample=6 * frame_samples,
+        frame_samples=frame_samples,
+    )
+
+    assert first_silence.speech_stopped is False
+    assert stopped.speech_stopped is True
+
+
+def test_threshold_endpoint_policy_honors_minimum_speech_duration():
+    policy = ThresholdEndpointPolicy(
+        ServerVADConfig(prefix_padding_ms=0, min_speech_duration_ms=25),
+        sample_rate_hz=16_000,
+    )
+    frame_samples = 160
+
+    assert not policy.update(0.9, frame_start_sample=0, frame_samples=frame_samples).speech_started
+    assert not policy.update(0.9, frame_start_sample=160, frame_samples=frame_samples).speech_started
+    started = policy.update(0.9, frame_start_sample=320, frame_samples=frame_samples)
+
+    assert started.speech_started is True
+    assert started.audio_start_ms == 0
+
+
+def test_threshold_endpoint_policy_clamps_silero_exit_threshold():
+    policy = ThresholdEndpointPolicy(
+        ServerVADConfig(threshold=0.1, silence_duration_ms=20),
+        sample_rate_hz=16_000,
+    )
+    frame_samples = 160
+
+    started = policy.update(0.9, frame_start_sample=0, frame_samples=frame_samples)
+    assert started.speech_started is True
+
+    first_silence = policy.update(
+        0.0,
+        frame_start_sample=frame_samples,
+        frame_samples=frame_samples,
+    )
+    stopped = policy.update(
+        0.0,
+        frame_start_sample=2 * frame_samples,
+        frame_samples=frame_samples,
+    )
+
+    assert first_silence.speech_stopped is False
+    assert stopped.speech_stopped is True
+    # The committed audio ends at 30 ms: 10 ms of speech followed by the
+    # 20 ms of silence required to detect the endpoint.
+    assert stopped.audio_end_ms == 30
+    assert stopped.endpoint_delay_ms == 20
+
+
+def test_silero_backend_matches_upstream_v62_streaming_contract(monkeypatch, tmp_path):
+    class FakeSessionOptions:
+        inter_op_num_threads = 0
+        intra_op_num_threads = 0
+
+    class FakeInferenceSession:
+        def __init__(self, path: str, *, providers: list[str], sess_options: object) -> None:
+            self.providers = providers
+            self.sess_options = sess_options
+            self.calls: list[dict[str, np.ndarray]] = []
+            self.run_barrier: threading.Barrier | None = None
+            sessions.append(self)
+
+        def get_inputs(self) -> list[SimpleNamespace]:
+            return [SimpleNamespace(name=name) for name in ("input", "state", "sr")]
+
+        def run(self, _outputs: object, inputs: dict[str, np.ndarray]) -> list[np.ndarray]:
+            self.calls.append({name: np.array(value, copy=True) for name, value in inputs.items()})
+            if self.run_barrier is not None:
+                self.run_barrier.wait(timeout=5)
+            return [
+                np.asarray([[0.75]], dtype=np.float32),
+                np.full((2, 1, 128), len(self.calls), dtype=np.float32),
+            ]
+
+    sessions: list[FakeInferenceSession] = []
+    monkeypatch.setitem(
+        sys.modules,
+        "onnxruntime",
+        SimpleNamespace(
+            SessionOptions=FakeSessionOptions,
+            InferenceSession=FakeInferenceSession,
+        ),
+    )
+
+    model_path = tmp_path / "silero_vad.onnx"
+    model_path.write_bytes(b"fake-model")
+    backend = SileroVADBackend(model_path)
+    session = sessions[0]
+
+    assert session.providers == ["CPUExecutionProvider"]
+    assert session.sess_options.inter_op_num_threads == 1
+    assert session.sess_options.intra_op_num_threads == 1
+    assert session.calls[0]["input"].shape == (1, 576)
+    assert session.calls[0]["state"].shape == (2, 1, 128)
+    np.testing.assert_array_equal(session.calls[0]["state"], np.zeros((2, 1, 128), dtype=np.float32))
+    assert session.calls[0]["sr"].shape == ()
+    assert session.calls[0]["sr"].item() == 16_000
+
+    state = backend.new_state()
+    first_frame = np.arange(backend.frame_samples, dtype=np.float32)
+    probability, state = backend.infer(first_frame, state)
+    first_call = session.calls[1]
+
+    assert probability == pytest.approx(0.75)
+    np.testing.assert_array_equal(
+        first_call["input"][:, : backend.context_samples],
+        np.zeros((1, backend.context_samples), dtype=np.float32),
+    )
+    np.testing.assert_array_equal(
+        first_call["input"][:, backend.context_samples :],
+        first_frame[None, :],
+    )
+    np.testing.assert_array_equal(first_call["state"], np.zeros((2, 1, 128), dtype=np.float32))
+
+    second_frame = -first_frame
+    backend.infer(second_frame, state)
+    second_call = session.calls[2]
+    np.testing.assert_array_equal(
+        second_call["input"][:, : backend.context_samples],
+        first_frame[None, -backend.context_samples :],
+    )
+    np.testing.assert_array_equal(
+        second_call["input"][:, backend.context_samples :],
+        second_frame[None, :],
+    )
+    np.testing.assert_array_equal(
+        second_call["state"],
+        np.full((2, 1, 128), 2, dtype=np.float32),
+    )
+
+    # Independent pipeline states may enter the shared ORT session concurrently.
+    session.run_barrier = threading.Barrier(2)
+    with ThreadPoolExecutor(max_workers=2) as executor:
+        futures = [
+            executor.submit(backend.infer, first_frame, backend.new_state()),
+            executor.submit(backend.infer, second_frame, backend.new_state()),
+        ]
+        assert [future.result()[0] for future in futures] == pytest.approx([0.75, 0.75])
+
+
+def test_duplex_session_owns_server_vad_prefix_and_utterance_audio():
+    session = DuplexSession("server-vad-session", DuplexSessionConfig())
+    frame = np.zeros(160, dtype=np.float32)
+
+    session.reserve_input_bytes(frame.nbytes * 3, limit=frame.nbytes * 4)
+    released = session.append_server_vad_frame(
+        frame,
+        speech_started=False,
+        speech_stopped=False,
+        prefix_samples=160,
+    )
+    assert released == 0
+    released = session.append_server_vad_frame(
+        frame,
+        speech_started=False,
+        speech_stopped=False,
+        prefix_samples=160,
+    )
+    assert released == frame.nbytes
+    session.release_input_bytes(released)
+    session.append_server_vad_frame(
+        frame,
+        speech_started=True,
+        speech_stopped=False,
+        prefix_samples=160,
+    )
+
+    assert session.server_vad_utterance_bytes == frame.nbytes * 2
+    assert session.stage_server_vad_audio_for_commit() is True
+    committed = session.commit_user_input()
+    assert committed is not None
+    assert committed.message["role"] == "user"
+    audio_url = committed.message["content"][0]["audio_url"]["url"]
+    assert audio_url.startswith("data:audio/wav;base64,")
+    wav_bytes = base64.b64decode(audio_url.partition(",")[2])
+    with wave.open(io.BytesIO(wav_bytes), "rb") as wav_file:
+        assert wav_file.getnchannels() == 1
+        assert wav_file.getsampwidth() == 2
+        assert wav_file.getframerate() == 16_000
+        assert wav_file.getnframes() == frame.size * 2
+
+
+def test_silero_provider_rejects_missing_or_invalid_local_artifact(tmp_path):
+    missing = SileroVADBackendProvider(model_path=str(tmp_path / "missing.onnx"))
+    with pytest.raises(RuntimeError, match="does not exist"):
+        missing.get()
+
+    invalid_path = tmp_path / "silero_vad.onnx"
+    invalid_path.write_bytes(b"not-the-pinned-model")
+    invalid = SileroVADBackendProvider(model_path=str(invalid_path))
+    with pytest.raises(RuntimeError, match="checksum mismatch"):
+        invalid.get()

tests/entrypoints/test_duplex_fence_propagation.py

@@ -74,7 +74,7 @@ async def test_openai_handler_passes_current_session_fence_for_runtime_controls(
     handler = OmniDuplexSessionHandler(chat_service=FakeChatService(engine))
     session = DuplexSession(
         session_id="sid",
-        config=DuplexSessionConfig(),
+        config=DuplexSessionConfig(extra_body={"native_duplex": True}),
         epoch=1,
         turn_id=2,
     )
@@ -92,7 +92,7 @@ async def test_openai_handler_passes_current_session_fence_for_runtime_controls(
 async def test_runtime_signal_uses_fence_captured_before_session_turn_advances():
     engine = FenceRecordingEngine()
     handler = OmniDuplexSessionHandler(chat_service=FakeChatService(engine))
-    session = DuplexSession(session_id="sid", config=DuplexSessionConfig())
+    session = DuplexSession(session_id="sid", config=DuplexSessionConfig(extra_body={"native_duplex": True}))
     captured_fence = DuplexFence("sid", epoch=0, turn_id=0)
     session.turn_id = 1
 
@@ -111,7 +111,7 @@ async def test_openai_handler_forwards_cancel_fence_through_async_omni_facade():
     app = object.__new__(AsyncOmni)
     app.engine = engine
     handler = OmniDuplexSessionHandler(chat_service=FakeChatService(app))
-    session = DuplexSession(session_id="sid-cancel", config=DuplexSessionConfig())
+    session = DuplexSession(session_id="sid-cancel", config=DuplexSessionConfig(extra_body={"native_duplex": True}))
     cancelled_fence = DuplexFence("sid-cancel", epoch=2, turn_id=3)
     next_fence = DuplexFence("sid-cancel", epoch=3, turn_id=3)
 

tests/entrypoints/test_utils.py

@@ -28,6 +28,7 @@ from vllm_omni.entrypoints.utils import (
     coerce_param_message_types,
     filter_dataclass_kwargs,
 )
+from vllm_omni.model_executor.models.qwen3_omni.pipeline import QWEN3_OMNI_PIPELINE
 
 pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
 
@@ -295,6 +296,31 @@ class TestFilterDataclassKwargs:
 
 
 class TestResolveOmniConfig:
+    def test_bare_deploy_name_returns_packaged_resolved_path(self, tmp_path, mocker: MockerFixture):
+        deploy_name = "qwen3_omni_moe.yaml"
+        deploy_path = tmp_path / deploy_name
+        deploy_path.write_text(
+            "duplex_session:\n  server_vad_model_path: /models/silero_vad.onnx\nstages: []\n",
+            encoding="utf-8",
+        )
+        mocker.patch("vllm_omni.config.config_factory._DEPLOY_DIR", tmp_path)
+        mocker.patch("vllm_omni.config.omni_config._DEPLOY_DIR", tmp_path)
+        mocker.patch(
+            "vllm_omni.config.config_factory.StageConfigFactory.get_pipeline_config",
+            return_value=QWEN3_OMNI_PIPELINE,
+        )
+
+        resolved = resolve_omni_config(
+            "dummy-model",
+            trust_remote_code=False,
+            deploy_config_path=deploy_name,
+            cli_overrides={},
+            stage_overrides=None,
+            strategy_config_path=None,
+        )
+
+        assert resolved.config_path == str(deploy_path)
+
     def test_stage_lookup_error_lists_resolved_ids(self):
         resolved = OmniConfigResolution(
             config_path=None,

vllm_omni/config/stage_config.py

@@ -509,6 +509,7 @@ class DuplexSessionRuntimeConfig:
     max_pending_turns_per_session: int = 4
     max_sessions: int = 1
     completed_append_cache_size: int = 256
+    server_vad_model_path: str | None = None
     # Startup warmup: run this many silent 80 ms-style frames through a
     # throwaway realtime session before real clients are admitted, so
     # one-time costs (kernel JIT, first prefill/decode paths, codec caches)
@@ -528,6 +529,10 @@ class DuplexSessionRuntimeConfig:
         }
         if self.idle_ttl_s is not None and self.idle_ttl_s <= 0:
             raise ValueError("duplex_session.idle_ttl_s must be positive or null")
+        if self.server_vad_model_path is not None and (
+            not isinstance(self.server_vad_model_path, str) or not self.server_vad_model_path.strip()
+        ):
+            raise ValueError("duplex_session.server_vad_model_path must be a non-empty string or null")
         for name, value in positive.items():
             if value <= 0:
                 raise ValueError(f"duplex_session.{name} must be positive")

vllm_omni/engine/async_omni_engine.py

@@ -224,7 +224,7 @@ class AsyncOmniEngine:
                 trust_remote_code=bool(trust_remote_code),
                 deploy_config_path=deploy_config_path,
             )
-            self._set_pipeline_runtime_config(pipeline_config, deploy_config_path)
+            self._set_pipeline_runtime_config(pipeline_config, self.config_path)
         else:
             self._set_pipeline_runtime_config(
                 self._config_resolution.pipeline_config,

vllm_omni/entrypoints/duplex/audio.py

@@ -21,8 +21,12 @@ MIN_INPUT_SAMPLE_RATE_HZ = 8_000
 MAX_INPUT_SAMPLE_RATE_HZ = 192_000
 
 
-def validate_input_sample_rate_hz(sample_rate_hz: int | float) -> int:
-    if isinstance(sample_rate_hz, bool) or (isinstance(sample_rate_hz, float) and not math.isfinite(sample_rate_hz)):
+def validate_input_sample_rate_hz(sample_rate_hz: object) -> int:
+    if (
+        isinstance(sample_rate_hz, bool)
+        or not isinstance(sample_rate_hz, int | float)
+        or (isinstance(sample_rate_hz, float) and not math.isfinite(sample_rate_hz))
+    ):
         raise ValueError("sample_rate_hz must be a finite integer")
     rate = int(sample_rate_hz)
     if rate != sample_rate_hz:
@@ -35,14 +39,27 @@ def validate_input_sample_rate_hz(sample_rate_hz: int | float) -> int:
 def resample_pcm16_mono(raw: bytes, *, source_rate_hz: int, target_rate_hz: int) -> bytes:
     if source_rate_hz <= 0 or target_rate_hz <= 0 or source_rate_hz == target_rate_hz:
         return raw
-    samples = np.frombuffer(raw, dtype="<i2").astype(np.float32)
-    if samples.size <= 1:
-        return raw
-    target_size = max(1, int(round(samples.size * target_rate_hz / source_rate_hz)))
-    source_x = np.linspace(0.0, 1.0, num=samples.size, endpoint=True)
-    target_x = np.linspace(0.0, 1.0, num=target_size, endpoint=True)
-    resampled = np.interp(target_x, source_x, samples)
-    return np.clip(resampled, -32768, 32767).astype("<i2").tobytes()
+    # Import lazily so the generic Duplex audio module does not pull in the
+    # OpenAI API server while the Duplex stack itself is being imported.
+    from vllm_omni.entrypoints.openai.audio_utils_mixin import StreamingAudioResampler
+
+    samples = np.frombuffer(raw, dtype="<i2")
+    rate_gcd = math.gcd(source_rate_hz, target_rate_hz)
+    reduced_factor = max(source_rate_hz // rate_gcd, target_rate_hz // rate_gcd)
+    if reduced_factor > StreamingAudioResampler._max_polyphase_factor:
+        # The FIR size scales with the reduced ratio. Keep unusual client
+        # rates bounded while common audio rates use the high-quality path.
+        if samples.size <= 1:
+            return raw
+        target_size = max(1, int(round(samples.size * target_rate_hz / source_rate_hz)))
+        source_x = np.linspace(0.0, 1.0, num=samples.size, endpoint=True)
+        target_x = np.linspace(0.0, 1.0, num=target_size, endpoint=True)
+        resampled = np.interp(target_x, source_x, samples)
+        return np.clip(resampled, -32_768, 32_767).astype("<i2").tobytes()
+    resampler = StreamingAudioResampler(source_rate_hz, target_rate_hz)
+    normalized = samples.astype(np.float32) * np.float32(1.0 / 32768.0)
+    resampled = resampler.process(normalized, final=True)
+    return np.clip(np.rint(resampled * 32768.0), -32768, 32767).astype("<i2").tobytes()
 
 
 def decode_g711_ulaw(raw: bytes) -> bytes:
@@ -116,6 +133,24 @@ def wav_payload_to_pcm16(raw: bytes) -> tuple[bytes | None, int | None]:
         return None, None
 
 
+def encode_float32_mono_wav_base64(
+    samples: np.ndarray,
+    *,
+    sample_rate_hz: int,
+) -> str:
+    """Package normalized mono float32 samples as a PCM16 WAV data payload."""
+    rate = validate_input_sample_rate_hz(sample_rate_hz)
+    normalized = np.ascontiguousarray(samples, dtype=np.float32).reshape(-1)
+    pcm16 = np.clip(np.rint(normalized * 32768.0), -32768, 32767).astype("<i2")
+    with io.BytesIO() as buffer:
+        with wave.open(buffer, "wb") as wav_file:
+            wav_file.setnchannels(1)
+            wav_file.setsampwidth(2)
+            wav_file.setframerate(rate)
+            wav_file.writeframes(pcm16.tobytes())
+        return base64.b64encode(buffer.getvalue()).decode("ascii")
+
+
 def convert_input_audio_with_rate(
     audio: object,
     fmt: object,

vllm_omni/entrypoints/duplex/capability.py

@@ -11,7 +11,7 @@ def should_enable_duplex_endpoint(
     *,
     config_path: str | None = None,
 ) -> bool:
-    """Enable duplex routes only for deployments that explicitly opt in."""
+    """Enable the realtime session handler for explicitly configured deployments."""
     if stage_configs:
         for stage in stage_configs:
             session_mode = (
@@ -21,14 +21,13 @@ def should_enable_duplex_endpoint(
                 return True
     if config_path:
         try:
-            from omegaconf import OmegaConf
+            from vllm_omni.config.stage_config import resolve_deploy_yaml
 
-            raw_config = OmegaConf.load(config_path)
-            session_mode = raw_config.get("session_mode") if hasattr(raw_config, "get") else None
-            if session_mode == "duplex":
+            raw_config = resolve_deploy_yaml(config_path)
+            if raw_config.get("session_mode") == "duplex" or isinstance(raw_config.get("duplex_session"), dict):
                 return True
         except Exception as exc:
-            logger.warning("Failed to inspect duplex session_mode from %s: %s", config_path, exc)
+            logger.warning("Failed to inspect realtime session configuration from %s: %s", config_path, exc)
     return False
 
 

vllm_omni/entrypoints/duplex/chat_fallback.py

@@ -38,7 +38,14 @@ class ChatFallbackProjectorMixin:
             request = self._build_chat_request(session, request_id)
             result = await self._chat_service.create_chat_completion(request, raw_request=None)
             if isinstance(result, ErrorResponse):
-                await send_json({"type": "error", "error": result.message, "code": result.type or "chat_error"})
+                error = result.error
+                await send_json(
+                    {
+                        "type": "error",
+                        "error": error.message if error else "Chat request failed",
+                        "code": error.type if error else "chat_error",
+                    }
+                )
                 session.end_response(commit_text=False)
                 return
             if hasattr(result, "__aiter__"):

vllm_omni/entrypoints/duplex/protocol.py

@@ -10,6 +10,16 @@ from enum import Enum
 from types import MappingProxyType
 from uuid import uuid4
 
+import numpy as np
+
+from vllm_omni.entrypoints.duplex.audio import (
+    encode_float32_mono_wav_base64,
+)
+from vllm_omni.entrypoints.duplex.server_vad import (
+    ServerVADConfig,
+    parse_session_turn_detection,
+)
+
 
 class DuplexOverlapPolicy(str, Enum):
     LISTEN_ONLY = "listen_only"
@@ -231,10 +241,12 @@ class DuplexSessionConfig:
     overlap_barge_in_ms: int = 1200
     overlap_silence_rms: float = 0.003
     playback_commit_policy: str = DuplexPlaybackCommitPolicy.COMMIT_ALL_ON_DONE.value
+    turn_detection_configured: bool = False
+    server_vad: ServerVADConfig | None = None
     extra_body: dict[str, object] = field(default_factory=dict)
 
     def as_dict(self) -> dict[str, object]:
-        return {
+        payload: dict[str, object] = {
             "model": self.model,
             "modalities": list(self.modalities),
             "instructions": self.instructions,
@@ -253,6 +265,9 @@ class DuplexSessionConfig:
             "playback_commit_policy": self.playback_commit_policy,
             "extra_body": dict(self.extra_body),
         }
+        if self.turn_detection_configured:
+            payload["turn_detection"] = self.server_vad.as_dict() if self.server_vad is not None else None
+        return payload
 
     @classmethod
     def from_event(cls, event: dict[str, object]) -> DuplexSessionConfig:
@@ -295,9 +310,13 @@ class DuplexSessionConfig:
             config.playback_commit_policy = cls._normalize_playback_commit_policy(source["playback_commit_policy"])
         if isinstance(source.get("modalities"), list) and all(isinstance(x, str) for x in source["modalities"]):
             config.modalities = list(source["modalities"])
+        turn_detection_configured, server_vad = parse_session_turn_detection(source)
         if isinstance(source.get("extra_body"), dict):
             config.extra_body = normalize_native_duplex_key(dict(source["extra_body"]))
             extra = config.extra_body
+            raw_realtime_session = extra.get("realtime_session_payload")
+            if not turn_detection_configured and isinstance(raw_realtime_session, dict):
+                turn_detection_configured, server_vad = parse_session_turn_detection(raw_realtime_session)
             if isinstance(extra.get("overlap_policy"), str):
                 config.overlap_policy = cls._normalize_overlap_policy(extra["overlap_policy"])
             if isinstance(extra.get("overlap_short_ack_ms"), int | float):
@@ -308,6 +327,9 @@ class DuplexSessionConfig:
                 config.overlap_silence_rms = max(0.0, float(extra["overlap_silence_rms"]))
             if isinstance(extra.get("playback_commit_policy"), str):
                 config.playback_commit_policy = cls._normalize_playback_commit_policy(extra["playback_commit_policy"])
+        if turn_detection_configured:
+            config.turn_detection_configured = True
+            config.server_vad = server_vad
         return config
 
     @staticmethod
@@ -354,6 +376,13 @@ class DuplexCommittedInput:
     input_commit_seq: int
 
 
+@dataclass
+class PendingServerVADTurn:
+    committed: DuplexCommittedInput
+    create_response: bool
+    item_id: str
+
+
 @dataclass
 class DuplexAssistantAudioTextMark:
     text_chars: int
@@ -368,6 +397,13 @@ class InputBufferState:
     overlap_speech_ms: int = 0
     reserved_input_bytes: int = 0
     pending_turns: int = 0
+    turn_detection_config_locked: bool = False
+    vad_prefix_audio: list[np.ndarray] = field(default_factory=list)
+    vad_prefix_samples: int = 0
+    vad_utterance_audio: list[np.ndarray] = field(default_factory=list)
+    vad_speech_active: bool = False
+    active_server_vad_item_id: str | None = None
+    pending_server_vad_turn: PendingServerVADTurn | None = None
 
 
 @dataclass
@@ -627,6 +663,9 @@ class DuplexSession:
         return active_turn_id is None or int(turn_id) == active_turn_id
 
     def append_history_message(self, message: dict[str, object]) -> None:
+        response_id = self.active_response_id
+        if response_id is not None:
+            self.reserve_history_item(f"item_{response_id}")
         self._conversation.messages.append(message)
 
     def stage_pending_history_item(self, item_id: str, message: dict[str, object]) -> None:
@@ -653,6 +692,13 @@ class DuplexSession:
         self._input.reserved_input_bytes += size
         return True
 
+    @property
+    def turn_detection_config_locked(self) -> bool:
+        return self._input.turn_detection_config_locked
+
+    def lock_turn_detection_config(self) -> None:
+        self._input.turn_detection_config_locked = True
+
     def release_input_bytes(self, size: int) -> None:
         self._input.reserved_input_bytes = max(0, self._input.reserved_input_bytes - max(0, int(size)))
 
@@ -677,6 +723,92 @@ class DuplexSession:
         self._input.pending_audio.append(DuplexAudioChunk(data=data, format=fmt, sample_rate_hz=sample_rate_hz))
         self.turn_state = DuplexTurnState.USER_SPEAKING
 
+    def append_server_vad_frame(
+        self,
+        samples: np.ndarray,
+        *,
+        speech_started: bool,
+        speech_stopped: bool,
+        prefix_samples: int,
+    ) -> int:
+        """Store one detector frame while keeping commit-eligible audio session-owned.
+
+        Returns the number of raw float32 bytes released when the rolling
+        pre-speech prefix is trimmed.
+        """
+        # Session-owned audio must not retain the batch backing a detector frame.
+        frame = np.array(samples, dtype=np.float32, copy=True).reshape(-1)
+        released_samples = 0
+        if not self._input.vad_speech_active:
+            if speech_started:
+                self._input.vad_utterance_audio = [*self._input.vad_prefix_audio, frame]
+                self._input.vad_prefix_audio = []
+                self._input.vad_prefix_samples = 0
+                self._input.vad_speech_active = True
+            else:
+                self._input.vad_prefix_audio.append(frame)
+                self._input.vad_prefix_samples += frame.size
+                while self._input.vad_prefix_samples > prefix_samples and self._input.vad_prefix_audio:
+                    overflow = self._input.vad_prefix_samples - prefix_samples
+                    first = self._input.vad_prefix_audio[0]
+                    if first.size <= overflow:
+                        self._input.vad_prefix_audio.pop(0)
+                        self._input.vad_prefix_samples -= first.size
+                        released_samples += first.size
+                    else:
+                        self._input.vad_prefix_audio[0] = first[overflow:].copy()
+                        self._input.vad_prefix_samples -= overflow
+                        released_samples += overflow
+        else:
+            self._input.vad_utterance_audio.append(frame)
+
+        if speech_stopped:
+            self._input.vad_speech_active = False
+        if self._input.vad_speech_active or speech_started:
+            self.turn_state = DuplexTurnState.USER_SPEAKING
+        return released_samples * np.dtype(np.float32).itemsize
+
+    def stage_server_vad_audio_for_commit(self) -> bool:
+        if not self._input.vad_utterance_audio:
+            return False
+        samples = np.concatenate(self._input.vad_utterance_audio)
+        encoded = encode_float32_mono_wav_base64(samples, sample_rate_hz=16_000)
+        self._input.pending_audio.append(DuplexAudioChunk(data=encoded, format="wav", sample_rate_hz=16_000))
+        self._input.vad_utterance_audio.clear()
+        self._input.vad_speech_active = False
+        return True
+
+    def clear_server_vad_audio(self) -> None:
+        self._input.vad_prefix_audio.clear()
+        self._input.vad_prefix_samples = 0
+        self._input.vad_utterance_audio.clear()
+        self._input.vad_speech_active = False
+        self._input.active_server_vad_item_id = None
+
+    @property
+    def active_server_vad_item_id(self) -> str | None:
+        return self._input.active_server_vad_item_id
+
+    @property
+    def server_vad_utterance_bytes(self) -> int:
+        return sum(frame.nbytes for frame in self._input.vad_utterance_audio)
+
+    def discard_uncommitted_server_vad_utterance(self) -> int:
+        released = self.server_vad_utterance_bytes
+        self._input.vad_utterance_audio.clear()
+        self._input.vad_speech_active = False
+        self._input.active_server_vad_item_id = None
+        self._input.pending_text.clear()
+        return released
+
+    def begin_server_vad_speech(self, item_id: str) -> None:
+        self._input.active_server_vad_item_id = item_id
+
+    def finish_server_vad_speech(self) -> str | None:
+        item_id = self._input.active_server_vad_item_id
+        self._input.active_server_vad_item_id = None
+        return item_id
+
     def mark_user_input_activity(self) -> None:
         self.turn_state = DuplexTurnState.USER_SPEAKING
 
@@ -692,10 +824,16 @@ class DuplexSession:
         self._input.pending_audio.clear()
         self._input.reserved_input_bytes = 0
         self._input.pending_turns = 0
+        self.clear_server_vad_audio()
         self.turn_state = DuplexTurnState.IDLE
         return cancelled
 
-    def commit_user_input(self) -> DuplexCommittedInput | None:
+    def commit_user_input(
+        self,
+        *,
+        append_history: bool = True,
+        retain_input_bytes: int = 0,
+    ) -> DuplexCommittedInput | None:
         text = "".join(self._input.pending_text).strip()
         audio_chunks = list(self._input.pending_audio)
         if not text and not audio_chunks:
@@ -722,10 +860,11 @@ class DuplexSession:
         self._bind_active_response_to_input_commit(self._input.commit_seq + 1)
         self._input.commit_seq += 1
         message = {"role": "user", "content": content}
-        self._conversation.messages.append(message)
+        if append_history:
+            self._conversation.messages.append(message)
         self._input.pending_text.clear()
         self._input.pending_audio.clear()
-        self._input.reserved_input_bytes = 0
+        self._input.reserved_input_bytes = max(0, int(retain_input_bytes))
         self.turn_state = DuplexTurnState.USER_COMMITTED
         return DuplexCommittedInput(
             message=message,
@@ -734,6 +873,40 @@ class DuplexSession:
             input_commit_seq=self._input.commit_seq,
         )
 
+    @property
+    def pending_server_vad_turn(self) -> PendingServerVADTurn | None:
+        return self._input.pending_server_vad_turn
+
+    def stage_server_vad_turn(
+        self,
+        committed: DuplexCommittedInput,
+        *,
+        create_response: bool,
+        item_id: str,
+    ) -> None:
+        if self._input.pending_server_vad_turn is not None:
+            raise RuntimeError("A Server VAD turn is already pending")
+        response_id = self.active_response_id
+        if response_id is not None:
+            # A pending user turn advances input_commit_seq before the active
+            # response completes. Reserve its current history position first.
+            self.reserve_history_item(f"item_{response_id}")
+        self._input.pending_server_vad_turn = PendingServerVADTurn(
+            committed=committed,
+            create_response=create_response,
+            item_id=item_id,
+        )
+
+    def activate_server_vad_turn(self) -> PendingServerVADTurn | None:
+        pending = self._input.pending_server_vad_turn
+        if pending is None:
+            return None
+        self._input.pending_server_vad_turn = None
+        self._conversation.messages.append(pending.committed.message)
+        self.register_history_item(pending.item_id, pending.committed.message)
+        self.turn_state = DuplexTurnState.USER_COMMITTED
+        return pending
+
     def commit_native_audio_input(
         self,
         *,
@@ -1020,7 +1193,14 @@ class DuplexSession:
         response_input_commit_seq = self._response.active_response_input_commit_seq
         if response_input_commit_seq is None:
             response_input_commit_seq = self.input_commit_seq
-        response_history_is_late = self.input_commit_seq > response_input_commit_seq
+        response_item_id = f"item_{response_id}" if response_id is not None else None
+        response_history_is_reserved = response_item_id is not None and (
+            response_item_id in self._conversation.item_ids
+            or response_item_id in self._conversation.history_item_placeholders
+        )
+        response_history_is_late = (
+            self.input_commit_seq > response_input_commit_seq and not response_history_is_reserved
+        )
         assistant_text = "".join(self._response.assistant_text_buffer).strip()
         message = None
         if assistant_text:
@@ -1041,9 +1221,8 @@ class DuplexSession:
             committed_text = ""
         if commit_text and committed_text:
             message = {"role": "assistant", "content": committed_text}
-            item_id = f"item_{response_id}" if response_id is not None else None
-            if item_id is not None and item_id in self._conversation.history_item_placeholders:
-                self._store_history_item_message(item_id, message)
+            if response_item_id is not None and response_history_is_reserved:
+                self._store_history_item_message(response_item_id, message)
             else:
                 self._conversation.messages.append(message)
         effective_playback_policy = playback_commit_policy or self.config.playback_commit_policy
@@ -1054,7 +1233,7 @@ class DuplexSession:
             and effective_playback_policy == DuplexPlaybackCommitPolicy.ACK_ONLY.value
         ):
             self.register_history_item(f"item_{response_id}", None)
-        elif response_id is not None and not assistant_text:
+        elif response_id is not None and (not assistant_text or not commit_text):
             self._discard_history_item_placeholder(f"item_{response_id}")
         self._response.assistant_text_buffer.clear()
         if not preserve_request:
@@ -1113,6 +1292,11 @@ class DuplexSession:
             )
 
     def delete_history_item(self, item_id: str) -> bool:
+        removed_pending_turn = False
+        pending_server_vad_turn = self._input.pending_server_vad_turn
+        if pending_server_vad_turn is not None and pending_server_vad_turn.item_id == item_id:
+            self._input.pending_server_vad_turn = None
+            removed_pending_turn = True
         response_id = item_id.removeprefix("item_") if item_id.startswith("item_") else None
         message = self._conversation.item_ids.pop(item_id, None)
         self._conversation.item_audio_text_marks.pop(item_id, None)
@@ -1125,7 +1309,7 @@ class DuplexSession:
         if response_id is not None:
             self._conversation.assistant_response_snapshots.pop(response_id, None)
         if message is None:
-            return pending is not None or removed_placeholder
+            return removed_pending_turn or pending is not None or removed_placeholder
         try:
             self._conversation.messages.remove(message)
         except ValueError:
@@ -1390,6 +1574,8 @@ class DuplexSession:
         self._response.active_response_turn_id = None
         self._response.active_response_input_commit_seq = None
         self._response.active_response_awaits_input_commit = False
+        self.clear_server_vad_audio()
+        self._input.pending_server_vad_turn = None
         self._clear_response_metrics()
         self._restore_response_config()
 
@@ -1420,6 +1606,8 @@ class DuplexSession:
             "playback": self.playback.as_dict(),
             "capabilities": self.capabilities.as_dict(),
         }
+        if self.config.turn_detection_configured:
+            payload["turn_detection"] = self.config.server_vad.as_dict() if self.config.server_vad is not None else None
         if isinstance(self.config.extra_body.get("realtime_tools"), list):
             payload["tools"] = self.config.extra_body["realtime_tools"]
         if isinstance(self.config.extra_body.get("realtime_tool_choice"), str | dict):

vllm_omni/entrypoints/duplex/realtime_output.py

@@ -39,6 +39,28 @@ class RealtimeOutputProjector:
         if event_type == "session.updated":
             session = self._realtime_session_payload(event.get("session"))
             return [{"type": "session.updated", "session": session}]
+        if event_type == "input_audio_buffer.speech_started":
+            item_id = event.get("item_id")
+            if isinstance(item_id, str):
+                self._active_input_item_id = item_id
+            self._input_speech_started = True
+            self._input_audio_buffer_has_audio = True
+            return [
+                {
+                    "type": "input_audio_buffer.speech_started",
+                    "audio_start_ms": max(0, int(event.get("audio_start_ms", 0) or 0)),
+                    "item_id": item_id,
+                }
+            ]
+        if event_type == "input_audio_buffer.speech_stopped":
+            self._input_speech_started = False
+            return [
+                {
+                    "type": "input_audio_buffer.speech_stopped",
+                    "audio_end_ms": max(0, int(event.get("audio_end_ms", 0) or 0)),
+                    "item_id": event.get("item_id") or self._active_input_item_id,
+                }
+            ]
         if event_type in {
             "session.resumed",
             "session.heartbeat_ack",
@@ -201,14 +223,15 @@ class RealtimeOutputProjector:
                 ),
             ]
         if event_type == "input.committed":
-            event_item_id = event.get("realtime_item_id")
-            item_id = (
-                event_item_id
-                if isinstance(event_item_id, str) and event_item_id
-                else self._pop_pending_commit_item_id()
-            )
+            event_item_id = event.get("item_id")
+            # Acknowledgements carry their own identity; untracked silence
+            # commits must not consume another turn's item ID.
+            item_id = event_item_id if isinstance(event_item_id, str) and event_item_id else f"item_{uuid4().hex}"
+            # The reader may already have accepted audio for the next turn.
+            if self._active_input_item_id == item_id:
+                self._reset_realtime_input_buffer_state()
             item = self._conversation_items.get(item_id)
-            created_payload: list[dict[str, object]] = []
+            payloads = [self._input_audio_buffer_committed_event(item_id=item_id)]
             if item is None:
                 message = event.get("message")
                 no_response = event.get("no_response") is True
@@ -226,19 +249,18 @@ class RealtimeOutputProjector:
                     ),
                 }
                 self._conversation_items[item_id] = item
-                created_payload.extend(self._conversation_item_added_events(item))
+                payloads.extend(self._conversation_item_added_events(item))
             item["status"] = "completed"
-            payloads = created_payload + [
-                self._input_audio_buffer_committed_event(item_id=item_id, event=event),
-            ]
             transcription_event = self._input_audio_transcription_completed_event(item_id, item)
             if transcription_event is not None:
                 payloads.append(transcription_event)
             payloads.append(self._conversation_item_done_event(item))
             return payloads
         if event_type == "input.cancelled":
+            self._reset_realtime_input_buffer_state()
             return [{"type": "input_audio_buffer.cleared"}]
         if event_type == "input_audio_buffer.cleared":
+            self._reset_realtime_input_buffer_state()
             return [{"type": "input_audio_buffer.cleared"}]
         if event_type == "audio.cancelled":
             response_id = event.get("response_id")
@@ -416,7 +438,14 @@ class RealtimeOutputProjector:
                 return [event]
             message = str(raw_error or event.get("message") or "Duplex runtime error")
             code = str(event.get("code") or "duplex_error")
-            return [self._realtime_error_payload(code, message, event_id=event.get("realtime_event_id"))]
+            return [
+                self._realtime_error_payload(
+                    code,
+                    message,
+                    event_id=event.get("realtime_event_id") or event.get("event_id"),
+                    param=event.get("param"),
+                )
+            ]
         if event_type == "session.closed":
             return [{"type": "session.closed", "event": event}]
         return [{"type": f"duplex.{event_type}", "event": event}]
@@ -490,7 +519,14 @@ class RealtimeOutputProjector:
                 },
             },
         )
-        payload.setdefault("turn_detection", payload.get("turn_detection"))
+        if self._turn_detection_configured or "turn_detection" in payload:
+            turn_detection = payload.get("turn_detection", self._turn_detection)
+            audio = dict(payload["audio"]) if isinstance(payload.get("audio"), dict) else {}
+            audio_input = dict(audio["input"]) if isinstance(audio.get("input"), dict) else {}
+            audio_input["turn_detection"] = turn_detection
+            audio["input"] = audio_input
+            payload["audio"] = audio
+            payload["turn_detection"] = turn_detection
         payload.setdefault("input_audio_transcription", payload.get("input_audio_transcription"))
         payload.setdefault("tracing", payload.get("tracing"))
         return payload

vllm_omni/entrypoints/duplex/realtime_session.py

@@ -42,6 +42,10 @@ class NativeRealtimeSessionProtocol(
     def bind_sender(self, send_realtime_json) -> None:
         self._send_realtime_json = send_realtime_json
 
+    def bind_native_input_append(self, enabled: bool) -> None:
+        """Use the serving-selected input path, independent of interruption policy."""
+        self._native_input_append = enabled
+
     def _default_session_payload(self) -> dict[str, object]:
         payload: dict[str, object] = {
             "model": self._default_model,

vllm_omni/entrypoints/duplex/realtime_state.py

@@ -24,6 +24,32 @@ REALTIME_OUTPUT_AUDIO_FORMATS = {
     "g711_ulaw",
     "g711_alaw",
 }
+REALTIME_PCM_DEFAULT_SAMPLE_RATE_HZ = 24_000
+REALTIME_PCM_F32_DEFAULT_SAMPLE_RATE_HZ = 16_000
+REALTIME_G711_DEFAULT_SAMPLE_RATE_HZ = 8_000
+
+_RealtimeProtocolConfig = tuple[str, int, str, int | None, float]
+_PendingTurnDetectionUpdate = tuple[
+    bool,
+    dict[str, object] | None,
+    asyncio.Event,
+    _RealtimeProtocolConfig,
+]
+
+
+def realtime_default_sample_rate_hz(fmt: object) -> int | None:
+    if not isinstance(fmt, str):
+        return None
+    normalized = fmt.lower()
+    if normalized in {"pcm16", "pcm_s16le", "s16le", "pcm", "audio/pcm"}:
+        return REALTIME_PCM_DEFAULT_SAMPLE_RATE_HZ
+    if normalized in {"pcm_f32le", "audio/pcm_f32le"}:
+        return REALTIME_PCM_F32_DEFAULT_SAMPLE_RATE_HZ
+    if normalized in {"g711_ulaw", "g711_alaw", "audio/g711_ulaw", "audio/g711_alaw"}:
+        return REALTIME_G711_DEFAULT_SAMPLE_RATE_HZ
+    return None
+
+
 REALTIME_ERROR_TYPES_BY_CODE = {
     "bad_event": "invalid_request_error",
     "bad_audio": "invalid_request_error",
@@ -43,12 +69,16 @@ REALTIME_ERROR_TYPES_BY_CODE = {
     "runtime_data_plane_text_without_audio": "server_error",
     "response_error": "server_error",
     "chat_error": "server_error",
+    "server_vad_initialization_failed": "server_error",
+    "server_vad_inference_failed": "server_error",
     "duplex_session_busy": "rate_limit_error",
     "resource_exhausted": "rate_limit_error",
+    "input_backpressure": "rate_limit_error",
     "response_already_active": "invalid_request_error",
     "response_not_active": "invalid_request_error",
     "response_create_without_input": "invalid_request_error",
     "input_audio_buffer_empty": "invalid_request_error",
+    "server_vad_manual_commit_unsupported": "invalid_request_error",
     "missing_item_id": "invalid_request_error",
     "item_not_found": "invalid_request_error",
     "playback_item_mismatch": "invalid_request_error",
@@ -63,7 +93,6 @@ REALTIME_ERROR_TYPES_BY_CODE = {
     "native_text_append_unsupported": "invalid_request_error",
     "runtime_native_stage_role_required": "server_error",
     "runtime_native_runner_kv_required": "server_error",
-    "server_vad_unavailable": "server_error",
 }
 
 
@@ -109,12 +138,12 @@ class RealtimeSessionState:
     _default_session_id: object | None = None
     _default_extra_body: dict[str, object] = field(default_factory=dict)
     _input_audio_format: str = "pcm16"
-    _input_sample_rate_hz: int = 16000
+    _input_sample_rate_hz: int = REALTIME_PCM_DEFAULT_SAMPLE_RATE_HZ
     _output_audio_format: str = "pcm16"
     _overlap_silence_rms: float = 0.003
     _turn_detection: dict[str, object] | None = None
-    _server_vad: object | None = None
-    _pending_turn_detection_update: tuple[dict[str, object] | None, object | None, asyncio.Event] | None = None
+    _native_input_append: bool = False
+    _pending_turn_detection_update: _PendingTurnDetectionUpdate | None = None
     _send_realtime_json: Any = None
     _initial_session_update: bool = False
     _input_speech_started: bool = False
@@ -123,13 +152,13 @@ class RealtimeSessionState:
     _active_response_id: str | None = None
     _last_response_id: str | None = None
     _conversation_items: dict[str, dict[str, object]] = field(default_factory=dict)
-    _pending_commit_item_ids: asyncio.Queue[str] = field(default_factory=asyncio.Queue)
     _last_conversation_item_id: str | None = None
     _output_sample_rate_hz: int | None = None
     _active_input_item_id: str | None = None
     _input_audio_buffer_has_audio: bool = False
     _input_audio_buffer_had_non_speech: bool = False
     _input_audio_buffer_transcript_parts: list[str] = field(default_factory=list)
+    _turn_detection_configured: bool = False
 
     @classmethod
     def from_query_params(cls, query_params: Any) -> RealtimeSessionState:
@@ -197,10 +226,8 @@ class RealtimeStateOwner:
     _output_audio_format: str = _RealtimeStateField()
     _overlap_silence_rms: float = _RealtimeStateField()
     _turn_detection: dict[str, object] | None = _RealtimeStateField()
-    _server_vad: object | None = _RealtimeStateField()
-    _pending_turn_detection_update: tuple[dict[str, object] | None, object | None, asyncio.Event] | None = (
-        _RealtimeStateField()
-    )
+    _native_input_append: bool = _RealtimeStateField()
+    _pending_turn_detection_update: _PendingTurnDetectionUpdate | None = _RealtimeStateField()
     _send_realtime_json: Any = _RealtimeStateField()
     _initial_session_update: bool = _RealtimeStateField()
     _input_speech_started: bool = _RealtimeStateField()
@@ -209,10 +236,10 @@ class RealtimeStateOwner:
     _active_response_id: str | None = _RealtimeStateField()
     _last_response_id: str | None = _RealtimeStateField()
     _conversation_items: dict[str, dict[str, object]] = _RealtimeStateField()
-    _pending_commit_item_ids: asyncio.Queue[str] = _RealtimeStateField()
     _last_conversation_item_id: str | None = _RealtimeStateField()
     _output_sample_rate_hz: int | None = _RealtimeStateField()
     _active_input_item_id: str | None = _RealtimeStateField()
     _input_audio_buffer_has_audio: bool = _RealtimeStateField()
     _input_audio_buffer_had_non_speech: bool = _RealtimeStateField()
     _input_audio_buffer_transcript_parts: list[str] = _RealtimeStateField()
+    _turn_detection_configured: bool = _RealtimeStateField()

vllm_omni/entrypoints/duplex/runtime_adapter.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 
 import asyncio
 from collections.abc import Awaitable, Callable, Iterable, Mapping, MutableMapping
+from dataclasses import dataclass, field
 from importlib import import_module
 from typing import Any, Protocol
 
@@ -106,6 +107,104 @@ class ServingRuntimeSessionState(Protocol):
     def clear_continuation(self) -> None: ...
 
 
+class _TurnBasedPcmAppendBuffer:
+    """Empty native-audio buffer for chat-fallback sessions.
+
+    Turn-based sessions keep input audio in ``DuplexSession`` and never use
+    the model-native append path.  The runner still shares lifecycle and
+    cancellation code with native sessions, so it needs an inert buffer for
+    those common operations.
+    """
+
+    @property
+    def pending_byte_count(self) -> int:
+        return 0
+
+    def clear(self) -> None:
+        return None
+
+    def clear_force_listen(self) -> None:
+        return None
+
+    def has_pending(self) -> bool:
+        return False
+
+    def has_reserved(self) -> bool:
+        return False
+
+    def prepare_append(
+        self,
+        payload: dict[str, object],
+        *,
+        operation_id: str,
+        chunk_period_ms: int,
+        allow_emit: bool,
+    ) -> PcmAppendReservation | None:
+        del payload, operation_id, chunk_period_ms, allow_emit
+        raise RuntimeError("Model-native audio append is not enabled for this session")
+
+    def prepare_commit(
+        self,
+        *,
+        operation_id: str,
+        chunk_period_ms: int,
+    ) -> PcmAppendReservation:
+        del operation_id, chunk_period_ms
+        raise RuntimeError("Model-native audio append is not enabled for this session")
+
+    def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None:
+        del chunk_period_ms
+        return None
+
+
+@dataclass(slots=True)
+class TurnBasedServingSessionState:
+    """Runner-local state for sessions served through chat fallback."""
+
+    audio_buffer: PcmAppendBuffer = field(default_factory=_TurnBasedPcmAppendBuffer)
+    input_since_commit: bool = False
+    speech_since_commit: bool = False
+    native_context_locked: bool = False
+    committed_audio_payload: dict[str, object] | None = None
+    committed_audio_operation_id: str | None = None
+    committed_audio_reserved_bytes: int = 0
+    deferred_response_create: bool = False
+    deferred_precreate_response: bool = False
+    data_plane_task: asyncio.Task[None] | None = None
+    data_plane_restart_requested: bool = False
+    continuation_owner_id: str | None = None
+    continuation_units: int = 0
+    pending_silence_task: asyncio.Task[bool] | None = None
+    pending_silence_owner_id: str | None = None
+    silence_continuation_scheduler: Callable[..., Awaitable[bool]] | None = None
+
+    def retain_committed_audio(
+        self,
+        payload: dict[str, object],
+        *,
+        operation_id: str | None,
+        reserved_bytes: int = 0,
+    ) -> None:
+        self.committed_audio_payload = payload
+        self.committed_audio_operation_id = operation_id
+        self.committed_audio_reserved_bytes += max(0, int(reserved_bytes))
+
+    def clear_committed_audio(self) -> int:
+        reserved_bytes = self.committed_audio_reserved_bytes
+        self.committed_audio_payload = None
+        self.committed_audio_operation_id = None
+        self.committed_audio_reserved_bytes = 0
+        self.deferred_response_create = False
+        self.deferred_precreate_response = False
+        return reserved_bytes
+
+    def clear_continuation(self) -> None:
+        self.continuation_owner_id = None
+        self.continuation_units = 0
+        self.pending_silence_task = None
+        self.pending_silence_owner_id = None
+
+
 class RuntimeDataPlane(Protocol):
     def begin_request(self, request_id: str) -> None: ...
 

vllm_omni/entrypoints/duplex/runtime_bridge.py

@@ -39,6 +39,11 @@ class NativeRuntimeBridgeMixin:
     }
 
     async def _open_runtime_session(self, session: DuplexSession, send_json) -> dict[str, object] | bool:
+        # Turn-based sessions use the chat-fallback path and do not own a
+        # model-native duplex runtime.  Do not send control messages merely
+        # because the shared engine client exposes the duplex RPC methods.
+        if not self._uses_serving_runtime_adapter(session.config):
+            return True
         contract_error = self._native_runtime_contract_error(session)
         if contract_error is not None:
             await send_json(
@@ -535,6 +540,8 @@ class NativeRuntimeBridgeMixin:
         session_config: dict[str, object] | None = None,
         runtime_config: dict[str, object] | None = None,
     ) -> bool:
+        if not self._uses_serving_runtime_adapter(session.config):
+            return True
         signal_turn = getattr(self._chat_service.engine_client, "signal_duplex_turn_async", None)
         if not callable(signal_turn):
             return True
@@ -582,6 +589,8 @@ class NativeRuntimeBridgeMixin:
         return True
 
     async def _close_runtime_session(self, session: DuplexSession, *, reason: str, send_json=None) -> bool:
+        if not self._uses_serving_runtime_adapter(session.config):
+            return True
         close_session = getattr(self._chat_service.engine_client, "close_duplex_session_async", None)
         if not callable(close_session):
             return True
@@ -1301,11 +1310,19 @@ class NativeRuntimeBridgeMixin:
 
     def _cleanup_duplex_session_state(self, session: DuplexSession) -> None:
         session_id = session.session_id
-        self._serving_runtime_adapter.remove_session_state(session_id)
-        self._serving_runtime_adapter.data_plane.close_session(
-            session_id,
-            active_request_id=session.active_request_id,
-        )
+        pipeline = self._server_vad_pipelines.pop(session_id, None)
+        if pipeline is not None:
+            pipeline.reset()
+            self._realtime_vad_metrics.session_finished()
+        adapter = self._serving_runtime_adapter
+        if adapter is None:
+            self._serving_session_states.pop(session_id, None)
+        else:
+            adapter.remove_session_state(session_id)
+            adapter.data_plane.close_session(
+                session_id,
+                active_request_id=session.active_request_id,
+            )
 
     def _encode_native_data_plane_audio(
         self,

vllm_omni/entrypoints/duplex/server_vad.py

@@ -0,0 +1,528 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+from __future__ import annotations
+
+import asyncio
+import hashlib
+import threading
+import time
+from collections.abc import Mapping
+from dataclasses import dataclass, replace
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal, Protocol
+
+import numpy as np
+from vllm.transformers_utils.repo_utils import try_get_local_file
+
+if TYPE_CHECKING:
+    from vllm_omni.entrypoints.openai.audio_utils_mixin import StreamingAudioResampler
+
+SILERO_VAD_REPO_ID = "istupakov/silero-vad-onnx"
+SILERO_VAD_REVISION = "8b14476858ef240c50b3884bb38cc67290c1cc70"
+SILERO_VAD_FILENAME = "silero_vad.onnx"
+SILERO_VAD_SHA256 = "1a153a22f4509e292a94e67d6f9b85e8deb25b4988682b7e174c65279d8788e3"
+SILERO_VAD_MIN_THRESHOLD = 0.15
+SILERO_VAD_DEFAULT_MIN_SPEECH_DURATION_MS = 96
+
+_SERVER_VAD_FIELDS = {
+    "type",
+    "threshold",
+    "prefix_padding_ms",
+    "silence_duration_ms",
+    "create_response",
+    "interrupt_response",
+    "min_speech_duration_ms",
+}
+
+
+@dataclass(frozen=True, slots=True)
+class ServerVADConfig:
+    """Validated OpenAI-compatible ``server_vad`` session configuration."""
+
+    type: Literal["server_vad"] = "server_vad"
+    threshold: float = 0.5
+    prefix_padding_ms: int = 300
+    silence_duration_ms: int = 500
+    create_response: bool = True
+    # Omission is resolved from runtime capabilities before the session is opened.
+    interrupt_response: bool | None = None
+    min_speech_duration_ms: int | None = None
+
+    @classmethod
+    def from_value(cls, value: object) -> ServerVADConfig:
+        if not isinstance(value, dict):
+            raise ValueError("turn_detection must be null or an object")
+        unknown = sorted(set(value) - _SERVER_VAD_FIELDS)
+        if unknown:
+            raise ValueError(f"Unknown server_vad field(s): {', '.join(unknown)}")
+        vad_type = value.get("type")
+        if vad_type != "server_vad":
+            raise ValueError("turn_detection.type must be 'server_vad'")
+
+        threshold = value.get("threshold", 0.5)
+        if isinstance(threshold, bool) or not isinstance(threshold, int | float) or not 0 <= threshold <= 1:
+            raise ValueError("server_vad.threshold must be a number between 0 and 1")
+
+        prefix_padding_ms = value.get("prefix_padding_ms", 300)
+        if isinstance(prefix_padding_ms, bool) or not isinstance(prefix_padding_ms, int) or prefix_padding_ms < 0:
+            raise ValueError("server_vad.prefix_padding_ms must be a non-negative integer")
+
+        silence_duration_ms = value.get("silence_duration_ms", 500)
+        if (
+            isinstance(silence_duration_ms, bool)
+            or not isinstance(silence_duration_ms, int)
+            or silence_duration_ms <= 0
+        ):
+            raise ValueError("server_vad.silence_duration_ms must be a positive integer")
+
+        create_response = value.get("create_response", True)
+        if not isinstance(create_response, bool):
+            raise ValueError("server_vad.create_response must be a boolean")
+
+        interrupt_response = value.get("interrupt_response")
+        if "interrupt_response" in value and not isinstance(interrupt_response, bool):
+            raise ValueError("server_vad.interrupt_response must be a boolean")
+
+        min_speech_duration_ms = value.get("min_speech_duration_ms")
+        if min_speech_duration_ms is not None and (
+            isinstance(min_speech_duration_ms, bool)
+            or not isinstance(min_speech_duration_ms, int | float)
+            or not np.isfinite(float(min_speech_duration_ms))
+            or min_speech_duration_ms < 0
+        ):
+            raise ValueError("server_vad.min_speech_duration_ms must be a non-negative number")
+
+        return cls(
+            type=vad_type,
+            threshold=float(threshold),
+            prefix_padding_ms=prefix_padding_ms,
+            silence_duration_ms=silence_duration_ms,
+            create_response=create_response,
+            interrupt_response=interrupt_response,
+            min_speech_duration_ms=(int(min_speech_duration_ms) if min_speech_duration_ms is not None else None),
+        )
+
+    def as_dict(self) -> dict[str, object]:
+        result: dict[str, object] = {
+            "type": self.type,
+            "threshold": self.threshold,
+            "prefix_padding_ms": self.prefix_padding_ms,
+            "silence_duration_ms": self.silence_duration_ms,
+            "create_response": self.create_response,
+        }
+        if self.interrupt_response is not None:
+            result["interrupt_response"] = self.interrupt_response
+        if self.min_speech_duration_ms is not None:
+            result["min_speech_duration_ms"] = self.min_speech_duration_ms
+        return result
+
+
+def parse_session_turn_detection(
+    payload: Mapping[str, object],
+) -> tuple[bool, ServerVADConfig | None]:
+    """Extract and validate OpenAI-compatible turn detection aliases."""
+    configured_values: list[tuple[str, object]] = []
+    if "turn_detection" in payload:
+        configured_values.append(("turn_detection", payload["turn_detection"]))
+    audio = payload.get("audio")
+    if isinstance(audio, Mapping):
+        audio_input = audio.get("input")
+        if isinstance(audio_input, Mapping) and "turn_detection" in audio_input:
+            configured_values.append(("audio.input.turn_detection", audio_input["turn_detection"]))
+    if not configured_values:
+        return False, None
+
+    first_path, first_value = configured_values[0]
+    first_config = None if first_value is None else ServerVADConfig.from_value(first_value)
+    for field_path, value in configured_values[1:]:
+        config = None if value is None else ServerVADConfig.from_value(value)
+        if first_config is not None and config is not None:
+            # An omitted option must not conflict with an explicit alias value.
+            if first_config.interrupt_response is None:
+                first_config = replace(first_config, interrupt_response=config.interrupt_response)
+            if config.interrupt_response is None:
+                config = replace(config, interrupt_response=first_config.interrupt_response)
+        if config != first_config:
+            raise ValueError(f"{first_path} and {field_path} must not specify conflicting values")
+    return True, first_config
+
+
+class SpeechDetectorBackend(Protocol):
+    """Detector contract; per-stream state is owned by the pipeline."""
+
+    frame_samples: int
+
+    def new_state(self) -> object: ...
+
+    def infer(self, frame: np.ndarray, state: object) -> tuple[float, object]: ...
+
+
+class SpeechDetectorBackendProvider(Protocol):
+    def get(self) -> SpeechDetectorBackend: ...
+
+
+@dataclass(frozen=True, slots=True)
+class SpeechEndpointDecision:
+    speech_started: bool = False
+    speech_stopped: bool = False
+    audio_start_ms: int | None = None
+    audio_end_ms: int | None = None
+    endpoint_delay_ms: int | None = None
+
+
+class SpeechEndpointPolicy(Protocol):
+    @property
+    def speech_active(self) -> bool: ...
+
+    def update(
+        self,
+        probability: float,
+        *,
+        frame_start_sample: int,
+        frame_samples: int,
+    ) -> SpeechEndpointDecision: ...
+
+    def reset(self) -> None: ...
+
+
+class ThresholdEndpointPolicy:
+    """Apply threshold and trailing-silence endpoint rules."""
+
+    def __init__(self, config: ServerVADConfig, *, sample_rate_hz: int) -> None:
+        self.config = config
+        self.sample_rate_hz = sample_rate_hz
+        self._speech_active = False
+        self._candidate_samples = 0
+        self._candidate_start_sample: int | None = None
+        self._silence_samples = 0
+
+    @property
+    def speech_active(self) -> bool:
+        return self._speech_active
+
+    def update(
+        self,
+        probability: float,
+        *,
+        frame_start_sample: int,
+        frame_samples: int,
+    ) -> SpeechEndpointDecision:
+        if probability >= self.config.threshold:
+            self._silence_samples = 0
+            if not self._speech_active:
+                if self._candidate_samples == 0:
+                    self._candidate_start_sample = frame_start_sample
+                self._candidate_samples += frame_samples
+                min_speech_duration_ms = self.config.min_speech_duration_ms or 0
+                min_speech_samples = max(1, round(min_speech_duration_ms * self.sample_rate_hz / 1000))
+                if self._candidate_samples < min_speech_samples:
+                    return SpeechEndpointDecision()
+                self._speech_active = True
+                detected_start_sample = (
+                    self._candidate_start_sample if self._candidate_start_sample is not None else frame_start_sample
+                )
+                detected_start_ms = round(detected_start_sample * 1000 / self.sample_rate_hz)
+                self._candidate_samples = 0
+                self._candidate_start_sample = None
+                return SpeechEndpointDecision(
+                    speech_started=True,
+                    audio_start_ms=max(0, detected_start_ms - self.config.prefix_padding_ms),
+                )
+            return SpeechEndpointDecision()
+
+        if not self._speech_active:
+            self._candidate_samples = 0
+            self._candidate_start_sample = None
+            return SpeechEndpointDecision()
+
+        # Match Silero v6.2's streaming VAD hysteresis: activation uses the
+        # configured threshold, while potential speech end starts only below
+        # ``max(threshold - 0.15, 0.01)``. Once a silence candidate exists,
+        # intermediate frames keep elapsed time moving but cannot themselves
+        # close the turn.
+        negative_threshold = max(self.config.threshold - SILERO_VAD_MIN_THRESHOLD, 0.01)
+        below_negative_threshold = probability < negative_threshold
+        if not below_negative_threshold and self._silence_samples == 0:
+            return SpeechEndpointDecision()
+
+        self._silence_samples += frame_samples
+        if not below_negative_threshold:
+            return SpeechEndpointDecision()
+        silence_limit = max(1, round(self.config.silence_duration_ms * self.sample_rate_hz / 1000))
+        if self._silence_samples < silence_limit:
+            return SpeechEndpointDecision()
+
+        # OpenAI defines ``audio_end_ms`` as the end of the audio sent to the
+        # model, including the trailing silence used for endpoint detection.
+        audio_end_sample = frame_start_sample + frame_samples
+        endpoint_delay_ms = round(self._silence_samples * 1000 / self.sample_rate_hz)
+        self.reset()
+        return SpeechEndpointDecision(
+            speech_stopped=True,
+            audio_end_ms=max(0, round(audio_end_sample * 1000 / self.sample_rate_hz)),
+            endpoint_delay_ms=endpoint_delay_ms,
+        )
+
+    def reset(self) -> None:
+        self._speech_active = False
+        self._candidate_samples = 0
+        self._candidate_start_sample = None
+        self._silence_samples = 0
+
+
+@dataclass(frozen=True, slots=True)
+class ServerVADFrame:
+    samples: np.ndarray
+    decision: SpeechEndpointDecision
+
+
+@dataclass(frozen=True, slots=True)
+class ServerVADBatch:
+    frames: tuple[ServerVADFrame, ...]
+    inference_ms: float
+
+
+class ServerVADPipeline:
+    """Frame continuous 16 kHz audio and apply acoustic endpointing.
+
+    The pipeline retains only input-resampling residuals, a partial frame,
+    detector state, and timing counters. Commit-eligible prefix and utterance
+    audio remain session-owned.
+    """
+
+    sample_rate_hz = 16_000
+
+    def __init__(
+        self,
+        backend: SpeechDetectorBackend,
+        config: ServerVADConfig,
+        *,
+        endpoint_policy: SpeechEndpointPolicy | None = None,
+    ) -> None:
+        self.backend = backend
+        self.config = config
+        self.endpoint_policy = endpoint_policy or ThresholdEndpointPolicy(
+            config,
+            sample_rate_hz=self.sample_rate_hz,
+        )
+        self._input_resampler: StreamingAudioResampler | None = None
+        self._source_sample_rate_hz: int | None = None
+        self._scratch = np.empty(0, dtype=np.float32)
+        self._detector_state = backend.new_state()
+        self._processed_samples = 0
+        self._stream_start_ms = 0
+
+    @property
+    def speech_active(self) -> bool:
+        return self.endpoint_policy.speech_active
+
+    @property
+    def source_sample_rate_hz(self) -> int | None:
+        return self._source_sample_rate_hz
+
+    @property
+    def scratch_bytes(self) -> int:
+        resampler_bytes = self._input_resampler.scratch_bytes if self._input_resampler is not None else 0
+        return int(self._scratch.nbytes) + resampler_bytes
+
+    async def push_pcm16(
+        self,
+        samples: bytes,
+        *,
+        source_sample_rate_hz: int,
+    ) -> ServerVADBatch:
+        """Normalize chunked mono PCM16 input and run endpoint detection."""
+        if isinstance(source_sample_rate_hz, bool) or source_sample_rate_hz not in {16_000, 24_000}:
+            raise ValueError("server_vad PCM16 input sample rate must be 16000 or 24000 Hz")
+
+        if len(samples) % np.dtype("<i2").itemsize:
+            raise ValueError("server_vad PCM16 input contains an incomplete sample")
+        pcm16 = np.frombuffer(samples, dtype="<i2")
+
+        if self._source_sample_rate_hz is not None and source_sample_rate_hz != self._source_sample_rate_hz:
+            raise ValueError("server_vad input sample rate cannot change within a continuous audio stream")
+        if pcm16.size and self._source_sample_rate_hz is None:
+            from vllm_omni.entrypoints.openai.audio_utils_mixin import StreamingAudioResampler
+
+            self._input_resampler = (
+                None
+                if source_sample_rate_hz == self.sample_rate_hz
+                else StreamingAudioResampler(source_sample_rate_hz, self.sample_rate_hz)
+            )
+            self._source_sample_rate_hz = source_sample_rate_hz
+
+        normalized = np.ascontiguousarray(pcm16, dtype=np.float32) * np.float32(1.0 / 32768.0)
+        if self._input_resampler is not None:
+            normalized = self._input_resampler.process(normalized)
+        return await self.push(normalized)
+
+    async def push(self, samples: np.ndarray) -> ServerVADBatch:
+        normalized = np.ascontiguousarray(samples, dtype=np.float32).reshape(-1)
+        return await asyncio.to_thread(self._push_sync, normalized)
+
+    def _push_sync(self, samples: np.ndarray) -> ServerVADBatch:
+        started_at = time.perf_counter()
+        if self._scratch.size:
+            samples = np.concatenate((self._scratch, samples))
+        frame_samples = int(self.backend.frame_samples)
+        complete_samples = samples.size - samples.size % frame_samples
+        # Retained state must not keep the whole input chunk alive through a view.
+        self._scratch = samples[complete_samples:].copy()
+
+        results: list[ServerVADFrame] = []
+        for offset in range(0, complete_samples, frame_samples):
+            frame = np.ascontiguousarray(samples[offset : offset + frame_samples], dtype=np.float32)
+            frame_start = self._processed_samples
+            probability, self._detector_state = self.backend.infer(frame, self._detector_state)
+            probability = min(1.0, max(0.0, float(probability)))
+            decision = self.endpoint_policy.update(
+                probability,
+                frame_start_sample=frame_start,
+                frame_samples=frame_samples,
+            )
+            if decision.speech_started and decision.audio_start_ms is not None:
+                # Prefix audio before the most recent reset is no longer available.
+                decision = replace(decision, audio_start_ms=max(self._stream_start_ms, decision.audio_start_ms))
+
+            self._processed_samples += frame_samples
+            results.append(
+                ServerVADFrame(
+                    samples=frame,
+                    decision=decision,
+                )
+            )
+        return ServerVADBatch(
+            frames=tuple(results),
+            inference_ms=(time.perf_counter() - started_at) * 1000,
+        )
+
+    def reset(self) -> None:
+        """Discard stream state while preserving the session-relative audio clock."""
+        self._stream_start_ms = round(self._processed_samples * 1000 / self.sample_rate_hz)
+        self._input_resampler = None
+        self._source_sample_rate_hz = None
+        self._scratch = np.empty(0, dtype=np.float32)
+        self._detector_state = self.backend.new_state()
+        self.endpoint_policy.reset()
+
+
+@dataclass(frozen=True, slots=True)
+class _SileroVADState:
+    model_state: np.ndarray
+    context: np.ndarray
+
+
+class SileroVADBackend:
+    """Shared ONNX Runtime Silero v6.2 detector running on CPU."""
+
+    sample_rate_hz = 16_000
+    frame_samples = 512
+    context_samples = 64
+    model_state_shape = (2, 1, 128)
+
+    def __init__(self, model_path: str | Path) -> None:
+        try:
+            import onnxruntime as ort
+        except ImportError as exc:  # pragma: no cover - platform packaging supplies ORT.
+            raise RuntimeError("server_vad requires ONNX Runtime") from exc
+
+        session_options = ort.SessionOptions()
+        session_options.inter_op_num_threads = 1
+        session_options.intra_op_num_threads = 1
+
+        self.model_path = Path(model_path)
+        self._session = ort.InferenceSession(
+            str(self.model_path),
+            providers=["CPUExecutionProvider"],
+            sess_options=session_options,
+        )
+        input_names = {item.name for item in self._session.get_inputs()}
+        if "input" not in input_names or "sr" not in input_names or "state" not in input_names:
+            raise RuntimeError(f"Unsupported Silero ONNX input contract: {sorted(input_names)}")
+        self._warm_up()
+
+    def _warm_up(self) -> None:
+        self.infer(np.zeros(self.frame_samples, dtype=np.float32), self.new_state())
+
+    def new_state(self) -> _SileroVADState:
+        return _SileroVADState(
+            model_state=np.zeros(self.model_state_shape, dtype=np.float32),
+            context=np.zeros((1, self.context_samples), dtype=np.float32),
+        )
+
+    def infer(self, frame: np.ndarray, state: object) -> tuple[float, object]:
+        if not isinstance(state, _SileroVADState):
+            raise TypeError("Silero detector state must be created by SileroVADBackend.new_state()")
+        model_state = np.ascontiguousarray(state.model_state, dtype=np.float32)
+        context = np.ascontiguousarray(state.context, dtype=np.float32)
+        if model_state.shape != self.model_state_shape:
+            raise ValueError(f"Silero model state must have shape {self.model_state_shape}, got {model_state.shape}")
+        expected_context_shape = (1, self.context_samples)
+        if context.shape != expected_context_shape:
+            raise ValueError(f"Silero context must have shape {expected_context_shape}, got {context.shape}")
+
+        audio = np.ascontiguousarray(frame, dtype=np.float32).reshape(1, -1)
+        if audio.shape[1] != self.frame_samples:
+            raise ValueError(
+                f"Silero detector frame must contain exactly {self.frame_samples} samples, got {audio.shape[1]}"
+            )
+        model_input = np.concatenate((context, audio), axis=1)
+        inputs = {
+            "input": model_input,
+            "state": model_state,
+            "sr": np.asarray(self.sample_rate_hz, dtype=np.int64),
+        }
+        # ONNX Runtime permits concurrent Run calls on one CPU session. Stream
+        # state is explicit, so independent pipelines can safely share the session.
+        output = self._session.run(None, inputs)
+        if len(output) < 2:
+            raise RuntimeError("Silero ONNX model did not return probability and model state")
+        probability = float(np.asarray(output[0]).reshape(-1)[0])
+        next_state = _SileroVADState(
+            model_state=np.ascontiguousarray(output[1], dtype=np.float32),
+            context=np.ascontiguousarray(model_input[:, -self.context_samples :], dtype=np.float32),
+        )
+        return probability, next_state
+
+
+class SileroVADBackendProvider:
+    """Resolve, verify, and load one Silero model instance per process."""
+
+    def __init__(self, *, model_path: str | None = None) -> None:
+        self.model_path = model_path
+        self._backend: SileroVADBackend | None = None
+        self._lock = threading.Lock()
+
+    def get(self) -> SileroVADBackend:
+        if self._backend is not None:
+            return self._backend
+        with self._lock:
+            if self._backend is None:
+                path = self._resolve_local_artifact()
+                self._verify_checksum(path)
+                self._backend = SileroVADBackend(path)
+        return self._backend
+
+    def _resolve_local_artifact(self) -> Path:
+        if self.model_path:
+            path = Path(self.model_path).expanduser()
+            if path.is_file():
+                return path
+            raise RuntimeError(f"Configured Silero VAD model does not exist: {path}")
+        cached = try_get_local_file(
+            model=SILERO_VAD_REPO_ID,
+            file_name=SILERO_VAD_FILENAME,
+            revision=SILERO_VAD_REVISION,
+        )
+        if isinstance(cached, Path) and cached.is_file():
+            return cached
+        raise RuntimeError(
+            "Silero VAD artifact is not available locally. Pre-download the pinned artifact "
+            "or configure duplex_session.server_vad_model_path."
+        )
+
+    def _verify_checksum(self, path: Path) -> None:
+        digest = hashlib.sha256(path.read_bytes()).hexdigest()
+        if digest != SILERO_VAD_SHA256:
+            raise RuntimeError(f"Silero VAD checksum mismatch for {path}: expected {SILERO_VAD_SHA256}, got {digest}")

vllm_omni/entrypoints/duplex/serving.py

@@ -8,9 +8,9 @@ import base64
 import binascii
 import inspect
 import json
-from collections.abc import Mapping
+from collections.abc import Mapping, MutableMapping
 from contextlib import suppress
-from dataclasses import dataclass
+from dataclasses import dataclass, replace
 from typing import TYPE_CHECKING
 
 import numpy as np
@@ -49,12 +49,21 @@ from vllm_omni.entrypoints.duplex.runtime_adapter import (
     ServingRuntimeAdapter,
     ServingRuntimeConfigError,
     ServingRuntimeSessionState,
+    TurnBasedServingSessionState,
     load_serving_runtime_adapter,
     validate_serving_runtime_adapter,
 )
 from vllm_omni.entrypoints.duplex.runtime_bridge import (
     NativeRuntimeBridgeMixin,
 )
+from vllm_omni.entrypoints.duplex.server_vad import (
+    SILERO_VAD_DEFAULT_MIN_SPEECH_DURATION_MS,
+    SILERO_VAD_MIN_THRESHOLD,
+    ServerVADPipeline,
+    SileroVADBackendProvider,
+    SpeechDetectorBackendProvider,
+    parse_session_turn_detection,
+)
 from vllm_omni.entrypoints.duplex.session_attachment import (
     DuplexJournalGapError,
     DuplexSessionAttachmentRegistry,
@@ -68,6 +77,7 @@ from vllm_omni.entrypoints.duplex.websocket import (
     DuplexSessionTasks,
     DuplexWebSocketActor,
 )
+from vllm_omni.metrics.realtime import RealtimeVADMetrics
 
 if TYPE_CHECKING:
     from vllm_omni.entrypoints.openai.serving_chat import OmniOpenAIServingChat
@@ -85,6 +95,7 @@ class _DuplexSessionHandshake:
     session: DuplexSession
     resumed: bool = False
     attachment_generation: int | None = None
+    event_id: str | None = None
 
 
 class OmniDuplexSessionHandler(
@@ -105,16 +116,27 @@ class OmniDuplexSessionHandler(
         self,
         *,
         chat_service: OmniOpenAIServingChat,
+        served_model_name: str | None = None,
         config_timeout_s: float = _DEFAULT_CONFIG_TIMEOUT_S,
         idle_timeout_s: float = _DEFAULT_IDLE_TIMEOUT_S,
+        log_stats: bool = True,
         duplex_session_config: DuplexSessionRuntimeConfig | None = None,
         serving_runtime_adapter: ServingRuntimeAdapter | None = None,
         serving_runtime_adapter_path: str | None = None,
+        server_vad_backend_provider: SpeechDetectorBackendProvider | None = None,
     ) -> None:
         self._chat_service = chat_service
         self._config_timeout_s = config_timeout_s
         self._idle_timeout_s = idle_timeout_s
         self._duplex_session_config = duplex_session_config or DuplexSessionRuntimeConfig()
+        self._server_vad_backend_provider = server_vad_backend_provider or SileroVADBackendProvider(
+            model_path=self._duplex_session_config.server_vad_model_path,
+        )
+        self._server_vad_pipelines: dict[str, ServerVADPipeline] = {}
+        self._realtime_vad_metrics = RealtimeVADMetrics(
+            served_model_name or chat_service.model_config.model,
+            log_stats=log_stats,
+        )
         self._registry = DuplexSessionRegistry(
             DuplexCapabilities(
                 supports_model_native_turn_policy=False,
@@ -131,6 +153,7 @@ class OmniDuplexSessionHandler(
             "duplex_serving_adapter_path",
             None,
         )
+        self._serving_runtime_adapter: ServingRuntimeAdapter | None
         if serving_runtime_adapter is not None:
             self._serving_runtime_adapter = validate_serving_runtime_adapter(serving_runtime_adapter)
         elif isinstance(adapter_path, str) and adapter_path:
@@ -139,7 +162,10 @@ class OmniDuplexSessionHandler(
                 self._encode_native_data_plane_audio,
             )
         else:
-            raise ValueError("A duplex serving runtime adapter must be explicitly configured")
+            self._serving_runtime_adapter = None
+        self._serving_session_states: MutableMapping[str, ServingRuntimeSessionState] = (
+            self._serving_runtime_adapter.session_states if self._serving_runtime_adapter is not None else {}
+        )
         self._session_tasks: dict[str, DuplexSessionTasks] = {}
         self._realtime_protocols: dict[str, NativeRealtimeSessionProtocol] = {}
         self._lease_generations: dict[str, int] = {}
@@ -223,7 +249,7 @@ class OmniDuplexSessionHandler(
                 active_response_task.cancel()
                 with suppress(asyncio.CancelledError):
                     await active_response_task
-        native = self._serving_runtime_adapter.session_states.get(session.session_id)
+        native = self._serving_session_states.get(session.session_id)
         if native is not None and native.data_plane_task is not None:
             data_plane_task = native.data_plane_task
             native.data_plane_task = None
@@ -794,8 +820,27 @@ class OmniDuplexSessionHandler(
         duration_ms = cls._input_audio_duration_ms(event, payload)
         return 0 < duration_ms <= session.config.overlap_short_ack_ms
 
+    def _create_runtime_session_state(self) -> ServingRuntimeSessionState:
+        adapter = self._serving_runtime_adapter
+        if adapter is not None:
+            return adapter.create_session_state()
+        return TurnBasedServingSessionState()
+
     def _runtime_session_state(self, session: DuplexSession) -> ServingRuntimeSessionState:
-        return self._serving_runtime_adapter.session_state(session.session_id)
+        adapter = self._serving_runtime_adapter
+        if adapter is not None:
+            return adapter.session_state(session.session_id)
+        state = self._serving_session_states.get(session.session_id)
+        if state is None:
+            state = TurnBasedServingSessionState()
+            self._serving_session_states[session.session_id] = state
+        return state
+
+    def _require_serving_runtime_adapter(self) -> ServingRuntimeAdapter:
+        adapter = self._serving_runtime_adapter
+        if adapter is None:
+            raise RuntimeError("Model-native duplex runtime is not configured for this session")
+        return adapter
 
     def _should_force_listen_for_auto_response_overlap(
         self,
@@ -823,21 +868,25 @@ class OmniDuplexSessionHandler(
         *,
         session: DuplexSession,
     ) -> bool:
-        for key in ("is_speech", "speech"):
-            value = event.get(key)
-            if isinstance(value, bool):
-                return value
-        vad = event.get("vad")
-        if isinstance(vad, dict):
-            value = vad.get("is_speech")
-            if isinstance(value, bool):
-                return value
-            probability = vad.get("speech_probability", vad.get("probability"))
+        # Prefer serving-derived fields in ``payload`` over optional client
+        # hints retained in ``event``. In server-VAD mode the pipeline, rather
+        # than the client or the RMS fallback below, owns speech classification.
+        for source in (payload, event):
+            for key in ("is_speech", "speech"):
+                value = source.get(key)
+                if isinstance(value, bool):
+                    return value
+            vad = source.get("vad")
+            if isinstance(vad, dict):
+                value = vad.get("is_speech")
+                if isinstance(value, bool):
+                    return value
+                probability = vad.get("speech_probability", vad.get("probability"))
+                if isinstance(probability, int | float):
+                    return float(probability) >= 0.5
+            probability = source.get("speech_probability")
             if isinstance(probability, int | float):
                 return float(probability) >= 0.5
-        probability = event.get("speech_probability")
-        if isinstance(probability, int | float):
-            return float(probability) >= 0.5
 
         fmt = payload.get("format")
         audio = payload.get("audio")
@@ -925,14 +974,26 @@ class OmniDuplexSessionHandler(
             )
             return None
 
-        config = DuplexSessionConfig.from_event(event)
+        try:
+            config = DuplexSessionConfig.from_event(event)
+        except ValueError as exc:
+            error: dict[str, object] = {
+                "type": "error",
+                "error": str(exc),
+                "code": "invalid_request_error",
+            }
+            event_id = event.get("_realtime_event_id")
+            if isinstance(event_id, str) and event_id:
+                error["event_id"] = event_id
+            await send_json(error)
+            return None
         if config.idle_timeout_s == _DEFAULT_IDLE_TIMEOUT_S:
             config.idle_timeout_s = self._idle_timeout_s
-        use_native_runtime = self._uses_serving_runtime_adapter(config)
+        runtime_adapter = self._serving_runtime_adapter if self._uses_serving_runtime_adapter(config) else None
         runtime_config: dict[str, object] = {}
-        if use_native_runtime:
+        if runtime_adapter is not None:
             try:
-                runtime_config = await self._serving_runtime_adapter.prepare_runtime_config(
+                runtime_config = await runtime_adapter.prepare_runtime_config(
                     config,
                     model_config=getattr(self._chat_service, "model_config", None),
                 )
@@ -944,14 +1005,20 @@ class OmniDuplexSessionHandler(
                 return None
         session_id = event.get("session_id") if isinstance(event.get("session_id"), str) else None
         session = self._registry.create(config=config, session_id=session_id)
-        if use_native_runtime:
+        if runtime_adapter is not None:
             session.replace_capabilities(
-                self._serving_runtime_adapter.capabilities(
-                    max_sessions=self._duplex_session_config.max_sessions,
-                )
+                runtime_adapter.capabilities(max_sessions=self._duplex_session_config.max_sessions)
             )
             session.replace_runtime_config(runtime_config)
-        return _DuplexSessionHandshake(session=session)
+        session_payload = event.get("session")
+        self._resolve_server_vad_defaults(
+            session, config, session_payload if isinstance(session_payload, dict) else event
+        )
+        event_id = event.get("_realtime_event_id")
+        return _DuplexSessionHandshake(
+            session=session,
+            event_id=event_id if isinstance(event_id, str) and event_id else None,
+        )
 
     async def _resume_session_handshake(
         self,
@@ -1151,7 +1218,8 @@ class OmniDuplexSessionHandler(
         return None
 
     def _uses_serving_runtime_adapter(self, config: DuplexSessionConfig) -> bool:
-        return self._serving_runtime_adapter.is_enabled(config)
+        adapter = self._serving_runtime_adapter
+        return adapter is not None and adapter.is_enabled(config)
 
     def _runtime_session_update_error(
         self,
@@ -1161,7 +1229,7 @@ class OmniDuplexSessionHandler(
         if not self._uses_native_input_append(session):
             return None
         try:
-            self._serving_runtime_adapter.validate_client_extra_body(payload.get("extra_body"))
+            self._require_serving_runtime_adapter().validate_client_extra_body(payload.get("extra_body"))
         except ServingRuntimeConfigError as exc:
             return {
                 "type": "error",
@@ -1178,24 +1246,109 @@ class OmniDuplexSessionHandler(
     ) -> dict[str, object]:
         if not self._uses_native_input_append(session):
             return dict(session.runtime_config)
-        return self._serving_runtime_adapter.runtime_config_for_update(
+        return self._require_serving_runtime_adapter().runtime_config_for_update(
             candidate_config,
             dict(session.runtime_config),
         )
 
+    def _resolve_server_vad_defaults(
+        self,
+        session: DuplexSession,
+        config: DuplexSessionConfig,
+        payload: dict[str, object],
+    ) -> None:
+        """Resolve omitted options without overriding explicit client policies."""
+        if config.server_vad is None:
+            return
+        if config.server_vad.interrupt_response is None:
+            config.server_vad = replace(config.server_vad, interrupt_response=self._uses_native_input_append(session))
+        extra = payload.get("extra_body")
+        overlap_policy = extra.get("overlap_policy") if isinstance(extra, dict) else None
+        if not isinstance(overlap_policy, str):
+            overlap_policy = payload.get("overlap_policy")
+        config.overlap_policy = (
+            overlap_policy
+            if isinstance(overlap_policy, str)
+            else "barge_in_on_speech"
+            if config.server_vad.interrupt_response
+            else "listen_only"
+        )
+
     def _runtime_session_candidate_update_error(
         self,
         session: DuplexSession,
         candidate_config: DuplexSessionConfig,
+        *,
+        realtime_protocol: NativeRealtimeSessionProtocol | None = None,
     ) -> dict[str, object] | None:
+        if candidate_config.server_vad is not None:
+            desired_policy = "barge_in_on_speech" if candidate_config.server_vad.interrupt_response else "listen_only"
+            if candidate_config.overlap_policy != desired_policy:
+                return {
+                    "type": "error",
+                    "code": "unsupported_turn_detection",
+                    "error": f"overlap_policy conflicts with turn_detection; expected {desired_policy!r}",
+                    "param": "overlap_policy",
+                }
         if not self._uses_native_input_append(session):
-            if candidate_config.overlap_policy != DuplexOverlapPolicy.BARGE_IN_ON_SPEECH.value:
-                return None
-            return {
-                "type": "error",
-                "code": "server_vad_requires_native_duplex",
-                "error": "Realtime server_vad requires a model-native duplex runtime",
-            }
+            if (
+                candidate_config.server_vad is not None
+                and candidate_config.server_vad.min_speech_duration_ms is not None
+            ):
+                return {
+                    "type": "error",
+                    "code": "unsupported_turn_detection",
+                    "error": "server_vad.min_speech_duration_ms is only supported by model-native duplex runtimes",
+                }
+            requests_interruption = (
+                candidate_config.server_vad is not None and candidate_config.server_vad.interrupt_response
+            ) or candidate_config.overlap_policy == DuplexOverlapPolicy.BARGE_IN_ON_SPEECH.value
+            if requests_interruption:
+                return {
+                    "type": "error",
+                    "code": "server_vad_requires_native_duplex",
+                    "error": (
+                        "Realtime server_vad interruption requires a model-native duplex runtime; "
+                        "set interrupt_response=false for turn-based models"
+                    ),
+                }
+            if candidate_config.server_vad is not None and realtime_protocol is not None:
+                session_payload = candidate_config.extra_body.get("realtime_session_payload")
+                audio_error = realtime_protocol.validate_server_vad_input_audio(
+                    session_payload if isinstance(session_payload, dict) else {}
+                )
+                if audio_error is not None:
+                    return {
+                        "type": "error",
+                        "code": "unsupported_turn_detection",
+                        "error": audio_error,
+                        "param": "audio.input.format",
+                    }
+            return None
+        if candidate_config.server_vad is not None:
+            if not candidate_config.server_vad.interrupt_response:
+                return {
+                    "type": "error",
+                    "session_id": session.session_id,
+                    "code": "unsupported_turn_detection",
+                    "error": "Model-native duplex server_vad requires interrupt_response=true",
+                }
+            if candidate_config.server_vad.threshold <= SILERO_VAD_MIN_THRESHOLD:
+                return {
+                    "type": "error",
+                    "session_id": session.session_id,
+                    "code": "unsupported_turn_detection",
+                    "error": (
+                        f"Model-native duplex server_vad.threshold must be greater than {SILERO_VAD_MIN_THRESHOLD}"
+                    ),
+                }
+            if not candidate_config.server_vad.create_response:
+                return {
+                    "type": "error",
+                    "session_id": session.session_id,
+                    "code": "unsupported_turn_detection",
+                    "error": "Model-native duplex server_vad requires create_response=true",
+                }
         if (
             candidate_config.instructions != session.config.instructions
             and self._runtime_session_state(session).native_context_locked
@@ -1280,7 +1433,7 @@ class OmniDuplexSessionHandler(
         session: DuplexSession,
         committed: DuplexCommittedInput,
         *,
-        realtime_item_id: object | None = None,
+        item_id: object | None = None,
     ) -> dict[str, object]:
         payload: dict[str, object] = {
             "type": "input.committed",
@@ -1290,15 +1443,15 @@ class OmniDuplexSessionHandler(
             "history_len": len(session.history),
             "message": committed.message,
         }
-        if isinstance(realtime_item_id, str) and realtime_item_id:
-            payload["realtime_item_id"] = realtime_item_id
+        if isinstance(item_id, str) and item_id:
+            payload["item_id"] = item_id
         return payload
 
     @staticmethod
     def _commit_native_audio_input(
         session: DuplexSession,
         *,
-        realtime_item_id: object | None = None,
+        item_id: object | None = None,
         transcript: object | None = None,
         turn_id: int | None = None,
     ) -> DuplexCommittedInput:
@@ -1307,8 +1460,8 @@ class OmniDuplexSessionHandler(
             transcript=clean_transcript or None,
             turn_id=turn_id,
         )
-        if isinstance(realtime_item_id, str) and realtime_item_id:
-            session.register_history_item(realtime_item_id, committed.message)
+        if isinstance(item_id, str) and item_id:
+            session.register_history_item(item_id, committed.message)
         return committed
 
     @staticmethod
@@ -1316,7 +1469,7 @@ class OmniDuplexSessionHandler(
         session: DuplexSession,
         *,
         committed: DuplexCommittedInput | None = None,
-        realtime_item_id: object | None = None,
+        item_id: object | None = None,
         transcript: object | None = None,
     ) -> dict[str, object]:
         message = committed.message if committed is not None else None
@@ -1342,12 +1495,15 @@ class OmniDuplexSessionHandler(
         }
         if isinstance(transcript, str) and transcript:
             payload["transcript"] = transcript
-        if isinstance(realtime_item_id, str) and realtime_item_id:
-            payload["realtime_item_id"] = realtime_item_id
+        if isinstance(item_id, str) and item_id:
+            payload["item_id"] = item_id
         return payload
 
-    @staticmethod
-    def _apply_session_update(session: DuplexSession, payload: dict[str, object]) -> dict[str, object] | None:
+    def _apply_session_update(
+        self,
+        session: DuplexSession,
+        payload: dict[str, object],
+    ) -> dict[str, object] | None:
         model = payload.get("model")
         audio_config = payload.get("audio")
         audio_input = audio_config.get("input") if isinstance(audio_config, dict) else None
@@ -1385,6 +1541,14 @@ class OmniDuplexSessionHandler(
             extra_body_payload = normalize_native_duplex_key(dict(extra_body_payload))
             payload = {**payload, "extra_body": extra_body_payload}
         if isinstance(extra_body_payload, dict) and NATIVE_DUPLEX_KEY in extra_body_payload:
+            requested_native_duplex = extra_body_payload[NATIVE_DUPLEX_KEY]
+            if not isinstance(requested_native_duplex, bool):
+                return {
+                    "type": "error",
+                    "session_id": session.session_id,
+                    "code": "invalid_request_error",
+                    "error": "extra_body.native_duplex must be a boolean",
+                }
             current_native_duplex = native_duplex_opt_in(session.config.extra_body)
             if current_native_duplex is not None and extra_body_payload[NATIVE_DUPLEX_KEY] != current_native_duplex:
                 return {
@@ -1498,8 +1662,69 @@ class OmniDuplexSessionHandler(
         session.config.extra_body["realtime_session_payload"] = (
             NativeRealtimeSessionProtocol._json_safe_realtime_payload(payload)
         )
+        try:
+            turn_detection_configured, server_vad = parse_session_turn_detection(payload)
+            if turn_detection_configured:
+                vad_config = replace(session.config, server_vad=server_vad)
+                self._resolve_server_vad_defaults(session, vad_config, payload)
+                server_vad = vad_config.server_vad
+                if (
+                    not self._uses_native_input_append(session)
+                    and session.turn_detection_config_locked
+                    and (not session.config.turn_detection_configured or server_vad != session.config.server_vad)
+                ):
+                    return {
+                        "type": "error",
+                        "session_id": session.session_id,
+                        "code": "unsupported_turn_detection",
+                        "error": "turn_detection cannot be changed after the first audio append",
+                    }
+                session.config.turn_detection_configured = True
+                session.config.server_vad = server_vad
+                session.config.overlap_policy = vad_config.overlap_policy
+        except ValueError as exc:
+            return {
+                "type": "error",
+                "session_id": session.session_id,
+                "code": "invalid_request_error",
+                "error": str(exc),
+            }
         return None
 
+    async def _configure_server_vad(self, session: DuplexSession) -> None:
+        pipeline = await self._prepare_server_vad_pipeline(session, session.config)
+        self._install_server_vad_pipeline(session, pipeline)
+
+    def _install_server_vad_pipeline(
+        self,
+        session: DuplexSession,
+        pipeline: ServerVADPipeline | None,
+    ) -> None:
+        existing = self._server_vad_pipelines.pop(session.session_id, None)
+        if existing is not None:
+            existing.reset()
+            self._realtime_vad_metrics.session_finished()
+        if pipeline is None:
+            return
+        self._server_vad_pipelines[session.session_id] = pipeline
+        self._realtime_vad_metrics.session_started()
+
+    async def _prepare_server_vad_pipeline(
+        self,
+        session: DuplexSession,
+        config: DuplexSessionConfig,
+    ) -> ServerVADPipeline | None:
+        if config.server_vad is None:
+            return None
+        backend = await asyncio.to_thread(self._server_vad_backend_provider.get)
+        pipeline_config = config.server_vad
+        if self._uses_native_input_append(session) and pipeline_config.min_speech_duration_ms is None:
+            pipeline_config = replace(
+                pipeline_config,
+                min_speech_duration_ms=SILERO_VAD_DEFAULT_MIN_SPEECH_DURATION_MS,
+            )
+        return ServerVADPipeline(backend, pipeline_config)
+
     def _apply_response_create_options(
         self,
         session: DuplexSession,
@@ -1580,7 +1805,7 @@ class OmniDuplexSessionHandler(
                 response_extra.update(
                     (key, value)
                     for key, value in extra_body.items()
-                    if key not in self._serving_runtime_adapter.private_runtime_config_keys
+                    if key not in self._require_serving_runtime_adapter().private_runtime_config_keys
                 )
             else:
                 response_extra.update(extra_body)
@@ -1841,7 +2066,8 @@ class OmniDuplexSessionHandler(
             # Epoch-scoped request ids prevent state reuse, but explicitly
             # release projector/parser cursors so cancelled epochs do not
             # accumulate until the whole session closes.
-            self._serving_runtime_adapter.data_plane.close_stream(old_request_id)
+            if self._serving_runtime_adapter is not None:
+                self._serving_runtime_adapter.data_plane.close_stream(old_request_id)
             await self._abort_request_background(
                 session,
                 old_request_id,

vllm_omni/entrypoints/duplex/session_runner.py

@@ -4,17 +4,23 @@
 from __future__ import annotations
 
 import asyncio
+import binascii
 import json
 import uuid
 from contextlib import suppress
 from copy import deepcopy
 
+import numpy as np
+import pybase64 as base64
 from fastapi import WebSocket, WebSocketDisconnect
 from vllm.logger import init_logger
 
 from vllm_omni.engine.duplex.lease import DuplexLeaseActivity
 from vllm_omni.engine.duplex.messages import DuplexFence
-from vllm_omni.entrypoints.duplex.audio import convert_input_audio_with_rate
+from vllm_omni.entrypoints.duplex.audio import (
+    convert_input_audio_with_rate,
+    validate_input_sample_rate_hz,
+)
 from vllm_omni.entrypoints.duplex.commit_policy import (
     CommitAction,
     CommitSnapshot,
@@ -36,6 +42,7 @@ from vllm_omni.entrypoints.duplex.runtime_adapter import (
     ServingRuntimeSessionState,
     payload_turn_id,
 )
+from vllm_omni.entrypoints.duplex.server_vad import ServerVADPipeline
 from vllm_omni.entrypoints.duplex.session_attachment import (
     DuplexJournalOverflowError,
 )
@@ -138,7 +145,7 @@ class DuplexSessionRunnerMixin:
                 await actor.send_json(raw_payload)
 
             realtime_protocol.bind_sender(send_realtime_raw)
-        native: ServingRuntimeSessionState = self._serving_runtime_adapter.create_session_state()
+        native: ServingRuntimeSessionState = self._create_runtime_session_state()
 
         def begin_close(reason: str) -> None:
             actor.closing = True
@@ -291,6 +298,120 @@ class DuplexSessionRunnerMixin:
                 return True
             return False
 
+        def start_chat_response() -> None:
+            assert session is not None
+
+            async def run_and_notify() -> None:
+                task = asyncio.current_task()
+                try:
+                    await self._run_response(session, emit_event)
+                finally:
+                    await actor.enqueue_event(
+                        {
+                            "type": "__chat_response_finished__",
+                            "_response_task": task,
+                        }
+                    )
+
+            actor.active_response_task = asyncio.create_task(run_and_notify())
+
+        def activate_waiting_server_vad_turn() -> None:
+            assert session is not None
+            pending_turn = session.activate_server_vad_turn()
+            if pending_turn is not None and pending_turn.create_response:
+                start_chat_response()
+
+        def correlate_realtime_error(
+            payload: dict[str, object],
+            event_id: object,
+        ) -> dict[str, object]:
+            if isinstance(event_id, str) and event_id:
+                payload["event_id"] = event_id
+            return payload
+
+        async def reject_and_clear_server_vad_input(
+            pipeline: ServerVADPipeline,
+            *,
+            code: str,
+            error: str,
+            metric_reason: str,
+            event_id: object,
+        ) -> None:
+            assert session is not None
+            session.cancel_pending_input()
+            pipeline.reset()
+            self._realtime_vad_metrics.error(metric_reason)
+            await emit_event(
+                correlate_realtime_error(
+                    {
+                        "type": "error",
+                        "session_id": session.session_id,
+                        "code": code,
+                        "error": error,
+                    },
+                    event_id,
+                )
+            )
+            await emit_event(
+                {
+                    "type": "input_audio_buffer.cleared",
+                    "session_id": session.session_id,
+                }
+            )
+
+        async def commit_server_vad_turn(
+            *,
+            item_id: str,
+            create_response: bool,
+            retain_input_bytes: int,
+            event_id: object = None,
+        ) -> None:
+            assert session is not None
+            # Retain ownership until the actor consumes the terminal event so
+            # queued audio cannot overtake pending-turn activation.
+            active_response = actor.active_response_task is not None
+            if active_response and session.pending_server_vad_turn is not None:
+                released = session.discard_uncommitted_server_vad_utterance()
+                session.release_input_bytes(released)
+                self._realtime_vad_metrics.error("pending_turn_backpressure")
+                await emit_event(
+                    correlate_realtime_error(
+                        {
+                            "type": "error",
+                            "session_id": session.session_id,
+                            "code": "input_backpressure",
+                            "error": "Only one committed Server VAD turn may wait for the active response",
+                        },
+                        event_id,
+                    )
+                )
+                return
+            if not session.stage_server_vad_audio_for_commit():
+                return
+            committed = session.commit_user_input(
+                append_history=not active_response,
+                retain_input_bytes=retain_input_bytes,
+            )
+            if committed is None:
+                return
+            if active_response:
+                session.stage_server_vad_turn(
+                    committed,
+                    create_response=create_response,
+                    item_id=item_id,
+                )
+            else:
+                session.register_history_item(item_id, committed.message)
+            await emit_event(
+                self._input_committed_payload(
+                    session,
+                    committed,
+                    item_id=item_id,
+                )
+            )
+            if create_response and not active_response:
+                start_chat_response()
+
         def clear_completed_pending_silence() -> None:
             task = native.pending_silence_task
             if task is not None and task.done():
@@ -679,11 +800,18 @@ class DuplexSessionRunnerMixin:
             if handshake is None:
                 return
             session = handshake.session
-            if initial_update_error := self._runtime_session_candidate_update_error(session, session.config):
-                await emit_event(initial_update_error)
+            if initial_update_error := self._runtime_session_candidate_update_error(
+                session, session.config, realtime_protocol=realtime_protocol
+            ):
+                await emit_event(
+                    correlate_realtime_error(
+                        initial_update_error,
+                        handshake.event_id,
+                    )
+                )
                 return
             if handshake.resumed:
-                native = self._serving_runtime_adapter.session_states[session.session_id]
+                native = self._serving_session_states[session.session_id]
                 actor.tasks = self._session_tasks[session.session_id]
                 persisted_protocol = self._realtime_protocols.get(session.session_id)
                 if persisted_protocol is None:
@@ -697,18 +825,36 @@ class DuplexSessionRunnerMixin:
                 self._ensure_lifecycle_listener()
                 reader_task = asyncio.create_task(read_event_loop(), name="duplex-session-reader")
             else:
-                self._serving_runtime_adapter.session_states[session.session_id] = native
+                self._serving_session_states[session.session_id] = native
                 self._session_tasks[session.session_id] = actor.tasks
                 if realtime_protocol is not None:
                     self._realtime_protocols[session.session_id] = realtime_protocol
             native.silence_continuation_scheduler = schedule_native_silence_continuation
             if realtime_protocol is not None:
+                realtime_protocol.bind_native_input_append(self._uses_native_input_append(session))
                 session.config.playback_commit_policy = DuplexPlaybackCommitPolicy.ACK_ONLY.value
             if not handshake.resumed:
                 open_result = await self._open_runtime_session(session, emit_event)
                 if open_result is False:
                     return
                 runtime_opened = True
+                try:
+                    await self._configure_server_vad(session)
+                except Exception as exc:
+                    logger.exception("Server VAD initialization failed: %s", exc)
+                    self._realtime_vad_metrics.error("initialization")
+                    await emit_event(
+                        correlate_realtime_error(
+                            {
+                                "type": "error",
+                                "session_id": session.session_id,
+                                "code": "server_vad_initialization_failed",
+                                "error": str(exc),
+                            },
+                            handshake.event_id,
+                        )
+                    )
+                    return
                 created_attachment = await self._attachment_registry.create(
                     session.session_id,
                     incarnation=session.incarnation,
@@ -734,7 +880,7 @@ class DuplexSessionRunnerMixin:
                 if isinstance(open_result, dict):
                     created_payload["runtime_control"] = self._redact_runtime_control_result(open_result)
                 if realtime_protocol is not None:
-                    realtime_protocol.commit_realtime_turn_detection_update()
+                    realtime_protocol.commit_realtime_turn_detection_update(session.config.server_vad)
                 await emit_event(created_payload)
                 resume_credential_delivered = session.capabilities.supports_session_resume
                 reader_task = asyncio.create_task(read_event_loop(), name="duplex-session-reader")
@@ -746,6 +892,14 @@ class DuplexSessionRunnerMixin:
                 if event_type == "__replaced_attachment__":
                     return
 
+                if event_type == "__chat_response_finished__":
+                    completed_task = event.get("_response_task")
+                    if completed_task is not actor.active_response_task:
+                        continue
+                    actor.active_response_task = None
+                    activate_waiting_server_vad_turn()
+                    continue
+
                 if event_type == "__timeout__":
                     begin_close("timeout")
                     native.audio_buffer.clear()
@@ -896,6 +1050,9 @@ class DuplexSessionRunnerMixin:
                 if event_type == "input_audio_buffer.clear":
                     native.audio_buffer.clear()
                     session.release_all_input_bytes()
+                    server_vad_pipeline = self._server_vad_pipelines.get(session.session_id)
+                    if server_vad_pipeline is not None:
+                        server_vad_pipeline.reset()
                     native.input_since_commit = False
                     native.speech_since_commit = False
                     native.clear_committed_audio()
@@ -932,10 +1089,8 @@ class DuplexSessionRunnerMixin:
                     if event_type == "response.cancel":
                         requested_response_id = event.get("response_id")
                         has_active_response_work = native_response_in_progress()
-                        if (
-                            isinstance(requested_response_id, str)
-                            and session.active_response_id is not None
-                            and requested_response_id != session.active_response_id
+                        if isinstance(requested_response_id, str) and (
+                            session.active_response_id is None or requested_response_id != session.active_response_id
                         ):
                             await emit_event(
                                 {
@@ -946,6 +1101,7 @@ class DuplexSessionRunnerMixin:
                                 }
                             )
                             continue
+                    if event_type == "response.cancel":
                         if not has_active_response_work:
                             if realtime_protocol is not None and isinstance(requested_response_id, str):
                                 continue
@@ -1056,6 +1212,7 @@ class DuplexSessionRunnerMixin:
                             }
                         )
                         actor.active_response_task = None
+                        activate_waiting_server_vad_turn()
                         continue
                     if not cancelled:
                         await self._cancel_pending_input(session, emit_event, reason="barge_in")
@@ -1077,6 +1234,7 @@ class DuplexSessionRunnerMixin:
                     ):
                         continue
                     actor.active_response_task = None
+                    activate_waiting_server_vad_turn()
                     continue
 
                 if event_type == "turn.signal":
@@ -1119,6 +1277,27 @@ class DuplexSessionRunnerMixin:
                                 )
                                 reject_update()
                                 continue
+                            if (
+                                realtime_protocol is not None
+                                and not self._uses_native_input_append(session)
+                                and session.config.server_vad is not None
+                                and session.turn_detection_config_locked
+                                and realtime_protocol.pending_realtime_input_audio_config_changed()
+                            ):
+                                await emit_update_event(
+                                    {
+                                        "type": "error",
+                                        "session_id": session.session_id,
+                                        "code": "unsupported_audio_format",
+                                        "error": (
+                                            "session.update cannot change Server VAD input format or sample rate "
+                                            "after the first audio append"
+                                        ),
+                                        "param": "audio.input.format",
+                                    }
+                                )
+                                reject_update()
+                                continue
                             runtime_update_error = self._runtime_session_update_error(session, payload)
                             if runtime_update_error is not None:
                                 await emit_update_event(runtime_update_error)
@@ -1131,6 +1310,7 @@ class DuplexSessionRunnerMixin:
                                 update_error = self._apply_session_update(session, payload)
                             finally:
                                 session.replace_config(previous_config)
+                            server_vad_changed = candidate_config.server_vad != previous_config.server_vad
                             if update_error is not None:
                                 await emit_update_event(update_error)
                                 reject_update()
@@ -1138,6 +1318,7 @@ class DuplexSessionRunnerMixin:
                             runtime_update_error = self._runtime_session_candidate_update_error(
                                 session,
                                 candidate_config,
+                                realtime_protocol=realtime_protocol,
                             )
                             if runtime_update_error is not None:
                                 await emit_update_event(runtime_update_error)
@@ -1159,19 +1340,46 @@ class DuplexSessionRunnerMixin:
                                 )
                                 reject_update()
                                 continue
-                            if not await self._signal_runtime_session(
+                            candidate_server_vad_pipeline = None
+                            if server_vad_changed:
+                                try:
+                                    candidate_server_vad_pipeline = await self._prepare_server_vad_pipeline(
+                                        session,
+                                        candidate_config,
+                                    )
+                                except Exception as exc:
+                                    logger.exception("Server VAD session update failed: %s", exc)
+                                    self._realtime_vad_metrics.error("initialization")
+                                    await emit_update_event(
+                                        {
+                                            "type": "error",
+                                            "session_id": session.session_id,
+                                            "code": "server_vad_initialization_failed",
+                                            "error": str(exc),
+                                        }
+                                    )
+                                    reject_update()
+                                    continue
+                            runtime_updated = await self._signal_runtime_session(
                                 session,
                                 turn_event,
                                 emit_update_event,
                                 session_config=candidate_config.as_dict(),
                                 runtime_config=candidate_runtime_config,
-                            ):
+                            )
+                            if not runtime_updated:
                                 reject_update()
                                 continue
                             session.replace_config(candidate_config)
                             session.replace_runtime_config(candidate_runtime_config)
+                            if server_vad_changed:
+                                self._install_server_vad_pipeline(
+                                    session,
+                                    candidate_server_vad_pipeline,
+                                )
                             if realtime_protocol is not None:
-                                realtime_protocol.commit_realtime_turn_detection_update()
+                                realtime_protocol.bind_native_input_append(self._uses_native_input_append(session))
+                                realtime_protocol.commit_realtime_turn_detection_update(session.config.server_vad)
                             await emit_update_event(
                                 {
                                     "type": "session.updated",
@@ -1229,7 +1437,10 @@ class DuplexSessionRunnerMixin:
                                     expected_model_turn_id=session.turn_id,
                                 )
                                 continue
-                            message = self._realtime_item_to_history_message(item)
+                            history_item = payload.get("history_item") if isinstance(payload, dict) else None
+                            if not isinstance(history_item, dict):
+                                history_item = item
+                            message = self._realtime_item_to_history_message(history_item)
                             item_id = item.get("id") if isinstance(item, dict) else None
                             if message is not None:
                                 session.append_history_message(message)
@@ -1346,24 +1557,30 @@ class DuplexSessionRunnerMixin:
                     default_sample_rate_hz = 16000
                     sr_raw = event.get("sample_rate_hz") or event.get("sample_rate")
                     sample_rate_hz = sr_raw if isinstance(sr_raw, int | float) else default_sample_rate_hz
-                    try:
-                        audio, fmt, sample_rate_hz = convert_input_audio_with_rate(
-                            audio,
-                            fmt,
-                            sample_rate_hz=sample_rate_hz,
-                        )
-                    except ValueError as exc:
-                        await emit_event({"type": "error", "error": str(exc), "code": "bad_event"})
-                        continue
-                    if isinstance(fmt, str) and fmt.lower() in {"pcm16", "pcm_s16le", "s16le"}:
-                        await emit_event(
-                            {
-                                "type": "error",
-                                "error": "input_audio_buffer.append pcm16 audio could not be decoded",
-                                "code": "bad_audio",
-                            }
-                        )
-                        continue
+                    native_input = self._uses_native_input_append(session)
+                    server_vad_config = session.config.server_vad
+                    turn_based_server_vad = server_vad_config is not None and not native_input
+                    if not turn_based_server_vad:
+                        try:
+                            audio, fmt, sample_rate_hz = convert_input_audio_with_rate(
+                                audio,
+                                fmt,
+                                sample_rate_hz=sample_rate_hz,
+                            )
+                        except ValueError as exc:
+                            await emit_event({"type": "error", "error": str(exc), "code": "bad_event"})
+                            continue
+                        if isinstance(fmt, str) and fmt.lower() in {"pcm16", "pcm_s16le", "s16le"}:
+                            await emit_event(
+                                {
+                                    "type": "error",
+                                    "error": "input_audio_buffer.append pcm16 audio could not be decoded",
+                                    "code": "bad_audio",
+                                }
+                            )
+                            continue
+                        if server_vad_config is None:
+                            session.lock_turn_detection_config()
                     force_listen = bool(event.get("force_listen", False))
                     payload = {
                         "type": "audio",
@@ -1377,11 +1594,186 @@ class DuplexSessionRunnerMixin:
                         frames = [frame for frame in video_frames if isinstance(frame, str) and frame]
                         if frames:
                             payload["video_frames"] = frames
+                    if server_vad_config is not None:
+                        realtime_event_id = event.get("_realtime_event_id")
+                        pipeline = self._server_vad_pipelines.get(session.session_id)
+                        if pipeline is None:
+                            await emit_event(
+                                correlate_realtime_error(
+                                    {
+                                        "type": "error",
+                                        "session_id": session.session_id,
+                                        "code": "internal_error",
+                                        "error": "server_vad is configured but its detector pipeline is unavailable",
+                                    },
+                                    realtime_event_id,
+                                )
+                            )
+                            continue
+                        speech_was_active = pipeline.speech_active
+                        reserved_bytes = 0
+                        previous_scratch_bytes = pipeline.scratch_bytes
+                        try:
+                            if native_input:
+                                if fmt != "pcm_f32le" or sample_rate_hz != pipeline.sample_rate_hz:
+                                    raise ValueError("native server_vad input must be mono PCM float32 at 16000 Hz")
+                                raw_audio = base64.b64decode(audio, validate=True)
+                                if len(raw_audio) % np.dtype("<f4").itemsize:
+                                    raise ValueError("native server_vad input contains an incomplete float32 sample")
+                                normalized_audio = np.frombuffer(raw_audio, dtype="<f4")
+                                accepted_audio = normalized_audio.size > 0
+                            else:
+                                if fmt != "pcm16":
+                                    raise ValueError("server_vad input must be mono PCM16")
+                                source_sample_rate_hz = validate_input_sample_rate_hz(sample_rate_hz)
+                                if source_sample_rate_hz not in {16_000, 24_000}:
+                                    raise ValueError("server_vad PCM16 input sample rate must be 16000 or 24000 Hz")
+                                if (
+                                    pipeline.source_sample_rate_hz is not None
+                                    and source_sample_rate_hz != pipeline.source_sample_rate_hz
+                                ):
+                                    raise ValueError(
+                                        "server_vad input sample rate cannot change within a continuous audio stream"
+                                    )
+                                try:
+                                    raw_audio = base64.b64decode(audio, validate=True)
+                                except (binascii.Error, ValueError) as exc:
+                                    raise ValueError(
+                                        "server_vad input audio must be valid base64-encoded PCM16"
+                                    ) from exc
+                                if len(raw_audio) % np.dtype("<i2").itemsize:
+                                    raise ValueError("server_vad PCM16 input contains an incomplete sample")
+                                source_samples = len(raw_audio) // np.dtype("<i2").itemsize
+                                accepted_audio = source_samples > 0
+                        except (binascii.Error, ValueError) as exc:
+                            await emit_event(
+                                correlate_realtime_error(
+                                    {
+                                        "type": "error",
+                                        "session_id": session.session_id,
+                                        "code": "bad_audio",
+                                        "error": str(exc),
+                                        "param": "audio",
+                                    },
+                                    realtime_event_id,
+                                )
+                            )
+                            continue
+                        if not native_input:
+                            normalized_sample_upper_bound = (
+                                source_samples
+                                if source_sample_rate_hz == pipeline.sample_rate_hz
+                                else ((source_samples + 2) // 3) * 2
+                            )
+                            reserved_bytes = normalized_sample_upper_bound * np.dtype(np.float32).itemsize
+                            if not session.reserve_input_bytes(
+                                reserved_bytes,
+                                limit=self._duplex_session_config.max_pending_input_bytes_per_session,
+                            ):
+                                await reject_and_clear_server_vad_input(
+                                    pipeline,
+                                    code="input_backpressure",
+                                    error="Server VAD input exceeds the per-session input limit",
+                                    metric_reason="input_backpressure",
+                                    event_id=realtime_event_id,
+                                )
+                                continue
+                        try:
+                            vad_batch = (
+                                await pipeline.push(normalized_audio)
+                                if native_input
+                                else await pipeline.push_pcm16(
+                                    raw_audio,
+                                    source_sample_rate_hz=source_sample_rate_hz,
+                                )
+                            )
+                        except Exception as exc:
+                            logger.exception("Server VAD inference failed: %s", exc)
+                            await reject_and_clear_server_vad_input(
+                                pipeline,
+                                code="server_vad_inference_failed",
+                                error=str(exc),
+                                metric_reason="inference",
+                                event_id=realtime_event_id,
+                            )
+                            continue
+                        if not native_input:
+                            retained_delta = (
+                                sum(frame.samples.nbytes for frame in vad_batch.frames)
+                                + pipeline.scratch_bytes
+                                - previous_scratch_bytes
+                            )
+                            session.release_input_bytes(max(0, reserved_bytes - retained_delta))
+                        self._realtime_vad_metrics.observe_inference(vad_batch.inference_ms)
+                        prefix_samples = round(server_vad_config.prefix_padding_ms * pipeline.sample_rate_hz / 1000)
+                        speech_started = False
+                        speech_stopped = False
+                        for frame_index, vad_frame in enumerate(vad_batch.frames):
+                            decision = vad_frame.decision
+                            if not native_input:
+                                released_bytes = session.append_server_vad_frame(
+                                    vad_frame.samples,
+                                    speech_started=decision.speech_started,
+                                    speech_stopped=decision.speech_stopped,
+                                    prefix_samples=prefix_samples,
+                                )
+                                session.release_input_bytes(released_bytes)
+                            if decision.speech_started:
+                                speech_started = True
+                                item_id = f"item_{uuid.uuid4().hex}"
+                                session.begin_server_vad_speech(item_id)
+                                await emit_event(
+                                    {
+                                        "type": "input_audio_buffer.speech_started",
+                                        "session_id": session.session_id,
+                                        "item_id": item_id,
+                                        "audio_start_ms": decision.audio_start_ms or 0,
+                                    }
+                                )
+                            if decision.speech_stopped:
+                                speech_stopped = True
+                                item_id = session.finish_server_vad_speech() or f"item_{uuid.uuid4().hex}"
+                                await emit_event(
+                                    {
+                                        "type": "input_audio_buffer.speech_stopped",
+                                        "session_id": session.session_id,
+                                        "item_id": item_id,
+                                        "audio_end_ms": decision.audio_end_ms or 0,
+                                    }
+                                )
+                                if decision.endpoint_delay_ms is not None:
+                                    self._realtime_vad_metrics.observe_endpoint_delay(decision.endpoint_delay_ms)
+                                if not native_input:
+                                    remaining_frame_bytes = sum(
+                                        frame.samples.nbytes for frame in vad_batch.frames[frame_index + 1 :]
+                                    )
+                                    await commit_server_vad_turn(
+                                        item_id=item_id,
+                                        create_response=server_vad_config.create_response,
+                                        retain_input_bytes=remaining_frame_bytes + pipeline.scratch_bytes,
+                                        event_id=realtime_event_id,
+                                    )
+                        if accepted_audio:
+                            session.lock_turn_detection_config()
+                        if not native_input:
+                            continue
+                        speech_active = pipeline.speech_active
+                        contained_speech = speech_was_active or speech_started or speech_stopped or speech_active
+                        payload["is_speech"] = contained_speech
+                        payload["vad"] = {
+                            "backend": "silero",
+                            "is_speech": contained_speech,
+                            "speech_active": speech_active,
+                            "speech_started": speech_started,
+                            "speech_stopped": speech_stopped,
+                        }
+                        if speech_active:
+                            payload["force_listen"] = True
                     # Speech/silence tag for the Stage0 turn-ended latch.
                     payload["is_speech"] = self._input_looks_like_speech(event, payload, session=session)
                     defer_native_append = False
                     buffer_overlap_audio = True
-                    if self._uses_native_input_append(session):
+                    if native_input:
                         mark_pending_silence_superseded()
                         overlap_active = native_response_in_progress() and (
                             not self._session_auto_responds(session)
@@ -1570,7 +1962,7 @@ class DuplexSessionRunnerMixin:
                         payload = pcm_reservation.payload
                     else:
                         session.append_audio(audio, fmt=fmt, sample_rate_hz=sample_rate_hz)
-                    if self._uses_native_input_append(session):
+                    if native_input:
                         await start_native_append(
                             payload,
                             final=False,
@@ -1582,11 +1974,23 @@ class DuplexSessionRunnerMixin:
                     continue
 
                 if event_type in {"input.commit", "input_audio_buffer.commit", "response.create"}:
-                    realtime_item_id = event.get("realtime_item_id")
+                    if (
+                        event_type == "response.create"
+                        and session.config.server_vad is not None
+                        and actor.active_response_task is not None
+                    ):
+                        await emit_event(
+                            {
+                                "type": "error",
+                                "session_id": session.session_id,
+                                "code": "response_already_active",
+                                "error": "response.create must wait for the active response to complete",
+                            }
+                        )
+                        continue
+                    item_id = event.get("item_id")
                     realtime_validated_audio_commit = (
-                        event_type == "input_audio_buffer.commit"
-                        and isinstance(realtime_item_id, str)
-                        and bool(realtime_item_id)
+                        event_type == "input_audio_buffer.commit" and isinstance(item_id, str) and bool(item_id)
                     )
                     if (
                         self._uses_native_input_append(session)
@@ -1604,6 +2008,7 @@ class DuplexSessionRunnerMixin:
                             {
                                 "type": "input.committed",
                                 "session_id": session.session_id,
+                                "item_id": item_id,
                                 "turn_id": session.turn_id,
                                 "epoch": session.epoch,
                                 "empty": True,
@@ -1699,6 +2104,7 @@ class DuplexSessionRunnerMixin:
                                     {
                                         "type": "input.committed",
                                         "session_id": session.session_id,
+                                        "item_id": item_id,
                                         "turn_id": session.turn_id,
                                         "epoch": session.epoch,
                                         "empty": True,
@@ -1736,13 +2142,13 @@ class DuplexSessionRunnerMixin:
                                 native.speech_since_commit = False
                                 committed = self._commit_native_audio_input(
                                     session,
-                                    realtime_item_id=event.get("realtime_item_id"),
+                                    item_id=event.get("item_id"),
                                     transcript=event.get("transcript"),
                                 )
                                 committed_payload = self._native_audio_committed_payload(
                                     session,
                                     committed=committed,
-                                    realtime_item_id=event.get("realtime_item_id"),
+                                    item_id=event.get("item_id"),
                                     transcript=event.get("transcript"),
                                 )
                                 committed_payload["overlap_deferred"] = True
@@ -1777,7 +2183,7 @@ class DuplexSessionRunnerMixin:
                             data_plane_turn_id = session.turn_id
                             committed = self._commit_native_audio_input(
                                 session,
-                                realtime_item_id=event.get("realtime_item_id"),
+                                item_id=event.get("item_id"),
                                 transcript=event.get("transcript"),
                                 turn_id=data_plane_turn_id,
                             )
@@ -1785,7 +2191,7 @@ class DuplexSessionRunnerMixin:
                                 self._native_audio_committed_payload(
                                     session,
                                     committed=committed,
-                                    realtime_item_id=event.get("realtime_item_id"),
+                                    item_id=event.get("item_id"),
                                     transcript=event.get("transcript"),
                                 )
                             )
@@ -1883,14 +2289,14 @@ class DuplexSessionRunnerMixin:
                             native.input_since_commit = False
                             committed = self._commit_native_audio_input(
                                 session,
-                                realtime_item_id=event.get("realtime_item_id"),
+                                item_id=event.get("item_id"),
                                 transcript=event.get("transcript"),
                             )
                             await emit_event(
                                 self._native_audio_committed_payload(
                                     session,
                                     committed=committed,
-                                    realtime_item_id=event.get("realtime_item_id"),
+                                    item_id=event.get("item_id"),
                                     transcript=event.get("transcript"),
                                 )
                             )
@@ -1949,7 +2355,7 @@ class DuplexSessionRunnerMixin:
                             committed = (
                                 self._commit_native_audio_input(
                                     session,
-                                    realtime_item_id=event.get("realtime_item_id"),
+                                    item_id=event.get("item_id"),
                                     transcript=event.get("transcript"),
                                 )
                                 if native_had_uncommitted_audio
@@ -1959,7 +2365,7 @@ class DuplexSessionRunnerMixin:
                                 self._native_audio_committed_payload(
                                     session,
                                     committed=committed,
-                                    realtime_item_id=event.get("realtime_item_id"),
+                                    item_id=event.get("item_id"),
                                     transcript=event.get("transcript"),
                                 )
                             )
@@ -1968,19 +2374,20 @@ class DuplexSessionRunnerMixin:
                                 {
                                     "type": "input.committed",
                                     "session_id": session.session_id,
+                                    "item_id": item_id,
                                     "empty": True,
                                 }
                             )
                         continue
                     if committed is not None:
-                        realtime_item_id = event.get("realtime_item_id")
-                        if isinstance(realtime_item_id, str):
-                            session.register_history_item(realtime_item_id, committed.message)
+                        item_id = event.get("item_id")
+                        if isinstance(item_id, str):
+                            session.register_history_item(item_id, committed.message)
                         await emit_event(
                             self._input_committed_payload(
                                 session,
                                 committed,
-                                realtime_item_id=realtime_item_id,
+                                item_id=item_id,
                             )
                         )
                     if not should_create_response:
@@ -1992,7 +2399,10 @@ class DuplexSessionRunnerMixin:
                             emit_event,
                             reason="new_response",
                         )
-                    actor.active_response_task = asyncio.create_task(self._run_response(session, emit_event))
+                    if session.config.server_vad is not None and not self._uses_native_input_append(session):
+                        start_chat_response()
+                    else:
+                        actor.active_response_task = asyncio.create_task(self._run_response(session, emit_event))
                     continue
 
                 await emit_event(

vllm_omni/entrypoints/duplex/vad.py

@@ -1,181 +0,0 @@
-# SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
-
-from __future__ import annotations
-
-import binascii
-from collections.abc import Callable
-from dataclasses import dataclass
-
-import numpy as np
-import pybase64 as base64
-
-SILERO_VAD_MIN_THRESHOLD = 0.15
-
-
-class ServerVADUnavailableError(RuntimeError):
-    pass
-
-
-@dataclass(frozen=True, slots=True)
-class SileroVADConfig:
-    threshold: float = 0.5
-    prefix_padding_ms: int = 300
-    silence_duration_ms: int = 500
-    min_speech_duration_ms: int = 96
-
-
-@dataclass(frozen=True, slots=True)
-class StreamingVADResult:
-    is_speech: bool
-    speech_active: bool
-    speech_started: bool = False
-    speech_stopped: bool = False
-    speech_probability: float = 0.0
-    speech_start_ms: int | None = None
-    speech_end_ms: int | None = None
-
-
-class SileroStreamingVAD:
-    _SAMPLE_RATE_HZ = 16_000
-    _WINDOW_SAMPLES = 512
-
-    def __init__(
-        self,
-        config: SileroVADConfig,
-        *,
-        frame_scorer: Callable[[np.ndarray], float] | None = None,
-    ) -> None:
-        self.config = config
-        self._frame_scorer = frame_scorer
-        self._model: object | None = None
-        self._clear_stream_state()
-
-    def _clear_stream_state(self) -> None:
-        self._pending = np.empty(0, dtype=np.float32)
-        self._speech_active = False
-        self._candidate_samples = 0
-        self._candidate_start_sample: int | None = None
-        self._silence_samples = 0
-        self._processed_samples = 0
-        self._resample_rate_hz: int | None = None
-        self._resample_remainder = 0
-
-    def reset(self) -> None:
-        self._clear_stream_state()
-        reset_states = getattr(self._model, "reset_states", None)
-        if callable(reset_states):
-            reset_states()
-
-    def process_base64(
-        self,
-        audio: object,
-        *,
-        fmt: object,
-        sample_rate_hz: object,
-    ) -> StreamingVADResult:
-        if fmt != "pcm_f32le" or not isinstance(audio, str):
-            raise ValueError("Silero server VAD requires decoded pcm_f32le audio")
-        try:
-            raw = base64.b64decode(audio, validate=True)
-        except (binascii.Error, ValueError) as exc:
-            raise ValueError("Silero server VAD received invalid base64 audio") from exc
-        if len(raw) % 4:
-            raise ValueError("Silero server VAD received an incomplete pcm_f32le frame")
-        samples = np.frombuffer(raw, dtype="<f4").astype(np.float32, copy=False)
-        rate = int(sample_rate_hz) if isinstance(sample_rate_hz, int | float) else self._SAMPLE_RATE_HZ
-        if rate <= 0:
-            raise ValueError("Silero server VAD requires a positive sample rate")
-        if rate != self._SAMPLE_RATE_HZ:
-            if self._resample_rate_hz != rate:
-                self._resample_rate_hz = rate
-                self._resample_remainder = 0
-            numerator = samples.size * self._SAMPLE_RATE_HZ + self._resample_remainder
-            target_size, self._resample_remainder = divmod(numerator, rate)
-            if target_size == 0:
-                samples = np.empty(0, dtype=np.float32)
-            elif samples.size == 1:
-                samples = np.full(target_size, samples[0], dtype=np.float32)
-            else:
-                source_x = np.linspace(0.0, 1.0, num=samples.size, endpoint=True)
-                target_x = np.linspace(0.0, 1.0, num=target_size, endpoint=True)
-                samples = np.interp(target_x, source_x, samples).astype(np.float32)
-        else:
-            self._resample_rate_hz = None
-            self._resample_remainder = 0
-        return self.process(samples)
-
-    def process(self, samples: np.ndarray) -> StreamingVADResult:
-        samples = np.asarray(samples, dtype=np.float32).reshape(-1)
-        if samples.size:
-            self._pending = np.concatenate((self._pending, samples))
-
-        started = False
-        stopped = False
-        start_ms: int | None = None
-        end_ms: int | None = None
-        max_probability = 0.0
-        contained_speech = self._speech_active
-        min_speech_samples = self.config.min_speech_duration_ms * 16
-        min_silence_samples = self.config.silence_duration_ms * 16
-        negative_threshold = max(0.0, self.config.threshold - SILERO_VAD_MIN_THRESHOLD)
-
-        while self._pending.size >= self._WINDOW_SAMPLES:
-            frame = self._pending[: self._WINDOW_SAMPLES]
-            self._pending = self._pending[self._WINDOW_SAMPLES :]
-            frame_start = self._processed_samples
-            self._processed_samples += self._WINDOW_SAMPLES
-            probability = min(1.0, max(0.0, float(self._score_frame(frame))))
-            max_probability = max(max_probability, probability)
-
-            if self._speech_active:
-                contained_speech = True
-                if probability < negative_threshold:
-                    self._silence_samples += self._WINDOW_SAMPLES
-                    if self._silence_samples >= min_silence_samples:
-                        speech_end_sample = self._processed_samples - self._silence_samples
-                        self._speech_active = False
-                        self._silence_samples = 0
-                        stopped = True
-                        end_ms = max(0, int(round(speech_end_sample / 16)))
-                else:
-                    self._silence_samples = 0
-                continue
-
-            if probability >= self.config.threshold:
-                if self._candidate_samples == 0:
-                    self._candidate_start_sample = frame_start
-                self._candidate_samples += self._WINDOW_SAMPLES
-                if self._candidate_samples >= min_speech_samples:
-                    candidate_start = self._candidate_start_sample or 0
-                    prefix_samples = self.config.prefix_padding_ms * 16
-                    self._speech_active = True
-                    self._candidate_samples = self._silence_samples = 0
-                    self._candidate_start_sample = None
-                    contained_speech = True
-                    started = True
-                    start_ms = max(0, int(round((candidate_start - prefix_samples) / 16)))
-            else:
-                self._candidate_samples = 0
-                self._candidate_start_sample = None
-
-        return StreamingVADResult(
-            contained_speech, self._speech_active, started, stopped, max_probability, start_ms, end_ms
-        )
-
-    def _score_frame(self, frame: np.ndarray) -> float:
-        if self._frame_scorer is not None:
-            return float(self._frame_scorer(frame))
-        if self._model is None:
-            try:
-                import torch
-                from silero_vad import load_silero_vad
-            except ImportError as exc:
-                raise ServerVADUnavailableError(
-                    "server_vad requires the optional 'silero-vad' package; install vllm-omni[server-vad]"
-                ) from exc
-            self._model = load_silero_vad(onnx=False)
-            self._frame_scorer = lambda value: float(
-                self._model(torch.from_numpy(np.ascontiguousarray(value)), self._SAMPLE_RATE_HZ).item()
-            )
-        return float(self._frame_scorer(frame))

vllm_omni/entrypoints/openai/api_server.py

@@ -1290,10 +1290,12 @@ async def omni_init_app_state(
     state.openai_serving_duplex = None
     if state.openai_serving_chat is not None and should_enable_duplex_endpoint(
         state.stage_configs,
-        config_path=getattr(args, "deploy_config", None),
+        config_path=getattr(engine_client, "config_path", None) or getattr(args, "deploy_config", None),
     ):
         state.openai_serving_duplex = OmniDuplexSessionHandler(
             chat_service=state.openai_serving_chat,
+            served_model_name=model_name,
+            log_stats=state.log_stats,
             duplex_session_config=getattr(engine_client, "duplex_session_config", None),
             serving_runtime_adapter_path=getattr(engine_client, "duplex_serving_adapter_path", None),
         )
@@ -1850,8 +1852,8 @@ async def realtime_websocket(websocket: WebSocket):
             logger.warning("Duplex warmup still running after 120 s; admitting the client anyway.")
     duplex_handler = getattr(websocket.app.state, "openai_serving_duplex", None)
     duplex_query = websocket.query_params.get("duplex")
-    use_duplex_realtime = (
-        duplex_handler is not None and isinstance(duplex_query, str) and duplex_query.lower() in {"1", "true", "on"}
+    use_duplex_realtime = duplex_handler is not None and (
+        duplex_query is None or (isinstance(duplex_query, str) and duplex_query.lower() in {"1", "true", "on"})
     )
     if use_duplex_realtime and duplex_handler is not None:
         await duplex_handler.handle_realtime_session(websocket)

vllm_omni/entrypoints/openai/audio_utils_mixin.py

@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: Apache-2.0
 # SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 
+import math
 from io import BytesIO
 
 import numpy as np
@@ -19,85 +20,152 @@ logger = init_logger(__name__)
 
 
 class StreamingAudioResampler:
-    """Stateful resampler for streaming mono audio.
+    """Stateful polyphase resampler for streaming mono float audio.
 
-    Only integer downsampling ratios are supported so every input chunk can
-    produce output without buffering the complete stream.
+    Retains filter state so output is invariant to input chunk boundaries.
     """
 
+    _half_filter_width = 10
+    _kaiser_beta = 5.0
+    _max_polyphase_factor = 2_048
+
     def __init__(self, source_rate: int, target_rate: int):
         if source_rate <= 0 or target_rate <= 0:
             raise ValueError("Audio sample rates must be positive")
         self.source_rate = source_rate
         self.target_rate = target_rate
-        if source_rate < target_rate or source_rate % target_rate != 0:
-            raise ValueError(
-                "Streaming audio resampling requires an integer downsampling ratio, "
-                f"got {source_rate} Hz to {target_rate} Hz"
+        rate_gcd = math.gcd(source_rate, target_rate)
+        self._up = target_rate // rate_gcd
+        self._down = source_rate // rate_gcd
+        if max(self._up, self._down) > self._max_polyphase_factor:
+            raise ValueError("source and target sample rates have an unsupported resampling ratio")
+        self._half_len, self._phase_kernels = self._design_polyphase_filter()
+        self._history_samples = self._phase_kernels.shape[1] - 1
+        self.reset()
+
+    def _design_polyphase_filter(self) -> tuple[int, np.ndarray]:
+        if self._up == self._down:
+            return 0, np.ones((1, 1), dtype=np.float32)
+
+        max_rate = max(self._up, self._down)
+        half_len = self._half_filter_width * max_rate
+        offsets = np.arange(-half_len, half_len + 1, dtype=np.float64)
+        cutoff = 1.0 / max_rate
+        taps = cutoff * np.sinc(cutoff * offsets)
+        taps *= np.kaiser(taps.size, self._kaiser_beta)
+        taps /= np.sum(taps)
+        taps *= self._up
+        taps = np.ascontiguousarray(taps, dtype=np.float32)
+
+        phases = [taps[phase :: self._up] for phase in range(self._up)]
+        phase_width = max(phase.size for phase in phases)
+        kernels = np.zeros((self._up, phase_width), dtype=np.float32)
+        for phase_index, phase in enumerate(phases):
+            kernels[phase_index, -phase.size :] = phase[::-1]
+        return half_len, kernels
+
+    @property
+    def scratch_bytes(self) -> int:
+        pending_samples = self._ceil_div(self._input_samples * self._up, self._down) - self._output_samples
+        return max(0, pending_samples) * np.dtype(np.float32).itemsize
+
+    @staticmethod
+    def _ceil_div(numerator: int, denominator: int) -> int:
+        return -(-numerator // denominator)
+
+    def _render_outputs(
+        self,
+        combined: np.ndarray,
+        *,
+        first_output: int,
+        output_count: int,
+        input_start: int,
+    ) -> np.ndarray:
+        if output_count <= 0:
+            return np.empty(0, dtype=np.float32)
+
+        output_indexes = np.arange(first_output, first_output + output_count, dtype=np.int64)
+        filter_indexes = output_indexes * self._down + self._half_len
+        source_indexes = filter_indexes // self._up
+        phases = filter_indexes % self._up
+        windows = np.lib.stride_tricks.sliding_window_view(
+            combined,
+            self._phase_kernels.shape[1],
+        )
+        selected_windows = windows[source_indexes - input_start]
+        return np.sum(
+            selected_windows * self._phase_kernels[phases],
+            axis=1,
+            dtype=np.float32,
+        )
+
+    def _push(self, chunk: np.ndarray) -> np.ndarray:
+        if chunk.size == 0:
+            return np.empty(0, dtype=np.float32)
+        if self._flushed:
+            raise RuntimeError("cannot process audio after the resampler has been finalized; call reset first")
+
+        combined = np.concatenate((self._history, chunk))
+        next_input_samples = self._input_samples + chunk.size
+        stable_output_samples = max(
+            0,
+            self._ceil_div(next_input_samples * self._up - self._half_len, self._down),
+        )
+        output = self._render_outputs(
+            combined,
+            first_output=self._output_samples,
+            output_count=stable_output_samples - self._output_samples,
+            input_start=self._input_samples,
+        )
+        if self._history_samples:
+            self._history = combined[-self._history_samples :].copy()
+        self._input_samples = next_input_samples
+        self._output_samples = stable_output_samples
+        return output
+
+    def _flush(self) -> np.ndarray:
+        if self._flushed:
+            return np.empty(0, dtype=np.float32)
+
+        total_output_samples = self._ceil_div(self._input_samples * self._up, self._down)
+        output_count = total_output_samples - self._output_samples
+        if output_count:
+            last_output = total_output_samples - 1
+            last_source_index = (last_output * self._down + self._half_len) // self._up
+            right_padding = max(0, last_source_index - self._input_samples + 1)
+            combined = np.concatenate((self._history, np.zeros(right_padding, dtype=np.float32)))
+            output = self._render_outputs(
+                combined,
+                first_output=self._output_samples,
+                output_count=output_count,
+                input_start=self._input_samples,
             )
-
-        self._ratio = source_rate // target_rate
-        self._buffer = np.empty((0,), dtype=np.float32)
-        self._buffer_start = 0
-        self._total_samples = 0
-        self._next_center = 0
-
-        if self._ratio == 1:
-            self._taps = np.ones((1,), dtype=np.float32)
         else:
-            # Use a Kaiser-windowed sinc and leave a small transition band
-            # below the target Nyquist frequency.
-            num_taps = 20 * self._ratio + 1
-            half = num_taps // 2
-            offsets = np.arange(num_taps, dtype=np.float64) - half
-            cutoff = 0.95 / self._ratio
-            taps = cutoff * np.sinc(cutoff * offsets) * np.kaiser(num_taps, 5.0)
-            self._taps = (taps / taps.sum()).astype(np.float32)
+            output = np.empty(0, dtype=np.float32)
+        self._output_samples = total_output_samples
+        self._flushed = True
+        return output
 
     def process(self, audio: np.ndarray, *, final: bool = False) -> np.ndarray:
         chunk = np.asarray(audio, dtype=np.float32)
         if chunk.ndim != 1:
             raise ValueError(f"Streaming audio resampling only supports mono audio, got shape {chunk.shape}")
+        chunk = np.ascontiguousarray(chunk, dtype=np.float32)
+        output = self._push(chunk)
+        if not final:
+            return output
+        tail = self._flush()
+        if not output.size:
+            return tail
+        if not tail.size:
+            return output
+        return np.concatenate((output, tail))
 
-        if self._ratio == 1:
-            return chunk
-
-        if chunk.size:
-            self._buffer = np.concatenate((self._buffer, chunk))
-            self._total_samples += int(chunk.size)
-
-        half = self._taps.size // 2
-        max_center = self._total_samples - 1 if final else self._total_samples - 1 - half
-        if self._next_center > max_center:
-            return np.empty((0,), dtype=np.float32)
-
-        centers = np.arange(self._next_center, max_center + 1, self._ratio, dtype=np.int64)
-        first_center = int(centers[0])
-        last_center = int(centers[-1])
-        window_start = first_center - half
-        window_end = last_center + half
-        segment = np.zeros((window_end - window_start + 1,), dtype=np.float32)
-
-        copy_start = max(window_start, self._buffer_start, 0)
-        copy_end = min(window_end + 1, self._buffer_start + self._buffer.size)
-        if copy_end > copy_start:
-            dst_start = copy_start - window_start
-            src_start = copy_start - self._buffer_start
-            segment[dst_start : dst_start + copy_end - copy_start] = self._buffer[
-                src_start : src_start + copy_end - copy_start
-            ]
-
-        windows = np.lib.stride_tricks.sliding_window_view(segment, self._taps.size)[:: self._ratio]
-        output = windows @ self._taps[::-1]
-        self._next_center = last_center + self._ratio
-
-        keep_from = max(0, self._next_center - half)
-        drop = min(max(keep_from - self._buffer_start, 0), self._buffer.size)
-        if drop:
-            self._buffer = self._buffer[drop:]
-            self._buffer_start += int(drop)
-
-        return np.asarray(output, dtype=np.float32)
+    def reset(self) -> None:
+        self._history = np.zeros(self._history_samples, dtype=np.float32)
+        self._input_samples = 0
+        self._output_samples = 0
+        self._flushed = False
 
 
 class AudioMixin:

vllm_omni/metrics/definitions.py

@@ -161,6 +161,12 @@ AUDIO_UNDERRUN_S = METRIC_PREFIX + AUDIO_UNDERRUN + "_s"
 AUDIO_CONTINUITY_OK_METRIC = METRIC_PREFIX + AUDIO_CONTINUITY_OK
 AUDIO_SKIPPED_REQUESTS_METRIC = METRIC_PREFIX + AUDIO_SKIPPED_REQUESTS
 
+# Realtime Server VAD serving metrics.
+REALTIME_VAD_ACTIVE_SESSIONS = METRIC_PREFIX + "realtime_vad_active_sessions"
+REALTIME_VAD_INFERENCE_LATENCY_S = METRIC_PREFIX + "realtime_vad_inference_latency_s"
+REALTIME_VAD_ENDPOINT_DELAY_S = METRIC_PREFIX + "realtime_vad_endpoint_delay_s"
+REALTIME_VAD_ERRORS = METRIC_PREFIX + "realtime_vad_errors"
+
 
 # ============================================================================
 # Diffusion family (per-stage + per-replica diffusion timing breakdowns)

vllm_omni/metrics/realtime.py

@@ -0,0 +1,60 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+from __future__ import annotations
+
+from prometheus_client import Counter, Gauge, Histogram
+
+from vllm_omni.metrics import definitions as defs
+
+_active_sessions = Gauge(
+    defs.REALTIME_VAD_ACTIVE_SESSIONS,
+    "Number of active Realtime sessions using server-side VAD.",
+    labelnames=list(defs.PIPELINE_LABELS),
+)
+_inference_latency = Histogram(
+    defs.REALTIME_VAD_INFERENCE_LATENCY_S,
+    "Server VAD inference time for one appended audio batch, in seconds.",
+    labelnames=list(defs.PIPELINE_LABELS),
+    buckets=defs.SECONDS_FAST_BUCKETS,
+)
+_endpoint_delay = Histogram(
+    defs.REALTIME_VAD_ENDPOINT_DELAY_S,
+    "Detected silence between the last speech frame and the committed endpoint, in seconds.",
+    labelnames=list(defs.PIPELINE_LABELS),
+    buckets=defs.SECONDS_FAST_BUCKETS,
+)
+_errors = Counter(
+    defs.REALTIME_VAD_ERRORS,
+    "Server VAD errors and input-overflow rejections by reason.",
+    labelnames=list(defs.FAILED_LABELS),
+)
+
+
+class RealtimeVADMetrics:
+    def __init__(self, model_name: str, log_stats: bool = True) -> None:
+        self._model_name = model_name
+        self._log_stats = log_stats
+        self._active_sessions = _active_sessions.labels(model_name=model_name)
+        self._inference_latency = _inference_latency.labels(model_name=model_name)
+        self._endpoint_delay = _endpoint_delay.labels(model_name=model_name)
+
+    def session_started(self) -> None:
+        if self._log_stats:
+            self._active_sessions.inc()
+
+    def session_finished(self) -> None:
+        if self._log_stats:
+            self._active_sessions.dec()
+
+    def observe_inference(self, latency_ms: float) -> None:
+        if self._log_stats:
+            self._inference_latency.observe(latency_ms / 1000)
+
+    def observe_endpoint_delay(self, delay_ms: int) -> None:
+        if self._log_stats:
+            self._endpoint_delay.observe(delay_ms / 1000)
+
+    def error(self, reason: str) -> None:
+        if self._log_stats:
+            _errors.labels(model_name=self._model_name, reason=reason).inc()