anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Model] Integrate AURA into Full-Duplex Runtime (#7633)

Browse Source
This commit is contained in:
NumberWan authored and GitHub committed 2026-09-21 09:26:54 -07:00
1 parent d397a2f547
commit 62b15142bd
77 files changed
+7695 -457
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/examples/online_serving/aura_omni.md
+31 -36
View File
@@ -6,29 +6,29 @@
ASR -> AURA -> Qwen3-TTS Talker -> Code2Wav
```
Qwen3-TTS remains two engine stages so the pipeline reuses the existing native
Talker and Code2Wav implementation.
**Supported online path:** Realtime duplex at `/v1/realtime?duplex=1`.
The default deploy profile (`vllm_omni/deploy/aura_omni.yaml`) sets
`session_mode: duplex`. The `aura_omni` pipeline declares `duplex_plugin`, and
`DuplexOmniEngine` requires that mode (same pattern as MiniCPM-o duplex
deploys). Turn-based `chat/completions` is not a supported AURA online mode
against this profile.
`async_chunk` defaults to `false` in that file (safer if a turn-based
`OmniOrchestrator` ever loads it). Duplex Realtime Stage2→3 still needs
async chunks — use the smoke deploy below, or set `async_chunk: true`
yourself.
```bash
vllm serve aurateam/AURA \
--omni \
--deploy-config vllm_omni/deploy/aura_omni.yaml \
--served-model-name aurateam/AURA \
--trust-remote-code
bash examples/online_serving/aura_omni/run_duplex_smoke_serve.sh
python examples/online_serving/aura_omni/smoke_duplex_realtime_client.py
```
Configure local checkpoints by editing per-stage `model` values in
`vllm_omni/deploy/aura_omni.yaml`. The deploy file sets
`pipeline: aura_omni`, so the four-stage topology is used even if the
command-line `--model` points at one of the component checkpoints.
Configure local checkpoints by editing per-stage `model` values in the deploy
or smoke YAML. The file sets `pipeline: aura_omni`, so the four-stage topology
is used even if the command-line `--model` points at one component checkpoint.
Send requests with `"model": "aurateam/AURA"`. The ASR, AURA, and Qwen3-TTS
checkpoint paths are internal stage models from the deploy YAML, not the
OpenAI-facing served model name.
The AURA stage can emit `<|silent|>`. Silent outputs are treated as a gate:
they produce no Qwen3-TTS Talker input, so no audio is synthesized for that
turn.
Silent Stage1 outputs (`<|silent|>` / id `151669`) skip TTS for that turn.
## GPU Utilization Recommendation
@@ -40,26 +40,21 @@ VRAM each stage can reserve. Start with this split for a single GPU:
- Stage 2 (Qwen3-TTS Talker): `0.20`
- Stage 3 (Qwen3-TTS Code2Wav): `0.20`
## TTS Modes
## TTS modes (stage extras)
`aura_omni` can pass AURA text to Qwen3-TTS in two task modes:
When the duplex session supplies TTS extras, AURA text can feed Qwen3-TTS as:
- `Base`: voice clone from `tts_ref_audio` with ICL enabled in the AURA
pipeline. Provide both `tts_ref_audio` and `tts_ref_text`. Set
`tts_x_vector_only_mode=true` to disable ICL and use speaker embedding only.
- `CustomVoice`: predefined speaker mode. Use a Qwen3-TTS CustomVoice
checkpoint for stages 2 and 3 in `aura_omni.yaml`, then pass
`tts_task_type=CustomVoice` and `tts_speaker`.
- `Base`: voice clone from `tts_ref_audio` (optional x-vector-only mode)
- `CustomVoice`: predefined speaker (`tts_speaker`) with a CustomVoice checkpoint
on stages 2 and 3
By default, AURA responses are passed to Qwen3-TTS as text. Set
`additional_information.tts_pass_token_ids=true` to pass AURA-generated
assistant token ids directly instead. Even when token passthrough is disabled,
the stage processor uses AURA token ids when available to estimate the Talker
prefill length, so scheduling does not rely on raw character length.
Optional `tts_pass_token_ids` passes AURA assistant token ids into Talker
instead of detokenized text.
The example client exposes this as:
## Old chat / Gradio / curl scripts (unsupported)
```bash
python examples/online_serving/aura_omni/openai_chat_completion_client.py \
--tts-pass-token-ids
```
`examples/online_serving/aura_omni/` still contains chat-completions, curl, and
Gradio helpers from an older turn-based path. They do **not** work against the
shipped duplex profile. That path was an incomplete stand-in before streaming
I/O and session history were available; it is not a supported AURA mode. Use
the Realtime duplex smoke client for online verification.
examples/online_serving/aura_omni/README.md
+63 -113
View File
@@ -6,139 +6,89 @@
Qwen3-ASR -> AURA/Qwen3-VL -> Qwen3-TTS Talker -> Qwen3-TTS Code2Wav
```
The pipeline has three semantic modules, but four engine stages because the
existing Qwen3-TTS implementation is natively split into Talker and Code2Wav.
**AURA's supported online path is Realtime duplex** (`/v1/realtime?duplex=1`).
The default deploy profile sets `session_mode: duplex` because the pipeline
declares `duplex_plugin` and `DuplexOmniEngine` requires that mode.
Start the server with the deploy profile:
This is AURA v1 / Qwen3-VL (silent / ChatML ids `151669` / `151645`), not
AURA v2 / Qwen3.5-VL.
## Duplex Realtime (only supported online path)
Stage2→3 codec handoff needs `async_chunk: true`. The default
`vllm_omni/deploy/aura_omni.yaml` keeps `async_chunk: false` as a safer
baseline (turn-based `OmniOrchestrator` never sees the duplex async-chunk
gate). For Realtime, use the smoke deploy (already `async_chunk: true`) or
set that flag yourself:
```bash
bash examples/online_serving/aura_omni/run_duplex_smoke_serve.sh
python examples/online_serving/aura_omni/smoke_duplex_realtime_client.py
```
Smoke deploy file: `examples/online_serving/aura_omni/aura_omni_duplex_smoke.yaml`.
Equivalent serve with the default topology and async chunks enabled:
```bash
# Copy or edit so async_chunk: true, then:
vllm serve aurateam/AURA \
--omni \
--port 8091 \
--deploy-config vllm_omni/deploy/aura_omni.yaml \
--deploy-config examples/online_serving/aura_omni/aura_omni_duplex_smoke.yaml \
--served-model-name aurateam/AURA \
--trust-remote-code
```
The deploy file sets per-stage model repos:
Connect clients to `/v1/realtime?duplex=1`. Silent Stage1 outputs gate TTS (no
audio for that turn). Overlapped input and vision-follow are AURA duplex
capabilities.
### Browser UI
AURA has no client VAD: control is **push-to-talk** on the shared shell from
[#7585](https://github.com/vllm-project/vllm-omni/pull/7585)
([`examples/online_serving/realtime_web/`](../realtime_web/README.md)), profile
`aura-ptt` (hold = `is_speech=true` PCM + sticky frames; release = `commit`;
silent+frames for vision-follow). Thin wrapper:
```bash
python -m examples.online_serving.aura_omni.realtime_web \
--ws-backend ws://127.0.0.1:8099 --model aurateam/AURA --port 7862
```
Do **not** fork a separate `aura_omni/realtime_web` asset tree; MiniCPM / Qwen
profiles are unchanged and hide the PTT control.
### Per-stage models
Edit `model` on each stage in `vllm_omni/deploy/aura_omni.yaml` (or the smoke
YAML) for local checkpoints:
- Stage 0 ASR: `Qwen/Qwen3-ASR-1.7B`
- Stage 1 AURA: `aurateam/AURA`
- Stage 2/3 TTS: `Qwen/Qwen3-TTS-12Hz-1.7B-Base`
- Stage 2/3 TTS: `Qwen/Qwen3-TTS-12Hz-1.7B-Base` (or CustomVoice)
For local weights, edit the `model` value on each stage in
`vllm_omni/deploy/aura_omni.yaml`. The deploy profile includes
`pipeline: aura_omni`, so the server uses this four-stage topology even when
the command-line model path points at one component checkpoint.
### GPU utilization
Expected request shape:
- Send microphone audio as the Stage 0 multimodal audio input.
- Include video frames in the original request `multi_modal_data`; the
`asr2aura` processor carries them forward to AURA.
- Optional `additional_information` keys:
- `aura_system_prompt`
- `tts_task_type`
- `tts_language`
- `tts_speaker`
- `tts_instruct`
- `tts_ref_audio`
- `tts_ref_text`
- `tts_x_vector_only_mode`
- `tts_pass_token_ids`
If AURA emits `<|silent|>`, the `aura2tts` processor returns no TTS request, so
the TTS stages are skipped for that turn.
## GPU Utilization Recommendation
Tune `gpu_memory_utilization` per stage in `vllm_omni/deploy/aura_omni.yaml`.
Recommended baseline on one GPU for H200
Tune `gpu_memory_utilization` per stage. Baseline on one large GPU:
- Stage 0 (ASR): `0.10`
- Stage 1 (AURA): `0.4`
- Stage 2 (Qwen3-TTS Talker): `0.20`
- Stage 3 (Qwen3-TTS Code2Wav): `0.20`
- Stage 1 (AURA): `0.40`
- Stage 2 (Talker): `0.20`
- Stage 3 (Code2Wav): `0.20`
## Python Client
## Old chat / Gradio / curl scripts (unsupported)
```bash
python examples/online_serving/aura_omni/openai_chat_completion_client.py \
--host localhost \
--port 8091 \
--model aurateam/AURA \
--modalities text,audio
```
`openai_chat_completion_client.py`, `run_curl_multimodal_generation.sh`, and
`run_gradio_demo.sh` target the older turn-based Omni path. They do **not**
work against the shipped `session_mode: duplex` profile.
Use local media:
```bash
python examples/online_serving/aura_omni/openai_chat_completion_client.py \
--audio-path /path/to/input.wav \
--video-path /path/to/video.mp4 \
--output-dir output_aura_omni_online
```
Base voice clone mode (default, recommended as x-vector while debugging ICL):
```bash
python examples/online_serving/aura_omni/openai_chat_completion_client.py \
--tts-task-type Base \
--tts-ref-audio vllm-omni/tests/assets/qwen3_tts/clone_2.wav \
--tts-ref-text "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."
```
Enable AURA token-id passthrough explicitly:
```bash
python examples/online_serving/aura_omni/openai_chat_completion_client.py \
--tts-pass-token-ids
```
CustomVoice mode requires stages 2 and 3 in `aura_omni.yaml` to point at a
Qwen3-TTS CustomVoice checkpoint:
```bash
python examples/online_serving/aura_omni/openai_chat_completion_client.py \
--tts-task-type CustomVoice \
--tts-speaker Vivian
```
By default, AURA responses are passed to Qwen3-TTS as text. Set
`tts_pass_token_ids=true` to pass AURA-generated assistant token ids directly
to Qwen3-TTS instead. The processor still uses AURA token ids, when available,
to estimate the Talker prompt length in the default text path.
## Curl
```bash
cd examples/online_serving/aura_omni
bash run_curl_multimodal_generation.sh
```
Set `PORT`, `MODEL`, or `OUTPUT_DIR` to override defaults:
```bash
PORT=8666 MODEL=aurateam/AURA bash run_curl_multimodal_generation.sh
TTS_PASS_TOKEN_IDS=true PORT=8666 MODEL=aurateam/AURA bash run_curl_multimodal_generation.sh
```
## Gradio
Launch the server and Gradio UI together:
```bash
cd examples/online_serving/aura_omni
bash run_gradio_demo.sh
```
If the server is already running:
```bash
python examples/online_serving/aura_omni/gradio_demo.py \
--model aurateam/AURA \
--api-base http://localhost:8091/v1
```
That turn-based path was an incomplete June-era stand-in: vLLM-Omni did not
yet support streaming I/O for this stack, and adding session history then would
have required a much larger orchestrator change. It is not a supported AURA
mode. Do not run those scripts against the default deploy. Prefer the duplex
smoke client above.
## Offline
examples/online_serving/aura_omni/aura_omni_duplex_smoke.yaml
+119
View File
@@ -0,0 +1,119 @@
# AURA duplex Realtime smoke (local weights, session_mode: duplex).
# Stage1 = /workspace/models/AURA + AuraQwen3VLForConditionalGeneration (AURA v1 / Qwen3-VL).
pipeline: aura_omni
# Qwen3-TTS Stage2→3 codec handoff requires async_chunk=true (same as native /
# duplex explore). async_chunk=false reaches talker/code_predictor then never
# delivers audio to Stage3/client.
async_chunk: true
session_mode: duplex
duplex_session:
max_sessions: 1
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
extra:
codec_streaming: true
connector_get_sleep_s: 0.01
connector_get_max_wait_first_chunk: 3000
connector_get_max_wait: 300
codec_chunk_frames: 25
codec_left_context_frames: 72
# Match native/duplex explore. initial=1 can stall Code2Wav on the first
# undersized chunk (observed hang after "Code2Wav codec: frames=1").
initial_codec_chunk_frames: 8
decode_cudagraph_capture_sizes: [25, 73, 97, 169, 325]
stages:
- stage_id: 0
max_num_seqs: 1
gpu_memory_utilization: 0.08
enforce_eager: true
trust_remote_code: true
enable_prefix_caching: true
async_scheduling: true
max_model_len: 4096
max_num_batched_tokens: 2048
devices: "0"
model: /workspace/models/Qwen3-ASR-1.7B
default_sampling_params:
temperature: 0.0
top_p: 1.0
top_k: -1
max_tokens: 256
- stage_id: 1
max_num_seqs: 1
gpu_memory_utilization: 0.55
enforce_eager: true
trust_remote_code: true
enable_prefix_caching: true
async_scheduling: true
max_model_len: 32768
max_num_batched_tokens: 8192
devices: "0"
model: /workspace/models/AURA
model_arch: AuraQwen3VLForConditionalGeneration
default_sampling_params:
temperature: 0.0
top_p: 1.0
top_k: -1
max_tokens: 256
repetition_penalty: 1.0
stop_token_ids: [151669, 151645]
- stage_id: 2
# Allow prior TTS to drain while the next Talker starts (not assumed 1).
max_num_seqs: 2
gpu_memory_utilization: 0.12
enforce_eager: true
trust_remote_code: true
enable_prefix_caching: false
async_scheduling: true
max_num_batched_tokens: 4096
max_model_len: 4096
devices: "0"
# Stage1→2 is orchestrator-fed via aura2tts (custom_process_input_func).
# A from_stage_1 SHM edge parks Talker in WAITING_FOR_CHUNK and clears
# additional_information.text → Missing Qwen3-TTS text conditioning.
# Only Stage2→3 codec handoff needs a connector.
output_connectors:
to_stage_3: connector_of_shared_memory
model: /workspace/models/Qwen3-TTS-12Hz-1.7B-CustomVoice
default_sampling_params:
temperature: 0.5
top_k: 50
# 12Hz × 240 ≈ 20s. Stay under the smoke 25s audio gate so a missed
# 2150 (English/quotes humming) cannot overlap the next vision-follow.
max_tokens: 240
min_tokens: 8
repetition_penalty: 1.05
# Qwen3-TTS Talker codec EOS — without this Talker can run to max_tokens
# (~26s garbage audio while Stage1 text was correct).
stop_token_ids: [2150]
subtalker_sampling_params:
do_sample: true
temperature: 0.5
top_k: 50
top_p: 1.0
- stage_id: 3
max_num_seqs: 2
gpu_memory_utilization: 0.12
enforce_eager: true
trust_remote_code: true
enable_prefix_caching: false
async_scheduling: true
max_num_batched_tokens: 16384
max_model_len: 16384
devices: "0"
input_connectors:
from_stage_2: connector_of_shared_memory
model: /workspace/models/Qwen3-TTS-12Hz-1.7B-CustomVoice
default_sampling_params:
temperature: 0.0
top_p: 1.0
top_k: -1
max_tokens: 16384
seed: 42
repetition_penalty: 1.0
examples/online_serving/aura_omni/realtime_web/__init__.py
+4
View File
@@ -0,0 +1,4 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""AURA push-to-talk browser call UI."""
examples/online_serving/aura_omni/realtime_web/__main__.py
+7
View File
@@ -0,0 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from examples.online_serving.realtime_web.server import main
if __name__ == "__main__":
main(default_profile="aura-ptt")
examples/online_serving/aura_omni/run_duplex_smoke_serve.sh
Executable
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
# Minimal AURA duplex Realtime server for smoke.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
cd "$REPO_ROOT"
PORT="${PORT:-8099}"
MODEL="${MODEL:-/workspace/models/AURA}"
DEPLOY_CONFIG="${DEPLOY_CONFIG:-$REPO_ROOT/examples/online_serving/aura_omni/aura_omni_duplex_smoke.yaml}"
# AURA v1 / Qwen3-VL ChatML ids (do not use AURA v2 248070/248046 here).
export VLLM_AURA_SILENT_TOKEN_ID="${VLLM_AURA_SILENT_TOKEN_ID:-151669}"
export VLLM_AURA_IM_END_TOKEN_ID="${VLLM_AURA_IM_END_TOKEN_ID:-151645}"
export VLLM_AURA_IM_START_TOKEN_ID="${VLLM_AURA_IM_START_TOKEN_ID:-151644}"
export VLLM_AURA_ASSISTANT_TOKEN_ID="${VLLM_AURA_ASSISTANT_TOKEN_ID:-77091}"
# Multi-stage colocated init can make earlier stages' Process.sentinel spuriously
# readable; prefer spawn for stage engine subprocesses (override if needed).
export VLLM_WORKER_MULTIPROC_METHOD="${VLLM_WORKER_MULTIPROC_METHOD:-spawn}"
# Do NOT set TORCHDYNAMO_DISABLE here: it breaks Stage0/1 AOT when those
# stages are not enforce_eager. Stage2/3 use enforce_eager; code_predictor
# honors that and skips its own torch.compile.
exec "$REPO_ROOT/.venv/bin/vllm-omni" serve "$MODEL" \
--omni \
--deploy-config "$DEPLOY_CONFIG" \
--trust-remote-code \
--host 0.0.0.0 \
--port "$PORT"
examples/online_serving/aura_omni/smoke_duplex_realtime_client.py
Executable
+153
View File
@@ -0,0 +1,153 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
"""Minimal AURA duplex Realtime smoke via DuplexClient."""
from __future__ import annotations
import argparse
import asyncio
import base64
import json
import sys
import time
from io import BytesIO
from pathlib import Path
from PIL import Image
REPO_ROOT = Path(__file__).resolve().parents[3]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from vllm_omni.clients.duplex import ( # noqa: E402
DuplexClient,
EventCollector,
SessionConfig,
read_pcm16_wav,
)
PROGRESS_TYPES = {
"response.created",
"response.done",
"response.audio.delta",
"response.output_audio.delta",
"response.audio_transcript.delta",
"response.output_text.delta",
"response.text.delta",
"response.listen",
"response.model_listen",
"error",
}
def _tiny_jpeg_b64() -> str:
img = Image.new("RGB", (64, 64), color=(200, 40, 40))
buf = BytesIO()
img.save(buf, format="JPEG", quality=85)
return base64.b64encode(buf.getvalue()).decode("ascii")
async def run(url: str, model: str, wav: Path, timeout_s: float) -> int:
pcm16 = read_pcm16_wav(wav)
if not pcm16:
print("empty wav", file=sys.stderr)
return 2
frame = _tiny_jpeg_b64()
collector = EventCollector()
config = SessionConfig(
modalities=("text", "audio"),
instructions="You are AURA. Reply briefly in Chinese.",
extra_body={
"aura_system_prompt": "You are AURA. Reply briefly in Chinese.",
"tts_task_type": "CustomVoice",
"tts_language": "Chinese",
"tts_speaker": "Vivian",
},
)
started = time.time()
client = DuplexClient(
url,
model=model,
config=config,
reconnect=None,
heartbeat_interval_s=None,
handshake_timeout_s=min(60.0, timeout_s),
)
async with client:
consume_task = asyncio.create_task(collector.consume(client))
collector.add({"type": "session.created", "session": getattr(client, "session_info", {})})
await client.append_audio(pcm16, is_speech=True, video_frames=[frame])
await client.commit(create_response=True)
progress = False
got_done = False
while time.time() - started < timeout_s:
types = [e.get("type") for e in collector.events]
if any(t in PROGRESS_TYPES for t in types):
progress = True
if "response.done" in types or "response.completed" in types:
got_done = True
progress = True
break
for e in collector.events:
meta = e.get("metadata") if isinstance(e.get("metadata"), dict) else {}
if meta.get("model_listen") or meta.get("listen_source") == "aura_silent":
progress = True
break
if collector.count("error") > 0 or collector.count("session.closed") > 0:
break
# Keep draining until response.done so TTS is not truncated.
await asyncio.sleep(0.2)
try:
await client.close(timeout_s=30.0)
except Exception as exc: # noqa: BLE001
print(f"close_warn={exc}", file=sys.stderr)
try:
await asyncio.wait_for(consume_task, timeout=5.0)
except (TimeoutError, asyncio.CancelledError):
consume_task.cancel()
event_types = [e.get("type") for e in collector.events]
errors = [json.dumps(e, ensure_ascii=False)[:500] for e in collector.errors()]
result = {
"ok": progress and (got_done or collector.count("response.done") > 0),
"got_response_progress": progress,
"got_response_done": got_done or collector.count("response.done") > 0,
"errors": errors,
"event_types": event_types,
"n_events": len(event_types),
"audio_bytes": len(collector.audio_bytes()),
"llm_text": collector.response_text(collector.response_ids[0]) if collector.response_ids else "",
"elapsed_s": round(time.time() - started, 2),
}
print(json.dumps(result, ensure_ascii=False, indent=2))
log_path = Path("/tmp/aura_duplex_smoke/client_events.jsonl")
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text(
"".join(json.dumps(e, ensure_ascii=False) + "\n" for e in collector.events),
encoding="utf-8",
)
print(f"events_log={log_path}", file=sys.stderr)
return 0 if progress else 1
def main() -> int:
p = argparse.ArgumentParser()
p.add_argument("--url", default="ws://127.0.0.1:8099/v1/realtime")
p.add_argument("--model", default="/workspace/models/AURA")
p.add_argument(
"--wav",
default=str(REPO_ROOT / "tests/assets/minicpmo_4_5/response_required_16k.wav"),
)
p.add_argument("--timeout-s", type=float, default=300.0)
args = p.parse_args()
return asyncio.run(run(args.url, args.model, Path(args.wav), args.timeout_s))
if __name__ == "__main__":
raise SystemExit(main())
examples/online_serving/realtime_web/README.md
+36 -2
View File
@@ -9,6 +9,7 @@ This implements [RFC #7222](https://github.com/vllm-project/vllm-omni/issues/722
| `minicpm-native` | Model-controlled listen/speak, continuous audio input | Frames accompany audio | Yes |
| `qwen3-turn --stt` (default) | User presses **Send turn** | No | No |
| `qwen3-turn --vad` | Server detects trailing silence; speech interrupts replies | Sampled frames with each spoken turn | Yes |
| `aura-ptt` | Hold **Hold to talk**; release commits. Hold also stops local playback | Sticky frames with speech; vision-follow every 2 frames only while unlocked (locked from PTT release until text-final / listen) | Yes |
Qwen3 VAD uses an engine-owned duplex plugin: microphone upload continues
while replies stream, and speech can interrupt generation and playback. The
@@ -29,6 +30,38 @@ The compatibility wrapper serves the shared assets with `minicpm-native`.
The current native query, `extra_body`, reference voice, continuous microphone
upload, camera frames and playback acknowledgements are retained.
## AURA: push-to-talk duplex
AURA has no client VAD. Start the duplex serve, then the thin wrapper:
```bash
bash examples/online_serving/aura_omni/run_duplex_smoke_serve.sh
python -m examples.online_serving.aura_omni.realtime_web \
--ws-backend ws://127.0.0.1:8099 --model aurateam/AURA --port 7862
```
Or select the profile on the shared host:
```bash
python -m examples.online_serving.realtime_web --profile aura-ptt \
--ws-backend ws://127.0.0.1:8099 --model aurateam/AURA --port 7862
```
Open the UI, start a session, optionally enable **Camera**, then **hold**
**Hold to talk** to stream `is_speech=true` PCM (with sticky camera frames).
**Release** to `input_audio_buffer.commit`. Holding the button stops playback
in the browser only; the in-flight response is not cancelled on the server.
With the camera on, frames are still captured at 2 fps. While the button is
up they are committed once per two frames (`is_speech=false` + silent PCM).
Both frames of that commit are sent and packed as one video (`<|video_pad|>`).
Turns still open from this client commit, not from a server-side auto trigger.
but only after the prior spoken turn unlocks (text-final or `response.listen`).
Release itself locks vision immediately so a follow-up frame cannot abort the
just-committed speech turn on Stage0/1 (`max_num_seqs=1`). Audio still playing
does not hold the next vision turn once text is done.
MiniCPM / Qwen profiles do not set `pushToTalk`, so the PTT control stays hidden.
## Qwen3: explicit-turn STT adapter
Start the backend, then the UI in a second terminal:
@@ -49,6 +82,7 @@ accepts only model selection, so system-prompt controls are hidden in this mode.
The adapter follows the shipped `qwen3_omni/openai_realtime_client.py` and
`vllm_omni/entrypoints/openai/realtime_connection.py`:
1. Explicit `duplex=0` selects the legacy STT handler on a turn deployment.
It does not enable STT on a duplex deployment.
2. Send `{type: "session.update", model: ...}` and `commit(final=false)`.
@@ -181,7 +215,7 @@ conversation API. This demo does not add WebRTC, semantic VAD, or tool calling.
## Shared host options
The shared entry point can select either profile explicitly:
The shared entry point can select a profile explicitly:
```bash
python -m examples.online_serving.realtime_web --profile qwen3-turn \
@@ -193,7 +227,7 @@ python -m examples.online_serving.realtime_web --profile qwen3-turn \
- `--public-realtime-url`: optional browser-visible WebSocket URL; otherwise the
static host proxies `/v1/realtime` on the same origin.
- `--host`, `--port`: UI bind address and port (default port 7862).
- `--ref-audio`: required for MiniCPM; rejected for Qwen3.
- `--ref-audio`: required for MiniCPM; rejected for Qwen3 and AURA.
Microphone and camera access require `localhost` or HTTPS. For a remote backend,
use an SSH tunnel to the UI host or an HTTPS reverse proxy with WebSocket support.
examples/online_serving/realtime_web/app/index.html
+2
View File
@@ -22,6 +22,7 @@
<div class="call-actions">
<button id="callButton" class="primary" type="button">Start session</button>
<button id="sendTurnButton" class="secondary" type="button" hidden disabled>Send turn</button>
<button id="pttButton" class="secondary" type="button" hidden disabled>Hold to talk</button>
<button id="muteButton" class="secondary" type="button" disabled>Mute</button>
<button id="cameraButton" class="secondary" type="button" disabled>Camera</button>
<video id="cameraPreview" muted playsinline style="display:none;width:96px;height:72px;border-radius:8px;object-fit:cover;background:#000"></video>
@@ -75,6 +76,7 @@
<script src="static/profiles/common.js?v=__FULL_DUPLEX_APP_VERSION__"></script>
<script src="static/profiles/minicpm_native.js?v=__FULL_DUPLEX_APP_VERSION__"></script>
<script src="static/profiles/qwen3_turn.js?v=__FULL_DUPLEX_APP_VERSION__"></script>
<script src="static/profiles/aura_ptt.js?v=__FULL_DUPLEX_APP_VERSION__"></script>
<script src="static/app.js?v=__FULL_DUPLEX_APP_VERSION__"></script>
</body>
</html>
examples/online_serving/realtime_web/app/static/app.js
+205 -14
View File
@@ -5,6 +5,7 @@
const profile = window.OmniRealtimeProfiles[config.profile || 'minicpm-native'](config);
const callButton = document.getElementById('callButton');
const sendTurnButton = document.getElementById('sendTurnButton');
const pttButton = document.getElementById('pttButton');
const muteButton = document.getElementById('muteButton');
const cameraButton = document.getElementById('cameraButton');
const cameraPreview = document.getElementById('cameraPreview');
@@ -28,6 +29,7 @@
const ECHO_GUARD_MS = 300;
const INITIAL_PLAYBACK_BUFFER_MS = 400;
const SESSION_CLOSE_TIMEOUT_MS = 1000;
const SILENT_PCM_SAMPLES = 160;
const PROMPT_PRESETS = profile.presets;
document.title = profile.title;
@@ -36,7 +38,21 @@
document.getElementById('profileDescription').textContent = profile.description;
document.getElementById('policyLabel').textContent = profile.policy;
cameraButton.hidden = !profile.camera;
if (profile.cameraPreviewLarge) {
cameraPreview.classList.add('camera-preview-large');
// Inline, so a cached stylesheet cannot leave the 96×72 HTML default.
cameraPreview.style.width = '100%';
cameraPreview.style.height = 'auto';
cameraPreview.style.maxHeight = '480px';
cameraPreview.style.flex = '0 0 100%';
const cameraActions = cameraPreview.closest('.call-actions');
if (cameraActions) {
cameraActions.style.flexWrap = 'wrap';
cameraActions.style.gridColumn = '1 / -1';
}
}
sendTurnButton.hidden = !profile.clientCommit;
pttButton.hidden = !profile.pushToTalk;
promptPreset.replaceChildren();
for (const name of [...Object.keys(PROMPT_PRESETS), 'custom']) {
const option = document.createElement('option');
@@ -64,6 +80,12 @@
let cameraStream = null;
let cameraTimer = null;
let cameraPendingFrame = null;
let cameraLastFrame = null;
// AURA vision clock only. Other profiles leave this empty and never read it.
let visionFollowQueue = [];
let visionTurnLocked = false;
let suppressedPlaybackId = null;
let pttHeld = false;
const cameraCanvas = document.createElement('canvas');
let playbackRate = OUTPUT_RATE;
let pendingCapture = [];
@@ -87,6 +109,8 @@
const interruptedResponses = new Set();
const pendingEvents = new Set();
let assistantTextChannel = null;
const assistantRowByResponse = new Map();
let lastClosedAssistantText = '';
let connectionReady = false;
let turnCounter = 0;
let stopping = null;
@@ -134,7 +158,7 @@
assistantTextChannel = null;
if (profile.halfDuplex) pendingCapture = [];
sendTurnButton.disabled = !profile.clientCommit;
setModel(profile.waiting);
if (!pttHeld) setModel(profile.waiting);
}, ECHO_GUARD_MS);
}
@@ -240,6 +264,7 @@
current.value = finalText;
current.text.textContent = finalText;
}
if (role === 'assistant' && current.value) lastClosedAssistantText = current.value;
current.row.classList.remove('turn-live');
if (role === 'user') liveUserTurn = null;
else liveAssistantTurn = null;
@@ -317,11 +342,67 @@
}
function microphoneUploadEnabled() {
return running && connectionReady && !muted && (!profile.halfDuplex || !assistantActive);
if (!running || !connectionReady || muted) return false;
if (profile.pushToTalk) return pttHeld;
return !profile.halfDuplex || !assistantActive;
}
function queueVisionFrame(frame) {
visionFollowQueue.push(frame);
if (visionFollowQueue.length > 2) visionFollowQueue.shift();
}
function commitVisionFollow() {
// Two frames still gate the turn (client commit). Both frames are one clip;
// the server packs them as a single <|video_pad|>, same as Native.
const frames = visionFollowQueue.slice();
visionFollowQueue = [];
const silent = new Int16Array(SILENT_PCM_SAMPLES);
const event = profile.append(int16ToBase64(silent), frames, { isSpeech: false });
socket.send(JSON.stringify(event));
for (const message of profile.commitMessages()) socket.send(JSON.stringify(message));
}
function releaseVisionTurn(responseId) {
if (!profile.visionFollowWhileSpeaking || !visionTurnLocked) return;
if (responseId && currentResponseId && responseId !== currentResponseId) return;
visionTurnLocked = false;
}
function suppressLocalPlayback() {
// Hold-to-talk cuts the speaker only. The in-flight response stays on the server.
if (currentResponseId) suppressedPlaybackId = currentResponseId;
if (playbackNode) playbackNode.port.postMessage({ type: 'clear' });
playbackComplete = true;
responseHasAudio = false;
setPlayback('Idle');
}
function playbackSuppressed(responseId) {
return Boolean(suppressedPlaybackId) && responseId === suppressedPlaybackId;
}
function flushCapture() {
if (!socket || socket.readyState !== WebSocket.OPEN || pendingCapture.length === 0) return;
if (!socket || socket.readyState !== WebSocket.OPEN) return;
// AURA: capture stays 2 fps, but a turn opens only every 2 frames, and
// not until the previous response's text is done (or it listened).
// Playback still running is not the lock. PTT speech is not batched here.
if (profile.pushToTalk && profile.visionFollowWhileSpeaking && !pttHeld) {
if (cameraPendingFrame) {
queueVisionFrame(cameraPendingFrame);
cameraPendingFrame = null;
}
if (!visionTurnLocked && visionFollowQueue.length >= 2) commitVisionFollow();
return;
}
if (profile.pushToTalk && !pttHeld) {
pendingCapture = [];
return;
}
if (pendingCapture.length === 0) return;
if (!microphoneUploadEnabled()) {
pendingCapture = [];
return;
@@ -335,11 +416,44 @@
}
pendingCapture = [];
const pcm = resampleInt16(merged, captureRate, profile.inputSampleRate || INPUT_RATE);
const appendEvent = profile.append(int16ToBase64(pcm), cameraPendingFrame);
const frame = cameraPendingFrame || (profile.stickyCamera ? cameraLastFrame : null);
cameraPendingFrame = null;
const appendEvent = profile.pushToTalk
? profile.append(int16ToBase64(pcm), frame, { isSpeech: true })
: profile.append(int16ToBase64(pcm), frame);
socket.send(JSON.stringify(appendEvent));
}
function setPttHeld(held) {
if (!profile.pushToTalk || !running || pttHeld === held) return;
if (held) {
pttHeld = true;
suppressLocalPlayback();
pttButton.classList.toggle('is-active', true);
pttButton.textContent = 'Release to send';
setModel('Talking');
appendLog('PTT down');
return;
}
// Flush remaining speech while still held, then commit.
flushCapture();
if (socket && socket.readyState === WebSocket.OPEN) {
for (const event of profile.commitMessages()) socket.send(JSON.stringify(event));
}
// Lock vision before clearing pttHeld. Stage0/1 are max_num_seqs=1: a
// vision-follow commit in the gap before response.begin aborts the speech
// turn (server sees a new ephemeral and kills the prior ASR request).
if (profile.visionFollowWhileSpeaking) {
visionTurnLocked = true;
visionFollowQueue = [];
}
pttHeld = false;
pttButton.classList.toggle('is-active', false);
pttButton.textContent = 'Hold to talk';
setModel(profile.waiting);
appendLog('PTT up · commit');
}
function beginAssistant(responseId) {
currentResponseId = responseId || currentResponseId;
responseHasAudio = false;
@@ -351,6 +465,7 @@
}
function feedPlayback(decoded, responseId) {
if (playbackSuppressed(responseId || currentResponseId)) return;
if (!decoded || !decoded.pcm || decoded.pcm.length === 0 || !playbackNode) return;
const pcm = resampleInt16(decoded.pcm, decoded.sourceRate, playbackRate);
responseHasAudio = true;
@@ -392,13 +507,18 @@
}
async function handleAudioEvent(action) {
const responseId = action.responseId || currentResponseId || `turn-${turnCounter}`;
if (playbackSuppressed(responseId)) return;
markBusy();
currentResponseId = action.responseId || currentResponseId || `turn-${turnCounter}`;
currentResponseId = responseId;
setModel('Speaking');
const generation = sessionGeneration;
const responseId = currentResponseId;
const decoded = await decodeAudioDelta(action.event);
if (generation === sessionGeneration && !interruptedResponses.has(responseId)) feedPlayback(decoded, responseId);
if (
generation === sessionGeneration
&& !interruptedResponses.has(responseId)
&& !playbackSuppressed(responseId)
) feedPlayback(decoded, responseId);
}
function handleTranscriptEvent(action) {
@@ -408,12 +528,43 @@
if (profile.deduplicateTranscript && assistantTextChannel && action.channel !== assistantTextChannel) return;
if (action.text) assistantTextChannel = action.channel || assistantTextChannel;
}
if (action.kind === 'text') addTranscript(action.role, action.text);
else if (profile.deduplicateTranscript && action.role === 'assistant') {
const responseId = action.responseId || null;
const existing = responseId && action.role === 'assistant' ? assistantRowByResponse.get(responseId) : null;
if (existing) {
if (action.kind === 'text') {
if (!action.text) return;
existing.value += action.text;
} else if (action.text) {
existing.value = action.text;
}
existing.text.textContent = existing.value;
liveAssistantTurn = existing;
return;
}
if (
action.role === 'assistant'
&& !liveAssistantTurn
&& action.text
&& action.text === lastClosedAssistantText
) {
// response.done already closed this sentence. A later full transcript,
// including one stamped with a newer vision-follow response id, must
// not open a second bubble with the same words.
return;
}
if (action.kind === 'text') {
addTranscript(action.role, action.text);
if (action.role === 'assistant' && responseId && liveAssistantTurn) {
assistantRowByResponse.set(responseId, liveAssistantTurn);
}
return;
}
if (profile.deduplicateTranscript && action.role === 'assistant') {
if (action.text) {
const turn = ensureTurn('assistant');
turn.value = action.text;
turn.text.textContent = action.text;
if (responseId) assistantRowByResponse.set(responseId, turn);
}
} else finishTranscript(action.role, action.text);
}
@@ -459,12 +610,14 @@
}
case 'listen':
assistantActive = false;
releaseVisionTurn(responseId);
setModel(profile.waiting);
break;
case 'begin':
if (profile.halfDuplex) armTurnTimeout();
if (echoTimer !== null) { clearTimeout(echoTimer); echoTimer = null; }
beginAssistant(responseId);
if (profile.visionFollowWhileSpeaking) visionTurnLocked = true;
break;
case 'audio':
await handleAudioEvent(action);
@@ -474,11 +627,13 @@
break;
case 'text': case 'text-final':
handleTranscriptEvent(action);
if (action.kind === 'text-final' && action.role === 'assistant') releaseVisionTurn(responseId);
break;
case 'done':
clearTimeout(turnTimeout);
responseComplete = true;
finishTranscript('assistant');
releaseVisionTurn(responseId);
requestPlaybackDrain(responseId);
finishResponseIfReady();
break;
@@ -671,7 +826,11 @@
await openSocket();
running = true;
muted = false;
pttHeld = false;
assistantActive = false;
visionFollowQueue = [];
visionTurnLocked = false;
suppressedPlaybackId = null;
sendTimer = window.setInterval(flushCapture, profile.sendIntervalMs || SEND_INTERVAL_MS);
startClock();
callButton.textContent = 'End session';
@@ -679,6 +838,7 @@
muteButton.disabled = false;
cameraButton.disabled = !profile.camera;
sendTurnButton.disabled = !profile.clientCommit;
pttButton.disabled = !profile.pushToTalk;
setConnection('Connected', 'online');
setModel(profile.waiting);
appendLog('session started');
@@ -698,8 +858,9 @@
cameraPreview.srcObject = cameraStream;
cameraPreview.style.display = '';
await cameraPreview.play().catch(() => {});
// Official omni-duplex cadence: one JPEG (quality 0.7) per ~1 s chunk,
// no client-side resize (the server normalizes at scale_resolution=448).
// JPEG quality 0.7; no client-side resize (server scale_resolution=448).
// Interval is per-profile: AURA 2 fps, MiniCPM/Qwen stay at 1 s.
const cameraIntervalMs = profile.cameraIntervalMs || 1000;
const captureCameraFrame = () => {
if (!cameraStream || cameraPreview.videoWidth === 0) return;
const scale = profile.cameraMaxDimension
@@ -712,14 +873,17 @@
if (connectionReady && socket?.readyState === WebSocket.OPEN) {
for (const event of profile.imageMessages(frame)) socket.send(JSON.stringify(event));
}
} else cameraPendingFrame = frame;
} else {
cameraPendingFrame = frame;
if (profile.stickyCamera) cameraLastFrame = frame;
}
};
// Do not make the first spoken turn race a one-second timer.
captureCameraFrame();
cameraTimer = window.setInterval(captureCameraFrame, 1000);
cameraTimer = window.setInterval(captureCameraFrame, cameraIntervalMs);
cameraButton.textContent = 'Camera off';
cameraButton.classList.add('is-active');
appendLog('camera on (1 fps omni frames)');
appendLog(`camera on (${(1000 / cameraIntervalMs).toFixed(0)} fps omni frames)`);
}
function stopCamera() {
@@ -730,6 +894,7 @@
}
cameraStream = null;
cameraPendingFrame = null;
cameraLastFrame = null;
cameraPreview.srcObject = null;
cameraPreview.style.display = 'none';
cameraButton.textContent = 'Camera';
@@ -778,9 +943,18 @@
playbackComplete = true;
turnSubmitted = false;
assistantTextChannel = null;
assistantRowByResponse.clear();
lastClosedAssistantText = '';
sendTurnButton.disabled = true;
pttHeld = false;
pttButton.disabled = true;
pttButton.classList.remove('is-active');
pttButton.textContent = 'Hold to talk';
running = false;
assistantActive = false;
visionFollowQueue = [];
visionTurnLocked = false;
suppressedPlaybackId = null;
pendingCapture = [];
if (sendTimer !== null) clearInterval(sendTimer);
if (clockTimer !== null) clearInterval(clockTimer);
@@ -854,6 +1028,23 @@
setModel('Thinking');
appendLog('turn submitted');
});
const pttDown = (event) => {
event.preventDefault();
setPttHeld(true);
};
const pttUp = (event) => {
event.preventDefault();
setPttHeld(false);
};
pttButton.addEventListener('pointerdown', pttDown);
pttButton.addEventListener('pointerup', pttUp);
pttButton.addEventListener('pointercancel', pttUp);
pttButton.addEventListener('pointerleave', () => {
if (pttHeld) setPttHeld(false);
});
window.addEventListener('pointerup', () => {
if (pttHeld) setPttHeld(false);
});
muteButton.addEventListener('click', toggleMute);
clearLogButton.addEventListener('click', () => {
eventLog.textContent = '';
examples/online_serving/realtime_web/app/static/profiles/aura_ptt.js
+79
View File
@@ -0,0 +1,79 @@
(function (root) {
'use strict';
const profiles = root.OmniRealtimeProfiles;
profiles['aura-ptt'] = () => ({
title: 'AURA Realtime',
eyebrow: 'Push-to-talk duplex',
policy: 'Hold to talk • release to commit',
description:
'Hold to talk, release to send. Camera stays on whether you speak or not. '
+ 'Holding the button stops local playback.',
waiting: 'Ready',
camera: true,
// Display only. Shared CSS puts .camera-preview-large on its own row.
// Other profiles keep the 96×72 preview in index.html.
cameraPreviewLarge: true,
playbackAck: true,
clientCommit: false,
pushToTalk: true,
stickyCamera: true,
// OmniInteract / Native AURA clock: 2 fps. Other profiles omit this and stay at 1 s.
cameraIntervalMs: 500,
visionFollowWhileSpeaking: true,
deduplicateTranscript: true,
halfDuplex: false,
closeSession: true,
reconnectEachTurn: false,
readyEvent: 'session.updated',
instructions: true,
sendIntervalMs: 200,
presets: {
omni: 'You are receiving a live video stream where the final frame is the present moment. Respond only when a response is needed based on the user\'s message or the visual context. Otherwise, output `<|silent|>` to signify silence.',
},
url(config, location) {
const url = profiles.url(config, location);
url.searchParams.set('duplex', '1');
url.searchParams.set('model', config.model || 'aurateam/AURA');
return url.toString();
},
initialMessages(config, instructions) {
const session = {
modalities: ['audio', 'text'],
extra_body: { auto_response: true },
};
if (instructions) session.instructions = instructions;
return [{ type: 'session.update', session }];
},
append(audio, frame, opts) {
const isSpeech = !opts || opts.isSpeech !== false;
const event = {
type: 'input_audio_buffer.append',
audio,
format: 'pcm16',
sample_rate_hz: 16000,
is_speech: isSpeech,
};
const frames = Array.isArray(frame) ? frame.filter(Boolean) : (frame ? [frame] : []);
if (frames.length) event.video_frames = frames;
return event;
},
commitMessages: () => [{ type: 'input_audio_buffer.commit', create_response: true }],
ack(responseId, playedMs) {
return {
type: 'playback.ack',
response_id: responseId,
item_id: `item_${responseId}`,
played_ms: playedMs,
committed_ms: playedMs,
};
},
mapEvent(event) {
if (event.type === 'response.listen') return { kind: 'listen' };
if (event.type === 'playback.acknowledged') {
return { kind: 'ack', committedMs: (event.event || event).committed_ms || 0 };
}
return profiles.event(event);
},
connectionHint: 'Check the AURA duplex serve (run_duplex_smoke_serve.sh) and --ws-backend port.',
});
})(globalThis);
examples/online_serving/realtime_web/app/static/styles.css
+12
View File
@@ -71,6 +71,18 @@ main { width: min(1040px, calc(100% - 40px)); margin: 28px auto 48px; }
}
.call-actions { display: flex; gap: 8px; }
/* AURA only: class is added solely by the aura-ptt profile. */
.call-bar:has(.camera-preview-large) .call-actions { grid-column: 1 / -1; }
.call-actions:has(.camera-preview-large) { flex-wrap: wrap; align-items: center; }
.call-actions:has(.camera-preview-large) .camera-preview-large {
order: -1;
flex: 0 0 100%;
width: 100% !important;
height: auto !important;
max-height: 480px;
aspect-ratio: 4 / 3;
}
.call-actions:has(.camera-preview-large) button { flex: 0 0 auto; }
button { border-radius: 5px; border: 1px solid transparent; cursor: pointer; }
button:disabled { opacity: .46; cursor: default; }
.primary { padding: 10px 15px; color: #ffffff; background: #16834b; font-weight: 700; }
examples/online_serving/realtime_web/server.py
+14 -6
View File
@@ -76,15 +76,20 @@ def build_app(
profile: str = "minicpm-native",
adapter: str = "stt",
) -> FastAPI:
if profile not in {"minicpm-native", "qwen3-turn"}:
if profile not in {"minicpm-native", "qwen3-turn", "aura-ptt"}:
raise ValueError(f"Unknown profile: {profile}")
if adapter not in {"stt", "vad"}:
raise ValueError(f"Unknown adapter: {adapter}")
if profile == "qwen3-turn" and ref_audio:
raise ValueError("Qwen3 profile does not accept --ref-audio")
if profile == "aura-ptt" and ref_audio:
raise ValueError("AURA profile does not accept --ref-audio")
native = profile == "minicpm-native"
ws_backend = ws_backend or ("ws://127.0.0.1:8099" if native else "ws://127.0.0.1:8091")
model = model or ("openbmb/MiniCPM-o-4_5" if native else "Qwen/Qwen3-Omni-30B-A3B-Instruct")
aura = profile == "aura-ptt"
ws_backend = ws_backend or ("ws://127.0.0.1:8099" if native or aura else "ws://127.0.0.1:8091")
model = model or (
"openbmb/MiniCPM-o-4_5" if native else "aurateam/AURA" if aura else "Qwen/Qwen3-Omni-30B-A3B-Instruct"
)
app = FastAPI(title="Omni Realtime Web UI")
index_path = APP_DIR / "index.html"
app_version_hash = hashlib.sha256()
@@ -182,7 +187,7 @@ def main(default_profile: str = "minicpm-native") -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--host", default="0.0.0.0")
parser.add_argument("--port", type=int, default=7862)
parser.add_argument("--profile", choices=("minicpm-native", "qwen3-turn"), default=default_profile)
parser.add_argument("--profile", choices=("minicpm-native", "qwen3-turn", "aura-ptt"), default=default_profile)
parser.add_argument("--ws-backend", "--backend", dest="ws_backend")
mode = parser.add_mutually_exclusive_group()
mode.add_argument("--stt", dest="adapter", action="store_const", const="stt")
@@ -202,12 +207,15 @@ def main(default_profile: str = "minicpm-native") -> None:
)
args = parser.parse_args()
native = args.profile == "minicpm-native"
aura = args.profile == "aura-ptt"
if native and not args.ref_audio:
parser.error("--ref-audio is required for the minicpm-native profile")
if not native and args.ref_audio:
parser.error("--ref-audio is only supported by minicpm-native")
args.model = args.model or ("openbmb/MiniCPM-o-4_5" if native else "Qwen/Qwen3-Omni-30B-A3B-Instruct")
args.ws_backend = args.ws_backend or ("ws://127.0.0.1:8099" if native else "ws://127.0.0.1:8091")
args.model = args.model or (
"openbmb/MiniCPM-o-4_5" if native else "aurateam/AURA" if aura else "Qwen/Qwen3-Omni-30B-A3B-Instruct"
)
args.ws_backend = args.ws_backend or ("ws://127.0.0.1:8099" if native or aura else "ws://127.0.0.1:8091")
logging.basicConfig(level=logging.INFO)
uvicorn.run(
tests/benchmarks/test_aura_omniinteract_clock.py
+157
View File
@@ -0,0 +1,157 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import asyncio
from pathlib import Path
import pytest
from vllm_omni.benchmarks.omniinteract import _session_capabilities, _wants_annotation_video_clock
from vllm_omni.model_executor.models.aura_omni.benchmarks.omniinteract_clock import (
stream_annotation_clock,
wants_annotation_video_clock,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
class _Events:
def __init__(self, events):
self.events = events
class _Client:
def __init__(self, capabilities):
self.events = _Events([{"type": "session.created", "session": {"capabilities": capabilities}}])
self._client = None
def test_annotation_clock_follows_capabilities_not_model_name():
aura = {"required_input_modalities": ["video"], "supports_turn_commit_only": True}
minicpm = {"required_input_modalities": ["audio"], "supports_turn_commit_only": False}
audio_commit = {"required_input_modalities": ["audio"], "supports_turn_commit_only": True}
assert wants_annotation_video_clock(aura)
assert not wants_annotation_video_clock(minicpm)
assert not wants_annotation_video_clock(audio_commit)
assert not wants_annotation_video_clock(None)
assert _wants_annotation_video_clock(_Client(minicpm)) is False
assert _wants_annotation_video_clock(_Client(aura)) is True
assert _session_capabilities(_Client(aura))["required_input_modalities"] == ["video"]
def test_annotation_clock_commits_only_the_due_wav(monkeypatch, tmp_path: Path):
video = tmp_path / "videos" / "0002.mp4"
video.parent.mkdir()
video.write_bytes(b"")
sent = []
class Client:
async def send(self, event):
sent.append(event)
async def commit(self):
sent.append({"type": "commit"})
class Playback:
async def acknowledge(self, client):
del client
monkeypatch.setattr(
"vllm_omni.model_executor.models.aura_omni.benchmarks.omniinteract_clock.load_annotation_utterances",
lambda path: [{"at_sec": 0.0, "pcm": b"\x01\x00" * 160, "sent": False, "qa_index": 0}],
)
chunks, frames, _, _ = asyncio.run(
stream_annotation_clock(
Client(),
["frame-a", "frame-b"],
Playback(),
video_path=video,
fps=2.0,
)
)
assert (chunks, frames) == (2, 2)
assert sent[0]["is_speech"] is True
assert sent[0]["video_frames"] == ["frame-a"]
assert sent[1] == {"type": "commit"}
# The trailing silent frame is one vision turn, committed after the clip.
assert sent[2]["is_speech"] is False
assert sent[2]["video_frames"] == ["frame-b"]
assert sent[3] == {"type": "commit"}
assert sent.count({"type": "commit"}) == 2
def test_annotation_clock_commits_silent_frames_in_pairs(monkeypatch, tmp_path: Path):
video = tmp_path / "videos" / "0002.mp4"
video.parent.mkdir()
video.write_bytes(b"")
sent = []
class Client:
async def send(self, event):
sent.append(event)
async def commit(self):
sent.append({"type": "commit"})
class Playback:
async def acknowledge(self, client):
del client
monkeypatch.setattr(
"vllm_omni.model_executor.models.aura_omni.benchmarks.omniinteract_clock.load_annotation_utterances",
lambda path: [{"at_sec": 100.0, "pcm": b"\x01\x00" * 160, "sent": False, "qa_index": 0}],
)
asyncio.run(
stream_annotation_clock(
Client(),
["f1", "f2", "f3", "f4"],
Playback(),
video_path=video,
fps=2.0,
)
)
vision = [event for event in sent if event.get("is_speech") is False]
assert [event["video_frames"] for event in vision] == [["f1", "f2"], ["f3", "f4"]]
assert sent.count({"type": "commit"}) == 2
def test_annotation_clock_holds_vision_while_a_response_is_open(monkeypatch, tmp_path: Path):
video = tmp_path / "videos" / "0002.mp4"
video.parent.mkdir()
video.write_bytes(b"")
sent = []
class Events:
def __init__(self):
self.events = [{"type": "response.created", "response": {"id": "resp-open"}}]
class Client:
def __init__(self):
self.events = Events()
async def send(self, event):
sent.append(event)
async def commit(self):
sent.append({"type": "commit"})
class Playback:
async def acknowledge(self, client):
del client
monkeypatch.setattr(
"vllm_omni.model_executor.models.aura_omni.benchmarks.omniinteract_clock.load_annotation_utterances",
lambda path: [{"at_sec": 100.0, "pcm": b"\x01\x00" * 160, "sent": False, "qa_index": 0}],
)
asyncio.run(
stream_annotation_clock(
Client(),
["f1", "f2", "f3", "f4"],
Playback(),
video_path=video,
fps=2.0,
)
)
assert sent == []
tests/config/test_environment_variables.py
+2 -2
View File
@@ -167,7 +167,7 @@ def test_inventory_matches_reviewed_snapshot_counts():
EnvironmentVariableCategory.PUBLIC_OMNI: 29,
EnvironmentVariableCategory.INHERITED_VLLM: 20,
EnvironmentVariableCategory.PLATFORM_EXTERNAL: 27,
EnvironmentVariableCategory.MODEL_SPECIFIC: 57,
EnvironmentVariableCategory.MODEL_SPECIFIC: 59,
EnvironmentVariableCategory.BENCHMARK_TRANSITIONAL: 21,
EnvironmentVariableCategory.INTERNAL: 2,
}
@@ -181,7 +181,7 @@ def test_inventory_matches_reviewed_snapshot_counts():
ModelEnvironmentVariableDisposition.PROMOTE: 31,
ModelEnvironmentVariableDisposition.REQUEST_SCOPE: 5,
ModelEnvironmentVariableDisposition.EXTERNAL: 0,
ModelEnvironmentVariableDisposition.INTERNALIZE: 14,
ModelEnvironmentVariableDisposition.INTERNALIZE: 16,
ModelEnvironmentVariableDisposition.DEPRECATE_REMOVE: 7,
}
tests/engine/duplex/test_aura_resumable_hooks.py
+725
View File
@@ -0,0 +1,725 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Public duplex hook: resumable defaults True; ephemeral ids when False."""
from __future__ import annotations
import pytest
from vllm.sampling_params import SamplingParams
from vllm_omni.engine.duplex.config import DuplexCapabilities
from vllm_omni.engine.duplex.contracts import (
DuplexFence,
DuplexStageRequestContext,
DuplexStageSubmission,
duplex_ephemeral_stage_request_id,
)
from vllm_omni.engine.duplex.session.manager import DuplexSessionManager
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def test_submission_resumable_default_preserves_minicpm() -> None:
ctx = DuplexStageRequestContext(
request_id="r",
session_id="s",
fence=DuplexFence("s"),
stage_id=0,
final_stage_id=2,
config_generation=0,
sampling_params=(SamplingParams(max_tokens=1),),
)
submission = DuplexStageSubmission(
context=ctx,
prompt={"prompt_token_ids": [1, 2]},
already_submitted=False,
)
assert submission.resumable is True
def test_stage_request_id_respects_resumable_flag() -> None:
fence = DuplexFence("sess", epoch=3, turn_id=9)
resumable_id = DuplexSessionManager.stage_request_id(fence, stage_id=0, resumable=True)
ephemeral_id = DuplexSessionManager.stage_request_id(fence, stage_id=0, resumable=False)
assert ephemeral_id == duplex_ephemeral_stage_request_id(fence, stage_id=0)
assert resumable_id != ephemeral_id
assert ephemeral_id.endswith("stage0-turn9")
from vllm_omni.engine.duplex.contracts import duplex_turn_id_from_request_id
assert duplex_turn_id_from_request_id(ephemeral_id) == 9
assert duplex_turn_id_from_request_id(resumable_id) is None
def test_capabilities_expose_concurrent_turn_requests_default_false() -> None:
caps = DuplexCapabilities()
assert caps.supports_concurrent_turn_requests is False
assert caps.allows_video_without_audio() is False
assert caps.required_input_modalities == frozenset({"audio"})
assert caps.optional_input_modalities == frozenset({"video"})
assert caps.supports_core_resumable_request is False
payload = caps.as_dict()
assert payload["supports_concurrent_turn_requests"] is False
assert payload["required_input_modalities"] == ["audio"]
assert payload["optional_input_modalities"] == ["video"]
def test_next_commit_allowed_soft_opens_on_r4_release() -> None:
from types import SimpleNamespace
from vllm_omni.engine.duplex.session import helpers
caps = DuplexCapabilities(supports_concurrent_turn_requests=True)
session = SimpleNamespace(active_response_id="resp", capabilities=caps)
tasks = SimpleNamespace(
active_response_task=None,
has_response_bound_append_tasks=lambda: False,
)
# Monkeypatch assistant_playback_active via response_in_progress path: active_response_id set.
assert helpers.response_in_progress(session, tasks) is True
assert helpers.next_commit_allowed(session, tasks, concurrent_turn_requests_released=False) is False
assert helpers.next_commit_allowed(session, tasks, concurrent_turn_requests_released=True) is True
caps_off = DuplexCapabilities(supports_concurrent_turn_requests=False)
session_off = SimpleNamespace(active_response_id="resp", capabilities=caps_off)
assert helpers.next_commit_allowed(session_off, tasks, concurrent_turn_requests_released=True) is False
def test_active_response_accepts_own_turn_when_concurrent_turn_requests() -> None:
"""Approach A: each response owns its turn; draining TTS uses request→response map."""
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
session = DuplexEngineSession(
session_id="s",
config=DuplexSessionConfig(model="m", modalities=["text"]),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True),
)
session.begin_response(turn_id=3)
assert session.active_response_accepts_model_turn(3)
assert not session.active_response_accepts_model_turn(4)
assert not session.active_response_accepts_model_turn(2)
session.bind_draining_request("req-old", "resp-old")
assert session.response_id_for_request("req-old") == "resp-old"
assert session.is_draining_request("req-old")
# Ending the active response must not wipe older draining bindings.
session.bind_draining_request("req-newer-tts", session.active_response_id or "resp-active")
ended = session.active_response_id
session.end_response(commit_text=False)
assert session.is_draining_request("req-old")
assert session.response_id_for_request("req-old") == "resp-old"
assert not session.is_draining_request("req-newer-tts")
session.clear_draining_for_response("resp-old")
assert not session.is_draining_request("req-old")
del ended
strict = DuplexEngineSession(
session_id="s2",
config=DuplexSessionConfig(model="m", modalities=["text"]),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=False),
)
strict.begin_response(turn_id=3)
assert strict.active_response_accepts_model_turn(3)
assert not strict.active_response_accepts_model_turn(4)
def test_draining_request_exempt_from_completed_turn_filter() -> None:
"""Draining ownership must win over the model_turn_id < turn_id late-audio guard."""
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
session = DuplexEngineSession(
session_id="s-drain",
config=DuplexSessionConfig(model="m", modalities=["text", "audio"]),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True),
)
session.begin_response(turn_id=1)
r1 = session.active_response_id
assert r1 is not None
session.bind_draining_request("req-r1-tts", r1)
# Overlapping R2 becomes active, then finishes — must not wipe R1 draining.
session.begin_response(turn_id=2)
session.end_response(commit_text=False)
session.turn_id = 2
assert session.active_response_id is None
assert session.is_draining_request("req-r1-tts")
assert session.response_id_for_request("req-r1-tts") == r1
draining_response_id = session.response_id_for_request("req-r1-tts")
model_turn_id = 1
drop = (
draining_response_id is None
and session.active_response_id is None
and model_turn_id is not None
and model_turn_id < session.turn_id
)
assert drop is False
def test_ephemeral_turn_id_parser_accepts_stage_turn_ids() -> None:
from vllm_omni.engine.duplex.contracts import duplex_turn_id_from_request_id
assert duplex_turn_id_from_request_id("duplex-s.x.e.0.r.stage0-turn12") == 12
assert duplex_turn_id_from_request_id("duplex-s.x.e.0.r.stage2-turn3") == 3
assert duplex_turn_id_from_request_id("duplex-s.x.e.0.r.stage0_t9") is None
assert duplex_turn_id_from_request_id("duplex-s.x.e.0.r.stage0") is None
def test_same_turn_request_ids_skip_other_turns_and_epochs() -> None:
from vllm_omni.engine.duplex.contracts import duplex_same_turn_request_ids
stage3 = "duplex-s.x.e.0.r.stage3-turn4"
found = duplex_same_turn_request_ids(
stage3,
[
"duplex-s.x.e.0.r.stage0-turn4",
"duplex-s.x.e.0.r.stage1-turn4",
"duplex-s.x.e.0.r.stage2-turn4",
stage3,
"duplex-s.x.e.0.r.stage2-turn5",
"duplex-s.x.e.1.r.stage0-turn4",
],
)
assert found == [
"duplex-s.x.e.0.r.stage0-turn4",
"duplex-s.x.e.0.r.stage1-turn4",
"duplex-s.x.e.0.r.stage2-turn4",
]
from vllm_omni.engine.duplex.contracts import duplex_session_id_from_request_id
fence_session = "sess id"
import base64
encoded = base64.urlsafe_b64encode(fence_session.encode()).decode("ascii").rstrip("=")
assert duplex_session_id_from_request_id(f"duplex-s.{encoded}.e.0.r.stage1-turn2") == fence_session
assert duplex_session_id_from_request_id("not-a-request") is None
def test_stale_keys_skip_already_draining_request_ids() -> None:
"""Rebinding must not move an older draining Stage2/3 id onto the newer response."""
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
session = DuplexEngineSession(
session_id="s-stale",
config=DuplexSessionConfig(model="m", modalities=["text", "audio"]),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True),
)
session.begin_response(turn_id=1)
r1 = session.active_response_id
assert r1 is not None
session.bind_draining_request("req-r1-talker", r1)
session.begin_response(turn_id=2)
r2 = session.active_response_id
assert r2 is not None and r2 != r1
# Simulate the concurrent-turn rebind loop: already-draining ids keep R1.
for rid in ("req-r1-talker", "req-r2-talker"):
if session.is_draining_request(rid):
continue
session.bind_draining_request(rid, r2)
assert session.response_id_for_request("req-r1-talker") == r1
assert session.response_id_for_request("req-r2-talker") == r2
def test_end_response_clears_only_own_draining_entries() -> None:
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
session = DuplexEngineSession(
session_id="s-end",
config=DuplexSessionConfig(model="m", modalities=["text", "audio"]),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True),
)
session.begin_response(turn_id=1)
r1 = session.active_response_id
session.bind_draining_request("req-r1", r1)
session.begin_response(turn_id=2)
r2 = session.active_response_id
session.bind_draining_request("req-r2", r2)
session.end_response(commit_text=False)
assert session.is_draining_request("req-r1")
assert session.response_id_for_request("req-r1") == r1
assert not session.is_draining_request("req-r2")
def test_on_stage_failure_resolves_draining_response_before_active() -> None:
"""Mirrors runner.on_stage_failure: draining request_id owns the failed done."""
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
session = DuplexEngineSession(
session_id="s-fail",
config=DuplexSessionConfig(model="m", modalities=["text", "audio"]),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True),
)
session.begin_response(turn_id=1)
r1 = session.active_response_id
assert r1 is not None
session.bind_draining_request("req-r1-talker", r1)
session.begin_response(turn_id=2)
r2 = session.active_response_id
assert r2 is not None and r2 != r1
request_id = "req-r1-talker"
draining_response_id = (
session.response_id_for_request(request_id) if session.is_draining_request(request_id) else None
)
response_id = draining_response_id or session.active_response_id
assert response_id == r1
assert response_id != r2
session.clear_draining_for_response(r1)
assert not session.is_draining_request(request_id)
assert session.active_response_id == r2
def test_concurrent_turn_requests_released_resets_when_new_stage0_binds() -> None:
"""After R4 release, a new ephemeral Stage0 bind must clear the gate flag."""
from types import SimpleNamespace
run = SimpleNamespace(concurrent_turn_requests_released=True)
# Simulate the concurrent-turn rebind arm in model_channel._append_via_data_plane.
concurrent_turn = True
if concurrent_turn:
run.concurrent_turn_requests_released = False
assert run.concurrent_turn_requests_released is False
def test_draining_stage_ids_come_from_the_plugin() -> None:
from types import SimpleNamespace
from vllm_omni.engine.duplex.session.model_channel import ModelChannel
channel = ModelChannel.__new__(ModelChannel)
channel._ctx = SimpleNamespace(
plugin=SimpleNamespace(draining_stage_ids=lambda *, stage_count: {stage_count - 1}),
stage_port=SimpleNamespace(stage_count=5),
)
assert channel._draining_stage_ids() == frozenset({4})
channel._ctx.plugin = SimpleNamespace()
assert channel._draining_stage_ids() == frozenset()
def test_draining_empty_eos_completes_owning_response_before_shortcut() -> None:
"""R2 finishing first must not swallow R1's empty Stage3 EOS."""
import asyncio
from types import SimpleNamespace
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
from vllm_omni.engine.duplex.session.model_channel import ModelChannel
session = DuplexEngineSession(
session_id="s-empty-eos",
config=DuplexSessionConfig(model="m", modalities=["text", "audio"], extra_body={"auto_response": True}),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True, supports_core_resumable_request=False),
)
session.begin_response(turn_id=1)
r1 = session.active_response_id
assert r1 is not None
r1_request = "duplex-s.x.e.0.r.stage3-turn1"
still_running = "duplex-s.x.e.0.r.stage1-turn2"
session.bind_draining_request(r1_request, r1)
session.begin_response(turn_id=2)
session.end_response(commit_text=False)
session.turn_id = 2
assert session.active_response_id is None
class _Plane:
def is_terminal(self, request_id: str | None) -> bool:
del request_id
return False
def close_stream(self, request_id: str) -> None:
del request_id
def mark_terminal(self, request_id: str) -> None:
del request_id
class _Port:
def __init__(self) -> None:
self.cleanups: list[list[str]] = []
async def cleanup(self, request_ids: list[str], *, abort: bool = False) -> None:
del abort
self.cleanups.append(list(request_ids))
class _Out:
def __init__(self) -> None:
self.events: list[dict[str, object]] = []
def auto_responds(self) -> bool:
return True
def emit(self, payload: dict[str, object]) -> None:
self.events.append(payload)
port = _Port()
out = _Out()
async def _noop(*_args: object, **_kwargs: object) -> None:
return None
async def _schedule(*_args: object, **_kwargs: object) -> bool:
return False
channel = ModelChannel(
SimpleNamespace(
session=session,
plugin=SimpleNamespace(data_plane=_Plane()),
stage_port=port,
model_state=SimpleNamespace(clear_continuation=lambda: None),
services=SimpleNamespace(spawn=lambda *_a, **_k: None),
),
out,
close_from_runtime=_noop,
schedule_silence_continuation=_schedule,
abort_request=_noop,
)
asyncio.run(
channel._send_one_model_output_event(
{
"data_plane_request_id": r1_request,
"end_of_turn": True,
"model_turn_id": 1,
"text": "",
"stage_role": "tts",
}
)
)
done = [event for event in out.events if event.get("type") == "response.done"]
assert done and done[-1].get("response_id") == r1
assert port.cleanups == [[r1_request]]
assert all(still_running not in ids for ids in port.cleanups)
assert not session.is_draining_request(r1_request)
def test_draining_completion_drops_finished_response_books_only() -> None:
"""R1 drain finish drops R1 books and leaves the live R2 snapshot."""
import asyncio
from types import SimpleNamespace
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
from vllm_omni.engine.duplex.session.model_channel import ModelChannel
session = DuplexEngineSession(
session_id="s-drain-books",
config=DuplexSessionConfig(model="m", modalities=["text", "audio"], extra_body={"auto_response": True}),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True, supports_core_resumable_request=False),
)
session.begin_response(turn_id=1)
r1 = session.active_response_id
assert r1 is not None
session.append_assistant_text("from-r1")
session.snapshot_active_response_for_drain()
r1_request = "duplex-s.x.e.0.r.stage3-turn1"
session.bind_draining_request(r1_request, r1)
session.begin_response(turn_id=2)
r2 = session.active_response_id
assert r2 is not None and r2 != r1
session.append_assistant_text("from-r2")
session.snapshot_active_response_for_drain()
class _Plane:
def is_terminal(self, request_id: str | None) -> bool:
del request_id
return False
def close_stream(self, request_id: str) -> None:
del request_id
def mark_terminal(self, request_id: str) -> None:
del request_id
class _Port:
async def cleanup(self, request_ids: list[str], *, abort: bool = False) -> None:
del request_ids, abort
class _Out:
def __init__(self) -> None:
self.events: list[dict[str, object]] = []
def auto_responds(self) -> bool:
return True
def emit(self, payload: dict[str, object]) -> None:
self.events.append(payload)
out = _Out()
async def _noop(*_args: object, **_kwargs: object) -> None:
return None
async def _schedule(*_args: object, **_kwargs: object) -> bool:
return False
channel = ModelChannel(
SimpleNamespace(
session=session,
plugin=SimpleNamespace(data_plane=_Plane()),
stage_port=_Port(),
model_state=SimpleNamespace(clear_continuation=lambda: None),
services=SimpleNamespace(spawn=lambda *_a, **_k: None),
),
out,
close_from_runtime=_noop,
schedule_silence_continuation=_schedule,
abort_request=_noop,
)
asyncio.run(
channel._send_one_model_output_event(
{
"data_plane_request_id": r1_request,
"end_of_turn": True,
"model_turn_id": 1,
"text": "",
"stage_role": "tts",
}
)
)
assert r1 not in session._conversation.assistant_response_snapshots
assert r1 not in session._playback.by_response
assert f"item_{r1}" not in session._conversation.history_item_placeholders
assert r2 in session._conversation.assistant_response_snapshots
assert session.active_response_id == r2
done = [event for event in out.events if event.get("type") == "response.done"]
assert done and done[-1].get("response_id") == r1
def test_stale_continue_does_not_close_the_new_response() -> None:
import asyncio
from types import SimpleNamespace
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
from vllm_omni.engine.duplex.session.model_channel import ModelChannel
session = DuplexEngineSession(
session_id="s-stale-continue",
config=DuplexSessionConfig(model="m", modalities=["text", "audio"], extra_body={"auto_response": True}),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True, supports_core_resumable_request=False),
)
session.begin_response(turn_id=2)
live = session.active_response_id
session.epoch = 1
class _Out:
def __init__(self) -> None:
self.events: list[dict[str, object]] = []
def auto_responds(self) -> bool:
return True
def emit(self, payload: dict[str, object]) -> None:
self.events.append(payload)
out = _Out()
cleared: list[bool] = []
async def _noop(*_args: object, **_kwargs: object) -> None:
return None
async def _schedule(*_args: object, **_kwargs: object) -> bool:
return False
channel = ModelChannel(
SimpleNamespace(
session=session,
plugin=SimpleNamespace(data_plane=SimpleNamespace()),
stage_port=SimpleNamespace(),
model_state=SimpleNamespace(clear_continuation=lambda: cleared.append(True)),
services=SimpleNamespace(spawn=lambda *_a, **_k: None),
run=SimpleNamespace(closing=False),
),
out,
close_from_runtime=_noop,
schedule_silence_continuation=_schedule,
abort_request=_noop,
)
asyncio.run(channel.maybe_continue_response(expected_epoch=0))
assert session.active_response_id == live
assert out.events == []
assert cleared == []
session.bind_draining_request("duplex-s.x.e.1.r.stage3-turn2", live or "")
asyncio.run(channel.maybe_continue_response(expected_epoch=1))
assert session.active_response_id == live
assert out.events == []
session.clear_draining_requests()
asyncio.run(channel.maybe_continue_response(expected_epoch=1))
assert session.active_response_id is None
assert [event.get("type") for event in out.events] == ["response.done"]
assert out.events[0].get("response_id") == live
def test_silent_listen_with_continuation_releases_ephemeral_request() -> None:
import asyncio
from collections.abc import Coroutine
from types import SimpleNamespace
from typing import Any
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
from vllm_omni.engine.duplex.session.model_channel import ModelChannel
session = DuplexEngineSession(
session_id="s-silent-continue",
config=DuplexSessionConfig(model="m", modalities=["text", "audio"]),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True, supports_core_resumable_request=False),
)
session.begin_response(turn_id=1)
response_id = session.active_response_id
request_id = "duplex-s.x.e.0.r.stage1-turn1"
session.bind_request(request_id)
class _Plane:
def __init__(self) -> None:
self.terminal: list[str] = []
def is_terminal(self, request_id: str | None) -> bool:
return request_id in self.terminal
def mark_terminal(self, request_id: str) -> None:
self.terminal.append(request_id)
class _Port:
def __init__(self) -> None:
self.cleanups: list[list[str]] = []
async def cleanup(self, request_ids: list[str], *, abort: bool = False) -> None:
del abort
self.cleanups.append(list(request_ids))
class _Out:
def __init__(self) -> None:
self.events: list[dict[str, object]] = []
def auto_responds(self) -> bool:
return False
def emit(self, payload: dict[str, object]) -> None:
self.events.append(payload)
port = _Port()
out = _Out()
spawned: list[Coroutine[Any, Any, None]] = []
async def _noop(*_args: object, **_kwargs: object) -> None:
return None
async def _schedule(*_args: object, **_kwargs: object) -> bool:
return False
channel = ModelChannel(
SimpleNamespace(
session=session,
plugin=SimpleNamespace(data_plane=_Plane()),
stage_port=port,
model_state=SimpleNamespace(
continuation_owner_id=None, continuation_units=0, clear_continuation=lambda: None
),
services=SimpleNamespace(spawn=lambda coro, **_k: spawned.append(coro)),
run=SimpleNamespace(closing=False),
),
out,
close_from_runtime=_noop,
schedule_silence_continuation=_schedule,
abort_request=_noop,
)
asyncio.run(
channel._send_one_model_output_event(
{
"data_plane_request_id": request_id,
"is_listen": True,
"end_of_turn": True,
"model_turn_id": 1,
"reason": "model_listen",
}
)
)
assert port.cleanups == [[request_id]]
assert session.active_request_id is None
assert spawned, "silent listen with unused continuation must schedule maybe_continue"
asyncio.run(spawned[0])
assert port.cleanups == [[request_id]]
assert session.active_response_id is None
assert [event.get("type") for event in out.events] == ["response.listen", "response.done"]
assert out.events[0].get("response_id") == response_id
assert out.events[1].get("response_id") == response_id
def test_silent_listen_aborts_data_plane_request_with_full_id() -> None:
"""abort_data_plane_request must pass [request_id], not a bare str."""
import asyncio
from types import SimpleNamespace
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.session.engine_session import DuplexEngineSession
from vllm_omni.engine.duplex.session.model_channel import ModelChannel
session = DuplexEngineSession(
session_id="s-silent-abort",
config=DuplexSessionConfig(model="m", modalities=["text", "audio"]),
capabilities=DuplexCapabilities(supports_concurrent_turn_requests=True, supports_core_resumable_request=False),
)
request_id = "duplex-s.x.e.0.r.stage0-turn7"
session.bind_request(request_id)
aborts: list[list[str]] = []
class _Plane:
def is_terminal(self, request_id: str | None) -> bool:
return False
def mark_terminal(self, request_id: str) -> None:
del request_id
class _Port:
async def cleanup(self, request_ids: list[str], *, abort: bool = False) -> None:
del request_ids, abort
class _Out:
def auto_responds(self) -> bool:
return True
def emit(self, payload: dict[str, object]) -> None:
del payload
async def _noop(*_args: object, **_kwargs: object) -> None:
return None
async def _abort(request_ids: list[str], *, notify: bool = False) -> None:
del notify
aborts.append(list(request_ids))
channel = ModelChannel(
SimpleNamespace(
session=session,
plugin=SimpleNamespace(data_plane=_Plane()),
stage_port=_Port(),
model_state=SimpleNamespace(
continuation_owner_id=None, continuation_units=0, clear_continuation=lambda: None
),
services=SimpleNamespace(spawn=lambda coro, **_k: None),
run=SimpleNamespace(closing=False),
),
_Out(),
close_from_runtime=_noop,
schedule_silence_continuation=_noop,
abort_request=_abort,
)
asyncio.run(
channel._send_one_model_output_event(
{
"data_plane_request_id": request_id,
"is_listen": True,
"end_of_turn": True,
"model_turn_id": 1,
"reason": "model_listen",
"abort_data_plane_request": True,
}
)
)
assert aborts == [[request_id]]
assert all(len(item) > 1 for batch in aborts for item in batch), "must not split request id into chars"
tests/engine/duplex/test_engine_session.py
+105
View File
@@ -242,6 +242,19 @@ def test_history_commit_with_ack_only_policy_defers_unacknowledged_text():
assert f"item_{response_id}" in session.pending_history_item_ids
def test_history_commit_with_ack_only_policy_keeps_only_acknowledged_prefix():
session = _session(config=DuplexSessionConfig(playback_commit_policy="ack_only"))
session.begin_response()
session.append_assistant_text("hello world")
session.mark_audio_sent(duration_ms=1_000, text_chars=6)
session.mark_audio_sent(duration_ms=2_000, text_chars=11)
session.acknowledge_playback(played_ms=1_000, committed_ms=1_000)
committed = session.end_response(commit_text=True)
assert committed == {"role": "assistant", "content": "hello"}
@pytest.mark.parametrize("audio_complete", [False, True])
def test_unaligned_response_keeps_empty_turn_before_later_user_input(audio_complete):
session = _session(config=DuplexSessionConfig(playback_commit_policy="ack_only"))
@@ -469,6 +482,36 @@ def test_cancel_fence_releases_stage_requests_and_advances_identity():
session.cancel_fence(DuplexFence("sid-other", epoch=1, turn_id=0), DuplexFence("sid-cancel", epoch=2))
def test_request_resource_keys_are_stage_id_and_request_id():
session = _session("sid-keys")
session.bind_stage_request(0, "req-a", fence=session.fence)
session.bind_stage_request(1, "req-b", fence=session.fence)
session.bind_stage_request(2, "req-tts", fence=session.fence)
stale_keys = list(session.request_resources.keys())
stale_ids = list(dict.fromkeys(rid for _, rid in stale_keys))
assert stale_ids == ["req-a", "req-b", "req-tts"]
for sid, rid in stale_keys:
if sid < 2:
session.request_resources.pop((sid, rid), None)
assert session.resource_request_ids() == ["req-tts"]
with pytest.raises(TypeError, match="unhashable"):
dict.fromkeys(rid for _, rid in session.request_resources.items())
def test_release_resources_for_request_ids_keeps_other_ids_on_the_same_fence():
session = _session("sid-drain-release")
fence = session.fence
session.bind_stage_request(2, "drain", fence=fence)
session.bind_stage_request(3, "drain", fence=fence)
session.bind_stage_request(0, "live", fence=fence)
released = session.release_resources_for_request_ids(["drain"])
assert released == ["drain"]
assert session.resource_request_ids() == ["live"]
assert session.release_resources_for_request_ids([]) == []
# ---- public view ----
@@ -688,3 +731,65 @@ def test_minicpmo_native_capabilities_do_not_overclaim_single_session_deployment
assert caps["supports_multi_session"] is False
assert caps["supports_multi_session_same_replica"] is False
def test_draining_response_stays_ack_admissible_after_next_begin_response() -> None:
from vllm_omni.engine.duplex.events import ErrorEvent
from vllm_omni.engine.duplex.session.playback_ledger import apply_playback_ack
session = _session()
first = session.begin_response(turn_id=1)
session.append_assistant_text("hello")
session.mark_audio_sent(400, text_chars=5)
session.snapshot_active_response_for_drain()
second = session.begin_response(turn_id=2)
assert second != first
assert session.playback.sent_ms == 0
events = apply_playback_ack(
session,
{
"type": "playback.ack",
"response_id": first,
"item_id": f"item_{first}",
"played_ms": 100,
"committed_ms": 100,
},
)
assert not any(isinstance(event, ErrorEvent) and event.code == "playback_item_not_found" for event in events)
session.mark_audio_sent(900, text_chars=5, response_id=first)
assert session.playback_for_response(first).sent_ms == 900
assert session.playback.sent_ms == 0
def test_finished_drain_keeps_sent_audio_ackable() -> None:
from vllm_omni.engine.duplex.events import ErrorEvent
from vllm_omni.engine.duplex.session.playback_ledger import apply_playback_ack
session = _session()
first = session.begin_response(turn_id=1)
session.append_assistant_text("hello")
session.mark_audio_sent(400, text_chars=5)
session.snapshot_active_response_for_drain()
second = session.begin_response(turn_id=2)
session.release_finished_drain_response(first)
assert first in session._conversation.assistant_response_snapshots
assert f"item_{first}" in session._conversation.history_item_placeholders
events = apply_playback_ack(
session,
{
"type": "playback.ack",
"response_id": first,
"item_id": f"item_{first}",
"played_ms": 400,
"committed_ms": 400,
},
)
assert not any(isinstance(event, ErrorEvent) for event in events)
assert session.active_response_id == second
silent = session.begin_response(turn_id=3)
session.snapshot_active_response_for_drain()
session.begin_response(turn_id=4)
session.release_finished_drain_response(silent)
assert silent not in session._conversation.assistant_response_snapshots
assert f"item_{silent}" not in session._conversation.history_item_placeholders
tests/engine/duplex/test_input_transcribed.py
+49
View File
@@ -0,0 +1,49 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import pytest
from vllm_omni.engine.duplex.realtime_events import (
RealtimeProjectionState,
project_internal_event,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def test_input_transcribed_is_a_user_transcript_event() -> None:
events = project_internal_event(
RealtimeProjectionState(session_id="s"),
{"type": "input.transcribed", "transcript": " 你好 "},
)
assert len(events) == 1
assert events[0].wire_type == "conversation.item.input_audio_transcription.completed"
assert events[0].transcript == "你好"
def test_input_transcribed_updates_the_committed_item() -> None:
state = RealtimeProjectionState(session_id="s")
project_internal_event(
state,
{
"type": "input.committed",
"realtime_item_id": "item_user_1",
"is_speech": True,
"message": {"role": "user"},
},
)
events = project_internal_event(
state,
{"type": "input.transcribed", "transcript": "你好", "realtime_item_id": "item_user_1"},
)
assert events[0].item_id == "item_user_1"
item = state.conversation_items["item_user_1"]
assert item["content"][0]["transcript"] == "你好"
def test_blank_input_transcribed_emits_nothing() -> None:
events = project_internal_event(
RealtimeProjectionState(session_id="s"),
{"type": "input.transcribed", "transcript": " "},
)
assert events == []
tests/engine/duplex/test_overlap_policy.py
+33
View File
@@ -79,6 +79,39 @@ def test_an_explicit_client_overlap_action_requests_barge_in() -> None:
assert decision["reason"] == "client_overlap_action"
def test_turn_mode_keeps_silent_vision_follow_during_overlap() -> None:
"""Vision-follow is is_speech=False + a frame; must not drop while TTS plays."""
session = _session()
session.capabilities = type(session.capabilities)(
required_input_modalities=frozenset({"video"}),
optional_input_modalities=frozenset({"audio"}),
)
payload = {"format": "pcm16", "audio": _pcm16(0.0), "video_frames": ["frame"], "is_speech": False}
decision = overlap_policy.decide(session, {"is_speech": False}, payload, auto_responds=False)
assert decision["action"] == "listen"
assert decision["reason"] == "vision_follow"
assert decision["buffer_audio"] is True
assert decision["defer_runtime_append"] is False
assert decision["force_listen"] is False
def test_turn_mode_drops_silent_vision_without_capability() -> None:
session = _session()
assert session.capabilities.allows_video_without_audio() is False
payload = {"format": "pcm16", "audio": _pcm16(0.0), "video_frames": ["frame"], "is_speech": False}
decision = overlap_policy.decide(session, {"is_speech": False}, payload, auto_responds=False)
assert decision["action"] == "drop"
assert decision["reason"] == "silence_or_noise"
def test_turn_mode_still_drops_silent_overlap_without_vision() -> None:
session = _session()
payload = {"format": "pcm16", "audio": _pcm16(0.0), "is_speech": False}
decision = overlap_policy.decide(session, {"is_speech": False}, payload, auto_responds=False)
assert decision["action"] == "drop"
assert decision["reason"] == "silence_or_noise"
def test_barge_in_is_deferred_when_the_session_cannot_support_it() -> None:
session = _session()
session.capabilities = type(session.capabilities)(supports_barge_in=False)
tests/engine/duplex/test_session_manager.py
+41 -2
View File
@@ -196,7 +196,11 @@ class FakePlugin(DuplexModelPlugin):
def capabilities(self, *, max_sessions: int) -> DuplexCapabilities:
del max_sessions
return DuplexCapabilities(supports_input_append=True, supports_core_resumable_request=True)
# Resident Stage0 ids (MiniCPM-shaped); AURA opts out via supports_core_resumable_request=False.
return DuplexCapabilities(
supports_input_append=True,
supports_core_resumable_request=True,
)
def validate_client_extra_body(self, extra_body: object) -> None:
pass
@@ -409,7 +413,8 @@ async def test_open_answers_with_capabilities_and_emits_session_created() -> Non
assert result.session_id == "sid-open"
assert result.lease_generation == 0
assert result.capabilities == DuplexCapabilities(
supports_input_append=True, supports_core_resumable_request=True
supports_input_append=True,
supports_core_resumable_request=True,
)
assert result.public_session is not None
assert result.public_session["id"] == "sid-open"
@@ -1028,6 +1033,40 @@ async def test_append_bytes_are_reserved_at_admission_until_the_runner_dequeues(
gate.set()
async def test_append_admission_counts_audio_and_video_frame_bytes() -> None:
"""Manager reserves len(audio)+Σlen(frame); video bytes count toward the same limit."""
async with Harness.create(max_sessions=1, max_pending_input_bytes_per_session=20) as harness:
await harness.open("sid-av")
harness.events()
session = harness.session("sid-av")
runner = harness.manager.runners["sid-av"]
gate = asyncio.Event()
runner._mailbox.put_nowait(_Internal("wait", {"gate": gate}))
runner._on_internal = lambda item: gate.wait() # type: ignore[method-assign]
await asyncio.sleep(0)
# Default caps require audio; attach video to a non-empty audio unit.
frame_a = "aaaa"
frame_b = "bbbbbb"
audio = b"1234"
expected = len(audio) + len(frame_a) + len(frame_b)
harness.command(
"sid-av",
AppendAudio(audio=audio, video_frames=(frame_a, frame_b), event_id="evt-av"),
)
assert session.pending_input_bytes == expected
# Second append that would exceed the limit is backpressured.
harness.command(
"sid-av",
AppendAudio(audio=b"x" * 10, video_frames=("yyyyyyyyyy",), event_id="evt-over"),
)
errors = [event for event in harness.events("sid-av") if isinstance(event, ErrorEvent)]
assert [error.code for error in errors] == ["input_backpressure"]
assert errors[0].related_event_id == "evt-over"
assert session.pending_input_bytes == expected
gate.set()
async def test_expired_session_retains_the_admission_slot_until_cleanup_succeeds() -> None:
async with Harness.create(max_sessions=1, idle_ttl_s=1.0) as harness:
await harness.open("sid-expired")
tests/engine/duplex/test_session_runner.py
+257 -2
View File
@@ -180,7 +180,10 @@ class Harness:
segment_token_ids=tuple(segment_token_ids),
segment_output_metadata=dict(segment_output_metadata or {}),
)
return self.runner.on_stage_output(stage_id, output, metrics, request_id=output.request_id, context=context)
request_id = getattr(output, "request_id", None)
if not isinstance(request_id, str):
raise AssertionError("stage output is missing request_id")
return self.runner.on_stage_output(stage_id, output, metrics, request_id=request_id, context=context)
async def deliver_and_settle(self, output: object, **kwargs: Any) -> list[DuplexEvent]:
self.deliver(output, **kwargs)
@@ -520,6 +523,28 @@ async def test_commit_without_audio_is_rejected() -> None:
await close_harness(h)
@pytest.mark.asyncio
async def test_empty_silent_commit_acks_without_opening_response() -> None:
"""#3: empty is_speech=False commit keeps the silent-ack pair, even auto-respond."""
h = await open_harness(auto_response=True)
try:
# resolve_commit only admits empty silent commits after non-speech was seen.
projector = h.runner._require_projector()
projector.input_audio_buffer_had_non_speech = True
events = await h.run(commands.Commit())
assert "error" not in types(events), types(events)
assert "input_audio_buffer.committed" in types(events), types(events)
assert "response.listen" in types(events), types(events)
listen = next(event for event in events if event.type == "response.listen")
assert listen.details["reason"] == "silence_or_noise"
committed = next(event for event in events if event.type == "input_audio_buffer.committed")
assert committed.details.get("empty") is True
assert committed.details.get("no_response") is True
assert h.port.submissions == []
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_commit_projects_the_user_item_and_does_not_resubmit_consumed_audio() -> None:
h = await open_harness()
@@ -674,6 +699,45 @@ async def test_stale_epoch_output_is_dropped_after_barge_in() -> None:
await close_harness(h)
async def test_barge_in_aborts_draining_tts_as_well_as_the_active_request() -> None:
h = await open_harness()
try:
await h.run(append_audio())
request_id = h.stage0_request_id()
await h.deliver_and_settle(tts_output(request_id, samples=24000, text="he"))
h.session.bind_draining_request("duplex-drain-tts", "resp-old")
# Draining TTS sits on an older turn fence. cancel_fence only drops
# the fence being cancelled, so these bindings must be popped by id.
from vllm_omni.engine.duplex.contracts import DuplexFence
from vllm_omni.engine.duplex.session.engine_session import DuplexRequestResource
older = DuplexFence(h.session.session_id, epoch=h.session.epoch, turn_id=h.session.turn_id + 5)
h.session.request_resources[(2, "duplex-drain-tts")] = DuplexRequestResource(
stage_id=2, request_id="duplex-drain-tts", fence=older, submitted=True
)
h.session.request_resources[(3, "duplex-drain-tts")] = DuplexRequestResource(
stage_id=3, request_id="duplex-drain-tts", fence=older, submitted=True
)
h.session.request_resources[(1, "still-live")] = DuplexRequestResource(
stage_id=1, request_id="still-live", fence=older, submitted=True
)
events = await h.run(commands.BargeIn())
assert h.port.aborts == [[request_id, "duplex-drain-tts"]]
assert not h.session.is_draining_request("duplex-drain-tts")
done_ids = [
event.response["id"]
for event in events
if getattr(event, "wire_type", "") == "response.done" and isinstance(getattr(event, "response", None), dict)
]
assert "resp-old" in done_ids
assert (2, "duplex-drain-tts") not in h.session.request_resources
assert (3, "duplex-drain-tts") not in h.session.request_resources
assert (0, request_id) not in h.session.request_resources
assert (1, "still-live") in h.session.request_resources
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_cancel_response_without_active_response_is_rejected() -> None:
h = await open_harness()
@@ -778,6 +842,46 @@ async def test_listen_decision_on_a_resumable_request_closes_the_bounded_respons
await close_harness(h)
@pytest.mark.asyncio
async def test_direct_response_listen_still_emits_response_done_after_continuation_clear() -> None:
"""AURA-like silent DIRECT_RESPONSE: after continuation budget is spent, emit response.done.
Non-resumable turn-commit still closes the active response on a listen/direct
decision so the client is not left without a terminal event.
"""
from dataclasses import replace
h = await open_harness()
try:
h.session.capabilities = replace(h.session.capabilities, supports_core_resumable_request=False)
await h.run(append_audio())
assert h.port.submissions, "ephemeral Stage0 must submit"
request_id = h.port.submissions[-1].context.request_id
assert "-turn" in request_id
await h.deliver_and_settle(tts_output(request_id, samples=24000, text="ok"))
response_id = h.session.active_response_id
assert response_id is not None
model_state = h.runner.model_state
model_state.continuation_owner_id = f"response:{response_id}"
model_state.continuation_units = h.runner.model._AUTO_RESPONSE_MAX_CONTINUATION_UNITS
listen = listen_output(request_id)
listen.finished = True
events = await h.deliver_and_settle(
listen,
stage_id=0,
segment_finished=True,
segment_token_ids=[11, 12, LISTEN_TOKEN_ID],
segment_output_metadata={"meta.listen_token_id": LISTEN_TOKEN_ID},
)
assert "response.listen" in types(events)
done = find(events, "response.done")
assert done.response_id == response_id
assert h.session.active_response_id is None
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_listen_decision_on_a_resumable_request_keeps_the_turn_going() -> None:
"""Same unfinished listen, budget left: it must schedule the next unit.
@@ -860,6 +964,38 @@ async def test_stage_failure_fails_the_active_response() -> None:
await close_harness(h)
@pytest.mark.asyncio
async def test_stage_failure_on_draining_request_fails_that_response_only() -> None:
"""Overlapped: a draining older request failure must not wipe the active response."""
from dataclasses import replace
h = await open_harness()
try:
h.session.capabilities = replace(h.session.capabilities, supports_concurrent_turn_requests=True)
await h.run(append_audio())
request_id = h.stage0_request_id()
await h.deliver_and_settle(tts_output(request_id, samples=24000, text="hello"))
r1 = h.session.active_response_id
assert r1 is not None
draining_req = "req-r1-talker-drain"
h.session.bind_draining_request(draining_req, r1)
# Open a newer active response while R1 TTS is still draining.
r2 = h.session.begin_response(turn_id=(h.session.turn_id or 0) + 1)
assert h.session.active_response_id == r2
assert r2 != r1
h.runner.on_stage_failure(2, RuntimeError("drain-boom"), request_id=draining_req)
events = await h.settle()
assert types(events)[0] == "error"
done = find(events, "response.done")
assert done.response_id == r1
assert done.status == "failed"
assert h.session.active_response_id == r2
assert not h.session.is_draining_request(draining_req)
finally:
await close_harness(h)
# --------------------------------------------------------------------------- #
# Turn mode (no auto response) #
# --------------------------------------------------------------------------- #
@@ -877,6 +1013,91 @@ async def test_turn_mode_skips_silent_chunks() -> None:
await close_harness(h)
@pytest.mark.asyncio
async def test_turn_mode_keeps_silent_chunks_with_video_frames() -> None:
"""Engine vision-follow buffers silent+frames; MiniCPM commit stays speech-gated."""
from dataclasses import replace
h = await open_harness(auto_response=False)
try:
h.session.capabilities = replace(
h.session.capabilities,
required_input_modalities=frozenset({"video"}),
optional_input_modalities=frozenset({"audio"}),
)
# Minimal 1x1 JPEG (base64) — wire validation only checks non-empty str.
frame = (
"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkS"
"Ew8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJ"
"CQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIy"
"MjIyMjIyMjIyMjIyMjL/wAARCAABAAEDASIAAhEBAxEB/8QAFQABAQAAAAAAAAAA"
"AAAAAAAAAAj/xAAUEAEAAAAAAAAAAAAAAAAAAAAA/8QAFQEBAQAAAAAAAAAAAAAA"
"AAAAAAD/xAAUEQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAGfAP/EABQQ"
"AQAAAAAAAAAAAAAAAAAAAAD/2gAIAQEAAQUCf//EABQRAQAAAAAAAAAAAAAAAAAA"
"AAD/2gAIAQMBAT8Bf//EABQRAQAAAAAAAAAAAAAAAAAAAAD/2gAIAQIBAT8Bf//E"
"ABQQAQAAAAAAAAAAAAAAAAAAAAD/2gAIAQEABj8Cf//EABQQAQAAAAAAAAAAAAAA"
"AAAAAAD/2gAIAQEAAT8hf//Z"
)
cmd = commands.AppendAudio(
audio=pcm_f32(160, value=0.0),
format="pcm_f32le",
sample_rate_hz=16000,
is_speech=False,
video_frames=(frame,),
)
events = await h.run(cmd)
assert "response.listen" not in types(events)
assert h.runner.model_state.audio_buffer.has_pending()
events = await h.run(commands.Commit(create_response=True))
# MiniCPM prepare_commit is speech-gated (no frames-only Stage0).
assert "input_audio_buffer.committed" in types(events), types(events)
assert h.port.submissions == []
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_turn_mode_keeps_silent_vision_while_response_in_progress() -> None:
"""TTS still playing: vision-follow must buffer, not drop as silence_or_noise."""
from dataclasses import replace
h = await open_harness(auto_response=False)
try:
h.session.capabilities = replace(
h.session.capabilities,
required_input_modalities=frozenset({"video"}),
optional_input_modalities=frozenset({"audio"}),
)
h.session._response.active_response_id = "resp-tts"
frame = (
"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkS"
"Ew8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJ"
"CQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIy"
"MjIyMjIyMjIyMjIyMjL/wAARCAABAAEDASIAAhEBAxEB/8QAFQABAQAAAAAAAAAA"
"AAAAAAAAAAj/xAAUEAEAAAAAAAAAAAAAAAAAAAAA/8QAFQEBAQAAAAAAAAAAAAAA"
"AAAAAAD/xAAUEQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAGfAP/EABQQ"
"AQAAAAAAAAAAAAAAAAAAAAD/2gAIAQMBAT8Bf//EABQRAQAAAAAAAAAAAAAAAAAA"
"AAD/2gAIAQIBAT8Bf//EABQQAQAAAAAAAAAAAAAAAAAAAAD/2gAIAQEABj8Cf//E"
"ABQQAQAAAAAAAAAAAAAAAAAAAAD/2gAIAQEAAT8hf//Z"
)
events = await h.run(
commands.AppendAudio(
audio=pcm_f32(160, value=0.0),
format="pcm_f32le",
sample_rate_hz=16000,
is_speech=False,
video_frames=(frame,),
)
)
listen_reasons = [
getattr(event, "details", {}) or {} for event in events if getattr(event, "type", None) == "response.listen"
]
assert all(details.get("reason") != "silence_or_noise" for details in listen_reasons)
assert h.runner.model_state.audio_buffer.has_pending()
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_turn_mode_commit_with_response_create_starts_one_response() -> None:
h = await open_harness(auto_response=False)
@@ -1033,7 +1254,7 @@ async def test_conversation_items_can_be_injected_and_deleted() -> None:
await close_harness(h)
def _stage_metrics_of(event: object) -> dict[str, dict[str, object]]:
def _stage_metrics_of(event: DuplexEvent) -> dict[str, dict[str, object]]:
"""Per-stage engine metrics as the client reads them off one wire event."""
payload = event.to_realtime()
metadata = payload.get("metadata")
@@ -1081,6 +1302,40 @@ async def test_stage0_metrics_reach_the_response_even_though_its_output_feeds_tt
await close_harness(h)
@pytest.mark.asyncio
async def test_projected_stage1_metrics_reach_the_spoken_audio_event() -> None:
"""Thinker text is projected, but its TTFT/TPOT still have to ride the audio event."""
h = await open_harness(stage_count=3)
try:
plugin = h.runner.model._ctx.plugin
def project_stage1(*, stage_id: int, output: object, context: object) -> bool:
del output, context
return stage_id == 1
plugin.project_intermediate_output = project_stage1
await h.run(append_audio())
request_id = h.stage0_request_id()
thinker = SimpleNamespace(
request_id=request_id,
finished=True,
outputs=[SimpleNamespace(text="hello", token_ids=[7, 8], multimodal_output={})],
multimodal_output={},
)
forwarded = h.deliver(
thinker,
stage_id=1,
segment_finished=False,
metrics=stage_stats(stage_id=1, request_id=request_id, num_tokens_out=2, itls_ms=[6.0]),
)
assert forwarded is False
events = await h.deliver_and_settle(tts_output(request_id, samples=24000, text="hello"), stage_id=2)
stage_metrics = _stage_metrics_of(find(events, "response.output_audio.delta"))
assert stage_metrics["1"]["num_tokens_out"] == 2
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_stage0_metrics_from_several_units_are_summed_into_one_response() -> None:
"""Two pass-through segments before the first audio are one response's tokens."""
tests/engine/duplex/test_session_runner_ephemeral_bind.py
+28
View File
@@ -0,0 +1,28 @@
# SPDX-License-Identifier: Apache-2.0
"""Ephemeral Stage0 bind uses r.stage0-turn{N}, not resumable r.stage0."""
from __future__ import annotations
from vllm_omni.engine.duplex.contracts import (
DuplexFence,
duplex_ephemeral_stage_request_id,
duplex_resource_request_id,
)
from vllm_omni.engine.duplex.session.manager import DuplexSessionManager
def test_stage0_request_id_ephemeral_when_not_resumable() -> None:
fence = DuplexFence("duplex-test", epoch=0, turn_id=3)
rid = DuplexSessionManager.stage_request_id(fence, stage_id=0, resumable=False)
expected = duplex_ephemeral_stage_request_id(fence, stage_id=0)
assert rid == expected
assert rid.endswith(".r.stage0-turn3")
assert not rid.endswith(".r.stage0")
def test_stage0_request_id_resumable_keeps_stable_role() -> None:
fence = DuplexFence("duplex-test", epoch=1, turn_id=3)
rid = DuplexSessionManager.stage_request_id(fence, stage_id=0, resumable=True)
expected = duplex_resource_request_id(fence, "stage0")
assert rid == expected
assert rid.endswith(".r.stage0")
tests/engine/test_duplex_orchestrator.py
+92 -2
View File
@@ -55,6 +55,7 @@ def _build(
stages: int = 1,
running_counter: FakeRunningCounter | None = None,
runtime_config: DuplexSessionRuntimeConfig | None = None,
plugin=None,
) -> tuple[DuplexOrchestrator, list[FakeStageClient], asyncio.Queue, asyncio.Queue]:
clients = [FakeStageClient(stage_type="llm", final_output=index == stages - 1) for index in range(stages)]
pools = _build_stage_pools(
@@ -70,7 +71,7 @@ def _build(
rpc_async_queue=rpc_q,
stage_pools=pools,
running_counter=running_counter,
plugin=MiniCPMO45DuplexPlugin(_encode_audio),
plugin=plugin or MiniCPMO45DuplexPlugin(_encode_audio),
duplex_session_config=runtime_config or DuplexSessionRuntimeConfig(reaper_interval_s=0.01),
model_config=None,
)
@@ -197,7 +198,8 @@ async def test_open_preregisters_the_stage0_request_and_close_releases_it() -> N
assert request_state.session_owned is True
assert request_state.session_id == SESSION_ID
assert request_state.fence == DuplexFence(SESSION_ID)
assert request_state.streaming.enabled is True
# Set on submit from submission.resumable, not again at preregister.
assert request_state.streaming.enabled is False
# Preregistration reserves the id; nothing is running until an append submits.
assert counter.value == 0
assert clients[0].add_request_calls == []
@@ -211,6 +213,18 @@ async def test_open_preregisters_the_stage0_request_and_close_releases_it() -> N
assert orchestrator.session_manager.active_count() == 0
@pytest.mark.asyncio
async def test_aura_ephemeral_preregister_disables_streaming() -> None:
from vllm_omni.model_executor.models.aura_omni.duplex.plugin import AuraDuplexPlugin
orchestrator, _, rpc_q, _ = _build(plugin=AuraDuplexPlugin(_encode_audio))
result = await _open(orchestrator, rpc_q)
assert result.ok is True
request_state = next(iter(orchestrator.request_states.values()))
assert request_state.streaming.enabled is False
await orchestrator.session_manager.shutdown()
@pytest.mark.asyncio
async def test_open_failure_rolls_back_session_and_reserved_request() -> None:
orchestrator, _, rpc_q, _ = _build()
@@ -275,6 +289,7 @@ async def test_append_submits_the_resumable_stage0_request_and_counts_it_running
assert submitted.resumable is True
assert submitted.sampling_params.max_tokens == 20
request_state = orchestrator.request_states[request_id]
assert request_state.streaming.enabled is True
assert request_state.stage_fences[0] == DuplexFence(SESSION_ID)
assert 0 in request_state.stage_submit_ts
assert counter.value == 1
@@ -390,6 +405,31 @@ async def test_forward_failure_closes_the_owning_session() -> None:
assert types[-1] == "session.expired"
@pytest.mark.asyncio
async def test_aura_forward_failure_keeps_the_session() -> None:
from vllm_omni.model_executor.models.aura_omni.duplex.plugin import AuraDuplexPlugin
orchestrator, clients, rpc_q, output_q = _build(stages=2, plugin=AuraDuplexPlugin(_encode_audio))
await _open(orchestrator, rpc_q)
request_id = next(iter(orchestrator.request_states))
await _submit(orchestrator, _append_audio())
request_state = orchestrator.request_states[request_id]
session = orchestrator.session_manager.get(SESSION_ID)
assert session is not None
absorbed = await orchestrator._handle_forward_failure(request_id, 1, request_state, ValueError("stale talker"))
await _settle(orchestrator)
assert absorbed is True
assert request_id not in orchestrator.request_states
assert SESSION_ID in orchestrator.session_manager.runners
assert session.state != DuplexSessionState.CLOSED
types = [message.event.type for message in [output_q.get_nowait() for _ in range(output_q.qsize())]]
assert "error" in types
assert "session.expired" not in types
await orchestrator.session_manager.shutdown()
@pytest.mark.asyncio
async def test_request_cleanup_failure_is_deferred_and_retried_by_the_reaper(monkeypatch) -> None:
orchestrator, clients, rpc_q, _ = _build()
@@ -560,3 +600,53 @@ async def test_turn_plugin_processes_multimodal_prompt_before_stage_submission(m
assert not orchestrator.request_states[submitted.request_id].streaming.enabled
finally:
await _close(orchestrator, rpc_q)
@pytest.mark.asyncio
async def test_sentence_partial_does_not_legacy_forward_the_full_stage_output() -> None:
from vllm_omni.engine.duplex.plugin import PartialStageForward
orchestrator, *_ = _build(stages=4)
forwarded: list[object] = []
async def record_forward(req_id, stage_id, output, req_state, **kwargs):
del req_id, stage_id, req_state, kwargs
forwarded.append(output)
async def no_intercept(*args, **kwargs):
del args, kwargs
return False
orchestrator._forward_to_next_stage = record_forward # type: ignore[method-assign]
orchestrator._intercept_stage_output = no_intercept # type: ignore[method-assign]
orchestrator.async_chunk = True
orchestrator._stage_receives_async_chunks = lambda stage_id: False # type: ignore[method-assign]
full = SimpleNamespace(request_id="r", finished=True, text="FULL")
chunk = SimpleNamespace(request_id="r", finished=True, text="CHUNK")
req_state = DuplexOrchestratorRequestState(
request_id="r",
final_stage_id=3,
session_owned=True,
sampling_params_list=[SimpleNamespace() for _ in range(4)],
)
orchestrator.plugin.plan_partial_stage_output = lambda *args, **kwargs: PartialStageForward( # type: ignore[method-assign]
output=chunk, is_final_update=True
)
orchestrator.plugin.partial_stage_followup = lambda *args, **kwargs: None # type: ignore[method-assign]
await orchestrator._route_output(1, 0, full, req_state, None)
assert forwarded == [chunk]
assert req_state.skip_legacy_stage_forward is False
forwarded.clear()
orchestrator.plugin.plan_partial_stage_output = lambda *args, **kwargs: None # type: ignore[method-assign]
await orchestrator._route_output(1, 0, full, req_state, None)
assert forwarded == [full]
def test_default_plugin_declares_no_draining_stages() -> None:
plugin = MiniCPMO45DuplexPlugin(_encode_audio)
assert plugin.draining_stage_ids(stage_count=4) == frozenset()
tests/engine/test_orchestrator_stage_input_bridge.py
+36
View File
@@ -14,6 +14,7 @@ import pytest
from vllm.outputs import CompletionOutput, RequestOutput
from vllm.sampling_params import SamplingParams
from vllm_omni.engine.duplex_orchestrator import DuplexOrchestrator
from vllm_omni.engine.orchestrator import (
Orchestrator,
OrchestratorRequestState,
@@ -249,6 +250,41 @@ async def test_async_prewarm_skips_outgoing_only_stage(payload_sender_info) -> N
)
@pytest.mark.asyncio
async def test_async_prewarm_skips_stage_with_custom_process_input_func() -> None:
"""AURA Stage1 has asr2aura: must not be zero-prewarmed under async_chunk.
The custom-process-input gate lives on DuplexOrchestrator (AURA path), not
the turn-based Orchestrator base.
"""
orchestrator = object.__new__(DuplexOrchestrator)
stage0 = FakePrewarmPool("sender")
stage1 = FakePrewarmPool("receiver") # would receive chunks if role alone decided
stage1.stage_client = SimpleNamespace(custom_process_input_func=lambda *a, **k: None)
stage2 = FakePrewarmPool("receiver")
orchestrator.stage_pools = [stage0, stage1, stage2]
orchestrator._emit_tx_edge = lambda **_kwargs: None
orchestrator._on_stage_submitted = MagicMock()
req_state = OrchestratorRequestState(
request_id="req-prewarm-custom",
prompt={"prompt_token_ids": [1, 2]},
sampling_params_list=[SamplingParams(max_tokens=1) for _ in range(3)],
final_stage_id=2,
)
prewarmed = await orchestrator._prewarm_async_chunk_stages(
"req-prewarm-custom",
SimpleNamespace(prompt_token_ids=[1, 2], resumable=True),
req_state,
)
assert prewarmed is True
assert stage1.submitted == []
assert len(stage2.submitted) == 1
assert 1 not in req_state.stage_submit_ts
assert 2 in req_state.stage_submit_ts
@pytest.mark.asyncio
async def test_async_route_forwards_to_outgoing_only_stage() -> None:
orchestrator = object.__new__(Orchestrator)
tests/entrypoints/duplex/test_startup_warmup.py
+56
View File
@@ -0,0 +1,56 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import pybase64 as base64
from vllm_omni.engine.duplex.config import DuplexCapabilities
from vllm_omni.entrypoints.duplex.warmup import (
_WARMUP_AUDIO_SECONDS,
_WARMUP_FRAME_HEIGHT,
_WARMUP_FRAME_WIDTH,
_WARMUP_SAMPLE_RATE_HZ,
_short_pcm_f32le_b64,
_warmup_jpeg_b64,
startup_warmup_kind,
)
class _Plugin:
def __init__(self, required: frozenset[str]) -> None:
self._required = required
def capabilities(self, *, max_sessions: int) -> DuplexCapabilities:
del max_sessions
return DuplexCapabilities(required_input_modalities=self._required)
def test_video_required_model_warms_up_one_turn_by_default() -> None:
plugin = _Plugin(frozenset({"video"}))
assert startup_warmup_kind(plugin, 0) == "video_turn"
def test_negative_warmup_frames_disables_the_video_turn() -> None:
plugin = _Plugin(frozenset({"video"}))
assert startup_warmup_kind(plugin, -1) is None
def test_audio_primary_model_stays_opt_in() -> None:
plugin = _Plugin(frozenset({"audio"}))
assert startup_warmup_kind(plugin, 0) is None
assert startup_warmup_kind(plugin, 4) == "silent_frames"
def test_startup_turn_audio_is_short_and_not_silent() -> None:
raw = base64.b64decode(_short_pcm_f32le_b64())
assert len(raw) == int(_WARMUP_SAMPLE_RATE_HZ * _WARMUP_AUDIO_SECONDS) * 4
assert any(raw)
def test_startup_turn_frame_is_a_camera_sized_jpeg() -> None:
from io import BytesIO
from PIL import Image
image = Image.open(BytesIO(base64.b64decode(_warmup_jpeg_b64())))
assert image.size == (_WARMUP_FRAME_WIDTH, _WARMUP_FRAME_HEIGHT)
assert image.format == "JPEG"
tests/examples/test_minicpmo_realtime_web_server.py
+12 -1
View File
@@ -113,12 +113,23 @@ def test_qwen_config_does_not_inject_reference_voice():
assert '"profile": "qwen3-turn"' in page
assert '"adapter": "vad"' in page
assert '"refAudio": null' in page
for name in ("common", "minicpm_native", "qwen3_turn"):
for name in ("common", "minicpm_native", "qwen3_turn", "aura_ptt"):
assert client.get(f"/static/profiles/{name}.js").status_code == 200
with pytest.raises(ValueError, match="ref-audio"):
build_app(profile="qwen3-turn", ref_audio="unused.wav")
def test_aura_ptt_config_defaults_and_rejects_ref_audio():
client = TestClient(build_app(profile="aura-ptt", model="aurateam/AURA"))
page = client.get("/").text
assert '"profile": "aura-ptt"' in page
assert '"refAudio": null' in page
assert "aura_ptt.js" in page
assert 'id="pttButton"' in page
with pytest.raises(ValueError, match="ref-audio"):
build_app(profile="aura-ptt", ref_audio="unused.wav")
def test_config_cannot_close_its_script_element():
page = TestClient(build_app(model="</script><script>alert(1)</script>")).get("/").text
assert "</script><script>alert(1)" not in page
tests/examples/test_minicpmo_realtime_web_static.py
+15
View File
@@ -240,3 +240,18 @@ def test_playback_worklet_fades_terminal_drain_to_zero():
capture_output=True,
text=True,
)
def test_aura_camera_preview_size_is_not_shared() -> None:
aura = (STATIC_ROOT / "profiles" / "aura_ptt.js").read_text(encoding="utf-8")
app = (STATIC_ROOT / "app.js").read_text(encoding="utf-8")
minicpm = (STATIC_ROOT / "profiles" / "minicpm_native.js").read_text(encoding="utf-8")
qwen = (STATIC_ROOT / "profiles" / "qwen3_turn.js").read_text(encoding="utf-8")
html = (APP_ROOT / "index.html").read_text(encoding="utf-8")
assert "cameraPreviewLarge: true" in aura
assert "cameraPreviewLarge" not in minicpm
assert "cameraPreviewLarge" not in qwen
assert "profile.cameraPreviewLarge" in app
assert "camera-preview-break" not in app
assert "width:96px;height:72px" in html
tests/examples/test_realtime_web_profiles.cjs
+153 -4
View File
@@ -5,7 +5,7 @@ const path = require('node:path');
const vm = require('node:vm');
const root = path.resolve(__dirname, '../../examples/online_serving/realtime_web/app/static');
const context = vm.createContext({ URL });
for (const name of ['common', 'minicpm_native', 'qwen3_turn']) {
for (const name of ['common', 'minicpm_native', 'qwen3_turn', 'aura_ptt']) {
vm.runInContext(fs.readFileSync(path.join(root, `profiles/${name}.js`), 'utf8'), context);
}
const profiles = context.OmniRealtimeProfiles;
@@ -83,6 +83,7 @@ function shell(profileName, adapter = 'stt', options = {}) {
class Element {
constructor() { this.style = {}; this.listeners = {}; this.children = []; this.value = ''; this.textContent = ''; this.classList = { add() {}, remove() {}, toggle() {} }; }
addEventListener(name, callback) { this.listeners[name] = callback; }
closest() { return null; }
append(...children) { this.children.push(...children); }
appendChild(child) { this.children.push(child); }
replaceChildren() { this.children = []; }
@@ -149,15 +150,16 @@ function shell(profileName, adapter = 'stt', options = {}) {
atob: (value) => Buffer.from(value, 'base64').toString('binary'),
});
ctx.window = ctx;
for (const name of ['common', 'minicpm_native', 'qwen3_turn']) {
for (const name of ['common', 'minicpm_native', 'qwen3_turn', 'aura_ptt']) {
vm.runInContext(fs.readFileSync(path.join(root, `profiles/${name}.js`), 'utf8'), ctx);
}
// Expose closure controls only in the test VM; production has no test API.
const source = fs.readFileSync(path.join(root, 'app.js'), 'utf8').replace(/\}\)\(\);\s*$/, `
globalThis.testUI = { startSession, stopSession, handleEvent, playbackDrained,
microphoneUploadEnabled, flushCapture, awaitQueue: () => audioChain,
microphoneUploadEnabled, flushCapture, setPttHeld, awaitQueue: () => audioChain,
capture() { pendingCapture.push(new Int16Array([100, 200])); },
state() { return { running, connectionReady, assistantActive }; }
setPendingFrame(frame) { cameraPendingFrame = frame; if (profile.stickyCamera) cameraLastFrame = frame; },
state() { return { running, connectionReady, assistantActive, pttHeld }; }
};
})();`);
vm.runInContext(source, ctx);
@@ -191,6 +193,7 @@ test('shared shell STT sends final commit and starts a fresh second turn only af
assert.equal(app.ui.microphoneUploadEnabled(), true);
assert.equal(app.elements.get('cameraButton').hidden, true);
assert.equal(app.sockets.flatMap(s => s.sent).some(e => e.type === 'playback.ack'), false);
assert.equal(app.elements.get('pttButton').hidden, true);
await app.ui.stopSession({ terminal: false });
});
@@ -247,6 +250,8 @@ test('an interrupted response disarms the turn watchdog it left armed', async ()
test('MiniCPM retains microphone upload while speaking and acknowledges speaker drain', async () => {
const app = shell('minicpm-native');
await app.ui.startSession();
assert.equal(app.elements.get('pttButton').hidden, true);
assert.equal(app.elements.get('sendTurnButton').hidden, true);
await app.ui.handleEvent({ type: 'response.speak', response_id: 'native-r1' });
assert.equal(app.ui.microphoneUploadEnabled(), true);
await app.ui.handleEvent({ type: 'response.output_audio.delta', delta: 'AAAAAA==', response_id: 'native-r1' });
@@ -428,6 +433,150 @@ test('the camera retires its oldest image instead of exhausting the session budg
assert.equal(plain(p.imageMessages('JPEG')).length, 1);
});
test('AURA PTT profile sets duplex, is_speech append, commit on release, and shell flags', () => {
const aura = profiles['aura-ptt']();
const url = new URL(aura.url({ ...config, model: 'aurateam/AURA' }, 'http://localhost/'));
assert.equal(url.searchParams.get('duplex'), '1');
assert.equal(url.searchParams.get('model'), 'aurateam/AURA');
assert.equal(aura.pushToTalk, true);
assert.equal(aura.clientCommit, false);
assert.equal(aura.camera, true);
assert.equal(aura.stickyCamera, true);
assert.equal(aura.cameraIntervalMs, 500);
assert.equal(aura.visionFollowWhileSpeaking, true);
assert.equal(aura.halfDuplex, false);
assert.equal(aura.playbackAck, true);
assert.equal(aura.append('PCM', 'JPEG', { isSpeech: true }).is_speech, true);
assert.equal(aura.append('PCM', 'JPEG', { isSpeech: false }).is_speech, false);
assert.equal(aura.append('PCM', 'JPEG', { isSpeech: false }).video_frames[0], 'JPEG');
assert.deepEqual(plain(aura.commitMessages()), [
{ type: 'input_audio_buffer.commit', create_response: true },
]);
assert.equal(aura.ack('r1', 50).type, 'playback.ack');
const [update] = aura.initialMessages({}, 'Be brief');
assert.equal(update.session.instructions, 'Be brief');
assert.equal(update.session.extra_body.auto_response, true);
});
test('AURA does not open a second bubble for the same sentence after response.done', async () => {
const app = shell('aura-ptt');
await app.ui.startSession();
const sentence = '你好，我看到一个戴眼镜、穿黑T恤的男生正对着镜头说话呢。';
await app.ui.handleEvent({
type: 'response.output_audio_transcript.delta', response_id: 'r1', delta: sentence,
});
await app.ui.handleEvent({ type: 'response.done', response_id: 'r1' });
await app.ui.handleEvent({
type: 'response.output_audio_transcript.done', response_id: 'r2', transcript: sentence,
});
const turns = [...app.elements.get('conversation').children].filter((row) => row.children.length);
assert.equal(turns.length, 1);
assert.equal(turns[0].children[1].textContent, sentence);
await app.ui.stopSession({ terminal: false });
});
function visionAppends(app) {
return app.sockets[0].sent.filter(e => e.type === 'input_audio_buffer.append' && e.is_speech === false);
}
test('AURA shell holds speech until PTT and opens one vision turn per two frames', async () => {
const app = shell('aura-ptt');
await app.ui.startSession();
assert.equal(app.elements.get('pttButton').hidden, false);
assert.equal(app.elements.get('sendTurnButton').hidden, true);
assert.equal(app.elements.get('cameraButton').hidden, false);
assert.equal(app.ui.microphoneUploadEnabled(), false);
app.ui.capture();
app.ui.flushCapture();
assert.equal(app.sockets[0].sent.some(e => e.type === 'input_audio_buffer.append'), false);
app.ui.setPendingFrame('IDLE1');
app.ui.flushCapture();
assert.equal(visionAppends(app).length, 0, 'one frame must not open a turn');
app.ui.setPendingFrame('IDLE2');
app.ui.flushCapture();
const idle = visionAppends(app).at(-1);
assert.equal(idle.is_speech, false);
assert.deepEqual(idle.video_frames, ['IDLE1', 'IDLE2'], 'two frames gate the turn and both go out as one clip');
assert.equal(app.sockets[0].sent.at(-1).type, 'input_audio_buffer.commit');
assert.equal(app.ui.state().assistantActive, false);
app.ui.setPttHeld(true);
assert.equal(app.ui.microphoneUploadEnabled(), true);
app.ui.setPendingFrame('FRAME1');
app.ui.capture();
app.ui.flushCapture();
const speech = app.sockets[0].sent.find(e => e.type === 'input_audio_buffer.append' && e.is_speech === true);
assert.equal(speech.video_frames[0], 'FRAME1');
assert.equal(visionAppends(app).length, 1, 'speech must not wait for a second frame');
app.ui.setPttHeld(false);
assert.equal(app.sockets[0].sent.at(-1).type, 'input_audio_buffer.commit');
assert.equal(app.ui.state().pttHeld, false);
await app.ui.handleEvent({ type: 'response.created', response: { id: 'aura-r1' } });
app.ui.setPendingFrame('LOCKED1');
app.ui.flushCapture();
app.ui.setPendingFrame('LOCKED2');
app.ui.flushCapture();
assert.equal(visionAppends(app).length, 1, 'open text turn blocks the next vision commit');
await app.ui.handleEvent({
type: 'response.output_audio_transcript.done', response_id: 'aura-r1', transcript: '看到了。',
});
assert.equal(app.ui.state().assistantActive, true, 'text end must not wait for playback');
app.ui.flushCapture();
assert.deepEqual(visionAppends(app).at(-1).video_frames, ['LOCKED1', 'LOCKED2']);
assert.equal(app.sockets[0].sent.at(-1).type, 'input_audio_buffer.commit');
await app.ui.handleEvent({ type: 'response.created', response: { id: 'aura-r2' } });
app.ui.setPendingFrame('SILENT1');
app.ui.flushCapture();
app.ui.setPendingFrame('SILENT2');
app.ui.flushCapture();
assert.equal(visionAppends(app).length, 2);
await app.ui.handleEvent({ type: 'response.listen', response_id: 'aura-r2' });
app.ui.flushCapture();
assert.deepEqual(visionAppends(app).at(-1).video_frames, ['SILENT1', 'SILENT2']);
await app.ui.stopSession({ terminal: false });
});
test('AURA hold-to-talk stops local playback and does not cancel the response', async () => {
const app = shell('aura-ptt', 'stt', { realPlayback: true });
await app.ui.startSession();
receive(app, { type: 'response.created', response: { id: 'old' } });
receive(app, audioChunk('old'));
await app.ui.awaitQueue();
const player = app.nodes.find(node => node.name === 'fullduplex-pcm-playback').player;
for (let index = 0; index < 75; index++) renderPlayback(player);
assert.ok(renderPlayback(player).some(sample => sample !== 0));
const sentBefore = app.sockets[0].sent.length;
app.ui.setPttHeld(true);
assertSilent(player);
const extra = app.sockets[0].sent.slice(sentBefore);
assert.ok(extra.every(event => event.type === 'playback.ack'), 'hold must not abort the server response');
receive(app, audioChunk('old'));
await app.ui.awaitQueue();
assertSilent(player);
await app.ui.stopSession({ terminal: false });
});
test('AURA sticky camera reuses the last frame on speech without a fresh pending', async () => {
const app = shell('aura-ptt');
await app.ui.startSession();
app.ui.setPendingFrame('STICKY');
app.ui.setPttHeld(true);
app.ui.capture();
app.ui.flushCapture();
assert.equal(app.sockets[0].sent.at(-1).video_frames[0], 'STICKY');
// Pending cleared; sticky last frame still available for the next speech flush.
app.ui.capture();
app.ui.flushCapture();
assert.equal(app.sockets[0].sent.at(-1).video_frames[0], 'STICKY');
await app.ui.stopSession({ terminal: false });
});
// Exercise the real WebSocket dispatch queue together with the actual playback
// processor: sending a clear message alone does not prove the speaker is silent.
function receive(app, event) {
tests/model_executor/models/aura_omni/duplex/test_aura_data_plane.py
+165
View File
@@ -0,0 +1,165 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""CPU tests for AuraDataPlaneSession (unwrap runner envelope; do not drop audio)."""
from __future__ import annotations
from types import SimpleNamespace
import numpy as np
from vllm_omni.model_executor.models.aura_omni.duplex.data_plane import AuraDataPlaneSession
def _encode_audio(audio: object, sample_rate: int, fmt: str, speed: float | None) -> str | None:
del sample_rate, fmt, speed
if audio is None:
return None
return "encoded-audio"
def test_project_unwraps_data_plane_outputs_and_emits_audio_then_done() -> None:
plane = AuraDataPlaneSession(_encode_audio)
request_id = "duplex-s.abc.e.0.r.stage0-turn1"
plane.begin_request(request_id)
audio = np.zeros(16, dtype=np.float32)
output = SimpleNamespace(
request_id=request_id,
finished=True,
stage_id=3,
outputs=[
SimpleNamespace(
text="",
cumulative_text="",
finished=True,
multimodal_output={"audio": audio, "sr": 24000},
)
],
multimodal_output={"audio": audio, "sr": 24000},
)
events = list(plane.project({"data_plane_outputs": [output]}))
assert events, "project must not silently drop the runner envelope"
assert any(event.get("audio") == "encoded-audio" for event in events)
assert events[-1].get("end_of_turn") is True
assert events[0].get("data_plane_request_id") == request_id
def test_project_sends_only_the_new_tail_of_a_growing_waveform() -> None:
seen: list[int] = []
def encode(audio: object, sample_rate: int, fmt: str, speed: float | None) -> str:
del sample_rate, fmt, speed
count = int(np.asarray(audio).reshape(-1).size)
seen.append(count)
return f"pcm-{count}"
plane = AuraDataPlaneSession(encode)
request_id = "duplex-s.abc.e.0.r.stage3-turn1"
plane.begin_request(request_id)
def chunk(n: int, *, finished: bool) -> SimpleNamespace:
audio = np.arange(n, dtype=np.float32)
return SimpleNamespace(
request_id=request_id,
finished=finished,
stage_id=3,
outputs=[
SimpleNamespace(
text="",
cumulative_text="",
finished=False,
multimodal_output={"audio": audio, "sr": 24000},
)
],
)
first = list(plane.project_output(chunk(8, finished=False)))
second = list(plane.project_output(chunk(12, finished=True)))
assert seen == [8, 4]
assert first[0]["audio_duration_ms"] == round(8 * 1000 / 24000)
assert second[0]["audio_duration_ms"] == round(4 * 1000 / 24000)
assert second[-1].get("end_of_turn") is True
def test_project_holds_silent_token_prefix() -> None:
plane = AuraDataPlaneSession(_encode_audio)
request_id = "duplex-s.abc.e.0.r.stage1-turn1"
plane.begin_request(request_id)
partial = SimpleNamespace(
request_id=request_id,
finished=False,
stage_id=1,
outputs=[SimpleNamespace(text="<|sil", cumulative_text="<|sil", finished=False)],
)
assert list(plane.project({"data_plane_outputs": [partial]})) == []
def test_project_emits_chinese_brackets_immediately() -> None:
"""[沉默] is ordinary text — do not hold as a <|silent|> prefix."""
plane = AuraDataPlaneSession(_encode_audio)
request_id = "duplex-s.abc.e.0.r.stage1_t2"
plane.begin_request(request_id)
partial = SimpleNamespace(
request_id=request_id,
finished=False,
stage_id=1,
outputs=[SimpleNamespace(text="[", cumulative_text="[", finished=False)],
)
events = list(plane.project({"data_plane_outputs": [partial]}))
assert len(events) == 1
assert events[0].get("text") == "["
def test_project_flushes_held_silent_prefix_when_text_becomes_speech() -> None:
plane = AuraDataPlaneSession(_encode_audio)
request_id = "duplex-s.abc.e.0.r.stage1-turn3"
plane.begin_request(request_id)
partial = SimpleNamespace(
request_id=request_id,
finished=False,
stage_id=1,
outputs=[SimpleNamespace(text="<|", cumulative_text="<|", finished=False)],
)
assert list(plane.project({"data_plane_outputs": [partial]})) == []
spoken = SimpleNamespace(
request_id=request_id,
finished=False,
stage_id=1,
outputs=[SimpleNamespace(text="<|hello", cumulative_text="<|hello", finished=False)],
)
events = list(plane.project({"data_plane_outputs": [spoken]}))
assert len(events) == 1
assert events[0].get("text") == "<|hello"
def test_project_marks_finished_silent_text_as_listen() -> None:
plane = AuraDataPlaneSession(_encode_audio)
request_id = "duplex-s.abc.e.0.r.stage1_t4"
plane.begin_request(request_id)
final = SimpleNamespace(
request_id=request_id,
finished=True,
stage_id=1,
outputs=[SimpleNamespace(text="<|silent|>", cumulative_text="<|silent|>", finished=True)],
)
events = list(plane.project({"data_plane_outputs": [final]}))
assert any(e.get("silent") and e.get("is_listen") for e in events)
assert any(e.get("abort_data_plane_request") is True for e in events)
def test_project_finished_spoken_turn_carries_model_context() -> None:
plane = AuraDataPlaneSession(_encode_audio)
request_id = "duplex-s.abc.e.0.r.stage1-turn5"
plane.begin_request(request_id)
final = SimpleNamespace(
request_id=request_id,
finished=True,
stage_id=1,
outputs=[SimpleNamespace(text="你好。", cumulative_text="你好。", finished=True)],
)
events = list(plane.project({"data_plane_outputs": [final]}))
context = [event for event in events if event.get("model_context_text")]
assert context and context[-1]["model_context_text"] == "你好。"
assert context[-1].get("is_listen") is False
assert context[-1].get("text") == ""
tests/model_executor/models/aura_omni/duplex/test_aura_data_plane_multimodal.py
+115
View File
@@ -0,0 +1,115 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Aura duplex data-plane must surface Code2Wav MultimodalPayload audio."""
from __future__ import annotations
from types import SimpleNamespace
import torch
from vllm_omni.model_executor.models.aura_omni.duplex.data_plane import (
AuraDataPlaneSession,
_audio_value,
_multimodal,
)
from vllm_omni.outputs.mm_outputs import MultimodalPayload
def _encode(audio: object, sample_rate: int, fmt: str, speed: float | None) -> str | None:
del sample_rate, fmt, speed
if audio is None:
return None
return "encoded-audio"
def test_audio_value_accepts_from_raw_audio_key() -> None:
raw = {"model_outputs": [torch.zeros(8)], "sr": [torch.tensor(24000)]}
mm = MultimodalPayload.from_raw(raw, "audio")
assert mm is not None
assert "audio" in mm
audio = _audio_value(dict(mm))
assert isinstance(audio, torch.Tensor)
assert int(audio.numel()) == 8
def test_audio_value_accepts_wrongly_tagged_text_key() -> None:
# If stage output_modality is wrong, from_raw remaps model_outputs → "text".
raw = {"model_outputs": torch.zeros(4), "sr": torch.tensor(24000)}
mm = MultimodalPayload.from_raw(raw, "text")
assert mm is not None
audio = _audio_value(dict(mm))
assert isinstance(audio, torch.Tensor)
def test_multimodal_skips_empty_outer_dict_and_reads_completion() -> None:
mm = MultimodalPayload.from_dict({"audio": torch.zeros(3), "sr": torch.tensor(24000)})
completion = SimpleNamespace(multimodal_output=mm)
# Empty Mapping on the outer object must not hide completion payload.
outer = SimpleNamespace(multimodal_output={})
got = _multimodal(outer, completion)
assert "audio" in got
def test_project_emits_audio_from_multimodal_completion() -> None:
plane = AuraDataPlaneSession(_encode)
request_id = "duplex-s.test.e.0.r.stage0-turn0"
plane.begin_request(request_id)
mm = MultimodalPayload.from_raw(
{"model_outputs": [torch.zeros(16)], "sr": [torch.tensor(24000)]},
"audio",
)
completion = SimpleNamespace(
text="",
cumulative_text="",
finished=False,
multimodal_output=mm,
)
output = SimpleNamespace(
request_id=request_id,
finished=False,
stage_id=3,
outputs=[completion],
multimodal_output={},
)
events = list(plane.project({"data_plane_outputs": [output]}))
assert events, "project must emit TTS audio for MultimodalPayload"
assert any(event.get("audio") == "encoded-audio" for event in events)
def test_streaming_chunk_completion_finished_is_not_turn_eos() -> None:
"""Code2Wav chunks set CompletionOutput.finished; that must not EOT.
Runner._send_one_model_output_event drops is_terminal request ids before
emit. Marking terminal (or end_of_turn) on the first 8-frame chunk made
WS response.* never leave the engine.
"""
plane = AuraDataPlaneSession(_encode)
request_id = "duplex-s.test.e.0.r.stage0-turn0"
plane.begin_request(request_id)
mm = MultimodalPayload.from_raw(
{"model_outputs": [torch.zeros(16)], "sr": [torch.tensor(24000)]},
"audio",
)
completion = SimpleNamespace(
text="",
cumulative_text="",
finished=True,
multimodal_output=mm,
)
output = SimpleNamespace(
request_id=request_id,
finished=False,
stage_id=3,
outputs=[completion],
multimodal_output={},
)
events: list[dict[str, object]] = []
for event in plane.project({"data_plane_outputs": [output]}):
assert plane.is_terminal(request_id) is False
events.append(event)
assert events
assert events[0].get("audio") == "encoded-audio"
assert events[0].get("end_of_turn") is False
assert plane.is_terminal(request_id) is False
tests/model_executor/models/aura_omni/duplex/test_aura_plugin_contract.py
+598
View File
@@ -0,0 +1,598 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""CPU contract tests for AURA duplex plugin seams."""
from __future__ import annotations
import base64
from types import SimpleNamespace
import numpy as np
import pytest
from vllm.sampling_params import SamplingParams
from vllm_omni.engine.duplex.config import DuplexSessionConfig
from vllm_omni.engine.duplex.contracts import (
DuplexFence,
DuplexStageRequestContext,
DuplexStageSubmission,
duplex_ephemeral_stage_request_id,
duplex_resource_request_id,
)
from vllm_omni.engine.duplex.plugin import load_duplex_plugin
from vllm_omni.model_executor.models.aura_omni.duplex.data_plane import (
AuraDataPlaneContext,
AuraDataPlaneSession,
)
from vllm_omni.model_executor.models.aura_omni.duplex.history import (
drop_session_history,
get_or_create_session_history,
)
from vllm_omni.model_executor.models.aura_omni.duplex.input import AuraPcmAppendBuffer
from vllm_omni.model_executor.models.aura_omni.duplex.plugin import (
AURA_SILENT_TOKEN_ID,
AuraDuplexPlugin,
)
from vllm_omni.model_executor.stage_input_processors.aura_omni import SILENT_TEXT
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def _encode_audio(audio: object, sample_rate: int, fmt: str, speed: float | None) -> str | None:
del audio, sample_rate, fmt, speed
return "ZmFrZQ=="
def test_load_aura_duplex_plugin_and_sampling_arity() -> None:
plugin = load_duplex_plugin(
"vllm_omni.model_executor.models.aura_omni.duplex.plugin.AuraDuplexPlugin",
_encode_audio,
)
assert isinstance(plugin, AuraDuplexPlugin)
assert plugin.plugin_id == "aura"
defaults = (
SamplingParams(max_tokens=16),
SamplingParams(max_tokens=16),
SamplingParams(max_tokens=4096),
SamplingParams(max_tokens=16),
)
configured = plugin.configure_sampling_params(runtime_config={}, defaults=defaults)
assert len(configured) == 4
assert configured[2].stop_token_ids == [2150]
assert configured[2].max_tokens == 4096
stage1_stops = set(configured[1].stop_token_ids or [])
assert {151669, 151645}.issubset(stage1_stops)
assert 248070 not in stage1_stops
caps = plugin.capabilities(max_sessions=1)
assert caps.supports_turn_commit_only is True
assert caps.supports_core_resumable_request is False
assert caps.supports_concurrent_turn_requests is True
assert caps.required_input_modalities == frozenset({"video"})
assert caps.optional_input_modalities == frozenset({"audio"})
assert caps.allows_video_without_audio() is True
assert caps.validate_append_modalities(has_audio=True, has_video=False) == (
"This duplex model requires video_frames on input_audio_buffer.append"
)
assert caps.validate_append_modalities(has_audio=True, has_video=True) is None
assert caps.validate_append_modalities(has_audio=False, has_video=True) is None
def test_commit_only_buffer_emits_on_commit() -> None:
buf = AuraPcmAppendBuffer()
samples = np.zeros(1600, dtype="<f4")
payload = {
"type": "audio",
"format": "pcm_f32le",
"sample_rate_hz": 16000,
"audio": base64.b64encode(samples.tobytes()).decode("ascii"),
"is_speech": True,
"video_frames": ["aGVsbG8="],
}
reservation = buf.prepare_append(
payload,
operation_id="op1",
chunk_period_ms=1000,
allow_emit=True,
)
assert reservation is None
assert buf.has_pending()
commit = buf.prepare_commit(operation_id="c1", chunk_period_ms=1000)
assert commit.payload is not None
assert commit.payload.get("final") is True
assert commit.payload.get("aura_turn_commit") is True
assert commit.payload.get("video_frames") == ["aGVsbG8="]
commit.commit()
def test_commit_reservation_includes_video_bytes() -> None:
buf = AuraPcmAppendBuffer()
pcm = np.zeros(1600, dtype="<f4").tobytes()
frames = ["aGVsbG8=", "d29ybGQ="]
payload = {
"type": "audio",
"format": "pcm_f32le",
"sample_rate_hz": 16000,
"audio": base64.b64encode(pcm).decode("ascii"),
"is_speech": True,
"video_frames": frames,
}
assert (
buf.prepare_append(
payload,
operation_id="op1",
chunk_period_ms=1000,
allow_emit=True,
)
is None
)
pending = buf.pending_byte_count
assert pending == len(pcm) + sum(len(frame) for frame in frames)
commit = buf.prepare_commit(operation_id="c1", chunk_period_ms=1000)
assert buf.pending_byte_count == 0
assert commit.byte_count == pending
commit.commit()
assert buf.pending_byte_count == 0
assert not buf.has_reserved()
def test_plan_append_commit_builds_stage0_prompt() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
samples = np.zeros(800, dtype="<f4")
payload = {
"type": "audio",
"format": "pcm_f32le",
"sample_rate_hz": 16000,
"audio": base64.b64encode(samples.tobytes()).decode("ascii"),
"final": True,
"aura_turn_commit": True,
"is_speech": True,
}
fence = DuplexFence("sess-1", epoch=2, turn_id=5)
plan = plugin.plan_append(
request_id="req",
fence=fence,
session_config={},
runtime_config={"aura_system_prompt": "sys"},
seq=1,
turn_seq=1,
payload=payload,
final=True,
sampling_params=SamplingParams(max_tokens=8),
)
info = plan.prompt["additional_information"]
assert info["aura_duplex"] is True
assert info["session_id"] == "sess-1"
assert "audio" in plan.prompt["multi_modal_data"]
assert "prompt_token_ids" not in plan.prompt
def test_video_frames_to_mm_packs_one_video() -> None:
"""Two JPEG frames are one video clip, not two image items."""
from io import BytesIO
from PIL import Image
from vllm_omni.model_executor.models.aura_omni.duplex.plugin import _video_frames_to_mm
frames: list[str] = []
for color in ((255, 0, 0), (0, 255, 0)):
buf = BytesIO()
Image.new("RGB", (8, 8), color).save(buf, format="JPEG")
frames.append(base64.b64encode(buf.getvalue()).decode("ascii"))
mm = _video_frames_to_mm(frames)
assert list(mm.keys()) == ["video"]
assert len(mm["video"]) == 1
video, metadata = mm["video"][0]
assert np.asarray(video).shape == (2, 8, 8, 3)
assert metadata["total_num_frames"] == 2
assert metadata["do_sample_frames"] is False
def test_plan_append_vision_empty_audio_never_leaves_empty_prompt() -> None:
"""Frames + empty/near-silent audio must still get a Stage0 ASR pad."""
from io import BytesIO
from PIL import Image
plugin = AuraDuplexPlugin(_encode_audio)
img = Image.new("RGB", (32, 32), (8, 16, 24))
buf = BytesIO()
img.save(buf, format="JPEG")
frame = base64.b64encode(buf.getvalue()).decode("ascii")
fence = DuplexFence("sess-v", epoch=0, turn_id=1)
near = np.linspace(1e-5, -1e-5, 80, dtype="<f4")
cases = [
("empty_speech_true", True, ""),
("empty_speech_false", False, ""),
("near_silent", False, base64.b64encode(near.tobytes()).decode("ascii")),
]
for name, is_speech, audio in cases:
payload = {
"type": "audio",
"format": "pcm_f32le",
"sample_rate_hz": 16000,
"audio": audio,
"final": True,
"aura_turn_commit": True,
"is_speech": is_speech,
"video_frames": [frame],
}
plan = plugin.plan_append(
request_id="req",
fence=fence,
session_config={},
runtime_config={},
seq=1,
turn_seq=1,
payload=payload,
final=True,
sampling_params=SamplingParams(max_tokens=8),
)
assert plan.prompt.get("prompt"), f"{name}: Stage0 prompt must be non-empty"
assert "audio" in plan.prompt["multi_modal_data"], f"{name}: padded audio required"
assert "deferred_multi_modal_data" in plan.prompt["additional_information"]
def test_project_intermediate_output_targets_stage1_only() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
assert plugin.project_intermediate_output(stage_id=1, output=object(), context=object()) is True
assert plugin.project_intermediate_output(stage_id=0, output=object(), context=object()) is False
assert plugin.project_intermediate_output(stage_id=3, output=object(), context=object()) is False
def test_release_concurrent_turn_requests_on_stage1_final() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
assert plugin.release_concurrent_turn_requests(stage_id=1, segment_finished=True, output=object(), context=object())
assert not plugin.release_concurrent_turn_requests(
stage_id=1, segment_finished=False, output=object(), context=object()
)
finished = type("Out", (), {"finished": True})()
assert plugin.release_concurrent_turn_requests(
stage_id=1, segment_finished=False, output=finished, context=object()
)
assert not plugin.release_concurrent_turn_requests(
stage_id=2, segment_finished=True, output=object(), context=object()
)
def test_configure_sampling_keeps_silent_stop_visible() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
defaults = (
SamplingParams(max_tokens=16),
SamplingParams(max_tokens=16),
SamplingParams(max_tokens=4096),
SamplingParams(max_tokens=16),
)
configured = plugin.configure_sampling_params(runtime_config={}, defaults=defaults)
stage1 = configured[1]
assert isinstance(stage1, SamplingParams)
assert stage1.include_stop_str_in_output is True
assert stage1.skip_special_tokens is False
assert AURA_SILENT_TOKEN_ID in (stage1.stop_token_ids or [])
def test_configure_sampling_keeps_explicit_talker_limits() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
defaults = (
SamplingParams(max_tokens=16),
SamplingParams(max_tokens=16),
SamplingParams(max_tokens=500, stop_token_ids=[7]),
SamplingParams(max_tokens=16),
)
configured = plugin.configure_sampling_params(runtime_config={}, defaults=defaults)
stage2 = configured[2]
assert isinstance(stage2, SamplingParams)
assert stage2.max_tokens == 500
assert stage2.stop_token_ids == [7]
unset = plugin.configure_sampling_params(
runtime_config={},
defaults=(
SamplingParams(max_tokens=16),
SamplingParams(max_tokens=16),
SamplingParams(max_tokens=None),
SamplingParams(max_tokens=16),
),
)
assert unset[2].max_tokens == 240
assert unset[2].stop_token_ids == [2150]
def test_decide_output_silent_short_circuits() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
class _Completion:
text = SILENT_TEXT
token_ids = [AURA_SILENT_TOKEN_ID]
finished = True
class _Output:
outputs = [_Completion()]
decision = plugin.decide_output(
stage_id=1,
final_stage_id=3,
segment_finished=True,
segment_token_ids=(AURA_SILENT_TOKEN_ID,),
segment_output_metadata={},
output=_Output(),
)
assert decision is not None
assert decision.metadata.get("model_listen") is True
def test_decide_output_chinese_silence_is_not_special_token() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
class _Completion:
text = "[沉默]"
token_ids = [58, 107107, 60]
finished = True
class _Output:
outputs = [_Completion()]
assert (
plugin.decide_output(
stage_id=1,
final_stage_id=3,
segment_finished=True,
segment_token_ids=(58, 107107, 60),
segment_output_metadata={},
output=_Output(),
)
is None
)
def test_ephemeral_request_id_includes_turn() -> None:
fence = DuplexFence("s", epoch=1, turn_id=7)
resumable = duplex_resource_request_id(fence, "stage0")
ephemeral = duplex_ephemeral_stage_request_id(fence, stage_id=0)
assert ephemeral.endswith("stage0-turn7")
assert resumable != ephemeral
fence2 = DuplexFence("s", epoch=1, turn_id=8)
assert duplex_ephemeral_stage_request_id(fence, stage_id=0) != duplex_ephemeral_stage_request_id(fence2, stage_id=0)
def test_stage_submission_defaults_resumable_true() -> None:
ctx = DuplexStageRequestContext(
request_id="r",
session_id="s",
fence=DuplexFence("s"),
stage_id=0,
final_stage_id=3,
config_generation=0,
sampling_params=(SamplingParams(max_tokens=1),),
)
submission = DuplexStageSubmission(
context=ctx,
prompt={"prompt_token_ids": [0]},
already_submitted=False,
)
assert submission.resumable is True
def test_data_plane_close_session_drops_history() -> None:
from vllm_omni.model_executor.models.aura_omni.duplex.data_plane import AuraDataPlaneSession
from vllm_omni.model_executor.models.aura_omni.duplex.history import get_or_create_session_history
drop_session_history("close-hist")
get_or_create_session_history("close-hist").begin_user_turn("hi")
AuraDataPlaneSession(encode_audio=_encode_audio).close_session("close-hist")
from vllm_omni.model_executor.models.aura_omni.duplex.history import _STORE
assert "close-hist" not in _STORE
def test_session_history_strips_oldest_silent_videos() -> None:
drop_session_history("hist-ut")
history = get_or_create_session_history("hist-ut")
history.max_video_rounds = 2
history.video_rounds_to_remove = 1
for index in range(3):
history.begin_user_turn("", video=(f"clip-{index}",))
history.commit_turn(SILENT_TEXT)
kept = [message["video"] for message in history.messages if message.get("video") is not None]
assert kept == [("clip-1",), ("clip-2",)]
assert not any(message.get("video") == ("clip-0",) for message in history.messages)
drop_session_history("hist-ut")
def test_pipeline_declares_aura_duplex_plugin() -> None:
from vllm_omni.model_executor.models.aura_omni.pipeline import AURA_OMNI_PIPELINE
assert AURA_OMNI_PIPELINE.duplex_plugin == (
"vllm_omni.model_executor.models.aura_omni.duplex.plugin.AuraDuplexPlugin"
)
def test_project_output_uses_requested_format_and_chunk_duration() -> None:
seen: dict[str, object] = {}
def encode(audio: object, sample_rate: int, fmt: str, speed: float | None) -> str:
seen["fmt"] = fmt
seen["rate"] = sample_rate
seen["speed"] = speed
del audio
return "AAAA"
plane = AuraDataPlaneSession(encode)
class _Output:
request_id = "aura-turn"
finished = False
stage_id = 3
multimodal_output = {"audio": np.zeros(4800, dtype=np.float32), "sr": 24000}
events = list(plane.project_output(_Output(), context=AuraDataPlaneContext(response_format="pcm16", speed=1.0)))
assert seen["fmt"] == "pcm16"
assert events[0]["audio_format"] == "pcm16"
assert events[0]["audio_duration_ms"] == 200
assert events[0]["sample_rate_hz"] == 24000
def encode_wav(audio: object, sample_rate: int, fmt: str, speed: float | None) -> str:
del audio, sample_rate, speed
assert fmt == "wav"
return "V0FW"
plane_wav = AuraDataPlaneSession(encode_wav)
wav_events = list(plane_wav.project_output(_Output(), context=AuraDataPlaneContext(response_format="wav")))
assert wav_events[0]["audio_format"] == "wav"
def test_instructions_update_replaces_effective_prompt() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
current = {"aura_system_prompt": "creation default", "instructions": "creation default"}
updated = plugin.runtime_config_for_update(
DuplexSessionConfig(model="aurateam/AURA", instructions="be brief"),
current,
)
assert updated["aura_system_prompt"] == "be brief"
assert updated["instructions"] == "be brief"
kept = plugin.runtime_config_for_update(
DuplexSessionConfig(
model="aurateam/AURA",
extra_body={"aura_system_prompt": "explicit"},
instructions="ignored when explicit",
),
updated,
)
assert kept["aura_system_prompt"] == "explicit"
def test_plan_partial_stage_output_hands_a_sentence_to_talker() -> None:
def aura2tts() -> None:
return None
class _Pool:
stage_client = SimpleNamespace(custom_process_input_func=aura2tts)
orchestrator = SimpleNamespace(
stage_pools={2: _Pool()},
_stage_receives_async_chunks=lambda stage_id: False,
)
req_state = SimpleNamespace(
session_owned=True,
final_stage_id=3,
request_id="req-1",
prompt={"additional_information": {}},
streaming=SimpleNamespace(bridge_states={}),
)
sentence = "这是一段用来直接测试句子级语音交接而且长度已经超过三十个字的内容。"
output = SimpleNamespace(
finished=False,
request_id="req-1",
outputs=[SimpleNamespace(cumulative_text=sentence)],
)
plan = AuraDuplexPlugin(_encode_audio).plan_partial_stage_output(orchestrator, 1, 0, output, req_state)
assert plan is not None
assert plan.close_only is False
assert plan.is_final_update is False
assert plan.output.text == sentence.strip()
assert req_state.prompt["additional_information"]["aura_tts_partial"] is True
assert AuraDuplexPlugin(_encode_audio).plan_partial_stage_output(orchestrator, 0, 0, output, req_state) is None
def _sentence_plan_fixture() -> tuple[SimpleNamespace, SimpleNamespace]:
def aura2tts() -> None:
return None
class _Pool:
stage_client = SimpleNamespace(custom_process_input_func=aura2tts)
orchestrator = SimpleNamespace(
stage_pools={2: _Pool()},
_stage_receives_async_chunks=lambda stage_id: False,
)
req_state = SimpleNamespace(
session_owned=True,
final_stage_id=3,
request_id="req-1",
prompt={"additional_information": {}},
streaming=SimpleNamespace(bridge_states={}),
)
return orchestrator, req_state
def _stage1_output(text: str, *, finished: bool) -> SimpleNamespace:
return SimpleNamespace(
finished=finished,
request_id="req-1",
outputs=[SimpleNamespace(cumulative_text=text)],
)
def test_finished_remainder_stays_resumable_until_close_sentinel() -> None:
"""A second sentence at Stage1 finish must not be the Talker end sentinel.
The first sentence is already a resumable Talker request. Marking the
remainder ``is_final_update`` makes ``StreamingUpdate.from_request``
return None, which aborts that in-flight sentence before Code2Wav has a
full codec group.
"""
orchestrator, req_state = _sentence_plan_fixture()
plugin = AuraDuplexPlugin(_encode_audio)
first = "我看到你身后是一个很明亮的办公室环境，天花板上能看到白色的管道和通风口，"
full = first + "远处还有办公桌和绿植。"
opened = plugin.plan_partial_stage_output(orchestrator, 1, 0, _stage1_output(first, finished=False), req_state)
assert opened is not None
assert opened.is_final_update is False
assert opened.queue_close_after is False
assert req_state.prompt["additional_information"]["aura_tts_close_only"] is False
tail = plugin.plan_partial_stage_output(orchestrator, 1, 0, _stage1_output(full, finished=True), req_state)
assert tail is not None
assert tail.close_only is False
assert tail.queue_close_after is True
assert tail.is_final_update is False
assert tail.output.text == "远处还有办公桌和绿植。"
assert req_state.prompt["additional_information"]["aura_tts_close_only"] is False
followup = plugin.partial_stage_followup(tail, req_state)
assert followup is not None
assert followup.is_final_update is True
assert followup.close_only is True
assert followup.output.text == ""
assert req_state.prompt["additional_information"]["aura_tts_close_only"] is True
def test_single_finished_sentence_is_one_non_resumable_submit() -> None:
orchestrator, req_state = _sentence_plan_fixture()
plugin = AuraDuplexPlugin(_encode_audio)
sentence = "你好，我看到一个戴眼镜、穿黑T恤的男生正对着镜头说话呢。"
plan = plugin.plan_partial_stage_output(orchestrator, 1, 0, _stage1_output(sentence, finished=True), req_state)
assert plan is not None
assert plan.queue_close_after is False
assert plan.is_final_update is True
assert plan.close_only is False
assert plugin.partial_stage_followup(plan, req_state) is None
def test_aura_draining_stages_are_declared_by_the_plugin() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
assert plugin.draining_stage_ids(stage_count=4) == frozenset({2, 3})
assert plugin.draining_stage_ids(stage_count=2) == frozenset()
def test_spoken_stage0_text_is_the_user_transcript() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
output = SimpleNamespace(
finished=True,
outputs=[
SimpleNamespace(text="language Chinese<asr_text>你好", cumulative_text="language Chinese<asr_text>你好")
],
)
prompt = {"additional_information": {"is_speech": True}}
assert plugin.user_transcript(stage_id=0, output=output, prompt=prompt, finished=True) == "你好"
def test_vision_follow_stage0_text_is_not_shown_as_user_speech() -> None:
plugin = AuraDuplexPlugin(_encode_audio)
output = SimpleNamespace(finished=True, text="嗯")
prompt = {"additional_information": {"is_speech": False}}
assert plugin.user_transcript(stage_id=0, output=output, prompt=prompt, finished=True) is None
assert plugin.user_transcript(stage_id=1, output=output, prompt=prompt, finished=True) is None
assert plugin.user_transcript(stage_id=0, output=output, prompt=prompt, finished=False) is None
tests/model_executor/models/aura_omni/duplex/test_sentence_decode.py
+62
View File
@@ -0,0 +1,62 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Growing Stage1 token ids must decode to the same text as one full decode."""
from __future__ import annotations
from vllm_omni.model_executor.models.aura_omni.duplex.sentence_tts import decode_growing_token_ids
def _assert_matches_full(decode, ids: list[int]) -> None:
cache: dict[str, object] = {}
for end in range(1, len(ids) + 1):
got = decode_growing_token_ids(decode, ids[:end], cache)
assert got == decode(ids[:end])
def test_growing_decode_matches_full_decode_for_concat_pieces() -> None:
vocab = {1: "你", 2: "好", 3: "。", 4: "世", 5: "界"}
def decode(token_ids: list[int], skip_special_tokens: bool = False) -> str:
del skip_special_tokens
return "".join(vocab[token_id] for token_id in token_ids)
_assert_matches_full(decode, [1, 2, 3, 4, 5])
def test_growing_decode_matches_full_decode_when_the_previous_piece_changes() -> None:
"""Token 9 rewrites the previous character. The tail window must reject a bad splice."""
def decode(token_ids: list[int], skip_special_tokens: bool = False) -> str:
del skip_special_tokens
chars: list[str] = []
for token_id in token_ids:
if token_id == 9:
if chars:
chars.pop()
chars.append("X")
else:
chars.append(chr(96 + token_id))
return "".join(chars)
_assert_matches_full(decode, [1, 2, 9, 3])
def test_growing_decode_matches_wordpiece_spacing() -> None:
vocab = {1: "hello", 2: "##world", 3: "。"}
def decode(token_ids: list[int], skip_special_tokens: bool = False) -> str:
del skip_special_tokens
parts: list[str] = []
for index, token_id in enumerate(token_ids):
piece = vocab[token_id]
if piece.startswith("##"):
parts.append(piece[2:])
else:
if index:
parts.append(" ")
parts.append(piece)
return "".join(parts)
_assert_matches_full(decode, [1, 2, 3, 1])
tests/model_executor/models/qwen3_tts/test_qwen3_tts_talker_preprocess.py
+32
View File
@@ -389,6 +389,38 @@ def test_decode_batch_preprocess_matches_decode_state_updates(mrv2):
assert updates[1]["hidden_states"]["trailing_text"].numel() == 0
def test_decode_batch_zero_fills_missing_last_hidden(monkeypatch):
tts_pad = torch.full((1, 4), -1.0, dtype=torch.bfloat16)
model = _make_minimal_talker(tts_pad_embed=tts_pad)
def fake_embed_input_ids(input_ids):
return input_ids.to(torch.float32).reshape(-1, 1, 1).expand(-1, 1, 4)
model.embed_input_ids = fake_embed_input_ids
warnings: list[tuple[object, ...]] = []
monkeypatch.setattr(
"vllm_omni.model_executor.models.qwen3_tts.qwen3_tts_talker.logger.warning_once",
lambda *args, **kwargs: warnings.append(args),
)
trailing = torch.arange(8, dtype=torch.float32).reshape(2, 4)
_, _, past_hidden, _, _ = model.preprocess_decode_batch(
input_ids=torch.tensor([101], dtype=torch.long),
req_infos=[
{
"text": ["hello"],
"task_type": ["CustomVoice"],
"hidden_states": {"trailing_text": trailing},
"meta": {"talker_text_offset": 0},
}
],
)
assert torch.equal(past_hidden.cpu(), torch.zeros((1, 4), dtype=torch.bfloat16))
assert len(warnings) == 1
assert "zero-filling" in str(warnings[0][0])
def _stub_text_embedding(device_param: torch.nn.Parameter):
"""Build a lambda that emulates ``nn.Embedding`` for the ``_device()`` helper.
tests/model_executor/stage_input_processors/test_aura_omni.py
+111 -2
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from types import SimpleNamespace
@@ -10,6 +10,9 @@ from vllm_omni.model_executor.models.qwen3_tts.prompt_embeds_builder import (
)
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
SILENT_TEXT,
_estimate_tts_prompt_len_from_token_ids,
_estimate_tts_prompt_len_official,
_normalize_asr_transcript,
asr2aura,
aura2tts,
)
@@ -166,7 +169,7 @@ def test_aura2tts_supports_custom_voice_mode():
assert tts_input["additional_information"]["task_type"] == ["CustomVoice"]
assert tts_input["additional_information"]["speaker"] == ["Vivian"]
assert "ref_audio" not in tts_input["additional_information"]
assert len(tts_input["prompt_token_ids"]) == 14
assert len(tts_input["prompt_token_ids"]) > 0
def test_aura2tts_passes_token_ids_to_qwen3_tts_when_enabled():
@@ -196,3 +199,109 @@ def test_aura2tts_passes_token_ids_to_qwen3_tts_when_enabled():
def test_aura2tts_drops_silent_response():
assert aura2tts([_source_output(SILENT_TEXT)]) == []
assert aura2tts([_source_output(f"{SILENT_TEXT}<|im_end|>")]) == []
def test_aura2tts_customvoice_uses_official_prompt_len_not_aura_token_count():
text = "当然可以，我正看着你呢。"
prompt = {
"additional_information": {
"tts_task_type": ["CustomVoice"],
"tts_speaker": ["Vivian"],
"tts_language": ["Chinese"],
}
}
aura_token_ids = list(range(80))
tts_info = {
"task_type": ["CustomVoice"],
"language": ["Chinese"],
"instruct": [""],
"text": [text],
"speaker": ["Vivian"],
}
official = _estimate_tts_prompt_len_official(tts_info, task_type="CustomVoice")
if official is None:
pytest.skip("Qwen3-TTS tokenizer unavailable for official prompt_len")
old_heuristic = _estimate_tts_prompt_len_from_token_ids(
aura_token_ids,
task_type="CustomVoice",
language="Chinese",
instruct="",
)
assert official != old_heuristic
[tts_input] = aura2tts(
[_source_output(text, token_ids=aura_token_ids)],
prompt=[prompt],
)
assert len(tts_input["prompt_token_ids"]) == official
assert tts_input["additional_information"]["instruct"] == [""]
def test_aura2tts_strips_im_end_from_spoken_text():
prompt = {
"additional_information": {
"tts_ref_audio": ["ref.wav"],
"tts_ref_text": ["Reference transcript sample."],
}
}
[tts_input] = aura2tts([_source_output("你好。<|im_end|>")], prompt=[prompt])
assert tts_input["additional_information"]["text"] == ["你好。"]
def test_aura2tts_does_not_treat_chinese_silence_as_special_token():
# [沉默] is not <|silent|>; prompt/session must instruct the real token.
prompt = {
"additional_information": {
"tts_ref_audio": ["ref.wav"],
"tts_ref_text": ["Reference transcript sample."],
}
}
assert len(aura2tts([_source_output("[沉默]")], prompt=[prompt])) == 1
def test_normalize_asr_transcript_strips_qwen3_asr_markup() -> None:
raw = "language Chinese<asr_text>出现《古韵》这本书的时候，提醒我。"
assert _normalize_asr_transcript(raw) == "出现《古韵》这本书的时候，提醒我。"
assert _normalize_asr_transcript("出现古韵这本书的时候提醒我。") == "出现古韵这本书的时候提醒我。"
def test_next_duplex_sentence_chunk_batches_until_min_chars(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("VLLM_AURA_SENTENCE_TTS", raising=False)
monkeypatch.delenv("VLLM_AURA_SENTENCE_TTS_MIN_CHARS", raising=False)
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
next_duplex_sentence_chunk,
)
state: dict[str, object] = {}
first = "甲" * 12 + "。"
assert next_duplex_sentence_chunk(state, first, finished=False) is None
second = first + "乙" * 20 + "。"
chunk = next_duplex_sentence_chunk(state, second, finished=False)
assert chunk is not None
assert chunk.startswith("甲" * 12)
assert "乙" * 20 in chunk
def test_next_duplex_sentence_chunk_holds_think_silent_and_flushes(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("VLLM_AURA_SENTENCE_TTS", "1")
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
next_duplex_sentence_chunk,
)
assert next_duplex_sentence_chunk({}, "<think>still thinking", finished=False) is None
assert next_duplex_sentence_chunk({}, "<|silent|>", finished=True) is None
assert next_duplex_sentence_chunk({}, '<tool_call>{"a":1}</tool_call>', finished=False) is None
state: dict[str, object] = {}
assert next_duplex_sentence_chunk(state, "你好。", finished=False) is None
assert next_duplex_sentence_chunk(state, "你好。", finished=True) == "你好。"
def test_next_duplex_sentence_chunk_disabled(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("VLLM_AURA_SENTENCE_TTS", "0")
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
next_duplex_sentence_chunk,
)
assert next_duplex_sentence_chunk({}, "甲" * 40 + "。", finished=False) is None
tests/model_executor/stage_input_processors/test_aura_omni_duplex_history.py
+273
View File
@@ -0,0 +1,273 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Logic-equivalent checks adapted from AURA_026_p0exp stage processor tests."""
from __future__ import annotations
from types import SimpleNamespace
import pytest
from vllm_omni.model_executor.models.aura_omni.duplex.history import (
drop_session_history,
get_or_create_session_history,
)
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
SILENT_TEXT,
asr2aura,
aura2tts,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def _source_output(text: str, request_id: str = "r0") -> SimpleNamespace:
return SimpleNamespace(
request_id=request_id,
outputs=[SimpleNamespace(text=text, token_ids=[1, 2, 3])],
)
def test_asr2aura_carries_video_and_transcript() -> None:
prompt = {
"additional_information": {
"aura_system_prompt": "sys",
"deferred_multi_modal_data": {"image": ["frame"]},
},
"multi_modal_data": {},
}
[next_input] = asr2aura([_source_output("What is happening now?")], prompt=[prompt])
assert "What is happening now?" in next_input["prompt"]
assert next_input["multi_modal_data"].get("image") == ["frame"]
def test_asr2aura_supports_video_only_observation() -> None:
prompt = {
"additional_information": {
"deferred_multi_modal_data": {"image": ["frame"]},
},
"multi_modal_data": {},
}
[next_input] = asr2aura([_source_output("")], prompt=[prompt])
assert next_input["multi_modal_data"].get("image") == ["frame"]
def test_asr2aura_duplex_empty_transcript_is_not_a_vision_user_turn() -> None:
drop_session_history("duplex-vision")
prompt = {
"additional_information": {
"aura_duplex": True,
"session_id": "duplex-vision",
"is_speech": False,
"deferred_multi_modal_data": {"image": ["frame"]},
},
"multi_modal_data": {},
}
[next_input] = asr2aura([_source_output("noise")], prompt=[prompt])
assert "[vision]" not in next_input["prompt"]
history = get_or_create_session_history("duplex-vision")
history.commit_turn("saw a book")
assert all(message.get("content") != "[vision]" for message in history.messages)
assert [message["role"] for message in history.messages] == ["assistant"]
drop_session_history("duplex-vision")
def test_asr2aura_duplex_history_videos_match_prompt_pads() -> None:
"""Retained clips then this clip: one <|video_pad|> each, same order as mm data."""
drop_session_history("duplex-pads")
history = get_or_create_session_history("duplex-pads")
history.begin_user_turn("看着手", video=("clip-a", {"fps": 2.0}))
history.commit_turn("好的")
prompt = {
"additional_information": {
"aura_duplex": True,
"session_id": "duplex-pads",
"is_speech": False,
"deferred_multi_modal_data": {"video": [("clip-b", {"fps": 2.0})]},
},
"multi_modal_data": {},
}
[next_input] = asr2aura([_source_output("")], prompt=[prompt])
text = next_input["prompt"]
assert text.count("<|video_pad|>") == 2
assert "clip-a" not in text and "clip-b" not in text
assert next_input["multi_modal_data"]["video"] == [
("clip-a", {"fps": 2.0}),
("clip-b", {"fps": 2.0}),
]
assert text.index("<|video_pad|>") < text.rindex("<|im_start|>user")
drop_session_history("duplex-pads")
def test_asr2aura_duplex_uses_session_history_prefix() -> None:
drop_session_history("duplex-hist")
history = get_or_create_session_history("duplex-hist")
history.begin_user_turn("prev")
history.commit_turn("prev-answer")
prompt = {
"additional_information": {
"aura_duplex": True,
"session_id": "duplex-hist",
"deferred_multi_modal_data": {"image": ["frame"]},
},
"multi_modal_data": {},
}
[next_input] = asr2aura([_source_output("next question")], prompt=[prompt])
assert "prev-answer" in next_input["prompt"]
assert "next question" in next_input["prompt"]
drop_session_history("duplex-hist")
def test_asr2aura_vision_follow_keeps_previous_assistant_in_prefix() -> None:
drop_session_history("duplex-watch")
history = get_or_create_session_history("duplex-watch")
history.begin_user_turn("盯着滑鼠")
history.commit_turn("好的，出现的时候我会说")
prompt = {
"additional_information": {
"aura_duplex": True,
"session_id": "duplex-watch",
"is_speech": False,
"deferred_multi_modal_data": {"image": ["frame"]},
},
"multi_modal_data": {},
}
[next_input] = asr2aura([_source_output("noise")], prompt=[prompt])
assert "好的，出现的时候我会说" in next_input["prompt"]
drop_session_history("duplex-watch")
def test_aura2tts_drops_silent_response() -> None:
prompt = {"additional_information": {"tts_task_type": "CustomVoice"}}
assert aura2tts([_source_output(SILENT_TEXT)], prompt=[prompt]) == []
def test_aura2tts_duplex_does_not_commit_history() -> None:
"""Stage1 history has one owner: commit_model_context. aura2tts must not write a second row."""
drop_session_history("duplex-silent")
history = get_or_create_session_history("duplex-silent")
history.begin_user_turn("look")
prompt = {
"additional_information": {
"aura_duplex": True,
"session_id": "duplex-silent",
"tts_task_type": "CustomVoice",
}
}
assert aura2tts([_source_output(SILENT_TEXT)], prompt=[prompt]) == []
assert all(message.get("role") != "assistant" for message in history.messages)
from vllm_omni.model_executor.models.aura_omni.duplex.plugin import AuraDuplexPlugin
AuraDuplexPlugin(encode_audio=lambda *_a, **_k: None).commit_model_context(
session_id="duplex-silent",
assistant_text="只記一次",
)
assert [message["role"] for message in history.messages] == ["user", "assistant"]
assert history.messages[-1]["content"] == "只記一次"
drop_session_history("duplex-silent")
def test_aura2tts_skips_history_on_sentence_partial() -> None:
drop_session_history("duplex-partial")
history = get_or_create_session_history("duplex-partial")
history.begin_user_turn("look")
prompt = {
"additional_information": {
"aura_duplex": True,
"aura_tts_partial": True,
"session_id": "duplex-partial",
"tts_task_type": "CustomVoice",
"tts_speaker": "Vivian",
}
}
assert len(aura2tts([_source_output("你好，这是一句测试。")], prompt=[prompt])) == 1
assert all(message.get("role") != "assistant" for message in history.messages)
drop_session_history("duplex-partial")
def test_aura2tts_close_only_does_not_repeat_the_sentence() -> None:
prompt = {
"additional_information": {
"aura_tts_partial": True,
"aura_tts_close_only": True,
"tts_task_type": "CustomVoice",
"tts_speaker": "Vivian",
}
}
[request] = aura2tts([_source_output("上一句不該再送。")], prompt=[prompt])
info = request["additional_information"]
assert info["text"] == [""]
assert info["max_new_tokens"] == [1]
assert request["prompt_token_ids"] == [0]
def test_duplex_interception_commits_silent_history_without_aura2tts() -> None:
"""Silent Stage1 uses DIRECT_RESPONSE; history must commit on the data-plane path."""
from vllm.outputs import CompletionOutput
from vllm_omni.engine.duplex.contracts import (
DuplexFence,
DuplexOutputAction,
DuplexOutputDecision,
duplex_ephemeral_stage_request_id,
)
from vllm_omni.model_executor.models.aura_omni.duplex.data_plane import AuraDataPlaneSession
from vllm_omni.outputs import OmniRequestOutput
from vllm_omni.outputs.duplex import attach_duplex_output_decision
drop_session_history("duplex-silent-dp")
history = get_or_create_session_history("duplex-silent-dp")
history.begin_user_turn("look")
fence = DuplexFence("duplex-silent-dp", epoch=0, turn_id=1)
request_id = duplex_ephemeral_stage_request_id(fence, stage_id=1)
output = OmniRequestOutput(
request_id=request_id,
finished=True,
stage_id=1,
outputs=[
CompletionOutput(
index=0,
text=SILENT_TEXT,
token_ids=[151669],
cumulative_logprob=None,
logprobs=None,
)
],
)
attach_duplex_output_decision(
output,
DuplexOutputDecision(
action=DuplexOutputAction.DIRECT_RESPONSE,
metadata={"model_listen": True, "duplex_direct_response": True},
),
)
plane = AuraDataPlaneSession(encode_audio=lambda *_a, **_k: None)
events = list(plane.project_output(output))
assert events and events[0].get("silent") is True
assert events[0].get("model_context_text") == SILENT_TEXT
assert events[0].get("abort_data_plane_request") is True
assert history.pending_user == "look"
from vllm_omni.model_executor.models.aura_omni.duplex.plugin import AuraDuplexPlugin
AuraDuplexPlugin(encode_audio=lambda *_a, **_k: None).commit_model_context(
session_id="duplex-silent-dp",
assistant_text=str(events[0]["model_context_text"]),
)
assert history.pending_user is None
assert history.messages[-1]["content"] == SILENT_TEXT
drop_session_history("duplex-silent-dp")
def test_proactive_silent_ticks_do_not_evict_a_real_turn() -> None:
drop_session_history("duplex-silent-ticks")
history = get_or_create_session_history("duplex-silent-ticks")
history.begin_user_turn("真正的問題")
history.commit_turn("真正的回答")
for _ in range(20):
history.begin_user_turn("")
history.commit_turn("<|silent|>")
assert history.messages[0]["content"] == "真正的問題"
assert history.messages[1]["content"] == "真正的回答"
assert len(history.messages) == 2
drop_session_history("duplex-silent-ticks")
tests/protocol/realtime/test_realtime_codec_single_source.py
+4 -1
View File
@@ -11,6 +11,7 @@ which a re-implementation cannot satisfy.
from __future__ import annotations
import base64
import dataclasses
import pytest
@@ -78,7 +79,9 @@ def test_a_duplex_command_error_is_a_realtime_protocol_error() -> None:
def test_the_duplex_append_command_is_built_from_the_shared_decoder() -> None:
defaults = protocol.RealtimeInputDefaults()
event = {"event_id": "event_1", "audio": "", "format": "pcm16", "duration_ms": 40}
# Empty ``audio`` is rejected (need bytes and/or video_frames); use silence.
silence = base64.b64encode(b"\x00\x00").decode("ascii")
event = {"event_id": "event_1", "audio": silence, "format": "pcm16", "duration_ms": 40}
decoded = protocol.decode_audio_append(event, defaults=defaults)
command = duplex_codec.build_append_audio(event, defaults=defaults)
vllm_omni/benchmarks/omniinteract.py
+67 -3
View File
@@ -968,6 +968,42 @@ def _populate_response_metrics(
result.duplex_session_metrics = bundle.session_metrics
def _session_capabilities(client: _RealtimeSession) -> dict[str, object]:
info = getattr(getattr(client, "_client", None), "session_info", None)
if isinstance(info, dict) and isinstance(info.get("capabilities"), dict):
return info["capabilities"]
for event in client.events.events:
if event.get("type") != "session.created":
continue
session = event.get("session")
if isinstance(session, dict) and isinstance(session.get("capabilities"), dict):
return session["capabilities"]
return {}
def _wants_annotation_video_clock(client: _RealtimeSession) -> bool:
"""Video-required turn-commit sessions use the AURA annotation clock.
The shared 1 FPS soundtrack path is unchanged unless both capability bits
are set. The AURA module is imported only in that case.
"""
caps = _session_capabilities(client)
required = caps.get("required_input_modalities") or ()
if isinstance(required, str):
required = (required,)
try:
video_required = "video" in {str(item) for item in required}
except TypeError:
return False
if not video_required or not caps.get("supports_turn_commit_only"):
return False
from vllm_omni.model_executor.models.aura_omni.benchmarks.omniinteract_clock import (
wants_annotation_video_clock,
)
return wants_annotation_video_clock(caps)
async def run_omniinteract_case(
case: OmniInteractCase,
config: OmniInteractBenchmarkConfig,
@@ -1020,10 +1056,38 @@ async def run_omniinteract_case(
try:
input_duration_s = len(pcm) / (PCM16_SAMPLE_RATE * PCM16_BYTES_PER_SAMPLE)
upload_timeout_s = config.timeout_s + input_duration_s
try:
chunks, frame_count, mean_lag, max_lag = await asyncio.wait_for(
stream_inputs(client, pcm, frames, playback), timeout=upload_timeout_s
if _wants_annotation_video_clock(client):
from vllm_omni.model_executor.models.aura_omni.benchmarks.omniinteract_clock import (
ANNOTATION_VIDEO_FPS,
stream_annotation_clock,
)
duration, _, frames = await asyncio.to_thread(
prepare_media,
case.video_path,
ANNOTATION_VIDEO_FPS,
timeout_s=config.media_timeout_s,
max_duration_s=config.max_video_duration_s,
)
if not any(frames):
raise ValueError(f"No video frames were decoded from {case.video_path}")
upload_timeout_s = config.timeout_s + duration
stream = stream_annotation_clock(
client,
frames,
playback,
video_path=case.video_path,
fps=ANNOTATION_VIDEO_FPS,
)
logger.info(
"OmniInteract annotation video clock: video=%s fps=%s",
case.video_path.name,
ANNOTATION_VIDEO_FPS,
)
else:
stream = stream_inputs(client, pcm, frames, playback)
try:
chunks, frame_count, mean_lag, max_lag = await asyncio.wait_for(stream, timeout=upload_timeout_s)
except asyncio.TimeoutError as exc:
raise TimeoutError(f"Realtime upload timed out after {upload_timeout_s:g}s") from exc
commit_from = len(client.events.events)
vllm_omni/config/environment_variable_inventory.py
+2
View File
@@ -200,6 +200,8 @@ _MODEL_INTERNALIZE = (
"VLLM_OMNI_QWEN3_CODE2WAV_CUDAGRAPH_STATS",
"VLLM_OMNI_QWEN3_CODE2WAV_CUDAGRAPH_STATS_FILE",
"VLLM_OMNI_QWEN3_CODE2WAV_CUDAGRAPH_STATS_LOG_EVERY",
"VLLM_AURA_SENTENCE_TTS",
"VLLM_AURA_SENTENCE_TTS_MIN_CHARS",
)
_MODEL_DEPRECATE_REMOVE = (
vllm_omni/config/stage_config.py
+6 -4
View File
@@ -525,10 +525,12 @@ class DuplexSessionRuntimeConfig:
# parse, and goes away with the PR that ports the last of them.
completed_append_cache_size: int = 256
server_vad_model_path: str | None = None
# Startup warmup: run this many silent 80 ms-style frames through a
# throwaway realtime session before real clients are admitted, so
# one-time costs (kernel JIT, first prefill/decode paths, codec caches)
# never land on the first user. 0 disables the warmup.
# Startup warmup, before real ``/v1/realtime`` clients are admitted.
# Audio-primary models run this many silent frames; 0 disables that path.
# Video-required models (AURA) still run one short non-silent audio chunk
# plus one image when this is 0, so ASR, vision, Talker and Code2Wav
# compile their real shapes. The empty per-stage JIT registry does not.
# A negative value disables every startup warmup.
warmup_frames: int = 0
def __post_init__(self) -> None:
vllm_omni/deploy/aura_omni.yaml
+14 -3
View File
@@ -1,11 +1,21 @@
# AURA Omni deploy profile: Qwen3-ASR -> AURA/Qwen3-VL -> Qwen3-TTS Talker -> Code2Wav.
#
# Supported online path is Realtime duplex (`/v1/realtime?duplex=1`).
# The aura_omni pipeline declares duplex_plugin; DuplexOmniEngine requires
# session_mode: duplex (same pattern as MiniCPM-o duplex deploys).
# For Realtime, set async_chunk: true (see aura_omni_duplex_smoke.yaml).
#
# Pass `--deploy-config vllm_omni/deploy/aura_omni.yaml` and point the parent
# `--model` at any local placeholder repo if needed; each stage overrides its
# own model below.
pipeline: aura_omni
# AURA has a semantic ASR -> AURA -> TTS dependency. Do not prewarm downstream
# stages before AURA text is converted into Qwen3-TTS metadata by aura2tts.
session_mode: duplex
duplex_session:
max_sessions: 1
# Default false: safer if this file is ever loaded by turn-based OmniOrchestrator,
# which does not apply DuplexOmniOrchestrator._stage_receives_async_chunks().
# Duplex Realtime Stage2→3 (Talker/Code2Wav) still needs async chunks — set
# async_chunk: true for /v1/realtime?duplex=1 (see aura_omni_duplex_smoke.yaml).
async_chunk: false
connectors:
@@ -18,7 +28,8 @@ connectors:
connector_get_max_wait: 300
codec_chunk_frames: 25
codec_left_context_frames: 72
initial_codec_chunk_frames: 1
# initial=1 can stall Code2Wav on the first undersized chunk under duplex.
initial_codec_chunk_frames: 8
decode_cudagraph_capture_sizes: [25, 73, 97, 169, 325]
stages:
vllm_omni/engine/duplex/commands.py
+6 -1
View File
@@ -94,6 +94,8 @@ class UpdateSession(DuplexCommand, _duplex_wire.UpdateSession):
@dataclass(frozen=True, slots=True, kw_only=True)
class AppendAudio(DuplexCommand, _duplex_wire.AppendAudio):
type: ClassVar[str] = "input_audio_buffer.append"
#: Empty ``audio`` with ``video_frames`` is legal when capabilities allow video without audio.
audio: bytes = b""
def payload(self) -> dict[str, object]:
data = DuplexCommand.payload(self)
@@ -108,7 +110,10 @@ class AppendAudio(DuplexCommand, _duplex_wire.AppendAudio):
merged: dict[str, object] = dict(hints)
merged.update(data)
data = merged
data["audio"] = base64.b64encode(self.audio).decode("ascii")
if self.audio:
data["audio"] = base64.b64encode(self.audio).decode("ascii")
else:
data.pop("audio", None)
if not data.get("video_frames"):
data.pop("video_frames", None)
return data
vllm_omni/engine/duplex/config.py
+33
View File
@@ -94,6 +94,13 @@ class DuplexCapabilities:
supports_stage_resumption: bool = False
supports_scheduler_native_append: bool = False
supports_core_resumable_request: bool = False
# Prior assistant TTS may keep draining while a new user commit is admitted.
# Not barge-in (which aborts prior TTS).
supports_concurrent_turn_requests: bool = False
#: Input modality contract. Defaults match audio-primary duplex (e.g. MiniCPM):
#: audio required, video optional when attached to an audio unit.
required_input_modalities: frozenset[str] = field(default_factory=lambda: frozenset({"audio"}))
optional_input_modalities: frozenset[str] = field(default_factory=lambda: frozenset({"video"}))
supports_stage_connector_handoff: bool = False
supports_independent_io_streams: bool = False
supports_realtime_endpoint: bool = False
@@ -142,6 +149,9 @@ class DuplexCapabilities:
"supports_stage_resumption": self.supports_stage_resumption,
"supports_scheduler_native_append": self.supports_scheduler_native_append,
"supports_core_resumable_request": self.supports_core_resumable_request,
"supports_concurrent_turn_requests": self.supports_concurrent_turn_requests,
"required_input_modalities": sorted(self.required_input_modalities),
"optional_input_modalities": sorted(self.optional_input_modalities),
"supports_stage_connector_handoff": self.supports_stage_connector_handoff,
"supports_independent_io_streams": self.supports_independent_io_streams,
"supports_realtime_endpoint": self.supports_realtime_endpoint,
@@ -166,6 +176,29 @@ class DuplexCapabilities:
"target_barge_in_latency_ms": self.target_barge_in_latency_ms,
}
def accepts_input_modality(self, modality: str) -> bool:
return modality in self.required_input_modalities or modality in self.optional_input_modalities
def allows_video_without_audio(self) -> bool:
"""Whether video without speech/audio is legal turn content (AURA video-compulsory)."""
return self.accepts_input_modality("video") and "audio" not in self.required_input_modalities
def validate_append_modalities(self, *, has_audio: bool, has_video: bool) -> str | None:
"""Return an error message when the append modalities violate this contract."""
if not has_audio and not has_video:
return "input_audio_buffer.append requires audio and/or video_frames"
if has_audio and not self.accepts_input_modality("audio"):
return "This duplex model does not accept audio input"
if has_video and not self.accepts_input_modality("video"):
return "This duplex model does not accept video_frames"
if "audio" in self.required_input_modalities and not has_audio:
return "This duplex model requires audio on input_audio_buffer.append"
if "video" in self.required_input_modalities and not has_video:
return "This duplex model requires video_frames on input_audio_buffer.append"
if has_video and not has_audio and not self.allows_video_without_audio():
return "This duplex model requires audio; video-only append is not allowed"
return None
@dataclass
class DuplexPlaybackCursor:
vllm_omni/engine/duplex/contracts.py
+74 -7
View File
@@ -11,11 +11,13 @@ from __future__ import annotations
import base64
from abc import ABC, abstractmethod
from collections.abc import Mapping
from collections.abc import Iterable, Mapping
from dataclasses import dataclass, field
from enum import Enum
from types import MappingProxyType
import regex as re
@dataclass(frozen=True, slots=True)
class DuplexFence:
@@ -78,6 +80,9 @@ class DuplexStageSubmission:
context: DuplexStageRequestContext
prompt: Mapping[str, object]
already_submitted: bool
# True: resume/update an existing stage0 id. False: open a new ephemeral id.
# Distinct from DuplexCapabilities.supports_core_resumable_request.
resumable: bool = True
def __post_init__(self) -> None:
object.__setattr__(self, "prompt", MappingProxyType(dict(self.prompt)))
@@ -157,22 +162,80 @@ def duplex_resource_request_id(fence: DuplexFence, role: str) -> str:
return f"duplex-s.{encoded_session_id}.e.{fence.epoch}.r.{role}"
def duplex_resource_request_belongs_to_session(request_id: str, session_id: str) -> bool:
"""Return whether a current-format resource request belongs to a session."""
def duplex_ephemeral_stage_request_id(fence: DuplexFence, *, stage_id: int) -> str:
"""Turn-scoped Stage request id for non-resumable (ephemeral) duplex models."""
return duplex_resource_request_id(fence, f"stage{stage_id}-turn{fence.turn_id}")
_EPHEMERAL_TURN_IN_REQUEST_ID = re.compile(r"\.r\.stage\d+-turn(\d+)$")
def duplex_turn_id_from_request_id(request_id: str | None) -> int | None:
"""Parse ephemeral ``…r.stage{N}-turn{T}`` ids."""
if not isinstance(request_id, str):
return None
match = _EPHEMERAL_TURN_IN_REQUEST_ID.search(request_id)
return int(match.group(1)) if match else None
def duplex_same_turn_request_ids(request_id: str, candidate_ids: Iterable[str]) -> list[str]:
"""Other ephemeral stage ids from the same session, epoch, and turn.
``request_id`` itself is not included. Non-ephemeral ids are ignored.
"""
turn_id = duplex_turn_id_from_request_id(request_id)
if turn_id is None or ".r." not in request_id:
return []
prefix = request_id.rsplit(".r.", 1)[0] + ".r."
return [
candidate
for candidate in candidate_ids
if candidate != request_id
and isinstance(candidate, str)
and candidate.startswith(prefix)
and duplex_turn_id_from_request_id(candidate) == turn_id
]
def _duplex_resource_request_fields(request_id: str | None) -> tuple[str, str, str] | None:
"""Split ``duplex-s.<b64url>.e.<epoch>.r.<role>`` or return None."""
if not isinstance(request_id, str):
return None
parts = request_id.split(".")
if len(parts) != 6 or parts[0] != "duplex-s" or parts[2] != "e" or parts[4] != "r":
return False
return None
return parts[1], parts[3], parts[5]
def duplex_session_id_from_request_id(request_id: str | None) -> str | None:
"""Decode the session id from ``duplex-s.<b64url>.e.<epoch>.r.<role>``."""
fields = _duplex_resource_request_fields(request_id)
if fields is None:
return None
encoded, _, _ = fields
pad = "=" * (-len(encoded) % 4)
try:
int(parts[3])
return base64.urlsafe_b64decode(encoded + pad).decode("utf-8")
except (ValueError, UnicodeDecodeError):
return None
def duplex_resource_request_belongs_to_session(request_id: str, session_id: str) -> bool:
"""Return whether a current-format resource request belongs to a session."""
fields = _duplex_resource_request_fields(request_id)
if fields is None:
return False
encoded, epoch, role = fields
try:
int(epoch)
except ValueError:
return False
role = parts[5]
if not role or any(
character not in "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-" for character in role
):
return False
encoded_session_id = base64.urlsafe_b64encode(session_id.encode("utf-8")).decode("ascii").rstrip("=")
return parts[1] == encoded_session_id
return encoded == encoded_session_id
__all__ = [
@@ -187,6 +250,10 @@ __all__ = [
"DuplexStageSubmission",
"DuplexStageSubmissionResult",
"duplex_data_plane_request_info",
"duplex_ephemeral_stage_request_id",
"duplex_turn_id_from_request_id",
"duplex_resource_request_belongs_to_session",
"duplex_resource_request_id",
"duplex_same_turn_request_ids",
"duplex_session_id_from_request_id",
]
vllm_omni/engine/duplex/pcm_reservation.py
+53
View File
@@ -0,0 +1,53 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Ordered PCM reservation ownership shared by duplex input buffers.
Models keep their own emit policy. This only transfers a reserved span:
commit drops it, rollback puts its bytes back.
"""
from __future__ import annotations
def commit_ordered_reservation(reservations: list, reservation: object, *, head_only: bool) -> None:
"""Drop one active reservation. ``head_only`` rejects an out-of-order commit."""
if not getattr(reservation, "_active", False):
return
if head_only:
if not reservations or reservations[0] is not reservation:
raise RuntimeError("PCM append reservations must commit in wire order")
reservations.pop(0)
elif reservation in reservations:
reservations.remove(reservation)
reservation._active = False
def rollback_ordered_reservation(
reservations: list,
reservation: object,
buffer: bytearray,
*,
active_only: bool,
) -> list:
"""Restore this reservation and every later one. Returns the restored items."""
if not getattr(reservation, "_active", False):
return []
try:
index = reservations.index(reservation)
except ValueError:
reservation._active = False
return []
rolled_back = list(reservations[index:])
restore = bytearray()
for item in rolled_back:
if active_only and not getattr(item, "_active", False):
continue
restore.extend(getattr(item, "_raw", b""))
item._active = False
del reservations[index:]
buffer[:0] = restore
reservation._active = False
return rolled_back
__all__ = ["commit_ordered_reservation", "rollback_ordered_reservation"]
vllm_omni/engine/duplex/plugin.py
+112
View File
@@ -15,6 +15,7 @@ from __future__ import annotations
import asyncio
from abc import ABC, abstractmethod
from collections.abc import Callable, Iterable, Mapping
from dataclasses import dataclass
from importlib import import_module
from typing import TYPE_CHECKING
@@ -170,6 +171,25 @@ class DuplexDataPlane(ABC):
EncodeAudio = Callable[[object, int, str, float | None], str | None]
@dataclass(frozen=True, slots=True)
class PartialStageForward:
"""One downstream update the orchestrator should submit.
``close_only`` is a final update with no new sentence. ``output`` is the
model-built payload; the orchestrator does not interpret its text.
``queue_close_after`` means this chunk still has text, but Stage1 has
finished and an earlier sentence is already in flight. The text must be
submitted resumable. A non-resumable submit is an end sentinel
(``StreamingUpdate.from_request`` returns None) and aborts that sentence.
"""
output: object
is_final_update: bool
close_only: bool = False
queue_close_after: bool = False
class DuplexModelPlugin(ABC):
"""Everything vLLM-Omni needs to know about one full-duplex model.
@@ -237,6 +257,98 @@ class DuplexModelPlugin(ABC):
output: object,
) -> DuplexOutputDecision | None: ...
def project_intermediate_output(
self,
*,
stage_id: int,
output: object,
context: object,
) -> bool:
"""Return True to project this intermediate stage to the client.
Unlike ``decide_output``, projecting does **not** short-circuit the
pipeline: the stage output is still forwarded to the next stage.
Default is off. Orthogonal to ``projects_intermediate_outputs``
(Qwen3 Stage0); this hook is per-stage.
"""
del stage_id, output, context
return False
def user_transcript(
self,
*,
stage_id: int,
output: object,
prompt: object,
finished: bool,
) -> str | None:
"""ASR text to show as the user's words, or None.
Default models do not surface Stage0. AURA uses this for a spoken
turn only; vision-follow commits stay off the transcript.
"""
del stage_id, output, prompt, finished
return None
def plan_partial_stage_output(
self,
orchestrator: object,
stage_id: int,
replica_id: int,
output: object,
req_state: object,
) -> PartialStageForward | None:
"""Return a Talker update the orchestrator should submit, or None.
Default models do not split Stage1 text. The orchestrator owns the
actual ``_forward_to_next_stage`` call.
"""
del orchestrator, stage_id, replica_id, output, req_state
return None
def partial_stage_followup(self, plan: PartialStageForward, req_state: object) -> PartialStageForward | None:
"""Optional second submit after ``plan`` has already been forwarded.
Default models have nothing to add. AURA uses this to queue the
end sentinel only after the last sentence text is already resumable.
"""
del plan, req_state
return None
def commit_model_context(self, *, session_id: str | None, assistant_text: str) -> None:
"""Persist model-context history at a turn boundary. Default is a no-op.
This is not playback-ACK history. A model that keeps its own prompt
transcript implements this; the session runner only decides when.
"""
del session_id, assistant_text
def release_concurrent_turn_requests(
self,
*,
stage_id: int,
segment_finished: bool,
output: object,
context: object,
) -> bool:
"""Return True when the next user commit may start while prior TTS drains.
The plugin chooses when that is safe. The runner must not hard-code a
stage id. Default off. Unlike barge-in, this path must not cancel the
old TTS.
"""
del stage_id, segment_finished, output, context
return False
def draining_stage_ids(self, *, stage_count: int) -> frozenset[int]:
"""Output stages that may keep running after the next user turn starts.
Empty means a concurrent turn does not overlap a previous output
stage. Shared lifecycle reads this instead of assuming a stage layout.
"""
del stage_count
return frozenset()
# ---- session policy (was ServingRuntimeAdapter) ----
@abstractmethod
vllm_omni/engine/duplex/realtime_events.py
+39 -2
View File
@@ -895,6 +895,33 @@ def _project(state: RealtimeProjectionState, event: dict[str, object]) -> list[D
state, event, response_id, status=status, status_details=status_details
),
]
if event_type == "input.transcribed":
transcript = event.get("transcript")
if not isinstance(transcript, str) or not transcript.strip():
return []
transcript = transcript.strip()
raw_item_id = event.get("realtime_item_id")
item_id = raw_item_id if isinstance(raw_item_id, str) and raw_item_id else f"item_{uuid4().hex}"
committed_item = state.conversation_items.get(item_id)
if isinstance(committed_item, dict):
content = committed_item.get("content")
if not isinstance(content, list):
content = []
committed_item["content"] = content
updated = False
for part in content:
if isinstance(part, dict) and part.get("type") == "input_audio":
part["transcript"] = transcript
updated = True
break
if not updated:
content.append({"type": "input_audio", "transcript": transcript})
return [
InputTranscriptionCompleted(
item_id=item_id,
transcript=transcript,
)
]
if event_type == "input.committed":
event_item_id = event.get("realtime_item_id")
item_id = (
@@ -1168,6 +1195,7 @@ def note_input_append(
payload: dict[str, object],
*,
vad_result: TurnDetectionResult | None = None,
allows_video_without_audio: bool = False,
) -> list[DuplexEvent]:
"""Update the input-buffer projection for one appended chunk; returns typed events.
@@ -1180,9 +1208,18 @@ def note_input_append(
audio = payload.get("audio")
looks_like_speech = bool(vad_result.is_speech) if vad_result is not None else payload.get("is_speech") is not False
has_audio = isinstance(audio, str) and bool(audio)
state.input_audio_buffer_has_audio = state.input_audio_buffer_has_audio or (looks_like_speech and has_audio)
video_frames = payload.get("video_frames")
has_video = isinstance(video_frames, list) and any(isinstance(frame, str) and frame for frame in video_frames)
# Vision-carrying silent appends are real turn content only when the model
# allows video without required audio. Without this gate, turn-mode
# camera sessions would treat silent+frames as buffer content and open a response.
state.input_audio_buffer_has_audio = (
state.input_audio_buffer_has_audio
or (looks_like_speech and has_audio)
or (has_video and allows_video_without_audio)
)
state.input_audio_buffer_had_non_speech = state.input_audio_buffer_had_non_speech or (
not looks_like_speech and has_audio
not looks_like_speech and has_audio and not (has_video and allows_video_without_audio)
)
events: list[DuplexEvent] = []
stop_ms: object = payload.get("audio_end_ms", payload.get("audio_ms", 0))
vllm_omni/engine/duplex/session/append_task.py
+9 -4
View File
@@ -158,10 +158,15 @@ class AppendAttempt:
self.ctx.run.runtime_closed = True
return False
if not emitted_response and session.epoch == self.epoch:
# Only if the session still points at this append's request:
# ``clear_request`` compares before it clears, and the id
# carries a turn suffix when the core request is not resumable.
session.clear_request(self.request_id)
# Resident Stage0 (``…r.stage0``): clear the listen-only bind.
# Ephemeral ``…r.stage{N}-turn{T}`` must stay bound after a
# listen-only append (``clear_request`` compares before clear).
if session.capabilities.supports_core_resumable_request:
session.clear_request(self.request_id)
else:
active = session.active_request_id
if isinstance(active, str) and active.endswith(".r.stage0"):
session.clear_request(active)
if self.final:
self.out.emit_events([session.signal_turn(DuplexTurnEventType.USER_STARTED.value)])
return append_ok
vllm_omni/engine/duplex/session/context.py
+4
View File
@@ -58,6 +58,10 @@ class DuplexRunState:
runtime_closed: bool = False
#: Request id of the resumable data-plane stream currently bound to the session.
stream_request_id: str | None = None
#: Plugin signaled that the next user commit may start while the current
#: assistant audio is still draining. Cleared on barge-in/cancel, or when
#: the next ephemeral turn begins. Orthogonal to barge-in (which aborts).
concurrent_turn_requests_released: bool = False
class RunnerServices(Protocol):
vllm_omni/engine/duplex/session/engine_session.py
+202 -2
View File
@@ -17,7 +17,7 @@ from __future__ import annotations
import copy
import time
from collections.abc import Callable, Mapping
from collections.abc import Callable, Iterable, Mapping
from dataclasses import dataclass, field
from types import MappingProxyType
from typing import TYPE_CHECKING, Any
@@ -105,6 +105,9 @@ class ResponseState:
active_response_input_commit_seq: int | None = None
active_response_awaits_input_commit: bool = False
last_response_id: str | None = None
#: Overlapped-input: prior-turn Stage2/3 request ids still draining under
#: their own ``response_id`` after the next turn opened a new response.
draining_response_by_request: dict[str, str] = field(default_factory=dict)
assistant_text_buffer: list[str] = field(default_factory=list)
assistant_audio_text_marks: list[DuplexAssistantAudioTextMark] = field(default_factory=list)
pending_options: ResponseCreateOptions | None = None
@@ -372,6 +375,28 @@ class DuplexEngineSession:
}
return stale
def release_resources_for_request_ids(self, request_ids: Iterable[str]) -> list[str]:
"""Drop bindings for these request ids, whichever fence they sit on.
``cancel_fence`` only releases the fence being cancelled. Overlapped
draining output stages belong to an older turn fence and would
otherwise stay until the session closes.
"""
wanted = {request_id for request_id in request_ids if isinstance(request_id, str) and request_id}
if not wanted:
return []
released = list(
dict.fromkeys(
resource.request_id for resource in self.request_resources.values() if resource.request_id in wanted
)
)
self.request_resources = {
resource_key: resource
for resource_key, resource in self.request_resources.items()
if resource.request_id not in wanted
}
return released
def cancel_fence(self, cancelled_fence: DuplexFence, next_fence: DuplexFence) -> list[str]:
stale = self.prepare_cancel_fence(cancelled_fence, next_fence)
self.release_fence(cancelled_fence)
@@ -587,8 +612,51 @@ class DuplexEngineSession:
return False
if turn_id is None:
return True
# Overlapped input opens a new response_id per released turn; each
# response only accepts its own turn. Draining prior TTS is keyed by
# request_id → response_id, not by this guard.
active_turn_id = self._response.active_response_turn_id
return active_turn_id is None or int(turn_id) == active_turn_id
if active_turn_id is None or int(turn_id) == int(active_turn_id):
return True
return False
def bind_draining_request(self, request_id: str, response_id: str) -> None:
"""Map a still-playing draining-stage request onto the response that owns it."""
if request_id and response_id:
self._response.draining_response_by_request[request_id] = response_id
def response_id_for_request(self, request_id: str | None) -> str | None:
if isinstance(request_id, str) and request_id in self._response.draining_response_by_request:
return self._response.draining_response_by_request[request_id]
return self._response.active_response_id
def pop_draining_request(self, request_id: str | None) -> str | None:
if not isinstance(request_id, str):
return None
return self._response.draining_response_by_request.pop(request_id, None)
def is_draining_request(self, request_id: str | None) -> bool:
return isinstance(request_id, str) and request_id in self._response.draining_response_by_request
def response_has_draining_request(self, response_id: str | None) -> bool:
if response_id is None:
return False
return any(resp == response_id for resp in self._response.draining_response_by_request.values())
def draining_request_ids(self) -> list[str]:
"""Request ids whose TTS is still running under a previous response."""
return list(self._response.draining_response_by_request)
def clear_draining_requests(self) -> None:
self._response.draining_response_by_request.clear()
def clear_draining_for_response(self, response_id: str | None) -> None:
"""Drop draining bindings owned by ``response_id``; leave other responses intact."""
if response_id is None:
return
self._response.draining_response_by_request = {
rid: resp for rid, resp in self._response.draining_response_by_request.items() if resp != response_id
}
def append_history_message(self, message: dict[str, object]) -> None:
self._conversation.messages.append(message)
@@ -707,6 +775,57 @@ class DuplexEngineSession:
self._response.active_options = None
self._response.pending_options = None
def snapshot_active_response_for_drain(self) -> None:
"""Keep the active response ACK-admissible after a later response starts.
Overlapped commit calls ``begin_response`` while prior TTS is still
draining. That clears the live text buffer and playback cursor.
``end_response`` is the normal snapshot, but it would also close the
response. Copy the same snapshot (and reserve the history slot) first
so a playback ACK for the draining response is not
``playback_item_not_found``.
"""
response_id = self.active_response_id
if response_id is None:
return
if response_id not in self._playback.by_response:
self._playback.by_response[response_id] = self._playback.current
assistant_text = "".join(self._response.assistant_text_buffer).strip()
message = _object_dict(role="assistant", content=assistant_text)
seq = self._response.active_response_input_commit_seq
if seq is None:
seq = self.input_commit_seq
self._conversation.assistant_response_snapshots[response_id] = (
copy.deepcopy(message),
tuple(copy.deepcopy(self._response.assistant_audio_text_marks)),
seq,
)
self.reserve_history_item(f"item_{response_id}")
def append_draining_assistant_text(self, response_id: str, text: str) -> None:
"""Append text onto a snapshotted draining response, not the active one."""
if not text:
return
snapshot = self._conversation.assistant_response_snapshots.get(response_id)
if snapshot is None:
return
message, marks, seq = snapshot
content = message.get("content")
if not isinstance(content, str):
content = ""
message["content"] = content + text
self._conversation.assistant_response_snapshots[response_id] = (message, marks, seq)
def assistant_transcript(self, response_id: str | None = None) -> str:
"""Joined assistant text for the active response, or a draining snapshot."""
if response_id is not None and response_id != self.active_response_id:
snapshot = self._conversation.assistant_response_snapshots.get(response_id)
if snapshot is None:
return ""
content = snapshot[0].get("content")
return content if isinstance(content, str) else ""
return "".join(self._response.assistant_text_buffer)
def begin_response(self, *, turn_id: int | None = None) -> str:
self._activate_response_options()
response_id = f"resp-{self.session_id}-{self.epoch}-{uuid4().hex[:8]}"
@@ -881,7 +1000,18 @@ class DuplexEngineSession:
audio_text_marks: list[dict[str, object]] | None = None,
text_requires_complete_audio: bool = False,
audio_complete: bool = False,
response_id: str | None = None,
) -> None:
if response_id is not None and response_id != self.active_response_id:
self._mark_draining_audio_sent(
response_id,
duration_ms,
text_chars=text_chars,
audio_text_marks=audio_text_marks,
text_requires_complete_audio=text_requires_complete_audio,
audio_complete=audio_complete,
)
return
playback = self._playback.current
playback.text_requires_complete_audio |= text_requires_complete_audio
playback.audio_complete |= audio_complete
@@ -911,6 +1041,52 @@ class DuplexEngineSession:
)
self.turn_state = DuplexTurnState.ASSISTANT_PLAYING
def _mark_draining_audio_sent(
self,
response_id: str,
duration_ms: int | None,
*,
text_chars: int | None,
audio_text_marks: list[dict[str, object]] | None,
text_requires_complete_audio: bool,
audio_complete: bool,
) -> None:
"""Attribute audio to a draining response, not the newly active cursor."""
playback = self._playback_cursor_for_response(response_id)
playback.text_requires_complete_audio |= text_requires_complete_audio
playback.audio_complete |= audio_complete
marks: list[DuplexAssistantAudioTextMark] = []
snapshot = self._conversation.assistant_response_snapshots.get(response_id)
if snapshot is not None:
marks = list(snapshot[1])
if duration_ms is not None:
playback.generated_ms = max(playback.generated_ms, duration_ms)
playback.sent_ms = max(playback.sent_ms, duration_ms)
if text_chars is not None and text_chars >= 0:
marks.append(
DuplexAssistantAudioTextMark(
text_chars=int(text_chars),
audio_end_ms=max(0, int(duration_ms)),
)
)
if audio_text_marks:
for raw_mark in audio_text_marks:
if not isinstance(raw_mark, dict):
continue
raw_text_chars = raw_mark.get("text_chars")
raw_audio_end_ms = raw_mark.get("audio_end_ms", raw_mark.get("audio_ms"))
if not isinstance(raw_text_chars, int | float) or not isinstance(raw_audio_end_ms, int | float):
continue
marks.append(
DuplexAssistantAudioTextMark(
text_chars=max(0, int(raw_text_chars)),
audio_end_ms=max(0, int(raw_audio_end_ms)),
)
)
if snapshot is not None:
message, _, seq = snapshot
self._conversation.assistant_response_snapshots[response_id] = (message, tuple(marks), seq)
def _playback_cursor_for_response(self, response_id: str | None = None) -> DuplexPlaybackCursor:
if response_id is None:
return self._playback.current
@@ -951,6 +1127,27 @@ class DuplexEngineSession:
return
self._playback.by_response.pop(response_id, None)
def release_finished_drain_response(self, response_id: str | None) -> None:
"""Drop unused books for a response whose TTS just finished.
Overlap leaves the next turn active, so this must not call
``end_response``. Audio that was already sent stays ACK-admissible:
the client may ``playback.ack`` after the drain ``response.done``.
A response that sent no audio has nothing to acknowledge, so its
snapshot, playback cursor, and unused history placeholder are dropped.
The live response is left untouched.
"""
if response_id is None or response_id == self.active_response_id:
return
playback = self._playback.by_response.get(response_id)
if playback is not None and max(playback.sent_ms, playback.generated_ms) > 0:
return
self._conversation.assistant_response_snapshots.pop(response_id, None)
self.release_response_playback(response_id)
item_id = f"item_{response_id}"
if item_id not in self._conversation.item_ids:
self._discard_history_item_placeholder(item_id)
def clear_playback_cursor(self) -> None:
self._playback.current = DuplexPlaybackCursor()
if self.active_response_id is not None:
@@ -1010,6 +1207,8 @@ class DuplexEngineSession:
self._response.active_response_turn_id = None
self._response.active_response_input_commit_seq = None
self._response.active_response_awaits_input_commit = False
# Keep draining bindings for other responses (older TTS may still play).
self.clear_draining_for_response(response_id)
self._clear_response_metrics()
self.turn_state = DuplexTurnState.IDLE
self._restore_response_config()
@@ -1344,6 +1543,7 @@ class DuplexEngineSession:
self._response.active_response_turn_id = None
self._response.active_response_input_commit_seq = None
self._response.active_response_awaits_input_commit = False
self._response.draining_response_by_request.clear()
self._clear_response_metrics()
self._restore_response_config()
self.turn_state = DuplexTurnState.BARGE_IN
vllm_omni/engine/duplex/session/helpers.py
+27 -1
View File
@@ -20,7 +20,7 @@ from typing import TYPE_CHECKING
import pybase64 as base64
from vllm_omni.engine.duplex.config import DuplexPlaybackCommitPolicy
from vllm_omni.engine.duplex.contracts import DuplexFence
from vllm_omni.engine.duplex.contracts import DuplexFence, duplex_resource_request_id
from vllm_omni.engine.duplex.events import ErrorEvent, OverlapDecision, error_event
from vllm_omni.engine.duplex.plugin import payload_turn_id
@@ -65,6 +65,14 @@ def append_fence(session: DuplexEngineSession, payload: object, *, epoch: int |
)
def stage0_request_id(session: DuplexEngineSession, epoch: int) -> str:
"""Stable ``stage0`` or ephemeral ``stage0-turn{T}`` request id for Stage0."""
fence = DuplexFence(session.session_id, epoch=epoch, turn_id=session.turn_id)
if session.capabilities.supports_core_resumable_request:
return duplex_resource_request_id(fence, "stage0")
return duplex_resource_request_id(fence, f"stage0-turn{fence.turn_id}")
def response_in_progress(session: DuplexEngineSession, tasks: DuplexSessionTasks) -> bool:
"""Whether the model still owns the turn.
@@ -81,6 +89,24 @@ def response_in_progress(session: DuplexEngineSession, tasks: DuplexSessionTasks
return tasks.has_response_bound_append_tasks()
def next_commit_allowed(
session: DuplexEngineSession,
tasks: DuplexSessionTasks,
*,
concurrent_turn_requests_released: bool,
) -> bool:
"""Whether a new commit may flush/submit now.
Idle sessions always allow it. When ``supports_concurrent_turn_requests`` is on and
the plugin has released the commit gate, a commit is allowed even though
prior assistant TTS/playback still counts as ``response_in_progress``.
Barge-in remains the abort path; this gate does not cancel anything.
"""
if not response_in_progress(session, tasks):
return True
return session.capabilities.supports_concurrent_turn_requests and concurrent_turn_requests_released
def assistant_playback_active(session: DuplexEngineSession) -> bool:
"""Whether the client is still playing audio the session has sent."""
return (
vllm_omni/engine/duplex/session/manager.py
+13 -4
View File
@@ -286,8 +286,18 @@ class DuplexSessionManager:
)
return
if isinstance(command, AppendAudio):
has_audio = bool(command.audio)
has_video = bool(command.video_frames)
modality_error = session.capabilities.validate_append_modalities(has_audio=has_audio, has_video=has_video)
if modality_error is not None:
self.emit(
session,
self._error_event("invalid_input_modality", modality_error, command=command),
)
return
limit = int(self.runtime_config.max_pending_input_bytes_per_session)
if not session.reserve_input_bytes(len(command.audio), limit=limit):
pending_bytes = len(command.audio) + sum(len(frame) for frame in command.video_frames)
if not session.reserve_input_bytes(pending_bytes, limit=limit):
self.emit(
session,
self._error_event(
@@ -425,9 +435,8 @@ class DuplexSessionManager:
if stage_id >= self.stage_port.stage_count:
return None
effective_fence = fence or session.fence
request_id = self.stage_request_id(
effective_fence, stage_id=stage_id, resumable=session.capabilities.supports_core_resumable_request
)
resumable = session.capabilities.supports_core_resumable_request
request_id = self.stage_request_id(effective_fence, stage_id=stage_id, resumable=resumable)
session.reserve_stage_request(stage_id, request_id, fence=effective_fence)
context = DuplexStageRequestContext(
request_id=request_id,
vllm_omni/engine/duplex/session/model_channel.py
+241 -29
View File
@@ -40,6 +40,8 @@ from vllm_omni.engine.duplex.contracts import (
DuplexOutputDecision,
DuplexStageSubmission,
duplex_data_plane_request_info,
duplex_same_turn_request_ids,
duplex_session_id_from_request_id,
)
from vllm_omni.engine.duplex.plugin import (
DuplexRuntimeConfigError,
@@ -99,6 +101,14 @@ class ModelChannel:
self._schedule_silence_continuation = schedule_silence_continuation
self._abort_request = abort_request
def _draining_stage_ids(self) -> frozenset[int]:
"""Stages the plugin keeps across a concurrent turn. Empty if undeclared."""
declare = getattr(self._ctx.plugin, "draining_stage_ids", None)
if not callable(declare):
return frozenset()
stage_count = int(getattr(self._ctx.stage_port, "stage_count", 0) or 0)
return frozenset(int(stage_id) for stage_id in declare(stage_count=stage_count))
@staticmethod
def should_commit_response_to_history(session: DuplexEngineSession, response_id: str | None) -> bool:
if response_id is not None and response_id != session.active_response_id:
@@ -214,17 +224,67 @@ class ModelChannel:
lease_operation_id = f"append:{operation_id or uuid.uuid4().hex}"
operation_started = False
stage_id = 0
request_id = self._ctx.manager.stage_request_id(
fence, stage_id=stage_id, resumable=session.capabilities.supports_core_resumable_request
)
resumable = session.capabilities.supports_core_resumable_request
request_id = self._ctx.manager.stage_request_id(fence, stage_id=stage_id, resumable=resumable)
# Ephemeral turn-commit cannot submit_update on a finished stage0 id.
# Bump turn_id and open a fresh ephemeral request instead.
if not resumable and session.stage_request_submitted(stage_id, request_id):
stale_ephemeral_id = request_id
# Keys are ``(stage_id, request_id)``; values are DuplexRequestResource.
stale_keys = list(session.request_resources.keys())
stale_ids = list(dict.fromkeys(rid for _, rid in stale_keys))
concurrent_turn = session.capabilities.supports_concurrent_turn_requests and (
self._ctx.run.concurrent_turn_requests_released
)
# Prior TTS may still drain under the same response_id; acceptance
# is gated by supports_concurrent_turn_requests, not a per-turn drain id.
session.complete_model_turn(fence.turn_id)
fence = DuplexFence(session.session_id, epoch=session.epoch, turn_id=session.turn_id)
request_id = self._ctx.manager.stage_request_id(fence, stage_id=stage_id, resumable=False)
session.request_resources.pop((stage_id, stale_ephemeral_id), None)
if concurrent_turn:
# Input gate already released after assistant text/silent final,
# so non-draining stages are idle. Drop their session bindings
# only — do not abort engine work. Stages the plugin marks as
# draining keep running under the prior response_id.
prior_response_id = session.active_response_id
draining_stages = self._draining_stage_ids()
for sid, rid in stale_keys:
if sid not in draining_stages:
session.request_resources.pop((sid, rid), None)
elif prior_response_id is not None and not session.is_draining_request(rid):
# Already-draining ids keep their original response_id.
session.bind_draining_request(rid, prior_response_id)
self._ctx.run.concurrent_turn_requests_released = False
if prior_response_id is not None:
session.snapshot_active_response_for_drain()
new_response_id = session.begin_response(turn_id=fence.turn_id)
self._out.emit(self.response_created_payload(new_response_id, epoch=session.epoch))
else:
session.bind_response_turn(fence.turn_id)
elif stale_ids:
# Input gate not released yet (e.g. commit while Stage1 is still
# running): abort the whole prior ephemeral so a later output
# stage is not left orphaned.
try:
await self._ctx.stage_port.cleanup(stale_ids, abort=True)
except Exception:
logger.warning(
"duplex abort of stale ephemeral request failed session=%s ids=%s",
session.session_id,
stale_ids,
exc_info=True,
)
try:
session.begin_lease_operation(fence, lease_operation_id)
operation_started = True
reservation = session.prepare_append(fence)
already_submitted = session.stage_request_submitted(stage_id, request_id)
already_submitted = False if not resumable else session.stage_request_submitted(stage_id, request_id)
request_context = self._ctx.manager.ensure_stage_request(session, stage_id=stage_id, fence=fence)
if request_context is None:
raise RuntimeError("duplex_data_plane_has_no_stage")
if request_context.request_id != request_id:
request_id = request_context.request_id
prompt_payload: dict[str, object] = (
{str(key): value for key, value in payload.items()} if isinstance(payload, Mapping) else {}
)
@@ -253,6 +313,7 @@ class ModelChannel:
context=request_context,
prompt=append_plan.prompt,
already_submitted=already_submitted,
resumable=resumable,
)
submission_result = await self._ctx.stage_port.submit(submission)
try:
@@ -273,6 +334,11 @@ class ModelChannel:
)
raise
session.touch_lease(DuplexLeaseActivity.APPEND)
session.bind_request(request_id)
# Consuming a Stage0 bind closes the concurrent-turn gate even when
# this turn used a fresh ephemeral id (not the reuse branch above).
if stage_id == 0:
self._ctx.run.concurrent_turn_requests_released = False
return {
"ok": True,
"operation": "append",
@@ -289,7 +355,7 @@ class ModelChannel:
"seq": update.seq,
"turn_id": update.turn_id,
"turn_seq": update.turn_seq,
"resumable": session.capabilities.supports_core_resumable_request,
"resumable": resumable,
},
}
],
@@ -386,6 +452,25 @@ class ModelChannel:
raise TypeError("duplex plugin decide_output() must return DuplexOutputDecision or None")
return decision
def project_intermediate_output(self, stage_id: int, output: RequestOutput, context: DuplexOutputContext) -> bool:
"""Project an intermediate stage without short-circuiting the pipeline."""
return self._ctx.plugin.project_intermediate_output(
stage_id=stage_id,
output=output,
context=context,
)
def release_concurrent_turn_requests(
self, stage_id: int, output: RequestOutput, context: DuplexOutputContext
) -> bool:
"""Ask the plugin whether the next commit may start while TTS drains."""
return self._ctx.plugin.release_concurrent_turn_requests(
stage_id=stage_id,
segment_finished=context.segment_finished,
output=output,
context=context,
)
@staticmethod
def stage_metrics_snapshot(stage_id: int, metrics: object, output: object) -> dict[str, dict[str, object]] | None:
if not isinstance(metrics, StageRequestStats):
@@ -464,7 +549,11 @@ class ModelChannel:
if self._out.auto_responds():
active_request_id = session.active_request_id
if active_request_id is not None and active_request_id != item.request_id:
return
if not (
session.capabilities.supports_concurrent_turn_requests
and session.is_draining_request(item.request_id)
):
return
engine_output = self._build_stage_output(item)
drain_result = {"data_plane_outputs": [engine_output]}
close_reason, emitted_response = await self._send_model_output_events(
@@ -474,12 +563,14 @@ class ModelChannel:
await self._close_from_runtime(close_reason)
return
finished = self._data_plane_outputs_finished(drain_result)
if finished and not session.capabilities.supports_core_resumable_request:
if self._ctx.run.stream_request_id == item.request_id:
self._ctx.run.stream_request_id = None
await self._ctx.stage_port.cleanup([item.request_id])
return
if finished and emitted_response and not self._out.auto_responds():
# Intermediate-stage projection finishing means text is done, not the duplex turn.
# Closing the stream here drops later Code2Wav chunks / next-turn bind.
if (
finished
and emitted_response
and not self._out.auto_responds()
and item.stage_id >= item.context.final_stage_id
):
# A finished, emitted response releases the per-request projector
# cursor on its way out and offers the model another
# silence unit.
@@ -579,6 +670,31 @@ class ModelChannel:
self._out.emit(payload)
return True
async def _release_ephemeral_request(self, request_id: object, *, whole_turn: bool = False) -> None:
"""Drop orchestrator state for a finished non-resumable stage request.
Resident Stage0 stays bound. Intermediate text must not call this;
only a completed turn (final stage, silent short-circuit, or a
draining TTS request) does. ``whole_turn`` also drops the other stage
ids of this turn. A request still marked draining is left registered.
"""
if not isinstance(request_id, str) or not request_id:
return
session = self._ctx.session
if session.capabilities.supports_core_resumable_request:
return
if session.is_draining_request(request_id):
await self._ctx.stage_port.cleanup([request_id])
return
release_ids = [request_id]
if whole_turn:
candidate_ids = [rid for _stage_id, rid in session.request_resources]
for candidate in duplex_same_turn_request_ids(request_id, candidate_ids):
if candidate not in release_ids and not session.is_draining_request(candidate):
release_ids.append(candidate)
session.release_resources_for_request_ids(release_ids)
await self._ctx.stage_port.cleanup(release_ids)
async def _on_model_listen(
self,
model_result: dict[str, object],
@@ -643,12 +759,18 @@ class ModelChannel:
self._attach_runtime_metadata(payload, model_result)
self._out.emit(payload)
if model_result.get("abort_data_plane_request") is True and isinstance(data_plane_request_id, str):
await self._abort_request(data_plane_request_id, notify=False)
# stage_port.abort_requests expects a list of ids; a bare str is
# iterated as characters and never matches the prewarmed binding.
await self._abort_request([data_plane_request_id], notify=False)
if response_id is not None:
if not auto_response and self.response_continuations_remaining(response_id):
self._ctx.services.spawn(
self.maybe_continue_response(expected_epoch=expected_epoch), name="duplex-continue"
)
# Continuation closes the response; it does not release the
# finished non-resumable request. Do that here so this early
# return does not leak orchestrator / manager state.
await self._release_ephemeral_request(data_plane_request_id)
return close_reason, emitted_response
if auto_response:
model_state.clear_continuation()
@@ -669,6 +791,7 @@ class ModelChannel:
"playback": session.playback.as_dict(),
}
)
await self._release_ephemeral_request(data_plane_request_id, whole_turn=True)
return close_reason, emitted_response
async def _send_one_model_output_event(
@@ -687,10 +810,13 @@ class ModelChannel:
if isinstance(data_plane_request_id, str) and data_plane.is_terminal(data_plane_request_id):
return close_reason, emitted_response
auto_response = self._out.auto_responds()
draining = session.is_draining_request(
data_plane_request_id if isinstance(data_plane_request_id, str) else None
)
active_request_matches = session.active_request_id == data_plane_request_id or (
auto_response and session.active_request_id is None
)
if isinstance(data_plane_request_id, str) and not active_request_matches:
if isinstance(data_plane_request_id, str) and not active_request_matches and not draining:
return close_reason, emitted_response
if isinstance(model_result.get("error_code"), str):
self._fail_response_from_model_error(model_result)
@@ -726,6 +852,12 @@ class ModelChannel:
self._attach_runtime_metadata(payload, model_result)
self._out.emit(payload)
return close_reason, emitted_response
context_text = model_result.get("model_context_text")
if isinstance(context_text, str) and isinstance(data_plane_request_id, str):
self._ctx.plugin.commit_model_context(
session_id=duplex_session_id_from_request_id(data_plane_request_id),
assistant_text=context_text,
)
if is_listen is True:
return await self._on_model_listen(
model_result,
@@ -753,26 +885,45 @@ class ModelChannel:
name="duplex-continue",
)
return close_reason, emitted_response
if end_of_turn and not has_text and not has_audio and session.active_response_id is None:
request_key = data_plane_request_id if isinstance(data_plane_request_id, str) else None
draining_response_id = (
session.response_id_for_request(request_key) if session.is_draining_request(request_key) else None
)
if (
end_of_turn
and not has_text
and not has_audio
and session.active_response_id is None
and draining_response_id is None
):
emitted_response = self._complete_model_turn_without_output(
model_result,
model_turn_id=model_turn_id,
data_plane_request_id=data_plane_request_id,
)
await self._release_ephemeral_request(data_plane_request_id, whole_turn=True)
return close_reason, emitted_response
if session.active_response_id is None and model_turn_id is not None and model_turn_id < session.turn_id:
# Late audio of a completed model turn must not reserve a second response.
return close_reason, emitted_response
self._end_active_response_before_future_model_turn(model_turn_id=model_turn_id)
if (
session.active_response_id is not None
draining_response_id is None
and session.active_response_id is None
and model_turn_id is not None
and model_turn_id < session.turn_id
):
# Late audio of a completed model turn must not reserve a second response.
# Draining requests are exempt: resolve ownership before this filter.
return close_reason, emitted_response
if draining_response_id is None:
self._end_active_response_before_future_model_turn(model_turn_id=model_turn_id)
if (
draining_response_id is None
and session.active_response_id is not None
and model_turn_id is not None
and not session.active_response_accepts_model_turn(model_turn_id)
):
return close_reason, emitted_response
emitted_response = True
response_created = False
response_id = session.active_response_id
response_id = draining_response_id or session.active_response_id
if response_id is None:
response_id = session.begin_response(turn_id=model_turn_id)
response_created = True
@@ -793,12 +944,16 @@ class ModelChannel:
}
self._attach_runtime_metadata(speak_payload, model_result, stage_metrics=response_stage_metrics)
self._out.emit(speak_payload)
previous_sent_ms = session.playback.sent_ms
text_chars_before_append = len("".join(session.assistant_text_buffer))
if isinstance(text, str):
session.append_assistant_text(text)
target_id = draining_response_id if draining_response_id not in (None, session.active_response_id) else None
previous_sent_ms = session.playback_for_response(target_id).sent_ms
text_chars_before_append = len(session.assistant_transcript(target_id))
if isinstance(text, str) and text:
if target_id is not None:
session.append_draining_assistant_text(target_id, text)
else:
session.append_assistant_text(text)
duration_ms = model_result.get("audio_duration_ms")
text_chars = len("".join(session.assistant_text_buffer))
text_chars = len(session.assistant_transcript(target_id))
mark_duration_ms = None
mark_text_chars: int | None = text_chars
if model_result.get("audio_text_mark") is False:
@@ -806,7 +961,10 @@ class ModelChannel:
if isinstance(duration_ms, int | float):
mark_duration_ms = int(duration_ms)
if model_result.get("audio_duration_is_cumulative") is not True:
mark_duration_ms += session.playback.sent_ms
# Deltas accumulate on the response that owns this chunk.
# session.playback is the active cursor and is 0 after overlap
# opens the next response.
mark_duration_ms += previous_sent_ms
audio_text_marks = model_result.get("audio_text_marks")
audio_text_marks = self._normalize_audio_text_marks(
audio_text_marks if isinstance(audio_text_marks, list) else None,
@@ -826,6 +984,7 @@ class ModelChannel:
audio_text_marks=audio_text_marks,
text_requires_complete_audio=model_result.get("text_requires_complete_audio") is True,
audio_complete=model_result.get("audio_complete") is True,
response_id=target_id,
)
payload = {
"type": "response.output_audio.delta",
@@ -850,7 +1009,7 @@ class ModelChannel:
payload["audio_text_marks"] = [
{"text_chars": max(0, int(mark_text_chars)), "audio_end_ms": max(0, int(mark_duration_ms))}
]
payload["playback"] = session.playback.as_dict()
payload["playback"] = session.playback_for_response(target_id).as_dict()
sample_rate_hz = model_result.get("sample_rate_hz") or model_result.get("audio_sample_rate_hz")
if isinstance(sample_rate_hz, int | float) and int(sample_rate_hz) > 0:
payload["sample_rate_hz"] = int(sample_rate_hz)
@@ -867,6 +1026,34 @@ class ModelChannel:
)
if end_of_turn:
data_plane_request_id = model_result.get("data_plane_request_id")
# Prior TTS finished under its own draining response_id while a
# newer turn already owns active_response_id — close that response.
if (
session.capabilities.supports_concurrent_turn_requests
and isinstance(data_plane_request_id, str)
and session.is_draining_request(data_plane_request_id)
):
drained_response_id = session.pop_draining_request(data_plane_request_id)
data_plane.close_stream(data_plane_request_id)
data_plane.mark_terminal(data_plane_request_id)
for stage_id in self._draining_stage_ids():
session.request_resources.pop((stage_id, data_plane_request_id), None)
await self._release_ephemeral_request(data_plane_request_id, whole_turn=True)
if drained_response_id is not None:
playback = session.playback_for_response(drained_response_id).as_dict()
session.release_finished_drain_response(drained_response_id)
self._out.emit(
{
"type": "response.done",
"session_id": session.session_id,
"response_id": drained_response_id,
"epoch": session.epoch,
"committed": False,
"status": "completed",
"playback": playback,
}
)
return close_reason, emitted_response
if isinstance(data_plane_request_id, str) and not auto_response:
data_plane.close_stream(data_plane_request_id)
if isinstance(data_plane_request_id, str):
@@ -891,6 +1078,7 @@ class ModelChannel:
"playback": session.playback.as_dict(),
}
)
await self._release_ephemeral_request(data_plane_request_id, whole_turn=True)
return close_reason, emitted_response
def _end_active_response_before_future_model_turn(self, *, model_turn_id: int | None) -> None:
@@ -1073,16 +1261,40 @@ class ModelChannel:
session = self._ctx.session
model_state = self._ctx.model_state
response_id = session.active_response_id
auto_response = self._out.auto_responds()
if session.state == DuplexSessionState.CLOSED or self._ctx.run.closing:
model_state.clear_continuation()
return
if expected_epoch is not None and session.epoch != expected_epoch:
return
# Non-resumable stage0 cannot submit_update after the request finishes;
# clear continuation and close the response (silent / listen final).
# A continue for a response that still has draining TTS must not emit
# response.done here; that finish owns the done event.
if not session.capabilities.supports_core_resumable_request:
model_state.clear_continuation()
if response_id is not None and not session.response_has_draining_request(response_id):
should_commit = self.should_commit_response_to_history(session, response_id)
committed_message = session.end_response(commit_text=should_commit, preserve_request=auto_response)
if should_commit and committed_message is not None:
session.register_history_item(f"item_{response_id}", committed_message)
self._out.emit(
{
"type": "response.done",
"session_id": session.session_id,
"response_id": response_id,
"epoch": session.epoch,
"committed": committed_message is not None if should_commit else False,
"playback": session.playback.as_dict(),
}
)
return
request_id = session.active_request_id
if request_id is None:
model_state.clear_continuation()
return
if expected_epoch is not None and session.epoch != expected_epoch:
return
auto_response = self._out.auto_responds()
response_owned = response_id is not None
if response_owned:
owner_id = f"response:{response_id}"
vllm_omni/engine/duplex/session/overlap_policy.py
+15
View File
@@ -111,6 +111,21 @@ def decide(
policy = session.config.overlap_policy
if not is_speech:
frames = payload.get("video_frames")
has_vision = isinstance(frames, list) and any(isinstance(frame, str) and frame for frame in frames)
if has_vision and session.capabilities.allows_video_without_audio():
# Same exception as the idle turn-mode path: video without speech is
# is_speech=False + a frame. Dropping it while TTS is still playing
# leaves Stage0 unsubmitted (empty llm).
return {
"action": "listen",
"reason": "vision_follow",
"duration_ms": duration_ms,
"overlap_speech_ms": session.overlap_speech_ms,
"buffer_audio": True,
"defer_runtime_append": False,
"force_listen": False,
}
if session.overlap_speech_ms <= 0:
session.reset_overlap_speech()
return {
vllm_omni/engine/duplex/session/runner.py
+261 -96
View File
@@ -247,21 +247,33 @@ class DuplexSessionRunner:
) -> bool:
"""Accept one stage output (orchestrator loop); return True when it must not be forwarded."""
decision: DuplexOutputDecision | None = None
project = False
if stage_id < context.final_stage_id:
decision = self.model.decide_output(stage_id, output, context)
# Optional mid-pipeline projection: client sees this stage; TTS still runs.
if decision is None:
project = self.model.project_intermediate_output(stage_id, output, context)
if self.session.capabilities.supports_concurrent_turn_requests and self.model.release_concurrent_turn_requests(
stage_id, output, context
):
self.run.concurrent_turn_requests_released = True
consume = decision is not None or stage_id >= context.final_stage_id
project_intermediate = self.plugin.projects_intermediate_outputs and stage_id == 0
if not consume and not project_intermediate:
if not consume and not project and not project_intermediate:
# Stage0 text without a direct decision feeds the TTS stage as before.
# Its metrics still have to reach the client: before sessions moved
# into the engine the orchestrator published them as a standalone
# ``StageMetricsMessage``, a path session-owned requests no longer
# take. Hand them to the session instead of dropping them, on the
# mailbox so they stay ordered with this session's other work.
snapshot = self.model.stage_metrics_snapshot(stage_id, metrics, output)
if snapshot is not None and not self.run.closing and self.session.state != DuplexSessionState.CLOSED:
self._mailbox.put_nowait(_Internal("stage_metrics", {"stage_metrics": snapshot}))
self._stash_stage_metrics(stage_id, metrics, output)
return False
if project and not consume:
# Stage1 thinker text is projected to the client, but that event
# does not carry engine metrics. Stash them on the same mailbox as
# a pass-through stage so the spoken-turn audio events include
# Stage1 TTFT/TPOT.
self._stash_stage_metrics(stage_id, metrics, output)
if self.run.closing or self.session.state == DuplexSessionState.CLOSED:
return True
self._mailbox.put_nowait(
@@ -274,10 +286,21 @@ class DuplexSessionRunner:
decision=decision,
)
)
# Projection-only must still forward to the next stage (return False).
return consume
def on_stage_failure(self, stage_id: int, exc: BaseException) -> None:
"""A stage rejected this session's request: fail the active response now.
def _stash_stage_metrics(self, stage_id: int, metrics: StageRequestStats | None, output: object) -> None:
snapshot = self.model.stage_metrics_snapshot(stage_id, metrics, output)
if snapshot is None or self.run.closing or self.session.state == DuplexSessionState.CLOSED:
return
self._mailbox.put_nowait(_Internal("stage_metrics", {"stage_metrics": snapshot}))
def on_stage_failure(self, stage_id: int, exc: BaseException, *, request_id: str | None = None) -> None:
"""A stage rejected this session's request: fail the owning response.
Under concurrent turn requests the failing request may belong to a draining
older response; resolve via ``response_id_for_request`` before falling
back to ``active_response_id``.
Runs synchronously on the loop (no mailbox hop): the orchestrator
expires the session right after this call, so a queued item could be
@@ -292,21 +315,32 @@ class DuplexSessionRunner:
"runtime_data_plane_stream_failed",
f"Stage-{stage_id} input processor failed: {type(exc).__name__}: {exc}",
)
response_id = session.active_response_id
if response_id is not None:
draining_response_id = (
session.response_id_for_request(request_id)
if isinstance(request_id, str) and session.is_draining_request(request_id)
else None
)
response_id = draining_response_id or session.active_response_id
if response_id is None:
return
if draining_response_id is not None:
session.clear_draining_for_response(draining_response_id)
if isinstance(request_id, str):
session.request_resources.pop((stage_id, request_id), None)
elif response_id == session.active_response_id:
session.end_response(commit_text=False)
self.emit(
{
"type": "response.done",
"session_id": session.session_id,
"response_id": response_id,
"epoch": session.epoch,
"committed": False,
"status": "failed",
"status_details": {"type": "failed", "reason": "runtime_data_plane_stream_failed"},
"playback": session.playback.as_dict(),
}
)
self.emit(
{
"type": "response.done",
"session_id": session.session_id,
"response_id": response_id,
"epoch": session.epoch,
"committed": False,
"status": "failed",
"status_details": {"type": "failed", "reason": "runtime_data_plane_stream_failed"},
"playback": session.playback.as_dict(),
}
)
@property
def closed_emitted(self) -> bool:
@@ -458,7 +492,8 @@ class DuplexSessionRunner:
if isinstance(item, Commit):
session.release_pending_turn()
elif isinstance(item, AppendAudio):
session.release_input_bytes(len(item.audio))
admission = len(item.audio) + sum(len(frame) for frame in item.video_frames)
session.release_input_bytes(admission)
return
await self._on_command(item)
@@ -514,9 +549,11 @@ class DuplexSessionRunner:
session = self.session
projector = self._require_projector()
if isinstance(command, AppendAudio):
# The manager reserved the wire size at admission; the handler
# re-reserves the decoded size around its PCM reservation.
session.release_input_bytes(len(command.audio))
# The manager reserved the wire size (audio + video) at admission.
# Release that full amount as the command leaves the mailbox; the
# handler re-reserves whatever the input buffer actually retains.
admission = len(command.audio) + sum(len(frame) for frame in command.video_frames)
session.release_input_bytes(admission)
await self._on_append_audio(command.payload())
elif isinstance(command, AppendText):
session.mark_user_input_activity()
@@ -761,9 +798,19 @@ class DuplexSessionRunner:
model_state = self.model_state
session.mark_user_input_activity()
audio = event.get("audio") or event.get("data")
if not isinstance(audio, str):
self._emit_error("bad_event", "input_audio_buffer.append requires audio")
video_frames_raw = event.get("video_frames")
video_frames = (
[frame for frame in video_frames_raw if isinstance(frame, str) and frame]
if isinstance(video_frames_raw, list)
else []
)
has_audio = isinstance(audio, str) and bool(audio)
has_video = bool(video_frames)
modality_error = session.capabilities.validate_append_modalities(has_audio=has_audio, has_video=has_video)
if modality_error is not None:
self._emit_error("invalid_input_modality", modality_error)
return
video_only = has_video and not has_audio
if not session.capabilities.supports_barge_in and overlap_policy.event_requests_barge_in(event):
self._emit_events([helpers.barge_in_unsupported_error()])
event = dict(event)
@@ -775,54 +822,77 @@ class DuplexSessionRunner:
fmt = event.get("format") if isinstance(event.get("format"), str) else "pcm16"
sr_raw = event.get("sample_rate_hz") or event.get("sample_rate")
sample_rate_hz = sr_raw if isinstance(sr_raw, int | float) else 16000
try:
converted: tuple[object, object, int | float | None] = await self.offload(
convert_input_audio_with_rate,
audio,
fmt,
sample_rate_hz=sample_rate_hz,
)
except ValueError as exc:
self._emit_error("bad_event", str(exc))
return
audio, fmt, converted_rate = converted
if converted_rate is not None:
sample_rate_hz = converted_rate
if isinstance(fmt, str) and fmt.lower() in {"pcm16", "pcm_s16le", "s16le"}:
self._emit_error("bad_audio", "input_audio_buffer.append pcm16 audio could not be decoded")
return
event["audio"] = audio
event["format"] = fmt
event["sample_rate_hz"] = sample_rate_hz
client_force_listen = bool(event.get("force_listen", False))
vad_result = await self.control.run_turn_detection(event)
if (
vad_result is not None
and not session.capabilities.supports_core_resumable_request
and not client_force_listen
):
# VAD's force_listen hint controls native model decoding. Committed
# turn models already buffer speech; it must not suppress barge-in.
event.pop("force_listen", None)
if video_only:
# No PCM to decode; frames alone are the turn content.
fmt = "pcm_f32le"
event = dict(event)
event.pop("audio", None)
event.pop("data", None)
event["format"] = fmt
event["sample_rate_hz"] = sample_rate_hz
event["is_speech"] = False
event["video_frames"] = video_frames
vad_result = None
else:
try:
converted: tuple[object, object, int | float | None] = await self.offload(
convert_input_audio_with_rate,
audio,
fmt,
sample_rate_hz=sample_rate_hz,
)
except ValueError as exc:
self._emit_error("bad_event", str(exc))
return
audio, fmt, converted_rate = converted
if converted_rate is not None:
sample_rate_hz = converted_rate
if isinstance(fmt, str) and fmt.lower() in {"pcm16", "pcm_s16le", "s16le"}:
self._emit_error("bad_audio", "input_audio_buffer.append pcm16 audio could not be decoded")
return
event["audio"] = audio
event["format"] = fmt
event["sample_rate_hz"] = sample_rate_hz
vad_result = await self.control.run_turn_detection(event)
if (
vad_result is not None
and not session.capabilities.supports_core_resumable_request
and not client_force_listen
):
# VAD's force_listen hint controls native model decoding. Committed
# turn models already buffer speech; it must not suppress barge-in.
event.pop("force_listen", None)
projector = self._require_projector()
self._emit_events(note_input_append(projector, event, vad_result=vad_result))
self._emit_events(
note_input_append(
projector,
event,
vad_result=vad_result,
allows_video_without_audio=session.capabilities.allows_video_without_audio(),
)
)
if self.run.closing or session.state != DuplexSessionState.OPEN:
return
force_listen = bool(event.get("force_listen", False))
payload: dict[str, object] = {
"type": "audio",
"audio": audio,
"format": fmt,
"sample_rate_hz": sample_rate_hz,
"force_listen": force_listen,
}
video_frames = event.get("video_frames")
if isinstance(video_frames, list):
frames = [frame for frame in video_frames if isinstance(frame, str) and frame]
if frames:
payload["video_frames"] = frames
payload["is_speech"] = overlap_policy.input_looks_like_speech(self.session, event, payload)
if video_only:
payload["format"] = fmt
payload["sample_rate_hz"] = sample_rate_hz
payload["is_speech"] = False
else:
payload["audio"] = audio
payload["format"] = fmt
payload["sample_rate_hz"] = sample_rate_hz
if video_frames:
payload["video_frames"] = video_frames
payload["is_speech"] = (
False if video_only else overlap_policy.input_looks_like_speech(self.session, event, payload)
)
auto_responds = self._session_auto_responds()
defer_append = False
buffer_overlap_audio = True
@@ -862,16 +932,21 @@ class DuplexSessionRunner:
defer_append = False
elif not auto_responds and not overlap_policy.input_looks_like_speech(self.session, event, payload):
# Turn-mode only: skip silent chunks so they don't open a response.
self.emit(
{
"type": "response.listen",
"session_id": session.session_id,
"epoch": session.epoch,
"reason": "silence_or_noise",
}
)
self._maybe_schedule_vad_commit(vad_result)
return
# Vision-carrying silent appends must still buffer when the model
# allows video without required audio.
frames = payload.get("video_frames")
has_vision = isinstance(frames, list) and any(isinstance(frame, str) and frame for frame in frames)
if not (has_vision and session.capabilities.allows_video_without_audio()):
self.emit(
{
"type": "response.listen",
"session_id": session.session_id,
"epoch": session.epoch,
"reason": "silence_or_noise",
}
)
self._maybe_schedule_vad_commit(vad_result)
return
if overlap_policy.should_force_listen_for_auto_response_overlap(event, payload, auto_responds=auto_responds):
payload["force_listen"] = True
if not buffer_overlap_audio:
@@ -883,6 +958,7 @@ class DuplexSessionRunner:
self.session, event, payload
)
raw_audio_bytes = helpers.audio_payload_size_bytes(payload)
pending_before = model_state.audio_buffer.pending_byte_count
try:
if not session.reserve_input_bytes(
raw_audio_bytes,
@@ -904,6 +980,19 @@ class DuplexSessionRunner:
self._emit_error("bad_event", str(exc))
return
if pcm_reservation is None:
# Commit-only buffers (AURA) accumulate in place and return None.
# Undo the speculative audio reserve and re-apply the exact pending
# delta so retained video frames are counted (and later released).
session.release_input_bytes(raw_audio_bytes)
pending_delta = model_state.audio_buffer.pending_byte_count - pending_before
if pending_delta > 0 and not session.reserve_input_bytes(
pending_delta,
limit=int(self.manager.runtime_config.max_pending_input_bytes_per_session),
):
self._emit_error("input_backpressure", "Duplex session pending input exceeds server limit")
return
if pending_delta < 0:
session.release_input_bytes(-pending_delta)
self._maybe_schedule_vad_commit(vad_result)
return
if pcm_reservation.byte_count == 0:
@@ -1338,13 +1427,18 @@ class DuplexSessionRunner:
) -> bool:
session = self.session
has_running_task = active_task is not None and not active_task.done()
if not has_running_task and session.active_request_id is None and session.active_response_id is None:
old_request_id = session.active_request_id
draining_ids = [
request_id for request_id in session.draining_request_ids() if request_id and request_id != old_request_id
]
if not has_running_task and old_request_id is None and session.active_response_id is None and not draining_ids:
return False
old_epoch = session.epoch
old_request_id = session.active_request_id
old_response_id = session.active_response_id
committed_ms = session.playback.committed_ms
# Barge-in / cancel aborts prior TTS; clear the concurrent-turn release.
self.run.concurrent_turn_requests_released = False
committed_message = session.end_response(
commit_text=self.model.should_commit_response_to_history(session, old_response_id),
playback_commit_policy=DuplexPlaybackCommitPolicy.ACK_ONLY.value,
@@ -1355,15 +1449,39 @@ class DuplexSessionRunner:
session.register_history_item(item_id, committed_message)
elif committed_ms > 0 and not session.playback_ack_is_too_late(old_response_id, item_id):
session.truncate_history_item(item_id, audio_end_ms=committed_ms)
draining_cancels: list[tuple[str, int, dict[str, int]]] = []
seen_responses: set[str] = set()
if isinstance(old_response_id, str) and old_response_id:
seen_responses.add(old_response_id)
for request_id in draining_ids:
response_id = session.response_id_for_request(request_id)
if not isinstance(response_id, str) or not response_id or response_id in seen_responses:
continue
seen_responses.add(response_id)
playback = session.playback_for_response(response_id).as_dict()
draining_cancels.append((response_id, int(playback.get("committed_ms") or 0), playback))
# The epoch bump is the atomic part: from here on every model output
# and append of the old epoch is dropped by the stale-epoch filter in
# ``emit`` / the append tail, whatever the awaits below interleave with.
# ``cancel_fence`` only drops the fence being cancelled, and that is
# what tells the orchestrator to cleanup(abort=True). Releasing those
# ids here first leaves the stage request in ``request_states``.
# Draining output stages sit on an older turn fence, so drop only those.
abort_ids = [request_id for request_id in (old_request_id, *draining_ids) if request_id]
cancelled_ids = set(session.resource_request_ids(session.fence))
older_abort_ids = [request_id for request_id in abort_ids if request_id not in cancelled_ids]
if older_abort_ids:
session.release_resources_for_request_ids(older_abort_ids)
new_epoch, old_playback = helpers.advance_barge_in_epoch(session)
if old_request_id is not None:
# Release projector/parser cursors so cancelled epochs do not
# accumulate until the whole session closes.
self.plugin.data_plane.close_stream(old_request_id)
await self._abort_request_background(old_request_id, notify=notify)
for request_id in draining_ids:
self.plugin.data_plane.close_stream(request_id)
session.clear_draining_requests()
if abort_ids:
await self._abort_request_background(abort_ids, notify=notify)
if has_running_task and active_task is not None:
active_task.cancel()
try:
@@ -1383,15 +1501,28 @@ class DuplexSessionRunner:
"playback": old_playback,
}
)
for response_id, drain_committed_ms, drain_playback in draining_cancels:
self.emit(
{
"type": "audio.cancelled",
"session_id": session.session_id,
"response_id": response_id,
"reason": reason,
"cancelled_epoch": old_epoch,
"epoch": new_epoch,
"committed_ms": drain_committed_ms,
"playback": drain_playback,
}
)
return True
async def _abort_request_background(self, request_id: str, *, notify: bool) -> None:
async def _abort_request_background(self, request_ids: list[str], *, notify: bool) -> None:
try:
await self.stage_port.abort_requests([request_id])
await self.stage_port.abort_requests(request_ids)
except asyncio.CancelledError:
raise
except Exception as exc:
logger.exception("Failed to abort duplex request %s: %s", request_id, exc)
logger.exception("Failed to abort duplex request %s: %s", request_ids, exc)
if notify and self.session.state != DuplexSessionState.CLOSED:
self.model.send_runtime_error("runtime_abort_failed", exc)
@@ -1465,10 +1596,22 @@ class DuplexSessionRunner:
if commit_reservation is not None:
commit_reservation.commit()
if flushed is None:
# Non-speech residual: prepare_commit refused a Stage0 unit. Clear
# the PCM and its byte reservation so silence does not leak into
# the next turn (MiniCPM-o / auto-response silent commit).
if event_type in {"input.commit", "input_audio_buffer.commit"} and not model_state.speech_since_commit:
pending_bytes = model_state.audio_buffer.pending_byte_count
model_state.audio_buffer.clear()
if pending_bytes:
session.release_input_bytes(pending_bytes)
model_state.input_since_commit = False
return False
if overlap_policy.should_force_listen_for_short_commit(self.session, event, flushed):
flushed = dict(flushed)
flushed["force_listen"] = True
if isinstance(realtime_item_id, str) and realtime_item_id:
flushed = dict(flushed)
flushed["realtime_item_id"] = realtime_item_id
model_state.input_since_commit = False
committed = helpers.commit_audio_input(
session,
@@ -1637,6 +1780,9 @@ class DuplexSessionRunner:
)
else:
retained_payload = deferred_payload
if isinstance(realtime_item_id, str) and realtime_item_id:
retained_payload = dict(retained_payload)
retained_payload["realtime_item_id"] = realtime_item_id
model_state.retain_committed_audio(
retained_payload,
operation_id=commit_reservation.operation_id,
@@ -1703,6 +1849,8 @@ class DuplexSessionRunner:
)
)
if final_payload is not None:
if isinstance(realtime_item_id, str) and realtime_item_id:
final_payload = {**final_payload, "realtime_item_id": realtime_item_id}
await self._start_append(
{**final_payload, "duplex_turn_id": data_plane_turn_id},
final=True,
@@ -1760,14 +1908,22 @@ class DuplexSessionRunner:
event_id=event.get("realtime_event_id"),
)
return
if event_type == "input_audio_buffer.commit" and event.get("is_speech") is False:
self._commit_silent_input()
return
should_create_response = (
event_type == "response.create"
or bool(event.get("response_create", event_type == "input.commit"))
or (event_type == "input_audio_buffer.commit" and self._session_auto_responds())
)
# Pure silence with nothing buffered: drop and keep listening.
# Pending silent+video (or an explicit create_response) must flush.
if event_type == "input_audio_buffer.commit" and event.get("is_speech") is False:
has_pending_turn = (
model_state.input_since_commit
or model_state.audio_buffer.has_pending()
or model_state.committed_audio_payload is not None
)
if not has_pending_turn:
self._commit_silent_input()
return
precreate_response_requested = event_type == "response.create" or bool(
event.get("response_create", event_type == "input.commit")
)
@@ -1808,32 +1964,41 @@ class DuplexSessionRunner:
)
)
if commit_action is CommitAction.DEFER_ACTIVE_RESPONSE:
if session.overlap_speech_ms <= session.config.overlap_short_ack_ms:
self._discard_short_overlap_ack()
return
if self._defer_commit_behind_active_response(
event,
realtime_item_id=realtime_item_id,
should_create_response=should_create_response,
precreate_response_requested=precreate_response_requested,
if not helpers.next_commit_allowed(
self.session,
self.tasks,
concurrent_turn_requests_released=self.run.concurrent_turn_requests_released,
):
return
if session.overlap_speech_ms <= session.config.overlap_short_ack_ms:
self._discard_short_overlap_ack()
return
if self._defer_commit_behind_active_response(
event,
realtime_item_id=realtime_item_id,
should_create_response=should_create_response,
precreate_response_requested=precreate_response_requested,
):
return
if commit_action is CommitAction.START_AUTO_RESPONSE:
await self._commit_and_start_auto_response(event, realtime_item_id=realtime_item_id)
return
if event_type == "response.create":
await self._start_response_from_committed_audio()
return
if not helpers.response_in_progress(self.session, self.tasks) and await self._flush_and_submit_committed_turn(
if helpers.next_commit_allowed(
self.session,
self.tasks,
concurrent_turn_requests_released=self.run.concurrent_turn_requests_released,
) and await self._flush_and_submit_committed_turn(
event,
event_type=event_type,
realtime_item_id=realtime_item_id,
should_create_response=should_create_response,
):
return
# Nothing flushed (or a response is still in progress): acknowledge the
# commit without starting a new response.
# Nothing flushed (or a response is still in progress without
# concurrent-turn release): acknowledge without starting a new response.
had_uncommitted_audio = (
model_state.input_since_commit
or model_state.audio_buffer.has_pending()
vllm_omni/engine/duplex_orchestrator.py
+88 -10
View File
@@ -154,13 +154,28 @@ class DuplexOrchestrator(Orchestrator, DuplexStagePort):
# Session gone: nothing may forward or reach a client.
return True
segment = req_state.streaming.segment(stage_id)
finished = bool(getattr(output, "finished", False)) or (req_state.streaming.enabled and segment.finished)
transcript = self.plugin.user_transcript(
stage_id=stage_id,
output=output,
prompt=getattr(req_state, "prompt", None),
finished=finished,
)
if isinstance(transcript, str) and transcript:
payload: dict[str, object] = {"type": "input.transcribed", "transcript": transcript}
prompt = getattr(req_state, "prompt", None)
info = prompt.get("additional_information") if isinstance(prompt, dict) else None
item_id = info.get("realtime_item_id") if isinstance(info, dict) else None
if isinstance(item_id, str) and item_id:
payload["realtime_item_id"] = item_id
runner.emit(payload)
context = DuplexOutputContext(
identity=DuplexRequestIdentity(
session_id=req_state.session_id,
fence=req_state.stage_fences.get(stage_id, req_state.fence),
),
final_stage_id=req_state.final_stage_id,
segment_finished=req_state.streaming.enabled and segment.finished,
segment_finished=finished,
segment_token_ids=tuple(segment.token_ids),
segment_output_metadata=segment.output_metadata,
)
@@ -181,13 +196,18 @@ class DuplexOrchestrator(Orchestrator, DuplexStagePort):
) -> bool:
if not req_state.session_owned:
return False
# Resumable resident Stage0: stage failure closes the session.
# Ephemeral turn-commit: free this turn's stages without tearing down WS.
close_session = self.plugin.capabilities(
max_sessions=self.duplex_session_config.max_sessions
).supports_core_resumable_request
runner = self.session_manager.runner_for_request_id(req_id)
if runner is not None:
runner.on_stage_failure(next_stage_id, exc)
runner.on_stage_failure(next_stage_id, exc, request_id=req_id)
await self._cleanup_request_ids(
[req_id, *self._cfg_tracker.cleanup_parent(req_id)],
abort=True,
release_owners=True,
release_owners=close_session,
)
return True
@@ -248,6 +268,19 @@ class DuplexOrchestrator(Orchestrator, DuplexStagePort):
defaults.append(client.default_sampling_params)
return tuple(defaults)
def _stage_receives_async_chunks(self, stage_id: int) -> bool:
"""Whether a stage's connector supplies its runtime inputs.
Stages with a custom orchestrator input processor must be fed via
process_engine_inputs, not zero-prewarm + connector chunks. Codec edges
without a custom processor still use async chunk transport.
"""
pool = self.stage_pools[stage_id]
client = getattr(pool, "stage_client", None)
if client is not None and getattr(client, "custom_process_input_func", None) is not None:
return False
return super()._stage_receives_async_chunks(stage_id)
@staticmethod
def _sync_bridge_state(
request_state: OrchestratorRequestState,
@@ -292,9 +325,6 @@ class DuplexOrchestrator(Orchestrator, DuplexStagePort):
fence=context.fence,
config_generation=context.config_generation,
)
request_state.streaming.enabled = self.plugin.capabilities(
max_sessions=self.duplex_session_config.max_sessions
).supports_core_resumable_request
self.request_states[context.request_id] = request_state
elif isinstance(request_state, DuplexOrchestratorRequestState):
if request_state.config_generation != context.config_generation:
@@ -326,9 +356,11 @@ class DuplexOrchestrator(Orchestrator, DuplexStagePort):
request_state = self.request_states.get(context.request_id)
if not isinstance(request_state, DuplexOrchestratorRequestState):
raise RuntimeError(f"duplex request was not preregistered: {context.request_id}")
if self.plugin.capabilities(
max_sessions=self.duplex_session_config.max_sessions
).supports_core_resumable_request:
request_state.streaming.enabled = submission.resumable
# Keep raw Stage0 prompt (additional_information / multi_modal_data) for
# stage input processors via process_engine_inputs.
request_state.prompt = dict(submission.prompt)
if submission.resumable:
request = build_engine_core_request_from_tokens(
request_id=context.request_id,
prompt=dict(submission.prompt),
@@ -349,10 +381,14 @@ class DuplexOrchestrator(Orchestrator, DuplexStagePort):
)
if self.request_states.get(context.request_id) is not request_state:
raise RuntimeError("duplex request cancelled during input preprocessing")
request_state.prompt = dict(submission.prompt)
request.external_req_id = request.request_id
mm_features = getattr(request, "mm_features", None)
if mm_features is not None:
request_state.mm_features = mm_features
pool = self.stage_pools[context.stage_id]
if submission.already_submitted:
if not submission.resumable:
raise RuntimeError(f"ephemeral duplex request cannot submit_update: {context.request_id}")
replica_id = await pool.submit_update(context.request_id, request_state, request)
else:
replica_id = await pool.submit_initial(context.request_id, request_state, request, prompt_text=None)
@@ -378,6 +414,48 @@ class DuplexOrchestrator(Orchestrator, DuplexStagePort):
replica_id=replica_id,
)
async def _route_output(
self,
stage_id: int,
replica_id: int,
output: Any,
req_state: OrchestratorRequestState,
stage_metrics: Any,
) -> None:
plan = self.plugin.plan_partial_stage_output(self, stage_id, replica_id, output, req_state)
if plan is not None:
# A text-bearing final is not itself the end sentinel. Submit the
# sentence resumable first; the follow-up, if any, closes the stream.
await self._forward_to_next_stage(
req_state.request_id,
stage_id,
plan.output,
req_state,
src_replica_id=replica_id,
is_streaming_session=True,
is_final_update=plan.is_final_update and not plan.queue_close_after,
)
followup = self.plugin.partial_stage_followup(plan, req_state)
if followup is not None:
await self._forward_to_next_stage(
req_state.request_id,
stage_id,
followup.output,
req_state,
src_replica_id=replica_id,
is_streaming_session=True,
is_final_update=followup.is_final_update,
)
# Sentence TTS already handed this Stage1 result to Talker. The
# legacy path below would run aura2tts on the original full text
# again when the next stage is not connector-fed (AURA Talker is
# a sender: Stage1→2 is orchestrator-fed).
req_state.skip_legacy_stage_forward = True
try:
await super()._route_output(stage_id, replica_id, output, req_state, stage_metrics)
finally:
req_state.skip_legacy_stage_forward = False
async def cleanup(self, request_ids: list[str], *, abort: bool = False) -> None:
await self._cleanup_request_ids(request_ids, abort=abort)
vllm_omni/engine/orchestrator.py
+4
View File
@@ -212,6 +212,9 @@ class OrchestratorRequestState:
# its own ``finished`` flag: no synthetic terminal, no client emission of
# stage-0 segment ends, no terminal re-forward, no cleanup on finish.
session_owned: bool = False
# Duplex sentence TTS already submitted this output. Do not also run the
# legacy full-text handoff for the same stage result.
skip_legacy_stage_forward: bool = False
running_counter_registered: bool = False
request_artifact_dirs: set[str] = field(default_factory=set)
native_kv_transfer_id: str | None = None
@@ -1722,6 +1725,7 @@ class OrchestratorBase:
if (
(finished or segment_finished)
and stage_id < req_state.final_stage_id
and not req_state.skip_legacy_stage_forward
and (not self.async_chunk or not self._stage_receives_async_chunks(stage_id + 1))
and (not self._next_stage_already_submitted(stage_id, req_state) or req_state.streaming.enabled)
):
vllm_omni/entrypoints/duplex/warmup.py
+161 -41
View File
@@ -5,12 +5,20 @@
Moved out of ``api_server.py`` under P0.2 of #5227. The
``omni_run_server_worker`` scheduling and ``/v1/realtime`` hold-clients
preamble stay in ``api_server.py`` until P0.3.
Video-required models default to one short audio-plus-frame turn so the four
pipeline stages compile real shapes before a client is accepted. That is
separate from the per-stage JIT kernel registry, which can finish in ~0.01 s
without compiling anything.
"""
import asyncio
import json
import math
import struct
import time
import uuid
from io import BytesIO
import httpx
import pybase64 as base64
@@ -18,16 +26,81 @@ from vllm.logger import init_logger
logger = init_logger(__name__)
# First-request compiles (Triton mRoPE, vision pos-embed, Talker torch.compile,
# Code2Wav) landed ~30–40 s after the socket was already "ready" on a cold
# process, then the short reply still has to be spoken. Hold clients past that.
DUPLEX_WARMUP_CLIENT_WAIT_S = 180
DUPLEX_WARMUP_AUDIO_WAIT_S = 150
_WARMUP_SAMPLE_RATE_HZ = 16000
_WARMUP_AUDIO_SECONDS = 0.4
# 640x480 is divisible by Qwen3-VL's patch*merge (32) and above the
# processor's shortest_edge, so the vision tower sees a real camera-sized grid
# rather than an upscaled thumbnail.
_WARMUP_FRAME_WIDTH = 640
_WARMUP_FRAME_HEIGHT = 480
_WARMUP_INSTRUCTIONS = "启动暖机。请只用这四个字回答，不要输出 silent：暖机完成。"
def lookup_duplex_plugin(engine_client: object) -> object | None:
engine = getattr(engine_client, "engine", None)
plugin = getattr(engine, "plugin", None) if engine is not None else None
if plugin is None:
plugin = getattr(engine_client, "plugin", None)
return plugin
def startup_warmup_kind(plugin: object | None, warmup_frames: int) -> str | None:
"""Which throwaway realtime session to run before admitting clients.
``video_turn`` is one short audio chunk plus one image. It is the default
for models that require video, including when ``warmup_frames`` is 0.
``silent_frames`` is the older audio-only path and stays opt-in.
A negative ``warmup_frames`` disables both. ``None`` means do not hold clients.
"""
if warmup_frames < 0:
return None
required: object = frozenset()
if plugin is not None and hasattr(plugin, "capabilities"):
caps = plugin.capabilities(max_sessions=1)
required = getattr(caps, "required_input_modalities", frozenset())
if isinstance(required, frozenset | set) and "video" in required:
return "video_turn"
if warmup_frames > 0:
return "silent_frames"
return None
def _short_pcm_f32le_b64() -> str:
"""Non-silent PCM so Stage0 ASR actually runs.
All-zero audio is either rejected or replaced before the ASR forward, which
would leave the mRoPE kernel uncompiled.
"""
n = int(_WARMUP_SAMPLE_RATE_HZ * _WARMUP_AUDIO_SECONDS)
pcm = bytearray()
for i in range(n):
sample = 0.2 * math.sin(2.0 * math.pi * 220.0 * i / _WARMUP_SAMPLE_RATE_HZ)
pcm += struct.pack("<f", sample)
return base64.b64encode(bytes(pcm)).decode("ascii")
def _warmup_jpeg_b64() -> str:
from PIL import Image
image = Image.new("RGB", (_WARMUP_FRAME_WIDTH, _WARMUP_FRAME_HEIGHT), (255, 255, 255))
image.paste((0, 0, 0), (_WARMUP_FRAME_WIDTH // 2, 0, _WARMUP_FRAME_WIDTH, _WARMUP_FRAME_HEIGHT))
buf = BytesIO()
image.save(buf, format="JPEG", quality=70)
return base64.b64encode(buf.getvalue()).decode("ascii")
async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
"""Run silent frames through a throwaway realtime session at startup.
"""Run one throwaway realtime session before real clients are admitted.
Self-connects over the server's own websocket endpoint so the warmup
exercises the exact production path (serving adapter, duplex control
plane, every pipeline stage). One-time costs — kernel JIT compilation,
first prefill/decode paths, codec caches — are paid here instead of by
the first real client; ``/v1/realtime`` holds other connections until
the warmup finishes (see ``duplex_warmup_done``).
Video-required models send one short non-silent audio chunk and one image
so each stage compiles the shape a real turn uses. Audio-primary models
keep the opt-in silent-frame path. ``/v1/realtime`` holds other connections
until this finishes (see ``duplex_warmup_done``).
"""
warmup_done = getattr(app.state, "duplex_warmup_done", None)
try:
@@ -36,6 +109,11 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
except ImportError:
logger.warning("Duplex warmup skipped: the 'websockets' package is not installed.")
return
engine_client = getattr(app.state, "engine_client", None)
plugin = lookup_duplex_plugin(engine_client) if engine_client is not None else None
kind = startup_warmup_kind(plugin, warmup_frames)
if kind is None:
return
served = getattr(args, "served_model_name", None)
if isinstance(served, list | tuple) and served:
model_name = served[0]
@@ -44,17 +122,23 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
else:
model_name = args.model
# One silence unit as the engine-side plugin defines it (DuplexOmniEngine.plugin).
plugin = getattr(getattr(app.state.engine_client, "engine", None), "plugin", None)
frame_samples = int(getattr(plugin, "silence_continuation_samples", 16000))
native_append = plugin.capabilities(max_sessions=1).supports_core_resumable_request if plugin else True
silence = base64.b64encode(bytes(frame_samples * 4)).decode("ascii")
from vllm_omni.clients.duplex import build_realtime_url
url = (
build_realtime_url(f"ws://127.0.0.1:{args.port}/v1/realtime", model_name, autostart=False)
+ "&vllm_omni_warmup=1"
)
logger.info("Duplex warmup starting: %d silent frames of %d samples.", warmup_frames, frame_samples)
if kind == "video_turn":
logger.info(
"Duplex warmup starting: one %.2f s audio chunk and one %dx%d frame.",
_WARMUP_AUDIO_SECONDS,
_WARMUP_FRAME_WIDTH,
_WARMUP_FRAME_HEIGHT,
)
else:
logger.info("Duplex warmup starting: %d silent frames of %d samples.", warmup_frames, frame_samples)
start = time.time()
# The socket is bound before uvicorn finishes starting, so a plain
# connect can be accepted and then reset. Wait for /health first.
@@ -69,24 +153,25 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
else:
logger.warning("Duplex warmup: /health never became ready; skipping warmup.")
return
session: dict[str, object] = {
"session_id": f"warmup-{uuid.uuid4().hex[:8]}",
"model": model_name,
"modalities": ["audio", "text"],
"input_audio_format": "pcm_f32le",
"output_audio_format": "pcm16",
"idle_timeout_s": 60,
"turn_detection": None,
"extra_body": {"auto_response": native_append},
}
if kind == "video_turn":
# The default AURA prompt answers with <|silent|> unless the user
# asked for something. This session is thrown away; force one short
# spoken reply so Talker and Code2Wav compile too. auto_response
# matches the web demo: a commit must actually start a turn.
session["instructions"] = _WARMUP_INSTRUCTIONS
session["extra_body"] = {"auto_response": True}
async with websockets.connect(url, max_size=64 * 1024 * 1024, open_timeout=10) as ws:
await ws.send(
json.dumps(
{
"type": "session.update",
"session": {
"session_id": f"warmup-{uuid.uuid4().hex[:8]}",
"model": model_name,
"modalities": ["audio", "text"],
"input_audio_format": "pcm_f32le",
"output_audio_format": "pcm16",
"idle_timeout_s": 60,
"turn_detection": None,
"extra_body": {"auto_response": native_append},
},
}
)
)
await ws.send(json.dumps({"type": "session.update", "session": session}))
saw_audio = False
sent = 0
@@ -105,25 +190,55 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
if not await _recv_until(lambda e: e.get("type") == "session.created", 30):
logger.warning("Duplex warmup: no session.created within 30 s; aborting warmup.")
return
while sent < warmup_frames:
await ws.send(json.dumps({"type": "input_audio_buffer.append", "audio": silence}))
sent += 1
await asyncio.sleep(0.08)
if kind == "video_turn":
await ws.send(
json.dumps(
{
"type": "input_audio_buffer.append",
"audio": _short_pcm_f32le_b64(),
"format": "pcm_f32le",
"sample_rate_hz": _WARMUP_SAMPLE_RATE_HZ,
"is_speech": True,
"video_frames": [_warmup_jpeg_b64()],
}
)
)
sent = 1
else:
silence = base64.b64encode(bytes(frame_samples * 4)).decode("ascii")
while sent < warmup_frames:
await ws.send(json.dumps({"type": "input_audio_buffer.append", "audio": silence}))
sent += 1
await asyncio.sleep(0.08)
if not native_append:
await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": True}))
# Wait for the pipeline's first audio output so every stage ran.
saw_audio = await _recv_until(lambda e: e.get("type") == "response.output_audio.delta", 30)
commit: dict[str, object] = {"type": "input_audio_buffer.commit", "final": True}
if kind == "video_turn":
commit["create_response"] = True
await ws.send(json.dumps(commit))
# First output audio means Code2Wav ran, so the Talker compile
# ahead of it has run too. Do not wait out the whole utterance.
saw_audio = await _recv_until(
lambda e: e.get("type") == "response.output_audio.delta",
DUPLEX_WARMUP_AUDIO_WAIT_S,
)
# Close the session EXPLICITLY: a bare websocket close parks the
# session in its disconnect grace window, where it would keep
# occupying the max_sessions=1 slot and block the first client.
await ws.send(json.dumps({"type": "session.close"}))
await _recv_until(lambda e: e.get("type") == "session.closed", 10)
logger.info(
"Duplex warmup finished in %.1f s (%d silent frames, first audio %s).",
time.time() - start,
sent,
"received" if saw_audio else "NOT received",
)
if kind == "video_turn":
logger.info(
"Duplex warmup finished in %.1f s (one audio+frame turn, first audio %s).",
time.time() - start,
"received" if saw_audio else "NOT received",
)
else:
logger.info(
"Duplex warmup finished in %.1f s (%d silent frames, first audio %s).",
time.time() - start,
sent,
"received" if saw_audio else "NOT received",
)
except Exception:
logger.exception("Duplex warmup failed; continuing to serve (first session pays cold-start costs).")
finally:
@@ -131,4 +246,9 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
warmup_done.set()
__all__ = ["_warmup_duplex_realtime"]
__all__ = [
"DUPLEX_WARMUP_CLIENT_WAIT_S",
"_warmup_duplex_realtime",
"lookup_duplex_plugin",
"startup_warmup_kind",
]
vllm_omni/entrypoints/openai/api_server.py
+19 -8
View File
@@ -95,7 +95,12 @@ from vllm_omni.engine.stage_init_utils import set_death_signal
from vllm_omni.engine.stage_runtime import OmniClientConfig
from vllm_omni.entrypoints.async_omni import ABORT_TIMEOUT_S, AsyncOmni
from vllm_omni.entrypoints.duplex.serving import OmniDuplexSessionHandler
from vllm_omni.entrypoints.duplex.warmup import _warmup_duplex_realtime
from vllm_omni.entrypoints.duplex.warmup import (
DUPLEX_WARMUP_CLIENT_WAIT_S,
_warmup_duplex_realtime,
lookup_duplex_plugin,
startup_warmup_kind,
)
from vllm_omni.entrypoints.duplex_omni import DuplexOmni
from vllm_omni.entrypoints.openai import app_state as openai_app_state
from vllm_omni.entrypoints.openai.app_state import (
@@ -407,16 +412,19 @@ async def omni_run_server_worker(
await self._inner(scope, receive, send)
# Startup duplex warmup (duplex_session.warmup_frames in the deploy
# yaml): real /v1/realtime connections wait on this event so the
# first client never pays cold-start costs.
# Startup duplex warmup: real /v1/realtime connections wait on this
# event. Video-required models (AURA) do this by default — one short
# audio chunk plus one frame — even when warmup_frames is 0. That is
# not another call to the empty 0.01 s JIT-kernel registry.
duplex_warmup_frames = 0
warmup_kind = None
if getattr(app.state, "openai_serving_duplex", None) is not None:
duplex_cfg = getattr(engine_client, "duplex_session_config", None)
duplex_warmup_frames = int(getattr(duplex_cfg, "warmup_frames", 0) or 0)
app.state.duplex_warmup_done = asyncio.Event() if duplex_warmup_frames > 0 else None
warmup_kind = startup_warmup_kind(lookup_duplex_plugin(engine_client), duplex_warmup_frames)
app.state.duplex_warmup_done = asyncio.Event() if warmup_kind is not None else None
warmup_task: asyncio.Task | None = None
if duplex_warmup_frames > 0:
if warmup_kind is not None:
# Scheduled before serve_http (which may not return until
# shutdown); the coroutine retries its self-connect until the
# server socket is accepting.
@@ -1760,9 +1768,12 @@ async def _wait_for_duplex_warmup(websocket: WebSocket) -> None:
warmup_done = getattr(websocket.app.state, "duplex_warmup_done", None)
if warmup_done is not None and not warmup_done.is_set() and websocket.query_params.get("vllm_omni_warmup") != "1":
try:
await asyncio.wait_for(warmup_done.wait(), timeout=120)
await asyncio.wait_for(warmup_done.wait(), timeout=DUPLEX_WARMUP_CLIENT_WAIT_S)
except (TimeoutError, asyncio.TimeoutError):
logger.warning("Duplex warmup still running after 120 s; admitting the client anyway.")
logger.warning(
"Duplex warmup still running after %d s; admitting the client anyway.",
DUPLEX_WARMUP_CLIENT_WAIT_S,
)
@router.websocket("/v1/duplex")
vllm_omni/model_executor/models/aura_omni/benchmarks/__init__.py
+4
View File
@@ -0,0 +1,4 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""AURA-only benchmark helpers. Not imported by the shared OmniInteract path."""
vllm_omni/model_executor/models/aura_omni/benchmarks/omniinteract_clock.py
+287
View File
@@ -0,0 +1,287 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""OmniInteract sender for video-required, turn-commit duplex sessions.
The shared harness still streams a 1 FPS soundtrack. This clock is the previous
AURA bench: frames are the clock, each append carries a frame, and each
annotation WAV is one append at ``question_time`` followed by ``commit``.
"""
from __future__ import annotations
import asyncio
import json
import logging
import subprocess
import time
from collections.abc import Sequence
from pathlib import Path
from typing import Any
import pybase64 as base64
logger = logging.getLogger(__name__)
ANNOTATION_VIDEO_FPS = 2.0
_PCM16_SAMPLE_RATE = 16_000
_PCM16_BYTES_PER_SAMPLE = 2
def wants_annotation_video_clock(capabilities: object) -> bool:
"""True when every append must carry video and a commit is one turn.
MiniCPM requires audio and is not turn-commit-only, so it stays on the
shared soundtrack clock. The check is the session capability set, not the
model name.
"""
if not isinstance(capabilities, dict):
return False
required = capabilities.get("required_input_modalities") or ()
if isinstance(required, str):
required = (required,)
try:
required_set = {str(item) for item in required}
except TypeError:
return False
return "video" in required_set and bool(capabilities.get("supports_turn_commit_only"))
def _parse_clock(value: object) -> float | None:
if isinstance(value, (int, float)) and not isinstance(value, bool):
return float(value)
if not isinstance(value, str) or not value.strip():
return None
parts = value.strip().split(":")
try:
nums = [float(part) for part in parts]
except ValueError:
return None
if len(nums) == 1:
return nums[0]
if len(nums) == 2:
minutes, seconds = nums
return minutes * 60.0 + seconds
if len(nums) == 3:
hours, minutes, seconds = nums
return hours * 3600.0 + minutes * 60.0 + seconds
return None
def _load_annotation_pcm(path: Path) -> bytes:
command = [
"ffmpeg",
"-loglevel",
"error",
"-i",
str(path),
"-f",
"s16le",
"-ac",
"1",
"-ar",
str(_PCM16_SAMPLE_RATE),
"pipe:1",
]
try:
result = subprocess.run(command, capture_output=True, timeout=60, check=False)
except subprocess.TimeoutExpired as exc:
raise TimeoutError(f"ffmpeg timed out reading {path}") from exc
except FileNotFoundError as exc:
raise RuntimeError("ffmpeg is required to read OmniInteract annotation WAV") from exc
if result.returncode:
error = result.stderr.decode("utf-8", "ignore").strip()
raise RuntimeError(f"ffmpeg failed for {path}: {error}")
return result.stdout
def load_annotation_utterances(video_path: Path) -> list[dict[str, Any]]:
"""Annotation WAV files injected at ``question_time``.
Layout is ``{subset}/annotations/{stem}.json`` and
``{subset}/audios/{stem}_{qa}.wav``. The video soundtrack is not used.
"""
subset_root = video_path.parent.parent
annotation_path = subset_root / "annotations" / f"{video_path.stem}.json"
if not annotation_path.is_file():
raise ValueError(f"Missing OmniInteract annotation for {video_path}: {annotation_path}")
annotation = json.loads(annotation_path.read_text(encoding="utf-8"))
if not isinstance(annotation, list):
raise ValueError(f"OmniInteract annotation is not a list: {annotation_path}")
items: list[dict[str, Any]] = []
for index, row in enumerate(annotation):
if not isinstance(row, dict):
continue
at_sec = _parse_clock(row.get("question_time"))
audio_path = subset_root / "audios" / f"{video_path.stem}_{index}.wav"
if at_sec is None or not audio_path.is_file():
continue
pcm = _load_annotation_pcm(audio_path)
if not pcm:
continue
items.append({"at_sec": at_sec, "pcm": pcm, "sent": False, "qa_index": index})
if not items:
raise ValueError(
f"No annotation WAV at question_time for {video_path.name}; "
f"expected {subset_root / 'audios' / (video_path.stem + '_*.wav')}"
)
items.sort(key=lambda item: float(item["at_sec"]))
return items
def _collector_events(client: Any) -> list[object] | None:
events = getattr(getattr(client, "events", None), "events", None)
return events if isinstance(events, list) else None
def _response_id(event: dict[str, object]) -> str | None:
raw = event.get("response_id")
if isinstance(raw, str) and raw:
return raw
response = event.get("response")
if isinstance(response, dict):
nested = response.get("id")
if isinstance(nested, str) and nested:
return nested
return None
def _open_response_count(client: Any) -> int | None:
"""Responses still occupying the input gate, or None when the client has no event log."""
events = _collector_events(client)
if events is None:
return None
open_ids: set[str] = set()
anon = 0
for event in events:
if not isinstance(event, dict):
continue
kind = event.get("type")
response_id = _response_id(event)
if kind == "response.created":
if response_id:
open_ids.add(response_id)
else:
anon += 1
elif kind in {"response.done", "response.listen"}:
if response_id:
open_ids.discard(response_id)
elif anon:
anon -= 1
elif open_ids:
open_ids.pop()
return len(open_ids) + anon
def _terminal_count(client: Any) -> int:
events = _collector_events(client) or []
return sum(
1 for event in events if isinstance(event, dict) and event.get("type") in {"response.done", "response.listen"}
)
_VISION_COMMIT_FRAMES = 2
async def stream_annotation_clock(
client: Any,
frames: Sequence[str | None],
playback: Any,
*,
video_path: Path,
fps: float = ANNOTATION_VIDEO_FPS,
) -> tuple[int, int, float, float]:
"""Send one frame per tick. Speech commits at question_time; vision every two frames.
Vision-only appends are not sent until they can be committed. The server
buffer keeps only the latest frames and will not infer until ``commit``.
A commit waits while a response is still open so it does not abort Stage0/1.
Clients without an event log (unit stubs) are not gated.
"""
utterances = await asyncio.to_thread(load_annotation_utterances, video_path)
frame_interval_s = 1.0 / fps
started_at = time.monotonic()
sent_frames = chunks = 0
lags: list[float] = []
held_frame: str | None = None
vision_pending: list[str] = []
local_commits = 0
terminals_at_start = 0
if _collector_events(client) is not None:
terminals_at_start = _terminal_count(client)
def _gate_blocked() -> bool:
if _collector_events(client) is None:
return False
open_responses = _open_response_count(client)
if open_responses:
return True
return local_commits > _terminal_count(client) - terminals_at_start
async def _commit_vision(frames_to_send: list[str]) -> None:
nonlocal chunks, sent_frames, local_commits
if not frames_to_send or _gate_blocked():
return
await client.send(
{
"type": "input_audio_buffer.append",
"is_speech": False,
"sample_rate_hz": _PCM16_SAMPLE_RATE,
"video_frames": list(frames_to_send),
}
)
chunks += 1
sent_frames += len(frames_to_send)
await playback.acknowledge(client)
await client.commit()
local_commits += 1
frames_to_send.clear()
for index, frame in enumerate(frames):
if frame:
held_frame = frame
if not held_frame:
continue
source_time = index / fps
due = [item for item in utterances if not item["sent"] and float(item["at_sec"]) <= source_time]
due_at = time.monotonic()
if due:
vision_pending.clear()
for item in due:
audio = item["pcm"]
assert isinstance(audio, bytes)
await client.send(
{
"type": "input_audio_buffer.append",
"audio": base64.b64encode(audio).decode("ascii"),
"input_audio_format": "pcm16",
"sample_rate_hz": _PCM16_SAMPLE_RATE,
"duration_ms": len(audio) * 1000 // (_PCM16_SAMPLE_RATE * _PCM16_BYTES_PER_SAMPLE),
"is_speech": True,
"video_frames": [held_frame],
}
)
chunks += 1
sent_frames += 1
await playback.acknowledge(client)
item["sent"] = True
await client.commit()
local_commits += 1
else:
vision_pending.append(held_frame)
if len(vision_pending) > _VISION_COMMIT_FRAMES:
vision_pending = vision_pending[-_VISION_COMMIT_FRAMES:]
if len(vision_pending) >= _VISION_COMMIT_FRAMES:
await _commit_vision(vision_pending)
lags.append(max(0.0, time.monotonic() - due_at))
await asyncio.sleep(max(0.0, started_at + (index + 1) * frame_interval_s - time.monotonic()))
if vision_pending:
await _commit_vision(vision_pending)
unsent = [item for item in utterances if not item["sent"]]
if unsent:
logger.warning(
"Annotation WAV past the last frame was not injected: video=%s qa=%s",
video_path,
[item["qa_index"] for item in unsent],
)
return chunks, sent_frames, sum(lags) / len(lags) if lags else 0.0, max(lags, default=0.0)
vllm_omni/model_executor/models/aura_omni/duplex/__init__.py
+6
View File
@@ -0,0 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from vllm_omni.model_executor.models.aura_omni.duplex.plugin import AuraDuplexPlugin
__all__ = ["AuraDuplexPlugin"]
vllm_omni/model_executor/models/aura_omni/duplex/capabilities.py
+48
View File
@@ -0,0 +1,48 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
from vllm_omni.engine.duplex.config import DuplexCapabilities
def aura_duplex_capabilities(*, max_sessions: int = 1) -> DuplexCapabilities:
"""AURA turn-commit duplex capabilities (not MiniCPM resident append)."""
supports_multi_session = max_sessions > 1
return DuplexCapabilities(
supports_model_native_turn_policy=False,
supports_barge_in=True,
supports_input_append=True,
supports_replace_latest_chunk=False,
supports_reencode_context=False,
supports_turn_commit_only=True,
supports_kv_lease=False,
supports_core_kv_lease=False,
supports_model_internal_state=True,
supports_stage_resumption=False,
supports_scheduler_native_append=False,
supports_core_resumable_request=False,
supports_concurrent_turn_requests=True,
# Video required, audio optional (vision-only / silent+frames legal).
required_input_modalities=frozenset({"video"}),
optional_input_modalities=frozenset({"audio"}),
supports_stage_connector_handoff=False,
supports_independent_io_streams=True,
supports_realtime_endpoint=True,
supports_multi_session=supports_multi_session,
supports_multi_session_same_replica=supports_multi_session,
supports_session_lease=True,
supports_session_resume=True,
session_admission_mode="engine_managed",
supports_audio_truncate=False,
requires_model_runner_kv=False,
requires_native_stage_role=False,
adapter_patterns=["turn_commit"],
signal_sources=["client_event", "server_policy"],
stage_handoff_transport="stage_connector",
chunk_period_ms=1000,
target_barge_in_latency_ms=None,
)
__all__ = ["aura_duplex_capabilities"]
vllm_omni/model_executor/models/aura_omni/duplex/data_plane.py
+381
View File
@@ -0,0 +1,381 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Project AURA stage outputs into duplex internal events."""
from __future__ import annotations
from collections.abc import Iterator, Mapping
from dataclasses import dataclass, field
import numpy as np
import torch
from vllm_omni.engine.duplex.contracts import (
duplex_resource_request_belongs_to_session,
duplex_turn_id_from_request_id,
)
from vllm_omni.engine.duplex.plugin import DuplexDataPlane, EncodeAudio
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
SILENT_TEXT,
is_effectively_silent,
is_silent_text_prefix,
)
from vllm_omni.outputs.duplex import get_duplex_output_decision
@dataclass(frozen=True, slots=True)
class AuraDataPlaneContext:
epoch: int = 0
turn_id: int = 0
auto_responds: bool = True
response_format: str = "wav"
speed: float | None = None
modalities: tuple[str, ...] = ("text", "audio")
@dataclass(slots=True)
class _RequestState:
text_sent: str = ""
text_emitted: str = ""
audio_offset: int = 0
silent: bool = False
terminal: bool = False
context_committed: bool = False
stage_seen: set[int] = field(default_factory=set)
def _unwrap(output: object) -> tuple[object, object | None, int | None]:
stage_id = getattr(output, "stage_id", None)
inner = getattr(output, "request_output", None)
if inner is not None and inner is not output:
output = inner
outputs = getattr(output, "outputs", None)
completion = outputs[0] if isinstance(outputs, list) and outputs else None
if stage_id is None:
stage_id = getattr(output, "stage_id", None)
return output, completion, int(stage_id) if isinstance(stage_id, int) else None
def _strip_think(text: str) -> str:
"""Drop Qwen3-VL ``<think>...</think>`` wrappers from assistant text."""
from vllm_omni.model_executor.stage_input_processors.aura_omni import _strip_assistant_text
return _strip_assistant_text(text)
def _text_from(output: object, completion: object | None) -> str:
for candidate in (completion, output):
if candidate is None:
continue
for attr in ("cumulative_text", "text"):
value = getattr(candidate, attr, None)
if isinstance(value, str) and value:
return _strip_think(value)
return ""
def _multimodal(output: object, completion: object | None) -> dict[str, object]:
"""Prefer decision metadata, then completion multimodal payload.
Empty mappings must not short-circuit: OmniRequestOutput.multimodal_output
returns ``{}`` when completion payloads are falsy, which would otherwise
hide a real MultimodalPayload on the completion (MiniCPM data-plane does
the same ``if not mm`` fallthrough).
"""
decision = get_duplex_output_decision(output)
metadata = getattr(decision, "metadata", None)
if isinstance(metadata, Mapping) and metadata:
return dict(metadata)
for candidate in (
getattr(output, "multimodal_output", None),
getattr(completion, "multimodal_output", None) if completion is not None else None,
):
if isinstance(candidate, Mapping) and candidate:
return dict(candidate)
return {}
def _audio_value(metadata: Mapping[str, object]) -> object | None:
"""Extract PCM/latent audio from a multimodal mapping.
Code2Wav wire payloads use producer key ``model_outputs``; after
``MultimodalPayload.from_raw(..., modality_key)`` that becomes ``audio``
when modality is audio, or another modality key (e.g. ``text``/``hidden``)
when the stage output_modality was wrongly tagged. Accept common aliases and
finally the first non-sr tensor so duplex still surfaces Stage3 PCM.
"""
value = next(
(metadata[key] for key in ("audio", "model_outputs", "latent", "hidden", "text") if key in metadata),
None,
)
if value is None:
primary = getattr(metadata, "primary_tensor", None)
if primary is not None:
value = primary
if value is None:
for key, candidate in metadata.items():
if key in {"sr", "sample_rate", "sample_rate_hz", "audio_sample_rate"}:
continue
if isinstance(candidate, torch.Tensor) or (
isinstance(candidate, list) and candidate and isinstance(candidate[0], torch.Tensor)
):
value = candidate
break
if isinstance(value, list) and len(value) == 1:
return value[0]
return value
def _sample_rate(metadata: Mapping[str, object]) -> int:
value = metadata.get("sr", metadata.get("sample_rate_hz", 24000))
if isinstance(value, list) and value:
value = value[0]
if hasattr(value, "item"):
value = value.item()
return int(value) if isinstance(value, int | float) else 24000
def _pcm_sample_count(audio: object) -> int:
if isinstance(audio, torch.Tensor):
return int(audio.detach().reshape(-1).numel())
try:
return int(np.asarray(audio, dtype=np.float32).reshape(-1).size)
except (TypeError, ValueError):
return 0
def _flat_audio(audio: object) -> torch.Tensor | np.ndarray | None:
if isinstance(audio, torch.Tensor):
return audio.detach().reshape(-1)
try:
return np.asarray(audio, dtype=np.float32).reshape(-1)
except (TypeError, ValueError):
return None
def _new_audio_samples(audio: object, already_sent: int) -> tuple[torch.Tensor | np.ndarray, int] | None:
"""Return only samples not yet sent.
Code2Wav hands back a growing waveform. A longer buffer is a cumulative
snapshot: send the tail. An equal buffer is a duplicate. A shorter buffer
is a real delta (or a restart) and is sent whole.
"""
flat = _flat_audio(audio)
if flat is None:
return None
count = int(flat.numel()) if isinstance(flat, torch.Tensor) else int(flat.size)
if count > already_sent:
return flat[already_sent:], count
if count == already_sent:
return None
return flat, already_sent + count
def _requested_audio_format(context: object | None) -> tuple[str, float | None]:
"""Honor the session response format. Missing context stays PCM16 (Realtime default)."""
fmt = getattr(context, "response_format", None)
speed = getattr(context, "speed", None)
if not isinstance(fmt, str) or not fmt:
fmt = "pcm16"
if not isinstance(speed, int | float):
speed = None
return fmt, float(speed) if speed is not None else None
class AuraDataPlaneSession(DuplexDataPlane):
"""Map Stage1 text + Stage3 audio (or silent) onto duplex events."""
def __init__(self, encode_audio: EncodeAudio) -> None:
self._encode_audio = encode_audio
self._requests: dict[str, _RequestState] = {}
self._closed: set[str] = set()
def begin_request(self, request_id: str) -> None:
self._closed.discard(request_id)
state = self._requests.setdefault(request_id, _RequestState())
state.terminal = False
def is_terminal(self, request_id: str | None) -> bool:
if request_id is None:
return False
if request_id in self._closed:
return True
state = self._requests.get(request_id)
return state is not None and state.terminal
def mark_terminal(self, request_id: str) -> None:
# The shared session already calls this when a turn finishes. Drop the
# projector row here so completed turns do not accumulate; the id stays
# closed so a late chunk cannot open a second turn.
self._requests.pop(request_id, None)
self._closed.add(request_id)
def close_stream(self, request_id: str) -> None:
state = self._requests.get(request_id)
if state is not None:
state.audio_offset = 0
def close_session(self, session_id: str, *, active_request_id: str | None = None) -> None:
if active_request_id is not None:
self._requests.pop(active_request_id, None)
for request_id in list(self._requests):
if duplex_resource_request_belongs_to_session(request_id, session_id):
self._requests.pop(request_id, None)
self._closed = {
request_id
for request_id in self._closed
if not duplex_resource_request_belongs_to_session(request_id, session_id)
}
from vllm_omni.model_executor.models.aura_omni.duplex.history import drop_session_history
drop_session_history(session_id)
def project(self, result: object, *, context: object | None = None) -> Iterator[dict[str, object]]:
if not isinstance(result, dict):
return
outputs = result.get("data_plane_outputs")
if not isinstance(outputs, list):
return
for output in outputs:
yield from self.project_output(output, context=context)
def project_output(self, result: object, *, context: object | None = None) -> Iterator[dict[str, object]]:
response_format, speed = _requested_audio_format(context)
request_id = getattr(result, "request_id", None)
if not isinstance(request_id, str) or not request_id:
return
if request_id in self._closed:
return
model_turn_id = duplex_turn_id_from_request_id(request_id)
outer_finished = bool(getattr(result, "finished", False))
output, completion, stage_id = _unwrap(result)
state = self._requests.setdefault(request_id, _RequestState())
if stage_id is not None:
state.stage_seen.add(stage_id)
def _event(**fields: object) -> dict[str, object]:
payload = dict(fields)
if model_turn_id is not None:
payload["model_turn_id"] = model_turn_id
return payload
decision = get_duplex_output_decision(result)
metadata = dict(getattr(decision, "metadata", {}) or {})
if metadata.get("model_listen") or metadata.get("duplex_native_decision") == "listen":
state.silent = True
# DIRECT_RESPONSE skips aura2tts. The session runner commits
# model context from model_context_text; this projector does not.
yield _event(
stage_role="thinker",
is_listen=True,
data_plane_request_id=request_id,
text="",
end_of_turn=True,
silent=True,
model_context_text=SILENT_TEXT,
# Prewarm already reserved Stage2/3 on Stage0 submit. Silent
# short-circuit never feeds codec chunks; abort frees those seats.
abort_data_plane_request=True,
)
state.terminal = True
return
text = _text_from(output, completion)
if text and text != state.text_sent:
state.text_sent = text
if is_effectively_silent(text):
state.silent = True
# Hold deltas that are still a <|silent|> prefix until short-circuit.
if not state.silent and not is_silent_text_prefix(text):
emit = (
text
if not state.text_emitted
else (text[len(state.text_emitted) :] if text.startswith(state.text_emitted) else text)
)
if emit:
state.text_emitted = text
yield _event(
stage_role="thinker",
is_listen=False,
data_plane_request_id=request_id,
text=emit,
end_of_turn=False,
)
# RequestOutput.finished / envelope finished is the stream EOS.
# CompletionOutput.finished is true on every Code2Wav chunk and must
# not close the duplex turn (runner would then drop via is_terminal).
finished = bool(outer_finished or getattr(output, "finished", False))
if (
finished
and stage_id == 1
and not state.silent
and not state.context_committed
and state.text_sent
and not is_effectively_silent(state.text_sent)
):
state.context_committed = True
yield _event(
stage_role="thinker",
is_listen=False,
data_plane_request_id=request_id,
text="",
end_of_turn=False,
model_context_text=state.text_sent,
)
is_final_audio_stage = stage_id is None or stage_id >= 3
mm = _multimodal(output, completion)
audio = _audio_value(mm)
if audio is not None and not state.silent:
sample_rate = _sample_rate(mm)
delta = _new_audio_samples(audio, state.audio_offset)
encoded = None
n = 0
if delta is not None:
samples, new_offset = delta
encoded = self._encode_audio(samples, sample_rate, response_format, speed)
if encoded:
n = _pcm_sample_count(samples)
state.audio_offset = new_offset
if encoded:
duration_ms = round(n * 1000 / max(1, sample_rate))
end_of_turn = bool(finished and is_final_audio_stage)
yield _event(
stage_role="tts",
is_listen=False,
data_plane_request_id=request_id,
audio=encoded,
audio_format=response_format,
sample_rate_hz=sample_rate,
audio_duration_ms=duration_ms,
end_of_turn=end_of_turn,
)
if end_of_turn:
state.terminal = True
return
if finished and is_final_audio_stage and not state.terminal:
yield _event(
stage_role="tts",
is_listen=False,
data_plane_request_id=request_id,
text="",
end_of_turn=True,
)
state.terminal = True
elif finished and state.silent and not state.terminal:
yield _event(
stage_role="thinker",
is_listen=True,
data_plane_request_id=request_id,
text="",
end_of_turn=True,
silent=True,
model_context_text=SILENT_TEXT,
abort_data_plane_request=True,
)
state.terminal = True
__all__ = ["AuraDataPlaneContext", "AuraDataPlaneSession"]
vllm_omni/model_executor/models/aura_omni/duplex/history.py
+128
View File
@@ -0,0 +1,128 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Stage1-local SessionHistory for duplex AURA (import from asr2aura / aura2tts)."""
from __future__ import annotations
from dataclasses import dataclass, field
from threading import Lock
# Native gateway (AURA_demo-main/gateway/actor.py): keep up to 45 video
# rounds, then strip the video payload from the oldest 30. Text stays.
# Empty user + <|silent|> after the strip is dropped as a pair.
# Total conversation rounds are effectively unbounded (Native default 999).
_MAX_VIDEO_ROUNDS = 45
_VIDEO_ROUNDS_TO_REMOVE = 30
_MAX_CONVERSATION_ROUNDS = 999
_SILENT = "<|silent|>"
_STORE: dict[str, SessionHistory] = {}
_LOCK = Lock()
@dataclass
class SessionHistory:
"""Per-session chat turns held in the Stage1 process."""
session_id: str
max_video_rounds: int = _MAX_VIDEO_ROUNDS
video_rounds_to_remove: int = _VIDEO_ROUNDS_TO_REMOVE
max_conversation_rounds: int = _MAX_CONVERSATION_ROUNDS
messages: list[dict[str, object]] = field(default_factory=list)
pending_user: str | None = None
pending_video: object | None = None
def render_prefix(self) -> str:
rendered: list[str] = []
for message in self.messages:
role = message.get("role")
content = message.get("content")
if role not in {"user", "assistant"} or not isinstance(content, str):
continue
body = content
if role == "user" and message.get("video") is not None:
body = "<|vision_start|><|video_pad|><|vision_end|>" + content
if not body:
continue
rendered.append(f"<|im_start|>{role}\n{body}<|im_end|>\n")
return "".join(rendered)
def retained_videos(self) -> list[object]:
"""Committed clips that still have pixels, in prompt order."""
return [
message["video"]
for message in self.messages
if message.get("role") == "user" and message.get("video") is not None
]
def begin_user_turn(self, transcript: str, video: object | None = None) -> None:
self.pending_user = transcript.strip()
self.pending_video = video
def commit_turn(self, assistant_text: str) -> None:
user = self.pending_user or ""
video = self.pending_video
self.pending_user = None
self.pending_video = None
text = (assistant_text or "").strip()
if not user and video is None:
if not text or text == _SILENT:
return
self.messages.append({"role": "assistant", "content": text})
self.prune()
return
user_message: dict[str, object] = {"role": "user", "content": user}
if video is not None:
user_message["video"] = video
self.messages.append(user_message)
self.messages.append({"role": "assistant", "content": text or _SILENT})
self.prune()
def prune(self) -> None:
video_indices = [
index
for index, message in enumerate(self.messages)
if message.get("role") == "user" and message.get("video") is not None
]
if len(video_indices) > self.max_video_rounds:
drop: set[int] = set()
for index in video_indices[: self.video_rounds_to_remove]:
user_message = self.messages[index]
user_message["video"] = None
text = user_message.get("content")
if isinstance(text, str) and text.strip():
continue
nxt = index + 1
if nxt >= len(self.messages) or self.messages[nxt].get("role") != "assistant":
continue
assistant = self.messages[nxt].get("content")
if isinstance(assistant, str) and assistant.strip() in {_SILENT, ""}:
drop.add(index)
drop.add(nxt)
if drop:
self.messages = [message for index, message in enumerate(self.messages) if index not in drop]
user_count = sum(1 for message in self.messages if message.get("role") == "user")
overflow = user_count - self.max_conversation_rounds
if overflow > 0:
self.messages = self.messages[overflow * 2 :]
def get_or_create_session_history(session_id: str) -> SessionHistory:
with _LOCK:
history = _STORE.get(session_id)
if history is None:
history = SessionHistory(session_id=session_id)
_STORE[session_id] = history
return history
def drop_session_history(session_id: str) -> None:
with _LOCK:
_STORE.pop(session_id, None)
__all__ = [
"SessionHistory",
"drop_session_history",
"get_or_create_session_history",
]
vllm_omni/model_executor/models/aura_omni/duplex/input.py
+186
View File
@@ -0,0 +1,186 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Commit-only PCM buffer for AURA duplex (emit whole utterance on commit)."""
from __future__ import annotations
import binascii
import pybase64 as base64
from vllm_omni.engine.duplex.pcm_reservation import (
commit_ordered_reservation,
rollback_ordered_reservation,
)
from vllm_omni.engine.duplex.plugin import PcmAppendBuffer, PcmAppendReservation
_SAMPLE_BYTES = 4
_DEFAULT_SAMPLE_RATE_HZ = 16000
class AuraPcmAppendReservation(PcmAppendReservation):
__slots__ = ("_active", "_owner", "_raw", "_reserved_bytes", "operation_id", "payload")
def __init__(
self,
*,
owner: AuraPcmAppendBuffer,
operation_id: str,
payload: dict[str, object] | None,
raw: bytes,
reserved_bytes: int | None = None,
) -> None:
self._owner = owner
self.operation_id = operation_id
self.payload = payload
self._raw = raw
self._reserved_bytes = len(raw) if reserved_bytes is None else reserved_bytes
self._active = True
@property
def active(self) -> bool:
return self._active
@property
def byte_count(self) -> int:
return self._reserved_bytes
def commit(self) -> None:
commit_ordered_reservation(self._owner._reservations, self, head_only=False)
def rollback(self) -> None:
rollback_ordered_reservation(
self._owner._reservations,
self,
self._owner._buffer,
active_only=True,
)
class AuraPcmAppendBuffer(PcmAppendBuffer):
"""Accumulate PCM until commit; ignore 1s auto-flush unit semantics."""
def __init__(self) -> None:
self._buffer = bytearray()
self._sample_rate_hz: int | None = None
self._frame_queue: list[str] = []
self._reservations: list[AuraPcmAppendReservation] = []
self._had_speech = False
@property
def pending_byte_count(self) -> int:
# Include retained video frame wire sizes so admission/release accounting
# stays balanced after the mailbox releases the full append reservation.
return len(self._buffer) + sum(len(frame) for frame in self._frame_queue)
def clear(self) -> None:
self._buffer.clear()
self._frame_queue.clear()
self._reservations.clear()
self._had_speech = False
def clear_force_listen(self) -> None:
return
def has_pending(self) -> bool:
return bool(self._buffer) or bool(self._frame_queue)
def has_reserved(self) -> bool:
return any(reservation.active for reservation in self._reservations)
def prepare_append(
self,
payload: dict[str, object],
*,
operation_id: str,
chunk_period_ms: int,
allow_emit: bool,
) -> AuraPcmAppendReservation | None:
del chunk_period_ms, allow_emit, operation_id
fmt = payload.get("format")
sample_rate_hz = payload.get("sample_rate_hz")
audio = payload.get("audio")
frames_in = payload.get("video_frames")
if isinstance(frames_in, list):
latest = [frame for frame in frames_in if isinstance(frame, str) and frame]
if latest:
self._frame_queue = latest[-2:]
if fmt != "pcm_f32le" or not isinstance(sample_rate_hz, int) or not isinstance(audio, str):
self._had_speech = self._had_speech or bool(payload.get("is_speech", False))
return None
try:
raw = base64.b64decode(audio, validate=True)
except (binascii.Error, ValueError):
return None
if len(raw) % _SAMPLE_BYTES:
return None
if self._sample_rate_hz is not None and self._sample_rate_hz != sample_rate_hz:
raise ValueError("AURA duplex audio append sample_rate_hz changed within a session")
self._sample_rate_hz = sample_rate_hz
self._buffer.extend(raw)
self._had_speech = self._had_speech or bool(payload.get("is_speech", True))
return None
def prepare_commit(
self,
*,
operation_id: str,
chunk_period_ms: int,
) -> AuraPcmAppendReservation:
del chunk_period_ms
if not self._buffer and not self._frame_queue:
reservation = AuraPcmAppendReservation(
owner=self,
operation_id=operation_id,
payload=None,
raw=b"",
reserved_bytes=0,
)
self._reservations.append(reservation)
return reservation
sample_rate_hz = self._sample_rate_hz or _DEFAULT_SAMPLE_RATE_HZ
raw = bytes(self._buffer)
self._buffer.clear()
payload: dict[str, object] = {
"type": "audio",
"audio": base64.b64encode(raw).decode("ascii") if raw else "",
"format": "pcm_f32le",
"sample_rate_hz": sample_rate_hz,
"final": True,
# Do not infer speech from non-empty PCM: vision-follow often sends
# 10ms zeros with is_speech=False so Stage0/1 treat it as vision-only
# rather than a new spoken utterance.
"is_speech": self._had_speech,
"aura_turn_commit": True,
}
video_bytes = 0
if self._frame_queue:
payload["video_frames"] = list(self._frame_queue)
video_bytes = sum(len(frame) for frame in self._frame_queue)
self._frame_queue.clear()
self._had_speech = False
reservation = AuraPcmAppendReservation(
owner=self,
operation_id=operation_id,
payload=payload,
raw=raw,
reserved_bytes=len(raw) + video_bytes,
)
self._reservations.append(reservation)
return reservation
def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None:
reservation = self.prepare_commit(
operation_id="flush",
chunk_period_ms=chunk_period_ms,
)
reservation.commit()
return reservation.payload
__all__ = ["AuraPcmAppendBuffer", "AuraPcmAppendReservation"]
vllm_omni/model_executor/models/aura_omni/duplex/plugin.py
+520
View File
@@ -0,0 +1,520 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""AURA full-duplex model plugin: turn-commit four-stage NewRequest."""
from __future__ import annotations
import binascii
from collections.abc import Mapping
from typing import Any
import numpy as np
import pybase64 as base64
from vllm.sampling_params import SamplingParams
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.contracts import (
DuplexAppendPlan,
DuplexFence,
DuplexOutputAction,
DuplexOutputDecision,
)
from vllm_omni.engine.duplex.plugin import DuplexModelPlugin, EncodeAudio
from vllm_omni.model_executor.models.aura_omni.duplex.capabilities import aura_duplex_capabilities
from vllm_omni.model_executor.models.aura_omni.duplex.data_plane import (
AuraDataPlaneContext,
AuraDataPlaneSession,
)
from vllm_omni.model_executor.models.aura_omni.duplex.session import AuraServingSessionState
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
DEFAULT_AURA_SYSTEM_PROMPT,
SILENT_TEXT,
is_effectively_silent,
)
# AURA v1 (Qwen3-VL) silent / ChatML turn-end ids.
AURA_SILENT_TOKEN_ID = 151669
AURA_IM_END_TOKEN_ID = 151645
AURA_SILENT_TOKEN_IDS = frozenset({AURA_SILENT_TOKEN_ID})
_PRIVATE_KEYS = frozenset(
{
"aura_system_prompt",
"aura_history_revision",
}
)
_TTS_EXTRA_KEYS = (
"tts_task_type",
"tts_language",
"tts_speaker",
"tts_instruct",
"tts_ref_audio",
"tts_ref_text",
"tts_x_vector_only_mode",
)
def _decode_pcm_f32le(payload: Mapping[str, object]) -> tuple[np.ndarray, int]:
audio = payload.get("audio")
sample_rate_hz = payload.get("sample_rate_hz", 16000)
if not isinstance(audio, str) or not isinstance(sample_rate_hz, int):
raise ValueError("AURA duplex commit payload requires pcm_f32le audio and sample_rate_hz")
try:
raw = base64.b64decode(audio, validate=True)
except (binascii.Error, ValueError) as exc:
raise ValueError("AURA duplex audio is not valid base64") from exc
if len(raw) % 4:
raise ValueError("AURA duplex pcm_f32le payload has a partial sample")
values = np.frombuffer(raw, dtype="<f4").copy()
return values, sample_rate_hz
def _video_frames_to_mm(frames: object) -> dict[str, object]:
if not isinstance(frames, list) or not frames:
return {}
from io import BytesIO
from PIL import Image
images: list[object] = []
for frame in frames:
if not isinstance(frame, str) or not frame:
continue
try:
raw = base64.b64decode(frame, validate=True)
except (binascii.Error, ValueError):
continue
try:
images.append(np.asarray(Image.open(BytesIO(raw)).convert("RGB")))
except Exception:
continue
if not images:
return {}
# One <|video_pad|> for the whole clip. Two separate images with one
# <|image_pad|> crash Stage1 (mm_items['image'][1]). Qwen3-VL's temporal
# patch wants at least two frames; a single sticky frame is duplicated.
if len(images) == 1:
images = [images[0], images[0]]
video = np.stack(images, axis=0)
n_frames = int(video.shape[0])
metadata = {
"fps": 2.0,
"duration": n_frames / 2.0,
"total_num_frames": n_frames,
"frames_indices": list(range(n_frames)),
"video_backend": "opencv",
"do_sample_frames": False,
}
return {"video": [(video, metadata)]}
def _completion_token_ids(completion: object | None) -> list[int]:
if completion is None:
return []
for attr in ("token_ids", "cumulative_token_ids"):
value = getattr(completion, attr, None)
if isinstance(value, list | tuple):
out: list[int] = []
for item in value:
try:
out.append(int(item))
except (TypeError, ValueError):
continue
if out:
return out
return []
def _first_completion(output: object) -> object | None:
outputs = getattr(output, "outputs", None)
if isinstance(outputs, list) and outputs:
return outputs[0]
return None
class AuraDuplexPlugin(DuplexModelPlugin):
"""Turn-commit AURA: one ephemeral four-stage request per utterance."""
plugin_id = "aura"
private_runtime_config_keys = _PRIVATE_KEYS
def __init__(self, encode_audio: EncodeAudio) -> None:
super().__init__(encode_audio)
self.data_plane = AuraDataPlaneSession(encode_audio)
def configure_sampling_params(
self,
*,
runtime_config: dict[str, object],
defaults: tuple[object, ...],
) -> tuple[object, ...]:
del runtime_config
configured = list(defaults)
if len(configured) > 1 and isinstance(configured[1], SamplingParams):
stage1 = configured[1].clone()
stop_ids = list(stage1.stop_token_ids or [])
for stop_id in (AURA_SILENT_TOKEN_ID, AURA_IM_END_TOKEN_ID):
if stop_id not in stop_ids:
stop_ids.append(stop_id)
stage1.stop_token_ids = stop_ids
# 151669 is a stop id: keep it in the completion so decide_output /
# aura2tts can see <|silent|> instead of an empty detokenized string.
stage1.include_stop_str_in_output = True
stage1.skip_special_tokens = False
configured[1] = stage1
# Codec EOS (2150) and the 240-token cap are defaults for an unset
# Talker config. A value already set by yaml or the caller is kept,
# even when max_tokens is longer than 240.
if len(configured) > 2 and isinstance(configured[2], SamplingParams):
stage2 = configured[2].clone()
if not stage2.stop_token_ids:
stage2.stop_token_ids = [2150]
if stage2.max_tokens is None:
stage2.max_tokens = 240
configured[2] = stage2
return tuple(configured)
def draining_stage_ids(self, *, stage_count: int) -> frozenset[int]:
# Talker is stage 2; Code2Wav is the last stage. Stage 0/1 are the
# input gate and are idle once a turn is released.
if stage_count <= 2:
return frozenset()
return frozenset(range(2, stage_count))
def plan_append(
self,
*,
request_id: str,
fence: DuplexFence,
session_config: dict[str, object],
runtime_config: dict[str, object],
seq: int,
turn_seq: int,
payload: object,
final: bool,
sampling_params: object,
) -> DuplexAppendPlan:
del request_id, session_config, seq, turn_seq, sampling_params
def _skip() -> DuplexAppendPlan:
return DuplexAppendPlan(
prompt={
"prompt_token_ids": [0],
"additional_information": {
"aura_skip_submit": True,
"session_id": fence.session_id,
"epoch": fence.epoch,
"turn_id": fence.turn_id,
},
}
)
if not isinstance(payload, Mapping):
if not final:
return _skip()
raise ValueError("AURA duplex plan_append expects a mapping payload")
if not final and not payload.get("aura_turn_commit"):
return _skip()
mm = _video_frames_to_mm(payload.get("video_frames"))
audio_b64 = payload.get("audio")
is_speech = bool(payload.get("is_speech", True))
has_audio = isinstance(audio_b64, str) and bool(audio_b64)
if has_audio:
wav, sample_rate_hz = _decode_pcm_f32le(payload)
else:
wav, sample_rate_hz = np.zeros(0, dtype=np.float32), int(payload.get("sample_rate_hz") or 16000)
# Vision-follow commits often carry empty or near-silent PCM with
# is_speech=False. All-zero audio crashes Qwen3ASRProcessor, and an
# empty Stage0 prompt is rejected ("decoder prompt cannot be empty").
# Also guard is_speech=True + empty-audio + frames (prompt was "").
sample_rate_hz = int(sample_rate_hz or payload.get("sample_rate_hz") or 16000)
wav_rms = float(np.sqrt(np.mean(np.square(wav.astype(np.float32, copy=False))))) if wav.size else 0.0
near_silent = wav.size == 0 or not np.any(wav) or wav_rms < 1e-3
if mm and (not has_audio or (not is_speech and near_silent)):
n = max(1600, int(sample_rate_hz) // 10)
wav = np.linspace(1e-4, -1e-4, n, dtype=np.float32)
has_audio = True
if not has_audio and not mm:
raise ValueError("AURA duplex commit requires audio and/or video_frames")
system_prompt = runtime_config.get("aura_system_prompt", DEFAULT_AURA_SYSTEM_PROMPT)
additional_information: dict[str, object] = {
"aura_system_prompt": system_prompt,
"aura_session_id": fence.session_id,
"aura_duplex": True,
"session_id": fence.session_id,
"epoch": fence.epoch,
"turn_id": fence.turn_id,
# asr2aura ignores Stage0 text when False (placeholder zeros).
"is_speech": is_speech,
"tts_task_type": runtime_config.get("tts_task_type", "CustomVoice"),
"tts_language": runtime_config.get("tts_language", "Chinese"),
"tts_speaker": runtime_config.get("tts_speaker", "Vivian"),
}
item_id = payload.get("realtime_item_id")
if isinstance(item_id, str) and item_id:
additional_information["realtime_item_id"] = item_id
for key in _TTS_EXTRA_KEYS:
if key in additional_information:
continue
value = runtime_config.get(key)
if value is not None:
additional_information[key] = value
if mm:
additional_information["deferred_multi_modal_data"] = mm
asr_prompt = (
"<|im_start|>user\n"
"<|audio_start|><|audio_pad|><|audio_end|>"
"Please transcribe this speech.<|im_end|>\n"
"<|im_start|>assistant\n"
)
multi_modal_data: dict[str, object] = {}
if has_audio:
multi_modal_data["audio"] = (wav, sample_rate_hz)
# Never submit an empty decoder prompt: Stage0 always needs the ASR pad
# when audio (or a vision-driven pad) is present.
prompt: dict[str, Any] = {
"prompt": asr_prompt if has_audio else "",
"multi_modal_data": multi_modal_data,
"additional_information": additional_information,
}
return DuplexAppendPlan(prompt=prompt)
def project_intermediate_output(
self,
*,
stage_id: int,
output: object,
context: object,
) -> bool:
"""Project Stage1 thinker text to the client without short-circuiting TTS."""
del output, context
return stage_id == 1
def user_transcript(
self,
*,
stage_id: int,
output: object,
prompt: object,
finished: bool,
) -> str | None:
"""Stage0 ASR text for a spoken turn, so the demo can show what was said.
Vision-follow sets ``is_speech`` false; that audio is a silent pad and
must not open a user bubble. The pipeline still forwards Stage0.
"""
if stage_id != 0 or not finished:
return None
info = prompt.get("additional_information") if isinstance(prompt, dict) else None
if isinstance(info, dict) and info.get("is_speech") is False:
return None
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
_extract_text,
_normalize_asr_transcript,
is_effectively_silent,
)
text = _normalize_asr_transcript(_extract_text(output))
if not text or is_effectively_silent(text):
return None
return text
def plan_partial_stage_output(
self,
orchestrator: Any,
stage_id: int,
replica_id: int,
output: Any,
req_state: Any,
):
from vllm_omni.model_executor.models.aura_omni.duplex.sentence_tts import (
plan_partial_stage_output,
)
return plan_partial_stage_output(orchestrator, stage_id, replica_id, output, req_state)
def partial_stage_followup(self, plan: Any, req_state: Any) -> Any:
"""Close the Talker stream after a resumable final sentence.
The sentence text was already forwarded with ``queue_close_after``.
Flipping the prompt flag here makes the second submit a close-only
sentinel instead of another copy of that sentence.
"""
if plan is None or not getattr(plan, "queue_close_after", False):
return None
prompt = getattr(req_state, "prompt", None)
info = prompt.get("additional_information") if isinstance(prompt, dict) else None
if isinstance(info, dict):
info["aura_tts_partial"] = True
info["aura_tts_close_only"] = True
from vllm_omni.engine.duplex.plugin import PartialStageForward
from vllm_omni.model_executor.models.aura_omni.duplex.sentence_tts import (
SentenceTtsOutput,
)
request_id = str(getattr(plan.output, "request_id", ""))
return PartialStageForward(
output=SentenceTtsOutput(request_id, ""),
is_final_update=True,
close_only=True,
)
def commit_model_context(self, *, session_id: str | None, assistant_text: str) -> None:
if not isinstance(session_id, str) or not session_id:
return
from vllm_omni.model_executor.models.aura_omni.duplex.history import get_or_create_session_history
get_or_create_session_history(session_id).commit_turn(assistant_text or SILENT_TEXT)
def release_concurrent_turn_requests(
self,
*,
stage_id: int,
segment_finished: bool,
output: object,
context: object,
) -> bool:
"""After Stage1 text/silent final, next commit may start while TTS drains."""
del context
if stage_id != 1:
return False
if segment_finished:
return True
return bool(getattr(output, "finished", False))
def decide_output(
self,
*,
stage_id: int,
final_stage_id: int,
segment_finished: bool,
segment_token_ids: tuple[int, ...],
segment_output_metadata: dict[str, object],
output: object,
) -> DuplexOutputDecision | None:
del final_stage_id, segment_output_metadata
# Stage1 silent → short-circuit TTS/Code2Wav.
if stage_id != 1 or not segment_finished:
return None
completion = _first_completion(output)
token_ids = _completion_token_ids(completion) or list(segment_token_ids)
text = getattr(completion, "text", None) if completion is not None else None
cumulative = getattr(completion, "cumulative_text", None) if completion is not None else None
text_blob = " ".join(part for part in (text, cumulative) if isinstance(part, str))
# Empty finished Stage1 is silent: stop_token 151669 is often omitted from
# token_ids/text unless include_stop_str_in_output is set.
is_silent = (
any(sid in token_ids for sid in AURA_SILENT_TOKEN_IDS)
or (isinstance(text, str) and is_effectively_silent(text))
or (isinstance(cumulative, str) and is_effectively_silent(cumulative))
or SILENT_TEXT in text_blob
)
if not is_silent and not token_ids and not text_blob.strip():
is_silent = True
if not is_silent and token_ids:
is_silent = token_ids[0] in AURA_SILENT_TOKEN_IDS or (
len(token_ids) > 3 and any(sid in token_ids[:6] for sid in AURA_SILENT_TOKEN_IDS)
)
if not is_silent:
return None
return DuplexOutputDecision(
action=DuplexOutputAction.DIRECT_RESPONSE,
metadata={
"duplex_direct_response": True,
"model_listen": True,
"listen_source": "aura_silent",
"silent_text": SILENT_TEXT,
},
)
def create_session_state(self) -> AuraServingSessionState:
return AuraServingSessionState()
def capabilities(self, *, max_sessions: int) -> DuplexCapabilities:
return aura_duplex_capabilities(max_sessions=max_sessions)
def validate_client_extra_body(self, extra_body: object) -> None:
if extra_body is None:
return
if not isinstance(extra_body, dict):
raise ValueError("AURA duplex extra_body must be an object")
async def prepare_runtime_config(
self, config: DuplexSessionConfig, *, model_config: object | None
) -> dict[str, object]:
del model_config
extra = config.extra_body if isinstance(config.extra_body, dict) else {}
system_prompt = extra.get("aura_system_prompt") or config.instructions or DEFAULT_AURA_SYSTEM_PROMPT
runtime: dict[str, object] = {
"aura_system_prompt": str(system_prompt),
"instructions": str(system_prompt),
"tts_task_type": str(extra.get("tts_task_type") or "CustomVoice"),
"tts_language": str(extra.get("tts_language") or "Chinese"),
"tts_speaker": str(extra.get("tts_speaker") or "Vivian"),
}
for key in _TTS_EXTRA_KEYS:
if key in runtime:
continue
if key in extra and extra[key] is not None:
runtime[key] = extra[key]
return runtime
def runtime_config_for_update(
self,
config: DuplexSessionConfig,
current: Mapping[str, object],
) -> dict[str, object]:
updated = dict(current)
extra = config.extra_body if isinstance(config.extra_body, dict) else {}
# Same precedence as prepare_runtime_config: explicit extra, then
# instructions, then the prompt already installed (or the default).
# setdefault would keep the creation-time prompt and ignore a later
# session.update.instructions.
explicit = extra.get("aura_system_prompt") if "aura_system_prompt" in extra else None
if isinstance(explicit, str) and explicit:
prompt = explicit
elif config.instructions:
prompt = str(config.instructions)
else:
current_prompt = current.get("aura_system_prompt")
prompt = (
str(current_prompt)
if isinstance(current_prompt, str) and current_prompt
else DEFAULT_AURA_SYSTEM_PROMPT
)
updated["aura_system_prompt"] = prompt
updated["instructions"] = str(config.instructions) if config.instructions else prompt
for key in _TTS_EXTRA_KEYS:
if key in extra and extra[key] is not None:
updated[key] = extra[key]
return updated
def data_plane_context(
self,
*,
epoch: int,
turn_id: int,
active_response_turn_id: int | None,
active_response_id: str | None,
auto_responds: bool,
response_format: str,
speed: float | None,
modalities: tuple[str, ...],
) -> AuraDataPlaneContext:
del active_response_turn_id, active_response_id
return AuraDataPlaneContext(
epoch=epoch,
turn_id=turn_id,
auto_responds=auto_responds,
response_format=response_format,
speed=speed,
modalities=modalities,
)
__all__ = ["AURA_SILENT_TOKEN_ID", "AuraDuplexPlugin"]
vllm_omni/model_executor/models/aura_omni/duplex/sentence_tts.py
+189
View File
@@ -0,0 +1,189 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Stage1 sentence handoff onto Talker. AURA-only; the orchestrator just calls it."""
from __future__ import annotations
from typing import Any
from vllm.logger import init_logger
from vllm_omni.engine.duplex.plugin import PartialStageForward
logger = init_logger(__name__)
class SentenceTtsOutput:
"""Stage1 view that exposes only the sentence Talker should speak."""
def __init__(self, request_id: str, text: str) -> None:
self.request_id = request_id
self.finished = False
self.text = text
self.cumulative_text = text
self.outputs = [self]
self.token_ids: list[int] = []
def _call_decode(decode: Any, token_ids: list[int]) -> str:
try:
decoded = decode(token_ids, skip_special_tokens=False)
except TypeError:
decoded = decode(token_ids)
return decoded if isinstance(decoded, str) else ""
def decode_growing_token_ids(decode: Any, token_ids: list[int], cache: dict[str, object]) -> str:
"""Decode a growing id list, re-decoding only the unstable tail.
The returned text matches ``decode(token_ids)`` when the tail window is a
suffix of that string. Otherwise the full id list is decoded.
"""
ids = list(token_ids)
prev_ids = cache.get("decoded_ids")
prev_text = cache.get("decoded_text")
overlap_text = cache.get("decoded_overlap_text")
candidate: str | None = None
if (
isinstance(prev_ids, list)
and isinstance(prev_text, str)
and isinstance(overlap_text, str)
and prev_ids
and len(ids) > len(prev_ids)
and ids[: len(prev_ids)] == prev_ids
and prev_text.endswith(overlap_text)
and overlap_text
):
extended = _call_decode(decode, [*prev_ids[-1:], *ids[len(prev_ids) :]])
if extended.startswith(overlap_text):
candidate = prev_text[: len(prev_text) - len(overlap_text)] + extended
window_ids = [*prev_ids[-2:], *ids[len(prev_ids) :]] if len(prev_ids) >= 2 else ids
window_text = _call_decode(decode, window_ids)
if window_text and not candidate.endswith(window_text):
candidate = None
if candidate is None:
candidate = _call_decode(decode, ids)
cache["decoded_ids"] = ids
cache["decoded_text"] = candidate
cache["decoded_overlap_text"] = _call_decode(decode, ids[-1:]) if ids else ""
return candidate
def stage1_tts_text(orchestrator: Any, output: Any, *, cache: dict[str, object] | None = None) -> str:
"""Stage1 text for sentence TTS.
``cumulative_text`` is attached only when the request finishes.
Mid-generation chunks still carry ``cumulative_token_ids``.
"""
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
_extract_output,
_extract_text,
)
completion = _extract_output(output)
cumulative = getattr(completion, "cumulative_text", None)
if isinstance(cumulative, str) and cumulative:
return cumulative
token_ids = getattr(completion, "cumulative_token_ids", None)
if isinstance(token_ids, list) and token_ids:
processor = orchestrator.stage_pools[1].output_processor
tokenizer = getattr(processor, "tokenizer", None)
decode = getattr(tokenizer, "decode", None)
if callable(decode):
if cache is None:
decoded = _call_decode(decode, list(token_ids))
else:
decoded = decode_growing_token_ids(decode, list(token_ids), cache)
if decoded:
return decoded
return _extract_text(output)
def plan_partial_stage_output(
orchestrator: Any,
stage_id: int,
replica_id: int,
output: Any,
req_state: Any,
) -> PartialStageForward | None:
"""Hand a finished sentence to Talker before Stage1 finishes.
Same request id as the turn (Code2Wav was prewarmed on it). Text goes
through ``aura2tts`` in the orchestrator, not the ``from_stage_1`` SHM
edge that clears ``additional_information.text``. Talker stays resumable
until Stage1 finishes. The orchestrator submits the returned plan.
"""
del replica_id
if not req_state.session_owned or stage_id != 1:
return
if stage_id + 1 > req_state.final_stage_id:
return
next_client = orchestrator.stage_pools[stage_id + 1].stage_client
processor = getattr(next_client, "custom_process_input_func", None)
if getattr(processor, "__name__", "") != "aura2tts":
return
if orchestrator._stage_receives_async_chunks(stage_id + 1):
return
from vllm_omni.model_executor.stage_input_processors.aura_omni import (
_sentence_tts_enabled,
next_duplex_sentence_chunk,
)
if not _sentence_tts_enabled():
return
bridge = req_state.streaming.bridge_states.setdefault("aura_sentence_tts", {})
if not isinstance(bridge, dict):
bridge = {}
req_state.streaming.bridge_states["aura_sentence_tts"] = bridge
if bridge.get("closed"):
return
finished = bool(getattr(output, "finished", False))
raw_text = stage1_tts_text(orchestrator, output, cache=bridge)
# Captured before the chunk helper increments ``emits``.
prior_emits = int(bridge.get("emits", 0))
chunk = next_duplex_sentence_chunk(bridge, raw_text, finished=finished)
close_only = False
if chunk is None:
if not (finished and int(bridge.get("emits", 0))):
return
chunk = ""
close_only = True
if not chunk and not close_only:
return
prompt = req_state.prompt if isinstance(req_state.prompt, dict) else None
if prompt is not None:
raw_info = prompt.get("additional_information")
if not isinstance(raw_info, dict):
raw_info = {}
prompt["additional_information"] = raw_info
# History is committed by the session runner from model_context_text.
# Partial and close-only updates must not commit inside aura2tts.
raw_info["aura_tts_partial"] = True
raw_info["aura_tts_close_only"] = close_only
view = SentenceTtsOutput(str(getattr(output, "request_id", req_state.request_id)), chunk)
# A later sentence on an already-running Talker must stay resumable.
# is_final_update=True is a non-resumable end sentinel: it discards this
# text and sets streaming_input=False on the in-flight sentence, so
# Code2Wav sees a short codec and the user hears a blip.
queue_close_after = bool(chunk) and finished and not close_only and prior_emits > 0
logger.info(
"[AURA] sentence TTS req=%s finished=%s close_only=%s text_len=%d queue_close_after=%s",
req_state.request_id,
finished,
close_only,
len(chunk),
queue_close_after,
)
if finished:
bridge["closed"] = True
return PartialStageForward(
output=view,
is_final_update=finished and not queue_close_after,
close_only=close_only,
queue_close_after=queue_close_after,
)
vllm_omni/model_executor/models/aura_omni/duplex/session.py
+58
View File
@@ -0,0 +1,58 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import asyncio
from dataclasses import dataclass, field
from vllm_omni.engine.duplex.plugin import DuplexModelSessionState
from vllm_omni.model_executor.models.aura_omni.duplex.input import AuraPcmAppendBuffer
@dataclass(slots=True)
class AuraServingSessionState(DuplexModelSessionState):
"""Per-session AURA duplex state (owned by SessionRunner)."""
audio_buffer: AuraPcmAppendBuffer = field(default_factory=AuraPcmAppendBuffer)
input_since_commit: bool = False
speech_since_commit: bool = False
context_locked: bool = False
committed_audio_payload: dict[str, object] | None = None
committed_audio_operation_id: str | None = None
committed_audio_reserved_bytes: int = 0
deferred_response_create: bool = False
deferred_precreate_response: bool = False
continuation_owner_id: str | None = None
continuation_units: int = 0
pending_silence_task: asyncio.Task[bool] | None = None
pending_silence_owner_id: str | None = None
def retain_committed_audio(
self,
payload: dict[str, object],
*,
operation_id: str | None,
reserved_bytes: int = 0,
) -> None:
self.committed_audio_payload = payload
self.committed_audio_operation_id = operation_id
self.committed_audio_reserved_bytes += max(0, int(reserved_bytes))
def clear_committed_audio(self) -> int:
reserved_bytes = self.committed_audio_reserved_bytes
self.committed_audio_payload = None
self.committed_audio_operation_id = None
self.committed_audio_reserved_bytes = 0
self.deferred_response_create = False
self.deferred_precreate_response = False
return reserved_bytes
def clear_continuation(self) -> None:
self.continuation_owner_id = None
self.continuation_units = 0
self.pending_silence_task = None
self.pending_silence_owner_id = None
__all__ = ["AuraServingSessionState"]
vllm_omni/model_executor/models/aura_omni/pipeline.py
+1
View File
@@ -26,6 +26,7 @@ AURA_OMNI_PIPELINE = PipelineConfig(
model_type="aura_omni",
default_deploy_config_name="aura_omni.yaml",
model_arch="Qwen3ASRForConditionalGeneration",
duplex_plugin="vllm_omni.model_executor.models.aura_omni.duplex.plugin.AuraDuplexPlugin",
stages=(
StagePipelineConfig(
stage_id=0,
vllm_omni/model_executor/models/minicpmo_4_5/duplex/capabilities.py
+3
View File
@@ -28,6 +28,9 @@ def minicpmo45_native_capabilities(*, max_sessions: int = 1) -> DuplexCapabiliti
supports_scheduler_native_append=False,
supports_core_resumable_request=True,
supports_stage_connector_handoff=True,
# Audio-primary: frames may ride an audio unit; video-only is illegal.
required_input_modalities=frozenset({"audio"}),
optional_input_modalities=frozenset({"video"}),
supports_independent_io_streams=True,
supports_realtime_endpoint=True,
supports_multi_session=supports_multi_session,
vllm_omni/model_executor/models/minicpmo_4_5/duplex/input.py
+12 -18
View File
@@ -10,6 +10,10 @@ from dataclasses import dataclass
import numpy as np
from numpy.typing import NDArray
from vllm_omni.engine.duplex.pcm_reservation import (
commit_ordered_reservation,
rollback_ordered_reservation,
)
from vllm_omni.engine.duplex.plugin import PcmAppendBuffer, PcmAppendReservation
@@ -389,33 +393,23 @@ class MiniCPMO45PcmAppendBuffer(PcmAppendBuffer):
return reservation.payload
def _commit_reservation(self, reservation: MiniCPMO45PcmAppendReservation) -> None:
if not reservation._active:
return
if not self._reservations or self._reservations[0] is not reservation:
raise RuntimeError("PCM append reservations must commit in wire order")
self._reservations.pop(0)
reservation._active = False
commit_ordered_reservation(self._reservations, reservation, head_only=True)
def _rollback_reservation(self, reservation: MiniCPMO45PcmAppendReservation) -> None:
if not reservation._active:
rolled_back = rollback_ordered_reservation(
self._reservations,
reservation,
self._buffer,
active_only=False,
)
if not rolled_back:
return
try:
index = self._reservations.index(reservation)
except ValueError:
reservation._active = False
return
rolled_back = self._reservations[index:]
restored = b"".join(item._raw for item in rolled_back)
self._buffer[:0] = restored
self._prepend_spans([span for item in rolled_back for span in item._spans])
restored_groups = [list(item._video_frames) for item in rolled_back if item._video_frames]
if restored_groups:
self._frame_queue[:0] = restored_groups
self._sample_rate_hz = self._sample_rate_hz or reservation._sample_rate_hz
self._turn_had_speech = self._turn_had_speech or any(item._turn_had_speech for item in rolled_back)
for item in rolled_back:
item._active = False
del self._reservations[index:]
def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None:
if not self._buffer:
vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py
+12 -3
View File
@@ -1151,9 +1151,18 @@ class Qwen3TTSTalkerForConditionalGeneration(nn.Module):
next_text_offset = text_offset
last_hidden = hs.get("last")
if not isinstance(last_hidden, torch.Tensor):
raise RuntimeError("Missing hidden_states['last'] in additional_information; postprocess must run.")
past_hidden_list.append(last_hidden.to(device=device, dtype=dtype).reshape(1, -1))
if isinstance(last_hidden, torch.Tensor):
past_hidden = last_hidden.to(device=device, dtype=dtype).reshape(1, -1)
else:
# Match scalar preprocess(): EOS / async-scheduling races can
# arrive before postprocess has written last; zeros are filtered
# downstream the same way as the single-request path.
logger.warning_once(
"Qwen3-TTS batched decode missing last hidden; zero-filling past_hidden "
"to match scalar preprocess()"
)
past_hidden = torch.zeros_like(text_step)
past_hidden_list.append(past_hidden)
text_step_list.append(text_step)
info_update: dict[str, Any] = {
vllm_omni/model_executor/stage_input_processors/aura_omni.py
+492 -23
View File
@@ -1,15 +1,19 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Stage processors for the AURA Omni pipeline."""
from __future__ import annotations
import json
import math
import os
from collections.abc import Mapping
from pathlib import Path
from typing import Any
import regex as re
import soundfile as sf
from vllm.logger import init_logger
from vllm_omni.inputs.data import OmniTokensPrompt
from vllm_omni.model_executor.models.qwen3_tts.prompt_embeds_builder import (
@@ -19,7 +23,7 @@ from vllm_omni.model_executor.models.qwen3_tts.prompt_embeds_builder import (
DEFAULT_AURA_SYSTEM_PROMPT = (
"You are receiving a live video stream where the final frame is the present moment. "
"Respond only when a response is needed based on the user's message or the visual context. "
"Otherwise, output '<|silent|>' to signify silence. Respond in Chinese."
"Otherwise, output `<|silent|>` to signify silence."
)
SILENT_TEXT = "<|silent|>"
@@ -39,6 +43,13 @@ DEFAULT_QWEN3_TTS_REF_AUDIO = "vllm-omni/tests/assets/qwen3_tts/clone_2.wav"
DEFAULT_QWEN3_TTS_REF_TEXT = (
"Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."
)
# Used only to size Talker prompt_token_ids placeholders (must match build_prompt_embeds).
DEFAULT_QWEN3_TTS_TOKENIZER = "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice"
logger = init_logger(__name__)
# Lazy cache: (path, tokenizer, codec_language_id, spk_is_dialect)
_qwen3_tts_prompt_len_cache: dict[str, Any] | None = None
def default_qwen3_tts_ref_audio_path() -> str:
@@ -72,6 +83,115 @@ def _first_bool(value: Any, default: bool = False) -> bool:
return bool(value)
def _resolve_qwen3_tts_tokenizer_path(additional_info: dict[str, Any] | None = None) -> str:
"""Resolve Qwen3-TTS tokenizer / config path for prompt_len parity."""
if additional_info:
for key in ("tts_tokenizer", "tts_model", "qwen3_tts_model"):
raw = _first_value(additional_info.get(key), None)
if isinstance(raw, str) and raw.strip():
return raw.strip()
for env_key in ("VLLM_AURA_TTS_TOKENIZER", "VLLM_AURA_TTS_MODEL"):
env = os.environ.get(env_key, "").strip()
if env:
return env
for candidate in (
"/workspace/models/Qwen3-TTS-12Hz-1.7B-CustomVoice",
"/workspace/models/hub/models--Qwen--Qwen3-TTS-12Hz-1.7B-CustomVoice/snapshots/0c0e3051f131929182e2c023b9537f8b1c68adfe",
str(Path.home() / ".cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-1.7B-CustomVoice"),
):
path = Path(candidate)
if path.is_dir() and (path / "tokenizer_config.json").is_file():
return candidate
snaps = path / "snapshots"
if snaps.is_dir():
for snap in sorted(snaps.iterdir()):
if (snap / "tokenizer_config.json").is_file():
return str(snap)
return DEFAULT_QWEN3_TTS_TOKENIZER
def _load_qwen3_tts_prompt_len_tools(
additional_info: dict[str, Any] | None = None,
) -> tuple[Any, Mapping[str, int] | None, Mapping[str, object] | None] | None:
"""Load tokenizer + talker dialect maps for official prompt_len estimate."""
global _qwen3_tts_prompt_len_cache
path = _resolve_qwen3_tts_tokenizer_path(additional_info)
if (
isinstance(_qwen3_tts_prompt_len_cache, dict)
and _qwen3_tts_prompt_len_cache.get("path") == path
and _qwen3_tts_prompt_len_cache.get("tokenizer") is not None
):
return (
_qwen3_tts_prompt_len_cache["tokenizer"],
_qwen3_tts_prompt_len_cache.get("codec_language_id"),
_qwen3_tts_prompt_len_cache.get("spk_is_dialect"),
)
try:
from transformers import AutoConfig, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
path,
trust_remote_code=True,
padding_side="left",
)
codec_language_id = None
spk_is_dialect = None
try:
hf_config = AutoConfig.from_pretrained(path, trust_remote_code=True)
talker_config = getattr(hf_config, "talker_config", None) or hf_config
codec_language_id = getattr(talker_config, "codec_language_id", None)
spk_is_dialect = getattr(talker_config, "spk_is_dialect", None)
except Exception as cfg_err:
cfg_path = Path(path) / "config.json"
if cfg_path.is_file():
raw = json.loads(cfg_path.read_text(encoding="utf-8"))
talker = raw.get("talker_config") if isinstance(raw.get("talker_config"), dict) else raw
codec_language_id = talker.get("codec_language_id")
spk_is_dialect = talker.get("spk_is_dialect")
else:
logger.warning("Qwen3-TTS talker_config unavailable for prompt_len (%s): %s", path, cfg_err)
_qwen3_tts_prompt_len_cache = {
"path": path,
"tokenizer": tokenizer,
"codec_language_id": codec_language_id,
"spk_is_dialect": spk_is_dialect,
}
return tokenizer, codec_language_id, spk_is_dialect
except Exception as e:
logger.warning("Failed to load Qwen3-TTS tokenizer for prompt_len (%s): %s", path, e)
return None
def _estimate_tts_prompt_len_official(
tts_info: dict[str, Any],
*,
task_type: str,
additional_info: dict[str, Any] | None = None,
) -> int | None:
"""Match standalone speech API: real BPE + estimate_prompt_len_from_additional_information."""
tools = _load_qwen3_tts_prompt_len_tools(additional_info)
if tools is None:
return None
tokenizer, codec_language_id, spk_is_dialect = tools
try:
from vllm_omni.model_executor.models.qwen3_tts.prompt_embeds_builder import (
Qwen3TTSPromptEmbedsBuilder,
)
return int(
Qwen3TTSPromptEmbedsBuilder.estimate_prompt_len_from_additional_information(
additional_information=tts_info,
task_type=task_type,
tokenize_prompt=lambda t: tokenizer(t, padding=False)["input_ids"],
codec_language_id=codec_language_id if isinstance(codec_language_id, dict) else None,
spk_is_dialect=spk_is_dialect if isinstance(spk_is_dialect, dict) else None,
)
)
except Exception as e:
logger.warning("Official Qwen3-TTS prompt_len estimate failed; falling back to heuristic: %s", e)
return None
def _normalize_qwen3_tts_speaker(speaker: Any) -> Any:
if not isinstance(speaker, str):
return speaker
@@ -143,6 +263,13 @@ def _source_prompt_by_request_id(source_outputs: list[Any], prompt: Any) -> dict
}
def _one_video_item(video: Any) -> Any:
"""The current clip, if ``multi_modal_data['video']`` is one ``(array, meta)``."""
if isinstance(video, list) and len(video) == 1 and isinstance(video[0], tuple):
return video[0]
return None
def _vision_placeholder(multi_modal_data: dict[str, Any]) -> str:
if "video" in multi_modal_data:
return "<|vision_start|><|video_pad|><|vision_end|>"
@@ -155,17 +282,217 @@ def _vision_multimodal_data(multi_modal_data: dict[str, Any]) -> dict[str, Any]:
return {key: value for key, value in multi_modal_data.items() if key in {"image", "video"}}
def _aura_prompt(system_prompt: str, transcript: str, multi_modal_data: dict[str, Any]) -> str:
def _aura_prompt(
system_prompt: str,
transcript: str,
multi_modal_data: dict[str, Any],
history_prefix: str = "",
) -> str:
vision = _vision_placeholder(multi_modal_data)
query = transcript.strip()
user_body = f"{vision}{query}" if query else vision
return (
f"<|im_start|>system\n{system_prompt}<|im_end|>\n"
f"{history_prefix}"
f"<|im_start|>user\n{user_body}<|im_end|>\n"
"<|im_start|>assistant\n"
)
def _strip_assistant_text(text: str) -> str:
"""Remove think wrappers and ChatML specials before TTS / client text.
Stage1 uses ``skip_special_tokens=False`` so ``<|silent|>`` / ``<|im_end|>``
remain visible for silent detection; strip ChatML markers (not silent)
before spoken TTS / transcript emission.
"""
cleaned = re.sub(r"<think>.*?</think>", "", text or "", flags=re.DOTALL | re.IGNORECASE)
# Drop an unclosed leading think block if the model is still inside it.
cleaned = re.sub(r"<think>.*$", "", cleaned, flags=re.DOTALL | re.IGNORECASE)
cleaned = cleaned.replace("</think>", "")
cleaned = re.sub(r"<\|im_end\|>|<\|im_start\|>|<\|endoftext\|>", "", cleaned)
return cleaned.strip()
def is_effectively_silent(text: str | None) -> bool:
"""True for empty / whitespace-only / exact ``<|silent|>`` Stage1 text."""
if not isinstance(text, str):
return False
stripped = _strip_assistant_text(text)
return not stripped or stripped == SILENT_TEXT
# Native AURA sentence boundaries for incremental Stage1→TTS handoff.
# Same rule as AURA_026 ``aura2tts_async_chunk``. Duplex applies it in the
# orchestrator (not SHM ``from_stage_1``, which clears Talker text).
_NATIVE_TTS_SENT_ENDS = frozenset("。！？；.!?;\n")
_NATIVE_TTS_COMMA_ENDS = frozenset("，,")
_NATIVE_TTS_MIN_CHARS = 10
_NATIVE_TTS_MIN_EMIT_CHARS = 30
def _sentence_tts_enabled() -> bool:
"""Emit TTS per sentence while Stage1 still generates.
Default on. Disable with ``VLLM_AURA_SENTENCE_TTS=0``.
"""
raw = (os.environ.get("VLLM_AURA_SENTENCE_TTS") or "1").strip().lower()
return raw not in {"0", "false", "off", "no"}
def _sentence_tts_min_emit_chars() -> int:
"""Minimum content chars before a mid-generation TTS handoff."""
raw = (os.environ.get("VLLM_AURA_SENTENCE_TTS_MIN_CHARS") or "").strip()
if not raw:
return _NATIVE_TTS_MIN_EMIT_CHARS
try:
return max(1, min(200, int(raw)))
except ValueError:
return _NATIVE_TTS_MIN_EMIT_CHARS
def _tts_content_char_count(text: str) -> int:
"""Count alphanumeric / CJK content chars (ignore punctuation/whitespace)."""
return sum(1 for ch in text if ch.isalnum() or "\u4e00" <= ch <= "\u9fff")
def _pop_native_tts_sentence(buf: str) -> tuple[str | None, str]:
"""Pop one sentence from ``buf``; return (sentence_or_None, rest)."""
if not buf:
return None, buf
split_pos = -1
for i, ch in enumerate(buf):
if ch in _NATIVE_TTS_SENT_ENDS:
split_pos = i + 1
break
if ch in _NATIVE_TTS_COMMA_ENDS and i + 1 >= _NATIVE_TTS_MIN_CHARS:
split_pos = i + 1
break
if split_pos < 0:
return None, buf
sentence = buf[:split_pos]
rest = buf[split_pos:]
if not sentence.strip():
return _pop_native_tts_sentence(rest)
return sentence, rest
def _pop_emit_ready_tts_text(
buf: str,
min_chars: int | None = None,
) -> tuple[str | None, str]:
"""Pop sentences until content length >= ``min_chars``."""
if min_chars is None:
min_chars = _sentence_tts_min_emit_chars()
parts: list[str] = []
rest = buf
while True:
sentence, rest = _pop_native_tts_sentence(rest)
if sentence is None:
break
parts.append(sentence)
if _tts_content_char_count("".join(parts)) >= min_chars:
return "".join(parts), rest
if not parts:
return None, buf
return None, "".join(parts) + rest
def _tool_marker_pending(text: str) -> bool:
lowered = text.lower()
return "<tool_call" in lowered or "</tool_call" in lowered
def next_duplex_sentence_chunk(state: dict[str, Any], raw_text: str, *, finished: bool) -> str | None:
"""Return the next Talker sentence, or None when this output must not start TTS.
``state`` is per Stage1 request and is mutated. Stage1 finish flushes any
leftover even under the min-char floor. Silent, an unclosed ``<think>``,
and tool markers do not emit mid-generation. A silent finish returns None
so the legacy full-text path can drop TTS.
"""
if not _sentence_tts_enabled():
return None
raw = raw_text or ""
already = int(state.get("emits", 0))
if already == 0 and (is_silent_text_prefix(raw) or is_effectively_silent(raw)):
return None
think_open = "<think>" in raw.lower() and "</think>" not in raw.lower()
tool_pending = _tool_marker_pending(raw)
if (think_open or tool_pending) and not finished:
return None
text = _strip_assistant_text(raw)
if is_effectively_silent(text) and already == 0:
return None
emitted_prefix = str(state.get("emitted_prefix", ""))
pending = str(state.get("pending", ""))
if text.startswith(emitted_prefix):
new_tail = text[len(emitted_prefix) :]
else:
new_tail = text
pending = ""
emitted_prefix = ""
if new_tail:
pending = pending + new_tail
emitted_prefix = text
state["emitted_prefix"] = emitted_prefix
if not finished:
sentence, pending = _pop_emit_ready_tts_text(pending)
state["pending"] = pending
if sentence is None:
return None
state["emits"] = already + 1
state["last"] = sentence.strip()
return state["last"]
state["pending"] = ""
remainder = _strip_assistant_text(pending).strip()
if not remainder or is_effectively_silent(remainder):
return None
state["emits"] = already + 1
state["last"] = remainder
return remainder
def is_silent_text_prefix(text: str | None) -> bool:
"""True while streamed text is still a prefix of ``<|silent|>``.
Holds Stage1 transcript deltas until the turn finishes (or diverges), so
partial ``<|sil`` fragments do not leak before a silent short-circuit.
"""
if not isinstance(text, str):
return False
stripped = text.strip()
if not stripped:
return True
return SILENT_TEXT.startswith(stripped)
def _normalize_asr_transcript(transcript: str) -> str:
"""Strip Qwen3-ASR markup wrappers so AURA sees plain user text.
Observed Stage0 text looks like:
``language Chinese<asr_text>出现《古韵》这本书的时候，提醒我。``
"""
text = (transcript or "").strip()
if not text:
return ""
marker = "<asr_text>"
if marker in text:
text = text.split(marker, 1)[1].strip()
# Drop a leading ``language <lang>`` line if still present.
if text.lower().startswith("language "):
parts = text.split(None, 2)
if len(parts) >= 3:
text = parts[2].strip()
elif len(parts) == 2:
text = ""
return text
def asr2aura(
source_outputs: list[Any],
prompt: Any = None,
@@ -178,7 +505,10 @@ def asr2aura(
src_prompt = prompt_by_request_id.get(str(getattr(source_output, "request_id", idx)), {})
additional_info = src_prompt.get("additional_information") or {}
system_prompt = _first_value(additional_info.get("aura_system_prompt"), DEFAULT_AURA_SYSTEM_PROMPT)
transcript = _extract_text(source_output)
transcript = _normalize_asr_transcript(_extract_text(source_output))
# Vision-follow: placeholder zeros may ASR into noise; trust client is_speech.
if additional_info.get("is_speech") is False:
transcript = ""
multi_modal_data = {}
source_multi_modal_data = src_prompt.get("multi_modal_data") or {}
if isinstance(source_multi_modal_data, dict):
@@ -188,9 +518,67 @@ def asr2aura(
multi_modal_data.update(deferred_multi_modal_data)
multi_modal_data = _vision_multimodal_data(multi_modal_data)
history_prefix = ""
prompt_mm = multi_modal_data
session_id = additional_info.get("session_id") or additional_info.get("aura_session_id")
if additional_info.get("aura_duplex") and isinstance(session_id, str) and session_id:
from vllm_omni.model_executor.models.aura_omni.duplex.history import (
get_or_create_session_history,
)
history = get_or_create_session_history(session_id)
current_video = _one_video_item(multi_modal_data.get("video"))
history.begin_user_turn(transcript, video=current_video)
history_prefix = history.render_prefix()
prior_videos = history.retained_videos()
# Native get_vllm_inputs: every retained clip is a <|video_pad|> in
# order, then this turn. The placeholder on the current user message
# is only this clip, so pad count matches multi_modal_data["video"].
if prior_videos or current_video is not None:
merged = list(prior_videos)
if current_video is not None:
merged.append(current_video)
multi_modal_data = dict(multi_modal_data)
multi_modal_data["video"] = merged
prompt_mm = dict(multi_modal_data)
if prior_videos:
if current_video is None:
prompt_mm.pop("video", None)
else:
prompt_mm["video"] = [current_video]
else:
prompt_mm = multi_modal_data
if not transcript:
last_assistant = next(
(
message["content"]
for message in reversed(history.messages)
if message.get("role") == "assistant" and isinstance(message.get("content"), str)
),
"",
)
logger.info(
"[asr2aura] vision-follow history_prefix_len=%d has_assistant=%s last_assistant_len=%d",
len(history_prefix),
bool(last_assistant),
len(last_assistant) if isinstance(last_assistant, str) else 0,
)
next_input: dict[str, Any] = {
"prompt": _aura_prompt(str(system_prompt), transcript, multi_modal_data),
"prompt": _aura_prompt(
str(system_prompt),
transcript,
prompt_mm,
history_prefix=history_prefix,
),
}
if isinstance(session_id, str) and session_id:
next_input.setdefault("additional_information", {})
# Preserve session id for aura2tts history commit.
info = dict(additional_info)
info["session_id"] = session_id
info["aura_duplex"] = bool(additional_info.get("aura_duplex"))
next_input["additional_information"] = info
if requires_multimodal_data:
next_input["multi_modal_data"] = multi_modal_data
if src_prompt.get("mm_processor_kwargs") is not None:
@@ -251,6 +639,49 @@ def _estimate_ref_code_len_from_ref_audio(ref_audio: Any) -> int | None:
return None
def _approx_qwen_token_count(text: str) -> int:
"""Rough Qwen BPE length without loading a tokenizer.
CJK / CJK punctuation ≈ 1 token each; contiguous non-CJK (incl. spaces)
≈ 1 token per 4 chars. Do **not** count spaces as their own tokens — that
inflated English ``tts_instruct`` (~144 chars) from real ~35 to ~62 and
zero-padded Talker prefill by ~100 (leading garbage / early cut).
"""
if not text:
return 0
n = 0
i = 0
while i < len(text):
code = ord(text[i])
if 0x4E00 <= code <= 0x9FFF or 0x3400 <= code <= 0x4DBF or 0x3000 <= code <= 0x303F or 0xFF00 <= code <= 0xFFEF:
n += 1
i += 1
continue
j = i + 1
while j < len(text):
cj = ord(text[j])
if 0x4E00 <= cj <= 0x9FFF or 0x3400 <= cj <= 0x4DBF or 0x3000 <= cj <= 0x303F or 0xFF00 <= cj <= 0xFFEF:
break
j += 1
n += max(1, (j - i + 3) // 4)
i = j
return n
def _estimate_instruct_prompt_tokens(instruct: str) -> int:
"""Token length of ``build_instruct_text(instruct)`` without a tokenizer."""
body = instruct.strip() if isinstance(instruct, str) else ""
if not body:
return 0
return 5 + _approx_qwen_token_count(body)
def _estimate_assistant_prompt_tokens(text: str) -> int:
"""Token length of ``build_assistant_text(text)`` without a tokenizer."""
body = text if isinstance(text, str) else ""
return max(8, 8 + _approx_qwen_token_count(body))
def _estimate_tts_prompt_len_from_token_ids(
token_ids: list[int],
*,
@@ -265,14 +696,17 @@ def _estimate_tts_prompt_len_from_token_ids(
This mirrors Qwen3-TTS prompt assembly at length level:
prompt_len = instruct_len + role_len + codec_prefix_len + text/icl term
``instruct_len`` must be a *token* estimate. Using ``len(instruct)`` chars
(e.g. 144 for the demo style string vs real ~40 tokens) zero-pads Talker
prefill by ~100 and causes leading garbage / truncated speech.
"""
# Official defaults: Base -> streaming, others -> non-streaming.
if non_streaming_mode is None:
non_streaming_mode = task_type in ("CustomVoice", "VoiceDesign")
# We do not have tokenizer here; use char length as a monotonic proxy.
instruct_len = len(instruct.strip()) if isinstance(instruct, str) else 0
instruct_len = _estimate_instruct_prompt_tokens(instruct) if isinstance(instruct, str) else 0
assistant_len = max(0, len(token_ids))
# role_len = 3; codec_prefix_len = (prefill_len + speaker_len + 2) - 1
@@ -322,12 +756,15 @@ def aura2tts(
prompt_by_request_id = _source_prompt_by_request_id(source_outputs, prompt)
next_inputs: list[OmniTokensPrompt] = []
for idx, source_output in enumerate(source_outputs):
text = _extract_text(source_output).strip()
if not text or text == SILENT_TEXT:
continue
raw_text = _extract_text(source_output).strip()
text = _strip_assistant_text(raw_text)
src_prompt = prompt_by_request_id.get(str(getattr(source_output, "request_id", idx)), {})
additional_info = src_prompt.get("additional_information") or {}
close_only = bool(additional_info.get("aura_tts_close_only"))
if is_effectively_silent(text) and not close_only:
continue
if close_only:
text = ""
task_type = _first_value(additional_info.get("tts_task_type"), "Base")
language = _first_value(additional_info.get("tts_language"), "English")
instruct = _first_value(additional_info.get("tts_instruct"), "")
@@ -348,7 +785,7 @@ def aura2tts(
"task_type": [task_type],
"language": [language],
"instruct": [instruct],
"max_new_tokens": [int(_first_value(additional_info.get("tts_max_new_tokens"), 2048))],
"max_new_tokens": [1 if close_only else int(_first_value(additional_info.get("tts_max_new_tokens"), 2048))],
}
if pass_token_ids and assistant_token_ids_for_len:
tts_info[PRECOMPUTED_TEXT_IDS_KEY] = [assistant_token_ids_for_len]
@@ -356,16 +793,6 @@ def aura2tts(
tts_info["text"] = [text]
if ref_code_len is not None:
tts_info["ref_code_length"] = [int(ref_code_len)]
prompt_len = _estimate_tts_prompt_len_from_token_ids(
assistant_token_ids_for_len if assistant_token_ids_for_len else [0] * max(0, len(text)),
task_type=str(task_type),
language=str(language),
instruct=str(instruct),
x_vector_only_mode=x_vector_only_mode,
non_streaming_mode=non_streaming_mode,
ref_code_len=ref_code_len,
)
if task_type == "Base":
ref_audio = ref_audio or _first_value(additional_info.get("tts_ref_audio"), None)
ref_text = _first_value(additional_info.get("tts_ref_text"), None)
@@ -379,10 +806,52 @@ def aura2tts(
tts_info["speaker"] = [
_normalize_qwen3_tts_speaker(_first_value(additional_info.get("tts_speaker"), "Vivian"))
]
if close_only:
prompt_len = 1
else:
prompt_len = None
if not (pass_token_ids and assistant_token_ids_for_len):
prompt_len = _estimate_tts_prompt_len_official(
tts_info,
task_type=str(task_type),
additional_info=additional_info if isinstance(additional_info, dict) else None,
)
if prompt_len is None:
if pass_token_ids and assistant_token_ids_for_len:
length_token_ids = assistant_token_ids_for_len
else:
length_token_ids = [0] * _estimate_assistant_prompt_tokens(text)
prompt_len = _estimate_tts_prompt_len_from_token_ids(
length_token_ids,
task_type=str(task_type),
language=str(language),
instruct=str(instruct),
x_vector_only_mode=x_vector_only_mode,
non_streaming_mode=non_streaming_mode,
ref_code_len=ref_code_len,
)
logger.info(
"[aura2tts] task=%s language=%s speaker=%s text_len=%d instruct_len=%d "
"prompt_len=%d close_only=%s text_preview=%r",
task_type,
language,
tts_info.get("speaker", [None])[0],
len(text),
len(str(instruct).strip()) if instruct else 0,
prompt_len,
close_only,
text[:120],
)
next_inputs.append(
OmniTokensPrompt(
prompt_token_ids=[0] * prompt_len,
prompt_token_ids=[0] if close_only else [0] * int(prompt_len or 1),
additional_information=tts_info,
# Prefer runner data-plane so Talker preprocess still sees
# text if legacy additional_information is wiped (e.g. a
# mistaken Stage2 chunk-receiver edge).
model_intermediate_buffer=dict(tts_info),
multi_modal_data=None,
mm_processor_kwargs=None,
)
vllm_omni/protocol/duplex/commands.py
+3 -1
View File
@@ -53,7 +53,9 @@ class AppendAudio(realtime_commands.AppendAudio):
OpenAI's append is just base64 audio. The duplex lane also lets a client
declare this chunk's own format and rate, whether it believes the chunk is
speech, camera frames captured alongside it, and its timing --- all
additive, all ignored by a stock client.
additive, all ignored by a stock client. Empty ``audio`` with
``video_frames`` is legal when session capabilities allow video without
required audio.
"""
#: Model-neutral hints carried through from the wire (rms, vad, transcript hints ...).
vllm_omni/protocol/duplex/errors.py
+1
View File
@@ -65,6 +65,7 @@ REALTIME_ERROR_TYPES_BY_CODE: dict[str, str] = {
"ref_audio_update_unsupported": "invalid_request_error",
"native_text_append_unsupported": "invalid_request_error",
"invalid_video_frames": "invalid_request_error",
"invalid_input_modality": "invalid_request_error",
"invalid_function_call_output": "invalid_request_error",
"server_vad_unavailable": "server_error",
}
vllm_omni/protocol/realtime/audio_input.py
+37 -9
View File
@@ -175,7 +175,10 @@ def decode_audio_append(
defaults: RealtimeInputDefaults,
hints_source: Mapping[str, object] | None = None,
) -> RealtimeAudioAppend:
"""Validate, convert (to 16 kHz ``pcm_f32le``) and pack one audio append.
"""Validate and pack one input append (audio and/or video frames).
Capability checks (required/optional modalities) run later on the session.
Audio path converts to 16 kHz ``pcm_f32le``.
``hints_source`` is the enclosing payload when the audio arrives inside
something larger than a bare append --- a ``conversation.item.create``
@@ -186,6 +189,39 @@ def decode_audio_append(
"""
event_id = cast("str", event.get("event_id")) if isinstance(event.get("event_id"), str) else None
audio = event.get("audio") or event.get("delta")
video_frames: tuple[str, ...] = ()
raw_frames = event.get("video_frames")
if raw_frames is not None:
frames_error = validate_realtime_video_frames(raw_frames, event.get("max_slice_nums"))
if frames_error is not None:
raise RealtimeProtocolError(frames_error, code="invalid_video_frames", event_id=event_id)
if isinstance(raw_frames, list):
video_frames = tuple(frame for frame in raw_frames if isinstance(frame, str) and frame)
has_audio_field = isinstance(audio, str) and bool(audio)
if not has_audio_field:
if not video_frames:
raise RealtimeProtocolError(
"input_audio_buffer.append requires audio and/or video_frames",
code="bad_event",
event_id=event_id,
)
hints: dict[str, object] = {}
if hints_source is not None:
copy_realtime_input_hints(hints_source, hints)
copy_realtime_input_hints(event, hints)
duration_ms = hints.get("duration_ms", hints.get("audio_duration_ms"))
audio_end_ms = hints.get("audio_end_ms")
return RealtimeAudioAppend(
event_id=event_id,
audio=b"",
format="pcm_f32le",
sample_rate_hz=defaults.input_sample_rate_hz,
is_speech=False,
video_frames=video_frames,
duration_ms=int(duration_ms) if isinstance(duration_ms, int | float) else None,
audio_end_ms=int(audio_end_ms) if isinstance(audio_end_ms, int | float) else None,
hints=hints,
)
fmt, format_rate = parse_realtime_audio_format(
event.get("format") or event.get("input_audio_format") or defaults.input_audio_format
)
@@ -216,14 +252,6 @@ def decode_audio_append(
fmt=fmt,
overlap_silence_rms=defaults.overlap_silence_rms,
)
video_frames: tuple[str, ...] = ()
raw_frames = event.get("video_frames")
if raw_frames is not None:
frames_error = validate_realtime_video_frames(raw_frames, event.get("max_slice_nums"))
if frames_error is not None:
raise RealtimeProtocolError(frames_error, code="invalid_video_frames", event_id=event_id)
if isinstance(raw_frames, list):
video_frames = tuple(frame for frame in raw_frames if isinstance(frame, str) and frame)
duration_ms = hints.get("duration_ms", hints.get("audio_duration_ms"))
audio_end_ms = hints.get("audio_end_ms")
try:
vllm_omni/protocol/realtime/commands.py
+2 -1
View File
@@ -54,7 +54,8 @@ class UpdateSession(RealtimeCommand):
class AppendAudio(RealtimeCommand):
wire_type: ClassVar[str] = "input_audio_buffer.append"
#: Raw audio bytes in ``format`` at ``sample_rate_hz`` (base64 only on the wire).
audio: bytes
#: Empty means video-only / frames-only append (validated against duplex capabilities).
audio: bytes = b""
@dataclass(frozen=True, slots=True, kw_only=True)