Merge remote-tracking branch 'origin/main' into codex-api-server-user-guide

Signed-off-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com>

# Conflicts:
#	docs/user_guide/examples/online_serving/text_to_audio.md
#	examples/online_serving/stable_audio/README.md
This commit is contained in:
hsliu_ustc committed 2026-09-01 11:23:08 +08:00
commit 7d41ea6183
1457 files changed
+164090 -40241

No files matched your search

+92 -5
View File
@@ -32,12 +32,14 @@ GPU 0 (90%) and colocates the Talker (55%) and Code2Wav (35%) on GPU 1. That
profile admits at most four concurrent sequences per stage.
| deploy config | GPUs | Notes |
|---|---|---|
| --- | --- | --- |
| `minicpmo_4_5.yaml` (default) | 1 | Memory-constrained compatibility layout. |
| `minicpmo_4_5_2gpu.yaml` | 2 | Recommended continuous-batching layout; Talker and Code2Wav share GPU 1. |
| `minicpmo_4_5_3gpu.yaml` | 3 | One GPU per stage. |
| `minicpmo_4_5_8x4090.yaml` | 8 | Full 8x4090 layout. |
| `minicpmo_4_5_duplex.yaml` | 1 | Experimental native full-duplex overlay. |
Every profile sets `session_mode: duplex`, so native full-duplex is served
from the same process as `/v1/chat/completions`.
The split pipeline preserves native-duplex epoch/turn identity, segment text,
turn completion, reference voice, and terminal-audio metadata through
@@ -55,8 +57,8 @@ vllm serve openbmb/MiniCPM-o-4_5 \
```
For local ModelScope checkpoints, replace `openbmb/MiniCPM-o-4_5` with the
checkpoint path. To start the experimental native duplex backend, use
`vllm_omni/deploy/minicpmo_4_5_duplex.yaml`.
checkpoint path. For native full-duplex, connect `/v1/realtime?duplex=1` on
this same server.
### Per-stage overrides
@@ -145,7 +147,7 @@ assistant template, so it is not an apples-to-apples accuracy run.
## Run the Realtime duplex CLI demo
After the duplex backend is running, stream one WAV through the Realtime
After the server is running, stream one WAV through the Realtime
WebSocket endpoint:
```bash
@@ -157,6 +159,33 @@ python examples/online_serving/minicpmo/realtime_duplex_demo.py \
--output-dir /tmp/minicpmo_realtime_duplex_demo
```
Video input uses the same session. PyAV demuxes JPEG frames at `--video-fps`
(default 1.0) and vLLM `load_audio` extracts a 16 kHz mono WAV unless
`--input-wav` overrides the soundtrack. Frames keep their capture size
(`--frame-max-side 0`); the server `process_image` normalizes at 448.
`--stack-frames N` raises the visual refresh rate the way official duplex does:
each 1 s unit also samples the `N-1` sub-frames captured inside it and tiles
them into a single composite image sent next to that unit's base frame. The
audio cadence never changes — a unit is always one second — and the wire always
carries 2 images per unit however large `N` is, because the sub-frames share one
composite. Official uses 5 for high refresh rate mode:
```bash
python examples/online_serving/minicpmo/realtime_duplex_demo.py \
--url ws://localhost:8099/v1/realtime?duplex=1 \
--model openbmb/MiniCPM-o-4_5 \
--input-video /path/to/clip.mp4 \
--stack-frames 5 \
--ref-audio /path/to/MiniCPM-o-Demo/assets/ref_audio/ref_minicpm_signature.wav \
--output-dir /tmp/minicpmo_realtime_duplex_video_demo
```
Detail inside a composite is capped by `scale_resolution=448` at
`max_slice_nums=1`: official suggests HD slicing (`max_slice_nums=[2, 1]`) for
stacked frames, which the duplex adapter does not implement yet. Reading small
text or digits out of a wide scene is limited by that, not by frame timing.
## Open the experimental browser client
The browser UI serves the page and proxies the same-origin Realtime WebSocket to
@@ -190,6 +219,64 @@ stronger `response-required` mode is diagnostic: it requires a purpose-built
two-response WAV, its `--input-sha256`, and an
`--expect-second-response-substring` value.
## Run Omni-DuplexEval
Omni-DuplexEval generation uses the vLLM-Omni native MiniCPM-o duplex
endpoint. Evaluation uses a separate OpenAI-compatible multimodal judge
served from the same vLLM-Omni environment. This validates the vLLM-Omni
duplex client and local judge integration; it does not reproduce the paper's
original MiniCPM-o inference implementation.
Start the duplex generation server:
```bash
vllm serve openbmb/MiniCPM-o-4_5 --omni --trust-remote-code \
--deploy-config vllm_omni/deploy/minicpmo_4_5_duplex.yaml \
--served-model-name openbmb/MiniCPM-o-4_5 \
--host 0.0.0.0 --port 8099
```
Start a separate multimodal judge. The allowed path must contain any local
videos passed with `--judge-video-mode video_url`:
```bash
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
--served-model-name Qwen/Qwen2.5-VL-7B-Instruct \
--allowed-local-media-path /data/omni-duplex-eval \
--host 0.0.0.0 --port 8000
```
Generate, evaluate, and summarize:
```bash
vllm bench omni-duplex-eval --omni generate \
--url ws://127.0.0.1:8099/v1/realtime?duplex=1 \
--model openbmb/MiniCPM-o-4_5 \
--ref-audio /data/ref.wav \
--dataset Hothan/Omni-DuplexEval \
--concurrency 2 \
--response-root /data/omni-duplex-eval/responses
vllm bench omni-duplex-eval --omni evaluate \
--dataset Hothan/Omni-DuplexEval \
--response-root /data/omni-duplex-eval/responses \
--score-root /data/omni-duplex-eval/scores \
--judge-base-url http://127.0.0.1:8000 \
--judge-model Qwen/Qwen2.5-VL-7B-Instruct \
--judge-video-mode video_url \
--eval-workers 4
vllm bench omni-duplex-eval --omni summarize \
--score-root /data/omni-duplex-eval/scores
```
The dataset names such as `RTD_OCR` and `PR_correction` are Hugging Face
splits, not dataset configurations. Pass one with `--split`, or omit it to
run all nine splits. `--limit 1` is useful for a smoke test. Realtime
generation records `clock=media`; artifacts generated with
`--pace as-fast-as-possible` record `clock=invalid` and evaluation rejects
them unless `--allow-invalid-clock` is explicit.
## Related examples
- [Offline MiniCPM-o inference](../../offline_inference/minicpmo/)
@@ -1,3 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Minimal single-input MiniCPM-o 4.5 Realtime duplex demo.
Run this after starting the duplex server. Strict lifecycle, overlap, and
@@ -9,7 +12,9 @@ from __future__ import annotations
import argparse
import asyncio
import base64
import io
import json
import math
import sys
import time
from pathlib import Path
@@ -28,6 +33,17 @@ from vllm_omni.experimental.fullduplex.client import ( # noqa: E402
wait_for,
write_pcm16_wav,
)
from vllm_omni.experimental.fullduplex.client import chunk_period_ms as _chunk_period_ms # noqa: E402
from vllm_omni.experimental.fullduplex.client import ( # noqa: E402
has_residual_model_unit as _has_residual_model_unit,
)
from vllm_omni.experimental.fullduplex.client import ( # noqa: E402
reference_audio_data_url as _ref_audio_data_url,
)
from vllm_omni.experimental.fullduplex.video_stacking import ( # noqa: E402
concat_frames_b64,
unit_subframe_offsets,
)
class _StreamingOutputWriter:
@@ -133,31 +149,29 @@ def _latest_model_decision(
return decision
def _chunk_period_ms(events: list[dict[str, object]]) -> int:
for event in reversed(events):
session = event.get("session")
if not isinstance(session, dict):
continue
capabilities = session.get("capabilities")
if not isinstance(capabilities, dict):
continue
chunk_period_ms = capabilities.get("chunk_period_ms")
if isinstance(chunk_period_ms, int) and chunk_period_ms > 0:
return chunk_period_ms
return 1000
def _has_residual_model_unit(pcm16: bytes, *, chunk_period_ms: int) -> bool:
unit_bytes = PCM16_SAMPLE_RATE * PCM16_BYTES_PER_SAMPLE * chunk_period_ms // 1000
return bool(unit_bytes > 0 and len(pcm16) % unit_bytes)
def _response_in_progress(events: list[dict[str, object]]) -> bool:
return sum(event.get("type") == "response.created" for event in events) > sum(
event.get("type") == "response.done" for event in events
)
async def _commit_input_after_playback_checkpoint(client: RealtimeDuplexClient) -> float:
"""Commit input only after preserving already-played assistant history.
Native duplex can finish several responses while the user is still
streaming. Under the ``ack_only`` playback policy, acknowledging those
responses after the final input commit is intentionally rejected: it could
append an old assistant message after the newly committed user message.
Checkpoint playback first so the server commits each assistant item at its
original history position. A second acknowledgement after the commit can
then safely advance response-local playback that drained in the meantime.
"""
await client.acknowledge_playback()
commit_sent_at_s = time.monotonic()
await client.commit()
return commit_sent_at_s
def _event_count_after(
events: list[dict[str, object]],
event_type: str,
@@ -168,21 +182,258 @@ def _event_count_after(
return sum(event.get("type") == event_type for event in events[index + 1 :])
def _ref_audio_data_url(path: str | None) -> str | None:
if path is None:
return None
ref_path = Path(path).expanduser()
return "data:audio/wav;base64," + base64.b64encode(ref_path.read_bytes()).decode("ascii")
def _probe_video_frames_and_fps(video_path: Path) -> tuple[int, float]:
"""Frame count and average FPS via PyAV (same approach as Daily-Omni)."""
import av
with av.open(str(video_path)) as container:
stream = container.streams.video[0]
avg_fps = float(stream.average_rate) if stream.average_rate else 0.0
num_frames = stream.frames or 0
if num_frames <= 0:
duration: float | None = None
if stream.duration is not None and stream.time_base is not None:
duration = float(stream.duration * stream.time_base)
elif container.duration is not None:
duration = container.duration / av.time_base
if duration and avg_fps:
num_frames = int(math.floor(duration * avg_fps))
if num_frames <= 0:
stream.thread_type = "AUTO"
num_frames = sum(1 for _ in container.decode(stream))
if num_frames <= 0:
raise ValueError(f"No decodable video frames in {video_path}")
return num_frames, avg_fps or 1.0
def _upright(image, rotation: int):
"""Apply a container display-matrix rotation, in degrees, to ``image``.
PyAV hands back the stored raster and leaves the display matrix for the
caller, unlike the ffmpeg CLI which autorotates. Phone clips routinely
carry +/-90, so skipping this feeds the model a sideways frame: arrows and
digits stop being readable and the model mirrors the motion it reports.
"""
from PIL import Image
quarter_turns = {
90: Image.Transpose.ROTATE_90,
180: Image.Transpose.ROTATE_180,
270: Image.Transpose.ROTATE_270,
}
turn = quarter_turns.get(int(rotation) % 360)
return image.transpose(turn) if turn is not None else image
def _decode_video_frames_rgb(video_path: Path, frame_idx: list[int]) -> list:
"""Decode presentation-order ``frame_idx`` into upright RGB ``PIL.Image``s."""
import av
wanted = sorted({int(i) for i in frame_idx})
if not wanted:
return []
decoded: dict[int, object] = {}
with av.open(str(video_path)) as container:
stream = container.streams.video[0]
stream.thread_type = "AUTO"
cursor = 0
for position, frame in enumerate(container.decode(stream)):
if wanted[cursor] > position:
continue
image = _upright(frame.to_image(), getattr(frame, "rotation", 0) or 0)
while cursor < len(wanted) and wanted[cursor] <= position:
decoded[wanted[cursor]] = image
cursor += 1
if cursor >= len(wanted):
break
if not decoded:
raise ValueError(f"Decoded no frames from {video_path} for indices {wanted[:8]}")
last_image = decoded[max(decoded)]
return [decoded.get(int(i), last_image) for i in frame_idx]
def _resize_max_side(image, max_side: int):
width, height = image.size
longest = max(width, height)
if longest <= max_side or max_side <= 0:
return image
scale = max_side / float(longest)
new_size = (max(1, int(round(width * scale))), max(1, int(round(height * scale))))
return image.resize(new_size)
def _jpeg_bytes(image, *, quality: int = 95) -> bytes:
buf = io.BytesIO()
image.convert("RGB").save(buf, format="JPEG", quality=quality)
return buf.getvalue()
def _to_pcm16_bytes(audio_np) -> bytes:
import numpy as np
arr = np.asarray(audio_np).reshape(-1)
if arr.dtype != np.int16:
arr = (np.clip(arr, -1.0, 1.0) * 32767.0).astype(np.int16)
return arr.tobytes()
def _pcm16_wav_duration_s(path: Path) -> float:
pcm16 = read_pcm16_wav(path)
return len(pcm16) / float(PCM16_SAMPLE_RATE * PCM16_BYTES_PER_SAMPLE)
def _extract_video_frames(
video_path: Path,
work_dir: Path,
*,
fps: float,
max_side: int,
duration_s: float,
stack_frames: int = 1,
) -> tuple[list[str], list[str | None]]:
"""Demux JPEG frames only. Frame count follows ``duration_s``, not soundtrack.
Returns the base frames (one per model unit) and the parallel stacked track.
With ``stack_frames=N`` each unit also samples its ``N-1`` interior
sub-frames and tiles them into one composite, which is how official duplex
raises the visual refresh rate while keeping one second of audio per unit.
A unit whose interior falls past the end of the clip stacks nothing.
"""
num_frames, avg_fps = _probe_video_frames_and_fps(video_path)
sample_fps = max(float(fps), 1e-6)
unit_s = 1.0 / sample_fps
timeline_s = max(float(duration_s), unit_s)
num_samples = max(1, int(math.ceil(timeline_s * sample_fps - 1e-9)))
offsets = unit_subframe_offsets(stack_frames, unit_s=unit_s)
def frame_at(seconds: float) -> int:
# Official get_video_frame_audio_segments uses truncating int(), not
# round(), so 29.97 fps lands on the same presentation index.
return min(int(seconds * avg_fps), num_frames - 1)
base_idx = [frame_at(unit * unit_s) for unit in range(num_samples)]
sub_times = [
[unit * unit_s + offset for offset in offsets if unit * unit_s + offset < timeline_s]
for unit in range(num_samples)
]
sub_idx = [frame_at(seconds) for times in sub_times for seconds in times]
images = _decode_video_frames_rgb(video_path, base_idx + sub_idx)
sub_images = iter(images[len(base_idx) :])
frame_dir = work_dir / "frames"
frame_dir.mkdir(parents=True, exist_ok=True)
for stale in frame_dir.glob("*.jpg"):
stale.unlink()
frames_b64: list[str] = []
stacked_b64: list[str | None] = []
for index, (image, times) in enumerate(zip(images[: len(base_idx)], sub_times), start=1):
jpeg = _jpeg_bytes(_resize_max_side(image, max_side))
(frame_dir / f"frame_{index:05d}.jpg").write_bytes(jpeg)
frames_b64.append(base64.b64encode(jpeg).decode("ascii"))
interior = [_resize_max_side(next(sub_images), max_side) for _ in times]
if not interior:
stacked_b64.append(None)
continue
composite = concat_frames_b64(interior)
(frame_dir / f"stack_{index:05d}.jpg").write_bytes(base64.b64decode(composite))
stacked_b64.append(composite)
return frames_b64, stacked_b64
def _extract_video_soundtrack(video_path: Path, work_dir: Path) -> Path:
"""Write a 16 kHz mono WAV from the video soundtrack."""
try:
from vllm.multimodal.media.audio import load_audio
except ImportError as exc: # pragma: no cover
raise SystemExit("--input-video needs vLLM audio helpers (load_audio)") from exc
audio_np, sr = load_audio(str(video_path), sr=PCM16_SAMPLE_RATE, mono=True)
work_dir.mkdir(parents=True, exist_ok=True)
audio_path = work_dir / "input_16k.wav"
write_pcm16_wav(audio_path, _to_pcm16_bytes(audio_np), sample_rate_hz=int(sr))
return audio_path
def _extract_video_input(
video_path: Path,
work_dir: Path,
*,
fps: float,
max_side: int,
stack_frames: int,
) -> tuple[Path, list[str], list[str | None]]:
"""Demux a video into duplex inputs: 16 kHz mono WAV plus JPEG frames.
Use this only when the video soundtrack is the audio source. When the
caller already has ``--input-wav``, extract frames against that WAV
duration instead so a silent clip cannot fail ``load_audio``.
"""
audio_path = _extract_video_soundtrack(video_path, work_dir)
frames_b64, stacked_b64 = _extract_video_frames(
video_path,
work_dir,
fps=fps,
max_side=max_side,
duration_s=_pcm16_wav_duration_s(audio_path),
stack_frames=stack_frames,
)
return audio_path, frames_b64, stacked_b64
def _resolve_duplex_av_inputs(
*,
input_wav: str | None,
input_video: str | None,
work_dir: Path,
fps: float,
max_side: int,
stack_frames: int = 1,
) -> tuple[str, list[str], list[str | None]]:
"""Return ``(wav_path, frames_b64, stacked_b64)``. External WAV wins over video audio."""
if input_video:
video_path = Path(input_video)
if input_wav:
frames, stacked = _extract_video_frames(
video_path,
work_dir,
fps=fps,
max_side=max_side,
duration_s=_pcm16_wav_duration_s(Path(input_wav)),
stack_frames=stack_frames,
)
return input_wav, frames, stacked
extracted_wav, frames, stacked = _extract_video_input(
video_path,
work_dir,
fps=fps,
max_side=max_side,
stack_frames=stack_frames,
)
return str(extracted_wav), frames, stacked
if not input_wav:
raise SystemExit("provide --input-wav, --input-video, or both")
return input_wav, [], []
async def run_demo(args: argparse.Namespace) -> dict[str, object]:
input_pcm16 = read_pcm16_wav(Path(args.input_wav))
if not input_pcm16:
raise ValueError("input WAV has no audio")
output_dir = Path(args.output_dir)
stream_writer = _StreamingOutputWriter(output_dir)
input_wav, video_frames, stacked_video_frames = _resolve_duplex_av_inputs(
input_wav=args.input_wav,
input_video=args.input_video,
work_dir=output_dir / "video_input",
fps=args.video_fps,
max_side=args.frame_max_side,
stack_frames=args.stack_frames,
)
input_pcm16 = read_pcm16_wav(Path(input_wav))
if not input_pcm16:
raise ValueError("input WAV has no audio")
url = build_realtime_url(
args.url,
args.model,
@@ -200,10 +451,12 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
timeout_s=args.timeout_s,
)
stream_event_cursor = len(client.events.events)
await client.stream_pcm16(
frames_sent = await client.stream_pcm16(
input_pcm16,
chunk_ms=args.chunk_ms,
realtime=not args.no_realtime_pacing,
video_frames=video_frames,
stacked_video_frames=stacked_video_frames,
)
commit_event_cursor = len(client.events.events)
stream_decision = _latest_model_decision(client.events.events, stream_event_cursor)
@@ -212,8 +465,7 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
chunk_period_ms=_chunk_period_ms(client.events.events),
)
wait_for_post_commit_decision = False
commit_sent_at_s = time.monotonic()
await client.commit()
commit_sent_at_s = await _commit_input_after_playback_checkpoint(client)
wait_error: str | None = None
committed_index: int | None = None
post_commit_decision: str | None = None
@@ -326,6 +578,10 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
committed_index,
),
},
"input_video": args.input_video,
"video_frames_extracted": len(video_frames),
"video_frames_sent": frames_sent,
"video_stacked_frames_extracted": sum(frame is not None for frame in stacked_video_frames),
"audio_bytes": len(audio),
"audio_chunk_count": len(stream_writer.audio_chunk_paths),
"audio_chunk_files": [str(path) for path in stream_writer.audio_chunk_paths],
@@ -372,7 +628,48 @@ def parse_args() -> argparse.Namespace:
parser.add_argument("--url", default="ws://localhost:8099/v1/realtime?duplex=1")
parser.add_argument("--model", default="openbmb/MiniCPM-o-4_5")
parser.add_argument("--session-id")
parser.add_argument("--input-wav", required=True)
parser.add_argument(
"--input-wav",
help="Mono 16 kHz PCM16 clip to stream. Optional when --input-video carries the audio track.",
)
parser.add_argument(
"--input-video",
help=(
"Video to stream as omni duplex input. PyAV demuxes frames and "
"vLLM load_audio extracts a 16 kHz mono WAV (unless --input-wav overrides the audio)."
),
)
parser.add_argument(
"--video-fps",
type=float,
default=1.0,
help="Frame extraction rate. The wire contract carries ~1 frame per second of audio.",
)
parser.add_argument(
"--frame-max-side",
type=int,
default=0,
help=(
"Optional client-side downscale of the longest side, in pixels. "
"0 (default) keeps the capture resolution so the server "
"process_image can normalize at scale_resolution=448 from a "
"higher-quality source. The web demo never resizes on the client."
),
)
parser.add_argument(
"--stack-frames",
type=int,
default=1,
help=(
"Frames sampled per 1 s audio unit. 1 (default) sends the unit's "
"frame alone. N > 1 also samples the N-1 sub-frames inside the unit "
"and tiles them into one composite sent next to the base frame, so "
"motion (arrows, doors, digits changing) is visible at N fps while "
"the audio stays 1 s per unit. Official uses 5 for high refresh "
"rate mode; the wire carries 2 images per unit at any N because the "
"sub-frames share one composite."
),
)
parser.add_argument(
"--ref-audio",
required=True,
@@ -388,7 +685,7 @@ def parse_args() -> argparse.Namespace:
"--temperature",
type=float,
default=None,
help="Stage0 sampling temperature; response-required soft-interrupt uses 0.0.",
help="Stage 0 sampling temperature; omit to preserve the model default.",
)
parser.add_argument("--no-realtime-pacing", action="store_true")
parser.add_argument("--require-audio", action="store_true")