[perf] Minicpm o4.5 optimization (#5228)

Signed-off-by: R2-Y <ruiruyang2@gmail.com>
Co-authored-by: Sy03 <1370724210@qq.com>

examples/online_serving/minicpmo/README.md

@@ -24,18 +24,28 @@ including `librosa`.
 
 ## Start the backend server
 
-Pick a deploy config that matches your GPU layout:
+The deploy config auto-loads via `--omni`.
+The default `vllm_omni/deploy/minicpmo_4_5.yaml` keeps all three stages on
+logical device 0 with memory budgets of 65%, 15%, and 15%. For throughput,
+`minicpmo_4_5_batching.yaml` gives the Thinker GPU 0 (90%) and colocates the
+Talker (55%) and Code2Wav (35%) on GPU 1. Each stage admits at most four
+concurrent sequences.
 
-| config | GPUs | TP | Notes |
-|---|---|---|---|
-| `minicpmo_4_5.yaml` | 1 | 1 | Thinker and talker+t2w co-located on GPU0. |
-| `minicpmo_4_5_2gpu.yaml` | 2 | 1 | Thinker on GPU0, talker+t2w on GPU1. |
-| `minicpmo_4_5_3gpu.yaml` | 3 | 2 | Thinker 2-way TP on GPU0/1, talker+t2w share GPU2. |
-| `minicpmo_4_5_8x4090.yaml` | 8 | 4 | Thinker 4-way TP on GPU0-3, talker+t2w on GPU4. |
-| `minicpmo_4_5_3gpu_stage1_replicas.yaml` | 3 | 1 | Experimental validation profile: Thinker on GPU0, two talker+Token2wav replicas on GPU1/2. |
-| `minicpmo_4_5_4gpu_stage1_replicas.yaml` | 4 | 1 | Experimental validation profile: Thinker on GPU0, three talker+Token2wav replicas on GPU1/2/3. |
-| `minicpmo_4_5_8x4090_stage1_replicas.yaml` | 8 | 4 | Experimental validation profile: Thinker 4-way TP on GPU0-3, four talker+Token2wav replicas on GPU4-7. |
-| `minicpmo_4_5_duplex.yaml` | 2 | 1 | Experimental native duplex profile. |
+| deploy config | GPUs | Notes |
+|---|---|---|
+| `minicpmo_4_5.yaml` (default) | 1 | Memory-constrained compatibility layout. |
+| `minicpmo_4_5_batching.yaml` | 2 | Recommended continuous-batching layout; Talker and Code2Wav share GPU 1. |
+| `minicpmo_4_5_2gpu.yaml` | 2 | Backward-compatible alias for `minicpmo_4_5_batching.yaml`. |
+| `minicpmo_4_5_3gpu.yaml` | 3 | One GPU per stage. |
+| `minicpmo_4_5_8x4090.yaml` | 8 | Full 8x4090 layout. |
+| `minicpmo_4_5_duplex.yaml` | 1 | Experimental native full-duplex overlay. |
+
+The split pipeline preserves native-duplex epoch/turn identity, segment text,
+turn completion, reference voice, and terminal-audio metadata through
+Code2Wav. Focused CPU regressions cover this envelope; run the Realtime
+scenario below for live barge-in validation on the target GPU.
+
+Default:
 
 ```bash
 vllm-omni serve openbmb/MiniCPM-o-4_5 \
@@ -49,13 +59,20 @@ For local ModelScope checkpoints, replace `openbmb/MiniCPM-o-4_5` with the
 checkpoint path. To start the experimental native duplex backend, use
 `vllm_omni/deploy/minicpmo_4_5_duplex.yaml`.
 
-The `*_stage1_replicas.yaml` files exercise composite Stage1 replica routing
-and failure recovery. They are validation profiles, not recommended production
-entrypoints.
+### Per-stage overrides
 
-## Send chat requests
+```bash
+vllm serve openbmb/MiniCPM-o-4_5 --omni --trust-remote-code --port 8099 \
+    --stage-overrides '{"0": {"gpu_memory_utilization": 0.65}}'
+```
 
-From this directory, run the MiniCPM-specific curl or Python clients:
+## Send multimodal requests
+
+```bash
+cd examples/online_serving/minicpmo
+```
+
+### curl
 
 ```bash
 bash run_curl_multimodal_generation.sh text
@@ -64,13 +81,40 @@ bash run_curl_multimodal_generation.sh use_audio '["text"]'
 
 python openai_chat_completion_client_for_multimodal_generation.py \
     --query-type use_image \
-    --host localhost \
+    --port 8099 \
+    --host localhost
+
+# Text-only (faster; no <|tts_bos|>)
+python openai_chat_completion_client_for_multimodal_generation.py \
+    --query-type text \
+    --modalities text \
+    --prompt "Briefly introduce yourself."
+```
+
+Streaming text + audio:
+
+```bash
+python streaming_chat_completion.py \
+    --base-url http://localhost:8099/v1 \
+    --output minicpmo_stream.wav
+```
+
+The example prints text deltas immediately and joins the independently encoded
+24 kHz WAV audio deltas into one valid WAV file. Add `--text-only` to skip
+Talker and Code2Wav.
+
+Shared helpers also work if you pass MiniCPM defaults yourself:
+
+```bash
+python ../openai_chat_completion_client_for_multimodal_generation.py \
+    --model openbmb/MiniCPM-o-4_5 \
+    --query-type text \
     --port 8099
 ```
 
-Speech output requires `chat_template_kwargs.use_tts_template=true`. Put that
-field at the request root for curl; the OpenAI Python SDK can merge it from
-`extra_body`.
+Speech output no longer depends on a MiniCPM-specific default in the generic
+serving layer. `chat_template_kwargs.use_tts_template=true` remains an
+explicitly supported model option.
 
 ## Launch the Gradio demo
 
@@ -86,6 +130,19 @@ python examples/online_serving/minicpmo/gradio_demo.py \
 
 Open `http://<host>:7862` in a browser.
 
+## Daily-Omni accuracy
+
+Daily-Omni requires one A–D letter. Set
+`chat_template_kwargs.enable_thinking=false` explicitly in `--extra-body`;
+the generic benchmark CLI does not inject a MiniCPM-specific default.
+Leaving reasoning enabled can exhaust the 256-token answer budget inside
+`<think>` and make first-letter extraction score reasoning text.
+
+For the established text benchmark, send
+`--extra_body '{"modalities":["text"],"chat_template_kwargs":{"enable_thinking":false}}'`.
+Requesting audio also benchmarks Talker and Code2Wav and changes the
+assistant template, so it is not an apples-to-apples accuracy run.
+
 ## Run the Realtime duplex CLI demo
 
 After the duplex backend is running, stream one WAV through the Realtime
@@ -137,3 +194,26 @@ two-response WAV, its `--input-sha256`, and an
 
 - [Offline MiniCPM-o inference](../../offline_inference/minicpmo/)
 - [MiniCPM-o 4.5 recipe](../../../recipes/OpenBMB/MiniCPM-o-4_5.md)
+
+## Pipeline notes
+
+- Stage 1 performs request-owned AR continuous batching. Stage 2 keeps
+  request-owned Flow/HiFT caches and batches exact-shape-compatible chunks.
+- Reference audio travels with the first codec chunk; Stage 2 owns its
+  temporary prompt WAV and evicts prompt features when the request finishes.
+- Codec sampling reads the checkpoint `tts_config` (default deterministic
+  seed 42). Stage-1 YAML sampling parameters govern only the binary
+  continue/stop token exposed to vLLM.
+- `StageRequestStats.batch_size` is request-scoped and does not report the
+  scheduler's execution batch.
+- Stage 0 and Stage 1 use vLLM CUDA Graph capture. Stage 2 remains eager until
+  a dedicated exact-shape graph wrapper owns static I/O buffers and copies
+  request cache state outside capture.
+- Co-locating all three stages minimizes hardware requirements but makes their
+  CUDA contexts contend for one GPU. Use the 8x4090 layout or a custom
+  multi-GPU deploy config when throughput is the primary goal.
+- Output audio is base64 WAV in `message.audio.data` (24 kHz mono).
+- Offline counterpart:
+  [`examples/offline_inference/minicpmo/`](../../offline_inference/minicpmo/)
+- Recipe:
+  [`recipes/OpenBMB/MiniCPM-o-4_5.md`](../../../recipes/OpenBMB/MiniCPM-o-4_5.md)

examples/online_serving/minicpmo/streaming_chat_completion.py

@@ -0,0 +1,71 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Stream MiniCPM-o 4.5 text and audio from /v1/chat/completions."""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import io
+import wave
+from pathlib import Path
+
+from openai import OpenAI
+
+
+def append_wav_chunk(encoded: str, pcm_parts: list[bytes]) -> tuple[int, int, int]:
+    """Decode one base64 WAV delta and append its PCM frames."""
+    with wave.open(io.BytesIO(base64.b64decode(encoded)), "rb") as wav:
+        channels = wav.getnchannels()
+        sample_width = wav.getsampwidth()
+        sample_rate = wav.getframerate()
+        pcm_parts.append(wav.readframes(wav.getnframes()))
+    return channels, sample_width, sample_rate
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base-url", default="http://localhost:8099/v1")
+    parser.add_argument("--model", default="openbmb/MiniCPM-o-4_5")
+    parser.add_argument("--prompt", default="Say hello, then introduce vLLM in one sentence.")
+    parser.add_argument("--output", type=Path, default=Path("minicpmo_stream.wav"))
+    parser.add_argument("--text-only", action="store_true")
+    args = parser.parse_args()
+
+    modalities = ["text"] if args.text_only else ["text", "audio"]
+    stream = OpenAI(base_url=args.base_url, api_key="EMPTY").chat.completions.create(
+        model=args.model,
+        messages=[{"role": "user", "content": args.prompt}],
+        modalities=modalities,
+        stream=True,
+        extra_body={"chat_template_kwargs": {"use_tts_template": not args.text_only}},
+    )
+
+    pcm_parts: list[bytes] = []
+    audio_format: tuple[int, int, int] | None = None
+    for chunk in stream:
+        for choice in chunk.choices:
+            content = getattr(choice.delta, "content", None)
+            if not content:
+                continue
+            if getattr(chunk, "modality", None) == "audio":
+                current_format = append_wav_chunk(content, pcm_parts)
+                if audio_format is not None and current_format != audio_format:
+                    raise ValueError(f"Audio format changed: {audio_format} -> {current_format}")
+                audio_format = current_format
+            elif getattr(chunk, "modality", None) == "text":
+                print(content, end="", flush=True)
+
+    print()
+    if pcm_parts and audio_format is not None:
+        channels, sample_width, sample_rate = audio_format
+        with wave.open(str(args.output), "wb") as wav:
+            wav.setnchannels(channels)
+            wav.setsampwidth(sample_width)
+            wav.setframerate(sample_rate)
+            wav.writeframes(b"".join(pcm_parts))
+        print(f"Audio saved to {args.output} ({sample_rate} Hz)")
+
+
+if __name__ == "__main__":
+    main()