vllm-omni/recipes/OpenBMB/MiniCPM-o-4_5.md

# MiniCPM-o 4.5

> Online full-duplex and chat-completions serving, and offline inference, for omni multimodal chat
> (text / image / audio / video → text + 24 kHz speech)

## Summary

- Vendor: OpenBMB
- Model: [`openbmb/MiniCPM-o-4_5`](https://huggingface.co/openbmb/MiniCPM-o-4_5)
- Task: Omni multimodal chat — accepts text / image / audio / video input;
  emits text and 24 kHz mono speech in the same response
- Mode: Online full-duplex serving over the OpenAI Realtime protocol
  (`/v1/realtime?duplex=1`, alias `/v1/duplex`), the OpenAI-compatible
  `/v1/chat/completions` API (plus Gradio demo) on the same server, and
  offline turn-based inference via `Omni.generate`. Besides those the server
  exposes only `/v1/models` and `/health`; the other turn-based HTTP routes
  are not available on a duplex deployment.
- Maintainer: [`@tc-mb`](https://github.com/tc-mb) (MiniCPM-V / MiniCPM-o team)

## When to use this recipe

Use this recipe as a known-good starting point for serving
`openbmb/MiniCPM-o-4_5` on vLLM-Omni. MiniCPM-o 4.5 is the omni member
of the MiniCPM-o family — it runs a multimodal thinker, a streaming
MiniCPMTTS codec talker, and a separate batched Code2Wav stage, so a live
duplex session streams text and 24 kHz speech while it keeps listening, and a
single `/v1/chat/completions` call returns both in one shot. The
recommended batching deploy isolates the Thinker on GPU 0 and
co-locates Talker and Code2Wav on GPU 1; 1-GPU, 3-GPU, and 8x4090 layouts are
also provided.

## References

- Default deploy configs (auto-loaded by HF `model_type=minicpmo` +
  `hf_config.version="4.5"`):
    - Default single-GPU compatibility layout (auto-loaded):
    [`vllm_omni/deploy/minicpmo_4_5.yaml`](../../vllm_omni/deploy/minicpmo_4_5.yaml)
    - Recommended 2-GPU continuous-batching layout:
    [`vllm_omni/deploy/minicpmo_4_5_2gpu.yaml`](../../vllm_omni/deploy/minicpmo_4_5_2gpu.yaml),
    - 3-GPU layout:
    [`vllm_omni/deploy/minicpmo_4_5_3gpu.yaml`](../../vllm_omni/deploy/minicpmo_4_5_3gpu.yaml)
    - 8x RTX 4090 layout:
    [`vllm_omni/deploy/minicpmo_4_5_8x4090.yaml`](../../vllm_omni/deploy/minicpmo_4_5_8x4090.yaml)
- Online examples (Realtime CLI demo, browser client, barge-in client,
  chat-completions client, curl script, Gradio demo):
  [`examples/online_serving/minicpmo/`](../../examples/online_serving/minicpmo/)
- Duplex API and Python client docs:
  [`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md)
- Offline end-to-end example:
  [`examples/offline_inference/minicpmo/`](../../examples/offline_inference/minicpmo/)
- Pipeline / talker source:
  [`vllm_omni/model_executor/models/minicpmo_4_5/`](../../vllm_omni/model_executor/models/minicpmo_4_5/)
- Stage-input processors (thinker → talker and talker → Code2Wav):
  [`vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py`](../../vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py)
- Upstream model card:
  [`openbmb/MiniCPM-o-4_5`](https://huggingface.co/openbmb/MiniCPM-o-4_5)
- Integration PR:
  [vllm-project/vllm-omni#3642](https://github.com/vllm-project/vllm-omni/pull/3642)

## Hardware Support

Four hardware layouts ship with deploy configs. Every layout uses the
same strict three-stage topology. The Talker emits codec chunks only;
Code2Wav consumes them through a shared-memory async connector.

| Layout | Thinker | Talker | Code2Wav | Typical hardware |
| --- | --- | --- | --- | --- |
| 1-GPU (default) | GPU 0 | GPU 0 | GPU 0 | 1x large-memory GPU |
| 2-GPU | GPU 0 | GPU 1 | GPU 1 | 2x large-memory GPU |
| 3-GPU | GPU 0 | GPU 1 | GPU 2 | 3x GPU |
| 8x RTX 4090 24GB | GPU 0–3 (TP=4) | GPU 4 | GPU 5 | 8x RTX 4090 consumer |

### Migration from the fused deployment

MiniCPM-o 4.5 now requires the three-stage topology: the Talker owns
request-local codec generation and Code2Wav owns waveform state and
reference-voice prompt features. `minicpmo_4_5.yaml` remains the stable
single-GPU entry point; `minicpmo_4_5_2gpu.yaml` is the recommended
two-GPU profile. The removed fused two-stage implementation is not retained as
a fallback because it would duplicate state machines and correctness paths.

### Graph execution

Graph boundaries follow each stage's state model:

| Stage | Graph mode on Ascend | Eager boundary |
| --- | --- | --- |
| 0 Thinker | vLLM `PIECEWISE` | multimodal preprocessing and output routing |
| 1 Talker | vLLM `PIECEWISE` | conditioning preprocessing, codec sampling, request-state updates |
| 2 Code2Wav | inner exact-shape NPUGraph for the CFM DiT estimator | encoder, timestep embedding, HiFT/RNG, request parsing, stream-state commit |

Stage 2 keeps `enforce_eager: true` for the generation runner because its
Python request metadata and per-request cache dictionaries are not valid
outer-graph inputs. The backend captures only the deterministic CFM DiT
estimator, with an exact graph key for each tensor shape. Timestep embedding
stays eager because the upstream implementation creates a CPU frequency tensor;
HiFT stays eager because it generates random phase. The graph cache stores up
to 32 entries by default, after which unseen shapes run eagerly.

An ACL graph capture failure is fatal to that Stage-2 process because older
torch-npu releases can leave allocator and RNG capture state invalid. Restart
the service after a capture failure. To run without capture, set
`code2wav_enable_npu_graph: false` under the Stage-2
`platforms.npu.stages[].additional_config` block before startup. Tune the cache
limit there with `code2wav_max_npu_graphs`. Graph mode also requires
`ASCEND_LAUNCH_BLOCKING` to be unset or set to `0`.

The inner NPUGraph is independent of the outer runner setting, so do not use a
global `--enforce-eager` override when Stage 0/1 `PIECEWISE` replay is desired.

## GPU

### 1 x GPU (default — single command)

The default
[`vllm_omni/deploy/minicpmo_4_5.yaml`](../../vllm_omni/deploy/minicpmo_4_5.yaml)
co-locates Thinker, codec-only Talker, and Code2Wav on GPU 0. Their
`gpu_memory_utilization` budgets are 0.55, 0.15, and 0.15. The remaining
device memory is left available for runtime workspaces such as the HiFi-GAN
vocoder's cuDNN kernels. All stages admit
up to four sequences. Startup video profiling is bounded to 32 frames per
video. Use the two-GPU profile for production throughput.

#### Environment

- OS: Linux
- Python: 3.10+
- vLLM / vLLM-Omni: >= 0.21.0 (or current `main`)
- Optional Talker dep: `stepaudio2-minicpmo` (see Notes for why this is
  required and how to install it)

#### Command

```bash
vllm serve openbmb/MiniCPM-o-4_5 --omni \
    --trust-remote-code \
    --chat-template vllm_omni/transformers_utils/chat_templates/minicpmo45_native.jinja \
    --chat-template-content-format openai \
    --host 0.0.0.0 --port 8099
```

Run these commands from the repository root. The bundled chat template
matches the Hugging Face `chat(omni_mode=True)` content layout: media and text
parts are concatenated without adding separators, while whitespace explicitly
included in a text part is preserved. This matters for speech generation because
an inserted newline changes the Thinker condition passed to Talker. For image
questions, place the image before the question, as in the native image-chat
layout. Continue to request `use_tts_template: true` and
`enable_thinking: false` for spoken answers.

The deploy config is auto-loaded by the model registry — no
`--deploy-config` flag needed for this default single-GPU layout.

For the recommended two-GPU layout, add:

```bash
--deploy-config vllm_omni/deploy/minicpmo_4_5_2gpu.yaml
```

#### Performance comparison

Compare text-only and text+audio separately. Text-only isolates Thinker
generation; text+audio also schedules Talker and Code2Wav. The following full
Daily-Omni runs used the same two GPUs, 1197 samples, concurrency 10, and
identical `enable_thinking=false` / `use_tts_template=true` request settings.
The `origin/main` fused Talker ran eager because its graph capture copied an
unpinned CPU metadata tensor.

| Metric | `origin/main` two-stage | Three-stage batching |
| --- | ---: | ---: |
| Accuracy | 64.83% | 64.83% |
| Throughput | 0.62 req/s | 1.97 req/s |
| Mean E2EL | 16.17 s | 5.07 s |
| Mean serving TTFT | 0.92 s | 1.28 s |
| Mean audio TTFP | 16.17 s | 3.24 s |
| Mean audio RTF | 5.97 | 2.11 |
| Stage 0 mean TPOT / ITL | 8.27 / 8.27 ms | 40.08 / 40.11 ms |
| Stage 0 median TPOT / ITL | 7.23 / 7.24 ms | 7.43 / 7.53 ms |

The split pipeline improves throughput 3.19x and lowers audio TTFP by 80%.
Isolating the Thinker on GPU 0 also removes the prior single-GPU TPOT
regression: 40.08 ms is slightly better than the pre-rebase report (~44 ms).
Its median TPOT is effectively the same as main; the higher mean is queueing
tail latency because this profile bounds each stage to four sequences while
main's Thinker admits 16. Global TPOT/ITL remains zero when serving emits text
as one aggregated chunk, so the table reports Stage 0 metrics.

#### Verification

**Health and model listing**:

```bash
curl http://localhost:8099/health
curl http://localhost:8099/v1/models
```

**Quick smoke test (text-only output)**:

```bash
curl http://localhost:8099/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "openbmb/MiniCPM-o-4_5",
        "messages": [{"role": "user", "content": "Briefly introduce yourself."}],
        "modalities": ["text"]
    }'
```

**One duplex turn over the Realtime WebSocket** (text + speech stream back
while the session keeps listening):

```bash
python examples/online_serving/minicpmo/realtime_duplex_demo.py \
    --url ws://localhost:8099/v1/realtime?duplex=1 \
    --model openbmb/MiniCPM-o-4_5 \
    --input-wav /path/to/input_16k_mono_pcm16.wav \
    --ref-audio /path/to/MiniCPM-o-Demo/assets/ref_audio/ref_minicpm_signature.wav \
    --output-dir /tmp/minicpmo_realtime_duplex_demo
```

The demo prints transcript deltas as they arrive and saves the streamed
24 kHz audio to WAV files. The same session shape is available from Python
through `vllm_omni.clients.duplex.DuplexClient` (over the server) or
`vllm_omni.clients.inline_duplex.InlineDuplexClient` (in-process, no server;
`examples/online_serving/barge_in_client.py --inline`).

**Barge-in**: `examples/online_serving/barge_in_client.py` plays a question,
interrupts the answer with a second WAV and checks that the first response is
cancelled and the second one is answered.

**Text + speech in one response over `/v1/chat/completions`** (the headline
4.5 feature, served by the ordinary chat service on the duplex engine). The
model bridge conditions the Talker from the generated assistant span, so the
generic serving layer does not inject MiniCPM-specific template defaults.
`use_tts_template=true` remains supported when explicitly requested:

```bash
curl http://localhost:8099/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "openbmb/MiniCPM-o-4_5",
        "messages": [{"role": "user", "content": "Say hello, then introduce vLLM in one sentence."}],
        "modalities": ["text", "audio"],
        "chat_template_kwargs": {"use_tts_template": true}
    }'
```

When using the OpenAI Python SDK, the same flag can also be sent as
`extra_body={"chat_template_kwargs": {"use_tts_template": True}}`
because the client merges `extra_body` into the request root.

Response carries text in one choice's `message.content` and base64 WAV
in another choice's `message.audio.data` (24 kHz mono, see Notes). With
`modalities: ["text", "audio"]` you typically get two `choices` entries
(one text, one audio).

**Streaming text + speech** (use `--stream`):

```bash
python examples/online_serving/minicpmo/openai_chat_completion_client_for_multimodal_generation.py \
    --query-type text \
    --prompt "Say hello, then introduce vLLM in one sentence." \
    --port 8099 \
    --stream
```

The client prints text deltas as they arrive and saves streamed audio chunks
to WAV files.

**Gradio demo (text + image + audio + video UI)**:

```bash
bash examples/online_serving/minicpmo/run_gradio_demo.sh
# or run the python entry point directly:
python examples/online_serving/minicpmo/gradio_demo.py \
    --minicpmo45-api-base http://localhost:8099/v1 \
    --minicpmo45-model openbmb/MiniCPM-o-4_5 \
    --port 7862
```

Open `http://<host>:7862` and try a text prompt with the **"Generate
speech output (TTS)"** checkbox on / off.

**Turn-based generation** is also available offline
(`examples/offline_inference/minicpmo/`, `Omni.generate`).

The Daily-Omni numbers above were measured on the `/v1/chat/completions`
route.

#### Notes

- Memory budget: Thinker, Talker, and Code2Wav reserve 0.55, 0.15, and
  0.15 of GPU 0. The larger Thinker share protects its multimodal KV cache,
  while the unreserved memory remains available to runtime kernels; all three
  model processes still share one CUDA device.
- `--trust-remote-code` is required — the HF repo ships a custom
  `MiniCPMO` config / model class.
- Stage 0 Thinker and Stage 1 Talker enable vLLM CUDA Graphs. Stage 2 keeps its
  request orchestration eager while using inner CFM and HiFT CUDA Graphs. On
  Ascend, the exact-shape CFM DiT estimator instead uses inner NPUGraph replay.
- All default stages use `max_num_seqs: 4` to reduce cross-process GPU
  contention. Talker AR
  state and Code2Wav caches are request-owned; Code2Wav batches only
  exact-shape-compatible chunks and does not fall back to serial decode.
- `limit_mm_per_prompt.video.num_frames: 32` bounds startup dummy profiling,
  not runtime media decoding. Use `media_io_kwargs.video.num_frames` when a
  matching URL/file video sampling limit is required.
- `StageRequestStats.batch_size` is a request-scoped placeholder, not the
  scheduler's execution batch.
- Single-GPU co-location trades throughput for hardware density: Stage 0/1
  CUDA Graph replay and Stage 2 vocoder kernels compete across three CUDA
  contexts. Use the 8x4090 config or a custom multi-GPU mapping for
  throughput-sensitive serving.

### 8 x RTX 4090 24GB (consumer-GPU layout)

Use
[`vllm_omni/deploy/minicpmo_4_5_8x4090.yaml`](../../vllm_omni/deploy/minicpmo_4_5_8x4090.yaml)
on an 8x RTX 4090 host. Thinker uses 4-way TP across GPUs 0–3
(`~85 %` mem each ≈ 20.4 GiB/card), Talker uses GPU 4, and Code2Wav
uses GPU 5. GPUs 6–7 are left free.

#### Command

```bash
vllm serve openbmb/MiniCPM-o-4_5 --omni \
    --deploy-config vllm_omni/deploy/minicpmo_4_5_8x4090.yaml \
    --trust-remote-code \
    --chat-template vllm_omni/transformers_utils/chat_templates/minicpmo45_native.jinja \
    --chat-template-content-format openai \
    --host 0.0.0.0 --port 8099
```

#### Notes

- `max_model_len` is capped at 4096 in this layout — 8192 still OOMs on
  4090s. Raise it if your cards have more headroom (e.g. 4090 D /
  custom 32 GB SKUs), but verify with a long-prompt run before
  promoting.
- All other knobs match the single-GPU section; the only difference is
  the per-card memory pressure on the thinker shards.

## Notes (applies to all layouts)

- **Code2Wav dependency**: Stage 2 loads `Token2wav` from the
  MiniCPM-o-flavored
  vocoder (PyPI package `stepaudio2-minicpmo` — NOT the upstream
  `stepfun-ai/Step-Audio2`, whose `Token2wav.__init__` signature
  rejects `n_timesteps`). Install via the published extra:

  ```bash
  pip install stepaudio2-minicpmo
  ```

  A missing dep raises `ImportError` at first request with the same
  install hint instead of silently emitting empty audio.

- **TTS conditioning**: the MiniCPM stage bridge can condition speech from
  the generated assistant span without changing shared serving code.
  `chat_template_kwargs.use_tts_template=true` remains supported when an
  explicit `<|tts_bos|>` boundary is desired. For **curl**, put
  `chat_template_kwargs` at the request root; the OpenAI Python SDK may use
  `extra_body` because it flattens those fields into the root.

- **Reference voice**: request audio is carried on the first codec chunk.
  Code2Wav owns the temporary prompt WAV and prompt-feature cache, and removes
  both when the stream ends.

- **Talker sampling**: codec-token sampling reads the checkpoint `tts_config`
  and defaults to deterministic seed 42. Stage-1 deploy sampling parameters
  control only vLLM's binary continue/stop token.

- **Output audio**: 24 kHz mono. On `/v1/chat/completions` it is base64 WAV
  inside the OpenAI-style `message.audio.data` (the Gradio demo's WAV player
  decodes this automatically); on a duplex session it streams as base64
  `response.audio.delta` events; offline `Omni.generate` returns WAV.
- **Response choices**: on `/v1/chat/completions` text and audio are separate
  choices. SDK clients should select the choice whose `message.audio.data` is
  populated rather than assuming `choices[0]` contains audio.

- **Routing**: MiniCPM-o 4.5 and 2.6 both ship `architectures=
  ["MiniCPMO"]` in HF config; routing is disambiguated by
  `hf_config.version == "4.5"` via the
  `hf_config_predicate` on the 4.5 pipeline. A 2.6 checkpoint loaded
  with this recipe's `--deploy-config` will be rejected at startup
  rather than silently misrouted.

- **Async chunking**: enabled in all deploy configs. Talker sends
  25-code chunks with three-code left context to Code2Wav through
  `SharedMemoryConnector`; terminal chunks flush held lookahead state.