vllm-omni/recipes/OpenBMB/MiniCPM-o-4_5.md

MiniCPM-o 4.5

Online serving and offline inference for omni multimodal chat (text / image / audio / video → text + 24 kHz speech)

Summary

• Vendor: OpenBMB
• Model: openbmb/MiniCPM-o-4_5
• Task: Omni multimodal chat — accepts text / image / audio / video input; emits text and 24 kHz mono speech in the same response
• Mode: Online serving via the OpenAI-compatible /v1/chat/completions API (plus Gradio demo), and offline inference via Omni.generate
• Maintainer: @tc-mb (MiniCPM-V / MiniCPM-o team)

When to use this recipe

Use this recipe as a known-good starting point for serving openbmb/MiniCPM-o-4_5 on vLLM-Omni. MiniCPM-o 4.5 is the omni member of the MiniCPM-o family — it runs a multimodal thinker, a streaming MiniCPMTTS codec talker, and a separate batched Code2Wav stage so a single /v1/chat/completions call can return text and 24 kHz speech in one shot. The recommended batching deploy isolates the Thinker on GPU 0 and co-locates Talker and Code2Wav on GPU 1; 1-GPU, 3-GPU, and 8x4090 layouts are also provided.

References

• Default deploy configs (auto-loaded by HF model_type=minicpmo + hf_config.version="4.5"):
  • Default single-GPU compatibility layout (auto-loaded): vllm_omni/deploy/minicpmo_4_5.yaml
  • Recommended 2-GPU continuous-batching layout: vllm_omni/deploy/minicpmo_4_5_2gpu.yaml,
  • 3-GPU layout: vllm_omni/deploy/minicpmo_4_5_3gpu.yaml
  • 8x RTX 4090 layout: vllm_omni/deploy/minicpmo_4_5_8x4090.yaml
• Online example + Gradio demo: examples/online_serving/minicpmo/
• Offline end-to-end example: examples/offline_inference/minicpmo/
• Pipeline / talker source: vllm_omni/model_executor/models/minicpmo_4_5/
• Stage-input processors (thinker → talker and talker → Code2Wav): vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py
• Upstream model card: openbmb/MiniCPM-o-4_5
• Integration PR: vllm-project/vllm-omni#3642

Hardware Support

Four hardware layouts ship with deploy configs. Every layout uses the same strict three-stage topology. The Talker emits codec chunks only; Code2Wav consumes them through a shared-memory async connector.

Layout	Thinker	Talker	Code2Wav	Typical hardware
1-GPU (default)	GPU 0	GPU 0	GPU 0	1x large-memory GPU
2-GPU	GPU 0	GPU 1	GPU 1	2x large-memory GPU
3-GPU	GPU 0	GPU 1	GPU 2	3x GPU
8x RTX 4090 24GB	GPU 0–3 (TP=4)	GPU 4	GPU 5	8x RTX 4090 consumer

Migration from the fused deployment

MiniCPM-o 4.5 now requires the three-stage topology: the Talker owns request-local codec generation and Code2Wav owns waveform state and reference-voice prompt features. minicpmo_4_5.yaml remains the stable single-GPU entry point; minicpmo_4_5_2gpu.yaml is the recommended two-GPU profile. The removed fused two-stage implementation is not retained as a fallback because it would duplicate state machines and correctness paths.

Graph execution

Graph boundaries follow each stage's state model:

Stage	Graph mode on Ascend	Eager boundary
0 Thinker	vLLM PIECEWISE	multimodal preprocessing and output routing
1 Talker	vLLM PIECEWISE	conditioning preprocessing, codec sampling, request-state updates
2 Code2Wav	inner exact-shape NPUGraph for the CFM DiT estimator	encoder, timestep embedding, HiFT/RNG, request parsing, stream-state commit

Stage 2 keeps enforce_eager: true for the generation runner because its Python request metadata and per-request cache dictionaries are not valid outer-graph inputs. The backend captures only the deterministic CFM DiT estimator, with an exact graph key for each tensor shape. Timestep embedding stays eager because the upstream implementation creates a CPU frequency tensor; HiFT stays eager because it generates random phase. The graph cache stores up to 32 entries by default, after which unseen shapes run eagerly.

An ACL graph capture failure is fatal to that Stage-2 process because older torch-npu releases can leave allocator and RNG capture state invalid. Restart the service after a capture failure. To run without capture, set code2wav_enable_npu_graph: false under the Stage-2 platforms.npu.stages[].additional_config block before startup. Tune the cache limit there with code2wav_max_npu_graphs. Graph mode also requires ASCEND_LAUNCH_BLOCKING to be unset or set to 0.

The inner NPUGraph is independent of the outer runner setting, so do not use a global --enforce-eager override when Stage 0/1 PIECEWISE replay is desired.

GPU

1 x GPU (default — single command)

The default vllm_omni/deploy/minicpmo_4_5.yaml co-locates Thinker, codec-only Talker, and Code2Wav on GPU 0. Their gpu_memory_utilization budgets are 0.55, 0.15, and 0.15. The remaining device memory is left available for runtime workspaces such as the HiFi-GAN vocoder's cuDNN kernels. All stages admit up to four sequences. Startup video profiling is bounded to 32 frames per video. Use the two-GPU profile for production throughput.

Environment

• OS: Linux
• Python: 3.10+
• vLLM / vLLM-Omni: >= 0.21.0 (or current main)
• Optional Talker dep: stepaudio2-minicpmo (see Notes for why this is required and how to install it)

Command

vllm serve openbmb/MiniCPM-o-4_5 --omni \
    --trust-remote-code \
    --host 0.0.0.0 --port 8099

The deploy config is auto-loaded by the model registry — no --deploy-config flag needed for this default single-GPU layout.

For the recommended two-GPU layout, add:

--deploy-config vllm_omni/deploy/minicpmo_4_5_2gpu.yaml

Performance comparison

Compare text-only and text+audio separately. Text-only isolates Thinker generation; text+audio also schedules Talker and Code2Wav. The following full Daily-Omni runs used the same two GPUs, 1197 samples, concurrency 10, and identical enable_thinking=false / use_tts_template=true request settings. The origin/main fused Talker ran eager because its graph capture copied an unpinned CPU metadata tensor.

Metric	origin/main two-stage	Three-stage batching
Accuracy	64.83%	64.83%
Throughput	0.62 req/s	1.97 req/s
Mean E2EL	16.17 s	5.07 s
Mean serving TTFT	0.92 s	1.28 s
Mean audio TTFP	16.17 s	3.24 s
Mean audio RTF	5.97	2.11
Stage 0 mean TPOT / ITL	8.27 / 8.27 ms	40.08 / 40.11 ms
Stage 0 median TPOT / ITL	7.23 / 7.24 ms	7.43 / 7.53 ms

The split pipeline improves throughput 3.19x and lowers audio TTFP by 80%. Isolating the Thinker on GPU 0 also removes the prior single-GPU TPOT regression: 40.08 ms is slightly better than the pre-rebase report (~44 ms). Its median TPOT is effectively the same as main; the higher mean is queueing tail latency because this profile bounds each stage to four sequences while main's Thinker admits 16. Global TPOT/ITL remains zero when serving emits text as one aggregated chunk, so the table reports Stage 0 metrics.

Verification

Quick smoke test (text-only output):

curl http://localhost:8099/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "openbmb/MiniCPM-o-4_5",
        "messages": [{"role": "user", "content": "Briefly introduce yourself."}],
        "modalities": ["text"]
    }'

Text + speech in one response (the headline 4.5 feature). The model bridge conditions the Talker from the generated assistant span, so the generic serving layer does not inject MiniCPM-specific template defaults. use_tts_template=true remains supported when explicitly requested:

curl http://localhost:8099/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "openbmb/MiniCPM-o-4_5",
        "messages": [{"role": "user", "content": "Say hello, then introduce vLLM in one sentence."}],
        "modalities": ["text", "audio"],
        "chat_template_kwargs": {"use_tts_template": true}
    }'

When using the OpenAI Python SDK, the same flag can also be sent as extra_body={"chat_template_kwargs": {"use_tts_template": True}} because the client merges extra_body into the request root.

Response carries text in one choice's message.content and base64 WAV in another choice's message.audio.data (24 kHz mono, see Notes). With modalities: ["text", "audio"] you typically get two choices entries (one text, one audio).

Streaming text + speech (use --stream):

python examples/online_serving/minicpmo/openai_chat_completion_client_for_multimodal_generation.py \
    --query-type text \
    --prompt "Say hello, then introduce vLLM in one sentence." \
    --port 8099 \
    --stream

The client prints text deltas as they arrive and saves streamed audio chunks to WAV files.

Gradio demo (text + image + audio + video UI):

bash examples/online_serving/minicpmo/run_gradio_demo.sh
# or run the python entry point directly:
python examples/online_serving/minicpmo/gradio_demo.py \
    --minicpmo45-api-base http://localhost:8099/v1 \
    --minicpmo45-model openbmb/MiniCPM-o-4_5 \
    --port 7862

Open http://<host>:7862 and try a text prompt with the "Generate speech output (TTS)" checkbox on / off.

Notes

• Memory budget: Thinker, Talker, and Code2Wav reserve 0.55, 0.15, and 0.15 of GPU 0. The larger Thinker share protects its multimodal KV cache, while the unreserved memory remains available to runtime kernels; all three model processes still share one CUDA device.
• --trust-remote-code is required — the HF repo ships a custom MiniCPMO config / model class.
• Stage 0 Thinker and Stage 1 Talker enable vLLM CUDA Graphs. Stage 2 keeps its request orchestration eager while using inner CFM and HiFT CUDA Graphs. On Ascend, the exact-shape CFM DiT estimator instead uses inner NPUGraph replay.
• All default stages use max_num_seqs: 4 to reduce cross-process GPU contention. Talker AR state and Code2Wav caches are request-owned; Code2Wav batches only exact-shape-compatible chunks and does not fall back to serial decode.
• limit_mm_per_prompt.video.num_frames: 32 bounds startup dummy profiling, not runtime media decoding. Use media_io_kwargs.video.num_frames when a matching URL/file video sampling limit is required.
• StageRequestStats.batch_size is a request-scoped placeholder, not the scheduler's execution batch.
• Single-GPU co-location trades throughput for hardware density: Stage 0/1 CUDA Graph replay and Stage 2 vocoder kernels compete across three CUDA contexts. Use the 8x4090 config or a custom multi-GPU mapping for throughput-sensitive serving.

8 x RTX 4090 24GB (consumer-GPU layout)

Use vllm_omni/deploy/minicpmo_4_5_8x4090.yaml on an 8x RTX 4090 host. Thinker uses 4-way TP across GPUs 0–3 (~85 % mem each ≈ 20.4 GiB/card), Talker uses GPU 4, and Code2Wav uses GPU 5. GPUs 6–7 are left free.

Command

vllm serve openbmb/MiniCPM-o-4_5 --omni \
    --deploy-config vllm_omni/deploy/minicpmo_4_5_8x4090.yaml \
    --trust-remote-code \
    --host 0.0.0.0 --port 8099

Notes

• max_model_len is capped at 4096 in this layout — 8192 still OOMs on 4090s. Raise it if your cards have more headroom (e.g. 4090 D / custom 32 GB SKUs), but verify with a long-prompt run before promoting.
• All other knobs match the single-GPU section; the only difference is the per-card memory pressure on the thinker shards.

Notes (applies to all layouts)

• Code2Wav dependency: Stage 2 loads Token2wav from the MiniCPM-o-flavored vocoder (PyPI package stepaudio2-minicpmo — NOT the upstream stepfun-ai/Step-Audio2, whose Token2wav.__init__ signature rejects n_timesteps). Install via the published extra:

  pip install stepaudio2-minicpmo
  

  A missing dep raises ImportError at first request with the same install hint instead of silently emitting empty audio.

• TTS conditioning: the MiniCPM stage bridge can condition speech from the generated assistant span without changing shared serving code. chat_template_kwargs.use_tts_template=true remains supported when an explicit <|tts_bos|> boundary is desired. For curl, put chat_template_kwargs at the request root; the OpenAI Python SDK may use extra_body because it flattens those fields into the root.

• Reference voice: request audio is carried on the first codec chunk. Code2Wav owns the temporary prompt WAV and prompt-feature cache, and removes both when the stream ends.

• Talker sampling: codec-token sampling reads the checkpoint tts_config and defaults to deterministic seed 42. Stage-1 deploy sampling parameters control only vLLM's binary continue/stop token.

• Output audio: 24 kHz mono WAV inside the OpenAI-style message.audio.data (base64). The Gradio demo's WAV player decodes this automatically.

• Routing: MiniCPM-o 4.5 and 2.6 both ship architectures= ["MiniCPMO"] in HF config; routing is disambiguated by hf_config.version == "4.5" via the hf_config_predicate on the 4.5 pipeline. A 2.6 checkpoint loaded with this recipe's --deploy-config will be rejected at startup rather than silently misrouted.

• Async chunking: enabled in all deploy configs. Talker sends 25-code chunks with three-code left context to Code2Wav through SharedMemoryConnector; terminal chunks flush held lookahead state.

• Response choices: text and audio are separate choices. SDK clients should select the choice whose message.audio.data is populated rather than assuming choices[0] contains audio.