vllm-omni/docs/analysis/minicpmo_edge_profile.md

MiniCPM-o 4.5 on the edge: measured profile (2026-09-11)

Companion to mobile_htp_simulation.md (Qwen3-TTS) using the same instruments: vLLM-Omni's StepStats for per-stage attribution, the Xeon 8480C for CPU, one H200 through canhazgpu for GPU, and Qualcomm AI Hub for real Snapdragon devices. Scripts and raw results: experiments/edge_engine_compare/minicpm/ and experiments/edge_engine_compare/results/minicpmo/.

1. What this model is, and why the shape matters

MiniCPM-o 4.5 is a three-stage pipeline in vLLM-Omni (MINICPMO_4_5_PIPELINE): an 8 B omni LLM (SigLIP vision tower + Whisper audio encoder + text backbone) → a TTS head → Code2Wav. Weights are local (19 GB fp16, ≈9.5 B parameters).

The speech half is much lighter than the thinker, and much lighter than Qwen3-TTS's:

component	parameters	rate
TTS head (Llama-style decoder, 20 layers × 768, 12 heads, no GQA, no q/k-norm)	194 M	1 token per 40 ms (25 Hz)
Code2Wav flow matching (CFM DiT, Step-Audio2 / CosyVoice2 lineage)	156 M	10 denoising steps per chunk
Code2Wav HiFT generator	21 M	1 chunk = 25 tokens = 1.0 s of audio
thinker (vision + audio + text, 8 B)	≈9.2 B	prompt-rate

num_vq = 1: one codebook, so one decode step per speech token. Qwen3-TTS emits 16 codebooks at 12.5 Hz, which costs one talker step plus fifteen code-predictor sub-steps per 80 ms frame. That single architectural choice is worth 3.5× on a phone (§4). The vocoder is also genuinely streaming already — 3 tokens of lookahead, persistent flow and HiFT caches — which is the design Qwen3-TTS's 72-frame-context vocoder still lacks.

2. GPU (one H200, 6 concurrent requests, text → speech)

Deploy vllm_omni/deploy/minicpmo_4_5.yaml (all three stages on one card), 6 prompts from benchmarks/tts/prompts_12.txt, max_tokens 192 for the thinker and the stage-1 deploy defaults for the TTS head.

metric	value
engine init (warm caches)	253 s
first text	p50 393 ms, p90 598 ms
complete audio per request	p50 5.81 s, p90 7.60 s (offline path yields the whole waveform, so this is completion, not first-chunk latency)
batch throughput	30.2 s of audio in 22.6 s wall → RTF 0.748 at concurrency 6

Per-stage step attribution (mean over the run):

stage	steps	mean step	mean model forward
0 · thinker	46	9.66 ms	2.31 ms
1 · TTS head	274	6.53 ms	1.72 ms
2 · Code2Wav	469	14.88 ms	–

The largest single number in the table is not compute: stage 2's hop.put_to_get averages 3.1 s, i.e. codec chunks wait in the shared-memory connector while the vocoder works through its backlog. On one card the vocoder is the queue.

3. CPU (Xeon 8480C, pinned)

stage	8 threads	16 threads	real-time factor at 16 threads
TTS head, per 40 ms token	17.1 ms	11.6 ms	0.29
Code2Wav, per 1.0 s of audio	1033 ms	833 ms	0.83
one-time speaker setup (x-vector + prompt mel + caches)	11.7 s	9.9 s	cacheable per voice

Inside the vocoder chunk the split is lopsided, and in the opposite direction from Qwen3-TTS:

half	per 1.0 s of audio, 16 threads	share
flow matching (10-step CFM DiT)	652 ms	91 %
HiFT generator	65 ms	9 %

Qwen3-TTS's vocoder was elementwise-bound (76 % in Snake activations on sample-rate tensors) and therefore wanted a GPU; MiniCPM-o's is matmul-bound in a DiT and therefore suits an NPU. The two models need opposite placements.

The step count is a config field (tts_config.s3_stream_n_timesteps), and it trades linearly:

flow steps	per 1.0 s of audio	mel fidelity vs the 10-step default
10 (default)	542 ms	reference
6	341 ms	30.6 dB
4	236 ms	16.5 dB
2	127 ms	9.6 dB

Six steps is a 1.6× vocoder speed-up at 30.6 dB on the mel spectrogram — worth a listening test, since it costs nothing but a config change.

4. Real Snapdragon devices (Qualcomm AI Hub)

The TTS head exported through vllm_omni/edge/qnn_export.py (Llama-style path: qk_norm=False, weight-normalised code head), 256-token KV cache, QNN context binary, every operator on the Hexagon NPU.

build	Galaxy S25 (8 Elite)	Galaxy S24 (8 Gen 3)
fp16, per 40 ms token	7.08 ms	8.18 ms
fp16 fidelity vs torch fp32	48.9 dB, top-1 token matches	49.2 dB, top-1 matches
int8 weights (w8a16), per token	4.88 ms	6.15 ms
int8 fidelity	2.3 dB, top-1 differs	0.1 dB, top-1 differs

Per second of speech, the comparison with Qwen3-TTS on the same phone is the headline:

model, S25 NPU fp16	per second of audio
MiniCPM-o 4.5 (1 codebook @ 25 Hz, 194 M)	177 ms
Qwen3-TTS 0.6B (16 codebooks @ 12.5 Hz: talker 15.8 + predictor 33.9 per 80 ms frame)	621 ms

Int8 fails here too. Per-tensor int8 weights destroy MiniCPM-o's code head exactly as they destroy Qwen3-TTS's code predictor (3.1 dB there, 2.3 dB here) — two unrelated families, same failure, so this is a property of speech-token heads rather than of one model or one runtime. The faster int8 rows stay timing-only.

Not yet measured on device: the Code2Wav flow + HiFT (its streaming caches need the same flattening the KV cache needed), and the 8 B thinker, which at fp16 is 16 GB and does not fit a phone at all — int4 would be mandatory, and §4 is the reason to expect trouble there.

5. Reading

• For a phone, MiniCPM-o's speech stack is the better architecture and Qwen3-TTS's is the better size. One codebook at 25 Hz costs 177 ms of NPU time per second of audio against 621 ms; but the thinker in front of it is 8 B, where Qwen3-TTS's whole pipeline is 0.6 B.
• Placement is model-specific. Qwen3-TTS wants transformer-on-NPU + vocoder-on-GPU. MiniCPM-o's vocoder is a DiT, so the NPU is plausible for both halves; that is the next measurement.
• The cheapest real win is the flow step count, not quantization: 10 → 6 steps is 1.6× at 30.6 dB, while every int8 path measured in this study has been numerically broken.
• Two bugs found while profiling, both in the offline path rather than the model: examples/offline_inference/minicpmo/end2end.py builds two sampling-parameter sets and fails on this three-stage pipeline with Expected 3 sampling params, got 2, and its talker parameters cap the codec LM at max_tokens=1, which silently yields zero audio. The driver in experiments/edge_engine_compare/minicpm/bench_minicpmo.py passes three sets and the stage-1 deploy defaults.