diff --git a/examples/online_serving/minicpmo/README.md b/examples/online_serving/minicpmo/README.md index 148fbb5e2..134ddd62c 100644 --- a/examples/online_serving/minicpmo/README.md +++ b/examples/online_serving/minicpmo/README.md @@ -23,10 +23,11 @@ including `librosa`. The deploy config auto-loads via `--omni`. The default `vllm_omni/deploy/minicpmo_4_5.yaml` keeps all three stages on -logical device 0 with memory budgets of 65%, 15%, and 15%. For throughput, -`minicpmo_4_5_batching.yaml` gives the Thinker GPU 0 (90%) and colocates the -Talker (55%) and Code2Wav (35%) on GPU 1. Each stage admits at most four -concurrent sequences. +logical device 0 with memory budgets of 55%, 22%, and 22%. The profile admits +at most four sequences per stage and bounds startup video profiling to 32 +frames per video. For throughput,`minicpmo_4_5_batching.yaml` gives the Thinker +GPU 0 (90%) and colocates the Talker (55%) and Code2Wav (35%) on GPU 1. That +profile admits at most four concurrent sequences per stage. | deploy config | GPUs | Notes | |---|---|---| @@ -60,7 +61,7 @@ checkpoint path. To start the experimental native duplex backend, use ```bash vllm serve openbmb/MiniCPM-o-4_5 --omni --trust-remote-code --port 8099 \ - --stage-overrides '{"0": {"gpu_memory_utilization": 0.65}}' + --stage-overrides '{"0": {"gpu_memory_utilization": 0.55}}' ``` ## Send multimodal requests diff --git a/recipes/OpenBMB/MiniCPM-o-4_5.md b/recipes/OpenBMB/MiniCPM-o-4_5.md index dddc9fac4..97b3c6d6d 100644 --- a/recipes/OpenBMB/MiniCPM-o-4_5.md +++ b/recipes/OpenBMB/MiniCPM-o-4_5.md @@ -79,9 +79,9 @@ a fallback because it would duplicate state machines and correctness paths. The default [`vllm_omni/deploy/minicpmo_4_5.yaml`](../../vllm_omni/deploy/minicpmo_4_5.yaml) co-locates Thinker, codec-only Talker, and Code2Wav on GPU 0. Their -`gpu_memory_utilization` budgets are 0.65, 0.15, and 0.15. This layout -minimizes the GPU count; use a large-memory accelerator and leave the -remaining 5% for runtime overhead. +`gpu_memory_utilization` budgets are 0.55, 0.22, and 0.22. All stages admit +up to four sequences. Startup video profiling is bounded to 32 frames per +video. Use the two-GPU profile for production throughput. #### Environment @@ -202,8 +202,8 @@ speech output (TTS)"** checkbox on / off. #### Notes -- Memory budget: Thinker, Talker, and Code2Wav reserve 0.65, 0.15, and - 0.15 of GPU 0. The larger Thinker share protects its multimodal KV cache; +- Memory budget: Thinker, Talker, and Code2Wav reserve 0.55, 0.22, and + 0.22 of GPU 0. The larger Thinker share protects its multimodal KV cache; all three model processes still share one CUDA device. - `--trust-remote-code` is required — the HF repo ships a custom `MiniCPMO` config / model class. @@ -214,6 +214,9 @@ speech output (TTS)"** checkbox on / off. contention. Talker AR state and Code2Wav caches are request-owned; Code2Wav batches only exact-shape-compatible chunks and does not fall back to serial decode. +- `limit_mm_per_prompt.video.num_frames: 32` bounds startup dummy profiling, + not runtime media decoding. Use `media_io_kwargs.video.num_frames` when a + matching URL/file video sampling limit is required. - `StageRequestStats.batch_size` is a request-scoped placeholder, not the scheduler's execution batch. - Single-GPU co-location trades throughput for hardware density: Stage 0/1 diff --git a/tests/model_executor/models/minicpmo_4_5/test_pipeline.py b/tests/model_executor/models/minicpmo_4_5/test_pipeline.py index 1b8bd669f..968df04a9 100644 --- a/tests/model_executor/models/minicpmo_4_5/test_pipeline.py +++ b/tests/model_executor/models/minicpmo_4_5/test_pipeline.py @@ -159,10 +159,11 @@ class TestDeployTopology: if filename == "minicpmo_4_5.yaml": assert [stage.yaml_engine_args["max_num_seqs"] for stage in stages] == [4, 4, 4] assert [stage.yaml_engine_args["gpu_memory_utilization"] for stage in stages] == [ - 0.65, - 0.15, - 0.15, + 0.55, + 0.22, + 0.22, ] + assert stages[0].yaml_engine_args["limit_mm_per_prompt"] == {"video": {"count": 1, "num_frames": 32}} elif filename in {"minicpmo_4_5_batching.yaml", "minicpmo_4_5_2gpu.yaml"}: assert [stage.yaml_engine_args["gpu_memory_utilization"] for stage in stages] == [ 0.9, diff --git a/vllm_omni/deploy/minicpmo_4_5.yaml b/vllm_omni/deploy/minicpmo_4_5.yaml index da84667af..ea1fdaf2f 100644 --- a/vllm_omni/deploy/minicpmo_4_5.yaml +++ b/vllm_omni/deploy/minicpmo_4_5.yaml @@ -18,13 +18,15 @@ connectors: stages: - stage_id: 0 max_num_seqs: 4 - gpu_memory_utilization: 0.65 + gpu_memory_utilization: 0.55 enforce_eager: false tensor_parallel_size: 1 max_num_batched_tokens: 16384 devices: "0" limit_mm_per_prompt: - video: 1 + video: + count: 1 + num_frames: 32 default_sampling_params: temperature: 0.0 top_p: 1.0 @@ -36,7 +38,7 @@ stages: - stage_id: 1 max_num_seqs: 4 - gpu_memory_utilization: 0.15 + gpu_memory_utilization: 0.22 enforce_eager: false max_num_batched_tokens: 8192 devices: "0" @@ -56,7 +58,7 @@ stages: - stage_id: 2 max_num_seqs: 4 - gpu_memory_utilization: 0.15 + gpu_memory_utilization: 0.22 enforce_eager: true enable_chunked_prefill: false max_num_batched_tokens: 65536