diff --git a/examples/offline_inference/minicpmo/README.md b/examples/offline_inference/minicpmo/README.md index b80fc0aa6..31120773c 100644 --- a/examples/offline_inference/minicpmo/README.md +++ b/examples/offline_inference/minicpmo/README.md @@ -1,8 +1,9 @@ # MiniCPM-o 4.5: Offline inference -Two-stage pipeline: **thinker** (multimodal understanding) → **talker + Token2Wav** +Three-stage pipeline: **Thinker** → **Talker** (codec) → **Code2Wav** (24 kHz speech). Deploy config auto-loads from -`vllm_omni/deploy/minicpmo_4_5.yaml` (2-GPU default). +`vllm_omni/deploy/minicpmo_4_5.yaml` (single-GPU default). For higher +throughput, pass `--deploy-config vllm_omni/deploy/minicpmo_4_5_2gpu.yaml`. ## Setup @@ -72,7 +73,7 @@ Supported `--query-type` values: ```bash python end2end.py --query-type text \ - --deploy-config /path/to/vllm_omni/deploy/minicpmo_4_5_8x4090.yaml + --deploy-config /path/to/vllm_omni/deploy/minicpmo_4_5_2gpu.yaml ``` ## Notes @@ -83,7 +84,8 @@ python end2end.py --query-type text \ - Output WAV is **24 kHz mono**. - Placeholders in the prompt are MiniCPM-style: `(./)`, `()`, `()`. -- Default layout needs **2 GPUs**. Async chunking is off in the bundled YAMLs. +- Default layout needs **1 GPU**. Use `minicpmo_4_5_2gpu.yaml` for the + recommended two-GPU continuous-batching layout. ## Online serving diff --git a/examples/online_serving/minicpmo/README.md b/examples/online_serving/minicpmo/README.md index 2f9929b26..5a9fd2be0 100644 --- a/examples/online_serving/minicpmo/README.md +++ b/examples/online_serving/minicpmo/README.md @@ -23,17 +23,17 @@ including `librosa`. The deploy config auto-loads via `--omni`. The default `vllm_omni/deploy/minicpmo_4_5.yaml` keeps all three stages on -logical device 0 with memory budgets of 55%, 22%, and 22%. The profile admits -at most four sequences per stage and bounds startup video profiling to 32 -frames per video. For throughput, `minicpmo_4_5_batching.yaml` gives the Thinker -GPU 0 (90%) and colocates the Talker (55%) and Code2Wav (35%) on GPU 1. That -profile admits at most four concurrent sequences per stage. +logical device 0 with memory budgets of 55%, 15%, and 18%, leaving headroom +for runtime kernels such as the HiFi-GAN vocoder's cuDNN workspace. The +profile admits at most four sequences per stage. For throughput, +`minicpmo_4_5_2gpu.yaml` gives the Thinker GPU 0 (90%) and colocates the +Talker (55%) and Code2Wav (35%) on GPU 1. That profile admits at most four +concurrent sequences per stage. | deploy config | GPUs | Notes | |---|---|---| | `minicpmo_4_5.yaml` (default) | 1 | Memory-constrained compatibility layout. | -| `minicpmo_4_5_batching.yaml` | 2 | Recommended continuous-batching layout; Talker and Code2Wav share GPU 1. | -| `minicpmo_4_5_2gpu.yaml` | 2 | Backward-compatible alias for `minicpmo_4_5_batching.yaml`. | +| `minicpmo_4_5_2gpu.yaml` | 2 | Recommended continuous-batching layout; Talker and Code2Wav share GPU 1. | | `minicpmo_4_5_3gpu.yaml` | 3 | One GPU per stage. | | `minicpmo_4_5_8x4090.yaml` | 8 | Full 8x4090 layout. | | `minicpmo_4_5_duplex.yaml` | 1 | Experimental native full-duplex overlay. | diff --git a/recipes/OpenBMB/MiniCPM-o-4_5.md b/recipes/OpenBMB/MiniCPM-o-4_5.md index d34376b9c..c643bc225 100644 --- a/recipes/OpenBMB/MiniCPM-o-4_5.md +++ b/recipes/OpenBMB/MiniCPM-o-4_5.md @@ -31,9 +31,7 @@ also provided. - Default single-GPU compatibility layout (auto-loaded): [`vllm_omni/deploy/minicpmo_4_5.yaml`](../../vllm_omni/deploy/minicpmo_4_5.yaml) - Recommended 2-GPU continuous-batching layout: - [`vllm_omni/deploy/minicpmo_4_5_batching.yaml`](../../vllm_omni/deploy/minicpmo_4_5_batching.yaml) - - 2-GPU and 3-GPU layouts: - [`vllm_omni/deploy/minicpmo_4_5_2gpu.yaml`](../../vllm_omni/deploy/minicpmo_4_5_2gpu.yaml), + [`vllm_omni/deploy/minicpmo_4_5_2gpu.yaml`](../../vllm_omni/deploy/minicpmo_4_5_2gpu.yaml) - 3-GPU layout: [`vllm_omni/deploy/minicpmo_4_5_3gpu.yaml`](../../vllm_omni/deploy/minicpmo_4_5_3gpu.yaml) - 8x RTX 4090 layout: @@ -69,8 +67,8 @@ Code2Wav consumes them through a shared-memory async connector. MiniCPM-o 4.5 now requires the three-stage topology: the Talker owns request-local codec generation and Code2Wav owns waveform state and reference-voice prompt features. `minicpmo_4_5.yaml` remains the stable -single-GPU entry point; `minicpmo_4_5_batching.yaml` is the recommended -two-GPU profile. The removed fused two-stage implementation is not retained as +single-GPU entry point; `minicpmo_4_5_2gpu.yaml` is the recommended +two-GPU continuous-batching profile. The removed fused two-stage implementation is not retained as a fallback because it would duplicate state machines and correctness paths. ## GPU @@ -80,9 +78,9 @@ a fallback because it would duplicate state machines and correctness paths. The default [`vllm_omni/deploy/minicpmo_4_5.yaml`](../../vllm_omni/deploy/minicpmo_4_5.yaml) co-locates Thinker, codec-only Talker, and Code2Wav on GPU 0. Their -`gpu_memory_utilization` budgets are 0.55, 0.22, and 0.22. All stages admit -up to four sequences. Startup video profiling is bounded to 32 frames per -video. Use the two-GPU profile for production throughput. +`gpu_memory_utilization` budgets are 0.55, 0.15, and 0.18 so concurrent +Code2Wav activations keep residual headroom. All stages admit up to four +sequences. Use the two-GPU profile for production throughput. #### Environment @@ -106,7 +104,7 @@ The deploy config is auto-loaded by the model registry — no For the recommended two-GPU layout, add: ```bash ---deploy-config vllm_omni/deploy/minicpmo_4_5_batching.yaml +--deploy-config vllm_omni/deploy/minicpmo_4_5_2gpu.yaml ``` #### Performance comparison @@ -205,8 +203,8 @@ speech output (TTS)"** checkbox on / off. #### Notes -- Memory budget: Thinker, Talker, and Code2Wav reserve 0.55, 0.22, and - 0.22 of GPU 0. The larger Thinker share protects its multimodal KV cache; +- Memory budget: Thinker, Talker, and Code2Wav reserve 0.55, 0.15, and + 0.18 of GPU 0 so concurrent Code2Wav activations keep residual headroom; all three model processes still share one CUDA device. - `--trust-remote-code` is required — the HF repo ships a custom `MiniCPMO` config / model class. @@ -217,14 +215,13 @@ speech output (TTS)"** checkbox on / off. contention. Talker AR state and Code2Wav caches are request-owned; Code2Wav batches only exact-shape-compatible chunks and does not fall back to serial decode. -- `limit_mm_per_prompt.video.num_frames: 32` bounds startup dummy profiling, - not runtime media decoding. Use `media_io_kwargs.video.num_frames` when a - matching URL/file video sampling limit is required. +- Stage 0 pins `interleave_mm_strings=true` and + `media_io_kwargs.video={fps:1,num_frames:128}` for Daily-Omni MiniCPM packs. - `StageRequestStats.batch_size` is a request-scoped placeholder, not the scheduler's execution batch. - Single-GPU co-location trades throughput for hardware density: Stage 0/1 CUDA Graph replay and eager Stage 2 vocoder kernels compete across three - CUDA contexts. Use the 8x4090 config or a custom multi-GPU mapping for + CUDA contexts. Use `minicpmo_4_5_2gpu.yaml` or a custom multi-GPU mapping for throughput-sensitive serving. ### 8 x RTX 4090 24GB (consumer-GPU layout) diff --git a/tests/dfx/reliability/invalid_param_test/test_invalid_minicpmo_4_5_omni.py b/tests/dfx/reliability/invalid_param_test/test_invalid_minicpmo_4_5_omni.py index c37b0b26a..df67b9922 100644 --- a/tests/dfx/reliability/invalid_param_test/test_invalid_minicpmo_4_5_omni.py +++ b/tests/dfx/reliability/invalid_param_test/test_invalid_minicpmo_4_5_omni.py @@ -14,7 +14,7 @@ from tests.helpers.stage_config import get_deploy_config_path pytestmark = [pytest.mark.slow, pytest.mark.omni, pytest.mark.full_model] -_MINICPMO_DEPLOY = get_deploy_config_path("minicpmo_4_5_batching.yaml") +_MINICPMO_DEPLOY = get_deploy_config_path("minicpmo_4_5_2gpu.yaml") _MINICPMO_SERVER = [ pytest.param( diff --git a/tests/e2e/online_serving/test_minicpmo_4_5.py b/tests/e2e/online_serving/test_minicpmo_4_5.py index e19ab6a09..f8b65fd0e 100644 --- a/tests/e2e/online_serving/test_minicpmo_4_5.py +++ b/tests/e2e/online_serving/test_minicpmo_4_5.py @@ -19,7 +19,7 @@ _MODEL = "openbmb/MiniCPM-o-4_5" _CI_DEPLOY = modify_stage_config( get_deploy_config_path("minicpmo_4_5.yaml"), updates={ - "stages": {0: {"default_sampling_params.max_tokens": 128}, 1: {"default_sampling_params.max_tokens": 512}} + "stages": {0: {"default_sampling_params.max_tokens": 128}, 1: {"default_sampling_params.max_tokens": 1024}} }, ) diff --git a/tests/model_executor/models/minicpmo_4_5/test_pipeline.py b/tests/model_executor/models/minicpmo_4_5/test_pipeline.py index bce016d67..ae121b118 100644 --- a/tests/model_executor/models/minicpmo_4_5/test_pipeline.py +++ b/tests/model_executor/models/minicpmo_4_5/test_pipeline.py @@ -34,7 +34,6 @@ _PIPELINE_KEY = "minicpmo_4_5" _DEPLOY_DIR = Path(__file__).resolve().parents[4] / "vllm_omni" / "deploy" _DEPLOY_LAYOUTS = { "minicpmo_4_5.yaml": ["0", "0", "0"], - "minicpmo_4_5_batching.yaml": ["0", "1", "1"], "minicpmo_4_5_2gpu.yaml": ["0", "1", "1"], "minicpmo_4_5_3gpu.yaml": ["0", "1", "2"], "minicpmo_4_5_8x4090.yaml": ["0,1,2,3", "4", "5"], diff --git a/tests/test_config_factory.py b/tests/test_config_factory.py index 1f7b1045d..78436eb1d 100644 --- a/tests/test_config_factory.py +++ b/tests/test_config_factory.py @@ -2191,6 +2191,16 @@ class TestPlatformOverrides: replica_stages = merge_pipeline_deploy(pipeline, replica) assert "kv_cache_memory_bytes" not in replica_stages[1].yaml_engine_args + def test_minicpmo_4_5_single_gpu_admits_four_sequences(self): + """Single-GPU shares leave room for Code2Wav peak activations at bs<=4.""" + deploy = load_deploy_config(Path(get_deploy_config_path("minicpmo_4_5.yaml"))) + assert [stage.max_num_seqs for stage in deploy.stages] == [4, 4, 4] + assert [stage.gpu_memory_utilization for stage in deploy.stages] == [ + 0.55, + 0.15, + 0.18, + ] + def test_npu_overrides(self): deploy_path = Path(__file__).parent.parent / "vllm_omni" / "deploy" / "qwen3_omni_moe.yaml" if not deploy_path.exists(): diff --git a/vllm_omni/deploy/minicpmo_4_5.yaml b/vllm_omni/deploy/minicpmo_4_5.yaml index ab79c37ca..fbc2853df 100644 --- a/vllm_omni/deploy/minicpmo_4_5.yaml +++ b/vllm_omni/deploy/minicpmo_4_5.yaml @@ -17,7 +17,9 @@ connectors: stages: - stage_id: 0 - max_num_seqs: 16 + # Four-way concurrency matches the single-GPU memory shares below; + # larger batches make Code2Wav activations OOM on one H100. + max_num_seqs: 4 # Leave headroom on the shared GPU for Talker KV + concurrent Code2Wav. gpu_memory_utilization: 0.55 enforce_eager: false @@ -48,7 +50,7 @@ stages: repetition_penalty: 1.0 - stage_id: 1 - max_num_seqs: 16 + max_num_seqs: 4 gpu_memory_utilization: 0.15 enforce_eager: false max_num_batched_tokens: 8192 @@ -70,7 +72,7 @@ stages: detokenize: false - stage_id: 2 - max_num_seqs: 16 + max_num_seqs: 4 gpu_memory_utilization: 0.18 enforce_eager: true enable_chunked_prefill: false diff --git a/vllm_omni/deploy/minicpmo_4_5_2gpu.yaml b/vllm_omni/deploy/minicpmo_4_5_2gpu.yaml index eddfd0877..1929e5fd1 100644 --- a/vllm_omni/deploy/minicpmo_4_5_2gpu.yaml +++ b/vllm_omni/deploy/minicpmo_4_5_2gpu.yaml @@ -1,2 +1,95 @@ -# Backward-compatible name for the canonical two-GPU batching profile. -base_config: minicpmo_4_5_batching.yaml +# MiniCPM-o 4.5 two-GPU continuous-batching layout. +# +# Thinker owns GPU 0. Talker and Code2Wav share GPU 1, keeping text decoding +# isolated from codec/vocoder contention while preserving stage-2 batching. +pipeline: minicpmo_4_5 +async_chunk: true + +trust_remote_code: true +enable_prefix_caching: false + +connectors: + connector_of_shared_memory: + name: SharedMemoryConnector + extra: + connector_get_sleep_s: 0.01 + connector_get_max_wait_first_chunk: 3000 + connector_get_max_wait: 300 + codec_chunk_frames: 25 + codec_left_context_frames: 3 + +stages: + - stage_id: 0 + max_num_seqs: 4 + gpu_memory_utilization: 0.9 + enforce_eager: false + tensor_parallel_size: 1 + max_num_batched_tokens: 16384 + devices: "0" + interleave_mm_strings: true + media_io_kwargs: + video: + fps: 1 + num_frames: 128 + limit_mm_per_prompt: + video: 1 + default_sampling_params: + temperature: 0.0 + top_p: 1.0 + top_k: -1 + max_tokens: 2048 + seed: 42 + detokenize: true + repetition_penalty: 1.1 + + - stage_id: 1 + max_num_seqs: 4 + gpu_memory_utilization: 0.55 + enforce_eager: false + max_num_batched_tokens: 8192 + # Talker context is 4096 (tts_config.max_position_embeddings). + max_model_len: 4096 + devices: "1" + output_connectors: + to_stage_2: connector_of_shared_memory + # Codec sampling is model-owned by the checkpoint's tts_config (defaults: + # T=0.8, top-k=25, top-p=0.85, repetition penalty=1.05, seed=42). + # These parameters only govern vLLM's binary continue/stop token. + default_sampling_params: + temperature: 0.0 + top_p: 1.0 + top_k: -1 + min_tokens: 50 + max_tokens: 2048 + stop_token_ids: [1] + seed: 42 + detokenize: false + + - stage_id: 2 + max_num_seqs: 4 + gpu_memory_utilization: 0.35 + enforce_eager: true + enable_chunked_prefill: false + max_num_batched_tokens: 65536 + max_model_len: 65536 + devices: "1" + input_connectors: + from_stage_1: connector_of_shared_memory + default_sampling_params: + temperature: 0.0 + top_p: 1.0 + top_k: -1 + max_tokens: 65536 + detokenize: true + +platforms: + npu: + stages: + - stage_id: 0 + max_num_batched_tokens: 8192 + compilation_config: + cudagraph_mode: PIECEWISE + - stage_id: 1 + max_num_batched_tokens: 8192 + compilation_config: + cudagraph_mode: PIECEWISE diff --git a/vllm_omni/deploy/minicpmo_4_5_batching.yaml b/vllm_omni/deploy/minicpmo_4_5_batching.yaml deleted file mode 100644 index 41ee72a39..000000000 --- a/vllm_omni/deploy/minicpmo_4_5_batching.yaml +++ /dev/null @@ -1,93 +0,0 @@ -# MiniCPM-o 4.5 two-GPU continuous-batching layout. -# -# Thinker owns GPU 0. Talker and Code2Wav share GPU 1, keeping text decoding -# isolated from codec/vocoder contention while preserving stage-2 batching. -pipeline: minicpmo_4_5 -async_chunk: true - -trust_remote_code: true -enable_prefix_caching: false - -connectors: - connector_of_shared_memory: - name: SharedMemoryConnector - extra: - connector_get_sleep_s: 0.01 - connector_get_max_wait_first_chunk: 3000 - connector_get_max_wait: 300 - codec_chunk_frames: 25 - codec_left_context_frames: 3 - -stages: - - stage_id: 0 - max_num_seqs: 4 - gpu_memory_utilization: 0.9 - enforce_eager: false - tensor_parallel_size: 1 - max_num_batched_tokens: 16384 - devices: "0" - interleave_mm_strings: true - media_io_kwargs: - video: - fps: 1 - num_frames: 128 - limit_mm_per_prompt: - video: 1 - default_sampling_params: - temperature: 0.0 - top_p: 1.0 - top_k: -1 - max_tokens: 2048 - seed: 42 - detokenize: true - repetition_penalty: 1.1 - - - stage_id: 1 - max_num_seqs: 4 - gpu_memory_utilization: 0.55 - enforce_eager: false - max_num_batched_tokens: 8192 - devices: "1" - output_connectors: - to_stage_2: connector_of_shared_memory - # Codec sampling is model-owned by the checkpoint's tts_config (defaults: - # T=0.8, top-k=25, top-p=0.85, repetition penalty=1.05, seed=42). - # These parameters only govern vLLM's binary continue/stop token. - default_sampling_params: - temperature: 0.0 - top_p: 1.0 - top_k: -1 - min_tokens: 50 - max_tokens: 2048 - stop_token_ids: [1] - seed: 42 - detokenize: false - - - stage_id: 2 - max_num_seqs: 4 - gpu_memory_utilization: 0.35 - enforce_eager: true - enable_chunked_prefill: false - max_num_batched_tokens: 65536 - max_model_len: 65536 - devices: "1" - input_connectors: - from_stage_1: connector_of_shared_memory - default_sampling_params: - temperature: 0.0 - top_p: 1.0 - top_k: -1 - max_tokens: 65536 - detokenize: true - -platforms: - npu: - stages: - - stage_id: 0 - max_num_batched_tokens: 8192 - compilation_config: - cudagraph_mode: PIECEWISE - - stage_id: 1 - max_num_batched_tokens: 8192 - compilation_config: - cudagraph_mode: PIECEWISE