diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260923/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260923/README.md index 46309a32f..669919380 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260923/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260923/README.md @@ -8,6 +8,8 @@ This extends the [60-cell device × model matrix](../e2e_profiling_20260922/evid | Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped complete three-stage text-to-text+speech path (P for this input only) | Coherent text with the same 32 thinker token IDs as WSL CUDA, and a nonzero 5.2 s mono 24 kHz WAV. First engine startup 297.20 s and request 78.75 s, including first-use graph/JIT work; separate 1-warmup/20-measured serial run: startup 239.05 s, request wall p50 18.38 s/p95 18.73 s. | Speech intelligibility/alignment, image/audio/video inputs, concurrency, interruption, loading peak and sustained behavior. | | WSL RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **image**-to-text+speech path (P for one synthetic image) | For a 448×448 red square on white, returned the correct visual description and a nonzero 5.2 s WAV. Engine startup 101.96 s; one request 17.49 s. | Broader visual quality, real images, audio/video inputs, speech intelligibility, repeated latency and cleanup reliability. | | WSL RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **audio**-to-text+speech path (P for one synthetic clip) | For a 2 s, 440 Hz tone at 16 kHz, answered “A loud whistling sound” and produced a nonzero 2.16 s WAV. Engine startup 105.94 s; one request 7.78 s. | Speech transcription/real audio quality, video input, speech intelligibility, repeated latency and shared-memory cleanup. | +| Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **image**-to-text+speech path (P for one synthetic image) | For the same 448×448 red square, returned exactly the same text and 27 thinker token IDs as WSL CUDA, plus a nonzero 5.2 s WAV. Engine startup 334.16 s; one request 17.15 s. | Real-image quality, audio/video inputs, speech intelligibility, repeated latency, memory and sustained behavior. | +| Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **audio**-to-text+speech path (P for one synthetic clip) | For the same 2 s, 440 Hz tone, returned exactly the same text and nine thinker token IDs as WSL CUDA, plus a nonzero 2.16 s WAV. Engine startup 268.47 s; one request 8.26 s. | Speech transcription/real-audio quality, video input, speech intelligibility, repeated latency, memory and sustained behavior. | | WSL CPU / InternVLA-A1-3B Place_Markpen, revision `0a557c6a503e6545b369bff2f34c76a410bbd190` | Strict real-weight load and full policy forward on **synthetic zero** observations/noise | Finite float32 actions `[1, 50, 32]`; CPU-only PyTorch startup 31.78 s and synchronized forward 3.25 s. Compared with a same-checkpoint CUDA run, relative L2 difference 1.3025% and cosine similarity 0.999917. | Real Place_Markpen observations and reference actions, task tolerance, units/timestamps, repeated latency, memory and deadlines. | | Native Windows CPU / InternVLA-A1-3B Place_Markpen, same revision | Strict real-weight load and full policy forward on **synthetic zero** observations/noise; policy and actions checked on CPU | Finite float32 actions `[1, 50, 32]`; placement-checked startup 61.66 s and forward 3.36 s. Relative L2 difference from WSL CPU 1.0824%; cosine 0.999942. | Real observations/reference actions and quality tolerance; repeated latency, memory, units/timestamps and deadlines. This is the direct policy CPU path, not native vLLM CPU attention. | | Native Windows RTX 5090 Laptop / InternVLA-A1-3B Place_Markpen, same revision | Strict real-weight load and full policy forward on **synthetic zero** observations/noise; policy and actions checked on CUDA | Finite float32 actions `[1, 50, 32]`; placement-checked startup 75.54 s and synchronized forward 1.12 s. Relative L2 difference from WSL CUDA 0.8315%; cosine 0.999966. | Real observations/reference actions and quality tolerance; repeated latency, memory, units/timestamps and deadlines. | @@ -24,6 +26,10 @@ The WSL RTX image path used the same real checkpoint and BF16 three-stage pipeli The separate WSL RTX audio path used the same checkpoint, BF16 three-stage pipeline, 2,048/512-token limits and 5 GiB thinker offload, with [one audio item admitted](evidence/minicpmo_wsl_audio/minicpmo_24gb_audio_offline.yaml) at a 0.61 thinker GPU budget. Its [input fixture](evidence/minicpmo_wsl_audio/tone_440hz_2s_16khz.wav) is a 2 s, 16 kHz mono sine tone at 440 Hz. The [raw log](evidence/minicpmo_wsl_audio/minicpmo_audio_success.log), [text/token IDs](evidence/minicpmo_wsl_audio/0_38cd53c3-5185-43ec-aeeb-6a40091cf9ad.txt), [output WAV](evidence/minicpmo_wsl_audio/output_0_38cd53c3-5185-43ec-aeeb-6a40091cf9ad.wav), and [artifact check](evidence/minicpmo_wsl_audio/minicpmo_audio_artifact_report.json) establish one audio-input to text+speech request. The output was “A loud whistling sound” and 51,840 mono samples at 24 kHz with nonzero RMS. This response is plausible for the tone but is not a transcription or general audio-quality pass. Shutdown again reported one shared-memory cleanup warning. +Native Windows RTX image execution reused the exact [WSL image configuration](evidence/minicpmo_wsl_image/minicpmo_24gb_image_offline.yaml) and [synthetic input](evidence/minicpmo_wsl_image/red_square.png) through local UNC paths. The three stages loaded the real BF16 checkpoint under vLLM 0.29.0. The [raw log](evidence/minicpmo_windows_image/minicpmo_native_image.log), [text/token IDs](evidence/minicpmo_windows_image/0_694a20bc-73ff-4659-8a2a-766061016444.txt), [WAV](evidence/minicpmo_windows_image/output_0_694a20bc-73ff-4659-8a2a-766061016444.wav), and [artifact check](evidence/minicpmo_windows_image/minicpmo_windows_image_artifact_report.json) retain the one-request evidence. Windows text and all 27 thinker token IDs matched WSL exactly; the 124,800-sample, 24 kHz WAV had nonzero RMS. Startup took 334.16 s and the request 17.15 s on this run. Workers shut down without a recorded force kill. This one image and one timing observation do not establish real-image quality, speech meaning, memory peak or sustained behavior. + +Native Windows RTX audio execution reused the exact [WSL audio configuration](evidence/minicpmo_wsl_audio/minicpmo_24gb_audio_offline.yaml) and [synthetic tone](evidence/minicpmo_wsl_audio/tone_440hz_2s_16khz.wav) through local UNC paths. The [raw log](evidence/minicpmo_windows_audio/minicpmo_native_audio.log), [text/token IDs](evidence/minicpmo_windows_audio/0_85a27d56-baee-49f0-a003-94069961be17.txt), [WAV](evidence/minicpmo_windows_audio/output_0_85a27d56-baee-49f0-a003-94069961be17.wav), and [artifact check](evidence/minicpmo_windows_audio/minicpmo_windows_audio_artifact_report.json) retain the one-request evidence. Windows returned “A loud whistling sound” with all nine thinker token IDs matching WSL; the 51,840-sample, 24 kHz WAV had nonzero RMS. Startup took 268.47 s and the request 8.26 s. Workers exited cleanly in the log. A tone description is not a speech-transcription or broad audio-quality pass; generated speech intelligibility and sustained behavior remain unverified. + The CPU-only InternVLA run used the same BF16 policy checkpoint, synthetic six-camera history/state input and zero diffusion noise as the CUDA reference. The raw [CPU JSON](evidence/internvla/internvla_pure_cpu_synthetic.json), [CPU log](evidence/internvla/internvla_pure_cpu_synthetic.log), [CPU actions](evidence/internvla/internvla_pure_cpu_actions.npy), [CUDA JSON](evidence/internvla/internvla_cuda_synthetic_with_actions.json), [CUDA actions](evidence/internvla/internvla_cuda_actions.npy) and [numerical comparison](evidence/internvla/internvla_pure_cpu_cuda_parity.json) are retained. The CUDA reference used the WSL RTX 5090 Laptop and PyTorch 2.13.0+cu130; its one forward took 1.33 s. The action difference has maximum absolute error 0.00993 and mean absolute error 0.00117. No reference-task tolerance is available, so this comparison is evidence of numerical proximity, not an accuracy pass. The native Windows runs used Windows 11 build 26200, NVIDIA driver 610.71 for the GPU path, Python 3.12.10, PyTorch 2.13.0+cu130, vLLM 0.29.0, Transformers 5.14.1, and the current Omni source. The CPU run explicitly selected the CPU policy despite the CUDA-capable wheel. The placement-checked reruns assert that all policy parameters and the output action tensor remain on the requested device. Both loaded the same checkpoint from the local WSL filesystem through a UNC path; Windows CPU and GPU runs used the same synthetic input and zero diffusion noise as the WSL reference. The native [CPU JSON](evidence/internvla/internvla_windows_cpu_synthetic.json), [GPU JSON](evidence/internvla/internvla_windows_gpu_synthetic.json), [CPU log](evidence/internvla/internvla_windows_cpu_synthetic.log), [GPU log](evidence/internvla/internvla_windows_gpu_synthetic.log), [CPU actions](evidence/internvla/internvla_windows_cpu_actions.npy), [GPU actions](evidence/internvla/internvla_windows_gpu_actions.npy), [CPU/WSL comparison](evidence/internvla/internvla_windows_cpu_wsl_cpu_parity.json), and [GPU/WSL comparison](evidence/internvla/internvla_windows_gpu_wsl_cuda_parity.json) are retained. Native vLLM import initially failed on a Linux-specific CuMem allocator patch; the added Windows guard resolved the import. Native default GBK source decoding also required `PYTHONUTF8=1`. Both platform differences are setup requirements, not model-quality evidence. Power mode and thermal state were not captured. @@ -89,3 +95,21 @@ $env:VLLM_ENABLE_V1_MULTIPROCESSING = '0' --warmups 1 --repeats 20 --init-timeout 900 --stage-init-timeout 600 ` --output 'C:\Users\zhout\w2\minicpmo_windows_gpu_profile20.json' ``` + +For the separate native Windows image and audio single-request probes, keep the same PowerShell environment above and run the example with the corresponding recorded deployment plan and fixture: + +```powershell +& 'C:\Users\zhout\w2\omni029venv\Scripts\python.exe' "$root\vllm-omni-edge\examples\offline_inference\minicpmo\end2end.py" ` + --model "$root\models\MiniCPM-o-4_5" ` + --deploy-config "$root\vllm-omni-edge\benchmarks\edge_harness\results\e2e_expansion_20260923\evidence\minicpmo_wsl_image\minicpmo_24gb_image_offline.yaml" ` + --query-type use_image --image-path "$root\vllm-omni-edge\benchmarks\edge_harness\results\e2e_expansion_20260923\evidence\minicpmo_wsl_image\red_square.png" ` + --modalities text,audio --num-prompts 1 --init-timeout 900 --stage-init-timeout 600 ` + --output-dir 'C:\Users\zhout\w2\minicpmo_native_image' + +& 'C:\Users\zhout\w2\omni029venv\Scripts\python.exe' "$root\vllm-omni-edge\examples\offline_inference\minicpmo\end2end.py" ` + --model "$root\models\MiniCPM-o-4_5" ` + --deploy-config "$root\vllm-omni-edge\benchmarks\edge_harness\results\e2e_expansion_20260923\evidence\minicpmo_wsl_audio\minicpmo_24gb_audio_offline.yaml" ` + --query-type use_audio --audio-path "$root\vllm-omni-edge\benchmarks\edge_harness\results\e2e_expansion_20260923\evidence\minicpmo_wsl_audio\tone_440hz_2s_16khz.wav" ` + --modalities text,audio --num-prompts 1 --init-timeout 900 --stage-init-timeout 600 ` + --output-dir 'C:\Users\zhout\w2\minicpmo_native_audio' +``` diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_windows_audio/0_85a27d56-baee-49f0-a003-94069961be17.txt b/benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_windows_audio/0_85a27d56-baee-49f0-a003-94069961be17.txt new file mode 100644 index 000000000..27050cf60 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_windows_audio/0_85a27d56-baee-49f0-a003-94069961be17.txt @@ -0,0 +1,16 @@ +Prompt: +<|im_start|>system +You are MiniCPM-o, a helpful multimodal assistant that can understand images, audio and video, and respond in text and speech.<|im_end|> +<|im_start|>user +() +What is the content of this audio?<|im_end|> +<|im_start|>assistant + + + + +<|tts_bos|> +vllm_text_output: +A loud whistling sound. +vllm_token_ids: +[32, 17361, 420, 380, 2718, 5112, 13, 151704, 151645] diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_windows_audio/minicpmo_native_audio.log b/benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_windows_audio/minicpmo_native_audio.log new file mode 100644 index 000000000..1b6838496 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_windows_audio/minicpmo_native_audio.log @@ -0,0 +1,272 @@ +INFO 09-23 17:19:45 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed. +INFO 09-23 17:19:45 [patch.py:510] inductor factorable-divisibility patch: installed. +==================== + vllm version: 0.29.0 + ==================== +INFO 09-23 17:19:48 [omni_base.py:216] [Omni] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5 +INFO 09-23 17:19:48 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5 +INFO 09-23 17:19:49 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 3 stages +INFO 09-23 17:19:50 [stage_init_utils.py:768] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn +INFO 09-23 17:19:50 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations +WARNING 09-23 17:19:50 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass) +WARNING 09-23 17:19:50 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass) +WARNING 09-23 17:19:50 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. +INFO 09-23 17:19:50 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration +INFO 09-23 17:19:50 [model.py:2021] Using max model len 2048 +WARNING 09-23 17:19:50 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False. +INFO 09-23 17:19:52 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=2048. +WARNING 09-23 17:19:52 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +WARNING 09-23 17:19:52 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +INFO 09-23 17:19:52 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +INFO 09-23 17:19:52 [vllm.py:1585] Cudagraph is disabled under eager mode +INFO 09-23 17:19:52 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +WARNING 09-23 17:19:52 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass) +WARNING 09-23 17:19:52 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass) +WARNING 09-23 17:19:52 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. +INFO 09-23 17:19:53 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration +INFO 09-23 17:19:53 [model.py:2021] Using max model len 512 +WARNING 09-23 17:19:53 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False. +INFO 09-23 17:19:53 [model.py:2021] Using max model len 512 +INFO 09-23 17:19:53 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=512. +INFO 09-23 17:19:53 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +WARNING 09-23 17:19:53 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass) +WARNING 09-23 17:19:53 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass) +WARNING 09-23 17:19:53 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. +INFO 09-23 17:19:53 [model.py:684] Resolved architecture: MiniCPMO45Code2Wav +INFO 09-23 17:19:53 [model.py:2021] Using max model len 4096 +WARNING 09-23 17:19:53 [arg_utils.py:2780] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs. +INFO 09-23 17:19:53 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +INFO 09-23 17:19:53 [stage_runtime.py:853] [stage_init] Stage-0 set runtime devices: 0 +INFO 09-23 17:20:06 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed. +INFO 09-23 17:20:06 [patch.py:510] inductor factorable-divisibility patch: installed. +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:09 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', speculative_config=None, tokenizer='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=\\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:12 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_7b13e02cd0c74a1ca40084f32dfc508e backend=nccl +(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:12 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo. +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:12 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:12 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks. +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:12 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0. +(StageEngineCoreProc_stage0_replica0 pid=38880) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead. +(StageEngineCoreProc_stage0_replica0 pid=38880) [transformers] Requested torchvision backend is not available. Falling back to pil backend. +(StageEngineCoreProc_stage0_replica0 pid=38880) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead. +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [base.py:138] Offloader set to UVAOffloader +(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:21 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:21 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5... +(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:21 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:21 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [vllm.py:1585] Cudagraph is disabled under eager mode +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:22 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:22 [flash_attn.py:897] Using FlashAttention version 2 +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:24 [uva.py:65] Total CPU offloaded parameters: 5.03 +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:24 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 25.00 GiB. +(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:24 [weight_utils.py:886] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(StageEngineCoreProc_stage0_replica0 pid=38880) +Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:22:54 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_b12cceabf1984b38b14ca47252b06883 backend=nccl +(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:22:54 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo. +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:22:54 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:22:55 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks. +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:22:55 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0. +(StageEngineCoreProc_stage1_replica0 pid=22944) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead. +(StageEngineCoreProc_stage1_replica0 pid=22944) [transformers] Requested torchvision backend is not available. Falling back to pil backend. +(StageEngineCoreProc_stage1_replica0 pid=22944) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead. +(StageEngineCoreProc_stage1_replica0 pid=22944) [transformers] Image processor MiniCPMVImageProcessor: kwargs ['max_slice_nums'] were applied for backward compatibility. To avoid this warning, add them to valid_kwargs: create a custom TypedDict extending ImagesKwargs with these keys and set it as the `valid_kwargs` class attribute. +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [factory.py:46] Created connector: SharedMemoryConnector +(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:11 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:11 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5... +(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:11 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:11 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [vllm.py:1585] Cudagraph is disabled under eager mode +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [flash_attn.py:897] Using FlashAttention version 2 +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 13.31 GiB. +(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (13.31 GiB). +(StageEngineCoreProc_stage1_replica0 pid=22944) +Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_4c8c7d9dfb3648ea8f6653978d6573f0 backend=nccl +(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:38 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo. +(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:38 [gpu_generation_worker.py:103] OMNI GPUGenerationWorker forces v1 model runner for omni hooks. +(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0. +(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [factory.py:46] Created connector: SharedMemoryConnector +(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:38 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:38 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5... +(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:39 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 11.09 GiB. +(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:39 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (11.09 GiB). +(StageEngineCoreProc_stage2_replica0 pid=42144) +Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00._kernel. This causes a latency spike; consider extending warmup to cover this shape/config. +Processed prompts: 0%| | 0/1 [00:00system +You are MiniCPM-o, a helpful multimodal assistant that can understand images, audio and video, and respond in text and speech.<|im_end|> +<|im_start|>user +(./) +What is the content of this image?<|im_end|> +<|im_start|>assistant + + + + +<|tts_bos|> +vllm_text_output: +The image shows a solid red square centered on a white background. There are no other elements, text, or details visible. +vllm_token_ids: +[785, 2168, 4933, 264, 6437, 2518, 9334, 30188, 389, 264, 4158, 4004, 13, 2619, 525, 902, 1008, 5424, 11, 1467, 11, 476, 3565, 9434, 13, 151704, 151645] diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_windows_image/minicpmo_native_image.log b/benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_windows_image/minicpmo_native_image.log new file mode 100644 index 000000000..52bb41b45 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_windows_image/minicpmo_native_image.log @@ -0,0 +1,274 @@ +INFO 09-23 17:12:37 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed. +INFO 09-23 17:12:37 [patch.py:510] inductor factorable-divisibility patch: installed. +==================== + vllm version: 0.29.0 + ==================== +INFO 09-23 17:12:41 [omni_base.py:216] [Omni] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5 +INFO 09-23 17:12:41 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5 +INFO 09-23 17:12:42 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 3 stages +INFO 09-23 17:12:42 [stage_init_utils.py:768] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn +INFO 09-23 17:12:42 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations +WARNING 09-23 17:12:42 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass) +WARNING 09-23 17:12:42 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass) +WARNING 09-23 17:12:42 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. +INFO 09-23 17:12:43 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration +INFO 09-23 17:12:43 [model.py:2021] Using max model len 2048 +WARNING 09-23 17:12:43 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False. +INFO 09-23 17:12:44 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=2048. +WARNING 09-23 17:12:44 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +WARNING 09-23 17:12:44 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +INFO 09-23 17:12:44 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +INFO 09-23 17:12:44 [vllm.py:1585] Cudagraph is disabled under eager mode +INFO 09-23 17:12:44 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +WARNING 09-23 17:12:44 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass) +WARNING 09-23 17:12:44 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass) +WARNING 09-23 17:12:45 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. +INFO 09-23 17:12:45 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration +INFO 09-23 17:12:45 [model.py:2021] Using max model len 512 +WARNING 09-23 17:12:45 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False. +INFO 09-23 17:12:45 [model.py:2021] Using max model len 512 +INFO 09-23 17:12:45 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=512. +INFO 09-23 17:12:45 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +WARNING 09-23 17:12:45 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass) +WARNING 09-23 17:12:45 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass) +WARNING 09-23 17:12:45 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. +INFO 09-23 17:12:46 [model.py:684] Resolved architecture: MiniCPMO45Code2Wav +INFO 09-23 17:12:46 [model.py:2021] Using max model len 4096 +WARNING 09-23 17:12:46 [arg_utils.py:2780] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs. +INFO 09-23 17:12:46 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +INFO 09-23 17:12:46 [stage_runtime.py:853] [stage_init] Stage-0 set runtime devices: 0 +INFO 09-23 17:13:03 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed. +INFO 09-23 17:13:03 [patch.py:510] inductor factorable-divisibility patch: installed. +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:07 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', speculative_config=None, tokenizer='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=\\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:10 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_431f956fcdf44084bd7cd31ffc24916b backend=nccl +(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:10 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo. +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:10 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:11 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks. +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:11 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0. +(StageEngineCoreProc_stage0_replica0 pid=24488) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead. +(StageEngineCoreProc_stage0_replica0 pid=24488) [transformers] Requested torchvision backend is not available. Falling back to pil backend. +(StageEngineCoreProc_stage0_replica0 pid=24488) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead. +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:18 [base.py:138] Offloader set to UVAOffloader +(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:18 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:18 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:18 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5... +(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:19 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:19 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:19 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:19 [vllm.py:1585] Cudagraph is disabled under eager mode +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:19 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:20 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:20 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:20 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:20 [flash_attn.py:897] Using FlashAttention version 2 +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:22 [uva.py:65] Total CPU offloaded parameters: 5.03 +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:23 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 8.21 GiB. +(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:23 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (8.21 GiB). +(StageEngineCoreProc_stage0_replica0 pid=24488) +Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:03 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_aa9f5ce7e8b54624b73d3744456143a5 backend=nccl +(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:03 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo. +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:03 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:04 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks. +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:04 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0. +(StageEngineCoreProc_stage1_replica0 pid=55724) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead. +(StageEngineCoreProc_stage1_replica0 pid=55724) [transformers] Requested torchvision backend is not available. Falling back to pil backend. +(StageEngineCoreProc_stage1_replica0 pid=55724) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead. +(StageEngineCoreProc_stage1_replica0 pid=55724) [transformers] Image processor MiniCPMVImageProcessor: kwargs ['max_slice_nums'] were applied for backward compatibility. To avoid this warning, add them to valid_kwargs: create a custom TypedDict extending ImagesKwargs with these keys and set it as the `valid_kwargs` class attribute. +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [factory.py:46] Created connector: SharedMemoryConnector +(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:19 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:19 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5... +(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:19 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:19 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [vllm.py:1585] Cudagraph is disabled under eager mode +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:20 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:20 [flash_attn.py:897] Using FlashAttention version 2 +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:20 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 17.15 GiB. +(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:20 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (17.15 GiB). +(StageEngineCoreProc_stage1_replica0 pid=55724) +Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:48 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_b39c4cfa1016448ba676b1a7438e3913 backend=nccl +(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:17:48 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo. +(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:48 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:17:49 [gpu_generation_worker.py:103] OMNI GPUGenerationWorker forces v1 model runner for omni hooks. +(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:49 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0. +(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:49 [factory.py:46] Created connector: SharedMemoryConnector +(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:17:49 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:17:49 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED. +(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:49 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5... +(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:50 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 23.12 GiB. +(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:50 [weight_utils.py:886] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(StageEngineCoreProc_stage2_replica0 pid=33204) +Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00._kernel. This causes a latency spike; consider extending warmup to cover this shape/config. +(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:42 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 18, 3840) +Processed prompts: 0%| | 0/1 [00:00