Document Windows MiniCPM-o image and audio end-to-end runs
This commit is contained in:
tzhouam committed
1 parent
a8fdb1babf
commit
b29a6ca99f
11 files changed
+639
-4
No files matched your search
@@ -8,6 +8,8 @@ This extends the [60-cell device × model matrix](../e2e_profiling_20260922/evid
|
||||
| Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped complete three-stage text-to-text+speech path (P for this input only) | Coherent text with the same 32 thinker token IDs as WSL CUDA, and a nonzero 5.2 s mono 24 kHz WAV. First engine startup 297.20 s and request 78.75 s, including first-use graph/JIT work; separate 1-warmup/20-measured serial run: startup 239.05 s, request wall p50 18.38 s/p95 18.73 s. | Speech intelligibility/alignment, image/audio/video inputs, concurrency, interruption, loading peak and sustained behavior. |
|
||||
| WSL RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **image**-to-text+speech path (P for one synthetic image) | For a 448×448 red square on white, returned the correct visual description and a nonzero 5.2 s WAV. Engine startup 101.96 s; one request 17.49 s. | Broader visual quality, real images, audio/video inputs, speech intelligibility, repeated latency and cleanup reliability. |
|
||||
| WSL RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **audio**-to-text+speech path (P for one synthetic clip) | For a 2 s, 440 Hz tone at 16 kHz, answered “A loud whistling sound” and produced a nonzero 2.16 s WAV. Engine startup 105.94 s; one request 7.78 s. | Speech transcription/real audio quality, video input, speech intelligibility, repeated latency and shared-memory cleanup. |
|
||||
| Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **image**-to-text+speech path (P for one synthetic image) | For the same 448×448 red square, returned exactly the same text and 27 thinker token IDs as WSL CUDA, plus a nonzero 5.2 s WAV. Engine startup 334.16 s; one request 17.15 s. | Real-image quality, audio/video inputs, speech intelligibility, repeated latency, memory and sustained behavior. |
|
||||
| Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **audio**-to-text+speech path (P for one synthetic clip) | For the same 2 s, 440 Hz tone, returned exactly the same text and nine thinker token IDs as WSL CUDA, plus a nonzero 2.16 s WAV. Engine startup 268.47 s; one request 8.26 s. | Speech transcription/real-audio quality, video input, speech intelligibility, repeated latency, memory and sustained behavior. |
|
||||
| WSL CPU / InternVLA-A1-3B Place_Markpen, revision `0a557c6a503e6545b369bff2f34c76a410bbd190` | Strict real-weight load and full policy forward on **synthetic zero** observations/noise | Finite float32 actions `[1, 50, 32]`; CPU-only PyTorch startup 31.78 s and synchronized forward 3.25 s. Compared with a same-checkpoint CUDA run, relative L2 difference 1.3025% and cosine similarity 0.999917. | Real Place_Markpen observations and reference actions, task tolerance, units/timestamps, repeated latency, memory and deadlines. |
|
||||
| Native Windows CPU / InternVLA-A1-3B Place_Markpen, same revision | Strict real-weight load and full policy forward on **synthetic zero** observations/noise; policy and actions checked on CPU | Finite float32 actions `[1, 50, 32]`; placement-checked startup 61.66 s and forward 3.36 s. Relative L2 difference from WSL CPU 1.0824%; cosine 0.999942. | Real observations/reference actions and quality tolerance; repeated latency, memory, units/timestamps and deadlines. This is the direct policy CPU path, not native vLLM CPU attention. |
|
||||
| Native Windows RTX 5090 Laptop / InternVLA-A1-3B Place_Markpen, same revision | Strict real-weight load and full policy forward on **synthetic zero** observations/noise; policy and actions checked on CUDA | Finite float32 actions `[1, 50, 32]`; placement-checked startup 75.54 s and synchronized forward 1.12 s. Relative L2 difference from WSL CUDA 0.8315%; cosine 0.999966. | Real observations/reference actions and quality tolerance; repeated latency, memory, units/timestamps and deadlines. |
|
||||
@@ -24,6 +26,10 @@ The WSL RTX image path used the same real checkpoint and BF16 three-stage pipeli
|
||||
|
||||
The separate WSL RTX audio path used the same checkpoint, BF16 three-stage pipeline, 2,048/512-token limits and 5 GiB thinker offload, with [one audio item admitted](evidence/minicpmo_wsl_audio/minicpmo_24gb_audio_offline.yaml) at a 0.61 thinker GPU budget. Its [input fixture](evidence/minicpmo_wsl_audio/tone_440hz_2s_16khz.wav) is a 2 s, 16 kHz mono sine tone at 440 Hz. The [raw log](evidence/minicpmo_wsl_audio/minicpmo_audio_success.log), [text/token IDs](evidence/minicpmo_wsl_audio/0_38cd53c3-5185-43ec-aeeb-6a40091cf9ad.txt), [output WAV](evidence/minicpmo_wsl_audio/output_0_38cd53c3-5185-43ec-aeeb-6a40091cf9ad.wav), and [artifact check](evidence/minicpmo_wsl_audio/minicpmo_audio_artifact_report.json) establish one audio-input to text+speech request. The output was “A loud whistling sound” and 51,840 mono samples at 24 kHz with nonzero RMS. This response is plausible for the tone but is not a transcription or general audio-quality pass. Shutdown again reported one shared-memory cleanup warning.
|
||||
|
||||
Native Windows RTX image execution reused the exact [WSL image configuration](evidence/minicpmo_wsl_image/minicpmo_24gb_image_offline.yaml) and [synthetic input](evidence/minicpmo_wsl_image/red_square.png) through local UNC paths. The three stages loaded the real BF16 checkpoint under vLLM 0.29.0. The [raw log](evidence/minicpmo_windows_image/minicpmo_native_image.log), [text/token IDs](evidence/minicpmo_windows_image/0_694a20bc-73ff-4659-8a2a-766061016444.txt), [WAV](evidence/minicpmo_windows_image/output_0_694a20bc-73ff-4659-8a2a-766061016444.wav), and [artifact check](evidence/minicpmo_windows_image/minicpmo_windows_image_artifact_report.json) retain the one-request evidence. Windows text and all 27 thinker token IDs matched WSL exactly; the 124,800-sample, 24 kHz WAV had nonzero RMS. Startup took 334.16 s and the request 17.15 s on this run. Workers shut down without a recorded force kill. This one image and one timing observation do not establish real-image quality, speech meaning, memory peak or sustained behavior.
|
||||
|
||||
Native Windows RTX audio execution reused the exact [WSL audio configuration](evidence/minicpmo_wsl_audio/minicpmo_24gb_audio_offline.yaml) and [synthetic tone](evidence/minicpmo_wsl_audio/tone_440hz_2s_16khz.wav) through local UNC paths. The [raw log](evidence/minicpmo_windows_audio/minicpmo_native_audio.log), [text/token IDs](evidence/minicpmo_windows_audio/0_85a27d56-baee-49f0-a003-94069961be17.txt), [WAV](evidence/minicpmo_windows_audio/output_0_85a27d56-baee-49f0-a003-94069961be17.wav), and [artifact check](evidence/minicpmo_windows_audio/minicpmo_windows_audio_artifact_report.json) retain the one-request evidence. Windows returned “A loud whistling sound” with all nine thinker token IDs matching WSL; the 51,840-sample, 24 kHz WAV had nonzero RMS. Startup took 268.47 s and the request 8.26 s. Workers exited cleanly in the log. A tone description is not a speech-transcription or broad audio-quality pass; generated speech intelligibility and sustained behavior remain unverified.
|
||||
|
||||
The CPU-only InternVLA run used the same BF16 policy checkpoint, synthetic six-camera history/state input and zero diffusion noise as the CUDA reference. The raw [CPU JSON](evidence/internvla/internvla_pure_cpu_synthetic.json), [CPU log](evidence/internvla/internvla_pure_cpu_synthetic.log), [CPU actions](evidence/internvla/internvla_pure_cpu_actions.npy), [CUDA JSON](evidence/internvla/internvla_cuda_synthetic_with_actions.json), [CUDA actions](evidence/internvla/internvla_cuda_actions.npy) and [numerical comparison](evidence/internvla/internvla_pure_cpu_cuda_parity.json) are retained. The CUDA reference used the WSL RTX 5090 Laptop and PyTorch 2.13.0+cu130; its one forward took 1.33 s. The action difference has maximum absolute error 0.00993 and mean absolute error 0.00117. No reference-task tolerance is available, so this comparison is evidence of numerical proximity, not an accuracy pass.
|
||||
|
||||
The native Windows runs used Windows 11 build 26200, NVIDIA driver 610.71 for the GPU path, Python 3.12.10, PyTorch 2.13.0+cu130, vLLM 0.29.0, Transformers 5.14.1, and the current Omni source. The CPU run explicitly selected the CPU policy despite the CUDA-capable wheel. The placement-checked reruns assert that all policy parameters and the output action tensor remain on the requested device. Both loaded the same checkpoint from the local WSL filesystem through a UNC path; Windows CPU and GPU runs used the same synthetic input and zero diffusion noise as the WSL reference. The native [CPU JSON](evidence/internvla/internvla_windows_cpu_synthetic.json), [GPU JSON](evidence/internvla/internvla_windows_gpu_synthetic.json), [CPU log](evidence/internvla/internvla_windows_cpu_synthetic.log), [GPU log](evidence/internvla/internvla_windows_gpu_synthetic.log), [CPU actions](evidence/internvla/internvla_windows_cpu_actions.npy), [GPU actions](evidence/internvla/internvla_windows_gpu_actions.npy), [CPU/WSL comparison](evidence/internvla/internvla_windows_cpu_wsl_cpu_parity.json), and [GPU/WSL comparison](evidence/internvla/internvla_windows_gpu_wsl_cuda_parity.json) are retained. Native vLLM import initially failed on a Linux-specific CuMem allocator patch; the added Windows guard resolved the import. Native default GBK source decoding also required `PYTHONUTF8=1`. Both platform differences are setup requirements, not model-quality evidence. Power mode and thermal state were not captured.
|
||||
@@ -89,3 +95,21 @@ $env:VLLM_ENABLE_V1_MULTIPROCESSING = '0'
|
||||
--warmups 1 --repeats 20 --init-timeout 900 --stage-init-timeout 600 `
|
||||
--output 'C:\Users\zhout\w2\minicpmo_windows_gpu_profile20.json'
|
||||
```
|
||||
|
||||
For the separate native Windows image and audio single-request probes, keep the same PowerShell environment above and run the example with the corresponding recorded deployment plan and fixture:
|
||||
|
||||
```powershell
|
||||
& 'C:\Users\zhout\w2\omni029venv\Scripts\python.exe' "$root\vllm-omni-edge\examples\offline_inference\minicpmo\end2end.py" `
|
||||
--model "$root\models\MiniCPM-o-4_5" `
|
||||
--deploy-config "$root\vllm-omni-edge\benchmarks\edge_harness\results\e2e_expansion_20260923\evidence\minicpmo_wsl_image\minicpmo_24gb_image_offline.yaml" `
|
||||
--query-type use_image --image-path "$root\vllm-omni-edge\benchmarks\edge_harness\results\e2e_expansion_20260923\evidence\minicpmo_wsl_image\red_square.png" `
|
||||
--modalities text,audio --num-prompts 1 --init-timeout 900 --stage-init-timeout 600 `
|
||||
--output-dir 'C:\Users\zhout\w2\minicpmo_native_image'
|
||||
|
||||
& 'C:\Users\zhout\w2\omni029venv\Scripts\python.exe' "$root\vllm-omni-edge\examples\offline_inference\minicpmo\end2end.py" `
|
||||
--model "$root\models\MiniCPM-o-4_5" `
|
||||
--deploy-config "$root\vllm-omni-edge\benchmarks\edge_harness\results\e2e_expansion_20260923\evidence\minicpmo_wsl_audio\minicpmo_24gb_audio_offline.yaml" `
|
||||
--query-type use_audio --audio-path "$root\vllm-omni-edge\benchmarks\edge_harness\results\e2e_expansion_20260923\evidence\minicpmo_wsl_audio\tone_440hz_2s_16khz.wav" `
|
||||
--modalities text,audio --num-prompts 1 --init-timeout 900 --stage-init-timeout 600 `
|
||||
--output-dir 'C:\Users\zhout\w2\minicpmo_native_audio'
|
||||
```
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
Prompt:
|
||||
<|im_start|>system
|
||||
You are MiniCPM-o, a helpful multimodal assistant that can understand images, audio and video, and respond in text and speech.<|im_end|>
|
||||
<|im_start|>user
|
||||
(<audio>./</audio>)
|
||||
What is the content of this audio?<|im_end|>
|
||||
<|im_start|>assistant
|
||||
<think>
|
||||
|
||||
</think>
|
||||
|
||||
<|tts_bos|>
|
||||
vllm_text_output:
|
||||
A loud whistling sound.
|
||||
vllm_token_ids:
|
||||
[32, 17361, 420, 380, 2718, 5112, 13, 151704, 151645]
|
||||
+272
@@ -0,0 +1,272 @@
|
||||
INFO 09-23 17:19:45 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:19:45 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
====================
|
||||
vllm version: 0.29.0
|
||||
====================
|
||||
INFO 09-23 17:19:48 [omni_base.py:216] [Omni] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5
|
||||
INFO 09-23 17:19:48 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5
|
||||
INFO 09-23 17:19:49 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 3 stages
|
||||
INFO 09-23 17:19:50 [stage_init_utils.py:768] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
INFO 09-23 17:19:50 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations
|
||||
WARNING 09-23 17:19:50 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:19:50 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:19:50 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:19:50 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:19:50 [model.py:2021] Using max model len 2048
|
||||
WARNING 09-23 17:19:50 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:19:52 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
WARNING 09-23 17:19:52 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-23 17:19:52 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-23 17:19:52 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 17:19:52 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
INFO 09-23 17:19:52 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-23 17:19:52 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:19:52 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:19:52 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:19:53 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:19:53 [model.py:2021] Using max model len 512
|
||||
WARNING 09-23 17:19:53 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:19:53 [model.py:2021] Using max model len 512
|
||||
INFO 09-23 17:19:53 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=512.
|
||||
INFO 09-23 17:19:53 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
WARNING 09-23 17:19:53 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:19:53 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:19:53 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:19:53 [model.py:684] Resolved architecture: MiniCPMO45Code2Wav
|
||||
INFO 09-23 17:19:53 [model.py:2021] Using max model len 4096
|
||||
WARNING 09-23 17:19:53 [arg_utils.py:2780] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs.
|
||||
INFO 09-23 17:19:53 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 17:19:53 [stage_runtime.py:853] [stage_init] Stage-0 set runtime devices: 0
|
||||
INFO 09-23 17:20:06 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:20:06 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:09 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', speculative_config=None, tokenizer='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=\\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:12 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_7b13e02cd0c74a1ca40084f32dfc508e backend=nccl
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:12 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:12 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:12 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:12 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [base.py:138] Offloader set to UVAOffloader
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:21 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:21 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:21 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:20:21 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:21 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:22 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:22 [flash_attn.py:897] Using FlashAttention version 2
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:24 [uva.py:65] Total CPU offloaded parameters: 5.03
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:24 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 25.00 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:20:24 [weight_utils.py:886] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880)
|
||||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880)
|
||||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 3.83it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880)
|
||||
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:00<00:00, 2.65it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880)
|
||||
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:00<00:00, 3.25it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.82it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.92it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:19 [default_loader.py:430] Loading weights took 114.68 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:20 [gpu_model_runner.py:5471] Model loading took 10.84 GiB memory and 117.294120 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:20 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:21 [gpu_model_runner.py:6546] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 1 audio items of the maximum feature size.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) [transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:31 [base.py:148] Available KV cache memory: 3.43 GiB (device-level profiling)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:31 [kv_cache_utils.py:2032] GPU KV cache size: 24,960 tokens, Maximum concurrency for 2,048 tokens per request: 12.19x
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:22:31 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:31 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880)
|
||||
JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 7.54kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 7.54kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:31 [kernel_warmup.py:134] JIT kernel warmup finished in 0.16s.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:32 [ll_bf16.py:30] cuteDSL (CUTLASS Python) not available, ll_bf16_gemm disabled
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:32 [gpu_worker.py:860] Free memory on device (22.61/23.89 GiB) on startup. Desired GPU memory utilization is (0.61, 14.57 GiB). Actual usage is 11.01 GiB for consumed memory (weights + non-torch), 0.22 GiB for peak activation, and 0.0 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=3432351273` (3.2 GiB) to fit into requested memory, or `--kv-cache-memory=12060912128` (11.23 GiB) to fully utilize gpu memory. Current kv cache memory in use is 3.43 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:32 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:22:32 [torch_utils.py:265] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:32 [core.py:368] init engine (profile, create kv cache, warmup model) took 12.17 s
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:22:32 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:22:33 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:33 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) INFO 09-23 17:22:33 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 17:22:33 [stage_runtime.py:943] [StageRuntime] Stage 0 engine startup completed
|
||||
INFO 09-23 17:22:33 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:22:33 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
|
||||
INFO 09-23 17:22:33 [stage_runtime.py:958] [StageRuntime] Stage 0 initialized
|
||||
INFO 09-23 17:22:33 [stage_runtime.py:853] [stage_init] Stage-1 set runtime devices: 0
|
||||
INFO 09-23 17:22:48 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:22:48 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:22:52 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', speculative_config=None, tokenizer='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=512, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=\\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:22:54 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_b12cceabf1984b38b14ca47252b06883 backend=nccl
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:22:54 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:22:54 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:22:55 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:22:55 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) [transformers] Image processor MiniCPMVImageProcessor: kwargs ['max_slice_nums'] were applied for backward compatibility. To avoid this warning, add them to valid_kwargs: create a custom TypedDict extending ImagesKwargs with these keys and set it as the `valid_kwargs` class attribute.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:11 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:11 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:11 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:11 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [flash_attn.py:897] Using FlashAttention version 2
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 13.31 GiB.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:11 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (13.31 GiB).
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944)
|
||||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944)
|
||||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 5.55it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944)
|
||||
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:00<00:00, 2.74it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944)
|
||||
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:01<00:00, 2.21it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.01it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.22it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:18 [default_loader.py:430] Loading weights took 6.72 seconds
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:19 [gpu_model_runner.py:5471] Model loading took 0.65 GiB memory and 7.083642 seconds
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:19 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:19 [gpu_model_runner.py:6519] Skipping memory profiling for multimodal encoder and encoder cache.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:20 [kv_cache_utils.py:2032] GPU KV cache size: 4,368 tokens, Maximum concurrency for 512 tokens per request: 8.53x
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:20 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:20 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944)
|
||||
JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 9.26kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 9.26kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:20 [kernel_warmup.py:134] JIT kernel warmup finished in 0.11s.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:20 [ll_bf16.py:30] cuteDSL (CUTLASS Python) not available, ll_bf16_gemm disabled
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:20 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:21 [torch_utils.py:265] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:21 [core.py:368] init engine (profile, create kv cache, warmup model) took 1.53 s
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:21 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) WARNING 09-23 17:23:21 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:21 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=22944) INFO 09-23 17:23:21 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 17:23:21 [stage_runtime.py:943] [StageRuntime] Stage 1 engine startup completed
|
||||
INFO 09-23 17:23:21 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-1 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:23:21 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-1 [rep-0] EngineCore running
|
||||
INFO 09-23 17:23:21 [stage_runtime.py:958] [StageRuntime] Stage 1 initialized
|
||||
INFO 09-23 17:23:21 [stage_runtime.py:853] [stage_init] Stage-2 set runtime devices: 0
|
||||
INFO 09-23 17:23:33 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:23:33 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:37 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', speculative_config=None, tokenizer='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=\\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=False, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_4c8c7d9dfb3648ea8f6653978d6573f0 backend=nccl
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:38 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:38 [gpu_generation_worker.py:103] OMNI GPUGenerationWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:38 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:38 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:38 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:39 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 11.09 GiB.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:39 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (11.09 GiB).
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144)
|
||||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144)
|
||||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 5.73it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144)
|
||||
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:00<00:00, 7.63it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 6.59it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 6.62it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:40 [step_audio2_token2wav.py:163] Loading Token2Wav models from: \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5\assets\token2wav
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:56 [step_audio2_token2wav.py:201] Token2Wav models loaded successfully
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:57 [batched_token2wav.py:227] Enabling HiFT CUDA Graph with batch sizes [1]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:57 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 50, 0)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:58 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 58, 3840)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:58 [batched_token2wav.py:235] HiFT CUDA Graph captured successfully
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:58 [batched_token2wav.py:246] CFM CUDA Graph enabled (max_graphs=32)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:58 [default_loader.py:430] Loading weights took 18.36 seconds
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:58 [gpu_model_runner.py:5471] Model loading took 3.36 GiB memory and 18.801405 seconds
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:59 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:59 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:59 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:59 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144)
|
||||
JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 142.85kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:59 [kernel_warmup.py:134] JIT kernel warmup finished in 0.01s.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:59 [ll_bf16.py:30] cuteDSL (CUTLASS Python) not available, ll_bf16_gemm disabled
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:23:59 [gpu_generation_model_runner.py:558] Dummy sampler run is not implemented for generation model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:23:59 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:24:00 [torch_utils.py:265] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:00 [core.py:368] init engine (profile, create kv cache, warmup model) took 0.97 s
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:24:01 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:24:01 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:01 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:24:01 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:24:01 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:01 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:01 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
INFO 09-23 17:24:01 [stage_runtime.py:943] [StageRuntime] Stage 2 engine startup completed
|
||||
INFO 09-23 17:24:01 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-2 [rep-0] initializing EngineCore
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:01 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
INFO 09-23 17:24:02 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-2 [rep-0] EngineCore running
|
||||
INFO 09-23 17:24:02 [stage_runtime.py:958] [StageRuntime] Stage 2 initialized
|
||||
[transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
[transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
[transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
INFO 09-23 17:24:17 [orchestrator.py:604] [Orchestrator] Starting event loop
|
||||
INFO 09-23 17:24:17 [async_omni_engine.py:323] [AsyncOmniEngine] Orchestrator ready with 3 stages
|
||||
INFO 09-23 17:24:17 [omni_base.py:235] [Omni] AsyncOmniEngine initialized in 268.47 seconds
|
||||
INFO 09-23 17:24:17 [omni_base.py:255] [Omni] Initialized with 3 stages for model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5
|
||||
WARNING 09-23 17:24:17 [input_processor.py:315] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
|
||||
|
||||
(StageEngineCoreProc_stage0_replica0 pid=38880) WARNING 09-23 17:24:46 [gpu_ar_model_runner.py:1464] prefer_model_sampler model MiniCPMO45OmniForConditionalGeneration returned None from sample(); falling back to the default sampler.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:52 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 100), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 0, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 100, 128), 'torch.float32', 'cuda:0')) (cache=1/32, stats={'calls': 1, 'hits': 0, 'captures': 1, 'flushes': 0, 'eager': 0, 'cache_size': 0}) [allocated 3573.7 MiB (+32.1), reserved 5392.0 MiB (+28.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:52 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 50), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 100, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 150, 128), 'torch.float32', 'cuda:0')) (cache=2/32, stats={'calls': 11, 'hits': 9, 'captures': 2, 'flushes': 0, 'eager': 0, 'cache_size': 1}) [allocated 3910.1 MiB (+32.0), reserved 5730.0 MiB (+26.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:52 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 50), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 150, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 200, 128), 'torch.float32', 'cuda:0')) (cache=3/32, stats={'calls': 21, 'hits': 18, 'captures': 3, 'flushes': 0, 'eager': 0, 'cache_size': 2}) [allocated 4122.7 MiB (+32.0), reserved 6032.0 MiB (+-102.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:53 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 8), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 200, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 208, 128), 'torch.float32', 'cuda:0')) (cache=4/32, stats={'calls': 31, 'hits': 27, 'captures': 4, 'flushes': 0, 'eager': 0, 'cache_size': 3}) [allocated 4335.3 MiB (+32.0), reserved 6190.0 MiB (+-350.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:53 [cuda_graph_wrapper.py:120] Lazily capturing HiFT CUDA Graph for shape (1, 16, 3840)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) WARNING 09-23 17:24:53 [jit_monitor.py:141] Triton kernel JIT compilation during inference: SnakeBeta._init_triton.<locals>._kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
|
||||
Processed prompts: 0%| | 0/1 [00:00<?, ?it/s]remark: snake_activation.py:48:0: 1658 instructions in function
|
||||
(StageEngineCoreProc_stage2_replica0 pid=42144) INFO 09-23 17:24:53 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 16, 3840)
|
||||
|
||||
Processed prompts: 100%|██████████| 1/1 [00:08<00:00, 8.26s/it]
|
||||
Processed prompts: 100%|██████████| 1/1 [00:08<00:00, 8.26s/it]
|
||||
query type: use_audio
|
||||
use_tts (tts_bos): True
|
||||
Request ID: 0_85a27d56-baee-49f0-a003-94069961be17, Text saved to C:\Users\zhout\w2\minicpmo_native_audio\0_85a27d56-baee-49f0-a003-94069961be17.txt
|
||||
Request ID: 0_85a27d56-baee-49f0-a003-94069961be17, Saved audio to C:\Users\zhout\w2\minicpmo_native_audio\output_0_85a27d56-baee-49f0-a003-94069961be17.wav
|
||||
INFO 09-23 17:24:53 [omni_base.py:781] [Omni] Shutting down
|
||||
INFO 09-23 17:24:53 [async_omni_engine.py:1807] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
INFO 09-23 17:24:53 [orchestrator.py:705] [Orchestrator] Received shutdown signal
|
||||
INFO 09-23 17:24:53 [orchestrator.py:3092] [Orchestrator] Shutting down all 3 client(s)
|
||||
INFO 09-23 17:24:53 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:24:53 [core_client.py:691] [shutdown] MPClient: stopping engine manager
|
||||
INFO 09-23 17:24:53 [utils.py:626] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
INFO 09-23 17:24:54 [core_client.py:693] [shutdown] MPClient: engine manager stopped
|
||||
INFO 09-23 17:24:54 [core_client.py:694] [shutdown] MPClient: cleaning up background resources
|
||||
INFO 09-23 17:24:54 [core_client.py:696] [shutdown] MPClient: complete
|
||||
INFO 09-23 17:24:54 [stage_pool.py:1354] [StagePool] Stage 0 replica 0 shut down
|
||||
INFO 09-23 17:24:54 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:24:54 [utils.py:626] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage1_replica0
|
||||
INFO 09-23 17:24:54 [stage_pool.py:1354] [StagePool] Stage 1 replica 0 shut down
|
||||
INFO 09-23 17:24:54 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:24:54 [utils.py:626] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage2_replica0
|
||||
INFO 09-23 17:24:55 [stage_pool.py:1354] [StagePool] Stage 2 replica 0 shut down
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"scope": "native Windows RTX 5090 Laptop, real MiniCPM-o checkpoint, one synthetic 440 Hz 2 s audio input to text+speech request",
|
||||
"text": "A loud whistling sound.",
|
||||
"token_count": 9,
|
||||
"text_match_wsl": true,
|
||||
"token_ids_match_wsl": true,
|
||||
"wav": {
|
||||
"sample_rate_hz": 24000,
|
||||
"frames": 51840,
|
||||
"duration_s": 2.16,
|
||||
"nonzero_samples": 51205,
|
||||
"rms_float": 0.03491710952527342,
|
||||
"sha256": "f558287642dbd6da52449ea51c5b1029a6ed375847b48b275d8b5284f77d04cc"
|
||||
}
|
||||
}
|
||||
BIN
Binary file not shown.
+16
@@ -0,0 +1,16 @@
|
||||
Prompt:
|
||||
<|im_start|>system
|
||||
You are MiniCPM-o, a helpful multimodal assistant that can understand images, audio and video, and respond in text and speech.<|im_end|>
|
||||
<|im_start|>user
|
||||
(<image>./</image>)
|
||||
What is the content of this image?<|im_end|>
|
||||
<|im_start|>assistant
|
||||
<think>
|
||||
|
||||
</think>
|
||||
|
||||
<|tts_bos|>
|
||||
vllm_text_output:
|
||||
The image shows a solid red square centered on a white background. There are no other elements, text, or details visible.
|
||||
vllm_token_ids:
|
||||
[785, 2168, 4933, 264, 6437, 2518, 9334, 30188, 389, 264, 4158, 4004, 13, 2619, 525, 902, 1008, 5424, 11, 1467, 11, 476, 3565, 9434, 13, 151704, 151645]
|
||||
+274
@@ -0,0 +1,274 @@
|
||||
INFO 09-23 17:12:37 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:12:37 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
====================
|
||||
vllm version: 0.29.0
|
||||
====================
|
||||
INFO 09-23 17:12:41 [omni_base.py:216] [Omni] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5
|
||||
INFO 09-23 17:12:41 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5
|
||||
INFO 09-23 17:12:42 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 3 stages
|
||||
INFO 09-23 17:12:42 [stage_init_utils.py:768] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
INFO 09-23 17:12:42 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations
|
||||
WARNING 09-23 17:12:42 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:12:42 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:12:42 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:12:43 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:12:43 [model.py:2021] Using max model len 2048
|
||||
WARNING 09-23 17:12:43 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:12:44 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
WARNING 09-23 17:12:44 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-23 17:12:44 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-23 17:12:44 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 17:12:44 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
INFO 09-23 17:12:44 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-23 17:12:44 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:12:44 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:12:45 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:12:45 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:12:45 [model.py:2021] Using max model len 512
|
||||
WARNING 09-23 17:12:45 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:12:45 [model.py:2021] Using max model len 512
|
||||
INFO 09-23 17:12:45 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=512.
|
||||
INFO 09-23 17:12:45 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
WARNING 09-23 17:12:45 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:12:45 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:12:45 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:12:46 [model.py:684] Resolved architecture: MiniCPMO45Code2Wav
|
||||
INFO 09-23 17:12:46 [model.py:2021] Using max model len 4096
|
||||
WARNING 09-23 17:12:46 [arg_utils.py:2780] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs.
|
||||
INFO 09-23 17:12:46 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 17:12:46 [stage_runtime.py:853] [stage_init] Stage-0 set runtime devices: 0
|
||||
INFO 09-23 17:13:03 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:13:03 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:07 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', speculative_config=None, tokenizer='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=\\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:10 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_431f956fcdf44084bd7cd31ffc24916b backend=nccl
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:10 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:10 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:11 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:11 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:18 [base.py:138] Offloader set to UVAOffloader
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:18 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:18 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:18 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:19 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:13:19 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:19 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:19 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:19 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:20 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:20 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:20 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:20 [flash_attn.py:897] Using FlashAttention version 2
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:22 [uva.py:65] Total CPU offloaded parameters: 5.03
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:23 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 8.21 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:13:23 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (8.21 GiB).
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488)
|
||||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488)
|
||||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 1.91it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488)
|
||||
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.67it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488)
|
||||
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:01<00:00, 2.20it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.44it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.22it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:22 [default_loader.py:430] Loading weights took 179.57 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:24 [gpu_model_runner.py:5471] Model loading took 11.8 GiB memory and 183.976371 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:25 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:25 [gpu_model_runner.py:6546] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 1 image items of the maximum feature size.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) [transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) C:\Users\zhout\w2\omni029venv\Lib\site-packages\transformers\modeling_attn_mask_utils.py:202: FutureWarning: The attention mask API under `transformers.modeling_attn_mask_utils` (`AttentionMaskConverter`) is deprecated and will be removed in Transformers v5.10. Please use the new API in `transformers.masking_utils`.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) warnings.warn(DEPRECATION_MESSAGE, FutureWarning)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:37 [base.py:148] Available KV cache memory: 0.82 GiB (device-level profiling)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:37 [kv_cache_utils.py:2032] GPU KV cache size: 5,936 tokens, Maximum concurrency for 2,048 tokens per request: 2.90x
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:16:37 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:37 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488)
|
||||
JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 5.93kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 5.93kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:37 [kernel_warmup.py:134] JIT kernel warmup finished in 0.18s.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:38 [ll_bf16.py:30] cuteDSL (CUTLASS Python) not available, ll_bf16_gemm disabled
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:38 [gpu_worker.py:860] Free memory on device (22.61/23.89 GiB) on startup. Desired GPU memory utilization is (0.61, 14.57 GiB). Actual usage is 12.3 GiB for consumed memory (weights + non-torch), 1.88 GiB for peak activation, and 0.0 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=266923049` (0.25 GiB) to fit into requested memory, or `--kv-cache-memory=8895483904` (8.28 GiB) to fully utilize gpu memory. Current kv cache memory in use is 0.82 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:38 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:16:38 [torch_utils.py:265] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:38 [core.py:368] init engine (profile, create kv cache, warmup model) took 13.25 s
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:16:38 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:16:38 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:38 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) INFO 09-23 17:16:38 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 17:16:38 [stage_runtime.py:943] [StageRuntime] Stage 0 engine startup completed
|
||||
INFO 09-23 17:16:38 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:16:39 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
|
||||
INFO 09-23 17:16:39 [stage_runtime.py:958] [StageRuntime] Stage 0 initialized
|
||||
INFO 09-23 17:16:39 [stage_runtime.py:853] [stage_init] Stage-1 set runtime devices: 0
|
||||
INFO 09-23 17:16:56 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:16:56 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:00 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', speculative_config=None, tokenizer='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=512, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=\\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:03 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_aa9f5ce7e8b54624b73d3744456143a5 backend=nccl
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:03 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:03 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:04 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:04 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) [transformers] Image processor MiniCPMVImageProcessor: kwargs ['max_slice_nums'] were applied for backward compatibility. To avoid this warning, add them to valid_kwargs: create a custom TypedDict extending ImagesKwargs with these keys and set it as the `valid_kwargs` class attribute.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:19 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:19 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:19 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:19 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:19 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:20 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:20 [flash_attn.py:897] Using FlashAttention version 2
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:20 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 17.15 GiB.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:20 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (17.15 GiB).
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724)
|
||||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724)
|
||||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 3.24it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724)
|
||||
Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:00<00:00, 3.98it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724)
|
||||
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:00<00:00, 4.64it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 3.04it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 3.34it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:27 [default_loader.py:430] Loading weights took 7.01 seconds
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:27 [gpu_model_runner.py:5471] Model loading took 0.65 GiB memory and 7.350049 seconds
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:28 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:28 [gpu_model_runner.py:6519] Skipping memory profiling for multimodal encoder and encoder cache.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:29 [kv_cache_utils.py:2032] GPU KV cache size: 4,368 tokens, Maximum concurrency for 512 tokens per request: 8.53x
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:29 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:29 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724)
|
||||
JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 8.97kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 8.97kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:29 [kernel_warmup.py:134] JIT kernel warmup finished in 0.11s.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:29 [ll_bf16.py:30] cuteDSL (CUTLASS Python) not available, ll_bf16_gemm disabled
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:29 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:30 [torch_utils.py:265] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:30 [core.py:368] init engine (profile, create kv cache, warmup model) took 1.65 s
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:30 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) WARNING 09-23 17:17:30 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:30 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=55724) INFO 09-23 17:17:30 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 17:17:30 [stage_runtime.py:943] [StageRuntime] Stage 1 engine startup completed
|
||||
INFO 09-23 17:17:30 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-1 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:17:30 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-1 [rep-0] EngineCore running
|
||||
INFO 09-23 17:17:30 [stage_runtime.py:958] [StageRuntime] Stage 1 initialized
|
||||
INFO 09-23 17:17:30 [stage_runtime.py:853] [stage_init] Stage-2 set runtime devices: 0
|
||||
INFO 09-23 17:17:43 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:17:43 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:47 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', speculative_config=None, tokenizer='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=\\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=False, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:48 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_b39c4cfa1016448ba676b1a7438e3913 backend=nccl
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:17:48 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:48 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:17:49 [gpu_generation_worker.py:103] OMNI GPUGenerationWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:49 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:49 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:17:49 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:17:49 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:49 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:50 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 17.46 GiB. Available RAM: 23.12 GiB.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:50 [weight_utils.py:886] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204)
|
||||
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204)
|
||||
Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 5.96it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204)
|
||||
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:00<00:00, 9.06it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 5.96it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 6.41it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:17:50 [step_audio2_token2wav.py:163] Loading Token2Wav models from: \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5\assets\token2wav
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:03 [step_audio2_token2wav.py:201] Token2Wav models loaded successfully
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:04 [batched_token2wav.py:227] Enabling HiFT CUDA Graph with batch sizes [1]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:04 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 50, 0)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:04 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 58, 3840)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:04 [batched_token2wav.py:235] HiFT CUDA Graph captured successfully
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:04 [batched_token2wav.py:246] CFM CUDA Graph enabled (max_graphs=32)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:04 [default_loader.py:430] Loading weights took 14.73 seconds
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:05 [gpu_model_runner.py:5471] Model loading took 3.36 GiB memory and 15.171103 seconds
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:05 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:05 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:18:05 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:06 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204)
|
||||
JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 166.62kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:06 [kernel_warmup.py:134] JIT kernel warmup finished in 0.01s.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:06 [ll_bf16.py:30] cuteDSL (CUTLASS Python) not available, ll_bf16_gemm disabled
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:18:06 [gpu_generation_model_runner.py:558] Dummy sampler run is not implemented for generation model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:06 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:18:06 [torch_utils.py:265] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:06 [core.py:368] init engine (profile, create kv cache, warmup model) took 0.72 s
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:18:07 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:18:07 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:07 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:18:07 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:18:07 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:07 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:07 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
INFO 09-23 17:18:07 [stage_runtime.py:943] [StageRuntime] Stage 2 engine startup completed
|
||||
INFO 09-23 17:18:07 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-2 [rep-0] initializing EngineCore
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:07 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
INFO 09-23 17:18:08 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-2 [rep-0] EngineCore running
|
||||
INFO 09-23 17:18:08 [stage_runtime.py:958] [StageRuntime] Stage 2 initialized
|
||||
[transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
[transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
[transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
INFO 09-23 17:18:15 [orchestrator.py:604] [Orchestrator] Starting event loop
|
||||
INFO 09-23 17:18:15 [async_omni_engine.py:323] [AsyncOmniEngine] Orchestrator ready with 3 stages
|
||||
INFO 09-23 17:18:15 [omni_base.py:235] [Omni] AsyncOmniEngine initialized in 334.16 seconds
|
||||
INFO 09-23 17:18:15 [omni_base.py:255] [Omni] Initialized with 3 stages for model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\MiniCPM-o-4_5
|
||||
WARNING 09-23 17:18:15 [input_processor.py:315] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
|
||||
|
||||
(StageEngineCoreProc_stage0_replica0 pid=24488) WARNING 09-23 17:18:25 [gpu_ar_model_runner.py:1464] prefer_model_sampler model MiniCPMO45OmniForConditionalGeneration returned None from sample(); falling back to the default sampler.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:40 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 302), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 0, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 302, 128), 'torch.float32', 'cuda:0')) (cache=1/32, stats={'calls': 1, 'hits': 0, 'captures': 1, 'flushes': 0, 'eager': 0, 'cache_size': 0}) [allocated 3633.9 MiB (+32.2), reserved 5464.0 MiB (+22.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:40 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 50), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 302, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 352, 128), 'torch.float32', 'cuda:0')) (cache=2/32, stats={'calls': 11, 'hits': 9, 'captures': 2, 'flushes': 0, 'eager': 0, 'cache_size': 1}) [allocated 4539.7 MiB (+32.0), reserved 6378.0 MiB (+28.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:40 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 50), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 352, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 402, 128), 'torch.float32', 'cuda:0')) (cache=3/32, stats={'calls': 21, 'hits': 18, 'captures': 3, 'flushes': 0, 'eager': 0, 'cache_size': 2}) [allocated 4803.9 MiB (+32.0), reserved 6956.0 MiB (+-352.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:41 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 50), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 402, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 452, 128), 'torch.float32', 'cuda:0')) (cache=4/32, stats={'calls': 31, 'hits': 27, 'captures': 4, 'flushes': 0, 'eager': 0, 'cache_size': 3}) [allocated 5079.1 MiB (+32.0), reserved 7096.0 MiB (+-876.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:41 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 10), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 402, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 412, 128), 'torch.float32', 'cuda:0')) (cache=5/32, stats={'calls': 61, 'hits': 56, 'captures': 5, 'flushes': 0, 'eager': 0, 'cache_size': 4}) [allocated 5206.6 MiB (+32.0), reserved 7190.0 MiB (+-1550.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:41 [cuda_graph_wrapper.py:120] Lazily capturing HiFT CUDA Graph for shape (1, 18, 3840)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) WARNING 09-23 17:18:41 [jit_monitor.py:141] Triton kernel JIT compilation during inference: SnakeBeta._init_triton.<locals>._kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=33204) INFO 09-23 17:18:42 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 18, 3840)
|
||||
Processed prompts: 0%| | 0/1 [00:00<?, ?it/s]
|
||||
Processed prompts: 100%|██████████| 1/1 [00:17<00:00, 17.15s/it]
|
||||
Processed prompts: 100%|██████████| 1/1 [00:17<00:00, 17.15s/it]
|
||||
query type: use_image
|
||||
use_tts (tts_bos): True
|
||||
Request ID: 0_694a20bc-73ff-4659-8a2a-766061016444, Text saved to C:\Users\zhout\w2\minicpmo_native_image\0_694a20bc-73ff-4659-8a2a-766061016444.txt
|
||||
Request ID: 0_694a20bc-73ff-4659-8a2a-766061016444, Saved audio to C:\Users\zhout\w2\minicpmo_native_image\output_0_694a20bc-73ff-4659-8a2a-766061016444.wav
|
||||
INFO 09-23 17:18:42 [omni_base.py:781] [Omni] Shutting down
|
||||
INFO 09-23 17:18:42 [async_omni_engine.py:1807] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
INFO 09-23 17:18:42 [orchestrator.py:705] [Orchestrator] Received shutdown signal
|
||||
INFO 09-23 17:18:42 [orchestrator.py:3092] [Orchestrator] Shutting down all 3 client(s)
|
||||
INFO 09-23 17:18:42 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:18:42 [core_client.py:691] [shutdown] MPClient: stopping engine manager
|
||||
INFO 09-23 17:18:42 [utils.py:626] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
INFO 09-23 17:18:42 [core_client.py:693] [shutdown] MPClient: engine manager stopped
|
||||
INFO 09-23 17:18:42 [core_client.py:694] [shutdown] MPClient: cleaning up background resources
|
||||
INFO 09-23 17:18:42 [core_client.py:696] [shutdown] MPClient: complete
|
||||
INFO 09-23 17:18:42 [stage_pool.py:1354] [StagePool] Stage 0 replica 0 shut down
|
||||
INFO 09-23 17:18:42 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:18:42 [utils.py:626] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage1_replica0
|
||||
INFO 09-23 17:18:43 [stage_pool.py:1354] [StagePool] Stage 1 replica 0 shut down
|
||||
INFO 09-23 17:18:43 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:18:43 [utils.py:626] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage2_replica0
|
||||
INFO 09-23 17:18:43 [stage_pool.py:1354] [StagePool] Stage 2 replica 0 shut down
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"scope": "native Windows RTX 5090 Laptop, real MiniCPM-o checkpoint, one synthetic 448x448 image to text+speech request",
|
||||
"text": "The image shows a solid red square centered on a white background. There are no other elements, text, or details visible.",
|
||||
"token_count": 27,
|
||||
"text_match_wsl": true,
|
||||
"token_ids_match_wsl": true,
|
||||
"wav": {
|
||||
"sample_rate_hz": 24000,
|
||||
"frames": 124800,
|
||||
"duration_s": 5.2,
|
||||
"nonzero_samples": 124412,
|
||||
"rms_float": 0.08686131048217999,
|
||||
"sha256": "9a819ed5d32331c59ee7e9f85c76ac1a3e52187f262ef252e34f05a38a3d4a9d"
|
||||
}
|
||||
}
|
||||
BIN
Binary file not shown.
@@ -11,7 +11,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i
|
||||
| PC HX370: CPU / WSL | profile completed | scoped E2E: short stream | NOT E2E | scoped E2E: text → speech | synthetic policy pass |
|
||||
| PC HX370: CPU / native Windows | NOT E2E | NOT E2E | NOT E2E | NOT E2E | synthetic policy pass |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | profile completed | profile completed | scoped E2E: text + image | scoped E2E: text/image/audio → speech | synthetic policy pass |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | profile completed | profile completed | NOT E2E | scoped E2E: text → speech | synthetic policy pass |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | profile completed | profile completed | NOT E2E | scoped E2E: text/image/audio → speech | synthetic policy pass |
|
||||
| PC HX370 + Radeon 890M / Windows worker | NOT E2E | NOT E2E | NOT E2E | NOT E2E | NOT E2E |
|
||||
| PC HX370 + AMD NPU / Windows worker | NOT E2E | NOT E2E | NOT E2E | NOT E2E | NOT E2E |
|
||||
| PC CPU+iGPU+NPU +/- discrete GPU, joint model execution | NOT E2E | NOT E2E | NOT E2E | NOT E2E | NOT E2E |
|
||||
@@ -43,7 +43,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | Spark-X2.5 | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Broaden only with a named workload/device/precision qualification. |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | Qwen3-TTS 0.6B CustomVoice | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Resolve and measure playback stalls; run quality, interruption and thermal tests. |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | Qwen3.8-27B | NOT_E2E_PROFILEABLE | The Omni pipeline binding now exists and passed scoped text/image cases in WSL, but no native Windows Omni run has verified the NVFP4 checkpoint, memory admission or modality behavior. | Run the same checkpoint and explicit context on native Windows, then broaden quality and profiling; [WSL-only evidence](../../../e2e_recovery_20260923/README.md). |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | MiniCPM-o 4.5 | SCOPED_E2E | The verified checkpoint completed native Windows three-stage BF16 text-to-text+speech requests with 5 GiB thinker host offload and explicit 2,048/512-token limits. Text and all 32 thinker token IDs matched WSL CUDA; a 5.2 s WAV was nonzero. After a 300 s default startup timeout on the first attempt, a 900/600 s limit allowed a functional pass. A separate 1-warmup/20-measured serial run had request wall p50 18.38 s/p95 18.73 s; sampled host available RAM fell to 0.78 GB and pagefile use reached 11.17 GB. Speech meaning remains unverified. | Check speech intelligibility/alignment, other input modalities, concurrency, interruption, loading-peak admission and sustained thermal/power behavior before broader qualification; [native evidence](../../../e2e_expansion_20260923/README.md). |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | MiniCPM-o 4.5 | SCOPED_E2E | The verified checkpoint completed native Windows three-stage BF16 text, one synthetic image, and one synthetic audio input to text+speech in separate constrained plans with 5 GiB thinker host offload and explicit 2,048/512-token limits. Text and all 32 thinker token IDs matched WSL CUDA; a 5.2 s WAV was nonzero. The red-square image produced the same description and 27 thinker token IDs as WSL plus a nonzero 5.2 s WAV in 17.15 s. The 2 s/440 Hz tone produced the same whistling description and nine IDs as WSL plus a nonzero 2.16 s WAV in 8.26 s. The initial text run needed 900/600 s startup limits after a 300 s timeout; a separate 1-warmup/20-measured serial text run had request wall p50 18.38 s/p95 18.73 s, with sampled host available RAM as low as 0.78 GB and pagefile use up to 11.17 GB. These synthetic inputs do not establish general image/audio or speech quality. | Check real image/audio suites, video input, speech intelligibility/alignment, concurrency, interruption, loading-peak admission and sustained thermal/power behavior before broader qualification; [native evidence](../../../e2e_expansion_20260923/README.md). |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | InternVLA-A1 | SYNTHETIC_POLICY_PASS | The Place_Markpen checkpoint strict-loaded from a UNC path and produced finite `[1, 50, 32]` actions on synthetic zero observations/noise. Policy/output CUDA placement was checked; one native forward took 1.12 s, with 6.68 GB peak allocated. Relative L2 difference from WSL CUDA was 0.83% (no task tolerance). | Validate real observations/reference actions, units/timestamps, memory and repeated latency before task qualification; [native evidence](../../../e2e_expansion_20260923/README.md). |
|
||||
| PC HX370 + Radeon 890M / Windows worker | Spark-X2.5 | NOT_E2E_PROFILEABLE | No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. |
|
||||
| PC HX370 + Radeon 890M / Windows worker | Qwen3-TTS 0.6B CustomVoice | NOT_E2E_PROFILEABLE | No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. |
|
||||
|
||||
@@ -33,7 +33,10 @@ previous 0.58 budget failed the 2,048-token KV admission gate; the successful
|
||||
run still force-killed stage 0 during shutdown and reported a shared-memory
|
||||
cleanup warning. A separate 2 s, 440 Hz synthetic tone now traverses audio
|
||||
input to text+speech and yielded a plausible whistling description, also with
|
||||
a shared-memory cleanup warning. Broader image/audio quality, video and clean
|
||||
a shared-memory cleanup warning. Native Windows RTX also completed these two
|
||||
separate synthetic image and tone inputs through all three stages, returning
|
||||
the same respective text and thinker token IDs as WSL with nonzero WAVs.
|
||||
Broader image/audio quality, video, repeated multimodal latency and clean
|
||||
lifecycle remain open.
|
||||
|
||||
## Architecture and prerequisites
|
||||
@@ -91,7 +94,7 @@ capability and lifecycle implementation; a state-handle type alone is insufficie
|
||||
| M1 | S25 Spark | Device-local Omni-compatible controller; embedding, prefill, continuous decode, ring/full KV, head and sampler; resident state | Reference checks, at least 128 output tokens, 512-window/1024-bucket transitions, cancellation, resident memory and sustained profile |
|
||||
| M2 | Qwen3-TTS | Resolve desktop playback and shutdown findings and CPU compatibility; then mobile talker/predictor plus GPU vocoder | Complete PCM and tail, history/chunk correctness, interruption/recovery, quality checks, no post-startup underruns and RTF below 1 in the declared workload |
|
||||
| M3 | AMD stages | Connect compatible real encoders or other useful coarse stages through current Omni workers; start from the existing 890M vision evidence | Actual node/device placement, numerical and task quality, complete downstream output, shared-memory and handoff costs; reject unhelpful splits |
|
||||
| M4a | MiniCPM-o 4.5 | Three-stage desktop text+WAV produced coherent text and nonzero WAV on RTX 5090 Laptop under WSL and native Windows, and WSL CPU, using separate constrained plans. One WSL RTX synthetic image passed through vision encoding to correct text and nonzero WAV after increasing the thinker GPU budget from 0.58 to 0.61; one synthetic tone passed through audio input to plausible text and nonzero WAV. Shutdown cleanup remains open. The CPU plan needed vLLM 0.28/0.29 processor compatibility and used swap; 20 serial CPU requests measured p50 21.86 s/p95 22.97 s. Native Windows needed longer startup limits; 20 serial requests measured p50 18.38 s/p95 18.73 s with host RAM/pagefile pressure. Add video encoder and persistent-state reference before mobile artifacts | Speech intelligibility/alignment, loading-peak admission, concurrency, broad image/audio-understanding suites, video, shutdown lifecycle, then combined streaming and interruption |
|
||||
| M4a | MiniCPM-o 4.5 | Three-stage desktop text+WAV produced coherent text and nonzero WAV on RTX 5090 Laptop under WSL and native Windows, and WSL CPU, using separate constrained plans. One synthetic red-square image and one synthetic tone each passed through the three-stage WSL RTX and native Windows RTX paths to text and nonzero WAV; matching inputs produced identical text and thinker token IDs across the two OS environments. The WSL image plan needed a thinker GPU budget increase from 0.58 to 0.61 for KV admission. WSL multimodal shutdown cleanup remains open. The CPU plan needed vLLM 0.28/0.29 processor compatibility and used swap; 20 serial CPU requests measured p50 21.86 s/p95 22.97 s. Native Windows needed longer startup limits; 20 serial text requests measured p50 18.38 s/p95 18.73 s with host RAM/pagefile pressure. Add video encoder and persistent-state reference before mobile artifacts | Speech intelligibility/alignment, loading-peak admission, concurrency, broad image/audio-understanding suites, video, shutdown lifecycle, then combined streaming and interruption |
|
||||
| M4b | InternVLA-A1 | Real weights, observation preprocessing, shared prefix state, iterative action head and versioned action metadata; strict-load synthetic policy forwards now pass on WSL and native Windows, each on CPU and RTX 5090 Laptop | Reference action agreement on real observations, units/order/timestamps, observation age, stale-input handling and application-specific deadline reporting |
|
||||
| Independent | Qwen3.8-27B | Public Omni pipeline binding; text, then image/video and longer contexts; same-model CPU/iGPU artifact and explicit offload evaluation | Modality-specific quality, actual loading/runtime memory and measured performance; each precision/context receives its own qualification |
|
||||
|
||||
|
||||
Reference in new issue
Block a user