Qualify scoped MiniCPM-o image and audio paths on WSL CPU
This commit is contained in:
tzhouam committed
1 parent
b29a6ca99f
commit
10221f0665
14 files changed
+1404
-6
No files matched your search
@@ -1,11 +1,13 @@
|
||||
# 2026-09-23 WSL CPU and native Windows expansion
|
||||
|
||||
This extends the [60-cell device × model matrix](../e2e_profiling_20260922/evidence/summary/README.md) with previously unrun WSL CPU and native Windows paths. The September 22 timing tables remain historical. MiniCPM-o now has separate 20-request serial text-to-speech timing runs on WSL CPU and native Windows GPU; each InternVLA timing remains one observation. None is a sustained or full-modality profile. The [checkpoint manifests](../e2e_recovery_20260923/README.md) pin the real weights. No model, precision or context was silently substituted.
|
||||
This extends the [60-cell device × model matrix](../e2e_profiling_20260922/evidence/summary/README.md) with previously unrun WSL CPU and native Windows paths. The September 22 timing tables remain historical. MiniCPM-o now has separate 20-request serial text-to-speech timing runs on WSL CPU and native Windows GPU, plus one synthetic image and one synthetic audio input on each platform; each InternVLA timing remains one observation. None is a sustained or full-modality profile. The [checkpoint manifests](../e2e_recovery_20260923/README.md) pin the real weights. No model, precision or context was silently substituted.
|
||||
|
||||
| Device / model | Verified depth | Observed output and time | Remaining gate |
|
||||
|---|---|---|---|
|
||||
| WSL CPU / MiniCPM-o 4.5, revision `503e754207c94da6bb26850b4469f367c9ea3582` | Scoped complete three-stage text-to-text+speech path (P for this input only) | Coherent text and a 5.2 s mono 24 kHz WAV. First request 27.09 s after 73.65 s startup; separate 1-warmup/20-measured serial run: startup 73.89 s, request wall p50 21.86 s/p95 22.97 s. | Speech intelligibility/alignment, image/audio/video inputs, concurrency, interruption, power and thermal behavior. |
|
||||
| Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped complete three-stage text-to-text+speech path (P for this input only) | Coherent text with the same 32 thinker token IDs as WSL CUDA, and a nonzero 5.2 s mono 24 kHz WAV. First engine startup 297.20 s and request 78.75 s, including first-use graph/JIT work; separate 1-warmup/20-measured serial run: startup 239.05 s, request wall p50 18.38 s/p95 18.73 s. | Speech intelligibility/alignment, image/audio/video inputs, concurrency, interruption, loading peak and sustained behavior. |
|
||||
| WSL CPU / MiniCPM-o 4.5, revision `503e754207c94da6bb26850b4469f367c9ea3582` | Scoped complete three-stage text-to-text+speech path (P for this input only) | Coherent text and a 5.2 s mono 24 kHz WAV. First request 27.09 s after 73.65 s startup; separate 1-warmup/20-measured serial run: startup 73.89 s, request wall p50 21.86 s/p95 22.97 s. | Speech intelligibility/alignment, real image/audio quality, video input, concurrency, interruption, power and thermal behavior. |
|
||||
| WSL CPU / MiniCPM-o 4.5, same revision | Scoped three-stage **image**-to-text+speech path (P for one synthetic image) | For the 448×448 red square, returned a correct description and nonzero 5.2 s WAV. Startup 112.44 s; one request 37.70 s. Text and token IDs differed from WSL GPU while describing the same visible content. | Real-image suite, speech meaning, repeated latency, memory admission and clean shutdown. |
|
||||
| WSL CPU / MiniCPM-o 4.5, same revision | Scoped three-stage **audio**-to-text+speech path (P for one synthetic clip) | For the 2 s, 440 Hz tone, answered “A loud and high pitched beep” and produced a nonzero 2.16 s WAV. Startup 101.22 s; one request 17.30 s. Text and token IDs differed from WSL GPU. | Real-audio and transcription suites, speech meaning, repeated latency, memory admission and clean shutdown. |
|
||||
| Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped complete three-stage text-to-text+speech path (P for this input only) | Coherent text with the same 32 thinker token IDs as WSL CUDA, and a nonzero 5.2 s mono 24 kHz WAV. First engine startup 297.20 s and request 78.75 s, including first-use graph/JIT work; separate 1-warmup/20-measured serial run: startup 239.05 s, request wall p50 18.38 s/p95 18.73 s. | Speech intelligibility/alignment, real image/audio quality, video input, concurrency, interruption, loading peak and sustained behavior. |
|
||||
| WSL RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **image**-to-text+speech path (P for one synthetic image) | For a 448×448 red square on white, returned the correct visual description and a nonzero 5.2 s WAV. Engine startup 101.96 s; one request 17.49 s. | Broader visual quality, real images, audio/video inputs, speech intelligibility, repeated latency and cleanup reliability. |
|
||||
| WSL RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **audio**-to-text+speech path (P for one synthetic clip) | For a 2 s, 440 Hz tone at 16 kHz, answered “A loud whistling sound” and produced a nonzero 2.16 s WAV. Engine startup 105.94 s; one request 7.78 s. | Speech transcription/real audio quality, video input, speech intelligibility, repeated latency and shared-memory cleanup. |
|
||||
| Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **image**-to-text+speech path (P for one synthetic image) | For the same 448×448 red square, returned exactly the same text and 27 thinker token IDs as WSL CUDA, plus a nonzero 5.2 s WAV. Engine startup 334.16 s; one request 17.15 s. | Real-image quality, audio/video inputs, speech intelligibility, repeated latency, memory and sustained behavior. |
|
||||
@@ -18,6 +20,8 @@ Both runs used Ryzen AI 9 HX 370 on Ubuntu 26.04/WSL2 kernel `6.18.33.2-microsof
|
||||
|
||||
The separate [20-request profile](evidence/minicpmo/minicpmo_cpu_profile_20.json) records every measured output and sampled memory; the [JSONL](evidence/minicpmo/minicpmo_cpu_profile_20.jsonl) survives interruption, and the [run log](evidence/minicpmo/minicpmo_cpu_profile_20.log) retains worker details. One warmup was excluded. All 20 measured requests returned the same thinker token sequence, nonzero 124,800-sample audio, and request wall times 20.95–23.33 s; nearest-rank p50/p95 were 21.86/22.97 s (real-time factor 4.20/4.42 for 5.2 s of audio). Sampled WSL available RAM reached a minimum of 3.80 GB and swap use a maximum of 5.75 GB; summed process-tree RSS reached 29.04 GB but double-counts shared pages. These are 0.5 s samples during requests, not load-peak or unique-resident-memory measurements. Audio and text were returned together at request completion, so the profile cannot establish playable-stream latency, and no concurrency or 30-minute thermal run was performed.
|
||||
|
||||
The WSL CPU image and audio runs reused the same real checkpoint, BF16 three-stage plan, eager execution, 2,048/512-token limits, 1 GiB CPU KV cache per stage and `OMP_NUM_THREADS=8` as the CPU text run. Their [image plan](evidence/minicpmo_wsl_cpu_image/minicpmo_cpu_image_offline.yaml) and [audio plan](evidence/minicpmo_wsl_cpu_audio/minicpmo_cpu_audio_offline.yaml) each admit only the named input. They used the same [red-square image](evidence/minicpmo_wsl_image/red_square.png) and [440 Hz tone](evidence/minicpmo_wsl_audio/tone_440hz_2s_16khz.wav) as the GPU runs. The image [raw log](evidence/minicpmo_wsl_cpu_image/minicpmo_cpu_image.log), [text/token IDs](evidence/minicpmo_wsl_cpu_image/0_54927839-1ab7-4d0d-86a3-0d6290a84f13.txt), [WAV](evidence/minicpmo_wsl_cpu_image/output_0_54927839-1ab7-4d0d-86a3-0d6290a84f13.wav) and [artifact report](evidence/minicpmo_wsl_cpu_image/minicpmo_cpu_image_artifact_report.json) show a correct visual description and a nonzero 124,800-frame, 24 kHz WAV. The audio [raw log](evidence/minicpmo_wsl_cpu_audio/minicpmo_cpu_audio.log), [text/token IDs](evidence/minicpmo_wsl_cpu_audio/0_40b94cec-371c-47a8-ba61-4729a0eb1285.txt), [WAV](evidence/minicpmo_wsl_cpu_audio/output_0_40b94cec-371c-47a8-ba61-4729a0eb1285.wav) and [artifact report](evidence/minicpmo_wsl_cpu_audio/minicpmo_cpu_audio_artifact_report.json) show a plausible tone description and a nonzero 51,840-frame, 24 kHz WAV. CPU text and token IDs differ from GPU for both inputs; no reference-quality threshold was applied. Each run force-killed stage 0 at shutdown and reported shared-memory cleanup warnings. Manual `free -h` snapshots during startup/request showed swap around 5.3 GiB for image and 4.1 GiB for audio, but these are neither load peaks nor model-attributed memory measurements. The first image attempt [failed](evidence/minicpmo_wsl_cpu_image/minicpmo_cpu_image_wrong_source.log) because the CPU venv loaded an older editable Omni checkout lacking the existing `cosyvoice2` YAML compatibility fix; setting `PYTHONPATH` explicitly to this checkout corrected the source path. This is a setup failure, not a model-input failure.
|
||||
|
||||
Native Windows MiniCPM-o used the same [constrained CUDA deployment config](../e2e_recovery_20260923/evidence/minicpmo/minicpmo_24gb_offline.yaml) as the WSL CUDA run: BF16, eager vLLM 0.29.0, 2,048-token thinker / 512-token talker limits, text input only, one request and 5 GiB thinker host offload. The local Windows 11 build 26200 host has 63.12 GiB visible RAM, an RTX 5090 Laptop GPU (driver 610.71), Python 3.12.10, PyTorch 2.13.0+cu130, Transformers 5.14.1, `stepaudio2-minicpmo` 0.1.1 and `pyarrow` 25.0.1. The checkpoint was read through a local WSL UNC path. The example's then-default 300 s overall startup timeout [expired](evidence/minicpmo_windows/minicpmo_native_gpu.log) after stage initialization had nearly completed; rerunning with `--init-timeout 900 --stage-init-timeout 600` [completed](evidence/minicpmo_windows/minicpmo_native_gpu_timeout900.log) in 297.20 s startup and one 78.75 s request. The example default has since been raised to 600 s, matching Omni's default. The text [and token IDs](evidence/minicpmo_windows/0_b4835d8d-f4f8-48b9-85d2-bb7ef06d9249.txt) exactly match the WSL CUDA result. The [WAV](evidence/minicpmo_windows/output_0_b4835d8d-f4f8-48b9-85d2-bb7ef06d9249.wav) has 124,800 mono PCM16 frames at 24 kHz; 124,560 samples are nonzero. The [artifact report](evidence/minicpmo_windows/minicpmo_windows_artifact_report.json) records its checksum and signal statistics. One first-use request is not a latency distribution or speech-quality assessment; the distinct Windows timing must not be transferred to WSL or CPU.
|
||||
|
||||
The separate native Windows [20-request profile](evidence/minicpmo_windows/minicpmo_windows_gpu_profile20.json), [per-request JSONL](evidence/minicpmo_windows/minicpmo_windows_gpu_profile20.jsonl), and [raw log](evidence/minicpmo_windows/minicpmo_windows_gpu_profile20.log) used one warmup followed by 20 measured serial requests in one engine. All measured requests returned one consistent thinker token sequence and 124,800-sample audio with nonzero RMS. Nearest-rank request wall p50/p95 were 18.38/18.73 s (real-time factor 3.53/3.60 for 5.2 s of audio), with observed range 18.22–19.79 s. Engine startup was 239.05 s on this later, warm-filesystem run. During measured requests, 0.5 s samples found native host available RAM as low as 0.78 GB and pagefile use as high as 11.17 GB; summed process-tree RSS reached 21.08 GB but double-counts shared pages. These are host-wide samples, not model-only RAM or loading peak. Neither a playable-stream timing nor speech intelligibility, concurrency, interruption, power or a 30-minute thermal run was measured.
|
||||
@@ -57,6 +61,24 @@ export CUDA_VISIBLE_DEVICES= VLLM_TARGET_DEVICE=cpu VLLM_CPU_KVCACHE_SPACE=1 VLL
|
||||
--deploy-config benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo/minicpmo_cpu_offline.yaml \
|
||||
--warmups 1 --repeats 20 --output /tmp/minicpmo-cpu-profile-20.json
|
||||
|
||||
# Pin this checkout: the CPU venv also contains an older editable Omni checkout.
|
||||
export PYTHONPATH=/home/zhout/project/edge_infer/vllm-omni-edge
|
||||
/home/zhout/project/edge_infer/.venvs/omni-cpu/bin/python examples/offline_inference/minicpmo/end2end.py \
|
||||
--model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5 \
|
||||
--deploy-config benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_wsl_cpu_image/minicpmo_cpu_image_offline.yaml \
|
||||
--query-type use_image \
|
||||
--image-path benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_wsl_image/red_square.png \
|
||||
--modalities text,audio --num-prompts 1 --init-timeout 900 --stage-init-timeout 600 \
|
||||
--output-dir /tmp/minicpmo-wsl-cpu-image
|
||||
|
||||
/home/zhout/project/edge_infer/.venvs/omni-cpu/bin/python examples/offline_inference/minicpmo/end2end.py \
|
||||
--model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5 \
|
||||
--deploy-config benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_wsl_cpu_audio/minicpmo_cpu_audio_offline.yaml \
|
||||
--query-type use_audio \
|
||||
--audio-path benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_wsl_audio/tone_440hz_2s_16khz.wav \
|
||||
--modalities text,audio --num-prompts 1 --init-timeout 900 --stage-init-timeout 600 \
|
||||
--output-dir /tmp/minicpmo-wsl-cpu-audio
|
||||
|
||||
CUDA_VISIBLE_DEVICES=0 VLLM_TARGET_DEVICE=cuda VLLM_USE_FLASHINFER_SAMPLER=0 \
|
||||
/home/zhout/project/edge_infer/.venvs/omni-cuda-029/bin/python examples/offline_inference/minicpmo/end2end.py \
|
||||
--model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5 \
|
||||
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
Prompt:
|
||||
<|im_start|>system
|
||||
You are MiniCPM-o, a helpful multimodal assistant that can understand images, audio and video, and respond in text and speech.<|im_end|>
|
||||
<|im_start|>user
|
||||
(<audio>./</audio>)
|
||||
What is the content of this audio?<|im_end|>
|
||||
<|im_start|>assistant
|
||||
<think>
|
||||
|
||||
</think>
|
||||
|
||||
<|tts_bos|>
|
||||
vllm_text_output:
|
||||
A loud and high pitched beep.
|
||||
vllm_token_ids:
|
||||
[32, 17361, 323, 1550, 48998, 90954, 13, 151704, 151645]
|
||||
+250
@@ -0,0 +1,250 @@
|
||||
W0923 17:38:24.658000 2302365 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:38:24.703000 2302365 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:38:26 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:38:26 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-23 17:38:27 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:38:28 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:38:28 [patch.py:646] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:38:28 [patch.py:756] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
====================
|
||||
vllm version: 0.28.0
|
||||
====================
|
||||
INFO 09-23 17:38:28 [omni_base.py:216] [Omni] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 17:38:28 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 17:38:28 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 3 stages
|
||||
INFO 09-23 17:38:28 [stage_init_utils.py:768] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
INFO 09-23 17:38:28 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations
|
||||
WARNING 09-23 17:38:28 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:38:28 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:38:28 [arg_utils.py:1719] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:38:29 [model.py:672] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:38:29 [model.py:1965] Using max model len 2048
|
||||
WARNING 09-23 17:38:29 [model.py:963] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:38:29 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
WARNING 09-23 17:38:29 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-23 17:38:29 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-23 17:38:29 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
WARNING 09-23 17:38:29 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
INFO 09-23 17:38:29 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-23 17:38:29 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:38:29 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:38:29 [arg_utils.py:1719] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:38:29 [model.py:672] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:38:29 [model.py:1965] Using max model len 512
|
||||
WARNING 09-23 17:38:29 [model.py:963] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:38:29 [model.py:1965] Using max model len 512
|
||||
INFO 09-23 17:38:29 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=512.
|
||||
INFO 09-23 17:38:29 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
WARNING 09-23 17:38:29 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:38:29 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:38:29 [arg_utils.py:1719] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:38:29 [model.py:672] Resolved architecture: MiniCPMO45Code2Wav
|
||||
INFO 09-23 17:38:29 [model.py:1965] Using max model len 4096
|
||||
WARNING 09-23 17:38:29 [arg_utils.py:2664] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs.
|
||||
INFO 09-23 17:38:29 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-23 17:38:29 [stage_runtime.py:853] [stage_init] Stage-0 set runtime devices: 0
|
||||
W0923 17:38:33.734000 2302392 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:38:33.766000 2302392 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:38:34 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:38:34 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-23 17:38:35 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:38:36 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:38:36 [patch.py:646] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:38:36 [patch.py:756] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:36 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
[W923 17:38:37.553020102 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:38:37 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:38:37 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:37 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_206b084e83e74e7ea4e80136de239ff3 backend=gloo
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:37 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:38:37 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:38:43 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:38:43 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:38:43 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:43 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:38:43 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:38:43 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:43 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:43 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:43 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:43 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:43 [selector.py:217] Using HND KV cache layout for CPU_ATTN backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:47 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 27.00 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:47 [weight_utils.py:881] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) ␍Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:02, 1.00it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) ␍Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:02<00:02, 1.08s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) ␍Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:03<00:01, 1.10s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.19s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.14s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:38:59 [default_loader.py:430] Loading weights took 12.90 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:39:02 [memory.py:91] Released 519 MiB after weight load (anonymous RSS 19931 -> 19413 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:39:02 [cpu_worker.py:255] Explicitly set (1.0/30.91) GiB for KV cache on node 0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:39:02 [kv_cache_utils.py:1869] GPU KV cache size: 7,168 tokens, Maximum concurrency for 2,048 tokens per request: 3.50x
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:39:02 [memory.py:91] Released 0 MiB after warm-up (anonymous RSS 20421 -> 20421 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:39:02 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.19 s
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:39:02 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:39:02 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:39:02 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:39:02 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) INFO 09-23 17:39:03 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-23 17:39:03 [stage_runtime.py:943] [StageRuntime] Stage 0 engine startup completed
|
||||
INFO 09-23 17:39:03 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:39:03 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
|
||||
INFO 09-23 17:39:03 [stage_runtime.py:958] [StageRuntime] Stage 0 initialized
|
||||
INFO 09-23 17:39:03 [stage_runtime.py:853] [stage_init] Stage-1 set runtime devices: 0
|
||||
W0923 17:39:08.135000 2302497 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:39:08.162000 2302497 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:39:09 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:39:09 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-23 17:39:10 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:39:10 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:39:10 [patch.py:646] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:39:10 [patch.py:756] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:11 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=512, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
[W923 17:39:12.211547924 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:12 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:12 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:12 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_61858aab472d470cb0cb86048a786e1b backend=gloo
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:12 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:12 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) [transformers] Image processor MiniCPMVImageProcessor: kwargs ['max_slice_nums'] were applied for backward compatibility. To avoid this warning, add them to valid_kwargs: create a custom TypedDict extending ImagesKwargs with these keys and set it as the `valid_kwargs` class attribute.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:20 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:20 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:20 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:20 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:20 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:20 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:20 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:20 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:20 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:20 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:20 [selector.py:217] Using HND KV cache layout for CPU_ATTN backend.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:20 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 7.86 GiB.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:20 [weight_utils.py:888] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (7.86 GiB).
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) ␍Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 3.43it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) ␍Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:02<00:02, 1.39s/it]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) ␍Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:02<00:00, 1.11it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.33it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.22it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:24 [default_loader.py:430] Loading weights took 3.44 seconds
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:24 [memory.py:91] Released 72 MiB after weight load (anonymous RSS 1534 -> 1461 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:24 [cpu_worker.py:255] Explicitly set (1.0/30.91) GiB for KV cache on node 0.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:24 [kv_cache_utils.py:1869] GPU KV cache size: 17,408 tokens, Maximum concurrency for 512 tokens per request: 34.00x
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:25 [memory.py:91] Released 0 MiB after warm-up (anonymous RSS 2484 -> 2484 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:25 [core.py:354] init engine (profile, create kv cache, warmup model) took 1.29 s
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:25 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:25 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) WARNING 09-23 17:39:25 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:25 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302497) INFO 09-23 17:39:26 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-23 17:39:26 [stage_runtime.py:943] [StageRuntime] Stage 1 engine startup completed
|
||||
INFO 09-23 17:39:26 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-1 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:39:26 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-1 [rep-0] EngineCore running
|
||||
INFO 09-23 17:39:26 [stage_runtime.py:958] [StageRuntime] Stage 1 initialized
|
||||
INFO 09-23 17:39:26 [stage_runtime.py:853] [stage_init] Stage-2 set runtime devices: 0
|
||||
W0923 17:39:32.405000 2302594 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:39:32.455000 2302594 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:39:34 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:39:34 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-23 17:39:35 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:39:35 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:39:35 [patch.py:646] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:39:35 [patch.py:756] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:36 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=False, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
[W923 17:39:36.885629043 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) WARNING 09-23 17:39:36 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) WARNING 09-23 17:39:36 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:36 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_ff43f47fca5a47759bce928b8ce6d28e backend=gloo
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:36 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:36 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:36 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:36 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 5.93 GiB.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:36 [weight_utils.py:888] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (5.93 GiB).
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 27.76it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 27.72it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:37 [step_audio2_token2wav.py:163] Loading Token2Wav models from: /home/zhout/project/edge_infer/models/MiniCPM-o-4_5/assets/token2wav
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:45 [step_audio2_token2wav.py:201] Token2Wav models loaded successfully
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:46 [default_loader.py:430] Loading weights took 9.22 seconds
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:46 [memory.py:91] Released 515 MiB after weight load (anonymous RSS 4479 -> 3964 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:46 [memory.py:91] Released 0 MiB after warm-up (anonymous RSS 3964 -> 3964 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:46 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.00 s
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) WARNING 09-23 17:39:46 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:46 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) WARNING 09-23 17:39:47 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) WARNING 09-23 17:39:47 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:47 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302594) INFO 09-23 17:39:47 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
INFO 09-23 17:39:47 [stage_runtime.py:943] [StageRuntime] Stage 2 engine startup completed
|
||||
INFO 09-23 17:39:47 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-2 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:39:47 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-2 [rep-0] EngineCore running
|
||||
INFO 09-23 17:39:47 [stage_runtime.py:958] [StageRuntime] Stage 2 initialized
|
||||
[transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
[transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
[transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
INFO 09-23 17:40:10 [orchestrator.py:604] [Orchestrator] Starting event loop
|
||||
INFO 09-23 17:40:10 [async_omni_engine.py:323] [AsyncOmniEngine] Orchestrator ready with 3 stages
|
||||
INFO 09-23 17:40:10 [omni_base.py:235] [Omni] AsyncOmniEngine initialized in 101.22 seconds
|
||||
INFO 09-23 17:40:10 [omni_base.py:255] [Omni] Initialized with 3 stages for model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
WARNING 09-23 17:40:10 [input_processor.py:296] Passing raw prompts to InputProcessor is deprecated and will be removed in v0.18. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
|
||||
␍Processed prompts: 0%| | 0/1 [00:00<?, ?it/s](StageEngineCoreProc_stage0_replica0 pid=2302392) [transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2302392) WARNING 09-23 17:40:38 [gpu_ar_model_runner.py:1464] prefer_model_sampler model MiniCPMO45OmniForConditionalGeneration returned None from sample(); falling back to the default sampler.
|
||||
␍Processed prompts: 100%|██████████| 1/1 [00:17<00:00, 17.30s/it]␍Processed prompts: 100%|██████████| 1/1 [00:17<00:00, 17.30s/it]
|
||||
query type: use_audio
|
||||
use_tts (tts_bos): True
|
||||
Request ID: 0_40b94cec-371c-47a8-ba61-4729a0eb1285, Text saved to /tmp/minicpmo-wsl-cpu-audio-20260923/0_40b94cec-371c-47a8-ba61-4729a0eb1285.txt
|
||||
Request ID: 0_40b94cec-371c-47a8-ba61-4729a0eb1285, Saved audio to /tmp/minicpmo-wsl-cpu-audio-20260923/output_0_40b94cec-371c-47a8-ba61-4729a0eb1285.wav
|
||||
INFO 09-23 17:40:46 [omni_base.py:781] [Omni] Shutting down
|
||||
INFO 09-23 17:40:46 [async_omni_engine.py:1807] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
INFO 09-23 17:40:46 [orchestrator.py:705] [Orchestrator] Received shutdown signal
|
||||
INFO 09-23 17:40:46 [orchestrator.py:3092] [Orchestrator] Shutting down all 3 client(s)
|
||||
INFO 09-23 17:40:46 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:40:46 [core_client.py:691] [shutdown] MPClient: stopping engine manager
|
||||
INFO 09-23 17:40:46 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
WARNING 09-23 17:40:51 [utils.py:635] [shutdown] Process manager: force killing remaining processes count=1
|
||||
WARNING 09-23 17:40:51 [utils.py:640] [shutdown] Process manager: force killing remaining process StageEngineCoreProc_stage0_replica0 pid 2302392
|
||||
INFO 09-23 17:40:51 [core_client.py:693] [shutdown] MPClient: engine manager stopped
|
||||
INFO 09-23 17:40:51 [core_client.py:694] [shutdown] MPClient: cleaning up background resources
|
||||
INFO 09-23 17:40:51 [core_client.py:696] [shutdown] MPClient: complete
|
||||
INFO 09-23 17:40:51 [stage_pool.py:1354] [StagePool] Stage 0 replica 0 shut down
|
||||
INFO 09-23 17:40:51 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:40:51 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage1_replica0
|
||||
INFO 09-23 17:40:55 [stage_pool.py:1354] [StagePool] Stage 1 replica 0 shut down
|
||||
INFO 09-23 17:40:55 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:40:55 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage2_replica0
|
||||
INFO 09-23 17:40:58 [stage_pool.py:1354] [StagePool] Stage 2 replica 0 shut down
|
||||
/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
|
||||
warnings.warn('resource_tracker: There appear to be %d '
|
||||
/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
|
||||
warnings.warn('resource_tracker: There appear to be %d '
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"scope": "WSL CPU, real MiniCPM-o checkpoint, one synthetic audio input to text+speech request",
|
||||
"text": "A loud and high pitched beep.",
|
||||
"token_count": 9,
|
||||
"text_match_wsl_gpu": false,
|
||||
"token_ids_match_wsl_gpu": false,
|
||||
"wav": {
|
||||
"sample_rate_hz": 24000,
|
||||
"frames": 51840,
|
||||
"duration_s": 2.16,
|
||||
"nonzero_samples": 50375,
|
||||
"rms_float": 0.02436868422594072,
|
||||
"sha256": "e6a6e2b1c09b578936ab8c571410d6b16560651abcc94355479cf0c7e6400fd1"
|
||||
}
|
||||
}
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
# MiniCPM-o 4.5 offline audio-to-text+speech probe on WSL CPU.
|
||||
# This limits context, concurrency and multimodal inputs. Run with
|
||||
# VLLM_TARGET_DEVICE=cpu and VLLM_CPU_KVCACHE_SPACE=1 (GiB per stage).
|
||||
# The CUDA/NPU overrides below are inherited but inactive on this run.
|
||||
pipeline: minicpmo_4_5
|
||||
async_chunk: true
|
||||
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
connector_get_sleep_s: 0.01
|
||||
connector_get_max_wait_first_chunk: 3000
|
||||
connector_get_max_wait: 300
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 3
|
||||
# HiFT capture buckets are derived from codec_chunk_frames and
|
||||
# codec_left_context_frames. (#5869)
|
||||
# A future initial_codec_chunk_frames shape will use lazy capture unless
|
||||
# HiFTGraphWrapper.derive_capture_bucket_size explicitly includes it.
|
||||
enable_hift_graph: false
|
||||
enable_cfm_graph: false
|
||||
cfm_max_graphs: 32
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_seqs: 1
|
||||
# Bound the thinker budget; no weights are offloaded from CPU.
|
||||
gpu_memory_utilization: 0.58
|
||||
cpu_offload_gb: 0
|
||||
enforce_eager: true
|
||||
tensor_parallel_size: 1
|
||||
max_num_batched_tokens: 2048
|
||||
max_model_len: 2048
|
||||
devices: "0"
|
||||
# Keep multimodal placeholders interleaved for the audio input.
|
||||
interleave_mm_strings: true
|
||||
media_io_kwargs:
|
||||
video:
|
||||
fps: 1
|
||||
num_frames: 128
|
||||
limit_mm_per_prompt:
|
||||
# One audio clip is admitted; no image or video inputs.
|
||||
image: 0
|
||||
audio: 1
|
||||
video: 0
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 64
|
||||
seed: 42
|
||||
detokenize: true
|
||||
# 1.0 for MCQ accuracy; 1.1 can bias short A–D answers.
|
||||
repetition_penalty: 1.0
|
||||
|
||||
- stage_id: 1
|
||||
max_num_seqs: 1
|
||||
# One short utterance; this does not qualify the full 4096-token Talker.
|
||||
gpu_memory_utilization: 0.10
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 512
|
||||
# The checkpoint supports 4096, but this probe explicitly admits 512.
|
||||
max_model_len: 512
|
||||
devices: "0"
|
||||
output_connectors:
|
||||
to_stage_2: connector_of_shared_memory
|
||||
# Talker is a single-vocab codec LM. These knobs go to vLLM's Sampler
|
||||
# (head_code logits). EOS is tts_config.eos_token_id (num_audio_tokens-1).
|
||||
# Values track upstream utils.TTSSamplingParams (min_p/win_size/tau_r are
|
||||
# declared there but never read, so they stay out). repetition_penalty is
|
||||
# consumed by the Talker, which rescores it over upstream's 16-frame window
|
||||
# instead of the Sampler's whole-stream presence penalty.
|
||||
# min_tokens is upstream's min_new_token.
|
||||
default_sampling_params:
|
||||
temperature: 0.8
|
||||
top_p: 0.85
|
||||
top_k: 25
|
||||
repetition_penalty: 1.05
|
||||
min_tokens: 50
|
||||
max_tokens: 128
|
||||
seed: 42
|
||||
detokenize: false
|
||||
|
||||
- stage_id: 2
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.10
|
||||
enforce_eager: true
|
||||
enable_chunked_prefill: false
|
||||
max_num_batched_tokens: 4096
|
||||
max_model_len: 4096
|
||||
devices: "0"
|
||||
input_connectors:
|
||||
from_stage_1: connector_of_shared_memory
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 4096
|
||||
detokenize: true
|
||||
|
||||
platforms:
|
||||
cuda:
|
||||
stages:
|
||||
- stage_id: 1
|
||||
# The cap must still admit the single 512-token Talker session.
|
||||
kv_cache_memory_bytes: 268435456 # 256 MiB for the 512-token probe
|
||||
|
||||
npu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_batched_tokens: 8192
|
||||
compilation_config:
|
||||
cudagraph_mode: PIECEWISE
|
||||
- stage_id: 1
|
||||
max_num_batched_tokens: 8192
|
||||
compilation_config:
|
||||
cudagraph_mode: PIECEWISE
|
||||
- stage_id: 2
|
||||
# Stage 2 stays outer-eager and captures only exact-shape CFM DiT
|
||||
# estimator kernels through the NPU platform patch.
|
||||
additional_config:
|
||||
code2wav_enable_npu_graph: true
|
||||
code2wav_max_npu_graphs: 32
|
||||
BIN
Binary file not shown.
+16
@@ -0,0 +1,16 @@
|
||||
Prompt:
|
||||
<|im_start|>system
|
||||
You are MiniCPM-o, a helpful multimodal assistant that can understand images, audio and video, and respond in text and speech.<|im_end|>
|
||||
<|im_start|>user
|
||||
(<image>./</image>)
|
||||
What is the content of this image?<|im_end|>
|
||||
<|im_start|>assistant
|
||||
<think>
|
||||
|
||||
</think>
|
||||
|
||||
<|tts_bos|>
|
||||
vllm_text_output:
|
||||
The image shows a solid red square centered on a white background. There are no other visible elements, text, or details.
|
||||
vllm_token_ids:
|
||||
[785, 2168, 4933, 264, 6437, 2518, 9334, 30188, 389, 264, 4158, 4004, 13, 2619, 525, 902, 1008, 9434, 5424, 11, 1467, 11, 476, 3565, 13, 151704, 151645]
|
||||
+248
@@ -0,0 +1,248 @@
|
||||
W0923 17:34:57.241000 2301854 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:34:57.265000 2301854 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:34:58 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:34:58 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-23 17:34:59 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:34:59 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:34:59 [patch.py:646] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:34:59 [patch.py:756] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
====================
|
||||
vllm version: 0.28.0
|
||||
====================
|
||||
INFO 09-23 17:34:59 [omni_base.py:216] [Omni] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 17:34:59 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 17:34:59 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 3 stages
|
||||
INFO 09-23 17:35:00 [stage_init_utils.py:768] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
INFO 09-23 17:35:00 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations
|
||||
WARNING 09-23 17:35:00 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:35:00 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:35:00 [arg_utils.py:1719] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:35:08 [model.py:672] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:35:08 [model.py:1965] Using max model len 2048
|
||||
WARNING 09-23 17:35:08 [model.py:963] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:35:09 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
WARNING 09-23 17:35:09 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-23 17:35:09 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-23 17:35:09 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
WARNING 09-23 17:35:09 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
INFO 09-23 17:35:09 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-23 17:35:09 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:35:09 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:35:09 [arg_utils.py:1719] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:35:09 [model.py:672] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:35:09 [model.py:1965] Using max model len 512
|
||||
WARNING 09-23 17:35:09 [model.py:963] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:35:09 [model.py:1965] Using max model len 512
|
||||
INFO 09-23 17:35:09 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=512.
|
||||
INFO 09-23 17:35:09 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
WARNING 09-23 17:35:09 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:35:09 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:35:09 [arg_utils.py:1719] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:35:17 [model.py:672] Resolved architecture: MiniCPMO45Code2Wav
|
||||
INFO 09-23 17:35:17 [model.py:1965] Using max model len 4096
|
||||
WARNING 09-23 17:35:17 [arg_utils.py:2664] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs.
|
||||
INFO 09-23 17:35:17 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-23 17:35:17 [stage_runtime.py:853] [stage_init] Stage-0 set runtime devices: 0
|
||||
W0923 17:35:21.189000 2301932 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:35:21.214000 2301932 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:35:22 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:35:22 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-23 17:35:23 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:35:23 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:35:23 [patch.py:646] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:35:23 [patch.py:756] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:23 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
[W923 17:35:24.784797782 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:24 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:24 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:24 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_2d48085d7a9347c8b8326460f316585f backend=gloo
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:24 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:24 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:27 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:27 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:27 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:27 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:27 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:27 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:27 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:27 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:31 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:31 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:31 [selector.py:217] Using HND KV cache layout for CPU_ATTN backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:35 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 25.93 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:35 [weight_utils.py:881] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) ␍Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:00<00:00, 7.24it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) ␍Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:00<00:00, 3.40it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.32it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.76it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:44 [default_loader.py:430] Loading weights took 9.27 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:46 [memory.py:91] Released 560 MiB after weight load (anonymous RSS 21021 -> 20462 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:46 [cpu_worker.py:255] Explicitly set (1.0/30.91) GiB for KV cache on node 0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:46 [kv_cache_utils.py:1869] GPU KV cache size: 7,168 tokens, Maximum concurrency for 2,048 tokens per request: 3.50x
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:47 [memory.py:91] Released -0 MiB after warm-up (anonymous RSS 21470 -> 21470 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:47 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.37 s
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:47 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:47 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:35:47 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:47 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) INFO 09-23 17:35:47 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-23 17:35:47 [stage_runtime.py:943] [StageRuntime] Stage 0 engine startup completed
|
||||
INFO 09-23 17:35:47 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:35:48 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
|
||||
INFO 09-23 17:35:48 [stage_runtime.py:958] [StageRuntime] Stage 0 initialized
|
||||
INFO 09-23 17:35:48 [stage_runtime.py:853] [stage_init] Stage-1 set runtime devices: 0
|
||||
W0923 17:35:53.309000 2302038 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:35:53.340000 2302038 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:35:54 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:35:54 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-23 17:35:56 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:35:56 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:35:56 [patch.py:646] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:35:56 [patch.py:756] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:35:56 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=512, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
[W923 17:35:57.014267214 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:35:57 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:35:57 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:35:57 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_a193d171d2d94114aacfb5dcecee4ed5 backend=gloo
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:35:57 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:35:57 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) [transformers] Image processor MiniCPMVImageProcessor: kwargs ['max_slice_nums'] were applied for backward compatibility. To avoid this warning, add them to valid_kwargs: create a custom TypedDict extending ImagesKwargs with these keys and set it as the `valid_kwargs` class attribute.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:36:06 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:36:06 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:36:06 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:06 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:06 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:36:06 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:36:06 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:06 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:06 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:06 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:06 [selector.py:217] Using HND KV cache layout for CPU_ATTN backend.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:07 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 6.75 GiB.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:07 [weight_utils.py:888] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (6.75 GiB).
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) ␍Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:00, 3.80it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) ␍Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:00<00:00, 3.56it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 4.53it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 4.32it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:09 [default_loader.py:430] Loading weights took 2.36 seconds
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:09 [memory.py:91] Released 74 MiB after weight load (anonymous RSS 1536 -> 1462 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:09 [cpu_worker.py:255] Explicitly set (1.0/30.91) GiB for KV cache on node 0.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:09 [kv_cache_utils.py:1869] GPU KV cache size: 17,408 tokens, Maximum concurrency for 512 tokens per request: 34.00x
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:10 [memory.py:91] Released 0 MiB after warm-up (anonymous RSS 2484 -> 2484 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:10 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.89 s
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:36:10 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:36:10 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) WARNING 09-23 17:36:10 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:10 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2302038) INFO 09-23 17:36:11 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-23 17:36:11 [stage_runtime.py:943] [StageRuntime] Stage 1 engine startup completed
|
||||
INFO 09-23 17:36:11 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-1 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:36:11 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-1 [rep-0] EngineCore running
|
||||
INFO 09-23 17:36:11 [stage_runtime.py:958] [StageRuntime] Stage 1 initialized
|
||||
INFO 09-23 17:36:11 [stage_runtime.py:853] [stage_init] Stage-2 set runtime devices: 0
|
||||
W0923 17:36:16.105000 2302144 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:36:16.133000 2302144 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:36:17 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:36:17 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-23 17:36:18 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:36:18 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:36:18 [patch.py:646] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:36:19 [patch.py:756] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:19 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=False, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
[W923 17:36:19.181532864 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) WARNING 09-23 17:36:19 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) WARNING 09-23 17:36:19 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:19 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_eac150642a1c4fc4a4d408f0589eaeda backend=gloo
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:20 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:20 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:20 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:20 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 5.85 GiB.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:20 [weight_utils.py:888] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (5.85 GiB).
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) ␍Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:00<00:00, 5.96it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) ␍Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:00<00:00, 4.00it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 1.69it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00, 2.14it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:22 [step_audio2_token2wav.py:163] Loading Token2Wav models from: /home/zhout/project/edge_infer/models/MiniCPM-o-4_5/assets/token2wav
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:29 [step_audio2_token2wav.py:201] Token2Wav models loaded successfully
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:29 [default_loader.py:430] Loading weights took 9.06 seconds
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:29 [memory.py:91] Released 512 MiB after weight load (anonymous RSS 4476 -> 3964 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:29 [memory.py:91] Released -0 MiB after warm-up (anonymous RSS 3964 -> 3964 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:29 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.01 s
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) WARNING 09-23 17:36:30 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:30 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) WARNING 09-23 17:36:30 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) WARNING 09-23 17:36:30 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:30 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2302144) INFO 09-23 17:36:30 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
INFO 09-23 17:36:30 [stage_runtime.py:943] [StageRuntime] Stage 2 engine startup completed
|
||||
INFO 09-23 17:36:30 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-2 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:36:30 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-2 [rep-0] EngineCore running
|
||||
INFO 09-23 17:36:30 [stage_runtime.py:958] [StageRuntime] Stage 2 initialized
|
||||
[transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
[transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
[transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
INFO 09-23 17:36:52 [orchestrator.py:604] [Orchestrator] Starting event loop
|
||||
INFO 09-23 17:36:52 [async_omni_engine.py:323] [AsyncOmniEngine] Orchestrator ready with 3 stages
|
||||
INFO 09-23 17:36:52 [omni_base.py:235] [Omni] AsyncOmniEngine initialized in 112.44 seconds
|
||||
INFO 09-23 17:36:52 [omni_base.py:255] [Omni] Initialized with 3 stages for model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
WARNING 09-23 17:36:52 [input_processor.py:296] Passing raw prompts to InputProcessor is deprecated and will be removed in v0.18. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
|
||||
␍Processed prompts: 0%| | 0/1 [00:00<?, ?it/s](StageEngineCoreProc_stage0_replica0 pid=2301932) [transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301932) WARNING 09-23 17:37:17 [gpu_ar_model_runner.py:1464] prefer_model_sampler model MiniCPMO45OmniForConditionalGeneration returned None from sample(); falling back to the default sampler.
|
||||
␍Processed prompts: 100%|██████████| 1/1 [00:37<00:00, 37.70s/it]␍Processed prompts: 100%|██████████| 1/1 [00:37<00:00, 37.70s/it]
|
||||
query type: use_image
|
||||
use_tts (tts_bos): True
|
||||
Request ID: 0_54927839-1ab7-4d0d-86a3-0d6290a84f13, Text saved to /tmp/minicpmo-wsl-cpu-image-20260923/0_54927839-1ab7-4d0d-86a3-0d6290a84f13.txt
|
||||
Request ID: 0_54927839-1ab7-4d0d-86a3-0d6290a84f13, Saved audio to /tmp/minicpmo-wsl-cpu-image-20260923/output_0_54927839-1ab7-4d0d-86a3-0d6290a84f13.wav
|
||||
INFO 09-23 17:37:39 [omni_base.py:781] [Omni] Shutting down
|
||||
INFO 09-23 17:37:39 [async_omni_engine.py:1807] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
INFO 09-23 17:37:39 [orchestrator.py:705] [Orchestrator] Received shutdown signal
|
||||
INFO 09-23 17:37:39 [orchestrator.py:3092] [Orchestrator] Shutting down all 3 client(s)
|
||||
INFO 09-23 17:37:39 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:37:39 [core_client.py:691] [shutdown] MPClient: stopping engine manager
|
||||
INFO 09-23 17:37:39 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
WARNING 09-23 17:37:44 [utils.py:635] [shutdown] Process manager: force killing remaining processes count=1
|
||||
WARNING 09-23 17:37:44 [utils.py:640] [shutdown] Process manager: force killing remaining process StageEngineCoreProc_stage0_replica0 pid 2301932
|
||||
INFO 09-23 17:37:44 [core_client.py:693] [shutdown] MPClient: engine manager stopped
|
||||
INFO 09-23 17:37:44 [core_client.py:694] [shutdown] MPClient: cleaning up background resources
|
||||
INFO 09-23 17:37:44 [core_client.py:696] [shutdown] MPClient: complete
|
||||
INFO 09-23 17:37:44 [stage_pool.py:1354] [StagePool] Stage 0 replica 0 shut down
|
||||
INFO 09-23 17:37:44 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:37:44 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage1_replica0
|
||||
INFO 09-23 17:37:48 [stage_pool.py:1354] [StagePool] Stage 1 replica 0 shut down
|
||||
INFO 09-23 17:37:48 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:37:48 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage2_replica0
|
||||
INFO 09-23 17:37:51 [stage_pool.py:1354] [StagePool] Stage 2 replica 0 shut down
|
||||
/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
|
||||
warnings.warn('resource_tracker: There appear to be %d '
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"scope": "WSL CPU, real MiniCPM-o checkpoint, one synthetic image input to text+speech request",
|
||||
"text": "The image shows a solid red square centered on a white background. There are no other visible elements, text, or details.",
|
||||
"token_count": 27,
|
||||
"text_match_wsl_gpu": false,
|
||||
"token_ids_match_wsl_gpu": false,
|
||||
"wav": {
|
||||
"sample_rate_hz": 24000,
|
||||
"frames": 124800,
|
||||
"duration_s": 5.2,
|
||||
"nonzero_samples": 123513,
|
||||
"rms_float": 0.09555768475943359,
|
||||
"sha256": "d1f5890a6aafa3c2dcea0ea4ae34e500d03ada1369ce61a8a005e9a35a5c3805"
|
||||
}
|
||||
}
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
# MiniCPM-o 4.5 offline image-to-text+speech probe on WSL CPU.
|
||||
# This limits context, concurrency and multimodal inputs. Run with
|
||||
# VLLM_TARGET_DEVICE=cpu and VLLM_CPU_KVCACHE_SPACE=1 (GiB per stage).
|
||||
# The CUDA/NPU overrides below are inherited but inactive on this run.
|
||||
pipeline: minicpmo_4_5
|
||||
async_chunk: true
|
||||
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
connector_get_sleep_s: 0.01
|
||||
connector_get_max_wait_first_chunk: 3000
|
||||
connector_get_max_wait: 300
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 3
|
||||
# HiFT capture buckets are derived from codec_chunk_frames and
|
||||
# codec_left_context_frames. (#5869)
|
||||
# A future initial_codec_chunk_frames shape will use lazy capture unless
|
||||
# HiFTGraphWrapper.derive_capture_bucket_size explicitly includes it.
|
||||
enable_hift_graph: false
|
||||
enable_cfm_graph: false
|
||||
cfm_max_graphs: 32
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_seqs: 1
|
||||
# Bound the thinker budget; no weights are offloaded from CPU.
|
||||
gpu_memory_utilization: 0.58
|
||||
cpu_offload_gb: 0
|
||||
enforce_eager: true
|
||||
tensor_parallel_size: 1
|
||||
max_num_batched_tokens: 2048
|
||||
max_model_len: 2048
|
||||
devices: "0"
|
||||
# Keep multimodal placeholders interleaved for the image input.
|
||||
interleave_mm_strings: true
|
||||
media_io_kwargs:
|
||||
video:
|
||||
fps: 1
|
||||
num_frames: 128
|
||||
limit_mm_per_prompt:
|
||||
# One image is admitted; no audio or video inputs.
|
||||
image: 1
|
||||
audio: 0
|
||||
video: 0
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 64
|
||||
seed: 42
|
||||
detokenize: true
|
||||
# 1.0 for MCQ accuracy; 1.1 can bias short A–D answers.
|
||||
repetition_penalty: 1.0
|
||||
|
||||
- stage_id: 1
|
||||
max_num_seqs: 1
|
||||
# One short utterance; this does not qualify the full 4096-token Talker.
|
||||
gpu_memory_utilization: 0.10
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 512
|
||||
# The checkpoint supports 4096, but this probe explicitly admits 512.
|
||||
max_model_len: 512
|
||||
devices: "0"
|
||||
output_connectors:
|
||||
to_stage_2: connector_of_shared_memory
|
||||
# Talker is a single-vocab codec LM. These knobs go to vLLM's Sampler
|
||||
# (head_code logits). EOS is tts_config.eos_token_id (num_audio_tokens-1).
|
||||
# Values track upstream utils.TTSSamplingParams (min_p/win_size/tau_r are
|
||||
# declared there but never read, so they stay out). repetition_penalty is
|
||||
# consumed by the Talker, which rescores it over upstream's 16-frame window
|
||||
# instead of the Sampler's whole-stream presence penalty.
|
||||
# min_tokens is upstream's min_new_token.
|
||||
default_sampling_params:
|
||||
temperature: 0.8
|
||||
top_p: 0.85
|
||||
top_k: 25
|
||||
repetition_penalty: 1.05
|
||||
min_tokens: 50
|
||||
max_tokens: 128
|
||||
seed: 42
|
||||
detokenize: false
|
||||
|
||||
- stage_id: 2
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.10
|
||||
enforce_eager: true
|
||||
enable_chunked_prefill: false
|
||||
max_num_batched_tokens: 4096
|
||||
max_model_len: 4096
|
||||
devices: "0"
|
||||
input_connectors:
|
||||
from_stage_1: connector_of_shared_memory
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 4096
|
||||
detokenize: true
|
||||
|
||||
platforms:
|
||||
cuda:
|
||||
stages:
|
||||
- stage_id: 1
|
||||
# The cap must still admit the single 512-token Talker session.
|
||||
kv_cache_memory_bytes: 268435456 # 256 MiB for the 512-token probe
|
||||
|
||||
npu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_batched_tokens: 8192
|
||||
compilation_config:
|
||||
cudagraph_mode: PIECEWISE
|
||||
- stage_id: 1
|
||||
max_num_batched_tokens: 8192
|
||||
compilation_config:
|
||||
cudagraph_mode: PIECEWISE
|
||||
- stage_id: 2
|
||||
# Stage 2 stays outer-eager and captures only exact-shape CFM DiT
|
||||
# estimator kernels through the NPU platform patch.
|
||||
additional_config:
|
||||
code2wav_enable_npu_graph: true
|
||||
code2wav_max_npu_graphs: 32
|
||||
+555
@@ -0,0 +1,555 @@
|
||||
W0923 17:31:25.620000 2301160 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:31:25.660000 2301160 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:31:27 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:31:27 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
INFO 09-23 17:31:29 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:31:29 [patch.py:503] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:31:29 [patch.py:633] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:31:29 [patch.py:743] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
====================
|
||||
vllm version: 0.28.0
|
||||
====================
|
||||
INFO 09-23 17:31:29 [omni_base.py:216] [Omni] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 17:31:29 [async_omni_engine.py:161] [AsyncOmniEngine] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 17:31:29 [async_omni_engine.py:286] [AsyncOmniEngine] Launching Orchestrator thread with 3 stages
|
||||
INFO 09-23 17:31:29 [stage_init_utils.py:765] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
INFO 09-23 17:31:29 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations
|
||||
WARNING 09-23 17:31:29 [utils.py:893] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:31:29 [utils.py:893] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:31:30 [arg_utils.py:1719] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:31:39 [model.py:672] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:31:39 [model.py:1965] Using max model len 2048
|
||||
WARNING 09-23 17:31:39 [model.py:963] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:31:39 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
WARNING 09-23 17:31:39 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-23 17:31:39 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-23 17:31:39 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
WARNING 09-23 17:31:39 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
INFO 09-23 17:31:39 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-23 17:31:39 [utils.py:893] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:31:39 [utils.py:893] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:31:39 [arg_utils.py:1719] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:31:39 [model.py:672] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 17:31:39 [model.py:1965] Using max model len 512
|
||||
WARNING 09-23 17:31:39 [model.py:963] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 17:31:39 [model.py:1965] Using max model len 512
|
||||
INFO 09-23 17:31:39 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=512.
|
||||
INFO 09-23 17:31:39 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
WARNING 09-23 17:31:39 [utils.py:893] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 17:31:39 [utils.py:893] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 17:31:39 [arg_utils.py:1719] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 17:31:48 [model.py:672] Resolved architecture: MiniCPMO45Code2Wav
|
||||
INFO 09-23 17:31:48 [model.py:1965] Using max model len 4096
|
||||
WARNING 09-23 17:31:48 [arg_utils.py:2664] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs.
|
||||
INFO 09-23 17:31:48 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-23 17:31:48 [stage_runtime.py:771] [stage_init] Stage-0 set runtime devices: 0
|
||||
W0923 17:31:51.896000 2301241 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:31:51.921000 2301241 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:31:53 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:31:53 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
INFO 09-23 17:31:53 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:31:54 [patch.py:503] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:31:54 [patch.py:633] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:31:54 [patch.py:743] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:31:54 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
[W923 17:31:55.592331371 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:31:55 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:31:55 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:31:55 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_1280a2b851d74b6494f5a36dd3bff3ba backend=gloo
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:31:55 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:31:55 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:31:58 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:31:58 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:31:58 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:31:58 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:31:58 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:31:58 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:31:58 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:31:58 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:01 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:01 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:01 [selector.py:217] Using HND KV cache layout for CPU_ATTN backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:06 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 25.88 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:06 [weight_utils.py:881] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) ␍Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:01<00:03, 1.02s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) ␍Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:02<00:02, 1.08s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) ␍Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:03<00:01, 1.11s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.22s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.17s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:20 [default_loader.py:430] Loading weights took 14.07 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:22 [memory.py:91] Released 563 MiB after weight load (anonymous RSS 21024 -> 20461 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:22 [cpu_worker.py:255] Explicitly set (1.0/30.91) GiB for KV cache on node 0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:22 [kv_cache_utils.py:1869] GPU KV cache size: 7,168 tokens, Maximum concurrency for 2,048 tokens per request: 3.50x
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:22 [memory.py:91] Released 0 MiB after warm-up (anonymous RSS 21469 -> 21469 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:22 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.23 s
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:32:22 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:32:22 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) WARNING 09-23 17:32:23 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:23 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage0_replica0 pid=2301241) INFO 09-23 17:32:23 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-23 17:32:23 [stage_runtime.py:871] [StageRuntime] Stage 0 engine startup completed
|
||||
INFO 09-23 17:32:23 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:32:23 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
|
||||
INFO 09-23 17:32:23 [stage_runtime.py:886] [StageRuntime] Stage 0 initialized
|
||||
INFO 09-23 17:32:23 [stage_runtime.py:771] [stage_init] Stage-1 set runtime devices: 0
|
||||
W0923 17:32:29.931000 2301347 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:32:29.964000 2301347 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:32:31 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:32:31 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
INFO 09-23 17:32:32 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:32:32 [patch.py:503] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:32:32 [patch.py:633] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:32:32 [patch.py:743] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:33 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=512, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
[W923 17:32:34.452742478 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:34 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:34 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:34 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_278a082a38b548ad9e3b89db9b034b71 backend=gloo
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:34 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:34 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) [transformers] Image processor MiniCPMVImageProcessor: kwargs ['max_slice_nums'] were applied for backward compatibility. To avoid this warning, add them to valid_kwargs: create a custom TypedDict extending ImagesKwargs with these keys and set it as the `valid_kwargs` class attribute.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:42 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:42 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:43 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:43 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:43 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:43 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:43 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:43 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:43 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:43 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:43 [selector.py:217] Using HND KV cache layout for CPU_ATTN backend.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:43 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 6.68 GiB.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:43 [weight_utils.py:888] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (6.68 GiB).
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) ␍Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 2.09it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) ␍Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.62it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) ␍Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:01<00:00, 2.11it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00, 1.78it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00, 1.83it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:46 [default_loader.py:430] Loading weights took 2.77 seconds
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:46 [memory.py:91] Released 70 MiB after weight load (anonymous RSS 1527 -> 1457 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:46 [cpu_worker.py:255] Explicitly set (1.0/30.91) GiB for KV cache on node 0.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:46 [kv_cache_utils.py:1869] GPU KV cache size: 17,408 tokens, Maximum concurrency for 512 tokens per request: 34.00x
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:47 [memory.py:91] Released 0 MiB after warm-up (anonymous RSS 2479 -> 2479 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:47 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.40 s
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:47 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:47 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) WARNING 09-23 17:32:47 [cpu_worker.py:153] CPU backend doesn't allow to use `torch.set_num_threads` after the thread binding, skip it.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:47 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=2301347) INFO 09-23 17:32:47 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-23 17:32:47 [stage_runtime.py:871] [StageRuntime] Stage 1 engine startup completed
|
||||
INFO 09-23 17:32:47 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-1 [rep-0] initializing EngineCore
|
||||
INFO 09-23 17:32:47 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-1 [rep-0] EngineCore running
|
||||
INFO 09-23 17:32:47 [stage_runtime.py:886] [StageRuntime] Stage 1 initialized
|
||||
INFO 09-23 17:32:47 [stage_runtime.py:771] [stage_init] Stage-2 set runtime devices: 0
|
||||
W0923 17:32:53.769000 2301454 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0923 17:32:53.808000 2301454 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
WARNING 09-23 17:32:55 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-23 17:32:55 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
INFO 09-23 17:32:56 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 17:32:56 [patch.py:503] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 17:32:56 [patch.py:633] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-23 17:32:56 [patch.py:743] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) INFO 09-23 17:32:57 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=False, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
[W923 17:32:57.791689740 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) WARNING 09-23 17:32:57 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) WARNING 09-23 17:32:57 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) INFO 09-23 17:32:57 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_596872673f5a47ea97847f3bf2204f58 backend=gloo
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) INFO 09-23 17:32:57 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) INFO 09-23 17:32:57 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) INFO 09-23 17:32:57 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) INFO 09-23 17:32:57 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 4.76 GiB.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) INFO 09-23 17:32:57 [weight_utils.py:888] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (4.76 GiB).
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ␍Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:01, 1.82it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ␍Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.05it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ␍Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:03<00:01, 1.11s/it]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.21it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00, 1.15it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) INFO 09-23 17:33:01 [step_audio2_token2wav.py:163] Loading Token2Wav models from: /home/zhout/project/edge_infer/models/MiniCPM-o-4_5/assets/token2wav
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] StageEngineCoreProc failed to start.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] Traceback (most recent call last):
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_engine_core_proc.py", line 189, in run_stage_core
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] engine_core = StageEngineCoreProc(
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1073, in __init__
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] super().__init__(
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 133, in __init__
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] self.model_executor = executor_class(vllm_config)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 110, in __init__
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] self._init_executor()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 74, in _init_executor
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] self.driver_worker.load_model()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/patch.py", line 731, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] out = base_load(self, *args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 457, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/init_timeline.py", line 343, in wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return fn(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/worker/gpu_model_runner.py", line 186, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] super().load_model(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/worker/cpu_model_runner.py", line 132, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] self.model = get_model(vllm_config=self.vllm_config)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/model_executor/model_loader/__init__.py", line 137, in get_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return loader.load_model(
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] self.load_weights(model, model_config)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py", line 823, in load_weights
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] self._build_backend()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py", line 864, in _build_backend
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] token2wav = Token2wav(
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_token2wav.py", line 79, in __init__
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] self._core._ensure_models_loaded()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/model_executor/models/step_audio2/step_audio2_token2wav.py", line 177, in _ensure_models_loaded
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] configs = load_hyperpyyaml(f)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/hyperpyyaml/core.py", line 189, in load_hyperpyyaml
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] hparams = yaml.load(yaml_stream, Loader=loader)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/yaml/__init__.py", line 81, in load
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return loader.get_single_data()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 119, in get_single_data
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return self.construct_document(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 123, in construct_document
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] data = self.construct_object(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 150, in construct_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] data = self.construct_non_recursive_object(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 191, in construct_non_recursive_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] for _dummy in generator:
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 629, in construct_yaml_map
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] value = self.construct_mapping(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 425, in construct_mapping
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] return BaseConstructor.construct_mapping(self, node, deep=deep)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 247, in construct_mapping
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] value = self.construct_object(value_node, deep=deep)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 150, in construct_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] data = self.construct_non_recursive_object(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 186, in construct_non_recursive_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] data = constructor(self, tag_suffix, node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/hyperpyyaml/core.py", line 473, in _construct_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] raise ImportError("There is no such class as %s" % callable_string)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ERROR 09-23 17:33:03 [stage_engine_core_proc.py:242] ImportError: There is no such class as cosyvoice2.flow.flow.CausalMaskedDiffWithXvec
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) Process StageEngineCoreProc_stage2_replica0:
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) Traceback (most recent call last):
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self.run()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self._target(*self._args, **self._kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_engine_core_proc.py", line 189, in run_stage_core
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) engine_core = StageEngineCoreProc(
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1073, in __init__
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) super().__init__(
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 133, in __init__
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self.model_executor = executor_class(vllm_config)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 110, in __init__
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self._init_executor()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 74, in _init_executor
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self.driver_worker.load_model()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/patch.py", line 731, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) out = base_load(self, *args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 457, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/init_timeline.py", line 343, in wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return fn(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/worker/gpu_model_runner.py", line 186, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) super().load_model(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/worker/cpu_model_runner.py", line 132, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self.model = get_model(vllm_config=self.vllm_config)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/model_executor/model_loader/__init__.py", line 137, in get_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return loader.load_model(
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self.load_weights(model, model_config)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py", line 823, in load_weights
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self._build_backend()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_code2wav.py", line 864, in _build_backend
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) token2wav = Token2wav(
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_token2wav.py", line 79, in __init__
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) self._core._ensure_models_loaded()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/model_executor/models/step_audio2/step_audio2_token2wav.py", line 177, in _ensure_models_loaded
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) configs = load_hyperpyyaml(f)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/hyperpyyaml/core.py", line 189, in load_hyperpyyaml
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) hparams = yaml.load(yaml_stream, Loader=loader)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/yaml/__init__.py", line 81, in load
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return loader.get_single_data()
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 119, in get_single_data
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return self.construct_document(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 123, in construct_document
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) data = self.construct_object(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 150, in construct_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) data = self.construct_non_recursive_object(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 191, in construct_non_recursive_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) for _dummy in generator:
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 629, in construct_yaml_map
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) value = self.construct_mapping(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 425, in construct_mapping
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) return BaseConstructor.construct_mapping(self, node, deep=deep)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 247, in construct_mapping
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) value = self.construct_object(value_node, deep=deep)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 150, in construct_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) data = self.construct_non_recursive_object(node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 186, in construct_non_recursive_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) data = constructor(self, tag_suffix, node)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/hyperpyyaml/core.py", line 473, in _construct_object
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) raise ImportError("There is no such class as %s" % callable_string)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=2301454) ImportError: There is no such class as cosyvoice2.flow.flow.CausalMaskedDiffWithXvec
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] [StageRuntime] Stage initialization failed; shutting down 2 initialized client(s)
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] Traceback (most recent call last):
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 246, in initialize
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] initialized_clients = self._initialize_stage_replicas(stage_plans, self._stage_init_timeout)
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 520, in _initialize_stage_replicas
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] raise primary_exc
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 481, in _init_group
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] client = self._initialize_replica(
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] ^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 746, in _initialize_replica
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] return self._initialize_local_llm_replica(plan, stage_init_timeout)
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 860, in _initialize_local_llm_replica
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] with launch_cm as resources:
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] ^^^^^^^^^
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] next(self.gen)
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_engine_startup.py", line 1211, in launch_stage_replica
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] wait_for_engine_startup(
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1286, in wait_for_engine_startup
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] raise RuntimeError(
|
||||
ERROR 09-23 17:33:05 [stage_runtime.py:259] RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
INFO 09-23 17:33:05 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:33:05 [core_client.py:691] [shutdown] MPClient: stopping engine manager
|
||||
INFO 09-23 17:33:05 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage1_replica0
|
||||
INFO 09-23 17:33:08 [core_client.py:693] [shutdown] MPClient: engine manager stopped
|
||||
INFO 09-23 17:33:08 [core_client.py:694] [shutdown] MPClient: cleaning up background resources
|
||||
INFO 09-23 17:33:08 [core_client.py:696] [shutdown] MPClient: complete
|
||||
INFO 09-23 17:33:08 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:33:08 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
WARNING 09-23 17:33:13 [utils.py:635] [shutdown] Process manager: force killing remaining processes count=1
|
||||
WARNING 09-23 17:33:13 [utils.py:640] [shutdown] Process manager: force killing remaining process StageEngineCoreProc_stage0_replica0 pid 2301241
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] [AsyncOmniEngine] Orchestrator thread crashed
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] Traceback (most recent call last):
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 463, in _bootstrap_orchestrator
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] loop.run_until_complete(_run_orchestrator())
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] return future.result()
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] ^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 420, in _run_orchestrator
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] self._initialize_stages(stage_init_timeout)
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 346, in _initialize_stages
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] self._initialize_stages_impl(stage_init_timeout)
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 367, in _initialize_stages_impl
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] self._runtime.initialize()
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 265, in initialize
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] raise exc
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 246, in initialize
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] initialized_clients = self._initialize_stage_replicas(stage_plans, self._stage_init_timeout)
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 520, in _initialize_stage_replicas
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] raise primary_exc
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 481, in _init_group
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] client = self._initialize_replica(
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] ^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 746, in _initialize_replica
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] return self._initialize_local_llm_replica(plan, stage_init_timeout)
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 860, in _initialize_local_llm_replica
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] with launch_cm as resources:
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] ^^^^^^^^^
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] next(self.gen)
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_engine_startup.py", line 1211, in launch_stage_replica
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] wait_for_engine_startup(
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1286, in wait_for_engine_startup
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] raise RuntimeError(
|
||||
ERROR 09-23 17:33:13 [async_omni_engine.py:469] RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
INFO 09-23 17:33:13 [async_omni_engine.py:2125] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
Exception in thread orchestrator:
|
||||
Traceback (most recent call last):
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
|
||||
self.run()
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
|
||||
self._target(*self._args, **self._kwargs)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 463, in _bootstrap_orchestrator
|
||||
loop.run_until_complete(_run_orchestrator())
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 420, in _run_orchestrator
|
||||
self._initialize_stages(stage_init_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 346, in _initialize_stages
|
||||
self._initialize_stages_impl(stage_init_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 367, in _initialize_stages_impl
|
||||
self._runtime.initialize()
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 265, in initialize
|
||||
raise exc
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 246, in initialize
|
||||
initialized_clients = self._initialize_stage_replicas(stage_plans, self._stage_init_timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 520, in _initialize_stage_replicas
|
||||
raise primary_exc
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 481, in _init_group
|
||||
client = self._initialize_replica(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 746, in _initialize_replica
|
||||
return self._initialize_local_llm_replica(plan, stage_init_timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 860, in _initialize_local_llm_replica
|
||||
with launch_cm as resources:
|
||||
^^^^^^^^^
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
|
||||
next(self.gen)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_engine_startup.py", line 1211, in launch_stage_replica
|
||||
wait_for_engine_startup(
|
||||
File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1286, in wait_for_engine_startup
|
||||
raise RuntimeError(
|
||||
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
Traceback (most recent call last):
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
|
||||
self.run()
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
|
||||
self._target(*self._args, **self._kwargs)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 463, in _bootstrap_orchestrator
|
||||
loop.run_until_complete(_run_orchestrator())
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 420, in _run_orchestrator
|
||||
self._initialize_stages(stage_init_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 346, in _initialize_stages
|
||||
self._initialize_stages_impl(stage_init_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 367, in _initialize_stages_impl
|
||||
self._runtime.initialize()
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 265, in initialize
|
||||
raise exc
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 246, in initialize
|
||||
initialized_clients = self._initialize_stage_replicas(stage_plans, self._stage_init_timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 520, in _initialize_stage_replicas
|
||||
raise primary_exc
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 481, in _init_group
|
||||
client = self._initialize_replica(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 746, in _initialize_replica
|
||||
return self._initialize_local_llm_replica(plan, stage_init_timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_runtime.py", line 860, in _initialize_local_llm_replica
|
||||
with launch_cm as resources:
|
||||
^^^^^^^^^
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
|
||||
next(self.gen)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/stage_engine_startup.py", line 1211, in launch_stage_replica
|
||||
wait_for_engine_startup(
|
||||
File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1286, in wait_for_engine_startup
|
||||
raise RuntimeError(
|
||||
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
|
||||
The above exception was the direct cause of the following exception:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/examples/offline_inference/minicpmo/end2end.py", line 519, in <module>
|
||||
main(args)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/examples/offline_inference/minicpmo/end2end.py", line 283, in main
|
||||
omni = Omni(**omni_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/entrypoints/omni_base.py", line 223, in __init__
|
||||
self.engine = AsyncOmniEngine(
|
||||
^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 301, in __init__
|
||||
self._wait_for_orchestrator_init(startup_future, startup_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 534, in _wait_for_orchestrator_init
|
||||
startup_future.result(
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 456, in result
|
||||
return self.__get_result()
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||||
raise self._exception
|
||||
RuntimeError: Orchestrator initialization failed: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
|
||||
warnings.warn('resource_tracker: There appear to be %d '
|
||||
BIN
Binary file not shown.
@@ -8,7 +8,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i
|
||||
|
||||
| Device | Spark-X2.5 | Qwen3-TTS 0.6B CustomVoice | Qwen3.8-27B | MiniCPM-o 4.5 | InternVLA-A1 |
|
||||
|---|---|---|---|---|---|
|
||||
| PC HX370: CPU / WSL | profile completed | scoped E2E: short stream | NOT E2E | scoped E2E: text → speech | synthetic policy pass |
|
||||
| PC HX370: CPU / WSL | profile completed | scoped E2E: short stream | NOT E2E | scoped E2E: text/image/audio → speech | synthetic policy pass |
|
||||
| PC HX370: CPU / native Windows | NOT E2E | NOT E2E | NOT E2E | NOT E2E | synthetic policy pass |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | profile completed | profile completed | scoped E2E: text + image | scoped E2E: text/image/audio → speech | synthetic policy pass |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | profile completed | profile completed | NOT E2E | scoped E2E: text/image/audio → speech | synthetic policy pass |
|
||||
@@ -28,7 +28,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i
|
||||
| PC HX370: CPU / WSL | Spark-X2.5 | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Qualify a matching vLLM 0.29 CPU build and token parity separately. |
|
||||
| PC HX370: CPU / WSL | Qwen3-TTS 0.6B CustomVoice | SCOPED_E2E | The 2026-09-23 compatibility fix enabled real-weight two-frame PCM and four short-stream requests, including cancellation recovery. One measured request generated 3.04 s of audio in 17.02 s (RTF 5.60) with four simulated playback underruns. The CPU wheel remains vLLM 0.28 versus Omni 0.29. | Qualify a matching CPU runtime, speech quality, uninterrupted playback and sustained load; [recovery evidence](../../../e2e_recovery_20260923/README.md). |
|
||||
| PC HX370: CPU / WSL | Qwen3.8-27B | NOT_E2E_PROFILEABLE | No compatible CPU artifact/deployment for this model was found in the checked model roots. Available Qwen 27B FP8/NVFP4 CUDA artifacts do not establish a CPU route. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. |
|
||||
| PC HX370: CPU / WSL | MiniCPM-o 4.5 | SCOPED_E2E | The verified checkpoint completed real three-stage text-to-text+speech requests on the vLLM 0.28 CPU backend: coherent text and nonzero 5.2 s WAVs. A separate 1-warmup/20-measured serial run had request wall p50 21.86 s/p95 22.97 s, with one consistent thinker token sequence. WSL swap reached 5.75 GB; speech meaning was not checked. | Validate speech intelligibility, other input modalities, concurrency, interruption, admission/loading peak and sustained thermal behavior before broader CPU qualification; [CPU evidence](../../../e2e_expansion_20260923/README.md). |
|
||||
| PC HX370: CPU / WSL | MiniCPM-o 4.5 | SCOPED_E2E | The verified checkpoint completed real three-stage text, one synthetic image and one synthetic audio input to text+speech on the vLLM 0.28 CPU backend in separate bounded plans. The 448×448 red square was described correctly with a nonzero 5.2 s WAV in one 37.70 s request; the 2 s/440 Hz tone was described as “A loud and high pitched beep” with a nonzero 2.16 s WAV in one 17.30 s request. CPU wording/token IDs differ from the WSL GPU runs, with no task-quality threshold established. A separate 1-warmup/20-measured serial **text** run had request wall p50 21.86 s/p95 22.97 s. WSL swap reached 5.75 GB during that text profile; manual snapshots in the multimodal runs showed heavy swap use but do not prove peak memory. Both multimodal runs force-killed stage 0 at shutdown and reported shared-memory cleanup warnings. | Validate real image/audio suites, speech intelligibility, video input, concurrency, interruption, admission/loading peak, clean shutdown and sustained thermal behavior before broader CPU qualification; [CPU evidence](../../../e2e_expansion_20260923/README.md). |
|
||||
| PC HX370: CPU / WSL | InternVLA-A1 | SYNTHETIC_POLICY_PASS | The Place_Markpen checkpoint strict-loaded in CPU-only PyTorch and produced finite `[1, 50, 32]` actions from zero observations/noise in one 3.25 s forward. Compared with the same CUDA checkpoint/input, relative L2 error was 1.30% and cosine similarity 0.999917; no task-quality tolerance is established. | Validate real observations against reference actions, units/timestamps, repeated latency and memory before robot-task qualification; [CPU evidence](../../../e2e_expansion_20260923/README.md). |
|
||||
| PC HX370: CPU / native Windows | Spark-X2.5 | NOT_E2E_PROFILEABLE | The installed native Windows vLLM extension lacks cpu_attn_reshape_and_cache and cpu_attn_get_scheduler_metadata. The separate ORT CPU graph worker is not a whole-model text/TTS adapter; a qualified CPU build/binding is missing. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. |
|
||||
| PC HX370: CPU / native Windows | Qwen3-TTS 0.6B CustomVoice | NOT_E2E_PROFILEABLE | The installed native Windows vLLM extension lacks cpu_attn_reshape_and_cache and cpu_attn_get_scheduler_metadata. The separate ORT CPU graph worker is not a whole-model text/TTS adapter; a qualified CPU build/binding is missing. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. |
|
||||
|
||||
@@ -39,6 +39,13 @@ the same respective text and thinker token IDs as WSL with nonzero WAVs.
|
||||
Broader image/audio quality, video, repeated multimodal latency and clean
|
||||
lifecycle remain open.
|
||||
|
||||
WSL CPU MiniCPM-o now also completes one synthetic red-square image and one
|
||||
440 Hz tone through thinker, talker and vocoder under separate one-item plans.
|
||||
Both return plausible text and nonzero WAVs, but CPU wording/token IDs differ
|
||||
from GPU, swap is heavily used, and stage 0 requires forced shutdown with
|
||||
shared-memory cleanup warnings. These single requests do not inherit the
|
||||
20-request text latency profile or establish real image/audio quality.
|
||||
|
||||
## Architecture and prerequisites
|
||||
|
||||
Keep Omni's PipelineConfig, StageRuntime, StageClient, orchestration and admission
|
||||
@@ -94,7 +101,7 @@ capability and lifecycle implementation; a state-handle type alone is insufficie
|
||||
| M1 | S25 Spark | Device-local Omni-compatible controller; embedding, prefill, continuous decode, ring/full KV, head and sampler; resident state | Reference checks, at least 128 output tokens, 512-window/1024-bucket transitions, cancellation, resident memory and sustained profile |
|
||||
| M2 | Qwen3-TTS | Resolve desktop playback and shutdown findings and CPU compatibility; then mobile talker/predictor plus GPU vocoder | Complete PCM and tail, history/chunk correctness, interruption/recovery, quality checks, no post-startup underruns and RTF below 1 in the declared workload |
|
||||
| M3 | AMD stages | Connect compatible real encoders or other useful coarse stages through current Omni workers; start from the existing 890M vision evidence | Actual node/device placement, numerical and task quality, complete downstream output, shared-memory and handoff costs; reject unhelpful splits |
|
||||
| M4a | MiniCPM-o 4.5 | Three-stage desktop text+WAV produced coherent text and nonzero WAV on RTX 5090 Laptop under WSL and native Windows, and WSL CPU, using separate constrained plans. One synthetic red-square image and one synthetic tone each passed through the three-stage WSL RTX and native Windows RTX paths to text and nonzero WAV; matching inputs produced identical text and thinker token IDs across the two OS environments. The WSL image plan needed a thinker GPU budget increase from 0.58 to 0.61 for KV admission. WSL multimodal shutdown cleanup remains open. The CPU plan needed vLLM 0.28/0.29 processor compatibility and used swap; 20 serial CPU requests measured p50 21.86 s/p95 22.97 s. Native Windows needed longer startup limits; 20 serial text requests measured p50 18.38 s/p95 18.73 s with host RAM/pagefile pressure. Add video encoder and persistent-state reference before mobile artifacts | Speech intelligibility/alignment, loading-peak admission, concurrency, broad image/audio-understanding suites, video, shutdown lifecycle, then combined streaming and interruption |
|
||||
| M4a | MiniCPM-o 4.5 | Three-stage desktop text+WAV produced coherent text and nonzero WAV on RTX 5090 Laptop under WSL and native Windows, and WSL CPU, using separate constrained plans. One synthetic red-square image and one synthetic tone each passed through the three-stage WSL RTX, native Windows RTX and WSL CPU paths to text and nonzero WAV. RTX text/token IDs matched across OS environments; CPU wording/token IDs differed while remaining plausible. The WSL RTX image plan needed a thinker GPU budget increase from 0.58 to 0.61 for KV admission. WSL multimodal shutdown cleanup remains open, including forced stage-0 termination on CPU. The CPU plan needed vLLM 0.28/0.29 processor compatibility and used swap; 20 serial **text** CPU requests measured p50 21.86 s/p95 22.97 s. Native Windows needed longer startup limits; 20 serial text requests measured p50 18.38 s/p95 18.73 s with host RAM/pagefile pressure. Add video encoder and persistent-state reference before mobile artifacts | Speech intelligibility/alignment, loading-peak admission, concurrency, broad image/audio-understanding suites, video, shutdown lifecycle, then combined streaming and interruption |
|
||||
| M4b | InternVLA-A1 | Real weights, observation preprocessing, shared prefix state, iterative action head and versioned action metadata; strict-load synthetic policy forwards now pass on WSL and native Windows, each on CPU and RTX 5090 Laptop | Reference action agreement on real observations, units/order/timestamps, observation age, stale-input handling and application-specific deadline reporting |
|
||||
| Independent | Qwen3.8-27B | Public Omni pipeline binding; text, then image/video and longer contexts; same-model CPU/iGPU artifact and explicit offload evaluation | Modality-specific quality, actual loading/runtime memory and measured performance; each precision/context receives its own qualification |
|
||||
|
||||
|
||||
Reference in new issue
Block a user