Validate MiniCPM-o image to speech on WSL GPU
This commit is contained in:
tzhouam committed
1 parent
8a4559c46c
commit
0d00ba4701
11 files changed
+929
-3
No files matched your search
@@ -6,6 +6,7 @@ This extends the [60-cell device × model matrix](../e2e_profiling_20260922/evid
|
||||
|---|---|---|---|
|
||||
| WSL CPU / MiniCPM-o 4.5, revision `503e754207c94da6bb26850b4469f367c9ea3582` | Scoped complete three-stage text-to-text+speech path (P for this input only) | Coherent text and a 5.2 s mono 24 kHz WAV. First request 27.09 s after 73.65 s startup; separate 1-warmup/20-measured serial run: startup 73.89 s, request wall p50 21.86 s/p95 22.97 s. | Speech intelligibility/alignment, image/audio/video inputs, concurrency, interruption, power and thermal behavior. |
|
||||
| Native Windows RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped complete three-stage text-to-text+speech path (P for this input only) | Coherent text with the same 32 thinker token IDs as WSL CUDA, and a nonzero 5.2 s mono 24 kHz WAV. First engine startup 297.20 s and request 78.75 s, including first-use graph/JIT work; separate 1-warmup/20-measured serial run: startup 239.05 s, request wall p50 18.38 s/p95 18.73 s. | Speech intelligibility/alignment, image/audio/video inputs, concurrency, interruption, loading peak and sustained behavior. |
|
||||
| WSL RTX 5090 Laptop / MiniCPM-o 4.5, same revision | Scoped three-stage **image**-to-text+speech path (P for one synthetic image) | For a 448×448 red square on white, returned the correct visual description and a nonzero 5.2 s WAV. Engine startup 101.96 s; one request 17.49 s. | Broader visual quality, real images, audio/video inputs, speech intelligibility, repeated latency and cleanup reliability. |
|
||||
| WSL CPU / InternVLA-A1-3B Place_Markpen, revision `0a557c6a503e6545b369bff2f34c76a410bbd190` | Strict real-weight load and full policy forward on **synthetic zero** observations/noise | Finite float32 actions `[1, 50, 32]`; CPU-only PyTorch startup 31.78 s and synchronized forward 3.25 s. Compared with a same-checkpoint CUDA run, relative L2 difference 1.3025% and cosine similarity 0.999917. | Real Place_Markpen observations and reference actions, task tolerance, units/timestamps, repeated latency, memory and deadlines. |
|
||||
| Native Windows CPU / InternVLA-A1-3B Place_Markpen, same revision | Strict real-weight load and full policy forward on **synthetic zero** observations/noise; policy and actions checked on CPU | Finite float32 actions `[1, 50, 32]`; placement-checked startup 61.66 s and forward 3.36 s. Relative L2 difference from WSL CPU 1.0824%; cosine 0.999942. | Real observations/reference actions and quality tolerance; repeated latency, memory, units/timestamps and deadlines. This is the direct policy CPU path, not native vLLM CPU attention. |
|
||||
| Native Windows RTX 5090 Laptop / InternVLA-A1-3B Place_Markpen, same revision | Strict real-weight load and full policy forward on **synthetic zero** observations/noise; policy and actions checked on CUDA | Finite float32 actions `[1, 50, 32]`; placement-checked startup 75.54 s and synchronized forward 1.12 s. Relative L2 difference from WSL CUDA 0.8315%; cosine 0.999966. | Real observations/reference actions and quality tolerance; repeated latency, memory, units/timestamps and deadlines. |
|
||||
@@ -18,6 +19,8 @@ Native Windows MiniCPM-o used the same [constrained CUDA deployment config](../e
|
||||
|
||||
The separate native Windows [20-request profile](evidence/minicpmo_windows/minicpmo_windows_gpu_profile20.json), [per-request JSONL](evidence/minicpmo_windows/minicpmo_windows_gpu_profile20.jsonl), and [raw log](evidence/minicpmo_windows/minicpmo_windows_gpu_profile20.log) used one warmup followed by 20 measured serial requests in one engine. All measured requests returned one consistent thinker token sequence and 124,800-sample audio with nonzero RMS. Nearest-rank request wall p50/p95 were 18.38/18.73 s (real-time factor 3.53/3.60 for 5.2 s of audio), with observed range 18.22–19.79 s. Engine startup was 239.05 s on this later, warm-filesystem run. During measured requests, 0.5 s samples found native host available RAM as low as 0.78 GB and pagefile use as high as 11.17 GB; summed process-tree RSS reached 21.08 GB but double-counts shared pages. These are host-wide samples, not model-only RAM or loading peak. Neither a playable-stream timing nor speech intelligibility, concurrency, interruption, power or a 30-minute thermal run was measured.
|
||||
|
||||
The WSL RTX image path used the same real checkpoint and BF16 three-stage pipeline with a declared one-image limit, 2,048-token thinker context, 512-token talker context and 5 GiB thinker host offload. The [synthetic 448×448 RGB fixture](evidence/minicpmo_wsl_image/red_square.png) contains a solid red square centered on white. The first [0.58 thinker-memory plan](evidence/minicpmo_wsl_image/minicpmo_24gb_image_failed_kv.yaml) failed before requests because its 0.10 GiB available KV was below the 0.28 GiB required for the declared context; the [raw failure](evidence/minicpmo_wsl_image/minicpmo_image_failed_kv.log) is retained. Raising only the thinker's `gpu_memory_utilization` to 0.61 in the [successful plan](evidence/minicpmo_wsl_image/minicpmo_24gb_image_offline.yaml) admitted the same context, image and model. The [raw successful log](evidence/minicpmo_wsl_image/minicpmo_image_061_success.log), [text/token IDs](evidence/minicpmo_wsl_image/0_effea4b3-d0b9-448a-8a48-4b43ed3d6556.txt), [WAV](evidence/minicpmo_wsl_image/output_0_effea4b3-d0b9-448a-8a48-4b43ed3d6556.wav), and [artifact check](evidence/minicpmo_wsl_image/minicpmo_image_artifact_report.json) establish one complete image-to-text+speech request. The answer describes the visible red square and white background; this does not qualify general image reasoning or speech meaning. Shutdown force-killed stage 0 and reported one shared-memory cleanup warning, which remains a reliability gate.
|
||||
|
||||
The CPU-only InternVLA run used the same BF16 policy checkpoint, synthetic six-camera history/state input and zero diffusion noise as the CUDA reference. The raw [CPU JSON](evidence/internvla/internvla_pure_cpu_synthetic.json), [CPU log](evidence/internvla/internvla_pure_cpu_synthetic.log), [CPU actions](evidence/internvla/internvla_pure_cpu_actions.npy), [CUDA JSON](evidence/internvla/internvla_cuda_synthetic_with_actions.json), [CUDA actions](evidence/internvla/internvla_cuda_actions.npy) and [numerical comparison](evidence/internvla/internvla_pure_cpu_cuda_parity.json) are retained. The CUDA reference used the WSL RTX 5090 Laptop and PyTorch 2.13.0+cu130; its one forward took 1.33 s. The action difference has maximum absolute error 0.00993 and mean absolute error 0.00117. No reference-task tolerance is available, so this comparison is evidence of numerical proximity, not an accuracy pass.
|
||||
|
||||
The native Windows runs used Windows 11 build 26200, NVIDIA driver 610.71 for the GPU path, Python 3.12.10, PyTorch 2.13.0+cu130, vLLM 0.29.0, Transformers 5.14.1, and the current Omni source. The CPU run explicitly selected the CPU policy despite the CUDA-capable wheel. The placement-checked reruns assert that all policy parameters and the output action tensor remain on the requested device. Both loaded the same checkpoint from the local WSL filesystem through a UNC path; Windows CPU and GPU runs used the same synthetic input and zero diffusion noise as the WSL reference. The native [CPU JSON](evidence/internvla/internvla_windows_cpu_synthetic.json), [GPU JSON](evidence/internvla/internvla_windows_gpu_synthetic.json), [CPU log](evidence/internvla/internvla_windows_cpu_synthetic.log), [GPU log](evidence/internvla/internvla_windows_gpu_synthetic.log), [CPU actions](evidence/internvla/internvla_windows_cpu_actions.npy), [GPU actions](evidence/internvla/internvla_windows_gpu_actions.npy), [CPU/WSL comparison](evidence/internvla/internvla_windows_cpu_wsl_cpu_parity.json), and [GPU/WSL comparison](evidence/internvla/internvla_windows_gpu_wsl_cuda_parity.json) are retained. Native vLLM import initially failed on a Linux-specific CuMem allocator patch; the added Windows guard resolved the import. Native default GBK source decoding also required `PYTHONUTF8=1`. Both platform differences are setup requirements, not model-quality evidence. Power mode and thermal state were not captured.
|
||||
@@ -44,6 +47,15 @@ export CUDA_VISIBLE_DEVICES= VLLM_TARGET_DEVICE=cpu VLLM_CPU_KVCACHE_SPACE=1 VLL
|
||||
--model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5 \
|
||||
--deploy-config benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo/minicpmo_cpu_offline.yaml \
|
||||
--warmups 1 --repeats 20 --output /tmp/minicpmo-cpu-profile-20.json
|
||||
|
||||
CUDA_VISIBLE_DEVICES=0 VLLM_TARGET_DEVICE=cuda VLLM_USE_FLASHINFER_SAMPLER=0 \
|
||||
/home/zhout/project/edge_infer/.venvs/omni-cuda-029/bin/python examples/offline_inference/minicpmo/end2end.py \
|
||||
--model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5 \
|
||||
--deploy-config benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_wsl_image/minicpmo_24gb_image_offline.yaml \
|
||||
--query-type use_image \
|
||||
--image-path benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/minicpmo_wsl_image/red_square.png \
|
||||
--modalities text,audio --num-prompts 1 --init-timeout 900 --stage-init-timeout 600 \
|
||||
--output-dir /tmp/minicpmo-wsl-image
|
||||
```
|
||||
|
||||
For native Windows InternVLA, set `PYTHONUTF8=1`, put this checkout on `PYTHONPATH`, and run `benchmarks/edge_harness/probe_internvla_actions.py` with the same model/processor paths as above but Windows-accessible UNC paths; use `--device cpu` or `--device cuda` and separate output paths.
|
||||
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
Prompt:
|
||||
<|im_start|>system
|
||||
You are MiniCPM-o, a helpful multimodal assistant that can understand images, audio and video, and respond in text and speech.<|im_end|>
|
||||
<|im_start|>user
|
||||
(<image>./</image>)
|
||||
What is the content of this image?<|im_end|>
|
||||
<|im_start|>assistant
|
||||
<think>
|
||||
|
||||
</think>
|
||||
|
||||
<|tts_bos|>
|
||||
vllm_text_output:
|
||||
The image shows a solid red square centered on a white background. There are no other elements, text, or details visible.
|
||||
vllm_token_ids:
|
||||
[785, 2168, 4933, 264, 6437, 2518, 9334, 30188, 389, 264, 4158, 4004, 13, 2619, 525, 902, 1008, 5424, 11, 1467, 11, 476, 3565, 9434, 13, 151704, 151645]
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
# MiniCPM-o 4.5 offline image-to-text+speech probe on WSL RTX 5090 Laptop.
|
||||
# This admits one image, limits context/concurrency, and offloads 5 GiB.
|
||||
# Duplex session controls are omitted for ordinary offline requests.
|
||||
pipeline: minicpmo_4_5
|
||||
async_chunk: true
|
||||
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
connector_get_sleep_s: 0.01
|
||||
connector_get_max_wait_first_chunk: 3000
|
||||
connector_get_max_wait: 300
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 3
|
||||
# HiFT capture buckets are derived from codec_chunk_frames and
|
||||
# codec_left_context_frames. (#5869)
|
||||
# A future initial_codec_chunk_frames shape will use lazy capture unless
|
||||
# HiFTGraphWrapper.derive_capture_bucket_size explicitly includes it.
|
||||
enable_hift_graph: true
|
||||
enable_cfm_graph: true
|
||||
cfm_max_graphs: 32
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_seqs: 1
|
||||
# Bound the thinker budget and offload weights for GPU co-residency.
|
||||
gpu_memory_utilization: 0.58
|
||||
cpu_offload_gb: 5
|
||||
enforce_eager: true
|
||||
tensor_parallel_size: 1
|
||||
max_num_batched_tokens: 2048
|
||||
max_model_len: 2048
|
||||
devices: "0"
|
||||
# The image probe submits one synthetic RGB image.
|
||||
interleave_mm_strings: true
|
||||
media_io_kwargs:
|
||||
video:
|
||||
fps: 1
|
||||
num_frames: 128
|
||||
limit_mm_per_prompt:
|
||||
# One image; no audio or video inputs.
|
||||
image: 1
|
||||
audio: 0
|
||||
video: 0
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 64
|
||||
seed: 42
|
||||
detokenize: true
|
||||
# 1.0 for MCQ accuracy; 1.1 can bias short A–D answers.
|
||||
repetition_penalty: 1.0
|
||||
|
||||
- stage_id: 1
|
||||
max_num_seqs: 1
|
||||
# One short utterance; this does not qualify the full 4096-token Talker.
|
||||
gpu_memory_utilization: 0.10
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 512
|
||||
# The checkpoint supports 4096, but this probe explicitly admits 512.
|
||||
max_model_len: 512
|
||||
devices: "0"
|
||||
output_connectors:
|
||||
to_stage_2: connector_of_shared_memory
|
||||
# Talker is a single-vocab codec LM. These knobs go to vLLM's Sampler
|
||||
# (head_code logits). EOS is tts_config.eos_token_id (num_audio_tokens-1).
|
||||
# Values track upstream utils.TTSSamplingParams (min_p/win_size/tau_r are
|
||||
# declared there but never read, so they stay out). repetition_penalty is
|
||||
# consumed by the Talker, which rescores it over upstream's 16-frame window
|
||||
# instead of the Sampler's whole-stream presence penalty.
|
||||
# min_tokens is upstream's min_new_token.
|
||||
default_sampling_params:
|
||||
temperature: 0.8
|
||||
top_p: 0.85
|
||||
top_k: 25
|
||||
repetition_penalty: 1.05
|
||||
min_tokens: 50
|
||||
max_tokens: 128
|
||||
seed: 42
|
||||
detokenize: false
|
||||
|
||||
- stage_id: 2
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.10
|
||||
enforce_eager: true
|
||||
enable_chunked_prefill: false
|
||||
max_num_batched_tokens: 4096
|
||||
max_model_len: 4096
|
||||
devices: "0"
|
||||
input_connectors:
|
||||
from_stage_1: connector_of_shared_memory
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 4096
|
||||
detokenize: true
|
||||
|
||||
platforms:
|
||||
cuda:
|
||||
stages:
|
||||
- stage_id: 1
|
||||
# The cap must still admit the single 512-token Talker session.
|
||||
kv_cache_memory_bytes: 268435456 # 256 MiB for the 512-token probe
|
||||
|
||||
npu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_batched_tokens: 8192
|
||||
compilation_config:
|
||||
cudagraph_mode: PIECEWISE
|
||||
- stage_id: 1
|
||||
max_num_batched_tokens: 8192
|
||||
compilation_config:
|
||||
cudagraph_mode: PIECEWISE
|
||||
- stage_id: 2
|
||||
# Stage 2 stays outer-eager and captures only exact-shape CFM DiT
|
||||
# estimator kernels through the NPU platform patch.
|
||||
additional_config:
|
||||
code2wav_enable_npu_graph: true
|
||||
code2wav_max_npu_graphs: 32
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
# MiniCPM-o 4.5 offline image-to-text+speech probe on WSL RTX 5090 Laptop.
|
||||
# This admits one image, limits context/concurrency, and offloads 5 GiB.
|
||||
# Duplex session controls are omitted for ordinary offline requests.
|
||||
pipeline: minicpmo_4_5
|
||||
async_chunk: true
|
||||
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
connector_get_sleep_s: 0.01
|
||||
connector_get_max_wait_first_chunk: 3000
|
||||
connector_get_max_wait: 300
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 3
|
||||
# HiFT capture buckets are derived from codec_chunk_frames and
|
||||
# codec_left_context_frames. (#5869)
|
||||
# A future initial_codec_chunk_frames shape will use lazy capture unless
|
||||
# HiFTGraphWrapper.derive_capture_bucket_size explicitly includes it.
|
||||
enable_hift_graph: true
|
||||
enable_cfm_graph: true
|
||||
cfm_max_graphs: 32
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_seqs: 1
|
||||
# Bound the thinker budget and offload weights for GPU co-residency.
|
||||
# The image encoder cache leaves insufficient thinker KV at 0.58.
|
||||
gpu_memory_utilization: 0.61
|
||||
cpu_offload_gb: 5
|
||||
enforce_eager: true
|
||||
tensor_parallel_size: 1
|
||||
max_num_batched_tokens: 2048
|
||||
max_model_len: 2048
|
||||
devices: "0"
|
||||
# The image probe submits one synthetic RGB image.
|
||||
interleave_mm_strings: true
|
||||
media_io_kwargs:
|
||||
video:
|
||||
fps: 1
|
||||
num_frames: 128
|
||||
limit_mm_per_prompt:
|
||||
# One image; no audio or video inputs.
|
||||
image: 1
|
||||
audio: 0
|
||||
video: 0
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 64
|
||||
seed: 42
|
||||
detokenize: true
|
||||
# 1.0 for MCQ accuracy; 1.1 can bias short A–D answers.
|
||||
repetition_penalty: 1.0
|
||||
|
||||
- stage_id: 1
|
||||
max_num_seqs: 1
|
||||
# One short utterance; this does not qualify the full 4096-token Talker.
|
||||
gpu_memory_utilization: 0.10
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 512
|
||||
# The checkpoint supports 4096, but this probe explicitly admits 512.
|
||||
max_model_len: 512
|
||||
devices: "0"
|
||||
output_connectors:
|
||||
to_stage_2: connector_of_shared_memory
|
||||
# Talker is a single-vocab codec LM. These knobs go to vLLM's Sampler
|
||||
# (head_code logits). EOS is tts_config.eos_token_id (num_audio_tokens-1).
|
||||
# Values track upstream utils.TTSSamplingParams (min_p/win_size/tau_r are
|
||||
# declared there but never read, so they stay out). repetition_penalty is
|
||||
# consumed by the Talker, which rescores it over upstream's 16-frame window
|
||||
# instead of the Sampler's whole-stream presence penalty.
|
||||
# min_tokens is upstream's min_new_token.
|
||||
default_sampling_params:
|
||||
temperature: 0.8
|
||||
top_p: 0.85
|
||||
top_k: 25
|
||||
repetition_penalty: 1.05
|
||||
min_tokens: 50
|
||||
max_tokens: 128
|
||||
seed: 42
|
||||
detokenize: false
|
||||
|
||||
- stage_id: 2
|
||||
max_num_seqs: 1
|
||||
gpu_memory_utilization: 0.10
|
||||
enforce_eager: true
|
||||
enable_chunked_prefill: false
|
||||
max_num_batched_tokens: 4096
|
||||
max_model_len: 4096
|
||||
devices: "0"
|
||||
input_connectors:
|
||||
from_stage_1: connector_of_shared_memory
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 4096
|
||||
detokenize: true
|
||||
|
||||
platforms:
|
||||
cuda:
|
||||
stages:
|
||||
- stage_id: 1
|
||||
# The cap must still admit the single 512-token Talker session.
|
||||
kv_cache_memory_bytes: 268435456 # 256 MiB for the 512-token probe
|
||||
|
||||
npu:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
max_num_batched_tokens: 8192
|
||||
compilation_config:
|
||||
cudagraph_mode: PIECEWISE
|
||||
- stage_id: 1
|
||||
max_num_batched_tokens: 8192
|
||||
compilation_config:
|
||||
cudagraph_mode: PIECEWISE
|
||||
- stage_id: 2
|
||||
# Stage 2 stays outer-eager and captures only exact-shape CFM DiT
|
||||
# estimator kernels through the NPU platform patch.
|
||||
additional_config:
|
||||
code2wav_enable_npu_graph: true
|
||||
code2wav_max_npu_graphs: 32
|
||||
+303
@@ -0,0 +1,303 @@
|
||||
INFO 09-23 16:58:34 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 16:58:34 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 16:58:34 [patch.py:589] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
|
||||
====================
|
||||
vllm version: 0.29.0
|
||||
====================
|
||||
INFO 09-23 16:58:35 [omni_base.py:216] [Omni] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 16:58:35 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 16:58:35 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 3 stages
|
||||
INFO 09-23 16:58:35 [stage_init_utils.py:768] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
INFO 09-23 16:58:35 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations
|
||||
WARNING 09-23 16:58:35 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 16:58:35 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 16:58:35 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 16:58:35 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 16:58:35 [model.py:2021] Using max model len 2048
|
||||
WARNING 09-23 16:58:35 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 16:58:36 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
WARNING 09-23 16:58:36 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-23 16:58:36 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-23 16:58:36 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 16:58:36 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
INFO 09-23 16:58:36 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-23 16:58:36 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 16:58:36 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 16:58:36 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 16:58:36 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 16:58:36 [model.py:2021] Using max model len 512
|
||||
WARNING 09-23 16:58:36 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 16:58:36 [model.py:2021] Using max model len 512
|
||||
INFO 09-23 16:58:36 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=512.
|
||||
INFO 09-23 16:58:36 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
WARNING 09-23 16:58:36 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 16:58:36 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 16:58:36 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 16:58:36 [model.py:684] Resolved architecture: MiniCPMO45Code2Wav
|
||||
INFO 09-23 16:58:36 [model.py:2021] Using max model len 4096
|
||||
WARNING 09-23 16:58:36 [arg_utils.py:2780] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs.
|
||||
INFO 09-23 16:58:36 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 16:58:36 [stage_utils.py:174] Stage 0 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:58:36 [stage_utils.py:174] Stage 1 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:58:36 [stage_utils.py:174] Stage 2 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:58:36 [stage_utils.py:174] Stage 0 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:58:36 [stage_runtime.py:853] [stage_init] Stage-0 set runtime devices: 0
|
||||
INFO 09-23 16:58:42 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 16:58:42 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 16:58:42 [patch.py:589] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:43 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:44 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_a4805aeaa1ac40ec90877afb5ffed118 backend=nccl
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:44 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:58:44 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:44 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:48 [base.py:138] Offloader set to UVAOffloader
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:58:48 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:58:48 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:48 [gpu_model_runner.py:5379] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:58:49 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:58:49 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:49 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:49 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:49 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:49 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:49 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:49 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:49 [flash_attn.py:897] Using FlashAttention version 2
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:54 [uva.py:65] Total CPU offloaded parameters: 5.03
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:54 [weight_utils.py:863] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 22.19 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:58:54 [weight_utils.py:886] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 238.11it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:03 [default_loader.py:430] Loading weights took 9.12 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:07 [gpu_model_runner.py:5471] Model loading took 11.8 GiB memory and 17.575991 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:08 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:08 [gpu_model_runner.py:6546] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 1 image items of the maximum feature size.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) [transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) /home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py:202: FutureWarning: The attention mask API under `transformers.modeling_attn_mask_utils` (`AttentionMaskConverter`) is deprecated and will be removed in Transformers v5.10. Please use the new API in `transformers.masking_utils`.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) warnings.warn(DEPRECATION_MESSAGE, FutureWarning)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:12 [base.py:148] Available KV cache memory: 0.82 GiB (device-level profiling)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:12 [kv_cache_utils.py:2032] GPU KV cache size: 5,952 tokens, Maximum concurrency for 2,048 tokens per request: 2.91x
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:12 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:12 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) ␍JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]␍JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 1.92kernel/s, ComputeSlotMappingKernel (1 keys)]␍JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 1.92kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:12 [kernel_warmup.py:134] JIT kernel warmup finished in 0.52s.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] Module vllm.third_party.deep_gemm was found but failed to import
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] Traceback (most recent call last):
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/utils/import_utils.py", line 406, in _has_module
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] importlib.import_module(module_name)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/importlib/__init__.py", line 90, in import_module
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] return _bootstrap._gcd_import(name[level:], package, level)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "<frozen importlib._bootstrap_external>", line 999, in exec_module
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/third_party/deep_gemm/__init__.py", line 126, in <module>
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] _find_cuda_home() # CUDA home
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] ^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/third_party/deep_gemm/__init__.py", line 120, in _find_cuda_home
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] assert cuda_home is not None
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:13 [import_utils.py:408] AssertionError
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:13 [gpu_worker.py:860] Free memory on device (22.58/23.89 GiB) on startup. Desired GPU memory utilization is (0.61, 14.57 GiB). Actual usage is 12.3 GiB for consumed memory (weights + non-torch), 1.88 GiB for peak activation, and 0.0 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=269020201` (0.25 GiB) to fit into requested memory, or `--kv-cache-memory=8864026624` (8.26 GiB) to fully utilize gpu memory. Current kv cache memory in use is 0.82 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:14 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:15 [torch_utils.py:265] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:15 [core.py:368] init engine (profile, create kv cache, warmup model) took 7.42 s
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:15 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 16:59:15 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:15 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1624223) INFO 09-23 16:59:15 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 16:59:15 [stage_runtime.py:943] [StageRuntime] Stage 0 engine startup completed
|
||||
INFO 09-23 16:59:15 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
|
||||
INFO 09-23 16:59:15 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
|
||||
INFO 09-23 16:59:15 [stage_runtime.py:958] [StageRuntime] Stage 0 initialized
|
||||
INFO 09-23 16:59:15 [stage_utils.py:174] Stage 1 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:59:15 [stage_runtime.py:853] [stage_init] Stage-1 set runtime devices: 0
|
||||
INFO 09-23 16:59:22 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 16:59:22 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 16:59:22 [patch.py:589] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:23 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=512, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:24 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_e10b7f035df8415ea4151996975d91fc backend=nccl
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:24 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:25 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:25 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) [transformers] Image processor MiniCPMVImageProcessor: kwargs ['max_slice_nums'] were applied for backward compatibility. To avoid this warning, add them to valid_kwargs: create a custom TypedDict extending ImagesKwargs with these keys and set it as the `valid_kwargs` class attribute.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:35 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:35 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:35 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:35 [gpu_model_runner.py:5379] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:35 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:35 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:35 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:35 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:35 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:35 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:36 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:36 [flash_attn.py:897] Using FlashAttention version 2
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:36 [weight_utils.py:863] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 20.37 GiB.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:36 [weight_utils.py:886] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 233.05it/s]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:37 [default_loader.py:430] Loading weights took 0.55 seconds
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:38 [gpu_model_runner.py:5471] Model loading took 0.65 GiB memory and 1.426251 seconds
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:38 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:38 [gpu_model_runner.py:6519] Skipping memory profiling for multimodal encoder and encoder cache.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:39 [kv_cache_utils.py:2032] GPU KV cache size: 4,368 tokens, Maximum concurrency for 512 tokens per request: 8.53x
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:39 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:39 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) ␍JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]␍JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 3.29kernel/s, ComputeSlotMappingKernel (1 keys)]␍JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 3.29kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:39 [kernel_warmup.py:134] JIT kernel warmup finished in 0.30s.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] Module vllm.third_party.deep_gemm was found but failed to import
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] Traceback (most recent call last):
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/utils/import_utils.py", line 406, in _has_module
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] importlib.import_module(module_name)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/importlib/__init__.py", line 90, in import_module
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] return _bootstrap._gcd_import(name[level:], package, level)
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "<frozen importlib._bootstrap_external>", line 999, in exec_module
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/third_party/deep_gemm/__init__.py", line 126, in <module>
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] _find_cuda_home() # CUDA home
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] ^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/third_party/deep_gemm/__init__.py", line 120, in _find_cuda_home
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] assert cuda_home is not None
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:40 [import_utils.py:408] AssertionError
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:40 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:41 [torch_utils.py:265] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:41 [core.py:368] init engine (profile, create kv cache, warmup model) took 2.41 s
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:41 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) WARNING 09-23 16:59:41 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:41 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=1624394) INFO 09-23 16:59:41 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 16:59:41 [stage_runtime.py:943] [StageRuntime] Stage 1 engine startup completed
|
||||
INFO 09-23 16:59:41 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-1 [rep-0] initializing EngineCore
|
||||
INFO 09-23 16:59:41 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-1 [rep-0] EngineCore running
|
||||
INFO 09-23 16:59:41 [stage_runtime.py:958] [StageRuntime] Stage 1 initialized
|
||||
INFO 09-23 16:59:41 [stage_utils.py:174] Stage 2 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:59:41 [stage_runtime.py:853] [stage_init] Stage-2 set runtime devices: 0
|
||||
INFO 09-23 16:59:50 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 16:59:50 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 16:59:50 [patch.py:589] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 16:59:51 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=False, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 16:59:51 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_370a1bbb63b64a499d39ccf5ce0e64e3 backend=nccl
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 16:59:51 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 16:59:52 [gpu_generation_worker.py:103] OMNI GPUGenerationWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 16:59:52 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 16:59:52 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 16:59:52 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 16:59:52 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 16:59:52 [gpu_model_runner.py:5379] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 16:59:53 [weight_utils.py:863] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 18.42 GiB.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 16:59:53 [weight_utils.py:893] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (17.46 GiB) exceeds 90% of available RAM (18.42 GiB).
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 221.82it/s]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 16:59:53 [step_audio2_token2wav.py:163] Loading Token2Wav models from: /home/zhout/project/edge_infer/models/MiniCPM-o-4_5/assets/token2wav
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:04 [step_audio2_token2wav.py:201] Token2Wav models loaded successfully
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:05 [batched_token2wav.py:227] Enabling HiFT CUDA Graph with batch sizes [1]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:06 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 50, 0)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:06 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 58, 3840)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:06 [batched_token2wav.py:235] HiFT CUDA Graph captured successfully
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:06 [batched_token2wav.py:246] CFM CUDA Graph enabled (max_graphs=32)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:06 [default_loader.py:430] Loading weights took 13.09 seconds
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:07 [gpu_model_runner.py:5471] Model loading took 3.36 GiB memory and 13.119221 seconds
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:08 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:08 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:08 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) ␍JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]␍JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 430.58kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:08 [kernel_warmup.py:134] JIT kernel warmup finished in 0.00s.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] Module vllm.third_party.deep_gemm was found but failed to import
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] Traceback (most recent call last):
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/utils/import_utils.py", line 406, in _has_module
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] importlib.import_module(module_name)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/importlib/__init__.py", line 90, in import_module
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] return _bootstrap._gcd_import(name[level:], package, level)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "<frozen importlib._bootstrap_external>", line 999, in exec_module
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/third_party/deep_gemm/__init__.py", line 126, in <module>
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] _find_cuda_home() # CUDA home
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] ^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/third_party/deep_gemm/__init__.py", line 120, in _find_cuda_home
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] assert cuda_home is not None
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [import_utils.py:408] AssertionError
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:08 [gpu_generation_model_runner.py:558] Dummy sampler run is not implemented for generation model
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:10 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:10 [torch_utils.py:265] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:10 [core.py:368] init engine (profile, create kv cache, warmup model) took 2.79 s
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:11 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:11 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:11 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:11 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:11 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:11 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:11 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
INFO 09-23 17:00:11 [stage_runtime.py:943] [StageRuntime] Stage 2 engine startup completed
|
||||
INFO 09-23 17:00:11 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-2 [rep-0] initializing EngineCore
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:11 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
INFO 09-23 17:00:11 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-2 [rep-0] EngineCore running
|
||||
INFO 09-23 17:00:11 [stage_runtime.py:958] [StageRuntime] Stage 2 initialized
|
||||
[transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
[transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
[transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
INFO 09-23 17:00:17 [orchestrator.py:604] [Orchestrator] Starting event loop
|
||||
INFO 09-23 17:00:17 [async_omni_engine.py:323] [AsyncOmniEngine] Orchestrator ready with 3 stages
|
||||
INFO 09-23 17:00:17 [omni_base.py:235] [Omni] AsyncOmniEngine initialized in 101.96 seconds
|
||||
INFO 09-23 17:00:17 [omni_base.py:255] [Omni] Initialized with 3 stages for model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
WARNING 09-23 17:00:17 [input_processor.py:315] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
|
||||
␍Processed prompts: 0%| | 0/1 [00:00<?, ?it/s](StageEngineCoreProc_stage0_replica0 pid=1624223) WARNING 09-23 17:00:28 [gpu_ar_model_runner.py:1464] prefer_model_sampler model MiniCPMO45OmniForConditionalGeneration returned None from sample(); falling back to the default sampler.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:42 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 302), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 0, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 302, 128), 'torch.float32', 'cuda:0')) (cache=1/32, stats={'calls': 1, 'hits': 0, 'captures': 1, 'flushes': 0, 'eager': 0, 'cache_size': 0}) [allocated 3633.9 MiB (+32.2), reserved 5464.0 MiB (+22.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:42 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 50), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 302, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 352, 128), 'torch.float32', 'cuda:0')) (cache=2/32, stats={'calls': 11, 'hits': 9, 'captures': 2, 'flushes': 0, 'eager': 0, 'cache_size': 1}) [allocated 4539.7 MiB (+32.0), reserved 6378.0 MiB (+28.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:42 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 50), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 352, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 402, 128), 'torch.float32', 'cuda:0')) (cache=3/32, stats={'calls': 21, 'hits': 18, 'captures': 3, 'flushes': 0, 'eager': 0, 'cache_size': 2}) [allocated 4803.9 MiB (+32.0), reserved 6956.0 MiB (+-352.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:43 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 50), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 402, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 452, 128), 'torch.float32', 'cuda:0')) (cache=4/32, stats={'calls': 31, 'hits': 27, 'captures': 4, 'flushes': 0, 'eager': 0, 'cache_size': 3}) [allocated 5079.1 MiB (+32.0), reserved 7096.0 MiB (+-876.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:44 [cuda_graph_wrapper.py:300] Captured CFM CUDA Graph for shape ('estimator_step', ((2, 320, 10), 'torch.float32', 'cuda:0'), ((2, 1, 512), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 402, 128), 'torch.float32', 'cuda:0'), ((16, 2, 1024, 2), 'torch.float32', 'cuda:0'), ((16, 2, 8, 412, 128), 'torch.float32', 'cuda:0')) (cache=5/32, stats={'calls': 61, 'hits': 56, 'captures': 5, 'flushes': 0, 'eager': 0, 'cache_size': 4}) [allocated 5206.6 MiB (+32.0), reserved 7190.0 MiB (+-1550.0)]
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:44 [cuda_graph_wrapper.py:120] Lazily capturing HiFT CUDA Graph for shape (1, 18, 3840)
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) WARNING 09-23 17:00:44 [jit_monitor.py:141] Triton kernel JIT compilation during inference: SnakeBeta._init_triton.<locals>._kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
|
||||
(StageEngineCoreProc_stage2_replica0 pid=1624538) INFO 09-23 17:00:44 [cuda_graph_wrapper.py:98] Captured HiFT CUDA Graph for shape (1, 18, 3840)
|
||||
␍Processed prompts: 100%|██████████| 1/1 [00:17<00:00, 17.49s/it]␍Processed prompts: 100%|██████████| 1/1 [00:17<00:00, 17.49s/it]
|
||||
query type: use_image
|
||||
use_tts (tts_bos): True
|
||||
Request ID: 0_effea4b3-d0b9-448a-8a48-4b43ed3d6556, Text saved to /home/zhout/project/edge_infer/analysis/experiments/e2e_expansion_20260923/minicpmo_wsl_image/0_effea4b3-d0b9-448a-8a48-4b43ed3d6556.txt
|
||||
Request ID: 0_effea4b3-d0b9-448a-8a48-4b43ed3d6556, Saved audio to /home/zhout/project/edge_infer/analysis/experiments/e2e_expansion_20260923/minicpmo_wsl_image/output_0_effea4b3-d0b9-448a-8a48-4b43ed3d6556.wav
|
||||
INFO 09-23 17:00:44 [omni_base.py:781] [Omni] Shutting down
|
||||
INFO 09-23 17:00:44 [async_omni_engine.py:1807] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
INFO 09-23 17:00:44 [orchestrator.py:705] [Orchestrator] Received shutdown signal
|
||||
INFO 09-23 17:00:44 [orchestrator.py:3092] [Orchestrator] Shutting down all 3 client(s)
|
||||
INFO 09-23 17:00:44 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:00:44 [core_client.py:691] [shutdown] MPClient: stopping engine manager
|
||||
INFO 09-23 17:00:44 [utils.py:620] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
WARNING 09-23 17:00:49 [utils.py:640] [shutdown] Process manager: force killing remaining processes count=1
|
||||
WARNING 09-23 17:00:49 [utils.py:645] [shutdown] Process manager: force killing remaining process StageEngineCoreProc_stage0_replica0 pid 1624223
|
||||
INFO 09-23 17:00:49 [core_client.py:693] [shutdown] MPClient: engine manager stopped
|
||||
INFO 09-23 17:00:49 [core_client.py:694] [shutdown] MPClient: cleaning up background resources
|
||||
INFO 09-23 17:00:49 [core_client.py:696] [shutdown] MPClient: complete
|
||||
INFO 09-23 17:00:49 [stage_pool.py:1354] [StagePool] Stage 0 replica 0 shut down
|
||||
INFO 09-23 17:00:49 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:00:49 [utils.py:620] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage1_replica0
|
||||
INFO 09-23 17:00:53 [stage_pool.py:1354] [StagePool] Stage 1 replica 0 shut down
|
||||
INFO 09-23 17:00:53 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-23 17:00:53 [utils.py:620] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage2_replica0
|
||||
INFO 09-23 17:00:58 [stage_pool.py:1354] [StagePool] Stage 2 replica 0 shut down
|
||||
/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
|
||||
warnings.warn('resource_tracker: There appear to be %d '
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"scope": "synthetic 448x448 RGB red-square image to text+speech, one request, WSL RTX 5090 Laptop",
|
||||
"model_response": "The image shows a solid red square centered on a white background. There are no other elements, text, or details visible.",
|
||||
"ground_truth_description": "solid red square centered on white background",
|
||||
"token_count": 27,
|
||||
"wav": {
|
||||
"sample_rate_hz": 24000,
|
||||
"frames": 124800,
|
||||
"duration_s": 5.2,
|
||||
"nonzero_samples": 124434,
|
||||
"rms_float": 0.08685403380620296,
|
||||
"sha256": "1f2a2763d072773f076afd621f5632d7883d880d61d5b0368c4933bd9596ccd5"
|
||||
},
|
||||
"shutdown_note": "stage 0 required force kill; Python resource_tracker reported one shared_memory object"
|
||||
}
|
||||
+321
@@ -0,0 +1,321 @@
|
||||
INFO 09-23 16:56:08 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 16:56:08 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 16:56:08 [patch.py:589] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
|
||||
====================
|
||||
vllm version: 0.29.0
|
||||
====================
|
||||
INFO 09-23 16:56:09 [omni_base.py:216] [Omni] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 16:56:09 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5
|
||||
INFO 09-23 16:56:09 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 3 stages
|
||||
INFO 09-23 16:56:09 [stage_init_utils.py:768] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
INFO 09-23 16:56:09 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations
|
||||
WARNING 09-23 16:56:09 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 16:56:09 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 16:56:09 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 16:56:09 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 16:56:09 [model.py:2021] Using max model len 2048
|
||||
WARNING 09-23 16:56:09 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 16:56:10 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
WARNING 09-23 16:56:10 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-23 16:56:10 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-23 16:56:10 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 16:56:10 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
INFO 09-23 16:56:10 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-23 16:56:10 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 16:56:10 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 16:56:10 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 16:56:10 [model.py:684] Resolved architecture: MiniCPMO45OmniForConditionalGeneration
|
||||
INFO 09-23 16:56:10 [model.py:2021] Using max model len 512
|
||||
WARNING 09-23 16:56:10 [model.py:981] Model does not support mm_device_do_normalize, forcing mm_device_do_normalize = False.
|
||||
INFO 09-23 16:56:10 [model.py:2021] Using max model len 512
|
||||
INFO 09-23 16:56:10 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=512.
|
||||
INFO 09-23 16:56:10 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
WARNING 09-23 16:56:10 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
WARNING 09-23 16:56:10 [utils.py:501] Dropping unknown OmniEngineArgs field 'shm_threshold_bytes' (not declared on the dataclass)
|
||||
WARNING 09-23 16:56:10 [arg_utils.py:1801] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
|
||||
INFO 09-23 16:56:10 [model.py:684] Resolved architecture: MiniCPMO45Code2Wav
|
||||
INFO 09-23 16:56:10 [model.py:2021] Using max model len 4096
|
||||
WARNING 09-23 16:56:10 [arg_utils.py:2780] This model does not officially support disabling chunked prefill. Disabling this manually may cause the engine to crash or produce incorrect outputs.
|
||||
INFO 09-23 16:56:10 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-23 16:56:10 [stage_utils.py:174] Stage 0 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:56:10 [stage_utils.py:174] Stage 1 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:56:10 [stage_utils.py:174] Stage 2 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:56:10 [stage_utils.py:174] Stage 0 logical-to-physical device mapping: 0->0
|
||||
INFO 09-23 16:56:10 [stage_runtime.py:853] [stage_init] Stage-0 set runtime devices: 0
|
||||
INFO 09-23 16:56:17 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-23 16:56:17 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-23 16:56:17 [patch.py:589] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:18 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/MiniCPM-o-4_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/MiniCPM-o-4_5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:19 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_932479702490457b841e10e45280634d backend=nccl
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:19 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) WARNING 09-23 16:56:19 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:19 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) [transformers] `MiniCPMOProcessor` defines `image_processor_class = 'AutoImageProcessor'`, which is deprecated. Register the correct mapping in `AutoImageProcessor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) [transformers] Requested torchvision backend is not available. Falling back to pil backend.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) [transformers] `MiniCPMOProcessor` defines `feature_extractor_class = 'WhisperFeatureExtractor'`, which is deprecated. Register the correct mapping in `AutoFeatureExtractor` instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:23 [base.py:138] Offloader set to UVAOffloader
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) WARNING 09-23 16:56:23 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) WARNING 09-23 16:56:23 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:23 [gpu_model_runner.py:5379] Starting to load model /home/zhout/project/edge_infer/models/MiniCPM-o-4_5...
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) WARNING 09-23 16:56:23 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) WARNING 09-23 16:56:23 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:23 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:23 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:23 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:24 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:24 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:25 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:25 [flash_attn.py:897] Using FlashAttention version 2
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:30 [uva.py:65] Total CPU offloaded parameters: 5.03
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:30 [weight_utils.py:863] Filesystem type for checkpoints: EXT4. Checkpoint size: 17.46 GiB. Available RAM: 22.28 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:56:30 [weight_utils.py:886] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ␍Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ␍Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 128.40it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:57:09 [default_loader.py:430] Loading weights took 38.36 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:57:12 [gpu_model_runner.py:5471] Model loading took 11.8 GiB memory and 47.933198 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:57:13 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:57:13 [gpu_model_runner.py:6546] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 1 image items of the maximum feature size.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) [transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) /home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py:202: FutureWarning: The attention mask API under `transformers.modeling_attn_mask_utils` (`AttentionMaskConverter`) is deprecated and will be removed in Transformers v5.10. Please use the new API in `transformers.masking_utils`.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) warnings.warn(DEPRECATION_MESSAGE, FutureWarning)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) INFO 09-23 16:57:17 [base.py:148] Available KV cache memory: 0.1 GiB (device-level profiling)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] StageEngineCoreProc failed to start.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] Traceback (most recent call last):
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_engine_core_proc.py", line 189, in run_stage_core
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] engine_core = StageEngineCoreProc(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1093, in __init__
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] super().__init__(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 145, in __init__
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] kv_cache_config = self._initialize_kv_caches(vllm_config)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 319, in _initialize_kv_caches
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] kv_cache_configs = get_kv_cache_configs(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/core/kv_cache_utils.py", line 2331, in get_kv_cache_configs
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] _check_enough_kv_cache_memory(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/core/kv_cache_utils.py", line 879, in _check_enough_kv_cache_memory
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] raise ValueError(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ERROR 09-23 16:57:17 [stage_engine_core_proc.py:242] ValueError: To serve at least one request with the model's max seq len (2048), (0.28 GiB KV cache is needed, which is larger than the available KV cache memory (0.1 GiB). Based on the available memory, the estimated maximum model length is 720. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_model_len` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) Process StageEngineCoreProc_stage0_replica0:
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) Traceback (most recent call last):
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) self.run()
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) self._target(*self._args, **self._kwargs)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_engine_core_proc.py", line 189, in run_stage_core
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) engine_core = StageEngineCoreProc(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1093, in __init__
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) super().__init__(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 145, in __init__
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) kv_cache_config = self._initialize_kv_caches(vllm_config)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) return func(*args, **kwargs)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 319, in _initialize_kv_caches
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) kv_cache_configs = get_kv_cache_configs(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/core/kv_cache_utils.py", line 2331, in get_kv_cache_configs
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) _check_enough_kv_cache_memory(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/core/kv_cache_utils.py", line 879, in _check_enough_kv_cache_memory
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) raise ValueError(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=1623961) ValueError: To serve at least one request with the model's max seq len (2048), (0.28 GiB KV cache is needed, which is larger than the available KV cache memory (0.1 GiB). Based on the available memory, the estimated maximum model length is 720. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_model_len` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.
|
||||
[rank0]:[W923 16:57:18.015861669 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
|
||||
INFO 09-23 16:57:20 [utils.py:620] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] [StageRuntime] Stage initialization failed; shutting down 0 initialized client(s)
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] Traceback (most recent call last):
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 252, in initialize
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] initialized_clients = self._initialize_stage_replicas(stage_plans, self._stage_init_timeout)
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 589, in _initialize_stage_replicas
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] raise primary_exc
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 550, in _init_group
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] client = self._initialize_replica(
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] ^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 828, in _initialize_replica
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] return self._initialize_local_llm_replica(plan, stage_init_timeout)
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 934, in _initialize_local_llm_replica
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] launch_stack.close()
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 618, in close
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] self.__exit__(None, None, None)
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 610, in __exit__
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] raise exc_details[1]
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 595, in __exit__
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] if cb(*exc_details):
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] ^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] next(self.gen)
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_engine_startup.py", line 1211, in launch_stage_replica
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] wait_for_engine_startup(
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1320, in wait_for_engine_startup
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] raise RuntimeError(
|
||||
ERROR 09-23 16:57:20 [stage_runtime.py:265] RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] [AsyncOmniEngine] Orchestrator thread crashed
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] Traceback (most recent call last):
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 468, in _bootstrap_orchestrator
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] loop.run_until_complete(_run_orchestrator())
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] return future.result()
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] ^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 424, in _run_orchestrator
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] self._initialize_stages(stage_init_timeout)
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 350, in _initialize_stages
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] self._initialize_stages_impl(stage_init_timeout)
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 371, in _initialize_stages_impl
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] self._runtime.initialize()
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 278, in initialize
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] raise exc
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 252, in initialize
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] initialized_clients = self._initialize_stage_replicas(stage_plans, self._stage_init_timeout)
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 589, in _initialize_stage_replicas
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] raise primary_exc
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 550, in _init_group
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] client = self._initialize_replica(
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] ^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 828, in _initialize_replica
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] return self._initialize_local_llm_replica(plan, stage_init_timeout)
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 934, in _initialize_local_llm_replica
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] launch_stack.close()
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 618, in close
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] self.__exit__(None, None, None)
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 610, in __exit__
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] raise exc_details[1]
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 595, in __exit__
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] if cb(*exc_details):
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] ^^^^^^^^^^^^^^^^
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] next(self.gen)
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_engine_startup.py", line 1211, in launch_stage_replica
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] wait_for_engine_startup(
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1320, in wait_for_engine_startup
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] raise RuntimeError(
|
||||
ERROR 09-23 16:57:20 [async_omni_engine.py:474] RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
INFO 09-23 16:57:20 [async_omni_engine.py:1807] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
Exception in thread orchestrator:
|
||||
Traceback (most recent call last):
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
|
||||
self.run()
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
|
||||
self._target(*self._args, **self._kwargs)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 468, in _bootstrap_orchestrator
|
||||
loop.run_until_complete(_run_orchestrator())
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 424, in _run_orchestrator
|
||||
self._initialize_stages(stage_init_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 350, in _initialize_stages
|
||||
self._initialize_stages_impl(stage_init_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 371, in _initialize_stages_impl
|
||||
self._runtime.initialize()
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 278, in initialize
|
||||
raise exc
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 252, in initialize
|
||||
initialized_clients = self._initialize_stage_replicas(stage_plans, self._stage_init_timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 589, in _initialize_stage_replicas
|
||||
raise primary_exc
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 550, in _init_group
|
||||
client = self._initialize_replica(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 828, in _initialize_replica
|
||||
return self._initialize_local_llm_replica(plan, stage_init_timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 934, in _initialize_local_llm_replica
|
||||
launch_stack.close()
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 618, in close
|
||||
self.__exit__(None, None, None)
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 610, in __exit__
|
||||
raise exc_details[1]
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 595, in __exit__
|
||||
if cb(*exc_details):
|
||||
^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
|
||||
next(self.gen)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_engine_startup.py", line 1211, in launch_stage_replica
|
||||
wait_for_engine_startup(
|
||||
File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1320, in wait_for_engine_startup
|
||||
raise RuntimeError(
|
||||
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
Traceback (most recent call last):
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
|
||||
self.run()
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
|
||||
self._target(*self._args, **self._kwargs)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 468, in _bootstrap_orchestrator
|
||||
loop.run_until_complete(_run_orchestrator())
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 424, in _run_orchestrator
|
||||
self._initialize_stages(stage_init_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 350, in _initialize_stages
|
||||
self._initialize_stages_impl(stage_init_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 371, in _initialize_stages_impl
|
||||
self._runtime.initialize()
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 278, in initialize
|
||||
raise exc
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 252, in initialize
|
||||
initialized_clients = self._initialize_stage_replicas(stage_plans, self._stage_init_timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 589, in _initialize_stage_replicas
|
||||
raise primary_exc
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 550, in _init_group
|
||||
client = self._initialize_replica(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 828, in _initialize_replica
|
||||
return self._initialize_local_llm_replica(plan, stage_init_timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_runtime.py", line 934, in _initialize_local_llm_replica
|
||||
launch_stack.close()
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 618, in close
|
||||
self.__exit__(None, None, None)
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 610, in __exit__
|
||||
raise exc_details[1]
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 595, in __exit__
|
||||
if cb(*exc_details):
|
||||
^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
|
||||
next(self.gen)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_engine_startup.py", line 1211, in launch_stage_replica
|
||||
wait_for_engine_startup(
|
||||
File "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1320, in wait_for_engine_startup
|
||||
raise RuntimeError(
|
||||
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
|
||||
The above exception was the direct cause of the following exception:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/examples/offline_inference/minicpmo/end2end.py", line 519, in <module>
|
||||
main(args)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/examples/offline_inference/minicpmo/end2end.py", line 283, in main
|
||||
omni = Omni(**omni_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni_base.py", line 223, in __init__
|
||||
self.engine = AsyncOmniEngine(
|
||||
^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 305, in __init__
|
||||
self._wait_for_orchestrator_init(startup_future, startup_timeout)
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py", line 539, in _wait_for_orchestrator_init
|
||||
startup_future.result(
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 456, in result
|
||||
return self.__get_result()
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||||
raise self._exception
|
||||
RuntimeError: Orchestrator initialization failed: Engine core initialization failed. See root cause above. Failed core proc(s): {}
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
|
After Width: | Height: | Size: 1.5 KiB |
@@ -10,7 +10,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i
|
||||
|---|---|---|---|---|---|
|
||||
| PC HX370: CPU / WSL | profile completed | scoped E2E: short stream | NOT E2E | scoped E2E: text → speech | synthetic policy pass |
|
||||
| PC HX370: CPU / native Windows | NOT E2E | NOT E2E | NOT E2E | NOT E2E | synthetic policy pass |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | profile completed | profile completed | scoped E2E: text + image | scoped E2E: text → speech | synthetic policy pass |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | profile completed | profile completed | scoped E2E: text + image | scoped E2E: text/image → speech | synthetic policy pass |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | profile completed | profile completed | NOT E2E | scoped E2E: text → speech | synthetic policy pass |
|
||||
| PC HX370 + Radeon 890M / Windows worker | NOT E2E | NOT E2E | NOT E2E | NOT E2E | NOT E2E |
|
||||
| PC HX370 + AMD NPU / Windows worker | NOT E2E | NOT E2E | NOT E2E | NOT E2E | NOT E2E |
|
||||
@@ -38,7 +38,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | Spark-X2.5 | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Broaden only with a named workload/device/precision qualification. |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | Qwen3-TTS 0.6B CustomVoice | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Resolve and measure playback stalls; run quality, interruption and thermal tests. |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | Qwen3.8-27B | SCOPED_E2E | A new one-stage Omni binding ran the real NVFP4 checkpoint: one 16-token text request and one synthetic red-image request (answer `Red`) at an explicit 512-token context with host offload. This is not the full multimodal, long-context or quality workload. | Qualify image/video suites, longer contexts, latency distributions and sustained power; [recovery evidence](../../../e2e_recovery_20260923/README.md). |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | MiniCPM-o 4.5 | SCOPED_E2E | The verified real checkpoint completed one three-stage text-to-text+speech request under a constrained 24 GiB profile with 5 GiB host offload. Text was coherent and a nonzero 5.2 s WAV was produced in a single 20.53 s request; speech meaning was not checked. | Validate speech intelligibility/alignment, image and audio input, interruption, sustained load and power; [raw log](../../../e2e_recovery_20260923/evidence/minicpmo/mini_fixed_e2e.log). |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | MiniCPM-o 4.5 | SCOPED_E2E | The verified checkpoint completed three-stage text-to-text+speech under a constrained 24 GiB profile with 5 GiB host offload. A separate one-image plan described a synthetic 448×448 red square on white correctly and produced a nonzero 5.2 s WAV in a 17.49 s request. The original 0.58 thinker GPU budget failed image/KV admission (0.10 GiB available versus 0.28 GiB required); 0.61 admitted the same context and image. Stage 0 needed a force kill at shutdown, and speech meaning was not checked. | Validate broader images, audio/video input, speech intelligibility, interruption, repeated latency, shutdown cleanup and sustained power; [text evidence](../../../e2e_recovery_20260923/README.md), [image evidence](../../../e2e_expansion_20260923/README.md). |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | InternVLA-A1 | SYNTHETIC_POLICY_PASS | Strict real-weight Place_Markpen load and one policy forward on zero observations/noise produced finite `[1, 50, 32]` actions in 1.222 s. Synthetic input and one timing sample do not establish robot-task E2E or action semantics. | Run real observation/reference-action cases and validate units, timestamps and deadline behavior; [raw result](../../../e2e_recovery_20260923/evidence/internvla/internvla_finetune_synthetic.json). |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | Spark-X2.5 | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Broaden only with a named workload/device/precision qualification. |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | Qwen3-TTS 0.6B CustomVoice | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Resolve and measure playback stalls; run quality, interruption and thermal tests. |
|
||||
|
||||
@@ -27,6 +27,12 @@ use reached 11.17 GB. Native Windows InternVLA direct-policy runs separately
|
||||
passed on CPU and RTX 5090 Laptop with synthetic inputs; AMD and mobile cells
|
||||
do not inherit these passes.
|
||||
|
||||
The WSL RTX MiniCPM-o image-to-text+speech path now passes for one 448×448
|
||||
synthetic red-square image under an explicit 0.61 thinker GPU budget. The
|
||||
previous 0.58 budget failed the 2,048-token KV admission gate; the successful
|
||||
run still force-killed stage 0 during shutdown and reported a shared-memory
|
||||
cleanup warning. Broader image quality and clean lifecycle remain open.
|
||||
|
||||
## Architecture and prerequisites
|
||||
|
||||
Keep Omni's PipelineConfig, StageRuntime, StageClient, orchestration and admission
|
||||
@@ -82,7 +88,7 @@ capability and lifecycle implementation; a state-handle type alone is insufficie
|
||||
| M1 | S25 Spark | Device-local Omni-compatible controller; embedding, prefill, continuous decode, ring/full KV, head and sampler; resident state | Reference checks, at least 128 output tokens, 512-window/1024-bucket transitions, cancellation, resident memory and sustained profile |
|
||||
| M2 | Qwen3-TTS | Resolve desktop playback and shutdown findings and CPU compatibility; then mobile talker/predictor plus GPU vocoder | Complete PCM and tail, history/chunk correctness, interruption/recovery, quality checks, no post-startup underruns and RTF below 1 in the declared workload |
|
||||
| M3 | AMD stages | Connect compatible real encoders or other useful coarse stages through current Omni workers; start from the existing 890M vision evidence | Actual node/device placement, numerical and task quality, complete downstream output, shared-memory and handoff costs; reject unhelpful splits |
|
||||
| M4a | MiniCPM-o 4.5 | Three-stage desktop text+WAV produced coherent text and nonzero WAV on RTX 5090 Laptop under WSL and native Windows, and WSL CPU, using separate constrained plans. The CPU plan needed vLLM 0.28/0.29 processor compatibility and used swap; 20 serial CPU requests measured p50 21.86 s/p95 22.97 s. Native Windows needed longer startup limits; 20 serial requests measured p50 18.38 s/p95 18.73 s with host RAM/pagefile pressure. Add encoder and persistent-state reference before mobile artifacts | Speech intelligibility/alignment, loading-peak admission, concurrency, separate text, image and audio-understanding suites, then combined streaming and interruption |
|
||||
| M4a | MiniCPM-o 4.5 | Three-stage desktop text+WAV produced coherent text and nonzero WAV on RTX 5090 Laptop under WSL and native Windows, and WSL CPU, using separate constrained plans. One WSL RTX synthetic image also passed through vision encoding to correct text and nonzero WAV after increasing the thinker GPU budget from 0.58 to 0.61; shutdown cleanup remains open. The CPU plan needed vLLM 0.28/0.29 processor compatibility and used swap; 20 serial CPU requests measured p50 21.86 s/p95 22.97 s. Native Windows needed longer startup limits; 20 serial requests measured p50 18.38 s/p95 18.73 s with host RAM/pagefile pressure. Add other encoders and persistent-state reference before mobile artifacts | Speech intelligibility/alignment, loading-peak admission, concurrency, broad image and audio-understanding suites, shutdown lifecycle, then combined streaming and interruption |
|
||||
| M4b | InternVLA-A1 | Real weights, observation preprocessing, shared prefix state, iterative action head and versioned action metadata; strict-load synthetic policy forwards now pass on WSL and native Windows, each on CPU and RTX 5090 Laptop | Reference action agreement on real observations, units/order/timestamps, observation age, stale-input handling and application-specific deadline reporting |
|
||||
| Independent | Qwen3.8-27B | Public Omni pipeline binding; text, then image/video and longer contexts; same-model CPU/iGPU artifact and explicit offload evaluation | Modality-specific quality, actual loading/runtime memory and measured performance; each precision/context receives its own qualification |
|
||||
|
||||
|
||||
Reference in new issue
Block a user