Audit all mobile and PC model pairings with archived evidence
This commit is contained in:
tzhouam committed
1 parent
1887d8294b
commit
ab2f2a3d65
48 files changed
+49116
-18
No files matched your search
@@ -0,0 +1,2 @@
|
||||
# Preserve raw experiment bytes for SHA256 verification, including original whitespace.
|
||||
evidence/** -text -whitespace
|
||||
@@ -0,0 +1,572 @@
|
||||
# Mobile / PC × model qualification audit
|
||||
|
||||
2026-09-22: **60 named pairings checked**, not 60 successful executions.
|
||||
|
||||
PASS = specified full workload passed; PARTIAL = functional output with acceptance gaps; COMPONENT = one model component only; PROFILE_ONLY = component timing without quality; FAILED = attempted execution/startup failed; REJECTED = specific artifact refused; BLOCKED = a named prerequisite is missing. Historical evidence is marked in each finding.
|
||||
|
||||
| Device / execution configuration | Spark-X2.5 | Qwen3-TTS 0.6B CustomVoice | Qwen3.8-27B | MiniCPM-o 4.5 | InternVLA-A1 |
|
||||
|---|---|---|---|---|---|
|
||||
| PC HX370: CPU / WSL | [PASS](#pc_cpu_wsl-spark) | [FAILED](#pc_cpu_wsl-tts) | [BLOCKED](#pc_cpu_wsl-qwen27) | [BLOCKED](#pc_cpu_wsl-minicpm) | [BLOCKED](#pc_cpu_wsl-vla) |
|
||||
| PC HX370: CPU / native Windows | [BLOCKED](#pc_cpu_windows-spark) | [BLOCKED](#pc_cpu_windows-tts) | [BLOCKED](#pc_cpu_windows-qwen27) | [BLOCKED](#pc_cpu_windows-minicpm) | [BLOCKED](#pc_cpu_windows-vla) |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | [PASS](#pc_cuda_wsl-spark) | [PARTIAL](#pc_cuda_wsl-tts) | [FAILED](#pc_cuda_wsl-qwen27) | [BLOCKED](#pc_cuda_wsl-minicpm) | [BLOCKED](#pc_cuda_wsl-vla) |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | [PASS](#pc_cuda_windows-spark) | [PARTIAL](#pc_cuda_windows-tts) | [FAILED](#pc_cuda_windows-qwen27) | [BLOCKED](#pc_cuda_windows-minicpm) | [BLOCKED](#pc_cuda_windows-vla) |
|
||||
| PC HX370 + Radeon 890M / Windows worker | [BLOCKED](#pc_igpu-spark) | [BLOCKED](#pc_igpu-tts) | [COMPONENT](#pc_igpu-qwen27) | [BLOCKED](#pc_igpu-minicpm) | [BLOCKED](#pc_igpu-vla) |
|
||||
| PC HX370 + AMD NPU / Windows worker | [BLOCKED](#pc_npu-spark) | [BLOCKED](#pc_npu-tts) | [REJECTED](#pc_npu-qwen27) | [BLOCKED](#pc_npu-minicpm) | [BLOCKED](#pc_npu-vla) |
|
||||
| PC CPU+iGPU+NPU +/- discrete GPU, joint model execution | [BLOCKED](#pc_joint-spark) | [BLOCKED](#pc_joint-tts) | [BLOCKED](#pc_joint-qwen27) | [BLOCKED](#pc_joint-minicpm) | [BLOCKED](#pc_joint-vla) |
|
||||
| PC Snapdragon X Elite CRD / AI Hub | [BLOCKED](#pc_xelite-spark) | [PROFILE_ONLY](#pc_xelite-tts) | [BLOCKED](#pc_xelite-qwen27) | [BLOCKED](#pc_xelite-minicpm) | [BLOCKED](#pc_xelite-vla) |
|
||||
| Mobile Galaxy S25 / Snapdragon 8 Elite for Galaxy | [COMPONENT](#mobile_s25-spark) | [COMPONENT](#mobile_s25-tts) | [BLOCKED](#mobile_s25-qwen27) | [COMPONENT](#mobile_s25-minicpm) | [BLOCKED](#mobile_s25-vla) |
|
||||
| Mobile Galaxy S24 / Snapdragon 8 Gen 3 | [BLOCKED](#mobile_s24-spark) | [COMPONENT](#mobile_s24-tts) | [BLOCKED](#mobile_s24-qwen27) | [COMPONENT](#mobile_s24-minicpm) | [BLOCKED](#mobile_s24-vla) |
|
||||
| Embedded SA8775P ADP / AI Hub | [BLOCKED](#embedded_sa8775p-spark) | [PROFILE_ONLY](#embedded_sa8775p-tts) | [BLOCKED](#embedded_sa8775p-qwen27) | [BLOCKED](#embedded_sa8775p-minicpm) | [BLOCKED](#embedded_sa8775p-vla) |
|
||||
| Embedded RB3 Gen 2 / QCS6490 / AI Hub | [BLOCKED](#embedded_rb3-spark) | [PROFILE_ONLY](#embedded_rb3-tts) | [BLOCKED](#embedded_rb3-qwen27) | [BLOCKED](#embedded_rb3-minicpm) | [BLOCKED](#embedded_rb3-vla) |
|
||||
|
||||
## Scope and reproduction
|
||||
|
||||
PC rows are execution configurations on the HX370/890M/AMD NPU/RTX 5090 Laptop machine. X Elite is a separate Windows ARM device accessed through AI Hub. The two embedded kits are not phones. Other SKUs are outside this finite audit, not implied supported.
|
||||
|
||||
The checked roots contain MiniCPM5-2B, which must not substitute for MiniCPM-o 4.5. Missing complete MiniCPM-o and InternVLA artifacts block those local runs. ADB reports no attached phone; Hub component jobs cannot prove a device-local controller or sustained co-residency.
|
||||
|
||||
The CPU venv is vLLM 0.28; the CUDA/native-Windows venvs are 0.29. The CPU TTS tuple failure is specific to the mixed-version environment. Preserve failures instead of describing these as universal model limitations.
|
||||
|
||||
Raw selected reports are in `evidence/`. `matrix.json` retains every source path/hash, cell finding and next step. Fresh probe scripts are in `repro/`; see their `--help` and the workspace experiment README for exact environments. No model weights or API credentials are included.
|
||||
|
||||
## Per-pair results
|
||||
|
||||
<a id="pc_cpu_wsl-spark"></a>
|
||||
### PC HX370: CPU / WSL × Spark-X2.5
|
||||
|
||||
**PASS — P**. Fresh Spark 1.7B INT8 acceptance: 12 prompts x 128 tokens; clean cancellation. Current Omni branch with the existing vLLM 0.28 CPU wheel; version mismatch warning retained, not a claim about a 0.29 CPU wheel.
|
||||
|
||||
Evidence: [spark-cpu](evidence/spark-cpu.json), [spark-cpu-log](evidence/spark-cpu-log.log).
|
||||
|
||||
Next: Qualify a matching vLLM 0.29 CPU build and token parity separately.
|
||||
|
||||
<a id="pc_cpu_wsl-tts"></a>
|
||||
### PC HX370: CPU / WSL × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**FAILED — execution**. Fresh real-weight two-frame probe loaded both stages but generation failed in sanitize_min_tokens_stop_ids: expected four tuple fields, got three. Existing CPU wheel is vLLM 0.28 versus Omni 0.29. No audio pass.
|
||||
|
||||
Evidence: [tts-cpu](evidence/tts-cpu.json), [tts-cpu-log](evidence/tts-cpu-log.log).
|
||||
|
||||
Next: Use a matching CPU runtime or explicitly validated compatibility fix; repeat real audio, then full streaming gates.
|
||||
|
||||
<a id="pc_cpu_wsl-qwen27"></a>
|
||||
### PC HX370: CPU / WSL × Qwen3.8-27B
|
||||
|
||||
**BLOCKED — none**. No compatible CPU artifact/deployment for this model was found in the checked model roots. Available Qwen 27B FP8/NVFP4 CUDA artifacts do not establish a CPU route.
|
||||
|
||||
Evidence: [host-wsl](evidence/host-wsl.json).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cpu_wsl-minicpm"></a>
|
||||
### PC HX370: CPU / WSL × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run.
|
||||
|
||||
Evidence: [host-wsl](evidence/host-wsl.json), [host-windows](evidence/host-windows.json).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cpu_wsl-vla"></a>
|
||||
### PC HX370: CPU / WSL × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run.
|
||||
|
||||
Evidence: [host-wsl](evidence/host-wsl.json), [host-windows](evidence/host-windows.json).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cpu_windows-spark"></a>
|
||||
### PC HX370: CPU / native Windows × Spark-X2.5
|
||||
|
||||
**BLOCKED — none**. The installed native Windows vLLM extension lacks cpu_attn_reshape_and_cache and cpu_attn_get_scheduler_metadata. The separate ORT CPU graph worker is not a whole-model text/TTS adapter; a qualified CPU build/binding is missing.
|
||||
|
||||
Evidence: [host-windows](evidence/host-windows.json), [windows-cpu-kernels](evidence/windows-cpu-kernels.json), [backend-factory](evidence/backend-factory.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cpu_windows-tts"></a>
|
||||
### PC HX370: CPU / native Windows × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**BLOCKED — none**. The installed native Windows vLLM extension lacks cpu_attn_reshape_and_cache and cpu_attn_get_scheduler_metadata. The separate ORT CPU graph worker is not a whole-model text/TTS adapter; a qualified CPU build/binding is missing.
|
||||
|
||||
Evidence: [host-windows](evidence/host-windows.json), [windows-cpu-kernels](evidence/windows-cpu-kernels.json), [backend-factory](evidence/backend-factory.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cpu_windows-qwen27"></a>
|
||||
### PC HX370: CPU / native Windows × Qwen3.8-27B
|
||||
|
||||
**BLOCKED — none**. The installed native Windows vLLM extension lacks cpu_attn_reshape_and_cache and cpu_attn_get_scheduler_metadata. The separate ORT CPU graph worker is not a whole-model text/TTS adapter; a qualified CPU build/binding is missing.
|
||||
|
||||
Evidence: [host-windows](evidence/host-windows.json), [windows-cpu-kernels](evidence/windows-cpu-kernels.json), [backend-factory](evidence/backend-factory.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cpu_windows-minicpm"></a>
|
||||
### PC HX370: CPU / native Windows × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run.
|
||||
|
||||
Evidence: [host-wsl](evidence/host-wsl.json), [host-windows](evidence/host-windows.json).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cpu_windows-vla"></a>
|
||||
### PC HX370: CPU / native Windows × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run.
|
||||
|
||||
Evidence: [host-wsl](evidence/host-wsl.json), [host-windows](evidence/host-windows.json).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cuda_wsl-spark"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / WSL × Spark-X2.5
|
||||
|
||||
**PASS — P**. Spark 4B BF16: 11 M0 tests passed on this branch for text, state, cancellation and memory; scoped to the recorded workload.
|
||||
|
||||
Evidence: [spark-cuda-wsl](evidence/spark-cuda-wsl.log).
|
||||
|
||||
Next: Broaden only with a named workload/device/precision qualification.
|
||||
|
||||
<a id="pc_cuda_wsl-tts"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / WSL × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**PARTIAL — P-limited**. Real-weight two-stage audio completes. One measured utterance has playback stalls; speech quality, long streaming, interruption/recovery and thermal gates remain open.
|
||||
|
||||
Evidence: [tts-cuda-wsl](evidence/tts-cuda-wsl.json).
|
||||
|
||||
Next: Resolve and measure playback stalls; run quality, interruption and thermal tests.
|
||||
|
||||
<a id="pc_cuda_wsl-qwen27"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / WSL × Qwen3.8-27B
|
||||
|
||||
**FAILED — startup**. Fresh public Omni startup rejects the real NVFP4 checkpoint: no registered Omni pipeline. Historical bare-vLLM short-text success does not qualify this integration or multimodal inference.
|
||||
|
||||
Evidence: [qwen-wsl](evidence/qwen-wsl.json), [qwen-text-historical](evidence/qwen-text-historical.json).
|
||||
|
||||
Next: Add/validate an Omni pipeline binding; rerun text, image, context and artifact-quality tests.
|
||||
|
||||
<a id="pc_cuda_wsl-minicpm"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / WSL × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run.
|
||||
|
||||
Evidence: [host-wsl](evidence/host-wsl.json), [host-windows](evidence/host-windows.json).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cuda_wsl-vla"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / WSL × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run.
|
||||
|
||||
Evidence: [host-wsl](evidence/host-wsl.json), [host-windows](evidence/host-windows.json).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cuda_windows-spark"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / Windows × Spark-X2.5
|
||||
|
||||
**PASS — P**. Spark 4B BF16: 11 M0 tests passed on this branch for text, state, cancellation and memory; scoped to the recorded workload.
|
||||
|
||||
Evidence: [spark-cuda-windows](evidence/spark-cuda-windows.log).
|
||||
|
||||
Next: Broaden only with a named workload/device/precision qualification.
|
||||
|
||||
<a id="pc_cuda_windows-tts"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / Windows × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**PARTIAL — P-limited**. Real-weight two-stage audio completes. One measured utterance has playback stalls; speech quality, long streaming, interruption/recovery and thermal gates remain open.
|
||||
|
||||
Evidence: [tts-cuda-windows](evidence/tts-cuda-windows.json).
|
||||
|
||||
Next: Resolve and measure playback stalls; run quality, interruption and thermal tests.
|
||||
|
||||
<a id="pc_cuda_windows-qwen27"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / Windows × Qwen3.8-27B
|
||||
|
||||
**FAILED — startup**. Fresh public Omni startup rejects the real NVFP4 checkpoint: no registered Omni pipeline. Historical bare-vLLM short-text success does not qualify this integration or multimodal inference.
|
||||
|
||||
Evidence: [qwen-windows](evidence/qwen-windows.json), [qwen-text-historical](evidence/qwen-text-historical.json).
|
||||
|
||||
Next: Add/validate an Omni pipeline binding; rerun text, image, context and artifact-quality tests.
|
||||
|
||||
<a id="pc_cuda_windows-minicpm"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / Windows × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run.
|
||||
|
||||
Evidence: [host-wsl](evidence/host-wsl.json), [host-windows](evidence/host-windows.json).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_cuda_windows-vla"></a>
|
||||
### PC HX370 + RTX 5090 Laptop / Windows × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run.
|
||||
|
||||
Evidence: [host-wsl](evidence/host-wsl.json), [host-windows](evidence/host-windows.json).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_igpu-spark"></a>
|
||||
### PC HX370 + Radeon 890M / Windows worker × Spark-X2.5
|
||||
|
||||
**BLOCKED — none**. No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support.
|
||||
|
||||
Evidence: [backend-dml](evidence/backend-dml.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_igpu-tts"></a>
|
||||
### PC HX370 + Radeon 890M / Windows worker × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**BLOCKED — none**. No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support.
|
||||
|
||||
Evidence: [backend-dml](evidence/backend-dml.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_igpu-qwen27"></a>
|
||||
### PC HX370 + Radeon 890M / Windows worker × Qwen3.8-27B
|
||||
|
||||
**COMPONENT — C-historical**. FP32 448x448 vision tower: DirectML 85/113 nodes; normalized max error 9.095e-06 versus torch CPU. Historical component path, not full 27B or current graph-stage image-to-text integration.
|
||||
|
||||
Evidence: [qwen-vision-dml](evidence/qwen-vision-dml.json).
|
||||
|
||||
Next: Recover the hashed tower bundle and qualify its current Omni graph handoff with the language stage.
|
||||
|
||||
<a id="pc_igpu-minicpm"></a>
|
||||
### PC HX370 + Radeon 890M / Windows worker × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support.
|
||||
|
||||
Evidence: [backend-dml](evidence/backend-dml.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_igpu-vla"></a>
|
||||
### PC HX370 + Radeon 890M / Windows worker × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support.
|
||||
|
||||
Evidence: [backend-dml](evidence/backend-dml.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_npu-spark"></a>
|
||||
### PC HX370 + AMD NPU / Windows worker × Spark-X2.5
|
||||
|
||||
**BLOCKED — none**. No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support.
|
||||
|
||||
Evidence: [backend-npu](evidence/backend-npu.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_npu-tts"></a>
|
||||
### PC HX370 + AMD NPU / Windows worker × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**BLOCKED — none**. No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support.
|
||||
|
||||
Evidence: [backend-npu](evidence/backend-npu.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_npu-qwen27"></a>
|
||||
### PC HX370 + AMD NPU / Windows worker × Qwen3.8-27B
|
||||
|
||||
**REJECTED — artifact**. The tested A16W8 whole-vision-tower export failed NPU placement. This conclusion is specific to that export, not all future Qwen/NPU combinations.
|
||||
|
||||
Evidence: [qwen-vision-npu](evidence/qwen-vision-npu.json).
|
||||
|
||||
Next: Validate a supported export or smaller useful component, including same-checkpoint numerical quality.
|
||||
|
||||
<a id="pc_npu-minicpm"></a>
|
||||
### PC HX370 + AMD NPU / Windows worker × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support.
|
||||
|
||||
Evidence: [backend-npu](evidence/backend-npu.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_npu-vla"></a>
|
||||
### PC HX370 + AMD NPU / Windows worker × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support.
|
||||
|
||||
Evidence: [backend-npu](evidence/backend-npu.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_joint-spark"></a>
|
||||
### PC CPU+iGPU+NPU +/- discrete GPU, joint model execution × Spark-X2.5
|
||||
|
||||
**BLOCKED — none**. No qualifying same-model multi-accelerator deployment/artifact plan or overlap/recovery evidence. Individual routes are not a joint pipeline.
|
||||
|
||||
Evidence: [backend-dml](evidence/backend-dml.json), [backend-npu](evidence/backend-npu.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_joint-tts"></a>
|
||||
### PC CPU+iGPU+NPU +/- discrete GPU, joint model execution × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**BLOCKED — none**. No qualifying same-model multi-accelerator deployment/artifact plan or overlap/recovery evidence. Individual routes are not a joint pipeline.
|
||||
|
||||
Evidence: [backend-dml](evidence/backend-dml.json), [backend-npu](evidence/backend-npu.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_joint-qwen27"></a>
|
||||
### PC CPU+iGPU+NPU +/- discrete GPU, joint model execution × Qwen3.8-27B
|
||||
|
||||
**BLOCKED — none**. No qualifying same-model multi-accelerator deployment/artifact plan or overlap/recovery evidence. Individual routes are not a joint pipeline.
|
||||
|
||||
Evidence: [backend-dml](evidence/backend-dml.json), [backend-npu](evidence/backend-npu.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_joint-minicpm"></a>
|
||||
### PC CPU+iGPU+NPU +/- discrete GPU, joint model execution × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No qualifying same-model multi-accelerator deployment/artifact plan or overlap/recovery evidence. Individual routes are not a joint pipeline.
|
||||
|
||||
Evidence: [backend-dml](evidence/backend-dml.json), [backend-npu](evidence/backend-npu.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_joint-vla"></a>
|
||||
### PC CPU+iGPU+NPU +/- discrete GPU, joint model execution × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No qualifying same-model multi-accelerator deployment/artifact plan or overlap/recovery evidence. Individual routes are not a joint pipeline.
|
||||
|
||||
Evidence: [backend-dml](evidence/backend-dml.json), [backend-npu](evidence/backend-npu.json), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_xelite-spark"></a>
|
||||
### PC Snapdragon X Elite CRD / AI Hub × Spark-X2.5
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_xelite-tts"></a>
|
||||
### PC Snapdragon X Elite CRD / AI Hub × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**PROFILE_ONLY — C-timing-only**. Historical QNN talker profile re-fetched successfully; no numerical/full-pipeline gate. Historical TFLite vocoder target was refused on this Windows ARM device.
|
||||
|
||||
Evidence: [tts-xelite](evidence/tts-xelite.json), [tts-xelite-vocoder-rejected](evidence/tts-xelite-vocoder-rejected.json), [tts-other-devices](evidence/tts-other-devices.json), [tts-other-device-code](evidence/tts-other-device-code.py).
|
||||
|
||||
Next: Validate target-specific numerical quality, complete local execution, memory and sustained behavior.
|
||||
|
||||
<a id="pc_xelite-qwen27"></a>
|
||||
### PC Snapdragon X Elite CRD / AI Hub × Qwen3.8-27B
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_xelite-minicpm"></a>
|
||||
### PC Snapdragon X Elite CRD / AI Hub × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="pc_xelite-vla"></a>
|
||||
### PC Snapdragon X Elite CRD / AI Hub × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="mobile_s25-spark"></a>
|
||||
### Mobile Galaxy S25 / Snapdragon 8 Elite for Galaxy × Spark-X2.5
|
||||
|
||||
**COMPONENT — C**. Fresh 2026-09-22 W8A16 full-attention component inference/profile passed; full1024 denotes one cache bucket. No full prefill/decode/KV/ring/sampling loop.
|
||||
|
||||
Evidence: [spark-s25](evidence/spark-s25.json), [spark-s25-parity](evidence/spark-s25-parity.json).
|
||||
|
||||
Next: Implement and validate local 128-token generation, 512/1024 transitions, cancellation and resident memory.
|
||||
|
||||
<a id="mobile_s25-tts"></a>
|
||||
### Mobile Galaxy S25 / Snapdragon 8 Elite for Galaxy × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**COMPONENT — C**. Fresh FP16 predictor inference passed; its one output exactly matches the same compiled artifact/dataset reference. Historical decoder parity exists. No complete co-resident NPU/GPU stream.
|
||||
|
||||
Evidence: [tts-s25](evidence/tts-s25.json), [hub-repeatability](evidence/hub-repeatability.json), [tts-mobile-parity](evidence/tts-mobile-parity.json).
|
||||
|
||||
Next: Validate the device-local talker/predictor/vocoder pipeline, output quality, recovery and 30-minute thermal behavior.
|
||||
|
||||
<a id="mobile_s25-qwen27"></a>
|
||||
### Mobile Galaxy S25 / Snapdragon 8 Elite for Galaxy × Qwen3.8-27B
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="mobile_s25-minicpm"></a>
|
||||
### Mobile Galaxy S25 / Snapdragon 8 Elite for Galaxy × MiniCPM-o 4.5
|
||||
|
||||
**COMPONENT — C**. Fresh FP16 speech-head inference passed; all 41 outputs exactly match the same artifact/dataset reference. Historical FP16 numerical parity passed; tested W8A16 top-1 parity failed. Thinker/encoders/vocoder are not qualified together.
|
||||
|
||||
Evidence: [minicpm-s25](evidence/minicpm-s25.json), [hub-repeatability](evidence/hub-repeatability.json), [minicpm-mobile-parity](evidence/minicpm-mobile-parity.json).
|
||||
|
||||
Next: Supply a qualified complete mobile checkpoint/export and validate the full multimodal/stateful pipeline.
|
||||
|
||||
<a id="mobile_s25-vla"></a>
|
||||
### Mobile Galaxy S25 / Snapdragon 8 Elite for Galaxy × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="mobile_s24-spark"></a>
|
||||
### Mobile Galaxy S24 / Snapdragon 8 Gen 3 × Spark-X2.5
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="mobile_s24-tts"></a>
|
||||
### Mobile Galaxy S24 / Snapdragon 8 Gen 3 × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**COMPONENT — C-historical**. Historical FP16 decoder parity has SNR 39.13 dB. Tested calibrated INT8 decoder parity failed with negative SNR. Neither result is a full local TTS stream.
|
||||
|
||||
Evidence: [tts-mobile-parity](evidence/tts-mobile-parity.json).
|
||||
|
||||
Next: Validate the complete local pipeline and a quality-passing artifact set.
|
||||
|
||||
<a id="mobile_s24-qwen27"></a>
|
||||
### Mobile Galaxy S24 / Snapdragon 8 Gen 3 × Qwen3.8-27B
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="mobile_s24-minicpm"></a>
|
||||
### Mobile Galaxy S24 / Snapdragon 8 Gen 3 × MiniCPM-o 4.5
|
||||
|
||||
**COMPONENT — C-historical**. Historical FP16 speech-head job re-fetched successfully; SNR 49.22 dB/top-1 match in the recorded parity test. W8A16 top-1 failed. Not complete MiniCPM-o.
|
||||
|
||||
Evidence: [minicpm-s24](evidence/minicpm-s24.json), [minicpm-mobile-parity](evidence/minicpm-mobile-parity.json).
|
||||
|
||||
Next: Qualify thinker/encoders/vocoder and stateful local integration.
|
||||
|
||||
<a id="mobile_s24-vla"></a>
|
||||
### Mobile Galaxy S24 / Snapdragon 8 Gen 3 × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="embedded_sa8775p-spark"></a>
|
||||
### Embedded SA8775P ADP / AI Hub × Spark-X2.5
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="embedded_sa8775p-tts"></a>
|
||||
### Embedded SA8775P ADP / AI Hub × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**PROFILE_ONLY — C-timing-only**. Historical NPU talker and GPU vocoder profiles re-fetched successfully; no complete pipeline, parity or co-residency qualification.
|
||||
|
||||
Evidence: [tts-sa8775p-npu](evidence/tts-sa8775p-npu.json), [tts-sa8775p-gpu](evidence/tts-sa8775p-gpu.json), [tts-other-devices](evidence/tts-other-devices.json), [tts-other-device-code](evidence/tts-other-device-code.py).
|
||||
|
||||
Next: Validate target-specific numerical quality, complete local execution, memory and sustained behavior.
|
||||
|
||||
<a id="embedded_sa8775p-qwen27"></a>
|
||||
### Embedded SA8775P ADP / AI Hub × Qwen3.8-27B
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="embedded_sa8775p-minicpm"></a>
|
||||
### Embedded SA8775P ADP / AI Hub × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="embedded_sa8775p-vla"></a>
|
||||
### Embedded SA8775P ADP / AI Hub × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="embedded_rb3-spark"></a>
|
||||
### Embedded RB3 Gen 2 / QCS6490 / AI Hub × Spark-X2.5
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="embedded_rb3-tts"></a>
|
||||
### Embedded RB3 Gen 2 / QCS6490 / AI Hub × Qwen3-TTS 0.6B CustomVoice
|
||||
|
||||
**PROFILE_ONLY — C-timing-only**. Historical GPU vocoder profile re-fetched successfully. The attempted NPU talker build was rejected for floating-point inputs. No full model pipeline.
|
||||
|
||||
Evidence: [tts-rb3-gpu](evidence/tts-rb3-gpu.json), [tts-rb3-talker-rejected](evidence/tts-rb3-talker-rejected.json), [tts-other-devices](evidence/tts-other-devices.json), [tts-other-device-code](evidence/tts-other-device-code.py).
|
||||
|
||||
Next: Validate target-specific numerical quality, complete local execution, memory and sustained behavior.
|
||||
|
||||
<a id="embedded_rb3-qwen27"></a>
|
||||
### Embedded RB3 Gen 2 / QCS6490 / AI Hub × Qwen3.8-27B
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="embedded_rb3-minicpm"></a>
|
||||
### Embedded RB3 Gen 2 / QCS6490 / AI Hub × MiniCPM-o 4.5
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
|
||||
<a id="embedded_rb3-vla"></a>
|
||||
### Embedded RB3 Gen 2 / QCS6490 / AI Hub × InternVLA-A1
|
||||
|
||||
**BLOCKED — none**. No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device.
|
||||
|
||||
Evidence: [hub-inventory](evidence/hub-inventory.json), [host-wsl](evidence/host-wsl.json), [backend-factory](evidence/backend-factory.py), [graph-contract](evidence/graph-contract.py).
|
||||
|
||||
Next: Supply and validate the missing matching artifact/backend, then run the complete specified workload.
|
||||
+183
@@ -0,0 +1,183 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"evidence": "B",
|
||||
"platform": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.43",
|
||||
"python": "/home/zhout/project/edge_infer/vllm-omni-edge/../.venvs/omni-cuda-029/bin/python",
|
||||
"workload": {
|
||||
"repeats": 20,
|
||||
"concurrency": 1
|
||||
},
|
||||
"artifact": "/home/zhout/project/edge_infer/analysis/experiments/omni_abstraction_20260922/artifacts/cpu_add/manifest.json",
|
||||
"samples_s": [
|
||||
0.0040255889762192965,
|
||||
0.0033558340510353446,
|
||||
0.0017729569808579981,
|
||||
0.0013527399860322475,
|
||||
0.002440641983412206,
|
||||
0.0013856349978595972,
|
||||
0.0013715020031668246,
|
||||
0.005610267980955541,
|
||||
0.0013573350152000785,
|
||||
0.0012883200543001294,
|
||||
0.0012204480008222163,
|
||||
0.0011961509590037167,
|
||||
0.0017715750145725906,
|
||||
0.0011716459994204342,
|
||||
0.0011642409954220057,
|
||||
0.001186243025586009,
|
||||
0.0011836979538202286,
|
||||
0.0013943579979240894,
|
||||
0.0030256269965320826,
|
||||
0.004145854036323726
|
||||
],
|
||||
"started_unix": 1790047607.8205295,
|
||||
"startup_s": 1.99209214001894,
|
||||
"execution_plan": {
|
||||
"backend": "external.graph.v1",
|
||||
"stage_id": 0,
|
||||
"worker_generation": "c2f7ca31da7e4afcb9751363336eead1",
|
||||
"artifact_sha256": "4314a53e6394d9be101ec3b05b1d7baeea58dbd314fe8d5d2c12d94a6017b051",
|
||||
"component": "cpu_add",
|
||||
"route": {
|
||||
"name": "amd-dml",
|
||||
"interpreter": "/mnt/c/Users/zhout/dml-ep/Scripts/python.exe",
|
||||
"worker": "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/edge/local/external/worker_ort.py",
|
||||
"ep": "dml",
|
||||
"is_windows": true,
|
||||
"available": true,
|
||||
"reason": "",
|
||||
"extra": null
|
||||
},
|
||||
"reserved_bytes": {
|
||||
"machine:ram": 536870912
|
||||
},
|
||||
"max_io_bytes": 8388608,
|
||||
"placement": {
|
||||
"ep": "dml",
|
||||
"graph_path": "/home/zhout/project/edge_infer/analysis/experiments/omni_abstraction_20260922/artifacts/cpu_add/graph.onnx",
|
||||
"session_providers": [
|
||||
"DmlExecutionProvider",
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"available_providers": [
|
||||
"DmlExecutionProvider",
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"node_counts": {
|
||||
"DmlExecutionProvider": 1
|
||||
},
|
||||
"total_nodes": 1,
|
||||
"target_nodes": 1,
|
||||
"fraction_on_target": 1.0,
|
||||
"session_create_s": 0.3837424000375904,
|
||||
"warmup_s": 0.0069688999792560935,
|
||||
"rss_bytes": 145149952,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "x",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
2,
|
||||
3
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "y",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
2,
|
||||
3
|
||||
]
|
||||
}
|
||||
],
|
||||
"profile_path": "\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\analysis\\experiments\\omni_abstraction_20260922\\graph-dml-crossdomain.placement_2026-09-22_11-26-49.json",
|
||||
"ep_directory": "",
|
||||
"requested_provider_missing": "",
|
||||
"fallback_note": "",
|
||||
"onnxruntime": "1.24.4",
|
||||
"note": ""
|
||||
},
|
||||
"worker": {
|
||||
"executable": "C:\\Users\\zhout\\dml-ep\\Scripts\\python.exe",
|
||||
"python": "3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]",
|
||||
"platform": "win32",
|
||||
"pid": 33568,
|
||||
"onnxruntime": "1.24.4",
|
||||
"numpy": "2.5.3",
|
||||
"available_providers": [
|
||||
"DmlExecutionProvider",
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"rss_bytes": 54603776,
|
||||
"process_identity": {
|
||||
"pid": 33568,
|
||||
"created_filetime": 134345212080641171
|
||||
}
|
||||
},
|
||||
"evidence": "B",
|
||||
"stateful": false
|
||||
},
|
||||
"worker_stats": {
|
||||
"runs": 20,
|
||||
"total_run_s": 0.011883700266480446,
|
||||
"rss_bytes": 145235968,
|
||||
"peak_rss_bytes": 145256448,
|
||||
"report": {
|
||||
"ep": "dml",
|
||||
"onnxruntime": "1.24.4",
|
||||
"available_providers": [
|
||||
"DmlExecutionProvider",
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"session_create_s": 0.3837424000375904,
|
||||
"session_providers": [
|
||||
"DmlExecutionProvider",
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"inputs": [
|
||||
{
|
||||
"name": "x",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
2,
|
||||
3
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "y",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
2,
|
||||
3
|
||||
]
|
||||
}
|
||||
],
|
||||
"warmup_s": 0.0069688999792560935,
|
||||
"profile_path": "\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\analysis\\experiments\\omni_abstraction_20260922\\graph-dml-crossdomain.placement_2026-09-22_11-26-49.json",
|
||||
"node_counts": {
|
||||
"DmlExecutionProvider": 1
|
||||
},
|
||||
"total_nodes": 1,
|
||||
"target_nodes": 1,
|
||||
"fraction_on_target": 1.0,
|
||||
"rss_bytes": 145149952
|
||||
}
|
||||
},
|
||||
"finite": true,
|
||||
"status": "passed",
|
||||
"ledger_after_shutdown": {
|
||||
"capacities": {
|
||||
"machine:ram": 1073741824
|
||||
},
|
||||
"reserved": {
|
||||
"machine:ram": 0
|
||||
},
|
||||
"owners": [],
|
||||
"quarantined": []
|
||||
},
|
||||
"ended_unix": 1790047610.5372653
|
||||
}
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Factories for non-vLLM stages owned by the existing StageRuntime."""
|
||||
|
||||
|
||||
def create_graph_client(metadata, config, ledger, reservation):
|
||||
from .graph import GraphStageClient
|
||||
|
||||
if config.get("backend", {}).get("name") != "external.graph.v1":
|
||||
raise ValueError("graph stage requires backend.name=external.graph.v1")
|
||||
return GraphStageClient(metadata, config["backend"], ledger, reservation)
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"evidence": "B",
|
||||
"platform": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.43",
|
||||
"python": "/home/zhout/project/edge_infer/vllm-omni-edge/../.venvs/omni-cuda-029/bin/python",
|
||||
"workload": {
|
||||
"repeats": 20,
|
||||
"concurrency": 1
|
||||
},
|
||||
"artifact": "/home/zhout/project/edge_infer/analysis/experiments/omni_abstraction_20260922/artifacts/amd_npu_a16w8/manifest.json",
|
||||
"samples_s": [
|
||||
0.014123815984930843,
|
||||
0.0061385720036923885,
|
||||
0.0058239150093868375,
|
||||
0.00862913700984791,
|
||||
0.006411314010620117,
|
||||
0.02813155995681882,
|
||||
0.005825735977850854,
|
||||
0.005439646018203348,
|
||||
0.006521859031636268,
|
||||
0.0055095599964261055,
|
||||
0.004404937033541501,
|
||||
0.006788280967157334,
|
||||
0.005359956994652748,
|
||||
0.0058780250255949795,
|
||||
0.0045300599886104465,
|
||||
0.006785528967157006,
|
||||
0.006678219011519104,
|
||||
0.0063079120009206235,
|
||||
0.0045537170371972024,
|
||||
0.006222707044798881
|
||||
],
|
||||
"started_unix": 1790047130.3935041,
|
||||
"startup_s": 5.256746579951141,
|
||||
"execution_plan": {
|
||||
"backend": "external.graph.v1",
|
||||
"stage_id": 0,
|
||||
"worker_generation": "7b8fc7c330a649f685c5c257778d0d57",
|
||||
"artifact_sha256": "63889910407277b0a557f7c58cfe6a4ce4b3884711961a90f3e9e52105c0f1ed",
|
||||
"component": "amd_npu_a16w8",
|
||||
"route": {
|
||||
"name": "amd-npu",
|
||||
"interpreter": "/mnt/c/Users/zhout/npu-ep/Scripts/python.exe",
|
||||
"worker": "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/edge/local/external/worker_ort.py",
|
||||
"ep": "vitisai",
|
||||
"is_windows": true,
|
||||
"available": true,
|
||||
"reason": "",
|
||||
"extra": null
|
||||
},
|
||||
"reserved_bytes": {
|
||||
"machine:ram": 1073741824
|
||||
},
|
||||
"max_io_bytes": 8388608,
|
||||
"placement": {
|
||||
"ep": "vitisai",
|
||||
"graph_path": "/home/zhout/project/edge_infer/analysis/experiments/omni_abstraction_20260922/artifacts/amd_npu_a16w8/graph.onnx",
|
||||
"session_providers": [
|
||||
"vitisai",
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"available_providers": [
|
||||
"AzureExecutionProvider",
|
||||
"CPUExecutionProvider",
|
||||
"vitisai"
|
||||
],
|
||||
"node_counts": {
|
||||
"CPUExecutionProvider": 2,
|
||||
"vitisai": 1
|
||||
},
|
||||
"total_nodes": 3,
|
||||
"target_nodes": 1,
|
||||
"fraction_on_target": 0.3333333333333333,
|
||||
"session_create_s": 2.7006354000186548,
|
||||
"warmup_s": 0.002900300023611635,
|
||||
"rss_bytes": 283721728,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "x",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
256,
|
||||
1024
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "y",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
256,
|
||||
1024
|
||||
]
|
||||
}
|
||||
],
|
||||
"profile_path": "onnxruntime_profile__2026-09-22_11-18-52_927.json",
|
||||
"ep_directory": "C:\\Program Files\\WindowsApps\\WindowsWorkload.EP.AMD.VitisAI.Framework.1.8_1.8.63.0_x64__8wekyb3d8bbwe\\ExecutionProvider",
|
||||
"requested_provider_missing": "",
|
||||
"fallback_note": "",
|
||||
"onnxruntime": "1.30.0",
|
||||
"note": ""
|
||||
},
|
||||
"worker": {
|
||||
"executable": "C:\\Users\\zhout\\npu-ep\\Scripts\\python.exe",
|
||||
"python": "3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]",
|
||||
"platform": "win32",
|
||||
"pid": 23080,
|
||||
"onnxruntime": "1.30.0",
|
||||
"numpy": "2.5.3",
|
||||
"available_providers": [
|
||||
"AzureExecutionProvider",
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"rss_bytes": 55009280,
|
||||
"process_identity": {
|
||||
"pid": 23080,
|
||||
"created_filetime": 134345207305451697
|
||||
}
|
||||
},
|
||||
"evidence": "B",
|
||||
"stateful": false
|
||||
},
|
||||
"worker_stats": {
|
||||
"runs": 20,
|
||||
"total_run_s": 0.020845199760515243,
|
||||
"rss_bytes": 285020160,
|
||||
"peak_rss_bytes": 288174080,
|
||||
"report": {
|
||||
"ep": "vitisai",
|
||||
"onnxruntime": "1.30.0",
|
||||
"ep_directory": "C:\\Program Files\\WindowsApps\\WindowsWorkload.EP.AMD.VitisAI.Framework.1.8_1.8.63.0_x64__8wekyb3d8bbwe\\ExecutionProvider",
|
||||
"available_providers": [
|
||||
"AzureExecutionProvider",
|
||||
"CPUExecutionProvider",
|
||||
"vitisai"
|
||||
],
|
||||
"session_create_s": 2.7006354000186548,
|
||||
"session_providers": [
|
||||
"vitisai",
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"inputs": [
|
||||
{
|
||||
"name": "x",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
256,
|
||||
1024
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "y",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
256,
|
||||
1024
|
||||
]
|
||||
}
|
||||
],
|
||||
"warmup_s": 0.002900300023611635,
|
||||
"profile_path": "onnxruntime_profile__2026-09-22_11-18-52_927.json",
|
||||
"node_counts": {
|
||||
"CPUExecutionProvider": 2,
|
||||
"vitisai": 1
|
||||
},
|
||||
"total_nodes": 3,
|
||||
"target_nodes": 1,
|
||||
"fraction_on_target": 0.3333333333333333,
|
||||
"rss_bytes": 283721728
|
||||
}
|
||||
},
|
||||
"finite": true,
|
||||
"status": "passed",
|
||||
"ledger_after_shutdown": {
|
||||
"capacities": {
|
||||
"machine:ram": 2147483648
|
||||
},
|
||||
"reserved": {
|
||||
"machine:ram": 0
|
||||
},
|
||||
"owners": [],
|
||||
"quarantined": []
|
||||
},
|
||||
"ended_unix": 1790047136.6063926
|
||||
}
|
||||
+219
@@ -0,0 +1,219 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""One bounded, non-preemptible graph stage using the existing worker.
|
||||
|
||||
The runtime owns startup and capacity. This adapter owns one in-flight call,
|
||||
including its output until acknowledged by the consumer. No token scheduler,
|
||||
provider fallback, or automatic worker replay is introduced here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import dataclasses
|
||||
import math
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
from omni_stage_contracts import ArtifactManifest, BufferRef, StageEvent, StageRequest
|
||||
from vllm_omni.edge.local.external.client import ExternalWorker
|
||||
from vllm_omni.engine.resource_ledger import ResourceUnavailable
|
||||
from vllm_omni.engine.stage_client import StageClientBase
|
||||
from vllm_omni.host.routes import resolve_route
|
||||
from vllm_omni.outputs import OmniRequestOutput
|
||||
|
||||
|
||||
class GraphStageClient(StageClientBase):
|
||||
def __init__(self, metadata, config, ledger, reservation) -> None:
|
||||
for name, value in vars(metadata).items():
|
||||
setattr(self, name, value)
|
||||
self.stage_type = "graph"
|
||||
self._ledger, self._reservation = ledger, reservation
|
||||
self._generation = uuid.uuid4().hex
|
||||
self._config = dict(config)
|
||||
self._worker = None
|
||||
self._closed = False
|
||||
self._active: str | None = None
|
||||
self._task: asyncio.Task | None = None
|
||||
self._output: OmniRequestOutput | None = None
|
||||
self._epoch = 0
|
||||
self._max_io_bytes = int(config.get("max_io_bytes", 8 << 20))
|
||||
if self._max_io_bytes <= 0 or self._max_io_bytes > min(reservation.demands.values()):
|
||||
raise ValueError("graph I/O limit must be positive and fit the stage reservation")
|
||||
try:
|
||||
manifest_path = Path(config["manifest"]).resolve()
|
||||
manifest = ArtifactManifest.read(manifest_path)
|
||||
graph_name = manifest.metadata["graph_file"]
|
||||
examples_name = manifest.metadata["example_inputs_file"]
|
||||
if graph_name not in manifest.files or examples_name not in manifest.files:
|
||||
raise ValueError("graph and placement examples must be hashed artifact payloads")
|
||||
if sum((manifest_path.parent / n).stat().st_size for n in manifest.files) > min(
|
||||
reservation.demands.values()
|
||||
):
|
||||
raise ResourceUnavailable("artifact files exceed stage memory reservation")
|
||||
route = resolve_route(config["route"])
|
||||
self._worker = ExternalWorker(
|
||||
route, start_timeout_s=float(config.get("start_timeout_s", 120)), max_payload_bytes=self._max_io_bytes
|
||||
)
|
||||
self._worker.start()
|
||||
with np.load(manifest_path.parent / examples_name, allow_pickle=False) as example:
|
||||
inputs = {name: example[name] for name in example.files}
|
||||
self._describe(inputs)
|
||||
self.load_report = self._worker.load(
|
||||
manifest_path.parent / graph_name,
|
||||
example_inputs=inputs,
|
||||
ep_dir=config.get("ep_dir"),
|
||||
intra_op_num_threads=config.get("intra_op_num_threads"),
|
||||
profile_prefix=config.get("profile_prefix"),
|
||||
artifact_files=[manifest_path.parent / name for name in manifest.files],
|
||||
)
|
||||
report = self.load_report
|
||||
minimum = float(config.get("min_fraction_on_target", 1.0))
|
||||
if not 0 < minimum <= 1:
|
||||
raise ValueError("placement threshold must be in (0, 1]")
|
||||
if (
|
||||
report.requested_provider_missing
|
||||
or report.fraction_on_target is None
|
||||
or not math.isfinite(report.fraction_on_target)
|
||||
or not 0 < report.fraction_on_target <= 1
|
||||
or report.target_nodes <= 0
|
||||
or report.fraction_on_target < minimum
|
||||
):
|
||||
raise RuntimeError(f"REFUSE_EP_PLACEMENT: {report.summary()}")
|
||||
allowed = set(config.get("allowed_providers", report.session_providers if route.ep == "cpu" else ()))
|
||||
if not allowed or set(report.node_counts) - allowed:
|
||||
raise RuntimeError("REFUSE_EP_PLACEMENT: declare all permitted execution providers")
|
||||
if report.rss_bytes > min(reservation.demands.values()):
|
||||
raise ResourceUnavailable("loaded worker RSS exceeds stage memory reservation")
|
||||
self._input_schema = {name: (value.dtype.str, value.shape) for name, value in inputs.items()}
|
||||
from omni_stage_contracts.types import file_digest
|
||||
|
||||
self.execution_plan = {
|
||||
"backend": "external.graph.v1",
|
||||
"stage_id": self.stage_id,
|
||||
"worker_generation": self._generation,
|
||||
"artifact_sha256": file_digest(manifest_path),
|
||||
"component": manifest.component,
|
||||
"route": route.to_dict(),
|
||||
"reserved_bytes": dict(reservation.demands),
|
||||
"max_io_bytes": self._max_io_bytes,
|
||||
"placement": report.to_dict(),
|
||||
"worker": {k: v for k, v in self._worker.hello.items() if k != "token"},
|
||||
"evidence": "B",
|
||||
"stateful": False,
|
||||
}
|
||||
except BaseException:
|
||||
self.shutdown()
|
||||
raise
|
||||
|
||||
def _describe(self, tensors: dict[str, np.ndarray]) -> tuple[BufferRef, ...]:
|
||||
refs = tuple(
|
||||
BufferRef(
|
||||
name, str(self.stage_id), self._generation, array.dtype.name, tuple(array.shape), int(array.nbytes)
|
||||
)
|
||||
for name, array in tensors.items()
|
||||
)
|
||||
if sum(r.nbytes for r in refs) > self._max_io_bytes:
|
||||
raise ResourceUnavailable("graph payload exceeds admitted I/O bound")
|
||||
return refs
|
||||
|
||||
async def add_request_async(self, request_id: str, prompt: Any, params: Any = None) -> None:
|
||||
self.check_health()
|
||||
if self._active is not None:
|
||||
raise ResourceUnavailable("graph stage has an unacknowledged request; capacity is one")
|
||||
if not isinstance(prompt, dict) or not isinstance(prompt.get("tensors"), dict):
|
||||
raise ValueError("graph prompt must contain a tensors mapping supplied by its model adapter")
|
||||
inputs = {name: np.asarray(value) for name, value in prompt["tensors"].items()}
|
||||
refs = self._describe(inputs)
|
||||
if {name: (value.dtype.str, value.shape) for name, value in inputs.items()} != self._input_schema:
|
||||
raise ValueError("input names/dtypes/shapes differ from the placement-verified artifact bucket")
|
||||
# Copy at submission: application mutation cannot change an admitted call.
|
||||
inputs = {name: value.copy() for name, value in inputs.items()}
|
||||
self._epoch += 1
|
||||
epoch = self._epoch
|
||||
request = StageRequest(request_id, self.stage_id, epoch, self._generation, inputs=refs)
|
||||
self._active = request_id
|
||||
self._task = asyncio.create_task(self._run(request, inputs), name=f"graph-{self.stage_id}-{request_id}")
|
||||
|
||||
async def _run(self, request: StageRequest, inputs: dict[str, np.ndarray]) -> None:
|
||||
try:
|
||||
outputs, timing = await asyncio.to_thread(self._worker.run, inputs)
|
||||
refs = self._describe(outputs)
|
||||
if sum(x.nbytes for x in outputs.values()) + sum(x.nbytes for x in inputs.values()) > self._max_io_bytes:
|
||||
raise ResourceUnavailable("combined graph input/output exceeds admitted I/O bound")
|
||||
event = StageEvent(
|
||||
request.request_id, self.stage_id, request.epoch, 1, "tensor", self._generation, refs, terminal=True
|
||||
)
|
||||
output = OmniRequestOutput(
|
||||
request_id=request.request_id,
|
||||
stage_id=self.stage_id,
|
||||
final_output_type=self.final_output_type or "latent",
|
||||
_custom_output={"tensors": outputs, "stage_event": dataclasses.asdict(event)},
|
||||
metrics={"graph_timing": timing.to_dict()},
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
output = OmniRequestOutput.from_error(request.request_id, str(exc))
|
||||
output.stage_id = self.stage_id
|
||||
if not self._closed and self._epoch == request.epoch:
|
||||
loop = asyncio.get_running_loop()
|
||||
output._stage_release = lambda: loop.call_soon_threadsafe(
|
||||
self.acknowledge, request.request_id, request.epoch, request.worker_generation
|
||||
)
|
||||
self._output = output
|
||||
|
||||
def get_graph_output_nowait(self):
|
||||
# Delivery is separate from release: the slot stays charged until ACK.
|
||||
output, self._output = self._output, None
|
||||
return output
|
||||
|
||||
def acknowledge(self, request_id: str, epoch: int, generation: str) -> None:
|
||||
if (
|
||||
self._active == request_id
|
||||
and epoch == self._epoch
|
||||
and generation == self._generation
|
||||
and (self._task is None or self._task.done())
|
||||
):
|
||||
self._active = None
|
||||
self._task = None
|
||||
|
||||
async def abort_requests_async(self, request_ids: list[str]) -> None:
|
||||
if self._active not in request_ids:
|
||||
return
|
||||
self._epoch += 1
|
||||
self._output = None
|
||||
# A completed call needs no worker kill. Otherwise cancellation retires
|
||||
# this single-session worker; the runtime will report it unavailable.
|
||||
if self._task is not None and not self._task.done():
|
||||
drained = await asyncio.to_thread(self._worker.terminate)
|
||||
self._closed = True
|
||||
done, _ = await asyncio.wait({self._task}, timeout=5)
|
||||
self._ledger.release(self._reservation, drained=drained and bool(done) and not self._task.cancelled())
|
||||
self._active = None
|
||||
|
||||
def check_health(self) -> None:
|
||||
if self._closed:
|
||||
from vllm.v1.engine.exceptions import EngineDeadError
|
||||
|
||||
raise EngineDeadError()
|
||||
|
||||
async def collective_rpc_async(self, method, timeout=None, args=(), kwargs=None):
|
||||
raise NotImplementedError(f"graph backend does not implement collective RPC {method}")
|
||||
|
||||
def shutdown(self) -> None:
|
||||
self._closed = True
|
||||
self._output = None
|
||||
drained = self._worker is None or self._worker.terminate()
|
||||
task = self._task
|
||||
if task is not None and not task.done():
|
||||
# Process exit does not release Python input/transport references.
|
||||
# Keep them charged until the call coroutine actually unwinds.
|
||||
self._ledger.release(self._reservation, drained=False)
|
||||
task.add_done_callback(
|
||||
lambda done: self._ledger.release(self._reservation, drained=drained and not done.cancelled())
|
||||
)
|
||||
else:
|
||||
self._ledger.release(self._reservation, drained=drained)
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"timestamp": "2026-09-22T05:37:31.241692+00:00",
|
||||
"platform": "Windows-11-10.0.26200-SP0",
|
||||
"python": "C:\\Users\\zhout\\w2\\omni029venv\\Scripts\\python.exe",
|
||||
"evidence": "preflight",
|
||||
"packages": {
|
||||
"vllm": "0.29.0+cu134",
|
||||
"torch": "2.13.0+cu130",
|
||||
"transformers": "5.14.1",
|
||||
"onnxruntime": null,
|
||||
"onnxruntime-directml": null
|
||||
},
|
||||
"search_roots": [
|
||||
{
|
||||
"path": "Microsoft.PowerShell.Core\\FileSystem::\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models",
|
||||
"exists": false
|
||||
},
|
||||
{
|
||||
"path": "C:\\Users\\zhout\\.cache\\huggingface\\hub",
|
||||
"exists": true
|
||||
},
|
||||
{
|
||||
"path": "C:\\Users\\zhout\\w2\\models",
|
||||
"exists": true
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"directory": "C:\\Users\\zhout\\w2\\models\\Qwen3-TTS-0.6B-85e237c",
|
||||
"name": "Qwen3-TTS-0.6B-85e237c",
|
||||
"model_type": "qwen3_tts",
|
||||
"architectures": [
|
||||
"Qwen3TTSForConditionalGeneration"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1811626576
|
||||
}
|
||||
],
|
||||
"vllm_platform": "cuda",
|
||||
"registered_pipelines": {
|
||||
"spark2_5": true,
|
||||
"qwen3_tts": true,
|
||||
"qwen3_5": false,
|
||||
"minicpmo_4_5": true
|
||||
},
|
||||
"nvidia-smi": {
|
||||
"returncode": 0,
|
||||
"stdout": "NVIDIA GeForce RTX 5090 Laptop GPU, 24463 MiB, 20263 MiB\n",
|
||||
"stderr": ""
|
||||
},
|
||||
"adb": {
|
||||
"error": "[WinError 2] \u7cfb\u7edf\u627e\u4e0d\u5230\u6307\u5b9a\u7684\u6587\u4ef6\u3002"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
{
|
||||
"timestamp": "2026-09-22T05:37:30.425887+00:00",
|
||||
"platform": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.43",
|
||||
"python": "/home/zhout/project/edge_infer/.venvs/omni-cuda-029/bin/python",
|
||||
"evidence": "preflight",
|
||||
"packages": {
|
||||
"vllm": "0.29.0",
|
||||
"torch": "2.13.0",
|
||||
"transformers": "5.14.1",
|
||||
"onnxruntime": "1.30.0",
|
||||
"onnxruntime-directml": null
|
||||
},
|
||||
"search_roots": [
|
||||
{
|
||||
"path": "/home/zhout/project/edge_infer/models",
|
||||
"exists": true
|
||||
},
|
||||
{
|
||||
"path": "/home/zhout/.cache/huggingface/hub",
|
||||
"exists": true
|
||||
},
|
||||
{
|
||||
"path": "/mnt/c/Users/zhout/w2/models",
|
||||
"exists": true
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/Qwen3.8-27B-FP8",
|
||||
"name": "Qwen3.8-27B-FP8",
|
||||
"model_type": "qwen3_5",
|
||||
"architectures": [
|
||||
"Qwen3_5ForConditionalGeneration"
|
||||
],
|
||||
"weight_shards": 66,
|
||||
"weight_bytes": 30866866928
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-gptq-g128",
|
||||
"name": "spark2_5-gptq-g128",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1287662032
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g32",
|
||||
"name": "spark2_5-int4-g32",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1068136688
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/Spark-X2.5-1.7B-int8",
|
||||
"name": "Spark-X2.5-1.7B-int8",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1979003288
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g64-rtn",
|
||||
"name": "spark2_5-int4-g64-rtn",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 961443656
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/Qwen3.8-27B-NVFP4",
|
||||
"name": "Qwen3.8-27B-NVFP4",
|
||||
"model_type": "qwen3_5",
|
||||
"architectures": [
|
||||
"Qwen3_5ForConditionalGeneration"
|
||||
],
|
||||
"weight_shards": 3,
|
||||
"weight_bytes": 21921697280
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g64",
|
||||
"name": "spark2_5-int4-g64",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 961443656
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g32-expanded",
|
||||
"name": "spark2_5-int4-g32-expanded",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 3415340632
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g128-nodequant",
|
||||
"error": "[Errno 2] No such file or directory: '/home/zhout/project/edge_infer/models/spark2_5-int4-g128-nodequant/model.safetensors'"
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g32-rtn",
|
||||
"name": "spark2_5-int4-g32-rtn",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1068136688
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g64-fpembed",
|
||||
"name": "spark2_5-int4-g64-fpembed",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1347320208
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g64-nocustomop",
|
||||
"error": "[Errno 2] No such file or directory: '/home/zhout/project/edge_infer/models/spark2_5-int4-g64-nocustomop/model.safetensors'"
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g64-expanded",
|
||||
"name": "spark2_5-int4-g64-expanded",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 3415340632
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-gptq-asym-g128",
|
||||
"name": "spark2_5-gptq-asym-g128",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1427131344
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-gptq-asym-g64",
|
||||
"name": "spark2_5-gptq-asym-g64",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1460472840
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-gptq-g128-head",
|
||||
"name": "spark2_5-gptq-g128-head",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1427131536
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/MiniCPM5-2B",
|
||||
"name": "MiniCPM5-2B",
|
||||
"model_type": "llama",
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 5033557096
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-int4-g128",
|
||||
"name": "spark2_5-int4-g128",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 908097200
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/spark2_5-gptq-g64",
|
||||
"name": "spark2_5-gptq-g64",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1460473032
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/project/edge_infer/models/Spark-X2.5-4B",
|
||||
"name": "Spark-X2.5-4B",
|
||||
"model_type": "spark2_5",
|
||||
"architectures": [
|
||||
"Spark2_5ForCausalLM"
|
||||
],
|
||||
"weight_shards": 5,
|
||||
"weight_bytes": 8224192408
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3.8-27B-FP8/snapshots/017b9c7af6b5689d5dd426a76e0bc077eb5ca20a",
|
||||
"name": "models--Qwen--Qwen3.8-27B-FP8",
|
||||
"model_type": "qwen3_5",
|
||||
"architectures": [
|
||||
"Qwen3_5ForConditionalGeneration"
|
||||
],
|
||||
"weight_shards": 0,
|
||||
"weight_bytes": 0
|
||||
},
|
||||
{
|
||||
"directory": "/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5",
|
||||
"name": "models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice",
|
||||
"model_type": "qwen3_tts",
|
||||
"architectures": [
|
||||
"Qwen3TTSForConditionalGeneration"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1811626576
|
||||
},
|
||||
{
|
||||
"directory": "/mnt/c/Users/zhout/w2/models/Qwen3-TTS-0.6B-85e237c",
|
||||
"name": "Qwen3-TTS-0.6B-85e237c",
|
||||
"model_type": "qwen3_tts",
|
||||
"architectures": [
|
||||
"Qwen3TTSForConditionalGeneration"
|
||||
],
|
||||
"weight_shards": 1,
|
||||
"weight_bytes": 1811626576
|
||||
}
|
||||
],
|
||||
"vllm_platform": "cuda",
|
||||
"registered_pipelines": {
|
||||
"spark2_5": true,
|
||||
"qwen3_tts": true,
|
||||
"qwen3_5": false,
|
||||
"minicpmo_4_5": true
|
||||
},
|
||||
"nvidia-smi": {
|
||||
"returncode": 0,
|
||||
"stdout": "NVIDIA GeForce RTX 5090 Laptop GPU, 24463 MiB, 20263 MiB\n",
|
||||
"stderr": ""
|
||||
},
|
||||
"adb": {
|
||||
"returncode": 0,
|
||||
"stdout": "List of devices attached\n\n",
|
||||
"stderr": ""
|
||||
}
|
||||
}
|
||||
+2608
File diff suppressed because it is too large.
Load diff
+523
@@ -0,0 +1,523 @@
|
||||
[
|
||||
{
|
||||
"model": "tts",
|
||||
"job": "j5wl18dzp",
|
||||
"reference_job": "jpez3xm7p",
|
||||
"model_id": "mmd5oy30q",
|
||||
"input_dataset_id": "d70qr8459",
|
||||
"scope": "same compiled component repeatability only",
|
||||
"outputs": [
|
||||
{
|
||||
"name": "output_0__0",
|
||||
"shape": [
|
||||
1,
|
||||
17,
|
||||
1024
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"model": "minicpm",
|
||||
"job": "jpe7l8r75",
|
||||
"reference_job": "jgol682xg",
|
||||
"model_id": "mn7zvvw8m",
|
||||
"input_dataset_id": "d7gwxe3y2",
|
||||
"scope": "same compiled component repeatability only",
|
||||
"outputs": [
|
||||
{
|
||||
"name": "output_0__0",
|
||||
"shape": [
|
||||
1,
|
||||
6562
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_1__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_2__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_3__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_4__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_5__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_6__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_7__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_8__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_9__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_10__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_11__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_12__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_13__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_14__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_15__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_16__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_17__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_18__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_19__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_20__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_21__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_22__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_23__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_24__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_25__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_26__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_27__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_28__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_29__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_30__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_31__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_32__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_33__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_34__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_35__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_36__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_37__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_38__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_39__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
},
|
||||
{
|
||||
"name": "output_40__0",
|
||||
"shape": [
|
||||
1,
|
||||
12,
|
||||
1,
|
||||
64
|
||||
],
|
||||
"finite": true,
|
||||
"exact": true,
|
||||
"max_abs_difference": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"S25|fp16": {
|
||||
"snr_db": 48.86255471981432,
|
||||
"corr": 0.999994236694899,
|
||||
"top1_match": true,
|
||||
"job": "jgol682xg"
|
||||
},
|
||||
"S25|w8a16": {
|
||||
"snr_db": 2.3150011558735324,
|
||||
"corr": 0.7138362463736316,
|
||||
"top1_match": false,
|
||||
"job": "jp4yr3nvp"
|
||||
},
|
||||
"S24|fp16": {
|
||||
"snr_db": 49.221230017543675,
|
||||
"corr": 0.9999949011919217,
|
||||
"top1_match": true,
|
||||
"job": "jp0m0646g"
|
||||
},
|
||||
"S24|w8a16": {
|
||||
"snr_db": 0.069410481494823,
|
||||
"corr": 0.5731221096245495,
|
||||
"top1_match": false,
|
||||
"job": "jgol68zxg"
|
||||
}
|
||||
}
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"timestamp_utc": "2026-09-22T05:41:07.867017+00:00",
|
||||
"qai_hub_version": "0.55.0",
|
||||
"facility": "Qualcomm AI Hub Workbench",
|
||||
"command": "collect",
|
||||
"job_id": "jp0m0646g",
|
||||
"url": "https://workbench.aihub.qualcomm.com/jobs/jp0m0646g/",
|
||||
"status": "SUCCESS",
|
||||
"status_message": "",
|
||||
"options": "",
|
||||
"hub_version": "aihub-2026.08.28.1",
|
||||
"device": {
|
||||
"name": "Samsung Galaxy S24",
|
||||
"os": "14",
|
||||
"attributes": [
|
||||
"os:android",
|
||||
"framework:tflite",
|
||||
"framework:onnx",
|
||||
"abi:aarch64-android",
|
||||
"vendor:samsung",
|
||||
"format:phone",
|
||||
"chipset:qualcomm-snapdragon-8gen3",
|
||||
"chipset:sm8650",
|
||||
"hexagon:v75",
|
||||
"soc-model:57",
|
||||
"htp-supports-fp16:true",
|
||||
"htp-supports-weight-sharing:true",
|
||||
"framework:qnn"
|
||||
]
|
||||
},
|
||||
"model_id": "mqp411wgq",
|
||||
"input_dataset_id": "d7zn8rk57",
|
||||
"outputs": "minicpm-s24-reference.npz"
|
||||
}
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"timestamp_utc": "2026-09-22T05:41:29.604567+00:00",
|
||||
"qai_hub_version": "0.55.0",
|
||||
"facility": "Qualcomm AI Hub Workbench",
|
||||
"command": "collect",
|
||||
"job_id": "jpe7l8r75",
|
||||
"url": "https://workbench.aihub.qualcomm.com/jobs/jpe7l8r75/",
|
||||
"status": "SUCCESS",
|
||||
"status_message": "",
|
||||
"options": "--compute_unit npu",
|
||||
"hub_version": "aihub-2026.09.11.0",
|
||||
"device": {
|
||||
"name": "Samsung Galaxy S25",
|
||||
"os": "15",
|
||||
"attributes": [
|
||||
"os:android",
|
||||
"framework:tflite",
|
||||
"framework:onnx",
|
||||
"abi:aarch64-android",
|
||||
"vendor:samsung",
|
||||
"format:phone",
|
||||
"chipset:qualcomm-snapdragon-8-elite-for-galaxy",
|
||||
"chipset:sm8750-ac",
|
||||
"hexagon:v79",
|
||||
"soc-model:69",
|
||||
"htp-supports-fp16:true",
|
||||
"htp-supports-weight-sharing:true",
|
||||
"framework:qnn"
|
||||
]
|
||||
},
|
||||
"model_id": "mn7zvvw8m",
|
||||
"input_dataset_id": "d7gwxe3y2",
|
||||
"outputs": "minicpm-s25-repeat.npz"
|
||||
}
|
||||
+386
@@ -0,0 +1,386 @@
|
||||
{
|
||||
"args": {
|
||||
"model": "Qwen3.8-27B-NVFP4",
|
||||
"tag": "nvfp4_fullgpu",
|
||||
"offload": 0.0,
|
||||
"max_len": 4096,
|
||||
"batch_tokens": 128,
|
||||
"contexts": "128,512,2048",
|
||||
"output_tokens": 16
|
||||
},
|
||||
"started_unix": 1789451559.825598,
|
||||
"status": "completed",
|
||||
"requests": [
|
||||
{
|
||||
"label": "warmup",
|
||||
"warmup": true,
|
||||
"input_tokens": 37,
|
||||
"requested_input_tokens": 64,
|
||||
"output_tokens": 4,
|
||||
"wall_s": 2.104777352999008,
|
||||
"ttft_s": 0.9845618159997684,
|
||||
"effective_prefill_tok_s": 37.58016957262204,
|
||||
"decode_tok_s": 2.678075334243908,
|
||||
"text": "A local inference engine",
|
||||
"token_events": [
|
||||
{
|
||||
"tokens": 1,
|
||||
"elapsed_s": 0.9845618159997684
|
||||
},
|
||||
{
|
||||
"tokens": 2,
|
||||
"elapsed_s": 1.4607479930000409
|
||||
},
|
||||
{
|
||||
"tokens": 3,
|
||||
"elapsed_s": 1.9160753899996052
|
||||
},
|
||||
{
|
||||
"tokens": 4,
|
||||
"elapsed_s": 2.1047692880001705
|
||||
}
|
||||
],
|
||||
"engine_metrics": {
|
||||
"num_generation_tokens": 4,
|
||||
"arrival_time": 1789451614.1118956,
|
||||
"queued_ts": 16132.809216287,
|
||||
"scheduled_ts": 16132.809240102,
|
||||
"first_token_ts": 16133.773814405,
|
||||
"last_token_ts": 16134.894747982,
|
||||
"first_token_latency": 0.9666800498962402,
|
||||
"is_corrupted": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "context_128",
|
||||
"warmup": false,
|
||||
"input_tokens": 101,
|
||||
"requested_input_tokens": 128,
|
||||
"output_tokens": 16,
|
||||
"wall_s": 7.534435138000845,
|
||||
"ttft_s": 0.8878744009998627,
|
||||
"effective_prefill_tok_s": 113.75482825753372,
|
||||
"decode_tok_s": 2.2568085983584214,
|
||||
"text": "A local inference engine schedules model stages and manages memory.\n",
|
||||
"token_events": [
|
||||
{
|
||||
"tokens": 1,
|
||||
"elapsed_s": 0.8878744009998627
|
||||
},
|
||||
{
|
||||
"tokens": 2,
|
||||
"elapsed_s": 1.35589112299931
|
||||
},
|
||||
{
|
||||
"tokens": 3,
|
||||
"elapsed_s": 1.8163898959992366
|
||||
},
|
||||
{
|
||||
"tokens": 4,
|
||||
"elapsed_s": 2.2728916380001465
|
||||
},
|
||||
{
|
||||
"tokens": 5,
|
||||
"elapsed_s": 2.740661313999226
|
||||
},
|
||||
{
|
||||
"tokens": 6,
|
||||
"elapsed_s": 3.191756124000676
|
||||
},
|
||||
{
|
||||
"tokens": 7,
|
||||
"elapsed_s": 3.6650295590006863
|
||||
},
|
||||
{
|
||||
"tokens": 8,
|
||||
"elapsed_s": 4.116174912000133
|
||||
},
|
||||
{
|
||||
"tokens": 9,
|
||||
"elapsed_s": 4.581921675999183
|
||||
},
|
||||
{
|
||||
"tokens": 10,
|
||||
"elapsed_s": 5.050290941999265
|
||||
},
|
||||
{
|
||||
"tokens": 11,
|
||||
"elapsed_s": 5.5145125439994445
|
||||
},
|
||||
{
|
||||
"tokens": 12,
|
||||
"elapsed_s": 5.978590863000136
|
||||
},
|
||||
{
|
||||
"tokens": 13,
|
||||
"elapsed_s": 6.428808316000868
|
||||
},
|
||||
{
|
||||
"tokens": 14,
|
||||
"elapsed_s": 6.903809924999223
|
||||
},
|
||||
{
|
||||
"tokens": 15,
|
||||
"elapsed_s": 7.354738661999363
|
||||
},
|
||||
{
|
||||
"tokens": 16,
|
||||
"elapsed_s": 7.534428304999892
|
||||
}
|
||||
],
|
||||
"engine_metrics": {
|
||||
"num_generation_tokens": 16,
|
||||
"arrival_time": 1789451616.200588,
|
||||
"queued_ts": 16134.897727778,
|
||||
"scheduled_ts": 16134.897737055,
|
||||
"first_token_ts": 16135.783655995,
|
||||
"last_token_ts": 16142.430436973,
|
||||
"first_token_latency": 0.8875420093536377,
|
||||
"is_corrupted": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "context_512",
|
||||
"warmup": false,
|
||||
"input_tokens": 485,
|
||||
"requested_input_tokens": 512,
|
||||
"output_tokens": 16,
|
||||
"wall_s": 9.34543342500001,
|
||||
"ttft_s": 2.698834197999531,
|
||||
"effective_prefill_tok_s": 179.70722334832524,
|
||||
"decode_tok_s": 2.2567944713093335,
|
||||
"text": "The passage consists of the single sentence \"A local inference engine schedules model stages and",
|
||||
"token_events": [
|
||||
{
|
||||
"tokens": 1,
|
||||
"elapsed_s": 2.698834197999531
|
||||
},
|
||||
{
|
||||
"tokens": 2,
|
||||
"elapsed_s": 3.153196487999594
|
||||
},
|
||||
{
|
||||
"tokens": 3,
|
||||
"elapsed_s": 3.620412973999919
|
||||
},
|
||||
{
|
||||
"tokens": 4,
|
||||
"elapsed_s": 4.0760655800004315
|
||||
},
|
||||
{
|
||||
"tokens": 5,
|
||||
"elapsed_s": 4.530665960000988
|
||||
},
|
||||
{
|
||||
"tokens": 6,
|
||||
"elapsed_s": 5.001150199001131
|
||||
},
|
||||
{
|
||||
"tokens": 7,
|
||||
"elapsed_s": 5.45293877900076
|
||||
},
|
||||
{
|
||||
"tokens": 8,
|
||||
"elapsed_s": 5.922671457999968
|
||||
},
|
||||
{
|
||||
"tokens": 9,
|
||||
"elapsed_s": 6.380228691999946
|
||||
},
|
||||
{
|
||||
"tokens": 10,
|
||||
"elapsed_s": 6.8396083309999085
|
||||
},
|
||||
{
|
||||
"tokens": 11,
|
||||
"elapsed_s": 7.307941597000536
|
||||
},
|
||||
{
|
||||
"tokens": 12,
|
||||
"elapsed_s": 7.773795327000698
|
||||
},
|
||||
{
|
||||
"tokens": 13,
|
||||
"elapsed_s": 8.249526080000578
|
||||
},
|
||||
{
|
||||
"tokens": 14,
|
||||
"elapsed_s": 8.700053388000015
|
||||
},
|
||||
{
|
||||
"tokens": 15,
|
||||
"elapsed_s": 9.167452538000362
|
||||
},
|
||||
{
|
||||
"tokens": 16,
|
||||
"elapsed_s": 9.345429708000665
|
||||
}
|
||||
],
|
||||
"engine_metrics": {
|
||||
"num_generation_tokens": 16,
|
||||
"arrival_time": 1789451623.7372339,
|
||||
"queued_ts": 16142.434277205,
|
||||
"scheduled_ts": 16142.434285471,
|
||||
"first_token_ts": 16145.130608161,
|
||||
"last_token_ts": 16151.777934355,
|
||||
"first_token_latency": 2.698073387145996,
|
||||
"is_corrupted": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "context_2048",
|
||||
"warmup": false,
|
||||
"input_tokens": 2021,
|
||||
"requested_input_tokens": 2048,
|
||||
"output_tokens": 16,
|
||||
"wall_s": 16.710036779999427,
|
||||
"ttft_s": 10.016890127999432,
|
||||
"effective_prefill_tok_s": 201.75922608463642,
|
||||
"decode_tok_s": 2.241099848233223,
|
||||
"text": "The passage consists of the single sentence \"A local inference engine schedules model stages and",
|
||||
"token_events": [
|
||||
{
|
||||
"tokens": 1,
|
||||
"elapsed_s": 10.016890127999432
|
||||
},
|
||||
{
|
||||
"tokens": 2,
|
||||
"elapsed_s": 10.475717346998863
|
||||
},
|
||||
{
|
||||
"tokens": 3,
|
||||
"elapsed_s": 10.951587396999457
|
||||
},
|
||||
{
|
||||
"tokens": 4,
|
||||
"elapsed_s": 11.405946809998568
|
||||
},
|
||||
{
|
||||
"tokens": 5,
|
||||
"elapsed_s": 11.883892373998606
|
||||
},
|
||||
{
|
||||
"tokens": 6,
|
||||
"elapsed_s": 12.338071106998541
|
||||
},
|
||||
{
|
||||
"tokens": 7,
|
||||
"elapsed_s": 12.809085696999318
|
||||
},
|
||||
{
|
||||
"tokens": 8,
|
||||
"elapsed_s": 13.279166899999836
|
||||
},
|
||||
{
|
||||
"tokens": 9,
|
||||
"elapsed_s": 13.738810389999344
|
||||
},
|
||||
{
|
||||
"tokens": 10,
|
||||
"elapsed_s": 14.21198344899858
|
||||
},
|
||||
{
|
||||
"tokens": 11,
|
||||
"elapsed_s": 14.666852663998725
|
||||
},
|
||||
{
|
||||
"tokens": 12,
|
||||
"elapsed_s": 15.146523536999666
|
||||
},
|
||||
{
|
||||
"tokens": 13,
|
||||
"elapsed_s": 15.601053123999009
|
||||
},
|
||||
{
|
||||
"tokens": 14,
|
||||
"elapsed_s": 16.074013286999616
|
||||
},
|
||||
{
|
||||
"tokens": 15,
|
||||
"elapsed_s": 16.53617112899883
|
||||
},
|
||||
{
|
||||
"tokens": 16,
|
||||
"elapsed_s": 16.710032341999977
|
||||
}
|
||||
],
|
||||
"engine_metrics": {
|
||||
"num_generation_tokens": 16,
|
||||
"arrival_time": 1789451633.0867412,
|
||||
"queued_ts": 16151.783924979,
|
||||
"scheduled_ts": 16151.783939036,
|
||||
"first_token_ts": 16161.797966789,
|
||||
"last_token_ts": 16168.492024295,
|
||||
"first_token_latency": 10.01593279838562,
|
||||
"is_corrupted": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"idle_cpu_pct": [
|
||||
4.0,
|
||||
0.0,
|
||||
1.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0,
|
||||
0.0
|
||||
],
|
||||
"wsl_memory_before": {
|
||||
"total": 33184985088,
|
||||
"available": 31711698944,
|
||||
"percent": 4.4,
|
||||
"used": 1473286144,
|
||||
"free": 31611633664,
|
||||
"active": 661553152,
|
||||
"inactive": 90931200,
|
||||
"buffers": 3821568,
|
||||
"cached": 469770240,
|
||||
"shared": 1900544,
|
||||
"slab": 468426752
|
||||
},
|
||||
"import_s": 10.525000752999404,
|
||||
"vllm_file": "/home/zhout/project/edge_infer/.venvs/omni-cuda/lib/python3.12/site-packages/vllm/__init__.py",
|
||||
"config": {
|
||||
"model": "/home/zhout/project/edge_infer/models/Qwen3.8-27B-NVFP4",
|
||||
"dtype": "bfloat16",
|
||||
"max_model_len": 4096,
|
||||
"max_num_seqs": 1,
|
||||
"max_num_batched_tokens": 128,
|
||||
"gpu_memory_utilization": 0.9,
|
||||
"enforce_eager": true,
|
||||
"cpu_offload_gb": 0.0,
|
||||
"language_model_only": true,
|
||||
"enable_prefix_caching": false,
|
||||
"seed": 42,
|
||||
"disable_log_stats": false
|
||||
},
|
||||
"engine_load_s": 42.73773059799896,
|
||||
"loaded_unix": 1789451614.0896077,
|
||||
"ended_unix": 1789451649.7974334,
|
||||
"peak": {
|
||||
"tree_rss": 13575667712,
|
||||
"tree_pss": 13079872512,
|
||||
"tree_rss_anon": 3798523904,
|
||||
"gpu_used": 25273769984,
|
||||
"gpu_power_mw": 39459,
|
||||
"gpu_temp": 49,
|
||||
"wsl_system_used": 5491564544
|
||||
}
|
||||
}
|
||||
+81
@@ -0,0 +1,81 @@
|
||||
{
|
||||
"runtime": {
|
||||
"python": "3.12.13",
|
||||
"platform": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.43",
|
||||
"torch": "2.13.0+cu130",
|
||||
"vllm": "0.28.0",
|
||||
"vllm_omni": "0.28.1.dev154+g3834de310.d20260914",
|
||||
"vllm_omni_file": "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/__init__.py",
|
||||
"vllm_file": "/home/zhout/project/edge_infer/.venvs/omni-cuda/lib/python3.12/site-packages/vllm/__init__.py",
|
||||
"checkouts": {
|
||||
"/home/zhout/project/edge_infer/vllm-omni": "3834de31051ae428beb2765915cdaff1772b9bbc-dirty",
|
||||
"/home/zhout/project/edge_infer": "758b3af238b19e2c2abc668f23a767bb835a4a22-dirty"
|
||||
}
|
||||
},
|
||||
"shape": {
|
||||
"image_size": 448,
|
||||
"grid_thw": [
|
||||
1,
|
||||
28,
|
||||
28
|
||||
],
|
||||
"patches": 784,
|
||||
"patch_dim": 1536,
|
||||
"merged_tokens": 196,
|
||||
"out_hidden_size": 5120
|
||||
},
|
||||
"artifact": {
|
||||
"sha256": "68c8ee4a9ad8509c3a17e4591d3ba4fa0f2b30a5b725c3dd4e93a4dd8ca5fa65",
|
||||
"bytes": 1838533565,
|
||||
"fmt": "onnx:fp32",
|
||||
"opset": 21
|
||||
},
|
||||
"reference": {
|
||||
"backend": "torch cpu fp32",
|
||||
"median_s": 1.3556148889983888,
|
||||
"threads": 12
|
||||
},
|
||||
"arms": {
|
||||
"igpu_dml_fp32": {
|
||||
"node_counts": {
|
||||
"DmlExecutionProvider": 85,
|
||||
"CPUExecutionProvider": 28
|
||||
},
|
||||
"fraction_on_target": 0.7522123893805309,
|
||||
"parity_vs_torch_cpu": {
|
||||
"max_abs": 0.00101470947265625,
|
||||
"mean_abs": 1.7557707678861334e-06,
|
||||
"normalized_max": 9.09517166292062e-06,
|
||||
"cosine": 0.9999999403953552
|
||||
},
|
||||
"runs": 10,
|
||||
"median_worker_s": 0.23143709999567363,
|
||||
"median_round_trip_s": 0.24530861800303683,
|
||||
"median_transport_s": 0.012688652001088485,
|
||||
"samples_worker_s": [
|
||||
0.21845380000013392,
|
||||
0.22525360000145156,
|
||||
0.2263950000051409,
|
||||
0.22689469999750145,
|
||||
0.2282961000018986,
|
||||
0.23143709999567363,
|
||||
0.2368339999957243,
|
||||
0.24216529999102931,
|
||||
0.24823209999885876,
|
||||
0.24826890000258572
|
||||
],
|
||||
"note": "medians over 10 runs after the load; the first call after a session is opened is materially slower and is excluded by the warm-up the load itself does"
|
||||
},
|
||||
"cuda_5090_fp32": {
|
||||
"median_s": 0.09445196900196606,
|
||||
"max_memory_allocated_bytes": 1952522240,
|
||||
"parity_vs_torch_cpu": {
|
||||
"max_abs": 0.06308555603027344,
|
||||
"mean_abs": 0.0001078158529708162,
|
||||
"normalized_max": 0.000565456401091069,
|
||||
"cosine": 0.9999995827674866
|
||||
},
|
||||
"note": "torch runs matmul in TF32 by default on CUDA; this is why its deviation from the CPU reference is larger than the iGPU's"
|
||||
}
|
||||
}
|
||||
}
|
||||
+355
@@ -0,0 +1,355 @@
|
||||
{
|
||||
"argv": [
|
||||
"/home/zhout/project/edge_infer/vllm-omni/vllm_omni/edge/local/__main__.py",
|
||||
"external",
|
||||
"--graph",
|
||||
"/tmp/claude-1000/-home-zhout-project-edge-infer/e1528a86-4037-4e80-b83f-3a2f6cd1e24f/scratchpad/tower_npu/tower_a16w8.onnx",
|
||||
"--format",
|
||||
"onnx:a16w8",
|
||||
"--opset",
|
||||
"21",
|
||||
"--component",
|
||||
"qwen3_5_vision_tower_448_a16w8",
|
||||
"--source-model",
|
||||
"models/Qwen3.8-27B-FP8",
|
||||
"--exporter",
|
||||
"vllm_omni.edge.local.artifacts.vision_tower",
|
||||
"--device",
|
||||
"npu:amd",
|
||||
"--min-placement",
|
||||
"0.30",
|
||||
"--runs",
|
||||
"5",
|
||||
"--inputs",
|
||||
"/tmp/claude-1000/-home-zhout-project-edge-infer/e1528a86-4037-4e80-b83f-3a2f6cd1e24f/scratchpad/tower_io.npz",
|
||||
"--profile-dir",
|
||||
"/tmp/claude-1000/-home-zhout-project-edge-infer/e1528a86-4037-4e80-b83f-3a2f6cd1e24f/scratchpad",
|
||||
"--json",
|
||||
"analysis/experiments/m3_amd_devices_20260915/m32_tower_npu.json"
|
||||
],
|
||||
"runtime": {
|
||||
"python": "3.12.13",
|
||||
"platform": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.43",
|
||||
"torch": "2.13.0+cu130",
|
||||
"vllm": "0.28.0",
|
||||
"vllm_omni": "0.28.1.dev154+g3834de310.d20260914",
|
||||
"vllm_omni_file": "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/__init__.py",
|
||||
"vllm_file": "/home/zhout/project/edge_infer/.venvs/omni-cuda/lib/python3.12/site-packages/vllm/__init__.py",
|
||||
"checkouts": {
|
||||
"/home/zhout/project/edge_infer/vllm-omni": "3834de31051ae428beb2765915cdaff1772b9bbc-dirty",
|
||||
"/home/zhout/project/edge_infer": "758b3af238b19e2c2abc668f23a767bb835a4a22-dirty"
|
||||
}
|
||||
},
|
||||
"devices": [
|
||||
{
|
||||
"device_id": "cuda:0",
|
||||
"kind": "gpu_discrete",
|
||||
"vendor": "nvidia",
|
||||
"name": "NVIDIA GeForce RTX 5090 Laptop GPU",
|
||||
"backend": "vllm:cuda",
|
||||
"runnable": true,
|
||||
"memory_pool": "vram",
|
||||
"memory_bytes": 25650855936,
|
||||
"compute_capability": [
|
||||
12,
|
||||
0
|
||||
],
|
||||
"weight_formats": [
|
||||
"awq",
|
||||
"compressed-tensors:fp8",
|
||||
"dense",
|
||||
"gptq",
|
||||
"modelopt_fp4"
|
||||
],
|
||||
"evidence": "B",
|
||||
"reason": "",
|
||||
"masked": false,
|
||||
"extra": {
|
||||
"sm_count": 82,
|
||||
"source": "torch.cuda"
|
||||
}
|
||||
},
|
||||
{
|
||||
"device_id": "npu:amd",
|
||||
"kind": "npu",
|
||||
"vendor": "amd",
|
||||
"name": "NPU Compute Accelerator Device",
|
||||
"backend": "ort:vitisai",
|
||||
"runnable": true,
|
||||
"memory_pool": "host_ram",
|
||||
"memory_bytes": 0,
|
||||
"compute_capability": null,
|
||||
"weight_formats": [
|
||||
"onnx:a16w8"
|
||||
],
|
||||
"evidence": "B",
|
||||
"reason": "An MCDM device; WSL2's GPU-PV forwards WDDM display adapters only, so it has no device node here and nothing installed inside WSL reaches it. It does work from a Windows-side process -- onnxruntime>=1.25 plus the WindowsWorkload.EP.AMD.VitisAI.Framework appx, loading onnxruntime_vitisai_ep.dll (not the RyzenAI *Light* EP beside it) with its own directory on the DLL search path, against an A16W8-quantized opset-21 graph. Measured 5.3x the CPU on a 1024x2048x2048 GEMM. See vllm_omni/edge/npu_ryzenai.py.",
|
||||
"masked": false,
|
||||
"extra": {
|
||||
"source": "wsl-interop",
|
||||
"probe_route": "ryzenai-ep-windows",
|
||||
"host_pool_bytes": 33184985088,
|
||||
"host_pool_available_bytes": 29899563008,
|
||||
"worker_route": "ort-vitisai",
|
||||
"worker_interpreter": "/mnt/c/Users/zhout/npu-ep/Scripts/python.exe",
|
||||
"routes": [
|
||||
{
|
||||
"name": "ort-vitisai",
|
||||
"interpreter": "/mnt/c/Users/zhout/npu-ep/Scripts/python.exe",
|
||||
"worker": "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/edge/local/external/worker_ort.py",
|
||||
"ep": "vitisai",
|
||||
"is_windows": true,
|
||||
"available": true,
|
||||
"reason": "",
|
||||
"extra": null
|
||||
}
|
||||
],
|
||||
"executes": "exported graphs, not checkpoints",
|
||||
"dedicated_vram_bytes": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"device_id": "gpu_integrated:amd",
|
||||
"kind": "gpu_integrated",
|
||||
"vendor": "amd",
|
||||
"name": "AMD Radeon(TM) 890M Graphics",
|
||||
"backend": "ort:dml",
|
||||
"runnable": true,
|
||||
"memory_pool": "host_ram",
|
||||
"memory_bytes": 0,
|
||||
"compute_capability": null,
|
||||
"weight_formats": [
|
||||
"onnx:fp16",
|
||||
"onnx:fp32"
|
||||
],
|
||||
"evidence": "B",
|
||||
"reason": "No /dev/dri under WSL2, but reachable through DirectML over /dev/dxg: pip install torch-directml in its own venv (it pins torch==2.4.1). Measured 1.12 TFLOP/s fp32 and 5-7.4 GB/s -- below the CPU on both, so route work here only when the CPU is the thing you are trying to free.",
|
||||
"masked": false,
|
||||
"extra": {
|
||||
"source": "wsl-interop",
|
||||
"probe_route": "directml",
|
||||
"host_pool_bytes": 33184985088,
|
||||
"host_pool_available_bytes": 29899563008,
|
||||
"worker_route": "ort-dml",
|
||||
"worker_interpreter": "/mnt/c/Users/zhout/dml-ep/Scripts/python.exe",
|
||||
"routes": [
|
||||
{
|
||||
"name": "ort-dml",
|
||||
"interpreter": "/mnt/c/Users/zhout/dml-ep/Scripts/python.exe",
|
||||
"worker": "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/edge/local/external/worker_ort.py",
|
||||
"ep": "dml",
|
||||
"is_windows": true,
|
||||
"available": true,
|
||||
"reason": "",
|
||||
"extra": null
|
||||
},
|
||||
{
|
||||
"name": "torch-dml",
|
||||
"interpreter": "/home/zhout/project/edge_infer/.venvs/dml/bin/python",
|
||||
"worker": "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/edge/local/external/worker_dml.py",
|
||||
"ep": "dml",
|
||||
"is_windows": false,
|
||||
"available": true,
|
||||
"reason": "",
|
||||
"extra": null
|
||||
}
|
||||
],
|
||||
"executes": "exported graphs, not checkpoints",
|
||||
"dedicated_vram_bytes": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"device_id": "cpu",
|
||||
"kind": "cpu",
|
||||
"vendor": "amd",
|
||||
"name": "AMD Ryzen AI 9 HX 370 w/ Radeon 890M",
|
||||
"backend": null,
|
||||
"runnable": false,
|
||||
"memory_pool": "host_ram",
|
||||
"memory_bytes": 33184985088,
|
||||
"compute_capability": null,
|
||||
"weight_formats": [
|
||||
"compressed-tensors:int8",
|
||||
"dense",
|
||||
"gptq"
|
||||
],
|
||||
"evidence": "D",
|
||||
"reason": "vllm.platforms.current_platform is 'cuda', not 'cpu': this interpreter's vLLM wheel has no CPU executor. The CPU build lives in a separate venv (.venvs/omni-cpu); a CPU plan from here is a plan for that interpreter, not something this one can run.",
|
||||
"masked": false,
|
||||
"extra": {
|
||||
"n_physical": 12,
|
||||
"n_logical": 24,
|
||||
"ram_available_bytes": 29899563008,
|
||||
"bf16": true
|
||||
}
|
||||
}
|
||||
],
|
||||
"plan": {
|
||||
"artifact": {
|
||||
"path": "/tmp/claude-1000/-home-zhout-project-edge-infer/e1528a86-4037-4e80-b83f-3a2f6cd1e24f/scratchpad/tower_npu/tower_a16w8.onnx",
|
||||
"sha256": "215746bbfece7b94540c8f833cc19f45b13e0664eaea34eeda4618a60eef3a2c",
|
||||
"bytes": 464117283,
|
||||
"fmt": "onnx:a16w8",
|
||||
"opset": 21,
|
||||
"source_model": "models/Qwen3.8-27B-FP8",
|
||||
"source_revision": null,
|
||||
"component": "qwen3_5_vision_tower_448_a16w8",
|
||||
"exporter": "vllm_omni.edge.local.artifacts.vision_tower",
|
||||
"input_spec": [],
|
||||
"output_spec": [],
|
||||
"calibration": null,
|
||||
"parity": null,
|
||||
"notes": ""
|
||||
},
|
||||
"device": {
|
||||
"device_id": "npu:amd",
|
||||
"kind": "npu",
|
||||
"vendor": "amd",
|
||||
"name": "NPU Compute Accelerator Device",
|
||||
"backend": "ort:vitisai",
|
||||
"runnable": true,
|
||||
"memory_pool": "host_ram",
|
||||
"memory_bytes": 0,
|
||||
"compute_capability": null,
|
||||
"weight_formats": [
|
||||
"onnx:a16w8"
|
||||
],
|
||||
"evidence": "B",
|
||||
"reason": "An MCDM device; WSL2's GPU-PV forwards WDDM display adapters only, so it has no device node here and nothing installed inside WSL reaches it. It does work from a Windows-side process -- onnxruntime>=1.25 plus the WindowsWorkload.EP.AMD.VitisAI.Framework appx, loading onnxruntime_vitisai_ep.dll (not the RyzenAI *Light* EP beside it) with its own directory on the DLL search path, against an A16W8-quantized opset-21 graph. Measured 5.3x the CPU on a 1024x2048x2048 GEMM. See vllm_omni/edge/npu_ryzenai.py.",
|
||||
"masked": false,
|
||||
"extra": {
|
||||
"source": "wsl-interop",
|
||||
"probe_route": "ryzenai-ep-windows",
|
||||
"host_pool_bytes": 33184985088,
|
||||
"host_pool_available_bytes": 29899563008,
|
||||
"worker_route": "ort-vitisai",
|
||||
"worker_interpreter": "/mnt/c/Users/zhout/npu-ep/Scripts/python.exe",
|
||||
"routes": [
|
||||
{
|
||||
"name": "ort-vitisai",
|
||||
"interpreter": "/mnt/c/Users/zhout/npu-ep/Scripts/python.exe",
|
||||
"worker": "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/edge/local/external/worker_ort.py",
|
||||
"ep": "vitisai",
|
||||
"is_windows": true,
|
||||
"available": true,
|
||||
"reason": "",
|
||||
"extra": null
|
||||
}
|
||||
],
|
||||
"executes": "exported graphs, not checkpoints",
|
||||
"dedicated_vram_bytes": 0
|
||||
}
|
||||
},
|
||||
"route": {
|
||||
"name": "ort-vitisai",
|
||||
"interpreter": "/mnt/c/Users/zhout/npu-ep/Scripts/python.exe",
|
||||
"worker": "/home/zhout/project/edge_infer/vllm-omni/vllm_omni/edge/local/external/worker_ort.py",
|
||||
"ep": "vitisai",
|
||||
"is_windows": true,
|
||||
"available": true,
|
||||
"reason": "",
|
||||
"extra": null
|
||||
},
|
||||
"reservations": [
|
||||
{
|
||||
"pool": "host_ram",
|
||||
"purpose": "graph weights",
|
||||
"bytes": 464117283,
|
||||
"lifetime": "session",
|
||||
"reclaimable": "no",
|
||||
"evidence": "D",
|
||||
"note": "the exported file, onnx:a16w8, sha 215746bbfece",
|
||||
"actual_upper_bound_bytes": null
|
||||
},
|
||||
{
|
||||
"pool": "host_ram",
|
||||
"purpose": "worker runtime",
|
||||
"bytes": 335544320,
|
||||
"lifetime": "session",
|
||||
"reclaimable": "on_release",
|
||||
"evidence": "E",
|
||||
"note": "interpreter plus execution provider in the worker process; measured 270 MiB for a VitisAI session on a 2 MiB graph and 369 MiB for a DirectML session on the 27B vision tower. NOTE: on an integrated GPU the device allocations come out of this same system RAM and do not appear in the worker's working set, so the measured figure is a lower bound for this pool, not a total",
|
||||
"actual_upper_bound_bytes": null
|
||||
},
|
||||
{
|
||||
"pool": "host_ram",
|
||||
"purpose": "worker workspace",
|
||||
"bytes": 268435456,
|
||||
"lifetime": "request",
|
||||
"reclaimable": "on_release",
|
||||
"evidence": "E",
|
||||
"note": "activations and EP scratch; not separable by any counter here",
|
||||
"actual_upper_bound_bytes": null
|
||||
},
|
||||
{
|
||||
"pool": "host_ram",
|
||||
"purpose": "external reserve",
|
||||
"bytes": 1073741824,
|
||||
"lifetime": "session",
|
||||
"reclaimable": "no",
|
||||
"evidence": "E",
|
||||
"note": "host RAM this engine does not control; the Windows side included",
|
||||
"actual_upper_bound_bytes": null
|
||||
}
|
||||
],
|
||||
"refusals": [
|
||||
{
|
||||
"code": "execution_provider_declined_graph",
|
||||
"device_id": "npu:amd",
|
||||
"message": "vitisai took 0 of 3246 nodes (0%, below the 30% required); the split was CPUExecutionProvider=3246",
|
||||
"remedy": "the graph is not in a shape XDNA2 partitions. Check, in this order: the quantization is A16W8 (QUInt16 activations, QInt8 weights) at opset >= 21; the EP loaded is onnxruntime_vitisai_ep.dll and not the RyzenAI *Light* provider beside it, which registers cleanly and claims zero nodes from every graph; and that the graph is not mostly requantization, since the partitioner leaves dequantize nodes on the CPU by its own account. Running it anyway would be a CPU fallback reported as NPU execution."
|
||||
}
|
||||
],
|
||||
"min_fraction_on_target": 0.3,
|
||||
"budget_bytes": 2141838883,
|
||||
"admitted": true,
|
||||
"report": {
|
||||
"ep": "vitisai",
|
||||
"graph_path": "/tmp/claude-1000/-home-zhout-project-edge-infer/e1528a86-4037-4e80-b83f-3a2f6cd1e24f/scratchpad/tower_npu/tower_a16w8.onnx",
|
||||
"session_providers": [
|
||||
"vitisai",
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"available_providers": [
|
||||
"AzureExecutionProvider",
|
||||
"CPUExecutionProvider",
|
||||
"vitisai"
|
||||
],
|
||||
"node_counts": {
|
||||
"CPUExecutionProvider": 3246
|
||||
},
|
||||
"total_nodes": 3246,
|
||||
"target_nodes": 0,
|
||||
"fraction_on_target": 0.0,
|
||||
"session_create_s": 75.55920579998929,
|
||||
"warmup_s": 2.251566899998579,
|
||||
"rss_bytes": 931209216,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "patches",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
784,
|
||||
1536
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "embeds",
|
||||
"type": "tensor(float)",
|
||||
"shape": [
|
||||
196,
|
||||
5120
|
||||
]
|
||||
}
|
||||
],
|
||||
"profile_path": "\\\\wsl.localhost\\Ubuntu\\tmp\\claude-1000\\-home-zhout-project-edge-infer\\e1528a86-4037-4e80-b83f-3a2f6cd1e24f\\scratchpad\\qwen3_5_vision_tower_448_a16w8__2026-09-16_09-44-15_178.json",
|
||||
"ep_directory": "C:\\Program Files\\WindowsApps\\WindowsWorkload.EP.AMD.VitisAI.Framework.1.8_1.8.63.0_x64__8wekyb3d8bbwe\\ExecutionProvider",
|
||||
"requested_provider_missing": "",
|
||||
"fallback_note": "",
|
||||
"onnxruntime": "1.30.0",
|
||||
"note": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"platform": "Windows-11-10.0.26200-SP0",
|
||||
"config": {
|
||||
"model": "\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\Qwen3.8-27B-NVFP4",
|
||||
"language_model_only": true,
|
||||
"max_model_len": 512,
|
||||
"max_num_seqs": 1,
|
||||
"max_num_batched_tokens": 128,
|
||||
"enforce_eager": true,
|
||||
"gpu_memory_utilization": 0.8,
|
||||
"cpu_offload_gb": 2,
|
||||
"dtype": "bfloat16",
|
||||
"stage_init_timeout": 120
|
||||
},
|
||||
"status": "failed",
|
||||
"scope": "public Omni, short text only; explicit 2 GiB native offload",
|
||||
"start": 1790055532.5138774,
|
||||
"error": "ValueError(\"Model '\\\\\\\\\\\\\\\\wsl.localhost\\\\\\\\Ubuntu\\\\\\\\home\\\\\\\\zhout\\\\\\\\project\\\\\\\\edge_infer\\\\\\\\models\\\\\\\\Qwen3.8-27B-NVFP4' did not resolve to a registered Omni pipeline or a supported diffusion model.\")",
|
||||
"traceback": "Traceback (most recent call last):\n File \"\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\analysis\\experiments\\model_device_matrix_20260922\\qwen_omni_probe.py\", line 30, in main\n engine = Omni(**config)\n ^^^^^^^^^^^^^^\n File \"\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\vllm-omni-edge\\vllm_omni\\entrypoints\\omni_base.py\", line 223, in __init__\n self.engine = AsyncOmniEngine(\n ^^^^^^^^^^^^^^^^\n File \"\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\vllm-omni-edge\\vllm_omni\\engine\\async_omni_engine.py\", line 228, in __init__\n self.config_path, self.stage_configs = self._resolve_stage_configs(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\vllm-omni-edge\\vllm_omni\\engine\\async_omni_engine.py\", line 1101, in _resolve_stage_configs\n load_and_resolve_stage_configs(\n File \"\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\vllm-omni-edge\\vllm_omni\\engine\\async_omni_engine.py\", line 106, in load_and_resolve_stage_configs\n return resolve_omni_config(\n ^^^^^^^^^^^^^^^^^^^^\n File \"\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\vllm-omni-edge\\vllm_omni\\config\\resolver.py\", line 254, in resolve_omni_config\n raise ValueError(\nValueError: Model '\\\\\\\\wsl.localhost\\\\Ubuntu\\\\home\\\\zhout\\\\project\\\\edge_infer\\\\models\\\\Qwen3.8-27B-NVFP4' did not resolve to a registered Omni pipeline or a supported diffusion model.\n",
|
||||
"end": 1790055533.0640798
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"platform": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.43",
|
||||
"config": {
|
||||
"model": "/home/zhout/project/edge_infer/models/Qwen3.8-27B-NVFP4",
|
||||
"language_model_only": true,
|
||||
"max_model_len": 512,
|
||||
"max_num_seqs": 1,
|
||||
"max_num_batched_tokens": 128,
|
||||
"enforce_eager": true,
|
||||
"gpu_memory_utilization": 0.8,
|
||||
"cpu_offload_gb": 2,
|
||||
"dtype": "bfloat16",
|
||||
"stage_init_timeout": 120
|
||||
},
|
||||
"status": "failed",
|
||||
"scope": "public Omni, short text only; explicit 2 GiB native offload",
|
||||
"start": 1790055472.1485431,
|
||||
"error": "ValueError(\"Model '/home/zhout/project/edge_infer/models/Qwen3.8-27B-NVFP4' did not resolve to a registered Omni pipeline or a supported diffusion model.\")",
|
||||
"traceback": "Traceback (most recent call last):\n File \"/home/zhout/project/edge_infer/analysis/experiments/model_device_matrix_20260922/qwen_omni_probe.py\", line 30, in main\n engine = Omni(**config)\n ^^^^^^^^^^^^^^\n File \"/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni_base.py\", line 223, in __init__\n self.engine = AsyncOmniEngine(\n ^^^^^^^^^^^^^^^^\n File \"/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py\", line 228, in __init__\n self.config_path, self.stage_configs = self._resolve_stage_configs(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py\", line 1101, in _resolve_stage_configs\n load_and_resolve_stage_configs(\n File \"/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/async_omni_engine.py\", line 106, in load_and_resolve_stage_configs\n return resolve_omni_config(\n ^^^^^^^^^^^^^^^^^^^^\n File \"/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/config/resolver.py\", line 254, in resolve_omni_config\n raise ValueError(\nValueError: Model '/home/zhout/project/edge_infer/models/Qwen3.8-27B-NVFP4' did not resolve to a registered Omni pipeline or a supported diffusion model.\n",
|
||||
"end": 1790055473.065098
|
||||
}
|
||||
+185
@@ -0,0 +1,185 @@
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
W0922 13:34:23.252000 629687 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0922 13:34:23.278000 629687 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
INFO 09-22 13:34:24 [importing.py:53] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
|
||||
WARNING 09-22 13:34:24 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-22 13:34:24 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
INFO 09-22 13:34:25 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 13:34:25 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 13:34:25 [patch.py:640] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-22 13:34:25 [patch.py:750] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
========================================================================
|
||||
capability-masked startup (--mask gpu_discrete): the no-NVIDIA class
|
||||
========================================================================
|
||||
ADMITTED on cpu (AMD Ryzen AI 9 HX 370 w/ Radeon 890M) via vllm:cpu
|
||||
artifact e25985ca3f2a44a2 compressed-tensors:int8 1.84GiB
|
||||
budget 6.55GiB of 27.96GiB host_ram (available)
|
||||
weights 1887 MiB host_ram session_weights D 1 safetensors shard(s), on-disk size
|
||||
kv_cache 280 MiB host_ram session E flat budget at 1x context; 96 MiB if the backend accounts 21 sliding layers separately
|
||||
activations 96 MiB host_ram request E 2048-token prefill chunk x 12 live tensors
|
||||
backend_workspace 512 MiB host_ram session E attention scratch, driver/allocator context
|
||||
load_transient 1887 MiB host_ram load E freed once the weights are resident; peaks with the largest shard
|
||||
external_reserve 1024 MiB host_ram session E memory this engine does not own on a shared pool
|
||||
safety_margin 1024 MiB host_ram session E fragmentation and allocator slack
|
||||
kv 280 MiB flat; 96 MiB if the backend accounts 21 sliding@512 layers separately (2.91x)
|
||||
not used: [no_exported_graph] npu:amd: npu:amd executes exported graphs (['onnx:a16w8']), not checkpoints, so a whole autoregressive text session cannot be placed on it
|
||||
not used: [no_exported_graph] gpu_integrated:amd: gpu_integrated:amd executes exported graphs (['onnx:fp16', 'onnx:fp32']), not checkpoints, so a whole autoregressive text session cannot be placed on it
|
||||
|
||||
========================================================================
|
||||
device-visible startup
|
||||
========================================================================
|
||||
ADMITTED on cpu (AMD Ryzen AI 9 HX 370 w/ Radeon 890M) via vllm:cpu
|
||||
artifact e25985ca3f2a44a2 compressed-tensors:int8 1.84GiB
|
||||
budget 6.55GiB of 27.99GiB host_ram (available)
|
||||
weights 1887 MiB host_ram session_weights D 1 safetensors shard(s), on-disk size
|
||||
kv_cache 280 MiB host_ram session E flat budget at 1x context; 96 MiB if the backend accounts 21 sliding layers separately
|
||||
activations 96 MiB host_ram request E 2048-token prefill chunk x 12 live tensors
|
||||
backend_workspace 512 MiB host_ram session E attention scratch, driver/allocator context
|
||||
load_transient 1887 MiB host_ram load E freed once the weights are resident; peaks with the largest shard
|
||||
external_reserve 1024 MiB host_ram session E memory this engine does not own on a shared pool
|
||||
safety_margin 1024 MiB host_ram session E fragmentation and allocator slack
|
||||
kv 280 MiB flat; 96 MiB if the backend accounts 21 sliding@512 layers separately (2.91x)
|
||||
not used: [no_exported_graph] npu:amd: npu:amd executes exported graphs (['onnx:a16w8']), not checkpoints, so a whole autoregressive text session cannot be placed on it
|
||||
not used: [no_exported_graph] gpu_integrated:amd: gpu_integrated:amd executes exported graphs (['onnx:fp16', 'onnx:fp32']), not checkpoints, so a whole autoregressive text session cannot be placed on it
|
||||
|
||||
INFO 09-22 13:34:27 [engine.py:392] [local] loading /home/zhout/project/edge_infer/models/Spark-X2.5-1.7B-int8 on cpu with {'max_model_len': 4096, 'max_num_seqs': 1, 'max_num_batched_tokens': 2048, 'enforce_eager': True, 'enable_prefix_caching': False, 'gpu_memory_utilization': 0.24, 'kv_cache_memory_bytes': 293601280, 'dtype': 'bfloat16'}
|
||||
INFO 09-22 13:34:27 [omni_base.py:216] [AsyncOmni] Initializing with model /home/zhout/project/edge_infer/models/Spark-X2.5-1.7B-int8
|
||||
INFO 09-22 13:34:27 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model /home/zhout/project/edge_infer/models/Spark-X2.5-1.7B-int8
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
INFO 09-22 13:34:27 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 1 stages
|
||||
INFO 09-22 13:34:27 [stage_init_utils.py:768] [stage_init] Set VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
WARNING 09-22 13:34:27 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
INFO 09-22 13:34:27 [model.py:672] Resolved architecture: Spark2_5ForCausalLM
|
||||
INFO 09-22 13:34:27 [model.py:1965] Using max model len 4096
|
||||
INFO 09-22 13:34:28 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
WARNING 09-22 13:34:28 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-22 13:34:28 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-22 13:34:28 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-22 13:34:28 [patch.py:692] Restored explicit kv_cache_memory_bytes=293601280 (0.273 GiB); VLLM_CPU_KVCACHE_SPACE would have overridden it.
|
||||
WARNING 09-22 13:34:28 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
INFO 09-22 13:34:28 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-22 13:34:28 [stage_runtime.py:778] [stage_init] Stage-0 replica 0 physical devices None are not integer GPU ids; all serial LLM replicas will share one init group
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
W0922 13:34:33.190000 629819 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0922 13:34:33.225000 629819 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
INFO 09-22 13:34:34 [importing.py:53] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
|
||||
WARNING 09-22 13:34:34 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-22 13:34:34 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
INFO 09-22 13:34:35 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 13:34:35 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 13:34:35 [patch.py:640] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-22 13:34:35 [patch.py:750] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:36 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/project/edge_infer/models/Spark-X2.5-1.7B-int8', speculative_config=None, tokenizer='/home/zhout/project/edge_infer/models/Spark-X2.5-1.7B-int8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=compressed-tensors, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/project/edge_infer/models/Spark-X2.5-1.7B-int8, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:36 [multiproc_executor.py:150] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.21.141.90 (local), world_size=1, local_world_size=1
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:36 [ompmultiprocessing.py:185] OpenMP thread binding info:
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:36 [ompmultiprocessing.py:185] VLLM_CPU_OMP_THREADS_BIND='auto', auto_setup=True, skip_setup=False
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:36 [ompmultiprocessing.py:185] local_world_size=1, reserve_cpu_num=1
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:36 [ompmultiprocessing.py:185] local_rank=0, core ids=[1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:36 [ompmultiprocessing.py:185] reserved_cpus=[23]
|
||||
W0922 13:34:40.073000 629853 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0922 13:34:40.099000 629853 .venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
INFO 09-22 13:34:41 [importing.py:53] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
|
||||
WARNING 09-22 13:34:41 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-22 13:34:41 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-22 13:34:42 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 13:34:42 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 13:34:42 [patch.py:640] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-22 13:34:42 [patch.py:750] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
[W922 13:34:42.576317158 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
WARNING 09-22 13:34:42 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(Worker pid=629853) WARNING 09-22 13:34:42 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(Worker pid=629853) INFO 09-22 13:34:42 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_18e922052e6e4cc9a96df0c8f74a41a3 backend=gloo
|
||||
(Worker pid=629853) INFO 09-22 13:34:43 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(Worker pid=629853) INFO 09-22 13:34:43 [cpu_model_runner.py:131] Starting to load model /home/zhout/project/edge_infer/models/Spark-X2.5-1.7B-int8...
|
||||
(Worker pid=629853) INFO 09-22 13:34:43 [__init__.py:755] Selected CPUInt8ScaledMMLinearKernel for CompressedTensorsW8A8Int8
|
||||
(Worker pid=629853) INFO 09-22 13:34:43 [selector.py:217] Using HND KV cache layout for CPU_ATTN backend.
|
||||
(Worker pid=629853) INFO 09-22 13:34:43 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 1.84 GiB. Available RAM: 26.90 GiB.
|
||||
(Worker pid=629853) INFO 09-22 13:34:43 [weight_utils.py:881] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(Worker pid=629853) ␍Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
|
||||
(Worker pid=629853) ␍Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.20s/it]
|
||||
(Worker pid=629853) ␍Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.20s/it]
|
||||
(Worker pid=629853)
|
||||
(Worker pid=629853) INFO 09-22 13:34:44 [default_loader.py:430] Loading weights took 1.22 seconds
|
||||
(Worker pid=629853) INFO 09-22 13:34:45 [memory.py:91] Released 104 MiB after weight load (anonymous RSS 3570 -> 3466 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) WARNING 09-22 13:34:45 [torch_utils.py:250] OMP_NUM_THREADS=8 is set; leaving Torch threads at 8 for serving. Multi-threaded torch CPU ops during serving can degrade performance through spin-wait contention and cgroup CPU-quota throttling.
|
||||
(Worker pid=629853) INFO 09-22 13:34:45 [cpu_worker.py:255] Explicitly set (0.27/30.91) GiB for KV cache on node 0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:45 [kv_cache_utils.py:1869] GPU KV cache size: 6,898 tokens, Maximum concurrency for 4,096 tokens per request: 1.68x
|
||||
(Worker pid=629853) INFO 09-22 13:34:45 [memory.py:91] Released -0 MiB after warm-up (anonymous RSS 3746 -> 3746 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:45 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.06 s
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) WARNING 09-22 13:34:46 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) WARNING 09-22 13:34:46 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) WARNING 09-22 13:34:46 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) WARNING 09-22 13:34:46 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:46 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:46 [patch.py:692] Restored explicit kv_cache_memory_bytes=293601280 (0.273 GiB); VLLM_CPU_KVCACHE_SPACE would have overridden it.
|
||||
INFO 09-22 13:34:46 [stage_runtime.py:943] [StageRuntime] Stage 0 engine startup completed
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:34:46 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
INFO 09-22 13:34:46 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
|
||||
INFO 09-22 13:34:46 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
|
||||
INFO 09-22 13:34:46 [stage_runtime.py:958] [StageRuntime] Stage 0 initialized
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
INFO 09-22 13:34:49 [orchestrator.py:604] [Orchestrator] Starting event loop
|
||||
INFO 09-22 13:34:49 [async_omni_engine.py:323] [AsyncOmniEngine] Orchestrator ready with 1 stages
|
||||
INFO 09-22 13:34:49 [omni_base.py:235] [AsyncOmni] AsyncOmniEngine initialized in 21.30 seconds
|
||||
INFO 09-22 13:34:49 [omni_base.py:255] [AsyncOmni] Initialized with 1 stages for model /home/zhout/project/edge_infer/models/Spark-X2.5-1.7B-int8
|
||||
loaded in 21.30 s
|
||||
WARNING 09-22 13:34:49 [input_processor.py:296] Passing raw prompts to InputProcessor is deprecated and will be removed in v0.18. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
|
||||
chat_short in=28 out=128 ttft=0.127s decode=16.41 tok/s
|
||||
chat_tools in=137 out=128 ttft=0.268s decode=16.29 tok/s
|
||||
chat_long in=2465 out=128 ttft=4.257s decode=15.51 tok/s
|
||||
plain_0 in=5 out=128 ttft=0.067s decode=16.66 tok/s
|
||||
plain_1 in=13 out=128 ttft=0.073s decode=16.43 tok/s
|
||||
plain_2 in=18 out=128 ttft=0.079s decode=16.38 tok/s
|
||||
plain_3 in=8 out=128 ttft=0.064s decode=15.84 tok/s
|
||||
plain_4 in=7 out=128 ttft=0.068s decode=16.78 tok/s
|
||||
plain_5 in=7 out=128 ttft=0.059s decode=16.51 tok/s
|
||||
plain_6 in=3 out=128 ttft=0.070s decode=16.03 tok/s
|
||||
plain_7 in=9 out=128 ttft=0.075s decode=16.22 tok/s
|
||||
plain_8 in=9 out=128 ttft=0.065s decode=16.48 tok/s
|
||||
INFO 09-22 13:36:28 [orchestrator.py:917] [Orchestrator] Aborted request(s) ['09e230fd09a74804-12-bce4064d']
|
||||
INFO 09-22 13:36:28 [orchestrator.py:917] [Orchestrator] Aborted request(s) ['09e230fd09a74804-12-bce4064d']
|
||||
INFO 09-22 13:36:28 [async_omni.py:665] [AsyncOmni] Request 09e230fd09a74804-12-bce4064d aborted.
|
||||
cancel: aborted=True after 8 events, leaked=0
|
||||
INFO 09-22 13:36:29 [omni_base.py:781] [AsyncOmni] Shutting down
|
||||
INFO 09-22 13:36:29 [async_omni_engine.py:1807] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
INFO 09-22 13:36:29 [orchestrator.py:705] [Orchestrator] Received shutdown signal
|
||||
INFO 09-22 13:36:29 [orchestrator.py:3092] [Orchestrator] Shutting down all 1 client(s)
|
||||
INFO 09-22 13:36:29 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-22 13:36:29 [core_client.py:691] [shutdown] MPClient: stopping engine manager
|
||||
INFO 09-22 13:36:29 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
(Worker pid=629853) INFO 09-22 13:36:29 [multiproc_executor.py:833] Parent process exited, terminating worker queues
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:36:29 [multiproc_executor.py:469] [shutdown] Executor: waiting for worker exit count=1
|
||||
(StageEngineCoreProc_stage0_replica0 pid=629819) INFO 09-22 13:36:31 [multiproc_executor.py:476] [shutdown] Executor: all workers exited gracefully
|
||||
INFO 09-22 13:36:34 [core_client.py:693] [shutdown] MPClient: engine manager stopped
|
||||
INFO 09-22 13:36:34 [core_client.py:694] [shutdown] MPClient: cleaning up background resources
|
||||
INFO 09-22 13:36:34 [core_client.py:696] [shutdown] MPClient: complete
|
||||
INFO 09-22 13:36:34 [stage_pool.py:1354] [StagePool] Stage 0 replica 0 shut down
|
||||
|
||||
acceptance: PASS ({'prompts': 12, 'min_tokens_required': 128, 'all_reached_min': True, 'cancel_clean': True})
|
||||
|
||||
wrote /home/zhout/project/edge_infer/analysis/experiments/model_device_matrix_20260922/spark-cpu-current.json
|
||||
+7812
File diff suppressed because it is too large.
Load diff
+149
@@ -0,0 +1,149 @@
|
||||
INFO 09-22 11:56:02 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 11:56:02 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 11:56:02 [patch.py:583] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
|
||||
INFO 09-22 11:56:08 [scheduler.py:277] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
INFO 09-22 11:56:08 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
--- Running test: test_the_masked_startup_reaches_a_decision
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
.--- Running test: test_a_refused_plan_cannot_be_started
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
.--- Running test: test_int8_is_refused_on_cuda_and_admitted_on_cpu
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
.INFO 09-22 11:56:26 [engine.py:392] [local] loading \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\Spark-X2.5-4B on cuda:0 with {'max_model_len': 4096, 'max_num_seqs': 1, 'max_num_batched_tokens': 2048, 'enforce_eager': True, 'enable_prefix_caching': False, 'gpu_memory_utilization': 0.53, 'kv_cache_memory_bytes': 754974720, 'dtype': 'bfloat16'}
|
||||
INFO 09-22 11:56:26 [omni_base.py:216] [AsyncOmni] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\Spark-X2.5-4B
|
||||
INFO 09-22 11:56:26 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\Spark-X2.5-4B
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
INFO 09-22 11:56:26 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 1 stages
|
||||
WARNING 09-22 11:56:27 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
INFO 09-22 11:56:27 [model.py:684] Resolved architecture: Spark2_5ForCausalLM
|
||||
INFO 09-22 11:56:27 [model.py:2021] Using max model len 4096
|
||||
WARNING 09-22 11:56:27 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-22 11:56:27 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-22 11:56:27 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
INFO 09-22 11:56:27 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
INFO 09-22 11:56:27 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-22 11:56:27 [stage_runtime.py:765] [stage_init] Stage-0 replica 0 physical devices None are not integer GPU ids; all serial LLM replicas will share one init group
|
||||
INFO 09-22 11:56:45 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 11:56:45 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 11:56:45 [patch.py:583] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:56:49 [core.py:123] Initializing a V1 LLM engine (v0.29.0) with config: model='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\Spark-X2.5-4B', speculative_config=None, tokenizer='\\\\wsl.localhost\\Ubuntu\\home\\zhout\\project\\edge_infer\\models\\Spark-X2.5-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=\\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\Spark-X2.5-4B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:56:50 [parallel_state.py:1775] world_size=1 rank=0 local_rank=0 distributed_init_method=file://C:\Users\zhout\AppData\Local\Temp/vllm_dist_cc490ec74fd54cbea88ba183b7556c4c backend=nccl
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) WARNING 09-22 11:56:50 [parallel_state.py:1788] Distributed backend nccl is not available; falling back to gloo.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:56:51 [parallel_state.py:2119] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) WARNING 09-22 11:56:51 [gpu_ar_worker.py:120] OMNI GPUARWorker forces v1 model runner for omni hooks.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:56:51 [topk_topp_sampler.py:46] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) WARNING 09-22 11:56:51 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) WARNING 09-22 11:56:51 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:56:51 [gpu_model_runner.py:5379] Starting to load model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\Spark-X2.5-4B...
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:56:52 [cuda.py:492] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:56:52 [flash_attn.py:897] Using FlashAttention version 2
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:56:53 [weight_utils.py:863] Filesystem type for checkpoints: unknown. Checkpoint size: 7.66 GiB. Available RAM: 9.69 GiB.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:56:53 [weight_utils.py:886] Auto-prefetch is disabled because the filesystem (unknown) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348)
|
||||
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348)
|
||||
Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:22<01:31, 23.00s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348)
|
||||
Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:47<01:11, 23.75s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348)
|
||||
Loading safetensors checkpoint shards: 60% Completed | 3/5 [01:10<00:47, 23.59s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348)
|
||||
Loading safetensors checkpoint shards: 80% Completed | 4/5 [01:34<00:23, 23.57s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 5/5 [01:37<00:00, 16.25s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348)
|
||||
Loading safetensors checkpoint shards: 100% Completed | 5/5 [01:37<00:00, 19.49s/it]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:30 [default_loader.py:430] Loading weights took 97.54 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:31 [gpu_model_runner.py:5471] Model loading took 7.66 GiB memory and 98.388669 seconds
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:31 [utils.py:306] Using LBNHC KV cache layout.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:32 [kv_cache_utils.py:2032] GPU KV cache size: 5,105 tokens, Maximum concurrency for 4,096 tokens per request: 1.25x
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) WARNING 09-22 11:58:32 [base.py:173] [LLM Worker 0] Sleep Mode DISABLED.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:32 [kernel_warmup.py:124] JIT kernel warmup starting.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348)
|
||||
JIT kernel warmup (1 compile keys): 0%| | 0/1 [00:00<?, ?kernel/s]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 1.31kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
JIT kernel warmup (1 compile keys): 100%|██████████| 1/1 [00:00<00:00, 1.31kernel/s, ComputeSlotMappingKernel (1 keys)]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:33 [kernel_warmup.py:134] JIT kernel warmup finished in 0.76s.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:34 [ll_bf16.py:30] cuteDSL (CUTLASS Python) not available, ll_bf16_gemm disabled
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:34 [jit_monitor.py:85] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:35 [torch_utils.py:277] Reducing Torch threads from 12 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:35 [core.py:368] init engine (profile, create kv cache, warmup model) took 3.41 s
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) WARNING 09-22 11:58:36 [scheduler.py:213] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARAsyncScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) WARNING 09-22 11:58:36 [omni_scheduler_mixin.py:118] OMNI scheduler forces v1 model runner for omni hooks.
|
||||
INFO 09-22 11:58:36 [stage_runtime.py:930] [StageRuntime] Stage 0 engine startup completed
|
||||
INFO 09-22 11:58:36 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) WARNING 09-22 11:58:36 [vllm.py:1371] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) WARNING 09-22 11:58:36 [vllm.py:1406] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:36 [kernel.py:369] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:36 [vllm.py:1585] Cudagraph is disabled under eager mode
|
||||
(StageEngineCoreProc_stage0_replica0 pid=13348) INFO 09-22 11:58:36 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
INFO 09-22 11:58:36 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
|
||||
INFO 09-22 11:58:36 [stage_runtime.py:945] [StageRuntime] Stage 0 initialized
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
|
||||
INFO 09-22 11:58:39 [orchestrator.py:604] [Orchestrator] Starting event loop
|
||||
INFO 09-22 11:58:39 [async_omni_engine.py:323] [AsyncOmniEngine] Orchestrator ready with 1 stages
|
||||
INFO 09-22 11:58:39 [omni_base.py:235] [AsyncOmni] AsyncOmniEngine initialized in 132.84 seconds
|
||||
INFO 09-22 11:58:39 [omni_base.py:255] [AsyncOmni] Initialized with 1 stages for model \\wsl.localhost\Ubuntu\home\zhout\project\edge_infer\models\Spark-X2.5-4B
|
||||
--- Running test: test_the_engine_loads_and_reports_where_it_landed
|
||||
.--- Running test: test_the_kv_pool_is_the_size_the_plan_asked_for
|
||||
.--- Running test: test_twelve_prompts_each_reach_128_tokens
|
||||
WARNING 09-22 11:58:39 [input_processor.py:315] Passing raw prompts to InputProcessor is deprecated and will be removed in the future. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
|
||||
.--- Running test: test_the_budget_bounded_the_measured_peak
|
||||
.--- Running test: test_usage_separates_what_was_measured_from_what_was_estimated
|
||||
.--- Running test: test_cancel_releases_and_delivers_nothing_from_the_retired_epoch
|
||||
INFO 09-22 11:59:34 [orchestrator.py:917] [Orchestrator] Aborted request(s) ['56e2a41427434ee8-12-b33cf904']
|
||||
INFO 09-22 11:59:34 [orchestrator.py:917] [Orchestrator] Aborted request(s) ['56e2a41427434ee8-12-b33cf904']
|
||||
INFO 09-22 11:59:34 [async_omni.py:665] [AsyncOmni] Request 56e2a41427434ee8-12-b33cf904 aborted.
|
||||
.--- Running test: test_a_retired_handle_cannot_be_resubmitted
|
||||
INFO 09-22 11:59:35 [orchestrator.py:917] [Orchestrator] Aborted request(s) ['d9c8163be55a49af-13-becca8b2']
|
||||
INFO 09-22 11:59:35 [orchestrator.py:917] [Orchestrator] Aborted request(s) ['d9c8163be55a49af-13-becca8b2']
|
||||
INFO 09-22 11:59:35 [async_omni.py:665] [AsyncOmni] Request d9c8163be55a49af-13-becca8b2 aborted.
|
||||
.--- Running test: test_the_engine_still_generates_after_a_cancel
|
||||
.INFO 09-22 11:59:37 [omni_base.py:781] [AsyncOmni] Shutting down
|
||||
INFO 09-22 11:59:37 [async_omni_engine.py:1807] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
INFO 09-22 11:59:37 [orchestrator.py:705] [Orchestrator] Received shutdown signal
|
||||
INFO 09-22 11:59:37 [orchestrator.py:3092] [Orchestrator] Shutting down all 1 client(s)
|
||||
INFO 09-22 11:59:37 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-22 11:59:37 [core_client.py:691] [shutdown] MPClient: stopping engine manager
|
||||
INFO 09-22 11:59:37 [utils.py:626] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
INFO 09-22 11:59:37 [core_client.py:693] [shutdown] MPClient: engine manager stopped
|
||||
INFO 09-22 11:59:37 [core_client.py:694] [shutdown] MPClient: cleaning up background resources
|
||||
INFO 09-22 11:59:37 [core_client.py:696] [shutdown] MPClient: complete
|
||||
INFO 09-22 11:59:37 [stage_pool.py:1354] [StagePool] Stage 0 replica 0 shut down
|
||||
|
||||
============================== warnings summary ===============================
|
||||
C:\Users\zhout\w2\omni029venv\Lib\site-packages\torch\jit\_script.py:365: 14 warnings
|
||||
C:\Users\zhout\w2\omni029venv\Lib\site-packages\torch\jit\_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
|
||||
warnings.warn(
|
||||
|
||||
tests/edge/local/test_e2e_local_text.py::test_the_masked_startup_reaches_a_decision
|
||||
C:\Users\zhout\w2\omni029venv\Lib\site-packages\pytest_asyncio\plugin.py:916: PytestDeprecationWarning: Overriding the "event_loop_policy" fixture is deprecated and will be removed in a future version of pytest-asyncio. Use the "pytest_asyncio_loop_factories" hook to customize event loop creation.
|
||||
warnings.warn(
|
||||
|
||||
tests/edge/local/test_e2e_local_text.py::test_the_masked_startup_reaches_a_decision
|
||||
tests\edge\local\test_e2e_local_text.py:109: PytestWarning: The test <Function test_the_masked_startup_reaches_a_decision> is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'.
|
||||
def test_the_masked_startup_reaches_a_decision(model_dir):
|
||||
|
||||
tests/edge/local/test_e2e_local_text.py::test_a_refused_plan_cannot_be_started
|
||||
tests\edge\local\test_e2e_local_text.py:124: PytestWarning: The test <Function test_a_refused_plan_cannot_be_started> is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'.
|
||||
def test_a_refused_plan_cannot_be_started(model_dir):
|
||||
|
||||
tests/edge/local/test_e2e_local_text.py::test_int8_is_refused_on_cuda_and_admitted_on_cpu
|
||||
tests\edge\local\test_e2e_local_text.py:133: PytestWarning: The test <Function test_int8_is_refused_on_cuda_and_admitted_on_cpu> is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'.
|
||||
@pytest.mark.skipif(
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
--- Running Summary
|
||||
11 passed, 18 warnings in 210.11s (0:03:30)
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
........... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
../.venvs/omni-cuda-029/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
|
||||
/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
|
||||
warnings.warn(
|
||||
|
||||
tests/edge/local/test_e2e_local_text.py::test_the_masked_startup_reaches_a_decision
|
||||
/home/zhout/project/edge_infer/.venvs/omni-cuda-029/lib/python3.12/site-packages/pytest_asyncio/plugin.py:916: PytestDeprecationWarning: Overriding the "event_loop_policy" fixture is deprecated and will be removed in a future version of pytest-asyncio. Use the "pytest_asyncio_loop_factories" hook to customize event loop creation.
|
||||
warnings.warn(
|
||||
|
||||
tests/edge/local/test_e2e_local_text.py::test_the_masked_startup_reaches_a_decision
|
||||
tests/edge/local/test_e2e_local_text.py:109: PytestWarning: The test <Function test_the_masked_startup_reaches_a_decision> is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'.
|
||||
def test_the_masked_startup_reaches_a_decision(model_dir):
|
||||
|
||||
tests/edge/local/test_e2e_local_text.py::test_a_refused_plan_cannot_be_started
|
||||
tests/edge/local/test_e2e_local_text.py:124: PytestWarning: The test <Function test_a_refused_plan_cannot_be_started> is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'.
|
||||
def test_a_refused_plan_cannot_be_started(model_dir):
|
||||
|
||||
tests/edge/local/test_e2e_local_text.py::test_int8_is_refused_on_cuda_and_admitted_on_cpu
|
||||
tests/edge/local/test_e2e_local_text.py:133: PytestWarning: The test <Function test_int8_is_refused_on_cuda_and_admitted_on_cpu> is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'.
|
||||
@pytest.mark.skipif(
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
--- Running Summary
|
||||
11 passed, 18 warnings in 261.25s (0:04:21)
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"scope": "repeatability of the same compiled W8A16 single attention component, not checkpoint quality or complete generation",
|
||||
"reference_job": "j57er3q9p",
|
||||
"job": "jp0mvkj6g",
|
||||
"model_id": "mnowl6xpq",
|
||||
"input_dataset_id": "d7m83gjy2",
|
||||
"outputs": [
|
||||
{
|
||||
"output": "output_0__0",
|
||||
"shape": [
|
||||
1,
|
||||
1,
|
||||
2048
|
||||
],
|
||||
"dtype": "float32",
|
||||
"finite": true,
|
||||
"max_absolute_difference": 0.0,
|
||||
"exact_equal": true
|
||||
},
|
||||
{
|
||||
"output": "output_1__0",
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
1,
|
||||
256
|
||||
],
|
||||
"dtype": "float32",
|
||||
"finite": true,
|
||||
"max_absolute_difference": 0.0,
|
||||
"exact_equal": true
|
||||
},
|
||||
{
|
||||
"output": "output_2__0",
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
1,
|
||||
256
|
||||
],
|
||||
"dtype": "float32",
|
||||
"finite": true,
|
||||
"max_absolute_difference": 0.0,
|
||||
"exact_equal": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"timestamp_utc": "2026-09-22T03:52:48.169689+00:00",
|
||||
"qai_hub_version": "0.55.0",
|
||||
"facility": "Qualcomm AI Hub Workbench",
|
||||
"command": "collect",
|
||||
"job_id": "jp0mvkj6g",
|
||||
"url": "https://workbench.aihub.qualcomm.com/jobs/jp0mvkj6g/",
|
||||
"status": "SUCCESS",
|
||||
"status_message": "",
|
||||
"options": "--compute_unit npu",
|
||||
"hub_version": "aihub-2026.09.11.0",
|
||||
"device": {
|
||||
"name": "Samsung Galaxy S25",
|
||||
"os": "15",
|
||||
"attributes": [
|
||||
"os:android",
|
||||
"framework:tflite",
|
||||
"framework:onnx",
|
||||
"abi:aarch64-android",
|
||||
"vendor:samsung",
|
||||
"format:phone",
|
||||
"chipset:qualcomm-snapdragon-8-elite-for-galaxy",
|
||||
"chipset:sm8750-ac",
|
||||
"hexagon:v79",
|
||||
"soc-model:69",
|
||||
"htp-supports-fp16:true",
|
||||
"htp-supports-weight-sharing:true",
|
||||
"framework:qnn"
|
||||
]
|
||||
},
|
||||
"model_id": "mnowl6xpq",
|
||||
"input_dataset_id": "d7m83gjy2",
|
||||
"outputs": "aihub-s25-infer.npz"
|
||||
}
|
||||
+419
@@ -0,0 +1,419 @@
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
W0922 13:40:19.393000 632538 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0922 13:40:19.417000 632538 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
INFO 09-22 13:40:20 [importing.py:53] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
|
||||
WARNING 09-22 13:40:20 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-22 13:40:20 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
INFO 09-22 13:40:21 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 13:40:21 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 13:40:21 [patch.py:640] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-22 13:40:21 [patch.py:750] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
INFO 09-22 13:40:22 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime.
|
||||
warnings.warn(
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:40:22 [configuration_qwen3_tts.py:483] talker_config is None. Initializing talker model with default values
|
||||
INFO 09-22 13:40:22 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:40:22 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
[transformers] Model config: codec_eos_token_id must be `None` or an integer within the vocabulary (between 0 and 3071), got 4198. This may result in unexpected behavior.
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:40:22 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
[transformers] You are using a model of type `qwen3_tts` to instantiate a model of type ``. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating.
|
||||
INFO 09-22 13:40:22 [omni_base.py:216] [Omni] Initializing with model /home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5
|
||||
INFO 09-22 13:40:22 [async_omni_engine.py:172] [AsyncOmniEngine] Initializing with model /home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5
|
||||
ERROR 09-22 13:40:22 [repo_utils.py:106] Error parsing config for /home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5: The checkpoint you are trying to load has model type `qwen3_tts` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
|
||||
ERROR 09-22 13:40:22 [repo_utils.py:106]
|
||||
ERROR 09-22 13:40:22 [repo_utils.py:106] You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`, retrying 1 of 2
|
||||
ERROR 09-22 13:40:24 [repo_utils.py:104] Error parsing config for /home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5: The checkpoint you are trying to load has model type `qwen3_tts` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
|
||||
ERROR 09-22 13:40:24 [repo_utils.py:104]
|
||||
ERROR 09-22 13:40:24 [repo_utils.py:104] You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`
|
||||
WARNING 09-22 13:40:24 [stage_config.py:845] Stage 0 env override replaces base env entirely (base type=NoneType, override type=dict)
|
||||
WARNING 09-22 13:40:24 [stage_config.py:845] Stage 1 env override replaces base env entirely (base type=NoneType, override type=dict)
|
||||
WARNING 09-22 13:40:24 [stage_config.py:845] Stage 0 env override replaces base env entirely (base type=NoneType, override type=dict)
|
||||
WARNING 09-22 13:40:24 [stage_config.py:845] Stage 1 env override replaces base env entirely (base type=NoneType, override type=dict)
|
||||
INFO 09-22 13:40:24 [async_omni_engine.py:290] [AsyncOmniEngine] Launching Orchestrator thread with 2 stages
|
||||
INFO 09-22 13:40:24 [initialization.py:356] Loaded OmniTransferConfig with 1 connector configurations
|
||||
WARNING 09-22 13:40:24 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:483] talker_config is None. Initializing talker model with default values
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
|
||||
INFO 09-22 13:40:24 [model.py:672] Resolved architecture: Qwen3TTSTalkerForConditionalGeneration
|
||||
INFO 09-22 13:40:24 [model.py:1965] Using max model len 2048
|
||||
INFO 09-22 13:40:24 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=2048.
|
||||
WARNING 09-22 13:40:24 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
WARNING 09-22 13:40:24 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
INFO 09-22 13:40:24 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
WARNING 09-22 13:40:24 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
INFO 09-22 13:40:24 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
WARNING 09-22 13:40:24 [utils.py:501] Dropping unknown OmniEngineArgs field 'inline_diffusion' (not declared on the dataclass)
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:483] talker_config is None. Initializing talker model with default values
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:40:24 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
INFO 09-22 13:40:24 [model.py:672] Resolved architecture: Qwen3TTSCode2Wav
|
||||
INFO 09-22 13:40:24 [model.py:1965] Using max model len 8192
|
||||
INFO 09-22 13:40:24 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=8192.
|
||||
INFO 09-22 13:40:24 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
WARNING 09-22 13:40:24 [stage_runtime.py:778] [stage_init] Stage-0 replica 0 physical devices None are not integer GPU ids; all serial LLM replicas will share one init group
|
||||
WARNING 09-22 13:40:24 [stage_runtime.py:778] [stage_init] Stage-1 replica 0 physical devices None are not integer GPU ids; all serial LLM replicas will share one init group
|
||||
INFO 09-22 13:40:24 [stage_init_utils.py:1038] [stage_init] Stage-0 applied runtime env keys: ['VLLM_CPU_OMP_THREADS_BIND']
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
W0922 13:40:29.596000 632642 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0922 13:40:29.619000 632642 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
INFO 09-22 13:40:30 [importing.py:53] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
|
||||
WARNING 09-22 13:40:30 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-22 13:40:30 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
INFO 09-22 13:40:31 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 13:40:31 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 13:40:31 [patch.py:640] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-22 13:40:31 [patch.py:750] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:32 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5', speculative_config=None, tokenizer='/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:32 [multiproc_executor.py:150] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.21.141.90 (local), world_size=1, local_world_size=1
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:32 [ompmultiprocessing.py:185] OpenMP thread binding info:
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:32 [ompmultiprocessing.py:185] VLLM_CPU_OMP_THREADS_BIND='0-11', auto_setup=False, skip_setup=False
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:32 [ompmultiprocessing.py:185] local_world_size=1, reserve_cpu_num=1
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:32 [ompmultiprocessing.py:185] local_rank=0, core ids=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:32 [ompmultiprocessing.py:185] reserved_cpus=[]
|
||||
W0922 13:40:36.009000 632722 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0922 13:40:36.032000 632722 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
INFO 09-22 13:40:36 [importing.py:53] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
|
||||
WARNING 09-22 13:40:36 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-22 13:40:36 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-22 13:40:38 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 13:40:38 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 13:40:38 [patch.py:640] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-22 13:40:38 [patch.py:750] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
[W922 13:40:38.266751546 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
WARNING 09-22 13:40:38 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(Worker pid=632722) WARNING 09-22 13:40:38 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(Worker pid=632722) INFO 09-22 13:40:38 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_f2ee15a3a46745df80d7c175951a35cb backend=gloo
|
||||
(Worker pid=632722) INFO 09-22 13:40:38 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(Worker pid=632722) INFO 09-22 13:40:38 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(Worker pid=632722) INFO 09-22 13:40:38 [cpu_model_runner.py:131] Starting to load model /home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5...
|
||||
(Worker pid=632722) INFO 09-22 13:40:38 [selector.py:217] Using HND KV cache layout for CPU_ATTN backend.
|
||||
(Worker pid=632722) INFO 09-22 13:40:40 [configuration_qwen3_tts_tokenizer_v2.py:156] encoder_config is None. Initializing encoder with default values
|
||||
(Worker pid=632722) INFO 09-22 13:40:40 [configuration_qwen3_tts_tokenizer_v2.py:159] decoder_config is None. Initializing decoder with default values
|
||||
(Worker pid=632722) INFO 09-22 13:40:41 [speaker_cache.py:239] Speaker cache ready (max_bytes=536870912)
|
||||
(Worker pid=632722) INFO 09-22 13:40:41 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 1.69 GiB. Available RAM: 25.49 GiB.
|
||||
(Worker pid=632722) INFO 09-22 13:40:41 [weight_utils.py:881] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(Worker pid=632722) ␍Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
|
||||
(Worker pid=632722) ␍Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.02s/it]
|
||||
(Worker pid=632722) ␍Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.02s/it]
|
||||
(Worker pid=632722)
|
||||
(Worker pid=632722) INFO 09-22 13:40:42 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 0.64 GiB. Available RAM: 24.75 GiB.
|
||||
(Worker pid=632722) ␍Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
|
||||
(Worker pid=632722) ␍Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.22it/s]
|
||||
(Worker pid=632722) ␍Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.21it/s]
|
||||
(Worker pid=632722)
|
||||
(Worker pid=632722) INFO 09-22 13:40:43 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 0.64 GiB. Available RAM: 24.75 GiB.
|
||||
(Worker pid=632722) ␍Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
|
||||
(Worker pid=632722) ␍Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 135.34it/s]
|
||||
(Worker pid=632722)
|
||||
(Worker pid=632722) INFO 09-22 13:40:43 [qwen3_tts_talker.py:1192] Loaded 604 weights for Qwen3TTSTalkerForConditionalGeneration
|
||||
(Worker pid=632722) INFO 09-22 13:40:43 [default_loader.py:430] Loading weights took 1.52 seconds
|
||||
(Worker pid=632722) INFO 09-22 13:40:43 [memory.py:91] Released 185 MiB after weight load (anonymous RSS 2683 -> 2498 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:43 [torch_utils.py:262] Reducing Torch threads from 12 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
|
||||
(Worker pid=632722) INFO 09-22 13:40:43 [cpu_worker.py:255] Explicitly set (0.62/30.91) GiB for KV cache on node 0.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:43 [kv_cache_utils.py:1869] GPU KV cache size: 5,760 tokens, Maximum concurrency for 2,048 tokens per request: 2.81x
|
||||
(Worker pid=632722) INFO 09-22 13:40:43 [memory.py:91] Released 0 MiB after warm-up (anonymous RSS 3128 -> 3128 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:43 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.11 s
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:43 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) /home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) warnings.warn(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) [transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:43 [configuration_qwen3_tts.py:483] talker_config is None. Initializing talker model with default values
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:43 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) [transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:43 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) [transformers] Model config: codec_eos_token_id must be `None` or an integer within the vocabulary (between 0 and 3071), got 4198. This may result in unexpected behavior.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) [transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:43 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) WARNING 09-22 13:40:43 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) WARNING 09-22 13:40:43 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:43 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) WARNING 09-22 13:40:44 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) WARNING 09-22 13:40:44 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:44 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
INFO 09-22 13:40:44 [stage_runtime.py:943] [StageRuntime] Stage 0 engine startup completed
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:40:44 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
INFO 09-22 13:40:44 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
|
||||
INFO 09-22 13:40:44 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
|
||||
INFO 09-22 13:40:44 [stage_runtime.py:958] [StageRuntime] Stage 0 initialized
|
||||
INFO 09-22 13:40:44 [stage_init_utils.py:1038] [stage_init] Stage-1 applied runtime env keys: ['VLLM_CPU_OMP_THREADS_BIND']
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
W0922 13:40:49.688000 632816 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0922 13:40:49.713000 632816 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
INFO 09-22 13:40:50 [importing.py:53] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
|
||||
WARNING 09-22 13:40:50 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-22 13:40:50 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
INFO 09-22 13:40:51 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 13:40:52 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 13:40:52 [patch.py:640] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-22 13:40:52 [patch.py:750] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:40:52 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5', speculative_config=None, tokenizer='/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:40:52 [multiproc_executor.py:150] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.21.141.90 (local), world_size=1, local_world_size=1
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:40:52 [ompmultiprocessing.py:185] OpenMP thread binding info:
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:40:52 [ompmultiprocessing.py:185] VLLM_CPU_OMP_THREADS_BIND='12-15', auto_setup=False, skip_setup=False
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:40:52 [ompmultiprocessing.py:185] local_world_size=1, reserve_cpu_num=1
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:40:52 [ompmultiprocessing.py:185] local_rank=0, core ids=[12, 13, 14, 15]
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:40:52 [ompmultiprocessing.py:185] reserved_cpus=[]
|
||||
W0922 13:40:56.415000 632906 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
W0922 13:40:56.439000 632906 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
|
||||
INFO 09-22 13:40:57 [importing.py:53] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
|
||||
WARNING 09-22 13:40:57 [importing.py:65] Triton is installed, but doesn't include CPU backend. Disabling Triton.
|
||||
INFO 09-22 13:40:57 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
|
||||
/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/version.py:55: RuntimeWarning: vLLM and vLLM-Omni appear to have mismatched major/minor versions:
|
||||
--> vLLM-Omni version 0.29.0rc2.dev71+g00510ce38.d20260922
|
||||
--> vLLM version 0.28.0
|
||||
This will likely cause compatibility issues.
|
||||
warn_if_misaligned_vllm_version()
|
||||
INFO 09-22 13:40:58 [patch.py:259] NVFP4 W4A4 weight_scale NaN-clamp: installed.
|
||||
INFO 09-22 13:40:58 [patch.py:510] inductor factorable-divisibility patch: installed.
|
||||
INFO 09-22 13:40:58 [patch.py:640] CPU AOT compile-cache load shim installed: compiled models are reused across engine starts instead of being recompiled.
|
||||
INFO 09-22 13:40:58 [patch.py:750] CPU weight-load memory release installed (set VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable).
|
||||
[W922 13:40:58.645946482 utils.cpp:68] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_memory_env)
|
||||
WARNING 09-22 13:40:59 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(Worker pid=632906) WARNING 09-22 13:40:59 [cpu_worker.py:127] libiomp is not found in LD_PRELOAD. For best performance, please follow the section `set LD_PRELOAD` in https://docs.vllm.ai/en/latest/getting_started/installation/cpu/ to setup required pre-loaded libraries.
|
||||
(Worker pid=632906) INFO 09-22 13:40:59 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_3961b9cb10014f5ea60bbb120b406eb5 backend=gloo
|
||||
(Worker pid=632906) INFO 09-22 13:40:59 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
|
||||
(Worker pid=632906) INFO 09-22 13:40:59 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(Worker pid=632906) INFO 09-22 13:40:59 [cpu_model_runner.py:131] Starting to load model /home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5...
|
||||
(Worker pid=632906) INFO 09-22 13:40:59 [configuration_qwen3_tts_tokenizer_v2.py:156] encoder_config is None. Initializing encoder with default values
|
||||
(Worker pid=632906) INFO 09-22 13:40:59 [configuration_qwen3_tts_tokenizer_v2.py:159] decoder_config is None. Initializing decoder with default values
|
||||
(Worker pid=632906) INFO 09-22 13:40:59 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 1.69 GiB. Available RAM: 22.89 GiB.
|
||||
(Worker pid=632906) INFO 09-22 13:40:59 [weight_utils.py:881] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
|
||||
(Worker pid=632906) ␍Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
|
||||
(Worker pid=632906) ␍Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 118.99it/s]
|
||||
(Worker pid=632906)
|
||||
(Worker pid=632906) INFO 09-22 13:40:59 [weight_utils.py:858] Filesystem type for checkpoints: EXT4. Checkpoint size: 0.64 GiB. Available RAM: 22.89 GiB.
|
||||
(Worker pid=632906) ␍Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
|
||||
(Worker pid=632906) ␍Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 13.44it/s]
|
||||
(Worker pid=632906)
|
||||
(Worker pid=632906) INFO 09-22 13:41:00 [modeling_qwen3_tts_tokenizer_v2.py:865] Precomputed exp caches for 29 SnakeBeta activations
|
||||
(Worker pid=632906) INFO 09-22 13:41:00 [default_loader.py:430] Loading weights took 0.18 seconds
|
||||
(Worker pid=632906) INFO 09-22 13:41:00 [memory.py:91] Released 36 MiB after weight load (anonymous RSS 1015 -> 979 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [torch_utils.py:262] Reducing Torch threads from 12 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
|
||||
(Worker pid=632906) INFO 09-22 13:41:00 [memory.py:91] Released 0 MiB after warm-up (anonymous RSS 979 -> 979 MiB) via ['libc.so.6:malloc_trim', 'libtcmalloc_minimal.so.4:MallocExtension_ReleaseFreeMemory']
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [core.py:354] init engine (profile, create kv cache, warmup model) took 0.00 s
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) /home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) warnings.warn(
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) [transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [configuration_qwen3_tts.py:483] talker_config is None. Initializing talker model with default values
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) [transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) [transformers] Model config: codec_eos_token_id must be `None` or an integer within the vocabulary (between 0 and 3071), got 4198. This may result in unexpected behavior.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) [transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) WARNING 09-22 13:41:00 [scheduler.py:182] Using custom scheduler class vllm_omni.core.sched.omni_generation_scheduler.OmniGenerationScheduler. This scheduler interface is not public and compatibility may not be maintained. If you have subclassed Scheduler instead of AsyncScheduler, you will see degraded performance due to async scheduling being disabled.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) WARNING 09-22 13:41:00 [core.py:154] Disabling chunked prefill for model without KVCache
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) WARNING 09-22 13:41:00 [vllm.py:652] Model Runner V2 requires Triton; using the V1 model runner instead.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [factory.py:46] Created connector: SharedMemoryConnector
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) WARNING 09-22 13:41:00 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) WARNING 09-22 13:41:00 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:00 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant
|
||||
INFO 09-22 13:41:00 [stage_runtime.py:943] [StageRuntime] Stage 1 engine startup completed
|
||||
INFO 09-22 13:41:00 [stage_engine_core_client.py:159] [StageEngineCoreClient] stage-1 [rep-0] initializing EngineCore
|
||||
INFO 09-22 13:41:00 [stage_engine_core_client.py:215] [StageEngineCoreClient] stage-1 [rep-0] EngineCore running
|
||||
INFO 09-22 13:41:00 [stage_runtime.py:958] [StageRuntime] Stage 1 initialized
|
||||
INFO 09-22 13:41:00 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:41:00 [configuration_qwen3_tts.py:483] talker_config is None. Initializing talker model with default values
|
||||
INFO 09-22 13:41:00 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:41:00 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
[transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
INFO 09-22 13:41:00 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
|
||||
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
|
||||
INFO 09-22 13:41:02 [orchestrator.py:604] [Orchestrator] Starting event loop
|
||||
INFO 09-22 13:41:02 [async_omni_engine.py:323] [AsyncOmniEngine] Orchestrator ready with 2 stages
|
||||
INFO 09-22 13:41:02 [omni_base.py:235] [Omni] AsyncOmniEngine initialized in 39.58 seconds
|
||||
INFO 09-22 13:41:02 [omni_base.py:255] [Omni] Initialized with 2 stages for model /home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5
|
||||
WARNING 09-22 13:41:02 [input_processor.py:296] Passing raw prompts to InputProcessor is deprecated and will be removed in v0.18. You should instead pass the outputs of Renderer.render_cmpl() or Renderer.render_chat().
|
||||
␍Processed prompts: 0%| | 0/1 [00:00<?, ?it/s](Worker pid=632722) INFO 09-22 13:41:02 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
(Worker pid=632722) /home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/transformers/modeling_rope_utils.py:1036: FutureWarning: `rope_config_validation` is deprecated and has been removed. Its functionality has been moved to RotaryEmbeddingConfigMixin.validate_rope method. PreTrainedConfig inherits this class, so please call self.validate_rope() instead. Also, make sure to use the new rope_parameters syntax. You can call self.standardize_rope_params() in the meantime.
|
||||
(Worker pid=632722) warnings.warn(
|
||||
(Worker pid=632722) [transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
(Worker pid=632722) INFO 09-22 13:41:02 [configuration_qwen3_tts.py:483] talker_config is None. Initializing talker model with default values
|
||||
(Worker pid=632722) INFO 09-22 13:41:02 [configuration_qwen3_tts.py:486] speaker_encoder_config is None. Initializing talker model with default values
|
||||
(Worker pid=632722) [transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
(Worker pid=632722) INFO 09-22 13:41:02 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
(Worker pid=632722) [transformers] Model config: codec_eos_token_id must be `None` or an integer within the vocabulary (between 0 and 3071), got 4198. This may result in unexpected behavior.
|
||||
(Worker pid=632722) [transformers] `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead
|
||||
(Worker pid=632722) INFO 09-22 13:41:02 [configuration_qwen3_tts.py:437] code_predictor_config is None. Initializing code_predictor model with default values
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] WorkerProc hit an exception.
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] Traceback (most recent call last):
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 1039, in worker_busy_loop
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] output = func(*args, **kwargs)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] return func(*args, **kwargs)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1049, in sample_tokens
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] return self.model_runner.sample_tokens(grammar_output)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] return func(*args, **kwargs)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/worker/gpu_ar_model_runner.py", line 2104, in sample_tokens
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] sanitize_min_tokens_stop_ids(
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/worker/sampling_utils.py", line 51, in sanitize_min_tokens_stop_ids
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] for _, _, stop_tok_ids, _ in min_toks.values():
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ValueError: not enough values to unpack (expected 4, got 3)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] Traceback (most recent call last):
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 1039, in worker_busy_loop
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] output = func(*args, **kwargs)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] return func(*args, **kwargs)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 1049, in sample_tokens
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] return self.model_runner.sample_tokens(grammar_output)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] return func(*args, **kwargs)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/worker/gpu_ar_model_runner.py", line 2104, in sample_tokens
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] sanitize_min_tokens_stop_ids(
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/worker/sampling_utils.py", line 51, in sanitize_min_tokens_stop_ids
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] for _, _, stop_tok_ids, _ in min_toks.values():
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ^^^^^^^^^^^^^^^^^^^^^
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047] ValueError: not enough values to unpack (expected 4, got 3)
|
||||
(Worker pid=632722) ERROR 09-22 13:41:02 [multiproc_executor.py:1047]
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.28.0) with config: model='/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5', speculative_config=None, tokenizer='/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto'),
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [dump_input.py:79] Dumping scheduler output for model execution: OmniSchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=0_0406560b-92cb-49a0-ab5a-adf87321faa0,prompt_token_ids_len=15,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.9, top_p=1.0, top_k=50, min_p=0.0, seed=None, stop=[], stop_token_ids=[2150], bad_words=[], thinking_token_budget=None, include_stop_str_in_output=False, ignore_eos=False, max_tokens=2, min_tokens=2, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, structured_outputs=None, extra_args=None),block_ids=([1],),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_scheduled_tokens={0_0406560b-92cb-49a0-ab5a-adf87321faa0: 15}, total_num_scheduled_tokens=15, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_mm_hashes=[], scheduled_encoder_input_stats=null, preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=null, ec_connector_metadata=null, ec_manager_metadata=null, new_block_ids_to_zero=null, kv_cache_block_copies=null, partial_tail_offloads=null, num_spec_tokens_to_schedule=0, finished_requests_needing_kv_transfer={}, pending_input_registrations=[])
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] StageEngineCoreProc encountered a fatal error.
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] Traceback (most recent call last):
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_engine_core_proc.py", line 235, in run_stage_core
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] engine_core.run_busy_loop()
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/fault_tolerance/engine_core_sentinel.py", line 179, in run_with_fault_tolerance
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] busy_loop_func(self)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1383, in run_busy_loop
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] self._process_engine_step()
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1436, in _process_engine_step
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] outputs, model_executed = self.step_fn()
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] ^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 603, in step
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] model_output = self.model_executor.sample_tokens(grammar_output)
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 353, in sample_tokens
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] return self.collective_rpc(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] ^^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 445, in collective_rpc
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] return future if non_block else future.result()
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] ^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 99, in result
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] return super().result()
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] ^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 449, in result
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] return self.__get_result()
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] ^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] raise self._exception
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 103, in _wait_for_response
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] response = self.aggregate(self.get_response())
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] ^^^^^^^^^^^^^^^^^^^
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 434, in get_response
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] raise RuntimeError(
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) ERROR 09-22 13:41:02 [stage_engine_core_proc.py:244] RuntimeError: Worker failed with error 'not enough values to unpack (expected 4, got 3)', please check the stack trace above for the root cause
|
||||
(Worker pid=632722) INFO 09-22 13:41:02 [multiproc_executor.py:833] Parent process exited, terminating worker queues
|
||||
(StageEngineCoreProc_stage0_replica0 pid=632642) INFO 09-22 13:41:02 [multiproc_executor.py:469] [shutdown] Executor: waiting for worker exit count=1
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] [StagePool] _poll_stage_raw failed for stage-0 replica-0
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] Traceback (most recent call last):
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_pool.py", line 1195, in poll_llm_raw_output
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] return await asyncio.wait_for(
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] ^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] File "/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] return await fut
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] ^^^^^^^^^
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/engine/stage_pool.py", line 1142, in _poll_stage_raw
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] outputs = await client.get_output_async()
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] File "/home/zhout/project/edge_infer/.venvs/omni-cpu/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1104, in get_output_async
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] raise self._format_exception(outputs) from None
|
||||
ERROR 09-22 13:41:02 [stage_pool.py:1204] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
|
||||
ERROR 09-22 13:41:02 [orchestrator.py:1604] [Orchestrator] Stage-0 replica-0 is dead; evicting it and failing its in-flight requests: EngineCore encountered an issue. See stack trace (above) for the root cause.
|
||||
INFO 09-22 13:41:02 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-22 13:41:02 [core_client.py:691] [shutdown] MPClient: stopping engine manager
|
||||
INFO 09-22 13:41:02 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage0_replica0
|
||||
INFO 09-22 13:41:02 [core_client.py:693] [shutdown] MPClient: engine manager stopped
|
||||
INFO 09-22 13:41:02 [core_client.py:694] [shutdown] MPClient: cleaning up background resources
|
||||
INFO 09-22 13:41:02 [core_client.py:696] [shutdown] MPClient: complete
|
||||
INFO 09-22 13:41:02 [stage_pool.py:1354] [StagePool] Stage 0 replica 0 shut down
|
||||
␍Processed prompts: 0%| | 0/1 [00:00<?, ?it/s]
|
||||
ERROR 09-22 13:41:02 [omni.py:92] [Omni] Failed to run generation: EngineCore encountered an issue. See stack trace (above) for the root cause.
|
||||
ERROR 09-22 13:41:02 [omni.py:92] Traceback (most recent call last):
|
||||
ERROR 09-22 13:41:02 [omni.py:92] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni.py", line 90, in generate
|
||||
ERROR 09-22 13:41:02 [omni.py:92] return list(self._run_generation(prompts, sampling_params_list, use_tqdm))
|
||||
ERROR 09-22 13:41:02 [omni.py:92] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-22 13:41:02 [omni.py:92] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni.py", line 173, in _run_generation
|
||||
ERROR 09-22 13:41:02 [omni.py:92] should_continue, req_id, stage_id, req_state = self._handle_output_message(msg)
|
||||
ERROR 09-22 13:41:02 [omni.py:92] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
ERROR 09-22 13:41:02 [omni.py:92] File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni_base.py", line 516, in _handle_output_message
|
||||
ERROR 09-22 13:41:02 [omni.py:92] raise OmniEngineDeadError(
|
||||
ERROR 09-22 13:41:02 [omni.py:92] vllm_omni.entrypoints.omni_base.OmniEngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
|
||||
INFO 09-22 13:41:02 [orchestrator.py:917] [Orchestrator] Aborted request(s) ['0_0406560b-92cb-49a0-ab5a-adf87321faa0']
|
||||
INFO 09-22 13:41:02 [omni_base.py:781] [Omni] Shutting down
|
||||
INFO 09-22 13:41:02 [async_omni_engine.py:1807] [AsyncOmniEngine] Shutting down Orchestrator
|
||||
INFO 09-22 13:41:02 [orchestrator.py:705] [Orchestrator] Received shutdown signal
|
||||
INFO 09-22 13:41:02 [orchestrator.py:3092] [Orchestrator] Shutting down all 1 client(s)
|
||||
INFO 09-22 13:41:02 [core_client.py:689] [shutdown] MPClient: start timeout=default
|
||||
INFO 09-22 13:41:02 [utils.py:615] [shutdown] Process manager: send sigterm to process StageEngineCoreProc_stage1_replica0
|
||||
(Worker pid=632906) INFO 09-22 13:41:02 [multiproc_executor.py:833] Parent process exited, terminating worker queues
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:02 [multiproc_executor.py:469] [shutdown] Executor: waiting for worker exit count=1
|
||||
(StageEngineCoreProc_stage1_replica0 pid=632816) INFO 09-22 13:41:04 [multiproc_executor.py:476] [shutdown] Executor: all workers exited gracefully
|
||||
INFO 09-22 13:41:07 [stage_pool.py:1354] [StagePool] Stage 1 replica 0 shut down
|
||||
Traceback (most recent call last):
|
||||
File "/home/zhout/project/edge_infer/analysis/experiments/model_device_matrix_20260922/tts_cpu_probe.py", line 36, in main
|
||||
outputs = engine.generate({"prompt_token_ids": [0] * length, "additional_information": info}, params)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni.py", line 90, in generate
|
||||
return list(self._run_generation(prompts, sampling_params_list, use_tqdm))
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni.py", line 173, in _run_generation
|
||||
should_continue, req_id, stage_id, req_state = self._handle_output_message(msg)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni_base.py", line 516, in _handle_output_message
|
||||
raise OmniEngineDeadError(
|
||||
vllm_omni.entrypoints.omni_base.OmniEngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
|
||||
INFO 09-22 13:41:07 [omni_base.py:781] [Omni] Shutting down
|
||||
failed
|
||||
/home/zhout/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
|
||||
warnings.warn('resource_tracker: There appear to be %d '
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"model": "/home/zhout/.cache/huggingface/hub/models--Qwen--Qwen3-TTS-12Hz-0.6B-CustomVoice/snapshots/85e237c12c027371202489a0ec509ded67b5e4b5",
|
||||
"scope": "two codec frames only; no speech-quality claim",
|
||||
"start": 1790055615.9570496,
|
||||
"status": "failed",
|
||||
"error": "OmniEngineDeadError('EngineCore encountered an issue. See stack trace (above) for the root cause.')",
|
||||
"traceback": "Traceback (most recent call last):\n File \"/home/zhout/project/edge_infer/analysis/experiments/model_device_matrix_20260922/tts_cpu_probe.py\", line 36, in main\n outputs = engine.generate({\"prompt_token_ids\": [0] * length, \"additional_information\": info}, params)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni.py\", line 90, in generate\n return list(self._run_generation(prompts, sampling_params_list, use_tqdm))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni.py\", line 173, in _run_generation\n should_continue, req_id, stage_id, req_state = self._handle_output_message(msg)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/home/zhout/project/edge_infer/vllm-omni-edge/vllm_omni/entrypoints/omni_base.py\", line 516, in _handle_output_message\n raise OmniEngineDeadError(\nvllm_omni.entrypoints.omni_base.OmniEngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.\n",
|
||||
"end": 1790055667.554795
|
||||
}
|
||||
+218
@@ -0,0 +1,218 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"meta": {
|
||||
"model": "C:/Users/zhout/w2/models/Qwen3-TTS-0.6B-85e237c",
|
||||
"deploy_config": null,
|
||||
"query_type": "CustomVoice",
|
||||
"txt_prompts": "../analysis/experiments/omni_abstraction_20260922/tts-prompts.txt",
|
||||
"git_sha": null,
|
||||
"gpu_name": "NVIDIA GeForce RTX 5090 Laptop GPU",
|
||||
"cuda_visible_devices": null,
|
||||
"timestamp": "20260922-123626",
|
||||
"argv": [
|
||||
"--query-type",
|
||||
"CustomVoice",
|
||||
"--txt-prompts",
|
||||
"../analysis/experiments/omni_abstraction_20260922/tts-prompts.txt"
|
||||
],
|
||||
"bench_args": {
|
||||
"model": "C:/Users/zhout/w2/models/Qwen3-TTS-0.6B-85e237c",
|
||||
"warmup": 1,
|
||||
"repeat": 1,
|
||||
"init_only": false,
|
||||
"tag": "abstraction",
|
||||
"output_dir": "../analysis/experiments/omni_abstraction_20260922/tts-stream-windows",
|
||||
"save_wav": true,
|
||||
"gpu_poll_interval_s": 0.2,
|
||||
"parallel_stage_init": false,
|
||||
"deploy_profile": "edge",
|
||||
"step_stats": false,
|
||||
"bench_help": false
|
||||
},
|
||||
"warmup": 1,
|
||||
"repeat": 1,
|
||||
"init_only": false
|
||||
},
|
||||
"init": {
|
||||
"init_s": 60.429562699981034,
|
||||
"init_timeline": null
|
||||
},
|
||||
"step_stats": null,
|
||||
"resources": {
|
||||
"peak_rss_bytes": 6449074176,
|
||||
"peak_gpu_mem_mib": 14545,
|
||||
"gpu_mem_samples": 114,
|
||||
"gpu_mem_mode": "whole-gpu"
|
||||
},
|
||||
"requests": [
|
||||
{
|
||||
"request_id": "w0",
|
||||
"sr": 24000,
|
||||
"chunks": [
|
||||
{
|
||||
"idx": 0,
|
||||
"t_ms": 693.0269000004046,
|
||||
"gap_ms": 693.0269000004046,
|
||||
"samples": 3840
|
||||
},
|
||||
{
|
||||
"idx": 1,
|
||||
"t_ms": 1116.0665999632329,
|
||||
"gap_ms": 423.0396999628283,
|
||||
"samples": 7680
|
||||
},
|
||||
{
|
||||
"idx": 2,
|
||||
"t_ms": 2043.6592999612913,
|
||||
"gap_ms": 927.5926999980584,
|
||||
"samples": 15360
|
||||
},
|
||||
{
|
||||
"idx": 3,
|
||||
"t_ms": 3894.4398000021465,
|
||||
"gap_ms": 1850.7805000408553,
|
||||
"samples": 30720
|
||||
},
|
||||
{
|
||||
"idx": 4,
|
||||
"t_ms": 6838.359799992759,
|
||||
"gap_ms": 2943.9199999906123,
|
||||
"samples": 48000
|
||||
},
|
||||
{
|
||||
"idx": 5,
|
||||
"t_ms": 9425.78130000038,
|
||||
"gap_ms": 2587.421500007622,
|
||||
"samples": 48000
|
||||
}
|
||||
],
|
||||
"t_final_ms": 11889.202099991962,
|
||||
"ttfa_ms": 693.0269000004046,
|
||||
"playback_start_ms": 5025.781300000381,
|
||||
"stall_at_ttfa_ms": 4332.754399999976,
|
||||
"n_chunks": 6,
|
||||
"streamed_audio_s": 6.4,
|
||||
"tail_audio_s": 1.76,
|
||||
"total_wall_s": 11.889202099991962,
|
||||
"rtf_streamed": 1.4727783281250595,
|
||||
"rtf_total": 1.4570100612735248,
|
||||
"gap_ms": [
|
||||
423.0396999628283,
|
||||
927.5926999980584,
|
||||
1850.7805000408553,
|
||||
2943.9199999906123,
|
||||
2587.421500007622
|
||||
],
|
||||
"text": "\u4eca\u5929\u6211\u4eec\u9a8c\u8bc1\u672c\u5730\u8bed\u97f3\u751f\u6210\u3002\u6a21\u578b\u5728\u540c\u4e00\u53f0\u8bbe\u5907\u4e0a\u8fd0\u884c\uff0c\u97f3\u9891\u6309\u7167\u987a\u5e8f\u8fde\u7eed\u8f93\u51fa\u3002",
|
||||
"warmup": true
|
||||
},
|
||||
{
|
||||
"request_id": "r0_0",
|
||||
"sr": 24000,
|
||||
"chunks": [
|
||||
{
|
||||
"idx": 0,
|
||||
"t_ms": 387.35440000891685,
|
||||
"gap_ms": 387.35440000891685,
|
||||
"samples": 3840
|
||||
},
|
||||
{
|
||||
"idx": 1,
|
||||
"t_ms": 765.4019000474364,
|
||||
"gap_ms": 378.0475000385195,
|
||||
"samples": 7680
|
||||
},
|
||||
{
|
||||
"idx": 2,
|
||||
"t_ms": 1700.6567000062205,
|
||||
"gap_ms": 935.2547999587841,
|
||||
"samples": 15360
|
||||
},
|
||||
{
|
||||
"idx": 3,
|
||||
"t_ms": 3457.911500008777,
|
||||
"gap_ms": 1757.2548000025563,
|
||||
"samples": 30720
|
||||
},
|
||||
{
|
||||
"idx": 4,
|
||||
"t_ms": 6172.567500034347,
|
||||
"gap_ms": 2714.6560000255704,
|
||||
"samples": 48000
|
||||
},
|
||||
{
|
||||
"idx": 5,
|
||||
"t_ms": 8866.196100017987,
|
||||
"gap_ms": 2693.62859998364,
|
||||
"samples": 48000
|
||||
}
|
||||
],
|
||||
"t_final_ms": 11508.378600003198,
|
||||
"ttfa_ms": 387.35440000891685,
|
||||
"playback_start_ms": 4466.196100017987,
|
||||
"stall_at_ttfa_ms": 4078.8417000090703,
|
||||
"n_chunks": 6,
|
||||
"streamed_audio_s": 6.4,
|
||||
"tail_audio_s": 1.92,
|
||||
"total_wall_s": 11.508378600003198,
|
||||
"rtf_streamed": 1.3853431406278105,
|
||||
"rtf_total": 1.3832185817311535,
|
||||
"gap_ms": [
|
||||
378.0475000385195,
|
||||
935.2547999587841,
|
||||
1757.2548000025563,
|
||||
2714.6560000255704,
|
||||
2693.62859998364
|
||||
],
|
||||
"text": "\u4eca\u5929\u6211\u4eec\u9a8c\u8bc1\u672c\u5730\u8bed\u97f3\u751f\u6210\u3002\u6a21\u578b\u5728\u540c\u4e00\u53f0\u8bbe\u5907\u4e0a\u8fd0\u884c\uff0c\u97f3\u9891\u6309\u7167\u987a\u5e8f\u8fde\u7eed\u8f93\u51fa\u3002",
|
||||
"warmup": false
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"n_requests": 1,
|
||||
"ttfa_ms": {
|
||||
"n": 1,
|
||||
"p50": 387.35440000891685,
|
||||
"p90": 387.35440000891685,
|
||||
"max": 387.35440000891685,
|
||||
"mean": 387.35440000891685
|
||||
},
|
||||
"playback_start_ms": {
|
||||
"n": 1,
|
||||
"p50": 4466.196100017987,
|
||||
"p90": 4466.196100017987,
|
||||
"max": 4466.196100017987,
|
||||
"mean": 4466.196100017987
|
||||
},
|
||||
"stall_at_ttfa_ms": {
|
||||
"n": 1,
|
||||
"p50": 4078.8417000090703,
|
||||
"p90": 4078.8417000090703,
|
||||
"max": 4078.8417000090703,
|
||||
"mean": 4078.8417000090703
|
||||
},
|
||||
"rtf_streamed": {
|
||||
"n": 1,
|
||||
"p50": 1.3853431406278105,
|
||||
"p90": 1.3853431406278105,
|
||||
"max": 1.3853431406278105,
|
||||
"mean": 1.3853431406278105
|
||||
},
|
||||
"rtf_total": {
|
||||
"n": 1,
|
||||
"p50": 1.3832185817311535,
|
||||
"p90": 1.3832185817311535,
|
||||
"max": 1.3832185817311535,
|
||||
"mean": 1.3832185817311535
|
||||
},
|
||||
"gap_ms": {
|
||||
"n": 5,
|
||||
"p50": 1757.2548000025563,
|
||||
"p90": 2714.6560000255704,
|
||||
"max": 2714.6560000255704,
|
||||
"mean": 1695.768340001814
|
||||
},
|
||||
"total_audio_s": 6.4,
|
||||
"total_wall_s": 11.508378600003198
|
||||
}
|
||||
}
|
||||
+218
@@ -0,0 +1,218 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"meta": {
|
||||
"model": "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice",
|
||||
"deploy_config": null,
|
||||
"query_type": "CustomVoice",
|
||||
"txt_prompts": "../analysis/experiments/omni_abstraction_20260922/tts-prompts.txt",
|
||||
"git_sha": "00510ce3",
|
||||
"gpu_name": "NVIDIA GeForce RTX 5090 Laptop GPU",
|
||||
"cuda_visible_devices": null,
|
||||
"timestamp": "20260922-120641",
|
||||
"argv": [
|
||||
"--query-type",
|
||||
"CustomVoice",
|
||||
"--txt-prompts",
|
||||
"../analysis/experiments/omni_abstraction_20260922/tts-prompts.txt"
|
||||
],
|
||||
"bench_args": {
|
||||
"model": "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice",
|
||||
"warmup": 1,
|
||||
"repeat": 1,
|
||||
"init_only": false,
|
||||
"tag": "abstraction",
|
||||
"output_dir": "../analysis/experiments/omni_abstraction_20260922/tts-stream",
|
||||
"save_wav": true,
|
||||
"gpu_poll_interval_s": 0.2,
|
||||
"parallel_stage_init": false,
|
||||
"deploy_profile": "edge",
|
||||
"step_stats": false,
|
||||
"bench_help": false
|
||||
},
|
||||
"warmup": 1,
|
||||
"repeat": 1,
|
||||
"init_only": false
|
||||
},
|
||||
"init": {
|
||||
"init_s": 40.22945002099732,
|
||||
"init_timeline": null
|
||||
},
|
||||
"step_stats": null,
|
||||
"resources": {
|
||||
"peak_rss_bytes": 8371363840,
|
||||
"peak_gpu_mem_mib": 14086,
|
||||
"gpu_mem_samples": 224,
|
||||
"gpu_mem_mode": "whole-gpu"
|
||||
},
|
||||
"requests": [
|
||||
{
|
||||
"request_id": "w0",
|
||||
"sr": 24000,
|
||||
"chunks": [
|
||||
{
|
||||
"idx": 0,
|
||||
"t_ms": 794.0313660074025,
|
||||
"gap_ms": 794.0313660074025,
|
||||
"samples": 3840
|
||||
},
|
||||
{
|
||||
"idx": 1,
|
||||
"t_ms": 1246.810203010682,
|
||||
"gap_ms": 452.7788370032795,
|
||||
"samples": 7680
|
||||
},
|
||||
{
|
||||
"idx": 2,
|
||||
"t_ms": 2253.012278000824,
|
||||
"gap_ms": 1006.2020749901421,
|
||||
"samples": 15360
|
||||
},
|
||||
{
|
||||
"idx": 3,
|
||||
"t_ms": 4120.435714023188,
|
||||
"gap_ms": 1867.4234360223636,
|
||||
"samples": 30720
|
||||
},
|
||||
{
|
||||
"idx": 4,
|
||||
"t_ms": 6991.494719986804,
|
||||
"gap_ms": 2871.059005963616,
|
||||
"samples": 48000
|
||||
},
|
||||
{
|
||||
"idx": 5,
|
||||
"t_ms": 9849.936566024553,
|
||||
"gap_ms": 2858.441846037749,
|
||||
"samples": 48000
|
||||
}
|
||||
],
|
||||
"t_final_ms": 12322.504625015426,
|
||||
"ttfa_ms": 794.0313660074025,
|
||||
"playback_start_ms": 5449.936566024553,
|
||||
"stall_at_ttfa_ms": 4655.9052000171505,
|
||||
"n_chunks": 6,
|
||||
"streamed_audio_s": 6.4,
|
||||
"tail_audio_s": 1.92,
|
||||
"total_wall_s": 12.322504625015426,
|
||||
"rtf_streamed": 1.5390525884413364,
|
||||
"rtf_total": 1.4810702674297387,
|
||||
"gap_ms": [
|
||||
452.7788370032795,
|
||||
1006.2020749901421,
|
||||
1867.4234360223636,
|
||||
2871.059005963616,
|
||||
2858.441846037749
|
||||
],
|
||||
"text": "\u4eca\u5929\u6211\u4eec\u9a8c\u8bc1\u672c\u5730\u8bed\u97f3\u751f\u6210\u3002\u6a21\u578b\u5728\u540c\u4e00\u53f0\u8bbe\u5907\u4e0a\u8fd0\u884c\uff0c\u97f3\u9891\u6309\u7167\u987a\u5e8f\u8fde\u7eed\u8f93\u51fa\u3002",
|
||||
"warmup": true
|
||||
},
|
||||
{
|
||||
"request_id": "r0_0",
|
||||
"sr": 24000,
|
||||
"chunks": [
|
||||
{
|
||||
"idx": 0,
|
||||
"t_ms": 411.0127830062993,
|
||||
"gap_ms": 411.0127830062993,
|
||||
"samples": 3840
|
||||
},
|
||||
{
|
||||
"idx": 1,
|
||||
"t_ms": 908.3849819726311,
|
||||
"gap_ms": 497.3721989663318,
|
||||
"samples": 7680
|
||||
},
|
||||
{
|
||||
"idx": 2,
|
||||
"t_ms": 1774.5298760128208,
|
||||
"gap_ms": 866.1448940401897,
|
||||
"samples": 15360
|
||||
},
|
||||
{
|
||||
"idx": 3,
|
||||
"t_ms": 3591.7348680086434,
|
||||
"gap_ms": 1817.2049919958226,
|
||||
"samples": 30720
|
||||
},
|
||||
{
|
||||
"idx": 4,
|
||||
"t_ms": 6472.409248002805,
|
||||
"gap_ms": 2880.6743799941614,
|
||||
"samples": 48000
|
||||
},
|
||||
{
|
||||
"idx": 5,
|
||||
"t_ms": 9228.608533972874,
|
||||
"gap_ms": 2756.1992859700695,
|
||||
"samples": 48000
|
||||
}
|
||||
],
|
||||
"t_final_ms": 11562.660126015544,
|
||||
"ttfa_ms": 411.0127830062993,
|
||||
"playback_start_ms": 4828.608533972874,
|
||||
"stall_at_ttfa_ms": 4417.595750966575,
|
||||
"n_chunks": 6,
|
||||
"streamed_audio_s": 6.4,
|
||||
"tail_audio_s": 1.92,
|
||||
"total_wall_s": 11.562660126015544,
|
||||
"rtf_streamed": 1.4419700834332616,
|
||||
"rtf_total": 1.3897428036076374,
|
||||
"gap_ms": [
|
||||
497.3721989663318,
|
||||
866.1448940401897,
|
||||
1817.2049919958226,
|
||||
2880.6743799941614,
|
||||
2756.1992859700695
|
||||
],
|
||||
"text": "\u4eca\u5929\u6211\u4eec\u9a8c\u8bc1\u672c\u5730\u8bed\u97f3\u751f\u6210\u3002\u6a21\u578b\u5728\u540c\u4e00\u53f0\u8bbe\u5907\u4e0a\u8fd0\u884c\uff0c\u97f3\u9891\u6309\u7167\u987a\u5e8f\u8fde\u7eed\u8f93\u51fa\u3002",
|
||||
"warmup": false
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"n_requests": 1,
|
||||
"ttfa_ms": {
|
||||
"n": 1,
|
||||
"p50": 411.0127830062993,
|
||||
"p90": 411.0127830062993,
|
||||
"max": 411.0127830062993,
|
||||
"mean": 411.0127830062993
|
||||
},
|
||||
"playback_start_ms": {
|
||||
"n": 1,
|
||||
"p50": 4828.608533972874,
|
||||
"p90": 4828.608533972874,
|
||||
"max": 4828.608533972874,
|
||||
"mean": 4828.608533972874
|
||||
},
|
||||
"stall_at_ttfa_ms": {
|
||||
"n": 1,
|
||||
"p50": 4417.595750966575,
|
||||
"p90": 4417.595750966575,
|
||||
"max": 4417.595750966575,
|
||||
"mean": 4417.595750966575
|
||||
},
|
||||
"rtf_streamed": {
|
||||
"n": 1,
|
||||
"p50": 1.4419700834332616,
|
||||
"p90": 1.4419700834332616,
|
||||
"max": 1.4419700834332616,
|
||||
"mean": 1.4419700834332616
|
||||
},
|
||||
"rtf_total": {
|
||||
"n": 1,
|
||||
"p50": 1.3897428036076374,
|
||||
"p90": 1.3897428036076374,
|
||||
"max": 1.3897428036076374,
|
||||
"mean": 1.3897428036076374
|
||||
},
|
||||
"gap_ms": {
|
||||
"n": 5,
|
||||
"p50": 1817.2049919958226,
|
||||
"p90": 2880.6743799941614,
|
||||
"max": 2880.6743799941614,
|
||||
"mean": 1763.519150193315
|
||||
},
|
||||
"total_audio_s": 6.4,
|
||||
"total_wall_s": 11.562660126015544
|
||||
}
|
||||
}
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"quantize_job": "jgzmkydzp",
|
||||
"profile_int8_calib_job": "jpez3xe0p",
|
||||
"compile_int8_calib_qio_S25": "jgzmkyo6p",
|
||||
"compile_int8_calib_qio_S24": "j5743nlr5",
|
||||
"decoder_fp16": {
|
||||
"snr_db": 39.164074983094494,
|
||||
"corr": 0.9999399789070609,
|
||||
"device": "Samsung Galaxy S25",
|
||||
"job": "jp39270xp"
|
||||
},
|
||||
"decoder_fp16_s24": {
|
||||
"snr_db": 39.13016700266315,
|
||||
"corr": 0.9999391875126119,
|
||||
"device": "Samsung Galaxy S24",
|
||||
"job": "jgj74yn7g"
|
||||
},
|
||||
"predictor_fp16": {
|
||||
"snr_db": 48.3393059438555,
|
||||
"corr": 0.999992670383219,
|
||||
"device": "Samsung Galaxy S25",
|
||||
"job": "jpez3xm7p"
|
||||
},
|
||||
"decoder_int8_calib_S25": {
|
||||
"snr_db": -3.3406415627342723,
|
||||
"corr": -0.002431700816898819,
|
||||
"device": "Samsung Galaxy S25",
|
||||
"job": "jgj74yz8g"
|
||||
},
|
||||
"decoder_int8_calib_S24": {
|
||||
"snr_db": -3.27255724475548,
|
||||
"corr": -0.003935559964945309,
|
||||
"device": "Samsung Galaxy S24",
|
||||
"job": "jgd3l4elp"
|
||||
},
|
||||
"profile_int8_calib_ms": 281.776
|
||||
}
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
"""Does the NPU-steps + GPU-vocoder split hold across the Snapdragon family? Same two graphs on a laptop-class part,
|
||||
an automotive board and a low-end IoT board, next to the two phones."""
|
||||
import json, qai_hub as hub
|
||||
S="/tmp/claude-2005/-data-zhoutaichang-embedding-infer/e695f57b-0163-4287-b1be-1d0ec9330768/scratchpad/qnn"
|
||||
meta=json.load(open(f"{S}/talker_step2_L256.json")); NL,NKV,D,H,L=meta["NL"],meta["NKV"],meta["D"],meta["H"],meta["L"]
|
||||
tspec={"x":((1,1,H),"float32"),"cos":((1,1,1,D),"float32"),"sin":((1,1,1,D),"float32"),"mask":((1,1,1,L+1),"float32")}
|
||||
for i in range(NL): tspec[f"k_cache_{i}"]=((1,NKV,L,D),"float32"); tspec[f"v_cache_{i}"]=((1,NKV,L,D),"float32")
|
||||
graphs={"talker_step2": (list(json.load(open(f"{S}/aihub/jobs7.json")).values())[0]["model_id"], tspec, "qnn_context_binary", "npu"),
|
||||
"code2wav": (json.load(open(f"{S}/aihub/jobs.json"))["code2wav_htp_c25@Samsung Galaxy S24"]["model_id"], {"quantized":((1,512,97),"float32")}, "tflite", "gpu")}
|
||||
devices=["Snapdragon X Elite CRD","SA8775P ADP","Dragonwing RB3 Gen 2 Vision Kit"]
|
||||
res={}; pend={}
|
||||
for dname in devices:
|
||||
for gname,(mid,spec,rt,unit) in graphs.items():
|
||||
try:
|
||||
dev=hub.Device(dname)
|
||||
cj=hub.submit_compile_job(model=hub.get_model(mid), device=dev, input_specs=spec, name=f"{gname} {rt}@{dname}", options=f"--target_runtime {rt}"+(" --quantize_full_type float16" if rt=="qnn_context_binary" else ""))
|
||||
pend[(dname,gname,rt,unit)]=cj; print("compile submitted", dname, gname, rt, cj.job_id, flush=True)
|
||||
except Exception as e: print("rejected", dname, gname, str(e)[:200], flush=True)
|
||||
prof={}
|
||||
for key,cj in pend.items():
|
||||
dname,gname,rt,unit=key; st=cj.wait(); print("compile", dname, gname, st.code, (st.message or "")[:120], flush=True)
|
||||
if not st.success: continue
|
||||
try:
|
||||
pj=hub.submit_profile_job(model=cj.get_target_model(), device=hub.Device(dname), name=f"{gname} on {dname} {unit}", options=f"--compute_unit {unit}")
|
||||
prof[key]=pj
|
||||
except Exception as e: print("profile rejected", dname, gname, str(e)[:160], flush=True)
|
||||
for key,pj in prof.items():
|
||||
dname,gname,rt,unit=key; st=pj.wait()
|
||||
if st.success:
|
||||
p=pj.download_profile(); ms=p["execution_summary"]["estimated_inference_time"]/1e3
|
||||
res[f"{gname}|{dname}|{rt}|{unit}"]=ms
|
||||
json.dump(p, open(f"{S}/aihub/profile_{gname}_{dname.replace(' ','_')}_{rt}_{unit}.json","w"))
|
||||
print(f" {gname} on {dname} ({rt}/{unit}): {ms:.1f} ms" + (f" -> {ms/25:.1f} ms per frame" if gname=="code2wav" else ""), flush=True)
|
||||
else: print(f" {gname} on {dname} failed:", (st.message or "")[:160], flush=True)
|
||||
json.dump(res, open(f"{S}/aihub/device_matrix.json","w"), indent=1)
|
||||
print("done", flush=True)
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"talker_step2|Snapdragon X Elite CRD|qnn_context_binary|npu": 19.583,
|
||||
"talker_step2|SA8775P ADP|qnn_context_binary|npu": 23.822,
|
||||
"code2wav|SA8775P ADP|tflite|gpu": 1700.577,
|
||||
"code2wav|Dragonwing RB3 Gen 2 Vision Kit|tflite|gpu": 3107.047
|
||||
}
|
||||
+5362
File diff suppressed because it is too large.
Load diff
+31
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"timestamp_utc": "2026-09-22T05:45:22.828581+00:00",
|
||||
"qai_hub_version": "0.55.0",
|
||||
"facility": "Qualcomm AI Hub Workbench",
|
||||
"command": "collect",
|
||||
"job_id": "jpx42jx1g",
|
||||
"url": "https://workbench.aihub.qualcomm.com/jobs/jpx42jx1g/",
|
||||
"status": "FAILED",
|
||||
"status_message": "Tensor 'x' has a floating-point type which is not supported by the targeted device.\nPlease quantize the model including its I/O and try again.",
|
||||
"options": "--target_runtime qnn_context_binary --quantize_full_type float16",
|
||||
"hub_version": "aihub-2026.08.28.1",
|
||||
"device": {
|
||||
"name": "Dragonwing RB3 Gen 2 Vision Kit",
|
||||
"os": "1.6",
|
||||
"attributes": [
|
||||
"os:qc_linux",
|
||||
"framework:tflite",
|
||||
"framework:onnx",
|
||||
"vendor:qualcomm",
|
||||
"abi:aarch64-oe-linux-gcc11",
|
||||
"format:iot",
|
||||
"chipset:qualcomm-qcs6490",
|
||||
"chipset:qcs6490",
|
||||
"chipset:qualcomm-dragonwing-qcs6490",
|
||||
"hexagon:v68",
|
||||
"soc-model:93",
|
||||
"framework:qnn"
|
||||
]
|
||||
},
|
||||
"model_id": "mq3xre4rq"
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"timestamp_utc": "2026-09-22T05:41:24.358293+00:00",
|
||||
"qai_hub_version": "0.55.0",
|
||||
"facility": "Qualcomm AI Hub Workbench",
|
||||
"command": "collect",
|
||||
"job_id": "j5wl18dzp",
|
||||
"url": "https://workbench.aihub.qualcomm.com/jobs/j5wl18dzp/",
|
||||
"status": "SUCCESS",
|
||||
"status_message": "",
|
||||
"options": "--compute_unit npu",
|
||||
"hub_version": "aihub-2026.09.11.0",
|
||||
"device": {
|
||||
"name": "Samsung Galaxy S25",
|
||||
"os": "15",
|
||||
"attributes": [
|
||||
"os:android",
|
||||
"framework:tflite",
|
||||
"framework:onnx",
|
||||
"abi:aarch64-android",
|
||||
"vendor:samsung",
|
||||
"format:phone",
|
||||
"chipset:qualcomm-snapdragon-8-elite-for-galaxy",
|
||||
"chipset:sm8750-ac",
|
||||
"hexagon:v79",
|
||||
"soc-model:69",
|
||||
"htp-supports-fp16:true",
|
||||
"htp-supports-weight-sharing:true",
|
||||
"framework:qnn"
|
||||
]
|
||||
},
|
||||
"model_id": "mmd5oy30q",
|
||||
"input_dataset_id": "d70qr8459",
|
||||
"outputs": "tts-s25-repeat.npz"
|
||||
}
|
||||
+5362
File diff suppressed because it is too large.
Load diff
+10982
File diff suppressed because it is too large.
Load diff
+31
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"timestamp_utc": "2026-09-22T05:45:21.128054+00:00",
|
||||
"qai_hub_version": "0.55.0",
|
||||
"facility": "Qualcomm AI Hub Workbench",
|
||||
"command": "collect",
|
||||
"job_id": "jgd3l92ep",
|
||||
"url": "https://workbench.aihub.qualcomm.com/jobs/jgd3l92ep/",
|
||||
"status": "FAILED",
|
||||
"status_message": "TargetRuntime.TFLITE is not a supported target for Snapdragon X Elite CRD.",
|
||||
"options": "--target_runtime tflite",
|
||||
"hub_version": "aihub-2026.08.28.1",
|
||||
"device": {
|
||||
"name": "Snapdragon X Elite CRD",
|
||||
"os": "11",
|
||||
"attributes": [
|
||||
"os:windows",
|
||||
"format:compute",
|
||||
"framework:onnx",
|
||||
"vendor:qualcomm",
|
||||
"abi:aarch64-windows",
|
||||
"chipset:qualcomm-snapdragon-x-elite",
|
||||
"chipset:sc8380xp",
|
||||
"hexagon:v73",
|
||||
"soc-model:60",
|
||||
"htp-supports-fp16:true",
|
||||
"htp-supports-weight-sharing:true",
|
||||
"framework:qnn"
|
||||
]
|
||||
},
|
||||
"model_id": "mqyer339n"
|
||||
}
|
||||
+10982
File diff suppressed because it is too large.
Load diff
+12
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"platform": "Windows-11-10.0.26200-SP0",
|
||||
"vllm": "0.29.0+cu134",
|
||||
"torch": "2.13.0+cu130",
|
||||
"extensions": {
|
||||
"vllm._C_stable_libtorch": "C:\\Users\\zhout\\w2\\omni029venv\\Lib\\site-packages\\vllm\\_C_stable_libtorch.pyd"
|
||||
},
|
||||
"cpu_attention_operators": {
|
||||
"cpu_attn_reshape_and_cache": false,
|
||||
"cpu_attn_get_scheduler_metadata": false
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
+29
@@ -0,0 +1,29 @@
|
||||
"""Inspect CPU attention operator availability; this is not a model execution."""
|
||||
|
||||
import argparse
|
||||
import importlib.metadata
|
||||
import json
|
||||
import platform
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
from vllm.platforms import current_platform
|
||||
|
||||
current_platform.import_kernels()
|
||||
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--out", type=Path, required=True)
|
||||
a = p.parse_args()
|
||||
names = ("cpu_attn_reshape_and_cache", "cpu_attn_get_scheduler_metadata")
|
||||
record = {
|
||||
"platform": platform.platform(),
|
||||
"vllm": importlib.metadata.version("vllm"),
|
||||
"torch": torch.__version__,
|
||||
"extensions": {
|
||||
name: getattr(module, "__file__", None) for name, module in sys.modules.items() if name.startswith("vllm._C")
|
||||
},
|
||||
"cpu_attention_operators": {name: hasattr(torch.ops._C, name) for name in names},
|
||||
}
|
||||
a.out.write_text(json.dumps(record, indent=2) + "\n")
|
||||
print(record["cpu_attention_operators"])
|
||||
@@ -0,0 +1,80 @@
|
||||
"""Read-only local artifact/runtime preflight; never claims inference success."""
|
||||
|
||||
import argparse
|
||||
import importlib.metadata
|
||||
import json
|
||||
import platform
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--workspace", type=Path, required=True)
|
||||
p.add_argument("--out", type=Path, required=True)
|
||||
a = p.parse_args()
|
||||
record = {
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"platform": platform.platform(),
|
||||
"python": sys.executable,
|
||||
"evidence": "preflight",
|
||||
"packages": {},
|
||||
"search_roots": [],
|
||||
"artifacts": [],
|
||||
}
|
||||
for name in ("vllm", "torch", "transformers", "onnxruntime", "onnxruntime-directml"):
|
||||
try:
|
||||
record["packages"][name] = importlib.metadata.version(name)
|
||||
except importlib.metadata.PackageNotFoundError:
|
||||
record["packages"][name] = None
|
||||
roots = [a.workspace / "models", Path.home() / ".cache/huggingface/hub"]
|
||||
if sys.platform == "win32":
|
||||
roots += [Path.home() / "w2/models"]
|
||||
else:
|
||||
roots += [Path("/mnt/c/Users/zhout/w2/models")]
|
||||
for root in roots:
|
||||
record["search_roots"].append({"path": str(root), "exists": root.is_dir()})
|
||||
if not root.is_dir():
|
||||
continue
|
||||
for item in root.iterdir():
|
||||
if not item.is_dir() or not any(x in item.name.lower() for x in ("spark", "qwen3", "minicpm", "internvla")):
|
||||
continue
|
||||
configs = list(item.glob("config.json")) + list(item.glob("snapshots/*/config.json"))
|
||||
for config in configs:
|
||||
try:
|
||||
data = json.loads(config.read_text(encoding="utf-8"))
|
||||
weights = list(config.parent.glob("*.safetensors"))
|
||||
record["artifacts"].append(
|
||||
{
|
||||
"directory": str(config.parent),
|
||||
"name": item.name,
|
||||
"model_type": data.get("model_type"),
|
||||
"architectures": data.get("architectures"),
|
||||
"weight_shards": len(weights),
|
||||
"weight_bytes": sum(f.stat().st_size for f in weights),
|
||||
}
|
||||
)
|
||||
except OSError as error:
|
||||
record["artifacts"].append({"directory": str(config.parent), "error": str(error)})
|
||||
try:
|
||||
from vllm.platforms import current_platform
|
||||
|
||||
record["vllm_platform"] = current_platform.device_type
|
||||
from vllm_omni.config.pipeline_registry import resolve_pipeline_config
|
||||
|
||||
record["registered_pipelines"] = {
|
||||
kind: resolve_pipeline_config(kind) is not None for kind in ("spark2_5", "qwen3_tts", "qwen3_5", "minicpmo_4_5")
|
||||
}
|
||||
except Exception as error:
|
||||
record["runtime_import_error"] = repr(error)
|
||||
for command in (
|
||||
["nvidia-smi", "--query-gpu=name,memory.total,memory.free", "--format=csv,noheader"],
|
||||
["adb", "devices", "-l"],
|
||||
):
|
||||
try:
|
||||
result = subprocess.run(command, capture_output=True, text=True, timeout=20)
|
||||
record[command[0]] = {"returncode": result.returncode, "stdout": result.stdout, "stderr": result.stderr}
|
||||
except (OSError, subprocess.TimeoutExpired) as error:
|
||||
record[command[0]] = {"error": str(error)}
|
||||
a.out.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
|
||||
print(f"Inventoried {len(record['artifacts'])} artifact directories; report: {a.out}")
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
"""Explicit AI Hub test facility; never imported by the local Omni runtime.
|
||||
|
||||
Pass an API token on stdin or use an already configured SDK profile. Secrets
|
||||
are never written to results. Inventory is read-only; profile/infer commands
|
||||
submit device jobs only for an explicitly supplied existing model ID.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.metadata
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import qai_hub as hub
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(__doc__)
|
||||
parser.add_argument("command", choices=("inventory", "profile", "infer", "collect"))
|
||||
parser.add_argument("--token-stdin", action="store_true")
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
parser.add_argument("--model")
|
||||
parser.add_argument("--device", default="Samsung Galaxy S25")
|
||||
parser.add_argument("--options", default="--compute_unit npu")
|
||||
parser.add_argument("--inputs", type=Path)
|
||||
parser.add_argument("--reference-job", help="Reuse a prior inference job's exact dataset")
|
||||
parser.add_argument("--job")
|
||||
args = parser.parse_args()
|
||||
client = hub.Client(hub.ClientConfig(api_token=sys.stdin.readline().strip())) if args.token_stdin else hub.Client()
|
||||
record = {
|
||||
"timestamp_utc": datetime.now(timezone.utc).isoformat(),
|
||||
"qai_hub_version": importlib.metadata.version("qai-hub"),
|
||||
"facility": "Qualcomm AI Hub Workbench",
|
||||
"command": args.command,
|
||||
}
|
||||
if args.command == "inventory":
|
||||
devices = client.get_devices()
|
||||
record["devices"] = [{"name": d.name, "os": d.os, "attributes": d.attributes} for d in devices]
|
||||
jobs = client.get_job_summaries(limit=100)
|
||||
record["recent_jobs"] = [
|
||||
{
|
||||
"job_id": j.job_id,
|
||||
"name": j.name,
|
||||
"date": str(j.date),
|
||||
"status": j.status.code,
|
||||
"type": str(j.job_type),
|
||||
"device": j.device_name,
|
||||
"url": j.url,
|
||||
}
|
||||
for j in jobs
|
||||
]
|
||||
models = client.get_models(limit=100)
|
||||
record["models"] = [{"model_id": m.model_id, "name": m.name} for m in models]
|
||||
print(f"Connected: {len(devices)} device entries, {len(jobs)} recent jobs, {len(models)} models")
|
||||
elif args.command in ("profile", "infer"):
|
||||
if not args.model:
|
||||
parser.error("--model is required")
|
||||
kwargs = dict(
|
||||
model=client.get_model(args.model),
|
||||
device=hub.Device(args.device),
|
||||
name=f"omni-boundary-validation-{args.command}",
|
||||
options=args.options,
|
||||
)
|
||||
if args.command == "infer":
|
||||
import numpy as np
|
||||
|
||||
if bool(args.inputs) == bool(args.reference_job):
|
||||
parser.error("provide exactly one of --inputs or --reference-job")
|
||||
if args.reference_job:
|
||||
reference = client.get_job(args.reference_job)
|
||||
if not isinstance(reference, hub.InferenceJob):
|
||||
parser.error("reference must be an inference job")
|
||||
kwargs["inputs"] = reference.inputs
|
||||
record["reference_job"] = reference.job_id
|
||||
record["input_dataset_id"] = reference.inputs.dataset_id
|
||||
else:
|
||||
with np.load(args.inputs, allow_pickle=False) as data:
|
||||
kwargs["inputs"] = {name: [data[name]] for name in data.files}
|
||||
job = client.submit_inference_job(**kwargs)
|
||||
else:
|
||||
job = client.submit_profile_job(**kwargs)
|
||||
record.update(
|
||||
job_id=job.job_id, url=job.url, model_id=args.model, requested_device=args.device, options=args.options
|
||||
)
|
||||
print(f"Submitted {args.command}: {job.job_id}")
|
||||
else:
|
||||
if not args.job:
|
||||
parser.error("--job is required")
|
||||
job = client.get_job(args.job)
|
||||
status = job.get_status()
|
||||
record.update(
|
||||
job_id=job.job_id,
|
||||
url=job.url,
|
||||
status=status.code,
|
||||
status_message=status.message,
|
||||
options=job.options,
|
||||
hub_version=job.hub_version,
|
||||
)
|
||||
if hasattr(job, "device"):
|
||||
record["device"] = {"name": job.device.name, "os": job.device.os, "attributes": job.device.attributes}
|
||||
if hasattr(job, "model"):
|
||||
record["model_id"] = job.model.model_id
|
||||
if isinstance(job, hub.ProfileJob):
|
||||
record["shapes"] = job.shapes
|
||||
if isinstance(job, hub.InferenceJob):
|
||||
record["input_dataset_id"] = job.inputs.dataset_id
|
||||
if status.success:
|
||||
if isinstance(job, hub.ProfileJob):
|
||||
record["profile"] = job.download_profile()
|
||||
elif isinstance(job, hub.InferenceJob):
|
||||
import numpy as np
|
||||
|
||||
outputs = job.download_output_data()
|
||||
output_path = args.out.with_suffix(".npz")
|
||||
np.savez(
|
||||
output_path,
|
||||
**{f"{key}__{i}": value for key, values in outputs.items() for i, value in enumerate(values)},
|
||||
)
|
||||
record["outputs"] = output_path.name
|
||||
elif isinstance(job, hub.CompileJob):
|
||||
target = job.get_target_model()
|
||||
record["target_model_id"] = target.model_id if target else None
|
||||
print(f"Job {job.job_id}: {status.code}")
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(json.dumps(record, indent=2, default=str) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Bounded, real-checkpoint public Omni startup/text probe (no model substitution)."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import time
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--model", required=True)
|
||||
p.add_argument("--out", type=Path, required=True)
|
||||
a = p.parse_args()
|
||||
os.environ.setdefault("VLLM_USE_FLASHINFER_SAMPLER", "0")
|
||||
os.environ.setdefault("VLLM_WSL2_ENABLE_PIN_MEMORY", "1")
|
||||
from vllm_omni.windows.aio import install_selector_policy
|
||||
|
||||
install_selector_policy()
|
||||
from vllm import SamplingParams
|
||||
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
|
||||
config = dict(
|
||||
model=a.model,
|
||||
language_model_only=True,
|
||||
max_model_len=512,
|
||||
max_num_seqs=1,
|
||||
max_num_batched_tokens=128,
|
||||
enforce_eager=True,
|
||||
gpu_memory_utilization=0.80,
|
||||
cpu_offload_gb=2,
|
||||
dtype="bfloat16",
|
||||
stage_init_timeout=120,
|
||||
)
|
||||
record = {
|
||||
"platform": platform.platform(),
|
||||
"config": config,
|
||||
"status": "starting",
|
||||
"scope": "public Omni, short text only; explicit 2 GiB native offload",
|
||||
"start": time.time(),
|
||||
}
|
||||
engine = None
|
||||
try:
|
||||
engine = Omni(**config)
|
||||
outputs = engine.generate(
|
||||
"Explain local inference in one sentence.", SamplingParams(temperature=0, max_tokens=16)
|
||||
)
|
||||
record["outputs"] = [str(output) for output in outputs]
|
||||
if not outputs:
|
||||
raise RuntimeError("No output")
|
||||
record["status"] = "completed"
|
||||
except Exception as error:
|
||||
record.update(status="failed", error=repr(error), traceback=traceback.format_exc())
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
if engine is not None:
|
||||
engine.close()
|
||||
record["end"] = time.time()
|
||||
a.out.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
|
||||
print(record["status"], flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Real-weight two-frame CPU TTS smoke probe, not full speech qualification."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--model", required=True)
|
||||
p.add_argument("--out", type=Path, required=True)
|
||||
a = p.parse_args()
|
||||
os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn")
|
||||
record = {"model": a.model, "scope": "two codec frames only; no speech-quality claim", "start": time.time()}
|
||||
engine = None
|
||||
try:
|
||||
import torch
|
||||
from transformers import AutoTokenizer
|
||||
from vllm.platforms import current_platform
|
||||
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.model_executor.models.qwen3_tts.configuration_qwen3_tts import Qwen3TTSConfig
|
||||
from vllm_omni.model_executor.models.qwen3_tts.prompt_embeds_builder import Qwen3TTSPromptEmbedsBuilder
|
||||
|
||||
assert current_platform.device_type == "cpu", "CPU probe must not silently use CUDA"
|
||||
config = Qwen3TTSConfig.from_pretrained(a.model)
|
||||
tok = AutoTokenizer.from_pretrained(a.model)
|
||||
info = {
|
||||
"task_type": ["CustomVoice"],
|
||||
"text": ["A short test."],
|
||||
"language": ["English"],
|
||||
"speaker": ["Vivian"],
|
||||
"max_new_tokens": [2],
|
||||
}
|
||||
length = Qwen3TTSPromptEmbedsBuilder.estimate_prompt_len_from_additional_information(
|
||||
additional_information=info,
|
||||
task_type="CustomVoice",
|
||||
tokenize_prompt=lambda x: tok(x, padding=False)["input_ids"],
|
||||
codec_language_id=config.talker_config.codec_language_id,
|
||||
spk_is_dialect=config.talker_config.spk_is_dialect,
|
||||
)
|
||||
engine = Omni(model=a.model, stage_init_timeout=120, parallel_stage_init=False)
|
||||
params = engine.default_sampling_params_list
|
||||
params[0].max_tokens = 2
|
||||
outputs = engine.generate({"prompt_token_ids": [0] * length, "additional_information": info}, params)
|
||||
chunks = []
|
||||
for output in outputs:
|
||||
if output.final_output_type == "audio":
|
||||
audio = output.outputs[0].multimodal_output.get("audio")
|
||||
chunks.extend(audio if isinstance(audio, list) else [audio])
|
||||
values = torch.cat([x.float().cpu().reshape(-1) for x in chunks if x is not None])
|
||||
assert values.numel() > 0 and torch.isfinite(values).all() and torch.any(values != 0)
|
||||
record.update(status="completed", samples=values.numel(), finite=True, nonzero=True)
|
||||
except Exception as error:
|
||||
record.update(status="failed", error=repr(error), traceback=traceback.format_exc())
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
if engine is not None:
|
||||
engine.close()
|
||||
record["end"] = time.time()
|
||||
a.out.write_text(json.dumps(record, indent=2) + "\n")
|
||||
print(record["status"], flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
"""Same compiled artifact and dataset repeatability, not FP checkpoint quality."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
root = Path(__file__).resolve().parent
|
||||
rows = []
|
||||
for model in ("tts", "minicpm"):
|
||||
reference = json.loads((root / f"{model}-s25-reference.json").read_text())
|
||||
current = json.loads((root / f"{model}-s25-repeat.json").read_text())
|
||||
assert current["status"] == "SUCCESS"
|
||||
assert current["model_id"] == reference["model_id"]
|
||||
assert current["input_dataset_id"] == reference["input_dataset_id"]
|
||||
with np.load(root / reference["outputs"]) as a, np.load(root / current["outputs"]) as b:
|
||||
assert set(a.files) == set(b.files)
|
||||
outputs = []
|
||||
for name in a.files:
|
||||
assert a[name].shape == b[name].shape
|
||||
outputs.append(
|
||||
{
|
||||
"name": name,
|
||||
"shape": list(b[name].shape),
|
||||
"finite": bool(np.isfinite(b[name]).all()),
|
||||
"exact": bool(np.array_equal(a[name], b[name])),
|
||||
"max_abs_difference": float(np.max(np.abs(a[name].astype(float) - b[name].astype(float)))),
|
||||
}
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"model": model,
|
||||
"job": current["job_id"],
|
||||
"reference_job": reference["job_id"],
|
||||
"model_id": current["model_id"],
|
||||
"input_dataset_id": current["input_dataset_id"],
|
||||
"scope": "same compiled component repeatability only",
|
||||
"outputs": outputs,
|
||||
}
|
||||
)
|
||||
assert all(output["finite"] for row in rows for output in row["outputs"])
|
||||
(root / "hub-repeatability.json").write_text(json.dumps(rows, indent=2) + "\n")
|
||||
print([(row["model"], len(row["outputs"]), all(o["exact"] for o in row["outputs"])) for row in rows])
|
||||
@@ -0,0 +1,39 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Check full matrix coverage and archived evidence integrity, not model quality.
|
||||
|
||||
Run from any directory with Python's standard library:
|
||||
python verify_qualification_matrix.py path/to/matrix.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import itertools
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def verify(path: Path):
|
||||
record = json.loads(path.read_text(encoding="utf-8"))
|
||||
pairs = [(case["device"], case["model"]) for case in record["cases"]]
|
||||
expected = set(itertools.product(record["devices"], record["models"]))
|
||||
assert len(pairs) == len(set(pairs)) and set(pairs) == expected, "missing or duplicate pairings"
|
||||
allowed = {"PASS", "PARTIAL", "COMPONENT", "PROFILE_ONLY", "FAILED", "REJECTED", "BLOCKED"}
|
||||
for case in record["cases"]:
|
||||
assert case["status"] in allowed
|
||||
assert all(case.get(field) for field in ("finding", "next_step", "check", "depth", "evidence"))
|
||||
assert case["status"] != "PASS" or case["depth"] == "P", "component pass promoted to full workload"
|
||||
assert all(ref in record["sources"] for ref in case["evidence"])
|
||||
for source in record["sources"].values():
|
||||
evidence = (path.parent / source["file"]).resolve()
|
||||
assert evidence.is_relative_to(path.parent.resolve()), "evidence escapes bundle"
|
||||
assert hashlib.sha256(evidence.read_bytes()).hexdigest() == source["sha256"], evidence
|
||||
counts = dict(Counter(case["status"] for case in record["cases"]))
|
||||
assert counts == record["counts"]
|
||||
print(f"Verified {len(pairs)} pairings and {len(record['sources'])} evidence hashes: {counts}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("matrix", type=Path)
|
||||
verify(parser.parse_args().matrix)
|
||||
@@ -40,25 +40,45 @@ non-migratable. Legacy M0 defaults are explicitly preserved in `contracts.legacy
|
||||
|
||||
### Mobile and PC × model support matrix
|
||||
|
||||
Rows identify the device class and execution configuration; columns identify the
|
||||
model family. **Not qualified** means the complete specified model path has not
|
||||
passed this release's acceptance gates, not that the hardware cannot run it.
|
||||
**All 60 named pairings were checked on 2026-09-22** through actual runs,
|
||||
raw-record review or artifact/runtime preflight. This is not 60 successful
|
||||
executions. Each cell links to its evidence, finding and next required step.
|
||||
|
||||
| Device / execution configuration | Spark-X2.5 | Qwen3-TTS 0.6B | Qwen3.8-27B | MiniCPM-o | InternVLA |
|
||||
PASS: specified complete workload passed. PARTIAL: functional output with
|
||||
acceptance gaps. COMPONENT: model component only. PROFILE_ONLY: component
|
||||
timing without quality qualification. FAILED: attempted startup/execution
|
||||
failed. REJECTED: specific artifact refused. BLOCKED: prerequisite missing.
|
||||
|
||||
| Device / execution configuration | Spark-X2.5 | Qwen3-TTS 0.6B CustomVoice | Qwen3.8-27B | MiniCPM-o 4.5 | InternVLA-A1 |
|
||||
|---|---|---|---|---|---|
|
||||
| PC: CPU only, WSL | 1.7B INT8 historical M0 pass; extraction rerun pending | Not qualified | Not qualified | Not qualified | Not qualified |
|
||||
| PC: CPU only, native Windows | Not qualified | Not qualified | Not qualified | Not qualified | Not qualified |
|
||||
| PC: CPU + NVIDIA discrete GPU, Windows/WSL | 4B BF16 M0 passed | Real-weight streaming completes; playback stalls remain | Complete multimodal pipeline not qualified | Not qualified | Not qualified |
|
||||
| PC: CPU + AMD Radeon 890M iGPU | Not qualified | Not qualified | Historical vision-tower component pass only | Not qualified | Not qualified |
|
||||
| PC: CPU + AMD NPU | Not qualified | Not qualified | Tested whole-vision-tower exports rejected | Not qualified | Not qualified |
|
||||
| PC: CPU + iGPU + NPU, with optional discrete GPU used jointly | Full heterogeneous model path not qualified | Full heterogeneous model path not qualified | Full heterogeneous model path not qualified | Not qualified | Not qualified |
|
||||
| Mobile: Galaxy S25 / Snapdragon 8 Elite for Galaxy, CPU + GPU + NPU | W8A16 attention component passed in AI Hub; full local generation pending | Local NPU/GPU pipeline not qualified | Not qualified | Not qualified | Not qualified |
|
||||
| Other mobile / embedded SoCs | Not qualified | Not qualified | Not qualified | Not qualified | Not qualified |
|
||||
| PC HX370: CPU / WSL | [PASS](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_wsl-spark) | [FAILED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_wsl-tts) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_wsl-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_wsl-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_wsl-vla) |
|
||||
| PC HX370: CPU / native Windows | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_windows-spark) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_windows-tts) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_windows-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_windows-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cpu_windows-vla) |
|
||||
| PC HX370 + RTX 5090 Laptop / WSL | [PASS](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_wsl-spark) | [PARTIAL](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_wsl-tts) | [FAILED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_wsl-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_wsl-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_wsl-vla) |
|
||||
| PC HX370 + RTX 5090 Laptop / Windows | [PASS](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_windows-spark) | [PARTIAL](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_windows-tts) | [FAILED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_windows-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_windows-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_cuda_windows-vla) |
|
||||
| PC HX370 + Radeon 890M / Windows worker | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_igpu-spark) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_igpu-tts) | [COMPONENT](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_igpu-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_igpu-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_igpu-vla) |
|
||||
| PC HX370 + AMD NPU / Windows worker | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_npu-spark) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_npu-tts) | [REJECTED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_npu-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_npu-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_npu-vla) |
|
||||
| PC CPU+iGPU+NPU +/- discrete GPU, joint model execution | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_joint-spark) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_joint-tts) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_joint-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_joint-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_joint-vla) |
|
||||
| PC Snapdragon X Elite CRD / AI Hub | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_xelite-spark) | [PROFILE_ONLY](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_xelite-tts) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_xelite-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_xelite-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#pc_xelite-vla) |
|
||||
| Mobile Galaxy S25 / Snapdragon 8 Elite for Galaxy | [COMPONENT](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s25-spark) | [COMPONENT](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s25-tts) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s25-qwen27) | [COMPONENT](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s25-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s25-vla) |
|
||||
| Mobile Galaxy S24 / Snapdragon 8 Gen 3 | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s24-spark) | [COMPONENT](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s24-tts) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s24-qwen27) | [COMPONENT](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s24-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#mobile_s24-vla) |
|
||||
| Embedded SA8775P ADP / AI Hub | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_sa8775p-spark) | [PROFILE_ONLY](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_sa8775p-tts) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_sa8775p-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_sa8775p-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_sa8775p-vla) |
|
||||
| Embedded RB3 Gen 2 / QCS6490 / AI Hub | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_rb3-spark) | [PROFILE_ONLY](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_rb3-tts) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_rb3-qwen27) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_rb3-minicpm) | [BLOCKED](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md#embedded_rb3-vla) |
|
||||
|
||||
The PC measurements below come from one Ryzen AI 9 HX370 laptop. Component
|
||||
success, backend probes and topology tests do not certify a full model pipeline.
|
||||
S25 AI Hub results do not qualify an on-device Omni controller, continuous
|
||||
generation, shared NPU/GPU execution or sustained thermal behavior.
|
||||
Counts: **3 PASS, 2 PARTIAL, 6 COMPONENT, 3 PROFILE_ONLY, 3 FAILED,
|
||||
1 REJECTED, 42 BLOCKED**. The PC CPU Spark pass uses the existing vLLM 0.28
|
||||
CPU wheel with the current Omni branch; CPU TTS fails a 3-versus-4-field
|
||||
sampling ABI mismatch in that environment. Current Windows/WSL Omni Qwen
|
||||
27B startup fails pipeline resolution; earlier bare-vLLM text evidence is
|
||||
not a pass for this integration. Full MiniCPM-o/InternVLA checkpoints are
|
||||
absent from the inventoried model roots. Native Windows CPU attention
|
||||
kernels are absent from the installed extension.
|
||||
|
||||
S25 Spark, TTS predictor and MiniCPM-o speech-head components have fresh Hub
|
||||
inference evidence. S24 and the other Hub targets retain explicitly marked
|
||||
historical component/profile evidence. No direct ADB device is attached.
|
||||
|
||||
See the [full audit](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md) and its hashed raw records. Other hardware SKUs
|
||||
remain outside this finite matrix; no universal device support is implied.
|
||||
|
||||
### Detailed PC model and backend pairings
|
||||
|
||||
@@ -71,13 +91,13 @@ Model support is specific to the following pairings:
|
||||
|---|---|---|---|
|
||||
| Spark-X2.5-4B BF16 | CPU + RTX 5090 Laptop GPU | WSL / vLLM CUDA | Current branch: M0 text/state/cancellation workload passed. |
|
||||
| Spark-X2.5-4B BF16 | CPU + RTX 5090 Laptop GPU | Native Windows / vLLM CUDA | Current branch: M0 text/state/cancellation workload passed. |
|
||||
| Spark-X2.5-1.7B INT8 | CPU execution; accelerators unused | WSL / vLLM CPU | Historical M0 acceptance and token parity passed on 2026-09-15; not freshly requalified on this extraction branch. |
|
||||
| Spark-X2.5-1.7B INT8 | CPU execution; accelerators unused | WSL / vLLM CPU | Fresh 12-prompt acceptance passed on this branch with the existing vLLM 0.28 CPU wheel; historical token parity is separate. |
|
||||
| Spark-X2.5-1.7B INT8 | CPU execution; no NVIDIA dependency | Native Windows | Model path not yet qualified; CPU Add graph success is not text-model support. |
|
||||
| Qwen3-TTS-12Hz-0.6B CustomVoice | CPU + RTX 5090 Laptop GPU; both stages on GPU | WSL / vLLM CUDA | Current branch: real-weight audio streaming completes; playback stalls and remaining M2 gates stay open. |
|
||||
| Qwen3-TTS-12Hz-0.6B CustomVoice | CPU + RTX 5090 Laptop GPU; both stages on GPU | Native Windows / vLLM CUDA | Current branch: real-weight audio streaming completes; playback stalls and remaining M2 gates stay open. |
|
||||
| Qwen3.8-27B vision tower, FP32 ONNX | CPU + Radeon 890M iGPU | Windows / ORT DirectML | Historical component validation only; not the complete 27B model or a newly qualified graph-stage vision-to-language pipeline. |
|
||||
| Qwen3.8-27B vision tower, tested A16W8 exports | CPU + AMD NPU | Windows / ORT VitisAI | Tested whole-tower artifacts were rejected by NPU partitioning. This does not rule out other exports or smaller components. |
|
||||
| Qwen3.8-27B complete multimodal pipeline | CPU/iGPU/NPU, with optional discrete GPU | Target-specific | Not qualified by this abstraction release; earlier short-text loading is not complete multimodal validation. |
|
||||
| Qwen3.8-27B complete multimodal pipeline | CPU/iGPU/NPU, with optional discrete GPU | Target-specific | Fresh public Omni startup fails pipeline resolution on Windows/WSL; earlier bare-vLLM short-text loading is separate and not multimodal validation. |
|
||||
| MiniCPM-o | CPU/iGPU/NPU, with optional discrete GPU | Target-specific | Full model × PC combinations remain unqualified for this release. |
|
||||
| InternVLA | CPU/iGPU/NPU, with optional discrete GPU | Target-specific | Full model × PC combinations remain unqualified for this release. |
|
||||
|
||||
|
||||
Reference in new issue
Block a user