diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md index 718f41a95..509e50cfb 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md @@ -8,4 +8,4 @@ The [talker decode-step attempt](evidence/qwen_tts_talker_qualcomm/README.md) pi The [MiniCPM-o speech-head study](evidence/minicpmo_tts_qualcomm/README.md) recovered the real 20-layer ONNX source and a matching retained fixture. S24 and S25 FP16 context-binary outputs preserved the ONNX CPU top-1 speech token with 0.346%/0.360% logits relative L2 on one step. Their 100-sample component profiles had p50/p95 8.537/9.006 ms and 7.197/7.615 ms, with all 1,029 execution-detail rows on NPU for each. FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3. SA8775P NPU-requested inference preserved CPU top-1 token 1867 with 0.3568% logits relative L2; X Elite same-fixture inference also preserved token 1867 with 0.3170% logits relative L2. SA and X Elite placement profiles remain pending. RB3 FP16 QNN inference failed at graph load despite compile success; an explicit TFLite CPU fallback compile passed with same-fixture CPU-requested inference pending. The complete multimodal model remains unqualified on those devices. -The [InternVLA Cosmos S25 attempt](evidence/internvla_cosmos_qualcomm/README.md) repackages the real policy's image encoder from its parity-tested ONNX export. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern. Injecting it into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component placement profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending. Device-local action-policy support remains unqualified. +The [InternVLA Cosmos S25 attempt](evidence/internvla_cosmos_qualcomm/README.md) repackages the real policy's image encoder from its parity-tested ONNX export. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern. Injecting it into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component placement profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. Device-local action-policy support remains unqualified. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md index b671f97ee..672255ecd 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md @@ -6,6 +6,8 @@ Workbench source model `mn4okj3rq` was [submitted for Galaxy S25 QNN DLC compila The same source model also [compiled to FP16 QNN DLC on Snapdragon X Elite CRD](xelite/compile_report.json), producing target `mmxj0w8kq` for Windows ARM. A [same-fixture NPU-requested inference](xelite/inference_submission.json) was submitted using dataset `d26qo8p57` and remained pending. Compilation alone does not establish numerical fidelity or actual NPU placement on X Elite. +The same source also [compiled to FP16 QNN DLC on SA8775P ADP](sa8775p/compile_report.json), producing target mn750wz8m for Android. [Same-fixture NPU-requested inference](sa8775p/inference_submission.json) was submitted using dataset d26qo8p57; numerical and placement outcomes remain pending. + The [generic collector](../../../../experiments/collect_aihub_component.py) preserves terminal jobs, and the [Cosmos auditor](../../../../experiments/audit_internvla_cosmos_qualcomm.py) checks source/fixture hashes, target/device/dataset identity, latent shape and finite numerical difference against the pinned source reference. A separate [synthetic action sensitivity check](s25/action_sensitivity.json) injected either the pinned source or S25 latent into the same real Place_Markpen policy on **local CPU**, with zero state/noise. The [reproducer](../../../../experiments/probe_internvla_hosted_latent_actions.py) returned finite `[1,50,32]` actions; hosted-vs-source action relative L2 was **0.742%**, maximum absolute difference **0.0115**, cosine **0.999973**. Repeating the source policy gave bitwise identical actions. This is a cross-device numerical handoff experiment, not a device-local policy or robot-task pass; no real-observation action tolerance is known. -Next: inspect X Elite same-fixture output, test representative real observations and action tolerances, then build and validate a device-local stateful policy. Full multimodal observation processing, admission, cancellation and sustained profiling remain open. +Next: inspect X Elite same-fixture output and SA8775P same-fixture output, test representative real observations and action tolerances, then build and validate a device-local stateful policy. Full multimodal observation processing, admission, cancellation and sustained profiling remain open. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/sa8775p/compile_report.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/sa8775p/compile_report.json new file mode 100644 index 000000000..52d4090f7 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/sa8775p/compile_report.json @@ -0,0 +1,31 @@ +{ + "retrieved_utc": "2026-09-24T00:36:15.467132+00:00", + "facility": "Qualcomm AI Hub Workbench", + "qai_hub_version": "0.55.0", + "job_id": "j5687wx6g", + "url": "https://workbench.aihub.qualcomm.com/jobs/j5687wx6g/", + "status": "SUCCESS", + "status_message": "", + "source_model_id": "mn4okj3rq", + "target_model_id": "mn750wz8m", + "device": { + "name": "SA8775P ADP", + "os": "14", + "attributes": [ + "os:android", + "framework:tflite", + "framework:onnx", + "vendor:qualcomm", + "abi:aarch64-android", + "format:auto", + "chipset:qualcomm-sa8775p", + "hexagon:v73", + "soc-model:52", + "htp-supports-fp16:true", + "htp-supports-weight-sharing:true", + "framework:qnn" + ] + }, + "options": "--target_runtime qnn_dlc --qnn_options default_graph_htp_precision=FLOAT16", + "hub_version": "aihub-2026.09.11.0" +} diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/sa8775p/compile_submission.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/sa8775p/compile_submission.json new file mode 100644 index 000000000..d738618d0 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/sa8775p/compile_submission.json @@ -0,0 +1,12 @@ +{ + "timestamp_utc": "2026-09-24T00:33:50.237703+00:00", + "facility": "Qualcomm AI Hub Workbench", + "qai_hub_version": "0.55.0", + "source_model_id": "mn4okj3rq", + "device": "SA8775P ADP", + "options": "--target_runtime qnn_dlc --qnn_options default_graph_htp_precision=FLOAT16", + "kind": "compile", + "status": "SUBMITTED", + "job_id": "j5687wx6g", + "url": "https://workbench.aihub.qualcomm.com/jobs/j5687wx6g/" +} diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/sa8775p/inference_submission.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/sa8775p/inference_submission.json new file mode 100644 index 000000000..bf4103906 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/sa8775p/inference_submission.json @@ -0,0 +1,14 @@ +{ + "timestamp_utc": "2026-09-24T00:36:17.520535+00:00", + "facility": "Qualcomm AI Hub Workbench", + "qai_hub_version": "0.55.0", + "source_compile_job_id": "j5687wx6g", + "target_model_id": "mn750wz8m", + "input_dataset_id": "d26qo8p57", + "device": "SA8775P ADP", + "options": "--compute_unit npu", + "kind": "inference", + "status": "SUBMITTED", + "job_id": "jp2rwe2rg", + "url": "https://workbench.aihub.qualcomm.com/jobs/jp2rwe2rg/" +} diff --git a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md index 37e99ceba..f1fc34a58 100644 --- a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md +++ b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md @@ -105,7 +105,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i | Embedded SA8775P ADP / AI Hub | Qwen3-TTS 0.6B CustomVoice | COMPONENT_CPU_VOCODER_PARITY_AND_PLACEMENT_PASS; QNN_NPU_VOCODER_COMPILE_PASS_INFERENCE_PENDING; COMPONENT_GPU_NUMERIC_FAIL; TALKER_FP16_NUMERIC_FAIL; NOT_E2E_PROFILEABLE | On the exact SA8775P, the same TFLite binary's CPU-requested Code2Wav inference matched original ONNX CPU within 4.13e-6 relative L2 / 107.67 dB SNR, without saturation; Its 100-sample CPU-requested component profile had p50/p95 5.384/5.534 s with all 738 execution-detail rows on CPU; the service estimated-time field 5.250 s is the sample minimum. The GPU-requested output saturated 96.63% of samples with 11.832 relative L2; its fresh 100-sample mixed CPU/GPU profile p50/p95 was 1.776/1.880 s, timing failed audio. Even with FP32-preserving GPU options, 84.89% of samples saturated with 11.389 relative L2. The 28-layer FP16 QNN talker step executed but changed CPU top-1 token 80 to 1720 with 9.43% logits relative L2. A TFLite CPU talker compile passed with same-fixture inference pending. A separate FP16 QNN DLC vocoder compile passed for SA8775P, with same-fixture NPU-requested inference pending. No complete pipeline exists. | Audit the pending SA FP16 QNN vocoder inference and talker CPU control, isolate GPU delegate and talker numerical failures, then establish listening quality, co-resident streaming, memory and sustained behavior; [vocoder](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) and [talker](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) evidence. | | Embedded SA8775P ADP / AI Hub | Qwen3.8-27B | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | Embedded SA8775P ADP / AI Hub | MiniCPM-o 4.5 | COMPONENT_SPEECH_HEAD_NUMERIC_MEASURED; PLACEMENT_PENDING; NOT_E2E_PROFILEABLE | The real 20-layer MiniCPM-o speech-head source compiled to FP16 QNN DLC for the exact SA8775P ADP. Same-fixture NPU-requested inference returned 41 finite tensors, preserved ONNX CPU top-1 speech token 1867, and had 0.3568% logits relative L2 and 1.225% maximum K/V relative L2. The requested NPU route awaits actual node-placement and timing profile; no complete thinker/encoders/vocoder bundle or device-local stateful controller exists. | Audit the [SA8775P speech-head placement profile](../../../e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md), then qualify the complete multimodal pipeline and sustained behavior. | -| Embedded SA8775P ADP / AI Hub | InternVLA-A1 | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | +| Embedded SA8775P ADP / AI Hub | InternVLA-A1 | COMPONENT_COSMOS_FP16_QNN_COMPILE_PASS_INFERENCE_PENDING; NOT_E2E_PROFILEABLE | The real Place_Markpen Cosmos image encoder source compiled to FP16 QNN DLC on exact SA8775P ADP. Same-fixture NPU-requested inference is pending; no numerical output, actual placement, full action policy or local stateful Omni backend exists yet. | Audit the [SA8775P Cosmos inference](../../../e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md), then obtain placement, real-observation action validation and a complete local policy. | | Embedded RB3 Gen 2 / QCS6490 / AI Hub | Spark-X2.5 | COMPONENT_CPU_PARITY_PASS; COMPONENT_NPU_NUMERIC_UNQUALIFIED; NOT_E2E_PROFILEABLE | On the exact hosted Dragonwing RB3 Gen 2 Vision Kit (QCS6490, QC Linux 1.6), the reused S24 W8A16 QNN DLC and a separately RB3-compiled W8A16 DLC each failed inference/profile at QnnModel_composeGraphsFromDlc: MODEL_GRAPH_ERROR. An FP32 ONNX one-layer CPU route passed with all 56 profile rows on CPU, source-relative hidden/K/V L2 of 8.73e-7/2.98e-7/3.37e-7 and 100-sample component p50/p95 19.304/21.113 ms. A calibrated W8A8 QNN DLC ran with all 61 rows on NPU and p50/p95 5.610/5.761 ms, but hidden/K/V relative L2 was 20.914%/1.702%/4.413% versus FP32 CPU, without a next-token quality gate. These are different precisions, not a qualified speedup; no complete 28-layer prefill/decode/KV-ring/sampling, local Omni controller or complete-request timing exists. | Improve representative W8A8 calibration or find another compatible quality-passing artifact; validate token parity, then full device-resident generation, admission/cancel and sustained power; [RB3 raw evidence](../../../e2e_expansion_20260923/evidence/spark_rb3_full_attention/README.md). | | Embedded RB3 Gen 2 / QCS6490 / AI Hub | Qwen3-TTS 0.6B CustomVoice | COMPONENT_MIXED_VOCODER_PROFILED_NUMERIC_UNQUALIFIED; NOT_E2E_PROFILEABLE | Fresh same-fixture TFLite GPU-requested vocoder inference returned a finite unsaturated waveform, but relative L2 versus original ONNX CPU was 15.73% (16.06 dB SNR), without a listening-quality tolerance; it is not a qualified audio path. The fresh 100-sample component profile had nearest-rank p50/p95 3.099/3.133 s and 403 CPU plus 335 GPU execution-detail rows. Its 3.070 s estimated-time field is the sample minimum, not median. The historical NPU talker build was rejected for floating-point inputs; no full model pipeline exists. | Establish waveform quality and a compatible integer talker artifact, then validate complete local execution, memory and sustained behavior; [vocoder evidence](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md). | | Embedded RB3 Gen 2 / QCS6490 / AI Hub | Qwen3.8-27B | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | diff --git a/docs/design/omni_edge_e2e_roadmap.md b/docs/design/omni_edge_e2e_roadmap.md index 9a1c236e1..bd7017690 100644 --- a/docs/design/omni_edge_e2e_roadmap.md +++ b/docs/design/omni_edge_e2e_roadmap.md @@ -23,7 +23,7 @@ A separate [28-layer talker decode-step attempt](../../benchmarks/edge_harness/r For M4, a [MiniCPM-o 4.5 speech-head study](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md) recovered the historical 20-layer ONNX source and matching S24/S25 fixtures. Both FP16 context-binary outputs preserved the ONNX CPU top-1 speech token on one decode step, with logits relative L2 0.346%/0.360%. Their 100-sample component profiles had p50/p95 8.537/9.006 ms and 7.197/7.615 ms, with all 1,029 execution-detail rows on NPU for each. New FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3. SA8775P same-fixture NPU-requested inference preserved CPU top-1 speech token 1867 with 0.3568% logits relative L2 and 1.225% maximum K/V relative L2; placement profiling is pending. X Elite same-fixture inference also preserved token 1867 with 0.3170% logits relative L2; both placement profiles remain pending. RB3 FP16 QNN inference failed at graph load despite compile success; a TFLite CPU fallback compile passed with same-fixture CPU-requested inference pending. This is component C numerical evidence only, not thinker/encoder/vocoder or complete multimodal support. -For M4, a [Galaxy S25 Cosmos encoder attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) repackages the real Place_Markpen FP32 ONNX export after retaining two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one synthetic pattern. Injecting that output into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU; the action policy did not run on S25. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending. This is component/action-sensitivity evidence, not device-local M4 qualification. +For M4, a [Galaxy S25 Cosmos encoder attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) repackages the real Place_Markpen FP32 ONNX export after retaining two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one synthetic pattern. Injecting that output into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU; the action policy did not run on S25. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. This is component/action-sensitivity evidence, not device-local M4 qualification. The [WSL CPU expansion](../../benchmarks/edge_harness/results/e2e_expansion_20260923/README.md) adds constrained three-stage MiniCPM-o text-to-speech requests and a CPU-only