diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md index 509e50cfb..b5872df1f 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md @@ -1,11 +1,11 @@ # 2026-09-24 Qualcomm component expansion -The [Qwen3-TTS vocoder study](evidence/qwen_tts_vocoder_qualcomm/README.md) reuses a retained fixed-shape Code2Wav export and historical Workbench input to test numerical behavior on SA8775P and other Qualcomm settings. The fresh SA8775P GPU-requested inference produces a saturated waveform, while the same binary's CPU-requested output on the exact SA device matches ONNX CPU within 4.13e-6 relative L2. Its 100-sample CPU-requested component profile had p50/p95 5.384/5.534 s with all 738 execution-detail rows on CPU. A FP32-preserving GPU-option control still saturated 84.89% of samples, so that option did not repair the tested path. RB3 produces an unsaturated but 15.73%-divergent waveform; its fresh 100-sample mixed CPU/GPU component p50/p95 was 3.099/3.133 s, not a usable TTS timing claim. X Elite ONNX CPU output matches the local ONNX Runtime reference on the retained fixture; its 94-sample component p50/p95 was 6.384/6.644 s with all execution-detail rows on CPU. Corrected X Elite DirectML inference and profile both failed during provider initialization; separate FP16 QNN DLC inference returned an unsaturated waveform at 1.138% relative L2 / 38.88 dB SNR against original ONNX CPU; placement profiling is pending. A separate SA8775P FP16 QNN vocoder compile passed; same-fixture NPU-requested inference is pending. No complete on-device TTS stream or model-quality qualification is claimed. +The [Qwen3-TTS vocoder study](evidence/qwen_tts_vocoder_qualcomm/README.md) reuses a retained fixed-shape Code2Wav export and historical Workbench input to test numerical behavior on SA8775P and other Qualcomm settings. The fresh SA8775P GPU-requested inference produces a saturated waveform, while the same binary's CPU-requested output on the exact SA device matches ONNX CPU within 4.13e-6 relative L2. Its 100-sample CPU-requested component profile had p50/p95 5.384/5.534 s with all 738 execution-detail rows on CPU. A FP32-preserving GPU-option control still saturated 84.89% of samples, so that option did not repair the tested path. RB3 produces an unsaturated but 15.73%-divergent waveform; its fresh 100-sample mixed CPU/GPU component p50/p95 was 3.099/3.133 s, not a usable TTS timing claim. X Elite ONNX CPU output matches the local ONNX Runtime reference on the retained fixture; its 94-sample component p50/p95 was 6.384/6.644 s with all execution-detail rows on CPU. Corrected X Elite DirectML inference and profile both failed during provider initialization; separate FP16 QNN DLC inference returned an unsaturated waveform at 1.138% relative L2 / 38.88 dB SNR against original ONNX CPU; placement profiling is pending. A separate SA8775P FP16 QNN vocoder compile passed, but same-fixture NPU-requested inference failed after compilation without a waveform or placement samples. No complete on-device TTS stream or model-quality qualification is claimed. A local HX370 x86-64 LiteRT CPU control of the identical SA/RB3 TFLite binary matched the original ONNX CPU waveform within 3.45e-6 relative L2. The SA saturation is therefore not universal to the converted artifact; the exact SA CPU control passed numerical and placement checks, while the faulty GPU delegate operation remains unidentified. -The [talker decode-step attempt](evidence/qwen_tts_talker_qualcomm/README.md) pins a separate 28-layer source and synthetic cache fixture. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU top-1 speech token with 8.57–9.43% logits relative L2. These artifacts are numerically unqualified on the fixture. RB3 rejected the floating-point-input artifact. Explicit X Elite ONNX CPU and SA8775P TFLite CPU fallback compiles passed. X Elite CPU-requested inference preserved CPU top-1 token 80 with logits relative L2 2.07e-5; its placement profile and SA inference remain pending. No complete talker loop or TTS stream passed. +The [talker decode-step attempt](evidence/qwen_tts_talker_qualcomm/README.md) pins a separate 28-layer source and synthetic cache fixture. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU top-1 speech token with 8.57–9.43% logits relative L2. These artifacts are numerically unqualified on the fixture. RB3 rejected the floating-point-input artifact. Explicit X Elite ONNX CPU and SA8775P TFLite CPU fallback compiles passed. X Elite CPU-requested inference preserved CPU top-1 token 80 with logits relative L2 2.07e-5; its placement profile remains pending. SA8775P TFLite CPU-requested talker inference also preserved token 80 with logits relative L2 3.33e-5; its placement profile is pending. No complete talker loop or TTS stream passed. The [MiniCPM-o speech-head study](evidence/minicpmo_tts_qualcomm/README.md) recovered the real 20-layer ONNX source and a matching retained fixture. S24 and S25 FP16 context-binary outputs preserved the ONNX CPU top-1 speech token with 0.346%/0.360% logits relative L2 on one step. Their 100-sample component profiles had p50/p95 8.537/9.006 ms and 7.197/7.615 ms, with all 1,029 execution-detail rows on NPU for each. FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3. SA8775P NPU-requested inference preserved CPU top-1 token 1867 with 0.3568% logits relative L2; X Elite same-fixture inference also preserved token 1867 with 0.3170% logits relative L2. SA and X Elite placement profiles remain pending. RB3 FP16 QNN inference failed at graph load despite compile success; an explicit TFLite CPU fallback compile passed with same-fixture CPU-requested inference pending. The complete multimodal model remains unqualified on those devices. -The [InternVLA Cosmos S25 attempt](evidence/internvla_cosmos_qualcomm/README.md) repackages the real policy's image encoder from its parity-tested ONNX export. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern. Injecting it into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component placement profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. Device-local action-policy support remains unqualified. +The [InternVLA Cosmos S25 attempt](evidence/internvla_cosmos_qualcomm/README.md) repackages the real policy's image encoder from its parity-tested ONNX export. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern. Injecting it into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component placement profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. Device-local action-policy support remains unqualified. The checkpoint-named Place_Markpen dataset returned HTTP 404 under the current authenticated Hugging Face account; [real-observation access evidence](evidence/internvla_real_observation_access/README.md) records the blocker without treating synthetic actions as task validation. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md new file mode 100644 index 000000000..f42485330 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md @@ -0,0 +1,5 @@ +# InternVLA real-observation data access check + +The pinned Place_Markpen checkpoint's `train_config.json` names Hugging Face dataset `a2d_real/a2d_pen_holder`. The [access record](access_report.json) pins the training-config hash and shows that `huggingface_hub` 1.30.0 received HTTP 404 for this dataset under the current authenticated account on 2026-09-24 UTC. A 404 does not distinguish an absent repository from a private repository inaccessible to this account. + +No representative real Place_Markpen observation/reference-action pairs have been obtained from that route. Existing synthetic policy outputs and hosted Cosmos component probes therefore cannot establish real-observation action accuracy, physical action units/order/step time, or task quality. The next gate is access to the named dataset or another authorized representative observation/action set, followed by a fixed reference-action comparison on each complete policy route. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/access_report.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/access_report.json new file mode 100644 index 000000000..f241e6913 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/access_report.json @@ -0,0 +1,13 @@ +{ + "timestamp_utc": "2026-09-24T00:40:50.575177+00:00", + "checkpoint_train_config_sha256": "e50ec42e40ba1925983e0e8db36558d3738bea64e1698045ff7cc08a130e1053", + "dataset_repo_id": "a2d_real/a2d_pen_holder", + "repo_type": "dataset", + "huggingface_hub_version": "1.30.0", + "authenticated": true, + "result": { + "status": "inaccessible_or_absent", + "http_status": 404, + "exception_type": "RepositoryNotFoundError" + } +} diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md index ebe0b5796..012ec4f27 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md @@ -6,7 +6,7 @@ The deterministic synthetic input dataset `d7dw6pgy2` has SHA256 `0bf538e1b64dfc Historical FP16 QNN context-binary [S24](s24/compile_report.json), [S25](s25/compile_report.json), [SA8775P](sa8775p/compile_report.json), and [Snapdragon X Elite](xelite/compile_report.json) compile jobs succeeded. Same-fixture NPU-requested inference succeeded on all four, but **all changed the CPU reference top-1 logit** on this synthetic step: CPU token 80 versus S24/SA/X Elite 1720 and S25 412. The [S24](s24/audit_report.json), [S25](s25/audit_report.json), [SA8775P](sa8775p/audit_report.json) and [X Elite](xelite/audit_report.json) audits report logits relative L2 **9.33%/8.99%/9.43%/8.57%**, respectively, with maximum K/V relative L2 **11.23%/11.71%/11.54%/11.50%**. These FP16 artifacts are numerically unqualified for this fixture; a successful graph execution is not speech-token parity. The [RB3 compile](rb3/compile_report.json) rejected floating-point input on this device. These are artifact-specific outcomes, not full-model support claims. -Explicit [X Elite ONNX CPU](xelite/cpu_fallback/compile_report.json) and [SA8775P TFLite CPU](sa8775p/cpu_fallback/compile_report.json) fallback compiles succeeded from the same source. The same-fixture [X Elite CPU-requested inference](xelite/cpu_fallback/inference_report.json) succeeded. Its [audit](xelite/cpu_fallback/audit_report.json) found logits relative L2 2.07e-5, maximum new-K/V relative L2 2.18e-5, and preserved ONNX CPU top-1 speech token 80. A [CPU-requested placement profile](xelite/cpu_fallback/profile_submission.json) was submitted and remains pending. [SA8775P CPU-requested inference](sa8775p/cpu_fallback/inference_submission.json) also remains pending. This is one numerically close decode step, not a talker loop or full TTS deployment. +Explicit [X Elite ONNX CPU](xelite/cpu_fallback/compile_report.json) and [SA8775P TFLite CPU](sa8775p/cpu_fallback/compile_report.json) fallback compiles succeeded from the same source. The same-fixture [X Elite CPU-requested inference](xelite/cpu_fallback/inference_report.json) succeeded. Its [audit](xelite/cpu_fallback/audit_report.json) found logits relative L2 2.07e-5, maximum new-K/V relative L2 2.18e-5, and preserved ONNX CPU top-1 speech token 80. A [CPU-requested placement profile](xelite/cpu_fallback/profile_submission.json) was submitted and remains pending. The [SA8775P TFLite CPU-requested inference](sa8775p/cpu_fallback/inference_report.json) also succeeded. Its [audit](sa8775p/cpu_fallback/audit_report.json) found logits relative L2 3.33e-5, maximum new-K/V relative L2 3.41e-5, and preserved CPU top-1 speech token 80. The [CPU-requested placement profile](sa8775p/cpu_fallback/profile_submission.json) is pending. Both explicit CPU routes are numerically close on one decode step, not complete talker loops or TTS deployments. The [collector](../../../../experiments/collect_aihub_component.py) retrieves terminal inference outputs without resubmission. The [tensor auditor](../../../../experiments/audit_qwen_tts_talker_qualcomm.py) enforces the source, fixture, compile target, exact device, output shape and finiteness contracts, and reports logits top-1 plus relative error for all 56 new K/V tensors. It does not convert those metrics to a quality pass without an explicit token/audio tolerance. Its 57-output contract was exercised against the pinned CPU reference as a local self-check; that self-check is not device evidence. Requested NPU placement still needs a profile. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/audit_report.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/audit_report.json new file mode 100644 index 000000000..58104b91d --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/audit_report.json @@ -0,0 +1,400 @@ +{ + "scope": "one synthetic fixed-cache 28-layer Qwen3-TTS talker decode step; no complete TTS stream", + "status": "component_executed_numeric_measured", + "device": { + "name": "SA8775P ADP", + "os": "14", + "attributes": [ + "os:android", + "framework:tflite", + "framework:onnx", + "vendor:qualcomm", + "abi:aarch64-android", + "format:auto", + "chipset:qualcomm-sa8775p", + "hexagon:v73", + "soc-model:52", + "htp-supports-fp16:true", + "htp-supports-weight-sharing:true", + "framework:qnn" + ] + }, + "compile_job_id": "jgolr4wqg", + "inference_job_id": "jpyox7el5", + "source_model_id": "mq3xre4rq", + "target_model_id": "mmd04edvn", + "input_dataset_id": "d7dw6pgy2", + "requested_run_options": "--compute_unit cpu", + "files": { + "source_onnx": { + "bytes": 1776377476, + "sha256": "41feb08517678e94d9301e992d0df610fc60e7c9faba5600703298c86554fd22" + }, + "fixture": { + "bytes": 54591500, + "sha256": "0bf538e1b64dfc9737dc4f925d61c6e4ffa5075a5dcd7afb1e85ac3bd2d9afea" + }, + "cpu_reference": { + "bytes": 237615, + "sha256": "ee0ead70e33c02c7d3d6bcb757c5cf0fcbb9eb6fc75c34abfbe8a4479657d5be" + }, + "device_output": { + "bytes": 238220, + "sha256": "d626313b519bb4fc4c831006fffaee38fd4aa70d722dd1538d8a6c01b70d7867" + } + }, + "tensor_comparison": { + "logits": { + "relative_l2": 3.331033983520798e-05, + "max_abs_error": 0.0005044937133789062, + "reference_rms": 3.7322664515389006, + "output_rms": 3.7322871955970216, + "cpu_top1": 80, + "device_top1": 80, + "top1_matches": true + }, + "k_new_0": { + "relative_l2": 6.41859715811326e-07, + "max_abs_error": 5.4836273193359375e-06, + "reference_rms": 1.1349352034374294, + "output_rms": 1.1349351709221114 + }, + "v_new_0": { + "relative_l2": 5.546668908680807e-07, + "max_abs_error": 4.172325134277344e-07, + "reference_rms": 0.1383137764927112, + "output_rms": 0.13831377250963336 + }, + "k_new_1": { + "relative_l2": 7.812657805080547e-07, + "max_abs_error": 6.67572021484375e-06, + "reference_rms": 1.4773734279001252, + "output_rms": 1.4773734396048948 + }, + "v_new_1": { + "relative_l2": 7.78749242327241e-07, + "max_abs_error": 5.364418029785156e-07, + "reference_rms": 0.162893856079411, + "output_rms": 0.16289386013484586 + }, + "k_new_2": { + "relative_l2": 1.186014449990387e-06, + "max_abs_error": 8.106231689453125e-06, + "reference_rms": 1.4749251890370425, + "output_rms": 1.47492518399937 + }, + "v_new_2": { + "relative_l2": 1.316425277428931e-06, + "max_abs_error": 1.4901161193847656e-06, + "reference_rms": 0.2996575863196337, + "output_rms": 0.29965761945737535 + }, + "k_new_3": { + "relative_l2": 1.6878881835662733e-06, + "max_abs_error": 1.0013580322265625e-05, + "reference_rms": 1.4529848472712377, + "output_rms": 1.4529848958161073 + }, + "v_new_3": { + "relative_l2": 1.725930691703213e-06, + "max_abs_error": 1.996755599975586e-06, + "reference_rms": 0.2917197159228712, + "output_rms": 0.2917197750423004 + }, + "k_new_4": { + "relative_l2": 1.8865986561149933e-06, + "max_abs_error": 1.3113021850585938e-05, + "reference_rms": 1.5413980317857385, + "output_rms": 1.5413980576547086 + }, + "v_new_4": { + "relative_l2": 2.3368984447896167e-06, + "max_abs_error": 4.649162292480469e-06, + "reference_rms": 0.41222657123793377, + "output_rms": 0.4122263786035369 + }, + "k_new_5": { + "relative_l2": 2.0853483454807167e-06, + "max_abs_error": 1.7642974853515625e-05, + "reference_rms": 1.6285454176549035, + "output_rms": 1.6285453502824736 + }, + "v_new_5": { + "relative_l2": 2.562214049334086e-06, + "max_abs_error": 3.2633543014526367e-06, + "reference_rms": 0.32638470026742417, + "output_rms": 0.3263846774219926 + }, + "k_new_6": { + "relative_l2": 2.4273033082861673e-06, + "max_abs_error": 1.7821788787841797e-05, + "reference_rms": 1.8000231156167423, + "output_rms": 1.800023142312508 + }, + "v_new_6": { + "relative_l2": 2.980671073998213e-06, + "max_abs_error": 2.562999725341797e-06, + "reference_rms": 0.2629708328253682, + "output_rms": 0.2629707830503696 + }, + "k_new_7": { + "relative_l2": 2.7560247397012616e-06, + "max_abs_error": 1.8775463104248047e-05, + "reference_rms": 1.6255722851184644, + "output_rms": 1.6255724328204177 + }, + "v_new_7": { + "relative_l2": 3.009527885433782e-06, + "max_abs_error": 3.7103891372680664e-06, + "reference_rms": 0.348368074126628, + "output_rms": 0.34836798650991385 + }, + "k_new_8": { + "relative_l2": 2.8886881992446257e-06, + "max_abs_error": 1.5497207641601562e-05, + "reference_rms": 1.516512353065435, + "output_rms": 1.5165123283605657 + }, + "v_new_8": { + "relative_l2": 3.5475424303688497e-06, + "max_abs_error": 3.5017728805541992e-06, + "reference_rms": 0.28027916002007797, + "output_rms": 0.28027912602147936 + }, + "k_new_9": { + "relative_l2": 3.5159477630415095e-06, + "max_abs_error": 3.0994415283203125e-05, + "reference_rms": 1.5960043895576705, + "output_rms": 1.5960044173808783 + }, + "v_new_9": { + "relative_l2": 4.651931229445717e-06, + "max_abs_error": 6.854534149169922e-06, + "reference_rms": 0.37541008682864946, + "output_rms": 0.37540993143659396 + }, + "k_new_10": { + "relative_l2": 4.900820154855892e-06, + "max_abs_error": 3.3736228942871094e-05, + "reference_rms": 1.6561161468173449, + "output_rms": 1.6561161320387674 + }, + "v_new_10": { + "relative_l2": 6.008033137848697e-06, + "max_abs_error": 7.659196853637695e-06, + "reference_rms": 0.26429778504066537, + "output_rms": 0.2642976398997342 + }, + "k_new_11": { + "relative_l2": 5.538137929150415e-06, + "max_abs_error": 4.100799560546875e-05, + "reference_rms": 1.5170462775366995, + "output_rms": 1.5170463147367288 + }, + "v_new_11": { + "relative_l2": 7.280477368164582e-06, + "max_abs_error": 5.751848220825195e-06, + "reference_rms": 0.22548908846990579, + "output_rms": 0.22548940933120534 + }, + "k_new_12": { + "relative_l2": 5.610020854635279e-06, + "max_abs_error": 4.4018030166625977e-05, + "reference_rms": 1.5731131793440647, + "output_rms": 1.5731131524644366 + }, + "v_new_12": { + "relative_l2": 7.293097494635962e-06, + "max_abs_error": 5.304813385009766e-06, + "reference_rms": 0.20839677682266325, + "output_rms": 0.20839708359611164 + }, + "k_new_13": { + "relative_l2": 6.5287105558340404e-06, + "max_abs_error": 3.6716461181640625e-05, + "reference_rms": 1.4939452330557514, + "output_rms": 1.493945267434375 + }, + "v_new_13": { + "relative_l2": 8.551514193977214e-06, + "max_abs_error": 8.13603401184082e-06, + "reference_rms": 0.24929462665621008, + "output_rms": 0.2492946827129621 + }, + "k_new_14": { + "relative_l2": 6.317748645220851e-06, + "max_abs_error": 5.221366882324219e-05, + "reference_rms": 1.4942429579997731, + "output_rms": 1.4942429074787813 + }, + "v_new_14": { + "relative_l2": 9.698764096173352e-06, + "max_abs_error": 1.0907649993896484e-05, + "reference_rms": 0.32824019866053383, + "output_rms": 0.32824087669987645 + }, + "k_new_15": { + "relative_l2": 7.545003744283523e-06, + "max_abs_error": 4.684925079345703e-05, + "reference_rms": 1.6561072116211841, + "output_rms": 1.656107586393447 + }, + "v_new_15": { + "relative_l2": 1.040090620524768e-05, + "max_abs_error": 7.987022399902344e-06, + "reference_rms": 0.24976283574047556, + "output_rms": 0.2497631348328584 + }, + "k_new_16": { + "relative_l2": 1.031865175719673e-05, + "max_abs_error": 6.902217864990234e-05, + "reference_rms": 1.617983543941328, + "output_rms": 1.6179835067073707 + }, + "v_new_16": { + "relative_l2": 1.2340793550880162e-05, + "max_abs_error": 1.4930963516235352e-05, + "reference_rms": 0.3826095020438836, + "output_rms": 0.382609431434366 + }, + "k_new_17": { + "relative_l2": 1.364752866368525e-05, + "max_abs_error": 0.00011086463928222656, + "reference_rms": 1.6828817809060599, + "output_rms": 1.6828815141215723 + }, + "v_new_17": { + "relative_l2": 1.5309009447573964e-05, + "max_abs_error": 2.4215085431933403e-05, + "reference_rms": 0.4656876133547257, + "output_rms": 0.465687659857562 + }, + "k_new_18": { + "relative_l2": 1.7327177701967923e-05, + "max_abs_error": 0.00010457634925842285, + "reference_rms": 1.5546463592216835, + "output_rms": 1.5546464752655822 + }, + "v_new_18": { + "relative_l2": 2.128800872936032e-05, + "max_abs_error": 4.41577285528183e-05, + "reference_rms": 0.5200020604619255, + "output_rms": 0.5200025142449275 + }, + "k_new_19": { + "relative_l2": 1.8042827459263795e-05, + "max_abs_error": 0.00013488531112670898, + "reference_rms": 1.5217070844504548, + "output_rms": 1.5217069635057079 + }, + "v_new_19": { + "relative_l2": 2.1659253215830757e-05, + "max_abs_error": 4.214048385620117e-05, + "reference_rms": 0.5402920798790236, + "output_rms": 0.5402928656440517 + }, + "k_new_20": { + "relative_l2": 2.0890461924866316e-05, + "max_abs_error": 0.0001780986785888672, + "reference_rms": 1.6813854859968058, + "output_rms": 1.681385616023159 + }, + "v_new_20": { + "relative_l2": 2.534319458677437e-05, + "max_abs_error": 6.563961505889893e-05, + "reference_rms": 0.6088730895277438, + "output_rms": 0.6088739110709308 + }, + "k_new_21": { + "relative_l2": 2.260457150040016e-05, + "max_abs_error": 0.00018072128295898438, + "reference_rms": 1.7370493419005983, + "output_rms": 1.7370492946923286 + }, + "v_new_21": { + "relative_l2": 2.797726809652519e-05, + "max_abs_error": 6.780028343200684e-05, + "reference_rms": 0.6072011016139925, + "output_rms": 0.6072000929155407 + }, + "k_new_22": { + "relative_l2": 2.5605046305493717e-05, + "max_abs_error": 0.00017595291137695312, + "reference_rms": 1.9661056736696203, + "output_rms": 1.9661055142914614 + }, + "v_new_22": { + "relative_l2": 3.0155327094988507e-05, + "max_abs_error": 7.873773574829102e-05, + "reference_rms": 0.7362191931025958, + "output_rms": 0.7362171698350103 + }, + "k_new_23": { + "relative_l2": 2.6671729116915016e-05, + "max_abs_error": 0.00019097328186035156, + "reference_rms": 1.6005997999291703, + "output_rms": 1.60060118565224 + }, + "v_new_23": { + "relative_l2": 3.4074008734635254e-05, + "max_abs_error": 6.61015510559082e-05, + "reference_rms": 0.6675198669572269, + "output_rms": 0.6675204903601081 + }, + "k_new_24": { + "relative_l2": 2.704267299460315e-05, + "max_abs_error": 0.0001538395881652832, + "reference_rms": 1.6337684549178757, + "output_rms": 1.6337684623266253 + }, + "v_new_24": { + "relative_l2": 3.2975925022545183e-05, + "max_abs_error": 9.268522262573242e-05, + "reference_rms": 0.6240472225015637, + "output_rms": 0.6240471864702112 + }, + "k_new_25": { + "relative_l2": 2.4551837497269556e-05, + "max_abs_error": 0.0001919269561767578, + "reference_rms": 1.527601662385539, + "output_rms": 1.5276006461430736 + }, + "v_new_25": { + "relative_l2": 2.8791585340796676e-05, + "max_abs_error": 0.00010532140731811523, + "reference_rms": 0.9133071626950193, + "output_rms": 0.9133053962861037 + }, + "k_new_26": { + "relative_l2": 2.5203233904938704e-05, + "max_abs_error": 0.00016385316848754883, + "reference_rms": 1.4905787414732559, + "output_rms": 1.490577617830345 + }, + "v_new_26": { + "relative_l2": 3.19054502054102e-05, + "max_abs_error": 9.194016456604004e-05, + "reference_rms": 0.7009595279765859, + "output_rms": 0.7009587143967237 + }, + "k_new_27": { + "relative_l2": 2.705629945468701e-05, + "max_abs_error": 0.00013589859008789062, + "reference_rms": 1.5453936433107067, + "output_rms": 1.5453921534264692 + }, + "v_new_27": { + "relative_l2": 3.162288021052051e-05, + "max_abs_error": 7.998943328857422e-05, + "reference_rms": 0.6731545154533489, + "output_rms": 0.6731549769955362 + } + }, + "cache_max_relative_l2": 3.4074008734635254e-05, + "limits": [ + "The historical source export does not attest an exact checkpoint revision.", + "The fixture is synthetic and has no token-to-audio quality tolerance.", + "Requested compute unit is not actual placement without a device profile.", + "No predictor, vocoder, persistent cache loop or complete TTS stream was tested." + ] +} diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/device_output.npz b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/device_output.npz new file mode 100644 index 000000000..6b152e95b Binary files /dev/null and b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/device_output.npz differ diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/inference_report.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/inference_report.json new file mode 100644 index 000000000..af384c1d8 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/inference_report.json @@ -0,0 +1,32 @@ +{ + "timestamp_utc": "2026-09-24T00:39:22.849748+00:00", + "qai_hub_version": "0.55.0", + "facility": "Qualcomm AI Hub Workbench", + "job_id": "jpyox7el5", + "url": "https://workbench.aihub.qualcomm.com/jobs/jpyox7el5/", + "status": "SUCCESS", + "status_message": "", + "options": "--compute_unit cpu", + "hub_version": "aihub-2026.09.11.0", + "device": { + "name": "SA8775P ADP", + "os": "14", + "attributes": [ + "os:android", + "framework:tflite", + "framework:onnx", + "vendor:qualcomm", + "abi:aarch64-android", + "format:auto", + "chipset:qualcomm-sa8775p", + "hexagon:v73", + "soc-model:52", + "htp-supports-fp16:true", + "htp-supports-weight-sharing:true", + "framework:qnn" + ] + }, + "model_id": "mmd04edvn", + "input_dataset_id": "d7dw6pgy2", + "outputs": "device_output.npz" +} diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/profile_submission.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/profile_submission.json new file mode 100644 index 000000000..5c3e1aad3 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/profile_submission.json @@ -0,0 +1,13 @@ +{ + "timestamp_utc": "2026-09-24T00:39:51.366579+00:00", + "facility": "Qualcomm AI Hub Workbench", + "qai_hub_version": "0.55.0", + "source_compile_job_id": "jgolr4wqg", + "target_model_id": "mmd04edvn", + "device": "SA8775P ADP", + "options": "--compute_unit cpu", + "kind": "profile", + "status": "SUBMITTED", + "job_id": "jgdd33x6g", + "url": "https://workbench.aihub.qualcomm.com/jobs/jgdd33x6g/" +} diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md index 0d3eaa1b6..9c6b29dfd 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md @@ -8,7 +8,7 @@ The [S25 historical inference](s25_reference_inference.json) on the same fixture The fresh [RB3 inference](rb3/inference_report.json) on the same fixture returned a finite, **unsaturated** waveform ([output](rb3/device_output.npz)). Its [comparison](rb3/inference_comparison.json) against original ONNX CPU was **15.73% relative L2 / 16.06 dB SNR**, still without a listening-quality threshold. The fresh [profile](rb3/profile_report.json) returned 100 component samples at nearest-rank p50/p95 **3.099/3.133 s** and 403 CPU plus 335 GPU execution-detail rows; the [audit](rb3/audit_report.json) ties the profile to this numerically unqualified output. Thus the bitwise identical SA and RB3 TFLite target downloads produce materially different outputs on their respective devices; target file identity does not establish runtime numerical parity. -A separate [local LiteRT 2.2.0 CPU control](local_litert_cpu/report.json) ran that **same pinned TFLite binary** on the WSL x86-64 HX370 host with four interpreter threads. Its [waveform](local_litert_cpu/device_output.npz) was finite and unsaturated, within **3.45e-6 relative L2 / 109.23 dB SNR** of the original ONNX Runtime CPU output. The [reproducer](../../../../experiments/probe_qwen_tts_vocoder_litert_cpu.py) checks binary/fixture/reference hashes and I/O contracts. This narrows the SA saturation to hosted runtime/device behavior rather than a universally bad TFLite conversion. The one local invoke is not a comparable performance profile; it does not prove either embedded device's CPU route or identify the specific delegate operation at fault. +A separate [local LiteRT 2.2.0 CPU control](local_litert_cpu/report.json) ran that **same pinned TFLite binary** on the WSL x86-64 HX370 host with four interpreter threads. Its [waveform](local_litert_cpu/device_output.npz) was finite and unsaturated, within **3.45e-6 relative L2 / 109.23 dB SNR** of the original ONNX Runtime CPU output. The [reproducer](../../../../experiments/probe_qwen_tts_vocoder_litert_cpu.py) checks binary/fixture/reference hashes and I/O contracts. This narrows the SA saturation to hosted runtime/device behavior rather than a universally bad TFLite conversion. The one local invoke is not a comparable performance profile and did not by itself establish embedded CPU placement or identify the specific delegate operation at fault. On the **exact hosted SA8775P**, a [CPU-requested inference](sa8775p/cpu_control/inference_report.json) of that binary also returned a finite unsaturated waveform. The [same-fixture comparison](sa8775p/cpu_control/inference_comparison.json) was **4.13e-6 relative L2 / 107.67 dB SNR** against original ONNX CPU. Its [CPU-requested profile](sa8775p/cpu_control/profile_report.json) attributed all **738 execution-detail rows to CPU**; 100 component samples had nearest-rank p50/p95 **5.384/5.534 s** (minimum 5.250 s). The [full audit](sa8775p/cpu_control/audit_report.json) ties the numerical and placement results to the same binary and fixture. This is a working **component CPU route**, roughly three times slower than the numerically failed GPU-requested component profile; neither is a complete TTS request. The GPU-requested SA output differed from this SA CPU-requested output by **11.832 relative L2** and saturated 96.63% of samples. A fresh [FP32-preserving GPU-option inference](sa8775p/gpu_fp32/inference_report.json) also failed numerically: its [comparison](sa8775p/gpu_fp32/inference_comparison.json) had **11.389 relative L2 / −21.13 dB SNR** versus original ONNX CPU and **84.89%** saturated samples. Disabling the FP32-to-FP16 allowance does not rescue this tested GPU route. The exact faulty delegate operation still requires isolation. @@ -16,6 +16,6 @@ The re-fetched *historical* [SA8775P profile](sa8775p/historical_profile_report. The X Elite [ONNX CPU inference](xelite/cpu/inference_report.json) succeeded on the pinned fixture. Its [waveform comparison](xelite/cpu/inference_comparison.json) to the original local ONNX Runtime CPU output was **8.09e-7 relative L2 / 121.84 dB SNR**, with no saturation. The fresh [CPU profile](xelite/cpu/profile_report.json) returned **94** component samples, nearest-rank p50/p95 **6.384/6.644 s**, with all **634** execution-detail rows on CPU; the service's 5.917 s estimated-time field is the sample minimum. The [full audit](xelite/cpu/audit_report.json) pins source, fixture, target, waveform, placement and percentiles. This is near-source component parity on one fixture, without a listening-quality or complete-stream gate. The first X Elite [GPU inference](xelite/gpu/inference_report.json) and [profile](xelite/gpu/profile_report.json) jobs failed because Workbench rejects specifying both compute unit and ONNX execution provider. The corrected [DirectML inference](xelite/directml/inference_report.json) and [profile](xelite/directml/profile_report.json) both failed during DML provider initialization with Windows error `80070057` (`The parameter is incorrect`), without an output or timing samples. A separate [FP16 QNN DLC compile](xelite/qnn_npu/compile_report.json) succeeded. Its [NPU-requested same-fixture inference](xelite/qnn_npu/inference_report.json) returned a finite, unsaturated waveform at **1.138% relative L2 / 38.88 dB SNR** versus original ONNX CPU ([comparison](xelite/qnn_npu/inference_comparison.json)). This is better waveform agreement on this fixture than the historical S25 GPU output, but lacks a listening-quality tolerance and verified NPU node placement; the [profile](xelite/qnn_npu/profile_submission.json) is pending. The [X Elite ONNX CPU compile](xelite/compile_report.json) passed after the older TFLite target was rejected on Windows ARM. These failures constrain the tested DirectML route, not all GPU or NPU artifacts. -A new SA8775P [FP16 QNN DLC compile](sa8775p/qnn_npu/compile_report.json) passed from the same source ONNX, producing target `mqvovl9lm`; [same-fixture NPU-requested inference](sa8775p/qnn_npu/inference_submission.json) was submitted and remains pending. This tests a separate accelerator artifact because the existing TFLite GPU output fails numerically. The compile alone has no numerical or placement conclusion. The [collector](../../../../experiments/collect_aihub_component.py) retrieves terminal inference/profile reports without resubmitting jobs, and the [audit script](../../../../experiments/audit_qwen_tts_vocoder_qualcomm.py) checks source/fixture hashes, device and model IDs, actual placement, waveform error, saturation and sample percentiles. +A new SA8775P [FP16 QNN DLC compile](sa8775p/qnn_npu/compile_report.json) passed from the same source ONNX, producing target `mqvovl9lm`; [same-fixture NPU-requested inference](sa8775p/qnn_npu/inference_report.json) failed with Workbench message "Failed to fully run the model, failed after compiling." It returned no waveform or placement samples. This separate accelerator artifact therefore does not rescue the numerically failed TFLite GPU route; the measured CPU component route remains available. The failure applies to this FP16 QNN DLC artifact, not every potential NPU export. The [collector](../../../../experiments/collect_aihub_component.py) retrieves terminal inference/profile reports without resubmitting jobs, and the [audit script](../../../../experiments/audit_qwen_tts_vocoder_qualcomm.py) checks source/fixture hashes, device and model IDs, actual placement, waveform error, saturation and sample percentiles. This is one fixed vocoder component with a retained input, not complete Qwen3-TTS. The next gates are a quality-passing SA/RB3/X Elite target, talker and code predictor numerical checks, device-resident handoff/co-residency, playable streaming, cancellation, memory admission and sustained power/thermal profiling. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/sa8775p/qnn_npu/inference_report.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/sa8775p/qnn_npu/inference_report.json new file mode 100644 index 000000000..467d50f48 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/sa8775p/qnn_npu/inference_report.json @@ -0,0 +1,31 @@ +{ + "timestamp_utc": "2026-09-24T00:41:44.947819+00:00", + "qai_hub_version": "0.55.0", + "facility": "Qualcomm AI Hub Workbench", + "job_id": "jp2rwemxg", + "url": "https://workbench.aihub.qualcomm.com/jobs/jp2rwemxg/", + "status": "FAILED", + "status_message": "Failed to fully run the model, failed after compiling.", + "options": "--compute_unit npu", + "hub_version": "aihub-2026.09.11.0", + "device": { + "name": "SA8775P ADP", + "os": "14", + "attributes": [ + "os:android", + "framework:tflite", + "framework:onnx", + "vendor:qualcomm", + "abi:aarch64-android", + "format:auto", + "chipset:qualcomm-sa8775p", + "hexagon:v73", + "soc-model:52", + "htp-supports-fp16:true", + "htp-supports-weight-sharing:true", + "framework:qnn" + ] + }, + "model_id": "mqvovl9lm", + "input_dataset_id": "d7m80jl32" +} diff --git a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md index f1fc34a58..c87b40de5 100644 --- a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md +++ b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md @@ -12,13 +12,13 @@ On a hosted [Galaxy S24 Spark attention-layer experiment](../../../e2e_expansion The same W8A16 layer also ran on hosted [Snapdragon X Elite CRD](../../../e2e_expansion_20260923/evidence/spark_xelite_full_attention/README.md) and [SA8775P ADP](../../../e2e_expansion_20260923/evidence/spark_sa8775_full_attention/README.md) NPUs, with pinned-fixture outputs identical to S24/S25; the 100-sample component p50/p95 was 1.433/1.606 ms and 1.837/2.079 ms, respectively. On [RB3 Gen 2 / QCS6490](../../../e2e_expansion_20260923/evidence/spark_rb3_full_attention/README.md), that W8A16 graph failed QNN DLC loading even after target-specific recompilation. An FP32 ONNX CPU layer then passed with source-relative L2 below 9e-7, while a calibrated W8A8 NPU layer executed but had 20.9% hidden-state relative L2 and remains numerically unqualified. These hosted component outcomes do not establish full generation or comparable device speed. -A [Qwen3-TTS Code2Wav follow-up](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) recovered source and target artifacts, a retained fixture, and raw historical profiles. On SA8775P, fresh GPU-requested inference saturated 96.63% of waveform samples and differed from original ONNX CPU by 11.832 relative L2; the same binary's CPU-requested output on the exact SA device matched ONNX CPU within 4.13e-6 relative L2. The CPU-requested route also has 100 component samples at p50/p95 5.384/5.534 s, with all 738 execution-detail rows on CPU; the GPU-requested path remains numerically failed. Fresh RB3 inference was unsaturated but differed from CPU by 15.73% relative L2 (16.06 dB SNR), without a listening-quality gate; its 100-sample mixed CPU/GPU component p50/p95 was 3.099/3.133 s. X Elite ONNX CPU vocoder inference passed on the same fixture with 8.09e-7 relative L2 versus local ONNX CPU; its 94-sample component p50/p95 was 6.384/6.644 s with all execution-detail rows on CPU. DirectML inference and profile both failed during provider initialization (Windows error 80070057); a separate FP16 QNN NPU vocoder inference returned an unsaturated waveform at 1.138% relative L2 / 38.88 dB SNR versus local ONNX CPU, with placement profiling pending. None is a complete TTS stream. +A [Qwen3-TTS Code2Wav follow-up](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) recovered source and target artifacts, a retained fixture, and raw historical profiles. On SA8775P, fresh GPU-requested inference saturated 96.63% of waveform samples and differed from original ONNX CPU by 11.832 relative L2; the same binary's CPU-requested output on the exact SA device matched ONNX CPU within 4.13e-6 relative L2. The CPU-requested route also has 100 component samples at p50/p95 5.384/5.534 s, with all 738 execution-detail rows on CPU; the GPU-requested path remains numerically failed. A separate SA FP16 QNN NPU vocoder artifact compiled but failed at inference without waveform output. Fresh RB3 inference was unsaturated but differed from CPU by 15.73% relative L2 (16.06 dB SNR), without a listening-quality gate; its 100-sample mixed CPU/GPU component p50/p95 was 3.099/3.133 s. X Elite ONNX CPU vocoder inference passed on the same fixture with 8.09e-7 relative L2 versus local ONNX CPU; its 94-sample component p50/p95 was 6.384/6.644 s with all execution-detail rows on CPU. DirectML inference and profile both failed during provider initialization (Windows error 80070057); a separate FP16 QNN NPU vocoder inference returned an unsaturated waveform at 1.138% relative L2 / 38.88 dB SNR versus local ONNX CPU, with placement profiling pending. None is a complete TTS stream. -The identical SA/RB3 TFLite binary matched original ONNX CPU within 3.45e-6 relative L2 in a separate local HX370 x86-64 LiteRT CPU control. This narrows the SA waveform saturation to hosted device/runtime behavior, without proving either embedded CPU route or a specific GPU-delegate fault. +The identical SA/RB3 TFLite binary matched original ONNX CPU within 3.45e-6 relative L2 in a separate local HX370 x86-64 LiteRT CPU control. Together with the exact SA CPU-requested parity result, this narrows the SA saturation to the GPU-requested route; the RB3 embedded CPU route and specific faulty delegate operation remain unverified. -A separate [Qwen3-TTS talker decode-step attempt](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) recovered a 28-layer ONNX export and synthetic fixed-cache fixture. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU reference top-1 speech token with 8.57–9.43% logits relative L2; they are numerically unqualified on this fixture. An RB3 compile rejected floating-point input. A separate X Elite ONNX CPU-requested talker step preserved CPU token 80 with logits relative L2 2.07e-5; placement profiling is pending. No complete talker loop or TTS stream follows. +A separate [Qwen3-TTS talker decode-step attempt](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) recovered a 28-layer ONNX export and synthetic fixed-cache fixture. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU reference top-1 speech token with 8.57–9.43% logits relative L2; they are numerically unqualified on this fixture. An RB3 compile rejected floating-point input. Separate CPU-requested talker steps on X Elite ONNX and SA8775P TFLite preserved CPU token 80 with logits relative L2 2.07e-5 and 3.33e-5; both placement profiles are pending. No complete talker loop or TTS stream follows. -A [Galaxy S25 InternVLA Cosmos attempt](../../../e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) packaged the real Place_Markpen image encoder into one ONNX file after two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern; its 100-sample six-frame component profile had p50/p95 49.665/51.382 ms and all 121 execution-detail rows on NPU. Injecting that latent into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending. This advances component and action-sensitivity evidence, not device-local full-policy qualification. +A [Galaxy S25 InternVLA Cosmos attempt](../../../e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) packaged the real Place_Markpen image encoder into one ONNX file after two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern; its 100-sample six-frame component profile had p50/p95 49.665/51.382 ms and all 121 execution-detail rows on NPU. Injecting that latent into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending. This advances component and action-sensitivity evidence, not device-local full-policy qualification. The Place_Markpen checkpoint names dataset a2d_real/a2d_pen_holder, which returned HTTP 404 under the current authenticated Hugging Face account; [the access record](../../../e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md) leaves real-observation reference-action validation open. A [MiniCPM-o speech-head study](../../../e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md) recovered the historical 20-layer source and matching S24/S25 fixtures. Both FP16 context-binary outputs preserved the ONNX CPU top-1 speech token on this one decode step with logits relative L2 of 0.346%/0.360%. S24/S25 100-sample component profiles attributed all 1,029 execution-detail rows to NPU, with p50/p95 8.537/9.006 ms and 7.197/7.615 ms. New FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3. SA8775P same-fixture NPU-requested inference returned finite outputs, preserved CPU top-1 token 1867 and had 0.3568% logits relative L2; actual NPU placement profiling is pending. X Elite same-fixture inference also preserved token 1867 with 0.3170% logits relative L2; its placement profile remains pending; RB3 FP16 QNN inference failed at graph load, and an explicit TFLite CPU fallback compile passed with same-fixture inference pending. These are speech-head component results, not multimodal MiniCPM-o requests. @@ -102,7 +102,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i | Mobile Galaxy S24 / Snapdragon 8 Gen 3 | MiniCPM-o 4.5 | COMPONENT_FP16_SPEECH_HEAD_NPU_NUMERIC_PROFILED; NOT_E2E_PROFILEABLE | The retained FP16 speech-head job used a dataset with identical tensors to the recovered S25 fixture. Against original ONNX CPU, logits relative L2 was 0.3459%, maximum new-K/V relative L2 1.235%, and top-1 speech token 1867 matched. Earlier parity reported 49.22 dB SNR/top-1 match; tested W8A16 top-1 failed. The 100-sample component profile had p50/p95 8.537/9.006 ms, with all 1,029 execution-detail rows on NPU and reported inference peak memory 165,433,344 bytes. Generated-audio quality and the rest of the model remain unqualified. | Qualify thinker/encoders/vocoder and stateful local integration; [speech-head evidence](../../../e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md). | | Mobile Galaxy S24 / Snapdragon 8 Gen 3 | InternVLA-A1 | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | Embedded SA8775P ADP / AI Hub | Spark-X2.5 | COMPONENT_NPU_INFERENCE_PROFILED; NOT_E2E_PROFILEABLE | A real Spark-X2.5-1.7B one-layer W8A16 QDQ attention artifact, compiled as QNN DLC, ran on the exact hosted SA8775P ADP (Android 14). Inference returned finite hidden/new-K/new-V tensors bitwise identical to the prior S24/S25 NPU outputs on the same fixture; the profile attributed all 73 execution-detail rows to NPU. Across 100 component samples, nearest-rank p50/p95 was 1.837/2.079 ms; the 1.749 ms estimated-time field is the minimum. Hidden/K/V relative L2 versus original FP32 ONNX CPU was 1.351%/0.377%/0.650%, without a next-token quality tolerance. There is no complete 28-layer prefill/decode/KV-ring/sampling loop, resident Omni controller or complete-request timing. | Qualify a complete embedded resident artifact/controller with state, token quality, memory admission/cancel and sustained power; [SA8775P NPU raw evidence](../../../e2e_expansion_20260923/evidence/spark_sa8775_full_attention/README.md). | -| Embedded SA8775P ADP / AI Hub | Qwen3-TTS 0.6B CustomVoice | COMPONENT_CPU_VOCODER_PARITY_AND_PLACEMENT_PASS; QNN_NPU_VOCODER_COMPILE_PASS_INFERENCE_PENDING; COMPONENT_GPU_NUMERIC_FAIL; TALKER_FP16_NUMERIC_FAIL; NOT_E2E_PROFILEABLE | On the exact SA8775P, the same TFLite binary's CPU-requested Code2Wav inference matched original ONNX CPU within 4.13e-6 relative L2 / 107.67 dB SNR, without saturation; Its 100-sample CPU-requested component profile had p50/p95 5.384/5.534 s with all 738 execution-detail rows on CPU; the service estimated-time field 5.250 s is the sample minimum. The GPU-requested output saturated 96.63% of samples with 11.832 relative L2; its fresh 100-sample mixed CPU/GPU profile p50/p95 was 1.776/1.880 s, timing failed audio. Even with FP32-preserving GPU options, 84.89% of samples saturated with 11.389 relative L2. The 28-layer FP16 QNN talker step executed but changed CPU top-1 token 80 to 1720 with 9.43% logits relative L2. A TFLite CPU talker compile passed with same-fixture inference pending. A separate FP16 QNN DLC vocoder compile passed for SA8775P, with same-fixture NPU-requested inference pending. No complete pipeline exists. | Audit the pending SA FP16 QNN vocoder inference and talker CPU control, isolate GPU delegate and talker numerical failures, then establish listening quality, co-resident streaming, memory and sustained behavior; [vocoder](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) and [talker](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) evidence. | +| Embedded SA8775P ADP / AI Hub | Qwen3-TTS 0.6B CustomVoice | COMPONENT_CPU_VOCODER_PARITY_AND_PLACEMENT_PASS; TALKER_TFLITE_CPU_NUMERIC_MEASURED; QNN_NPU_VOCODER_INFERENCE_FAILED; COMPONENT_GPU_NUMERIC_FAIL; TALKER_FP16_NUMERIC_FAIL; NOT_E2E_PROFILEABLE | On the exact SA8775P, the same TFLite binary's CPU-requested Code2Wav inference matched original ONNX CPU within 4.13e-6 relative L2 / 107.67 dB SNR, without saturation. Its 100-sample CPU-requested component profile had p50/p95 5.384/5.534 s with all 738 execution-detail rows on CPU; the service estimated-time field 5.250 s is the sample minimum. The GPU-requested output saturated 96.63% of samples with 11.832 relative L2; its fresh 100-sample mixed CPU/GPU profile p50/p95 was 1.776/1.880 s, timing failed audio. Even with FP32-preserving GPU options, 84.89% of samples saturated with 11.389 relative L2. The 28-layer FP16 QNN talker step executed but changed CPU top-1 token 80 to 1720 with 9.43% logits relative L2. A separate TFLite CPU-requested talker step preserved CPU top-1 token 80 with logits relative L2 3.33e-5 and maximum K/V relative L2 3.41e-5; its placement profile is pending. A separate FP16 QNN DLC vocoder compile passed, but same-fixture NPU-requested inference failed after compilation without a waveform or placement samples. No complete pipeline exists. | Audit talker CPU placement and isolate the FP16 QNN vocoder runtime failure, GPU delegate failure and FP16 talker numerical drift, then establish listening quality, co-resident streaming, memory and sustained behavior; [vocoder](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) and [talker](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) evidence. | | Embedded SA8775P ADP / AI Hub | Qwen3.8-27B | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | Embedded SA8775P ADP / AI Hub | MiniCPM-o 4.5 | COMPONENT_SPEECH_HEAD_NUMERIC_MEASURED; PLACEMENT_PENDING; NOT_E2E_PROFILEABLE | The real 20-layer MiniCPM-o speech-head source compiled to FP16 QNN DLC for the exact SA8775P ADP. Same-fixture NPU-requested inference returned 41 finite tensors, preserved ONNX CPU top-1 speech token 1867, and had 0.3568% logits relative L2 and 1.225% maximum K/V relative L2. The requested NPU route awaits actual node-placement and timing profile; no complete thinker/encoders/vocoder bundle or device-local stateful controller exists. | Audit the [SA8775P speech-head placement profile](../../../e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md), then qualify the complete multimodal pipeline and sustained behavior. | | Embedded SA8775P ADP / AI Hub | InternVLA-A1 | COMPONENT_COSMOS_FP16_QNN_COMPILE_PASS_INFERENCE_PENDING; NOT_E2E_PROFILEABLE | The real Place_Markpen Cosmos image encoder source compiled to FP16 QNN DLC on exact SA8775P ADP. Same-fixture NPU-requested inference is pending; no numerical output, actual placement, full action policy or local stateful Omni backend exists yet. | Audit the [SA8775P Cosmos inference](../../../e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md), then obtain placement, real-observation action validation and a complete local policy. | diff --git a/docs/design/omni_edge_e2e_roadmap.md b/docs/design/omni_edge_e2e_roadmap.md index bd7017690..3095c6f47 100644 --- a/docs/design/omni_edge_e2e_roadmap.md +++ b/docs/design/omni_edge_e2e_roadmap.md @@ -15,15 +15,15 @@ remain open; the 2026-09-22 audit remains the historical baseline. The [Spark hosted Qualcomm component expansion](../../benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/spark_s24_full_attention/README.md) advances M1 only to one real fixed-shape decoder attention layer. The same W8A16 QNN DLC ran on S24, Snapdragon X Elite CRD and SA8775P ADP NPUs with identical outputs on one pinned fixture; S25 had prior separate component evidence. On [RB3 Gen 2](../../benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/spark_rb3_full_attention/README.md), the tested W8A16 graph failed QNN loading even after an exact-device compile. FP32 ONNX ran on CPU with near-source parity, while calibrated W8A8 ran on NPU but had 20.9% hidden-state relative L2 and remains numerically unqualified. These are component C results, not full M1: device-local 28-layer prefill/continuous decode, KV/ring, sampling, token quality, admission/cancellation, complete-request timing and sustained power/thermal gates remain open. AI Hub is the test facility, not a deployment dependency. -The [Qwen3-TTS Code2Wav Qualcomm follow-up](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) adds a retained same-fixture numerical check to M2. A fresh SA8775P GPU-requested output saturated 96.63% of samples and differs from original ONNX CPU by 11.832 relative L2, so that route fails the numerical gate despite profiling. The same binary's CPU-requested output on the exact SA device matched original ONNX CPU within 4.13e-6 relative L2; its 100-sample CPU-requested component profile passed placement (738/738 execution-detail rows on CPU) at nearest-rank p50/p95 5.384/5.534 s, still without a full stream. A FP32-preserving GPU-option control still saturated 84.89% of samples, so that option did not fix the tested route. RB3 TFLite GPU-requested inference returned an unsaturated but 15.73%-relative-L2 divergent waveform; its fresh 100-sample mixed CPU/GPU component p50/p95 was 3.099/3.133 s. X Elite ONNX CPU inference matched the local ONNX Runtime output with 8.09e-7 relative L2; its 94-sample component profile was p50/p95 6.384/6.644 s with all execution-detail rows on CPU. Corrected DirectML inference and profile both failed during provider initialization; separate FP16 QNN DLC inference returned an unsaturated waveform at 1.138% relative L2 / 38.88 dB SNR versus local ONNX CPU, with placement profiling pending. A separate SA8775P FP16 QNN DLC vocoder compile passed, with NPU-requested same-fixture inference pending. M2 still requires a quality-passing talker/predictor/vocoder set, device-local co-residency and handoff, playable streaming and sustained memory/power validation. +The [Qwen3-TTS Code2Wav Qualcomm follow-up](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) adds a retained same-fixture numerical check to M2. A fresh SA8775P GPU-requested output saturated 96.63% of samples and differs from original ONNX CPU by 11.832 relative L2, so that route fails the numerical gate despite profiling. The same binary's CPU-requested output on the exact SA device matched original ONNX CPU within 4.13e-6 relative L2; its 100-sample CPU-requested component profile passed placement (738/738 execution-detail rows on CPU) at nearest-rank p50/p95 5.384/5.534 s, still without a full stream. A FP32-preserving GPU-option control still saturated 84.89% of samples, so that option did not fix the tested route. RB3 TFLite GPU-requested inference returned an unsaturated but 15.73%-relative-L2 divergent waveform; its fresh 100-sample mixed CPU/GPU component p50/p95 was 3.099/3.133 s. X Elite ONNX CPU inference matched the local ONNX Runtime output with 8.09e-7 relative L2; its 94-sample component profile was p50/p95 6.384/6.644 s with all execution-detail rows on CPU. Corrected DirectML inference and profile both failed during provider initialization; separate FP16 QNN DLC inference returned an unsaturated waveform at 1.138% relative L2 / 38.88 dB SNR versus local ONNX CPU, with placement profiling pending. A separate SA8775P FP16 QNN DLC vocoder compile passed, but NPU-requested same-fixture inference failed after compilation without waveform or placement samples. M2 still requires a quality-passing talker/predictor/vocoder set, device-local co-residency and handoff, playable streaming and sustained memory/power validation. The identical SA/RB3 TFLite vocoder binary also matched original ONNX CPU within 3.45e-6 relative L2 on a separate local HX370 x86-64 LiteRT 2.2.0 CPU control. Exact-SA CPU-requested parity narrows the failure to the GPU-requested route. The FP32-preserving GPU option did not fix it, so per-node placement and delegate-operation isolation remain open. -A separate [28-layer talker decode-step attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) retains a synthetic fixed-cache fixture and CPU reference. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU top-1 speech token with 8.57–9.43% logits relative L2. RB3 rejected that floating-point-input artifact. Explicit X Elite ONNX CPU and SA8775P TFLite CPU fallback compiles passed. X Elite CPU-requested inference preserved CPU top-1 token 80 with logits relative L2 2.07e-5; its placement profile and SA inference remain pending. None advances M2 to numerical talker or full-stream qualification yet. +A separate [28-layer talker decode-step attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) retains a synthetic fixed-cache fixture and CPU reference. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU top-1 speech token with 8.57–9.43% logits relative L2. RB3 rejected that floating-point-input artifact. Explicit X Elite ONNX CPU and SA8775P TFLite CPU fallback compiles passed. X Elite CPU-requested inference preserved CPU top-1 token 80 with logits relative L2 2.07e-5; its placement profile remains pending. SA8775P TFLite CPU-requested talker inference also preserved token 80 with logits relative L2 3.33e-5; its placement profile is pending. None advances M2 to numerical talker or full-stream qualification yet. For M4, a [MiniCPM-o 4.5 speech-head study](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md) recovered the historical 20-layer ONNX source and matching S24/S25 fixtures. Both FP16 context-binary outputs preserved the ONNX CPU top-1 speech token on one decode step, with logits relative L2 0.346%/0.360%. Their 100-sample component profiles had p50/p95 8.537/9.006 ms and 7.197/7.615 ms, with all 1,029 execution-detail rows on NPU for each. New FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3. SA8775P same-fixture NPU-requested inference preserved CPU top-1 speech token 1867 with 0.3568% logits relative L2 and 1.225% maximum K/V relative L2; placement profiling is pending. X Elite same-fixture inference also preserved token 1867 with 0.3170% logits relative L2; both placement profiles remain pending. RB3 FP16 QNN inference failed at graph load despite compile success; a TFLite CPU fallback compile passed with same-fixture CPU-requested inference pending. This is component C numerical evidence only, not thinker/encoder/vocoder or complete multimodal support. -For M4, a [Galaxy S25 Cosmos encoder attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) repackages the real Place_Markpen FP32 ONNX export after retaining two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one synthetic pattern. Injecting that output into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU; the action policy did not run on S25. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. This is component/action-sensitivity evidence, not device-local M4 qualification. +For M4, a [Galaxy S25 Cosmos encoder attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) repackages the real Place_Markpen FP32 ONNX export after retaining two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one synthetic pattern. Injecting that output into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU; the action policy did not run on S25. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. This is component/action-sensitivity evidence, not device-local M4 qualification. The checkpoint train config names a2d_real/a2d_pen_holder; an [authenticated access check](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md) returned HTTP 404, so real-observation reference-action quality is still unmeasured. The [WSL CPU expansion](../../benchmarks/edge_harness/results/e2e_expansion_20260923/README.md) adds constrained three-stage MiniCPM-o text-to-speech requests and a CPU-only