From f04fdf79da38781b237e63352a4d20cd0942d17f Mon Sep 17 00:00:00 2001 From: tzhouam Date: Thu, 24 Sep 2026 08:01:05 +0800 Subject: [PATCH] Track CPU talker controls and mobile speech-head profiles --- .../results/e2e_expansion_20260924/README.md | 2 +- .../evidence/minicpmo_tts_qualcomm/README.md | 2 +- .../s24/profile_submission.json | 13 ++++++++ .../s25/profile_submission.json | 13 ++++++++ .../qwen_tts_talker_qualcomm/README.md | 2 +- .../sa8775p/cpu_fallback/compile_report.json | 31 +++++++++++++++++++ .../cpu_fallback/inference_submission.json | 14 +++++++++ .../evidence/summary/README.md | 4 +-- docs/design/omni_edge_e2e_roadmap.md | 2 +- 9 files changed, 77 insertions(+), 6 deletions(-) create mode 100644 benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/s24/profile_submission.json create mode 100644 benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/s25/profile_submission.json create mode 100644 benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/compile_report.json create mode 100644 benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/inference_submission.json diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md index cd2f65311..fd41eee70 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md @@ -4,7 +4,7 @@ The [Qwen3-TTS vocoder study](evidence/qwen_tts_vocoder_qualcomm/README.md) reus A local HX370 x86-64 LiteRT CPU control of the identical SA/RB3 TFLite binary matched the original ONNX CPU waveform within 3.45e-6 relative L2. The SA saturation is therefore not universal to the converted artifact; hosted CPU and GPU precision controls are still needed to isolate its device/runtime cause. -The [talker decode-step attempt](evidence/qwen_tts_talker_qualcomm/README.md) pins a separate 28-layer source and synthetic cache fixture. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU top-1 speech token with 8.57–9.43% logits relative L2. These artifacts are numerically unqualified on the fixture. RB3 rejected the floating-point-input artifact, and an X Elite ONNX CPU fallback compile is pending. No complete talker loop or TTS stream passed. +The [talker decode-step attempt](evidence/qwen_tts_talker_qualcomm/README.md) pins a separate 28-layer source and synthetic cache fixture. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU top-1 speech token with 8.57–9.43% logits relative L2. These artifacts are numerically unqualified on the fixture. RB3 rejected the floating-point-input artifact. Explicit X Elite ONNX CPU and SA8775P TFLite CPU fallback compiles passed, with same-fixture inference pending. No complete talker loop or TTS stream passed. The [MiniCPM-o speech-head study](evidence/minicpmo_tts_qualcomm/README.md) recovered the real 20-layer ONNX source and a matching retained fixture. S24 and S25 FP16 context-binary outputs preserved the ONNX CPU top-1 speech token with 0.346%/0.360% logits relative L2 on one step. FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3, with same-fixture NPU-requested inference pending. The complete multimodal model remains unqualified on those devices. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md index bbcb4c556..c8c3cc036 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md @@ -4,7 +4,7 @@ This is a **single fixed-cache 20-layer speech-head step**, not the complete thi The [S25 historical inference](s25/inference_report.json) used dataset `d7gwxe3y2`; the [S24 historical inference](s24/inference_report.json) used `d7zn8rk57`. Downloaded inputs from both dataset IDs produced the same 44-array `.npz` SHA256 `65919ea1f351feec33c1415a3e3db76d60bc9c2beb3d49a72af75dcb418e83b4`. The retained fixture is outside Git under `/home/zhout/project/edge_infer/models/minicpmo45_tts_step_fixture_s25_20260924.npz`. ONNX Runtime 1.29 CPU produced a 41-output reference (SHA256 `902adcceadbe761d938ef2ade51cf953c8917ea1abfaee33a10c84e300c1bd96`), also outside Git. -The historical [S24](s24/compile_report.json) and [S25](s25/compile_report.json) FP16 QNN context-binary compiles and inference jobs succeeded. Against the **same original ONNX CPU fixture**, [S24](s24/audit_report.json) logits relative L2 was **0.3459%** and max K/V relative L2 **1.235%**; [S25](s25/audit_report.json) logits relative L2 was **0.3605%** and max K/V relative L2 **1.311%**. Both preserved the CPU top-1 speech token **1867** on this one synthetic step. The raw [S24](s24/device_output.npz) and [S25](s25/device_output.npz) arrays are retained. These values are numerical component evidence, without a generated-audio quality threshold or actual NPU placement profile. +The historical [S24](s24/compile_report.json) and [S25](s25/compile_report.json) FP16 QNN context-binary compiles and inference jobs succeeded. Against the **same original ONNX CPU fixture**, [S24](s24/audit_report.json) logits relative L2 was **0.3459%** and max K/V relative L2 **1.235%**; [S25](s25/audit_report.json) logits relative L2 was **0.3605%** and max K/V relative L2 **1.311%**. Both preserved the CPU top-1 speech token **1867** on this one synthetic step. The raw [S24](s24/device_output.npz) and [S25](s25/device_output.npz) arrays are retained. [S24](s24/profile_submission.json) and [S25](s25/profile_submission.json) placement profiles were submitted and pending. These values remain numerical component evidence, without a generated-audio quality threshold or actual NPU placement proof. New FP16 QNN DLC [SA8775P](sa8775p/compile_report.json), [Snapdragon X Elite](xelite/compile_report.json) and [RB3 Gen 2](rb3/compile_report.json) compiles passed from the same source. Same-fixture NPU-requested inference jobs were [submitted for SA8775P](sa8775p/inference_submission.json), [X Elite](xelite/inference_submission.json) and [RB3](rb3/inference_submission.json), but were pending at this snapshot. A compile cannot establish device execution or numerical parity. The [collector](../../../../experiments/collect_aihub_component.py) retrieves terminal jobs, and the [auditor](../../../../experiments/audit_minicpmo_tts_qualcomm.py) pins hashes and checks all 41 output tensors. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/s24/profile_submission.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/s24/profile_submission.json new file mode 100644 index 000000000..4eff22419 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/s24/profile_submission.json @@ -0,0 +1,13 @@ +{ + "timestamp_utc": "2026-09-23T23:56:58.273469+00:00", + "facility": "Qualcomm AI Hub Workbench", + "qai_hub_version": "0.55.0", + "source_compile_job_id": "jp2ryv3qg", + "target_model_id": "mqp411wgq", + "device": "Samsung Galaxy S24", + "options": "--compute_unit npu", + "kind": "profile", + "status": "SUBMITTED", + "job_id": "jglyx1qj5", + "url": "https://workbench.aihub.qualcomm.com/jobs/jglyx1qj5/" +} diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/s25/profile_submission.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/s25/profile_submission.json new file mode 100644 index 000000000..5ab403c98 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/s25/profile_submission.json @@ -0,0 +1,13 @@ +{ + "timestamp_utc": "2026-09-23T23:56:59.942164+00:00", + "facility": "Qualcomm AI Hub Workbench", + "qai_hub_version": "0.55.0", + "source_compile_job_id": "jp4yr9j1p", + "target_model_id": "mn7zvvw8m", + "device": "Samsung Galaxy S25", + "options": "--compute_unit npu", + "kind": "profile", + "status": "SUBMITTED", + "job_id": "j5687d06g", + "url": "https://workbench.aihub.qualcomm.com/jobs/j5687d06g/" +} diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md index a01155bc1..b72540f69 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md @@ -6,7 +6,7 @@ The deterministic synthetic input dataset `d7dw6pgy2` has SHA256 `0bf538e1b64dfc Historical FP16 QNN context-binary [S24](s24/compile_report.json), [S25](s25/compile_report.json), [SA8775P](sa8775p/compile_report.json), and [Snapdragon X Elite](xelite/compile_report.json) compile jobs succeeded. Same-fixture NPU-requested inference succeeded on all four, but **all changed the CPU reference top-1 logit** on this synthetic step: CPU token 80 versus S24/SA/X Elite 1720 and S25 412. The [S24](s24/audit_report.json), [S25](s25/audit_report.json), [SA8775P](sa8775p/audit_report.json) and [X Elite](xelite/audit_report.json) audits report logits relative L2 **9.33%/8.99%/9.43%/8.57%**, respectively, with maximum K/V relative L2 **11.23%/11.71%/11.54%/11.50%**. These FP16 artifacts are numerically unqualified for this fixture; a successful graph execution is not speech-token parity. The [RB3 compile](rb3/compile_report.json) rejected floating-point input on this device. These are artifact-specific outcomes, not full-model support claims. -Explicit [X Elite ONNX CPU](xelite/cpu_fallback/compile_submission.json) and [SA8775P TFLite CPU](sa8775p/cpu_fallback/compile_submission.json) fallback compiles were submitted from the same source to test separate numerical routes. Their results were pending at this snapshot; neither is an NPU pass or a full TTS deployment. +Explicit [X Elite ONNX CPU](xelite/cpu_fallback/compile_report.json) and [SA8775P TFLite CPU](sa8775p/cpu_fallback/compile_report.json) fallback compiles succeeded from the same source. Same-fixture [X Elite](xelite/cpu_fallback/inference_submission.json) and [SA8775P](sa8775p/cpu_fallback/inference_submission.json) CPU-requested inference jobs were pending at this snapshot. Neither compile alone is a numerical pass or a full TTS deployment. The [collector](../../../../experiments/collect_aihub_component.py) retrieves terminal inference outputs without resubmission. The [tensor auditor](../../../../experiments/audit_qwen_tts_talker_qualcomm.py) enforces the source, fixture, compile target, exact device, output shape and finiteness contracts, and reports logits top-1 plus relative error for all 56 new K/V tensors. It does not convert those metrics to a quality pass without an explicit token/audio tolerance. Its 57-output contract was exercised against the pinned CPU reference as a local self-check; that self-check is not device evidence. Requested NPU placement still needs a profile. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/compile_report.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/compile_report.json new file mode 100644 index 000000000..d79adcd36 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/compile_report.json @@ -0,0 +1,31 @@ +{ + "retrieved_utc": "2026-09-23T23:58:28.870827+00:00", + "facility": "Qualcomm AI Hub Workbench", + "qai_hub_version": "0.55.0", + "job_id": "jgolr4wqg", + "url": "https://workbench.aihub.qualcomm.com/jobs/jgolr4wqg/", + "status": "SUCCESS", + "status_message": "", + "source_model_id": "mq3xre4rq", + "target_model_id": "mmd04edvn", + "device": { + "name": "SA8775P ADP", + "os": "14", + "attributes": [ + "os:android", + "framework:tflite", + "framework:onnx", + "vendor:qualcomm", + "abi:aarch64-android", + "format:auto", + "chipset:qualcomm-sa8775p", + "hexagon:v73", + "soc-model:52", + "htp-supports-fp16:true", + "htp-supports-weight-sharing:true", + "framework:qnn" + ] + }, + "options": "--target_runtime tflite", + "hub_version": "aihub-2026.09.11.0" +} diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/inference_submission.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/inference_submission.json new file mode 100644 index 000000000..978201467 --- /dev/null +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/sa8775p/cpu_fallback/inference_submission.json @@ -0,0 +1,14 @@ +{ + "timestamp_utc": "2026-09-23T23:58:30.656978+00:00", + "facility": "Qualcomm AI Hub Workbench", + "qai_hub_version": "0.55.0", + "source_compile_job_id": "jgolr4wqg", + "target_model_id": "mmd04edvn", + "input_dataset_id": "d7dw6pgy2", + "device": "SA8775P ADP", + "options": "--compute_unit cpu", + "kind": "inference", + "status": "SUBMITTED", + "job_id": "jpyox7el5", + "url": "https://workbench.aihub.qualcomm.com/jobs/jpyox7el5/" +} diff --git a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md index 0380258c0..36f61df4f 100644 --- a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md +++ b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md @@ -87,7 +87,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i | PC CPU+iGPU+NPU +/- discrete GPU, joint model execution | MiniCPM-o 4.5 | NOT_E2E_PROFILEABLE | No qualifying same-model multi-accelerator deployment/artifact plan or overlap/recovery evidence. Individual routes are not a joint pipeline. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | PC CPU+iGPU+NPU +/- discrete GPU, joint model execution | InternVLA-A1 | NOT_E2E_PROFILEABLE | No qualifying same-model multi-accelerator deployment/artifact plan or overlap/recovery evidence. Individual routes are not a joint pipeline. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | PC Snapdragon X Elite CRD / AI Hub | Spark-X2.5 | COMPONENT_NPU_INFERENCE_PROFILED; NOT_E2E_PROFILEABLE | A real Spark-X2.5-1.7B one-layer W8A16 QDQ attention artifact, compiled as QNN DLC, ran on the exact hosted Snapdragon X Elite CRD (Windows 11 ARM64). Inference returned finite hidden/new-K/new-V tensors bitwise identical to the prior S24/S25 NPU outputs on the same fixture; the profile attributed all 73 execution-detail rows to NPU. Across 100 component samples, nearest-rank p50/p95 was 1.433/1.606 ms; the 1.346 ms estimated-time field is the minimum. Hidden/K/V relative L2 versus original FP32 ONNX CPU was 1.351%/0.377%/0.650%, without a next-token quality tolerance. There is no complete 28-layer prefill/decode/KV-ring/sampling loop, resident Omni controller or complete-request timing. | Qualify a complete X Elite resident artifact/controller with state, token quality, memory admission/cancel and sustained power; [X Elite NPU raw evidence](../../../e2e_expansion_20260923/evidence/spark_xelite_full_attention/README.md). | -| PC Snapdragon X Elite CRD / AI Hub | Qwen3-TTS 0.6B CustomVoice | COMPONENT_ONNX_CPU_VOCODER_PARITY_PROFILED; DIRECTML_RUNTIME_FAILED; TALKER_FP16_NUMERIC_FAIL; QNN_NPU_VOCODER_COMPILE_PASS; NOT_E2E_PROFILEABLE | The fresh ONNX vocoder inference on the exact Windows 11 ARM64 X Elite CRD requested CPU and returned an unsaturated waveform on the pinned fixture, matching local ONNX Runtime CPU with 8.09e-7 relative L2 and 121.84 dB SNR. Its 94-sample component profile had nearest-rank p50/p95 6.384/6.644 s and all 634 execution-detail rows on CPU; 5.917 s estimated time is the sample minimum. This is one component, without a listening-quality gate. Corrected DirectML inference and profile both failed at provider initialization with Windows error 80070057. A separate FP16 QNN DLC vocoder compile passed, but NPU-requested inference was pending. The 28-layer FP16 QNN talker step executed, yet changed CPU top-1 token 80 to 1720 with 8.57% logits relative L2 on a synthetic cache fixture; a separate ONNX CPU talker compile was submitted. The older TFLite vocoder target was rejected on Windows ARM; no complete stream exists. | Audit QNN vocoder and CPU talker controls, investigate DirectML failure, then qualify complete local streaming, memory and sustained behavior; [vocoder](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) and [talker](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) evidence. | +| PC Snapdragon X Elite CRD / AI Hub | Qwen3-TTS 0.6B CustomVoice | COMPONENT_ONNX_CPU_VOCODER_PARITY_PROFILED; DIRECTML_RUNTIME_FAILED; TALKER_FP16_NUMERIC_FAIL; QNN_NPU_VOCODER_COMPILE_PASS; NOT_E2E_PROFILEABLE | The fresh ONNX vocoder inference on the exact Windows 11 ARM64 X Elite CRD requested CPU and returned an unsaturated waveform on the pinned fixture, matching local ONNX Runtime CPU with 8.09e-7 relative L2 and 121.84 dB SNR. Its 94-sample component profile had nearest-rank p50/p95 6.384/6.644 s and all 634 execution-detail rows on CPU; 5.917 s estimated time is the sample minimum. This is one component, without a listening-quality gate. Corrected DirectML inference and profile both failed at provider initialization with Windows error 80070057. A separate FP16 QNN DLC vocoder compile passed, but NPU-requested inference was pending. The 28-layer FP16 QNN talker step executed, yet changed CPU top-1 token 80 to 1720 with 8.57% logits relative L2 on a synthetic cache fixture; a separate ONNX CPU talker compile passed, with inference pending. The older TFLite vocoder target was rejected on Windows ARM; no complete stream exists. | Audit QNN vocoder and CPU talker controls, investigate DirectML failure, then qualify complete local streaming, memory and sustained behavior; [vocoder](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) and [talker](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) evidence. | | PC Snapdragon X Elite CRD / AI Hub | Qwen3.8-27B | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | PC Snapdragon X Elite CRD / AI Hub | MiniCPM-o 4.5 | COMPONENT_SPEECH_HEAD_QNN_COMPILE_PASS; INFERENCE_PENDING; NOT_E2E_PROFILEABLE | The real 20-layer MiniCPM-o speech-head source compiled to FP16 QNN DLC for the exact X Elite CRD. Same-fixture NPU-requested inference was pending; no numerical output or placement profile exists. Thinker/encoders/vocoder and a device-local stateful Omni backend remain unqualified. | Audit the [X Elite speech-head inference](../../../e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md), then validate the complete multimodal pipeline, state, quality, memory and sustained behavior. | | PC Snapdragon X Elite CRD / AI Hub | InternVLA-A1 | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | @@ -102,7 +102,7 @@ NOT E2E means no complete device/model workload was verified. “Scoped E2E” i | Mobile Galaxy S24 / Snapdragon 8 Gen 3 | MiniCPM-o 4.5 | COMPONENT_FP16_SPEECH_HEAD_NUMERIC_MEASURED; NOT_E2E_PROFILEABLE | The retained FP16 speech-head job used a dataset with identical tensors to the recovered S25 fixture. Against original ONNX CPU, logits relative L2 was 0.3459%, maximum new-K/V relative L2 1.235%, and top-1 speech token 1867 matched. Earlier parity reported 49.22 dB SNR/top-1 match; tested W8A16 top-1 failed. Actual NPU placement, generated-audio quality and the rest of the model remain unqualified. | Qualify placement, thinker/encoders/vocoder and stateful local integration; [speech-head evidence](../../../e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md). | | Mobile Galaxy S24 / Snapdragon 8 Gen 3 | InternVLA-A1 | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | Embedded SA8775P ADP / AI Hub | Spark-X2.5 | COMPONENT_NPU_INFERENCE_PROFILED; NOT_E2E_PROFILEABLE | A real Spark-X2.5-1.7B one-layer W8A16 QDQ attention artifact, compiled as QNN DLC, ran on the exact hosted SA8775P ADP (Android 14). Inference returned finite hidden/new-K/new-V tensors bitwise identical to the prior S24/S25 NPU outputs on the same fixture; the profile attributed all 73 execution-detail rows to NPU. Across 100 component samples, nearest-rank p50/p95 was 1.837/2.079 ms; the 1.749 ms estimated-time field is the minimum. Hidden/K/V relative L2 versus original FP32 ONNX CPU was 1.351%/0.377%/0.650%, without a next-token quality tolerance. There is no complete 28-layer prefill/decode/KV-ring/sampling loop, resident Omni controller or complete-request timing. | Qualify a complete embedded resident artifact/controller with state, token quality, memory admission/cancel and sustained power; [SA8775P NPU raw evidence](../../../e2e_expansion_20260923/evidence/spark_sa8775_full_attention/README.md). | -| Embedded SA8775P ADP / AI Hub | Qwen3-TTS 0.6B CustomVoice | COMPONENT_CPU_REQUESTED_VOCODER_PARITY_PASS; COMPONENT_GPU_NUMERIC_FAIL; TALKER_FP16_NUMERIC_FAIL; NOT_E2E_PROFILEABLE | On the exact SA8775P, the same TFLite binary's CPU-requested Code2Wav inference matched original ONNX CPU within 4.13e-6 relative L2 / 107.67 dB SNR, without saturation; CPU placement/timing profile was pending. The GPU-requested output saturated 96.63% of samples with 11.832 relative L2; its fresh 100-sample mixed CPU/GPU profile p50/p95 was 1.776/1.880 s, timing failed audio. Even with FP32-preserving GPU options, 84.89% of samples saturated with 11.389 relative L2. The 28-layer FP16 QNN talker step executed but changed CPU top-1 token 80 to 1720 with 9.43% logits relative L2. No complete pipeline exists. | Isolate GPU delegate and talker numerical failures, audit pending CPU profile, then establish listening quality, co-resident streaming, memory and sustained behavior; [vocoder](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) and [talker](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) evidence. | +| Embedded SA8775P ADP / AI Hub | Qwen3-TTS 0.6B CustomVoice | COMPONENT_CPU_REQUESTED_VOCODER_PARITY_PASS; COMPONENT_GPU_NUMERIC_FAIL; TALKER_FP16_NUMERIC_FAIL; NOT_E2E_PROFILEABLE | On the exact SA8775P, the same TFLite binary's CPU-requested Code2Wav inference matched original ONNX CPU within 4.13e-6 relative L2 / 107.67 dB SNR, without saturation; CPU placement/timing profile was pending. The GPU-requested output saturated 96.63% of samples with 11.832 relative L2; its fresh 100-sample mixed CPU/GPU profile p50/p95 was 1.776/1.880 s, timing failed audio. Even with FP32-preserving GPU options, 84.89% of samples saturated with 11.389 relative L2. The 28-layer FP16 QNN talker step executed but changed CPU top-1 token 80 to 1720 with 9.43% logits relative L2. A TFLite CPU talker compile passed with same-fixture inference pending. No complete pipeline exists. | Isolate GPU delegate and talker numerical failures, audit pending CPU controls, then establish listening quality, co-resident streaming, memory and sustained behavior; [vocoder](../../../e2e_expansion_20260924/evidence/qwen_tts_vocoder_qualcomm/README.md) and [talker](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) evidence. | | Embedded SA8775P ADP / AI Hub | Qwen3.8-27B | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | Embedded SA8775P ADP / AI Hub | MiniCPM-o 4.5 | COMPONENT_SPEECH_HEAD_QNN_COMPILE_PASS; INFERENCE_PENDING; NOT_E2E_PROFILEABLE | The real 20-layer MiniCPM-o speech-head source compiled to FP16 QNN DLC for the exact SA8775P ADP. Same-fixture NPU-requested inference was pending, so output parity and actual placement are unknown. No complete thinker/encoders/vocoder bundle or device-local stateful controller exists. | Audit the [SA8775P speech-head inference](../../../e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md), then qualify the complete multimodal pipeline and sustained behavior. | | Embedded SA8775P ADP / AI Hub | InternVLA-A1 | NOT_E2E_PROFILEABLE | No complete qualified target artifact and device-local stateful Omni backend. Hub jobs provide component execution, not an on-device generation/streaming controller; ADB has no attached device. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | diff --git a/docs/design/omni_edge_e2e_roadmap.md b/docs/design/omni_edge_e2e_roadmap.md index b3305f5fd..d4b118d15 100644 --- a/docs/design/omni_edge_e2e_roadmap.md +++ b/docs/design/omni_edge_e2e_roadmap.md @@ -19,7 +19,7 @@ The [Qwen3-TTS Code2Wav Qualcomm follow-up](../../benchmarks/edge_harness/result The identical SA/RB3 TFLite vocoder binary also matched original ONNX CPU within 3.45e-6 relative L2 on a separate local HX370 x86-64 LiteRT 2.2.0 CPU control. Exact-SA CPU-requested parity narrows the failure to the GPU-requested route. The FP32-preserving GPU option did not fix it, so per-node placement and delegate-operation isolation remain open. -A separate [28-layer talker decode-step attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) retains a synthetic fixed-cache fixture and CPU reference. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU top-1 speech token with 8.57–9.43% logits relative L2. RB3 rejected that floating-point-input artifact. An explicit X Elite ONNX CPU fallback compile was pending. None advances M2 to numerical talker or full-stream qualification yet. +A separate [28-layer talker decode-step attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) retains a synthetic fixed-cache fixture and CPU reference. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU top-1 speech token with 8.57–9.43% logits relative L2. RB3 rejected that floating-point-input artifact. Explicit X Elite ONNX CPU and SA8775P TFLite CPU fallback compiles passed, with same-fixture inference pending. None advances M2 to numerical talker or full-stream qualification yet. For M4, a [MiniCPM-o 4.5 speech-head study](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md) recovered the historical 20-layer ONNX source and matching S24/S25 fixtures. Both FP16 context-binary outputs preserved the ONNX CPU top-1 speech token on one decode step, with logits relative L2 0.346%/0.360%. New FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3, while same-fixture NPU-requested inference remained pending. This is component C numerical evidence only, not thinker/encoder/vocoder or complete multimodal support.