From d8995de9ec565b15ebcec3e131cbad5273fdbd0c Mon Sep 17 00:00:00 2001 From: tzhouam Date: Thu, 24 Sep 2026 08:49:40 +0800 Subject: [PATCH] Record official InternVLA dataset access gate --- .../results/e2e_expansion_20260924/README.md | 2 +- .../internvla_real_observation_access/README.md | 4 +++- .../access_report.json | 12 ++++++++++++ .../evidence/summary/README.md | 2 +- docs/design/omni_edge_e2e_roadmap.md | 2 +- 5 files changed, 18 insertions(+), 4 deletions(-) diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md index b5872df1f..d61bf78bf 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/README.md @@ -8,4 +8,4 @@ The [talker decode-step attempt](evidence/qwen_tts_talker_qualcomm/README.md) pi The [MiniCPM-o speech-head study](evidence/minicpmo_tts_qualcomm/README.md) recovered the real 20-layer ONNX source and a matching retained fixture. S24 and S25 FP16 context-binary outputs preserved the ONNX CPU top-1 speech token with 0.346%/0.360% logits relative L2 on one step. Their 100-sample component profiles had p50/p95 8.537/9.006 ms and 7.197/7.615 ms, with all 1,029 execution-detail rows on NPU for each. FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3. SA8775P NPU-requested inference preserved CPU top-1 token 1867 with 0.3568% logits relative L2; X Elite same-fixture inference also preserved token 1867 with 0.3170% logits relative L2. SA and X Elite placement profiles remain pending. RB3 FP16 QNN inference failed at graph load despite compile success; an explicit TFLite CPU fallback compile passed with same-fixture CPU-requested inference pending. The complete multimodal model remains unqualified on those devices. -The [InternVLA Cosmos S25 attempt](evidence/internvla_cosmos_qualcomm/README.md) repackages the real policy's image encoder from its parity-tested ONNX export. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern. Injecting it into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component placement profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. Device-local action-policy support remains unqualified. The checkpoint-named Place_Markpen dataset returned HTTP 404 under the current authenticated Hugging Face account; [real-observation access evidence](evidence/internvla_real_observation_access/README.md) records the blocker without treating synthetic actions as task validation. +The [InternVLA Cosmos S25 attempt](evidence/internvla_cosmos_qualcomm/README.md) repackages the real policy's image encoder from its parity-tested ONNX export. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern. Injecting it into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component placement profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. Device-local action-policy support remains unqualified. The checkpoint-named Place_Markpen dataset returned HTTP 404, and the official vLLM-Omni example archive in InternData-A1 returned HTTP 403 gated-access denial under the current authenticated account; [real-observation access evidence](evidence/internvla_real_observation_access/README.md) records the blocker without treating synthetic actions as task validation. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md index f42485330..c6d259184 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md @@ -2,4 +2,6 @@ The pinned Place_Markpen checkpoint's `train_config.json` names Hugging Face dataset `a2d_real/a2d_pen_holder`. The [access record](access_report.json) pins the training-config hash and shows that `huggingface_hub` 1.30.0 received HTTP 404 for this dataset under the current authenticated account on 2026-09-24 UTC. A 404 does not distinguish an absent repository from a private repository inaccessible to this account. -No representative real Place_Markpen observation/reference-action pairs have been obtained from that route. Existing synthetic policy outputs and hosted Cosmos component probes therefore cannot establish real-observation action accuracy, physical action units/order/step time, or task quality. The next gate is access to the named dataset or another authorized representative observation/action set, followed by a fixed reference-action comparison on each complete policy route. +The [official vLLM-Omni InternVLA example](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/examples/offline_inference/internvla_a1/) names an alternative real-observation archive: `InternRobotics/InternData-A1`, path `real_lerobotv30/genie1/Genie1-Place_Markpen.tar.gz`. A path-info request under the same authenticated account returned HTTP **403 GatedRepoError**; the archive size and content could not be inspected or downloaded. This is a concrete access gate, not evidence that the archive is absent. + +No representative real Place_Markpen observation/reference-action pairs have been obtained from either route. Existing synthetic policy outputs and hosted Cosmos component probes therefore cannot establish real-observation action accuracy, physical action units/order/step time, or task quality. The next gate is access to the official archive or another authorized representative observation/action set, followed by a fixed reference-action comparison on each complete policy route. diff --git a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/access_report.json b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/access_report.json index f241e6913..382cccbee 100644 --- a/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/access_report.json +++ b/benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/access_report.json @@ -9,5 +9,17 @@ "status": "inaccessible_or_absent", "http_status": 404, "exception_type": "RepositoryNotFoundError" + }, + "official_example_archive": { + "checked_utc": "2026-09-24T00:48:33.364235+00:00", + "documentation_url": "https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/examples/offline_inference/internvla_a1/", + "dataset_repo_id": "InternRobotics/InternData-A1", + "repo_type": "dataset", + "path": "real_lerobotv30/genie1/Genie1-Place_Markpen.tar.gz", + "result": { + "status": "gated_access_denied", + "http_status": 403, + "exception_type": "GatedRepoError" + } } } diff --git a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md index d8d4b2cdd..14bc517da 100644 --- a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md +++ b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md @@ -18,7 +18,7 @@ The identical SA/RB3 TFLite binary matched original ONNX CPU within 3.45e-6 rela A separate [Qwen3-TTS talker decode-step attempt](../../../e2e_expansion_20260924/evidence/qwen_tts_talker_qualcomm/README.md) recovered a 28-layer ONNX export and synthetic fixed-cache fixture. Historical FP16 QNN context-binary inference executed on S24, S25, SA8775P and X Elite, but all four changed the CPU reference top-1 speech token with 8.57–9.43% logits relative L2; they are numerically unqualified on this fixture. An RB3 compile rejected floating-point input. Separate CPU-requested talker steps on X Elite ONNX and SA8775P TFLite preserved CPU token 80 with logits relative L2 2.07e-5 and 3.33e-5; both placement profiles are pending. No complete talker loop or TTS stream follows. -A [Galaxy S25 InternVLA Cosmos attempt](../../../e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) packaged the real Place_Markpen image encoder into one ONNX file after two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern; its 100-sample six-frame component profile had p50/p95 49.665/51.382 ms and all 121 execution-detail rows on NPU. Injecting that latent into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending. This advances component and action-sensitivity evidence, not device-local full-policy qualification. The Place_Markpen checkpoint names dataset a2d_real/a2d_pen_holder, which returned HTTP 404 under the current authenticated Hugging Face account; [the access record](../../../e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md) leaves real-observation reference-action validation open. +A [Galaxy S25 InternVLA Cosmos attempt](../../../e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) packaged the real Place_Markpen image encoder into one ONNX file after two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one pattern; its 100-sample six-frame component profile had p50/p95 49.665/51.382 ms and all 121 execution-detail rows on NPU. Injecting that latent into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending. This advances component and action-sensitivity evidence, not device-local full-policy qualification. The Place_Markpen checkpoint names dataset a2d_real/a2d_pen_holder, which returned HTTP 404 under the current authenticated Hugging Face account. The official vLLM-Omni example names a different Place_Markpen archive in InternData-A1, but access to that gated dataset returned HTTP 403; [the access record](../../../e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md) leaves real-observation reference-action validation open. A [MiniCPM-o speech-head study](../../../e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md) recovered the historical 20-layer source and matching S24/S25 fixtures. Both FP16 context-binary outputs preserved the ONNX CPU top-1 speech token on this one decode step with logits relative L2 of 0.346%/0.360%. S24/S25 100-sample component profiles attributed all 1,029 execution-detail rows to NPU, with p50/p95 8.537/9.006 ms and 7.197/7.615 ms. New FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3. SA8775P same-fixture NPU-requested inference returned finite outputs, preserved CPU top-1 token 1867 and had 0.3568% logits relative L2; actual NPU placement profiling is pending. X Elite same-fixture inference also preserved token 1867 with 0.3170% logits relative L2; its placement profile remains pending; RB3 FP16 QNN inference failed at graph load, and an explicit TFLite CPU fallback compile passed with same-fixture inference pending. These are speech-head component results, not multimodal MiniCPM-o requests. diff --git a/docs/design/omni_edge_e2e_roadmap.md b/docs/design/omni_edge_e2e_roadmap.md index 3095c6f47..ae1dc1edf 100644 --- a/docs/design/omni_edge_e2e_roadmap.md +++ b/docs/design/omni_edge_e2e_roadmap.md @@ -23,7 +23,7 @@ A separate [28-layer talker decode-step attempt](../../benchmarks/edge_harness/r For M4, a [MiniCPM-o 4.5 speech-head study](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/minicpmo_tts_qualcomm/README.md) recovered the historical 20-layer ONNX source and matching S24/S25 fixtures. Both FP16 context-binary outputs preserved the ONNX CPU top-1 speech token on one decode step, with logits relative L2 0.346%/0.360%. Their 100-sample component profiles had p50/p95 8.537/9.006 ms and 7.197/7.615 ms, with all 1,029 execution-detail rows on NPU for each. New FP16 QNN DLC compiles passed for SA8775P, X Elite and RB3. SA8775P same-fixture NPU-requested inference preserved CPU top-1 speech token 1867 with 0.3568% logits relative L2 and 1.225% maximum K/V relative L2; placement profiling is pending. X Elite same-fixture inference also preserved token 1867 with 0.3170% logits relative L2; both placement profiles remain pending. RB3 FP16 QNN inference failed at graph load despite compile success; a TFLite CPU fallback compile passed with same-fixture CPU-requested inference pending. This is component C numerical evidence only, not thinker/encoder/vocoder or complete multimodal support. -For M4, a [Galaxy S25 Cosmos encoder attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) repackages the real Place_Markpen FP32 ONNX export after retaining two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one synthetic pattern. Injecting that output into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU; the action policy did not run on S25. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. This is component/action-sensitivity evidence, not device-local M4 qualification. The checkpoint train config names a2d_real/a2d_pen_holder; an [authenticated access check](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md) returned HTTP 404, so real-observation reference-action quality is still unmeasured. +For M4, a [Galaxy S25 Cosmos encoder attempt](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_cosmos_qualcomm/README.md) repackages the real Place_Markpen FP32 ONNX export after retaining two CPU parity fixtures. FLOAT16 QNN DLC inference returned a latent within 0.0524% relative L2 of the source on one synthetic pattern. Injecting that output into the real policy on local CPU changed synthetic actions by 0.742% relative L2, without a task tolerance. S25 six-frame component profile returned 100 samples at p50/p95 49.665/51.382 ms, with all 121 execution-detail rows on NPU; the action policy did not run on S25. The same encoder compiled to FP16 QNN DLC for X Elite CRD, with same-fixture NPU-requested inference pending; an SA8775P FP16 QNN DLC compile passed, with same-fixture NPU-requested inference pending. This is component/action-sensitivity evidence, not device-local M4 qualification. The checkpoint train config names a2d_real/a2d_pen_holder; an [authenticated access check](../../benchmarks/edge_harness/results/e2e_expansion_20260924/evidence/internvla_real_observation_access/README.md) returned HTTP 404. The official vLLM-Omni example names an InternData-A1 Place_Markpen archive, but that dataset returned HTTP 403 gated-access denial under the current account, so real-observation reference-action quality is still unmeasured. The [WSL CPU expansion](../../benchmarks/edge_harness/results/e2e_expansion_20260923/README.md) adds constrained three-stage MiniCPM-o text-to-speech requests and a CPU-only