diff --git a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md index b5ad5970e..5b69fb569 100644 --- a/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md +++ b/benchmarks/edge_harness/results/e2e_profiling_20260922/evidence/summary/README.md @@ -1,16 +1,16 @@ # E2E check and profiling -All 60 pairings have a disposition. Profiling completion does not imply full release qualification. +All 60 pairings have a disposition. The timing tables below remain the 2026-09-22 measurements. The configuration matrix and affected per-cell reasons incorporate the [2026-09-23 scoped recovery runs](../../../e2e_recovery_20260923/README.md); those single-request or short-stream runs are not additional completed profiling protocols. Profiling completion does not imply full release qualification. ## Configuration × model -NOT E2E means the current artifacts/runtime cannot run the complete requested pipeline. See the per-cell reasons below. +NOT E2E means no complete device/model workload was verified. “Scoped E2E” identifies only the named input/output path; “synthetic policy pass” does not establish a real robot-task result. See the per-cell reasons below. | Device | Spark-X2.5 | Qwen3-TTS 0.6B CustomVoice | Qwen3.8-27B | MiniCPM-o 4.5 | InternVLA-A1 | |---|---|---|---|---|---| -| PC HX370: CPU / WSL | profile completed | NOT E2E | NOT E2E | NOT E2E | NOT E2E | +| PC HX370: CPU / WSL | profile completed | scoped E2E: short stream | NOT E2E | NOT E2E | NOT E2E | | PC HX370: CPU / native Windows | NOT E2E | NOT E2E | NOT E2E | NOT E2E | NOT E2E | -| PC HX370 + RTX 5090 Laptop / WSL | profile completed | profile completed | NOT E2E | NOT E2E | NOT E2E | +| PC HX370 + RTX 5090 Laptop / WSL | profile completed | profile completed | scoped E2E: text + image | scoped E2E: text → speech | synthetic policy pass | | PC HX370 + RTX 5090 Laptop / Windows | profile completed | profile completed | NOT E2E | NOT E2E | NOT E2E | | PC HX370 + Radeon 890M / Windows worker | NOT E2E | NOT E2E | NOT E2E | NOT E2E | NOT E2E | | PC HX370 + AMD NPU / Windows worker | NOT E2E | NOT E2E | NOT E2E | NOT E2E | NOT E2E | @@ -26,25 +26,25 @@ NOT E2E means the current artifacts/runtime cannot run the complete requested pi | Device | Model | Current profiling disposition | Reason / remaining work | Next step | |---|---|---|---|---| | PC HX370: CPU / WSL | Spark-X2.5 | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Qualify a matching vLLM 0.29 CPU build and token parity separately. | -| PC HX370: CPU / WSL | Qwen3-TTS 0.6B CustomVoice | NOT_E2E_PROFILEABLE | Fresh real-weight two-frame probe loaded both stages but generation failed in sanitize_min_tokens_stop_ids: expected four tuple fields, got three. Existing CPU wheel is vLLM 0.28 versus Omni 0.29. No audio pass. | Use a matching CPU runtime or explicitly validated compatibility fix; repeat real audio, then full streaming gates. | +| PC HX370: CPU / WSL | Qwen3-TTS 0.6B CustomVoice | SCOPED_E2E | The 2026-09-23 compatibility fix enabled real-weight two-frame PCM and four short-stream requests, including cancellation recovery. One measured request generated 3.04 s of audio in 17.02 s (RTF 5.60) with four simulated playback underruns. The CPU wheel remains vLLM 0.28 versus Omni 0.29. | Qualify a matching CPU runtime, speech quality, uninterrupted playback and sustained load; [recovery evidence](../../../e2e_recovery_20260923/README.md). | | PC HX370: CPU / WSL | Qwen3.8-27B | NOT_E2E_PROFILEABLE | No compatible CPU artifact/deployment for this model was found in the checked model roots. Available Qwen 27B FP8/NVFP4 CUDA artifacts do not establish a CPU route. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | -| PC HX370: CPU / WSL | MiniCPM-o 4.5 | NOT_E2E_PROFILEABLE | No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | -| PC HX370: CPU / WSL | InternVLA-A1 | NOT_E2E_PROFILEABLE | No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | +| PC HX370: CPU / WSL | MiniCPM-o 4.5 | NOT_E2E_PROFILEABLE | The exact MiniCPM-o 4.5 checkpoint is now downloaded and SHA-256 verified, but no CPU memory admission or complete CPU pipeline has been run. The RTX result does not establish CPU support. | Validate a CPU execution plan and complete text/speech path under its actual memory budget; [weight evidence](../../../e2e_recovery_20260923/evidence/weights/minicpmo.json). | +| PC HX370: CPU / WSL | InternVLA-A1 | NOT_E2E_PROFILEABLE | Base and Place_Markpen checkpoints are now SHA-256 verified, but the policy has only run on CUDA with synthetic inputs. No CPU policy forward or task-quality result exists. | Admit the full CPU model and run the policy on real observations with reference actions; [recovery evidence](../../../e2e_recovery_20260923/README.md). | | PC HX370: CPU / native Windows | Spark-X2.5 | NOT_E2E_PROFILEABLE | The installed native Windows vLLM extension lacks cpu_attn_reshape_and_cache and cpu_attn_get_scheduler_metadata. The separate ORT CPU graph worker is not a whole-model text/TTS adapter; a qualified CPU build/binding is missing. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | PC HX370: CPU / native Windows | Qwen3-TTS 0.6B CustomVoice | NOT_E2E_PROFILEABLE | The installed native Windows vLLM extension lacks cpu_attn_reshape_and_cache and cpu_attn_get_scheduler_metadata. The separate ORT CPU graph worker is not a whole-model text/TTS adapter; a qualified CPU build/binding is missing. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | PC HX370: CPU / native Windows | Qwen3.8-27B | NOT_E2E_PROFILEABLE | The installed native Windows vLLM extension lacks cpu_attn_reshape_and_cache and cpu_attn_get_scheduler_metadata. The separate ORT CPU graph worker is not a whole-model text/TTS adapter; a qualified CPU build/binding is missing. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | -| PC HX370: CPU / native Windows | MiniCPM-o 4.5 | NOT_E2E_PROFILEABLE | No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | -| PC HX370: CPU / native Windows | InternVLA-A1 | NOT_E2E_PROFILEABLE | No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | +| PC HX370: CPU / native Windows | MiniCPM-o 4.5 | NOT_E2E_PROFILEABLE | The exact checkpoint is verified in the local Linux model store, but the installed native Windows vLLM extension lacks required CPU attention operators; no native full-model run or memory admission exists. | Establish a native Windows CPU backend and model access, then run the full requested path; [weight evidence](../../../e2e_recovery_20260923/evidence/weights/minicpmo.json). | +| PC HX370: CPU / native Windows | InternVLA-A1 | NOT_E2E_PROFILEABLE | Base and fine-tuned checkpoints are verified in the local Linux model store. No native Windows policy run, model access or qualified CPU backend exists; the installed vLLM CPU attention operators are incomplete. | Validate a native policy backend, real observations and reference actions; [recovery evidence](../../../e2e_recovery_20260923/README.md). | | PC HX370 + RTX 5090 Laptop / WSL | Spark-X2.5 | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Broaden only with a named workload/device/precision qualification. | | PC HX370 + RTX 5090 Laptop / WSL | Qwen3-TTS 0.6B CustomVoice | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Resolve and measure playback stalls; run quality, interruption and thermal tests. | -| PC HX370 + RTX 5090 Laptop / WSL | Qwen3.8-27B | NOT_E2E_PROFILEABLE | Fresh public Omni startup rejects the real NVFP4 checkpoint: no registered Omni pipeline. Historical bare-vLLM short-text success does not qualify this integration or multimodal inference. | Add/validate an Omni pipeline binding; rerun text, image, context and artifact-quality tests. | -| PC HX370 + RTX 5090 Laptop / WSL | MiniCPM-o 4.5 | NOT_E2E_PROFILEABLE | No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | -| PC HX370 + RTX 5090 Laptop / WSL | InternVLA-A1 | NOT_E2E_PROFILEABLE | No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | +| PC HX370 + RTX 5090 Laptop / WSL | Qwen3.8-27B | SCOPED_E2E | A new one-stage Omni binding ran the real NVFP4 checkpoint: one 16-token text request and one synthetic red-image request (answer `Red`) at an explicit 512-token context with host offload. This is not the full multimodal, long-context or quality workload. | Qualify image/video suites, longer contexts, latency distributions and sustained power; [recovery evidence](../../../e2e_recovery_20260923/README.md). | +| PC HX370 + RTX 5090 Laptop / WSL | MiniCPM-o 4.5 | SCOPED_E2E | The verified real checkpoint completed one three-stage text-to-text+speech request under a constrained 24 GiB profile with 5 GiB host offload. Text was coherent and a nonzero 5.2 s WAV was produced in a single 20.53 s request; speech meaning was not checked. | Validate speech intelligibility/alignment, image and audio input, interruption, sustained load and power; [raw log](../../../e2e_recovery_20260923/evidence/minicpmo/mini_fixed_e2e.log). | +| PC HX370 + RTX 5090 Laptop / WSL | InternVLA-A1 | SYNTHETIC_POLICY_PASS | Strict real-weight Place_Markpen load and one policy forward on zero observations/noise produced finite `[1, 50, 32]` actions in 1.222 s. Synthetic input and one timing sample do not establish robot-task E2E or action semantics. | Run real observation/reference-action cases and validate units, timestamps and deadline behavior; [raw result](../../../e2e_recovery_20260923/evidence/internvla/internvla_finetune_synthetic.json). | | PC HX370 + RTX 5090 Laptop / Windows | Spark-X2.5 | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Broaden only with a named workload/device/precision qualification. | | PC HX370 + RTX 5090 Laptop / Windows | Qwen3-TTS 0.6B CustomVoice | PROFILE_COMPLETED | not fully qualified: reference-quality and remaining reliability/performance gates must be reviewed | Resolve and measure playback stalls; run quality, interruption and thermal tests. | -| PC HX370 + RTX 5090 Laptop / Windows | Qwen3.8-27B | NOT_E2E_PROFILEABLE | Fresh public Omni startup rejects the real NVFP4 checkpoint: no registered Omni pipeline. Historical bare-vLLM short-text success does not qualify this integration or multimodal inference. | Add/validate an Omni pipeline binding; rerun text, image, context and artifact-quality tests. | -| PC HX370 + RTX 5090 Laptop / Windows | MiniCPM-o 4.5 | NOT_E2E_PROFILEABLE | No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | -| PC HX370 + RTX 5090 Laptop / Windows | InternVLA-A1 | NOT_E2E_PROFILEABLE | No complete matching checkpoint found in the inventoried workspace, HF cache and native model roots. MiniCPM5-2B is not MiniCPM-o 4.5. Runtime registration alone is not a model run. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | +| PC HX370 + RTX 5090 Laptop / Windows | Qwen3.8-27B | NOT_E2E_PROFILEABLE | The Omni pipeline binding now exists and passed scoped text/image cases in WSL, but no native Windows Omni run has verified the NVFP4 checkpoint, memory admission or modality behavior. | Run the same checkpoint and explicit context on native Windows, then broaden quality and profiling; [WSL-only evidence](../../../e2e_recovery_20260923/README.md). | +| PC HX370 + RTX 5090 Laptop / Windows | MiniCPM-o 4.5 | NOT_E2E_PROFILEABLE | The exact checkpoint is verified and one constrained three-stage text/speech request passed in WSL; no native Windows execution, memory admission or speech-quality check exists. | Repeat the real three-stage pipeline on native Windows and verify speech and sustained behavior; [WSL-only evidence](../../../e2e_recovery_20260923/README.md). | +| PC HX370 + RTX 5090 Laptop / Windows | InternVLA-A1 | NOT_E2E_PROFILEABLE | Base and fine-tuned checkpoints are verified, but the only policy forward used synthetic observations under WSL/CUDA; no native Windows run or real-task qualification exists. | Run a native Windows policy forward, then compare real observations against reference actions; [WSL-only evidence](../../../e2e_recovery_20260923/README.md). | | PC HX370 + Radeon 890M / Windows worker | Spark-X2.5 | NOT_E2E_PROFILEABLE | No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | PC HX370 + Radeon 890M / Windows worker | Qwen3-TTS 0.6B CustomVoice | NOT_E2E_PROFILEABLE | No placement- and quality-verified whole-model graph bundle/stateful adapter for this model on this AMD route. Small backend graphs do not establish model support. | Supply and validate the missing matching artifact/backend, then run the complete specified workload. | | PC HX370 + Radeon 890M / Windows worker | Qwen3.8-27B | NOT_E2E_PROFILEABLE | FP32 448x448 vision tower: DirectML 85/113 nodes; normalized max error 9.095e-06 versus torch CPU. Historical component path, not full 27B or current graph-stage image-to-text integration. | Recover the hashed tower bundle and qualify its current Omni graph handoff with the language stage. |