vllm-omni/docs/design/omni_edge_e2e_roadmap.md

# Local PC/mobile end-to-end support and profiling roadmap

This plan covers the [60 named device/model pairings](../../benchmarks/edge_harness/results/model_device_matrix_20260922/README.md).
Each pairing must finish with either a qualified local pipeline or a documented
feasibility/implementation blocker. It does not promise that every model fits
every device. The [current profiling review](../../benchmarks/edge_harness/results/e2e_profiling_20260922/README.md)
records measured execution separately from release qualification.

The [2026-09-23 recovery run](../../benchmarks/edge_harness/results/e2e_recovery_20260923/README.md)
adds a functional CPU TTS stream with playback underruns, a scoped RTX
Qwen3.8 text/image pass, verified MiniCPM-o and InternVLA weights, a MiniCPM-o
three-stage text-to-speech functional pass after correcting request embeddings,
and an InternVLA real-weight synthetic action forward. Model-quality and device gates
remain open; the 2026-09-22 audit remains the historical baseline.

The [WSL CPU expansion](../../benchmarks/edge_harness/results/e2e_expansion_20260923/README.md)
adds constrained three-stage MiniCPM-o text-to-speech requests and a CPU-only
InternVLA synthetic policy forward. A subsequent 20-request serial MiniCPM-o
profile measured p50 21.86 s/p95 22.97 s request wall time after one warmup, with
swap use under the 30.91 GiB WSL limit. It does not qualify speech quality,
concurrency, streaming, loading peak or sustained behavior. A separate native
Windows RTX 5090 Laptop MiniCPM-o three-stage text-to-speech request passed
with explicit 900/600 s startup limits after the default 300 s overall timeout
had expired. A later native Windows 20-request serial run measured p50 18.38 s
and p95 18.73 s, but sampled host available RAM fell to 0.78 GB and pagefile
use reached 11.17 GB. Native Windows InternVLA direct-policy runs separately
passed on CPU and RTX 5090 Laptop with synthetic inputs; AMD and mobile cells
do not inherit these passes.

The WSL RTX MiniCPM-o image-to-text+speech path now passes for one 448×448
synthetic red-square image under an explicit 0.61 thinker GPU budget. The
previous 0.58 budget failed the 2,048-token KV admission gate; the successful
run still force-killed stage 0 during shutdown and reported a shared-memory
cleanup warning. A separate 2 s, 440 Hz synthetic tone now traverses audio
input to text+speech and yielded a plausible whistling description, also with
a shared-memory cleanup warning. Native Windows RTX also completed these two
separate synthetic image and tone inputs through all three stages, returning
the same respective text and thinker token IDs as WSL with nonzero WAVs.
Broader image/audio quality, video, repeated multimodal latency and clean
lifecycle remain open.

WSL CPU MiniCPM-o now also completes one synthetic red-square image and one
440 Hz tone through thinker, talker and vocoder under separate one-item plans.
Both return plausible text and nonzero WAVs, but CPU wording/token IDs differ
from GPU, swap is heavily used, and stage 0 requires forced shutdown with
shared-memory cleanup warnings. These single requests do not inherit the
20-request text latency profile or establish real image/audio quality.

Native Windows HX370 CPU now also completes two Qwen3-TTS CustomVoice
text-to-WAV requests through Qwen's standalone PyTorch wrapper using the pinned
0.6B checkpoint. A separate 1-warmup/20-measured serial profile of one short
prompt took p50/p95 13.94/14.68 s to generate 4.56 s of audio. Whisper tiny.en
ASR gave word error rates 0.20 and 0.00 for the two WAVs, an intelligibility
proxy rather than a quality pass. The [raw record](../../benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/qwen_tts_native_cpu/README.md)
documents the isolated Transformers 4.57.3 dependency and short process-memory
trace. The subsequent [Omni CPU whole-session stage](../../benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/qwen_tts_omni_cpu/README.md)
matched both standalone PCM hashes exactly, returned a public `AsyncOmni`
audio event, and completed one warmup plus 20 serial measured requests at
p50/p95 15.32/15.60 s for 4.56 s of audio. It reserves 10 GiB host RAM,
rejects an insufficient 4 GiB demand before worker launch, and drains an
in-flight cancellation without stale output or remaining reservation. This
closes the scoped CPU stage-binding task. Playable streaming, post-cancel
restart, loading-peak admission, broad speech quality and sustained
power/thermal tests remain M2 work.

A separate native Windows CPU+Radeon 890M Qwen3-TTS route now completes two
named text-to-WAV requests in standalone CrispASR using a Q8_0 talker, F16
codec, Vulkan talker/codec and CPU FP32 code predictor. Whisper tiny.en returned
both reference sentences exactly; one warmup and 20 serial resident-server
requests measured p50/p95 3.310/3.345 s for 2.56 s of audio. The
[placement and failed-route record](../../benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/qwen_tts_radeon890m_joint/README.md)
shows that default Vulkan silently becomes CPU for this backend, all-Vulkan
generation ran away, and DirectML did not pass quality. The explicit hybrid
route remains outside Omni and slower than real time. Its CPU+iGPU evidence
does not advance the NPU joint-execution row or mobile M2 gate.

## Architecture and prerequisites

Keep Omni's PipelineConfig, StageRuntime, StageClient, orchestration and admission
as the control plane. Keep vLLM's model execution, caches and internal batching.
Other runtimes implement stage backends. Model adapters own preprocessing,
sampling, modality alignment, state semantics and output interpretation.

The portable boundary is request/events, tensor or buffer contracts, opaque
state handles, device capabilities, artifact manifests and execution plans.
Persistent weights, KV/ring/recurrent state and workspaces stay with the backend.
Prefer one backend per autoregressive session; split coarse stages only after
measuring a complete-chain benefit.

Android may use a small native controller derived from Omni's stage, session and
streaming semantics, constrained by the same contract tests. A full Python/vLLM
service port is not a prerequisite. AI Hub remains a validation facility rather
than a deployment dependency.

Before advancing individual cells:

1. Pin installed runtime builds and checkpoint revisions. Repair the CPU
   vLLM/Omni mismatch and native Windows CPU attention-operator gap independently.
2. The complete MiniCPM-o 4.5 and InternVLA-A1 source checkpoints are now
   SHA-256 verified. Record export, quantization, calibration, compiler, shape
   buckets, state layout and quality evidence before target-specific claims.
3. Integrate native and external stages into the shared admission ledger.
   CPU/iGPU/NPU share physical RAM; Windows availability and WSL quota are
   separate constraints on that RAM. Budget loading peaks, weights, state,
   activations, workspace, transfers and safety margin. Discrete VRAM is separate.
4. Require bounded queues, consumer acknowledgement, cancellation propagation,
   sequence/epoch fencing, state retirement and actual placement reporting.
   No silent precision/model/context changes or CPU fallback labelled as NPU.
5. Record exact SKU/RAM, OS, drivers, installed packages, power state and device
   access. A local mobile application needs full on-device execution access;
   AI Hub component jobs alone do not establish that access or an E2E pipeline.

Implementation ownership follows the existing source boundaries:
[portable contracts](../../packages/omni-stage-contracts/omni_stage_contracts),
[host routes and process lifetime](../../vllm_omni/host),
[StageClient](../../vllm_omni/engine/stage_client.py) and
[backend adapters](../../vllm_omni/engine/backends),
[StageRuntime](../../vllm_omni/engine/stage_runtime.py), and the
[resource ledger](../../vllm_omni/engine/resource_ledger.py).
The current `GraphStageClient` represents one bounded, non-preemptible graph
call. Persistent mobile sessions require an explicitly negotiated backend
capability and lifecycle implementation; a state-handle type alone is insufficient.

## Work packages and completion gates

| Order | Work package | Implementation | Gate before wider rollout |
|---|---|---|---|
| Foundation | Runtime, contracts and accounting | Close the prerequisites above; reuse the existing qualification runners | Reproducible launch or explicit pre-load refusal, correct state ownership and placement, auditable memory budget |
| M1 | S25 Spark | Device-local Omni-compatible controller; embedding, prefill, continuous decode, ring/full KV, head and sampler; resident state | Reference checks, at least 128 output tokens, 512-window/1024-bucket transitions, cancellation, resident memory and sustained profile |
| M2 | Qwen3-TTS | Resolve desktop playback and shutdown findings and CPU compatibility; then mobile talker/predictor plus GPU vocoder | Complete PCM and tail, history/chunk correctness, interruption/recovery, quality checks, no post-startup underruns and RTF below 1 in the declared workload |
| M3 | AMD stages | Connect compatible real encoders or other useful coarse stages through current Omni workers; start from the existing 890M vision evidence | Actual node/device placement, numerical and task quality, complete downstream output, shared-memory and handoff costs; reject unhelpful splits |
| M4a | MiniCPM-o 4.5 | Three-stage desktop text+WAV produced coherent text and nonzero WAV on RTX 5090 Laptop under WSL and native Windows, and WSL CPU, using separate constrained plans. One synthetic red-square image and one synthetic tone each passed through the three-stage WSL RTX, native Windows RTX and WSL CPU paths to text and nonzero WAV. RTX text/token IDs matched across OS environments; CPU wording/token IDs differed while remaining plausible. The WSL RTX image plan needed a thinker GPU budget increase from 0.58 to 0.61 for KV admission. WSL multimodal shutdown cleanup remains open, including forced stage-0 termination on CPU. The CPU plan needed vLLM 0.28/0.29 processor compatibility and used swap; 20 serial **text** CPU requests measured p50 21.86 s/p95 22.97 s. Native Windows needed longer startup limits; 20 serial text requests measured p50 18.38 s/p95 18.73 s with host RAM/pagefile pressure. Add video encoder and persistent-state reference before mobile artifacts | Speech intelligibility/alignment, loading-peak admission, concurrency, broad image/audio-understanding suites, video, shutdown lifecycle, then combined streaming and interruption |
| M4b | InternVLA-A1 | Real weights, observation preprocessing, shared prefix state, iterative action head and versioned action metadata; strict-load synthetic policy forwards now pass on WSL and native Windows, each on CPU and RTX 5090 Laptop | Reference action agreement on real observations, units/order/timestamps, observation age, stale-input handling and application-specific deadline reporting |
| Independent | Qwen3.8-27B | Public Omni pipeline binding; text, then image/video and longer contexts; same-model CPU/iGPU artifact and explicit offload evaluation | Modality-specific quality, actual loading/runtime memory and measured performance; each precision/context receives its own qualification |

The initial M0 desktop text acceptance remains scoped to its tested checkpoint,
precision and workload. New concurrency, state-boundary or quality evidence is
required before broadening that acceptance. Current batch-dependent greedy
outputs need reference analysis; timing alone cannot identify their cause.

The 2026-09-23 [Spark GGUF Omni follow-up](../../benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/spark_omni_llamacpp/README.md) adds a separate native Windows CPU/Radeon 890M whole-session option. Its pinned 1.7B Q4_K_M artifact passed two named complete text requests and 20 serial inventory requests per device through `StageRuntime`/`StagePool`; public `AsyncOmni` short requests passed on both devices. It verifies CPU or Vulkan1 placement, explicit 4 GiB host reservation, terminal event/acknowledgement and cancellation drain. This does not change the vLLM M0 acceptance or establish M1 mobile generation. Its next gates are incremental output, post-cancel restart, broader token/quality parity, loading/device-memory peaks, concurrency and sustained power/thermal behavior.

The [Qwen3-TTS Omni hybrid follow-up](../../benchmarks/edge_harness/results/e2e_expansion_20260923/evidence/qwen_tts_omni_hybrid/README.md) adds one complete-request stage for the native Windows HX370 CPU+Radeon 890M route. A pinned Q8_0 talker and F16 codec produced two named PCM outputs exactly matching the prior standalone backend; one warmup plus 20 serial requests measured 3.298/3.336 s p50/p95 for 2.56 s of audio. Public `AsyncOmni` audio, cancellation drain and explicit 8 GiB host admission passed. This is progress toward M2 integration, not M2 acceptance: RTF remains above 1, and playable chunk streaming, post-cancel restart, loading/iGPU peaks, broader voice quality and 30-minute thermal behavior remain open. No NPU participated.

VLA qualification here ends at action data. Physical robot control, collision
avoidance and actuator safety belong to the external controller.

## Hardware routes

Every row below covers all five model families through the work packages above.
Specific artifact/backend feasibility is a gate, not assumed from the route.

| Execution configuration | Route to qualify | Main additional gate |
|---|---|---|
| CPU only, WSL | Compatible vLLM CPU; a model-specific external CPU backend where required | ISA, complete model operations, artifact format and total RAM |
| CPU only, native Windows | Native CPU worker implementing the same contracts | Executable attention/state operations and matching model adapter; CUDA success does not cover this row |
| CPU + NVIDIA, WSL | Complete-model reference using vLLM/Omni CUDA | Full modalities and quality, stable power conditions, loading and sustained memory |
| CPU + NVIDIA, native Windows | Native installation of the same declared pipeline | IPC/process lifetime, filesystem/startup scope, streaming and shutdown validated separately |
| CPU + Radeon 890M | Compatible whole-model backend or CPU model plus coarse iGPU stages | Full downstream model result and transfers, not only an encoder result |
| CPU + AMD NPU | CPU model plus compiled NPU stages | Accepted artifact, actual NPU placement, quality and net E2E contribution |
| CPU+iGPU+NPU, with/without NVIDIA | Select stage placements after establishing individual routes | Co-resident memory/power; identical-workload single-backend versus split-serial versus split-overlapped comparison |
| Snapdragon X Elite PC | Windows ARM controller and target-specific CPU/GPU/QNN stages | Full application access, Windows ARM artifacts and RAM; Android binaries are not interchangeable |
| Galaxy S25 | Spark first, TTS second; memory-qualified MiniCPM-o/VLA afterward | Complete device-local state/streaming loop and sustained measurements |
| Galaxy S24 | Rebuild and revalidate the selected mobile pipeline | Independent SoC/runtime/quality qualification; S25 results do not transfer |
| SA8775P ADP | Target-local controller and compatible compiled stages | Complete application access, parity, memory and sustained co-residency |
| RB3 Gen 2 / QCS6490 | Supported integer NPU artifacts or an explicitly chosen CPU/GPU route | Resolve rejected artifact constraints, then full-model quality and performance |

Expand the joint-execution row into separate deployments with and without
NVIDIA when implementing it. Likewise, checkpoint size, precision, RAM SKU and
OS variants require separate execution plans and results beneath each overview
cell; a pass for one variant does not qualify the entire model or device family.

For Qwen27, nominal four-bit language weights alone are about 12.6 GiB;
the actual artifact, remaining weights, state, workspace and OS require more.
Gate small-memory targets before export/integration investment. Record an
artifact-specific capacity rejection rather than substituting a smaller model.
Treat unenumerated mobile/embedded SKUs as new qualification rows.

## Profiling protocol for every executable cell

Use [the profiling harness](../../benchmarks/edge_harness/PROFILING.md) and keep
raw samples, outputs, configuration, logs and traces. Record the actual installed
runtime as well as the source revision. Run competing workloads serially on
the same physical machine.

| Dimension | Required measurements |
|---|---|
| Startup | Compilation, process/import/preparation, constructor/load, first output and warm restart separately; explicit cache conditions |
| Workload | Short/medium/long inputs, concurrency 1 then 2/4 where admitted, at least 20 measured requests per group; separate warmup and state-boundary cases |
| Text | TTFT, delivery/token intervals, decode rate, complete request time; effective prefill separated from isolated device compute |
| Speech | First playable PCM, RTF, chunk arrival gaps, playback deficit count/duration, tail/flush and interruption latency |
| Multimodal/VLA | Preprocessing, encoders, downstream completion, modality alignment, observation-to-action age and deadline misses |
| Memory | Loading/runtime peaks, state growth with context/concurrency, workspace and transfers; physical RAM, WSL quota and VRAM kept distinct |
| Placement | Actual executed devices and fallback; copies, conversions, synchronization, stage queues and handoff time |
| Sustained behavior | At least 30 minutes; power mode, clocks, temperature, memory and latency trends; energy only within available sensor scope |
| Reliability | Slow consumers, cancellation, repeated sessions, memory admission and owned-worker failure; no stale/duplicate output and verified retirement |
| Instrumentation | Separate matched before/traced/after diagnostics; profiler RPC/export costs; retain uninstrumented timing baseline |

Report p50/p95 with sample counts and measurement boundaries. Preserve failures
and retries. Component sums are not E2E measurements, and whole-GPU power is
not CPU/NPU or whole-device energy. Low-clock/power-limited measurements remain
valid for those observed conditions; they do not establish optimized performance.

For heterogeneous candidates, hold inputs, artifact quality, context, concurrency
and power conditions fixed. Keep a split only if the full chain improves latency,
capacity or energy after transfers, synchronization, initialization and contention.

## Per-cell completion and repair record

Progress each cell through artifact readiness, correctness, complete local
execution, profiling and qualification. Functional and performance qualification
are separate fields. A final record contains:

- Exact device/OS and artifact/backend identities; declared modalities and limits.
- Reproducible command and explicit execution plan, including actual placement.
- Quality, state/streaming and memory-admission outcomes with raw evidence.
- Performance distributions, sustained behavior and sensor limitations.
- On failure: failing layer, configuration, observed reason, evidence and the
  smallest next repair/experiment. Distinguish missing prerequisites from an
  attempted run that failed, a rejected artifact and an unqualified result.

Text latency and VLA deadlines require workload-specific product targets.
Do not invent universal thresholds. A completed timing protocol alone is not
a release pass. The current 60-cell matrix and its profiling follow-up are the
starting backlog; subsequent repairs should replace only the affected cells
after repeating their complete gates.