vllm-omni/docs/design/omni_edge_backends.md

Local edge backends in Omni

The first backend is external.graph.v1: one complete, stateless graph operation per request, with a fixed input bucket and copied host tensors. It uses the normal Omni / AsyncOmni APIs, PipelineConfig, StageRuntime, StagePool and orchestrator. Native vLLM stages retain their model execution, KV cache, sampling and batching. No accelerator is mandatory.

Boundaries

Owner	Responsibility
omni_stage_contracts	Version/features, buffer descriptors, request/event identities, opaque state, physical device descriptors, artifact manifests. Root import uses only the standard library.
vllm_omni.host	Explicit interpreter/OS routes, process bootstrap, process identity and bounded tree retirement.
vllm_omni.engine.backends	Adapt backend operations to StageClientBase; observe placement; fence epochs and worker generations.
StageRuntime	Resolve all replicas, reserve the whole plan before loading, create clients, unwind failures.
ResourceLedger	Atomically charge every affected memory constraint; keep uncertain allocations quarantined.
StagePool / orchestrator	Existing stage dispatch, handoff, polling, metrics, cancellation and request cleanup.
Model adapter	Tokenization, media preparation, stage transforms, sampling semantics, artifact export and numerical/task validation.

Native Windows device-init locks use the host temporary directory, independent of a local or UNC checkout. This coordinates same-user native processes. It does not establish a cross-user or Windows/WSL global lock authority; cross-domain workers in one graph plan are coordinated by that controller's reservations.

Native Windows SHM transport also uses a host-local directory for locks and acknowledgements. A producer retains each named mapping until its single consumer copies the payload and acknowledges its generation, or the producer cancels/closes it. A length header handles Windows page-rounded mapping sizes. Retained mappings are limited to 64 MiB and 256 buffers per producer process; exceeding either limit fails the write explicitly. This host transport fix does not enable cross-OS or accelerator zero-copy. POSIX keeps its existing unlink-based lifetime.

edge.local.session and edge.local.external.protocol retain compatibility imports. The old direct ExternalStage API remains available; new pipelines should use the runtime-owned graph backend. New state handles default to non-replayable and non-migratable. Legacy M0 defaults are explicitly preserved in contracts.legacy.

Hardware composition

Mobile and PC × model support matrix

All 60 named pairings were checked on 2026-09-22 through actual runs, raw-record review or artifact/runtime preflight. This is not 60 successful executions. Each cell links to its evidence, finding and next required step.

PASS: specified complete workload passed. PARTIAL: functional output with acceptance gaps. COMPONENT: model component only. PROFILE_ONLY: component timing without quality qualification. FAILED: attempted startup/execution failed. REJECTED: specific artifact refused. BLOCKED: prerequisite missing.

Device / execution configuration	Spark-X2.5	Qwen3-TTS 0.6B CustomVoice	Qwen3.8-27B	MiniCPM-o 4.5	InternVLA-A1
PC HX370: CPU / WSL	PASS	FAILED	BLOCKED	BLOCKED	BLOCKED
PC HX370: CPU / native Windows	BLOCKED	BLOCKED	BLOCKED	BLOCKED	BLOCKED
PC HX370 + RTX 5090 Laptop / WSL	PASS	PARTIAL	FAILED	BLOCKED	BLOCKED
PC HX370 + RTX 5090 Laptop / Windows	PASS	PARTIAL	FAILED	BLOCKED	BLOCKED
PC HX370 + Radeon 890M / Windows worker	BLOCKED	BLOCKED	COMPONENT	BLOCKED	BLOCKED
PC HX370 + AMD NPU / Windows worker	BLOCKED	BLOCKED	REJECTED	BLOCKED	BLOCKED
PC CPU+iGPU+NPU +/- discrete GPU, joint model execution	BLOCKED	BLOCKED	BLOCKED	BLOCKED	BLOCKED
PC Snapdragon X Elite CRD / AI Hub	BLOCKED	PROFILE_ONLY	BLOCKED	BLOCKED	BLOCKED
Mobile Galaxy S25 / Snapdragon 8 Elite for Galaxy	COMPONENT	COMPONENT	BLOCKED	COMPONENT	BLOCKED
Mobile Galaxy S24 / Snapdragon 8 Gen 3	BLOCKED	COMPONENT	BLOCKED	COMPONENT	BLOCKED
Embedded SA8775P ADP / AI Hub	BLOCKED	PROFILE_ONLY	BLOCKED	BLOCKED	BLOCKED
Embedded RB3 Gen 2 / QCS6490 / AI Hub	BLOCKED	PROFILE_ONLY	BLOCKED	BLOCKED	BLOCKED

Counts: 3 PASS, 2 PARTIAL, 6 COMPONENT, 3 PROFILE_ONLY, 3 FAILED, 1 REJECTED, 42 BLOCKED. The PC CPU Spark pass uses the existing vLLM 0.28 CPU wheel with the current Omni branch; CPU TTS fails a 3-versus-4-field sampling ABI mismatch in that environment. Current Windows/WSL Omni Qwen 27B startup fails pipeline resolution; earlier bare-vLLM text evidence is not a pass for this integration. Full MiniCPM-o/InternVLA checkpoints are absent from the inventoried model roots. Native Windows CPU attention kernels are absent from the installed extension.

S25 Spark, TTS predictor and MiniCPM-o speech-head components have fresh Hub inference evidence. S24 and the other Hub targets retain explicitly marked historical component/profile evidence. No direct ADB device is attached.

See the full audit and its hashed raw records. Other hardware SKUs remain outside this finite matrix; no universal device support is implied.

Detailed PC model and backend pairings

The tested PC is a Ryzen AI 9 HX370 laptop with Radeon 890M, AMD NPU and RTX 5090 Laptop GPU. Selecting CPU-only or a subset of those accelerators is an execution configuration on that PC, not evidence from a different PC SKU. Model support is specific to the following pairings:

Model / artifact	PC execution configuration	OS / backend	Qualification
Spark-X2.5-4B BF16	CPU + RTX 5090 Laptop GPU	WSL / vLLM CUDA	Current branch: M0 text/state/cancellation workload passed.
Spark-X2.5-4B BF16	CPU + RTX 5090 Laptop GPU	Native Windows / vLLM CUDA	Current branch: M0 text/state/cancellation workload passed.
Spark-X2.5-1.7B INT8	CPU execution; accelerators unused	WSL / vLLM CPU	Fresh 12-prompt acceptance passed on this branch with the existing vLLM 0.28 CPU wheel; historical token parity is separate.
Spark-X2.5-1.7B INT8	CPU execution; no NVIDIA dependency	Native Windows	Model path not yet qualified; CPU Add graph success is not text-model support.
Qwen3-TTS-12Hz-0.6B CustomVoice	CPU + RTX 5090 Laptop GPU; both stages on GPU	WSL / vLLM CUDA	Current branch: real-weight audio streaming completes; playback stalls and remaining M2 gates stay open.
Qwen3-TTS-12Hz-0.6B CustomVoice	CPU + RTX 5090 Laptop GPU; both stages on GPU	Native Windows / vLLM CUDA	Current branch: real-weight audio streaming completes; playback stalls and remaining M2 gates stay open.
Qwen3.8-27B vision tower, FP32 ONNX	CPU + Radeon 890M iGPU	Windows / ORT DirectML	Historical component validation only; not the complete 27B model or a newly qualified graph-stage vision-to-language pipeline.
Qwen3.8-27B vision tower, tested A16W8 exports	CPU + AMD NPU	Windows / ORT VitisAI	Tested whole-tower artifacts were rejected by NPU partitioning. This does not rule out other exports or smaller components.
Qwen3.8-27B complete multimodal pipeline	CPU/iGPU/NPU, with optional discrete GPU	Target-specific	Fresh public Omni startup fails pipeline resolution on Windows/WSL; earlier bare-vLLM short-text loading is separate and not multimodal validation.
MiniCPM-o	CPU/iGPU/NPU, with optional discrete GPU	Target-specific	Full model × PC combinations remain unqualified for this release.
InternVLA	CPU/iGPU/NPU, with optional discrete GPU	Target-specific	Full model × PC combinations remain unqualified for this release.

ORT Add and the small AMD NPU graph are backend probes, so they are deliberately not listed as supported application models. Likewise, a Galaxy S25 Spark attention component run in AI Hub is a separate model component × phone pairing, not a PC pairing or proof of complete mobile Spark execution. Other model × PC × backend combinations require their own artifact and workload qualification.

Hardware layouts represented by the abstraction

The abstraction supports these single-machine compositions. This table separates topology/admission support from successful execution of a particular model; it does not promise that every listed accelerator can run every model.

Composition	Resource accounting	Current execution evidence
CPU only	Host RAM; WSL quota when applicable	ORT CPU graphs through public Omni on Windows and WSL. Native Windows CPU text remains unqualified.
CPU + integrated GPU	Shared host RAM	Radeon 890M DirectML graph through Omni. This extraction's probe is not a complete vision/language pipeline.
CPU + integrated NPU	Shared host RAM	AMD NPU VitisAI graph through Omni; permitted CPU partitions are explicit in the placement report.
CPU + integrated GPU + integrated NPU	Shared RAM plus optional observed package/power/bandwidth relationships	Topology and admission contracts tested; individual laptop routes execute. Simultaneous full-model overlap is not qualified.
CPU + integrated GPU + integrated NPU + discrete GPU	Shared RAM and separate discrete VRAM	Current Ryzen AI 9 HX370 / Radeon 890M / AMD NPU / RTX 5090 Laptop machine. Spark CUDA text and native two-stage TTS run on Windows and WSL; using all accelerators together is not qualified.
CPU + discrete GPU, with integrated devices unused or absent	Host RAM plus discrete VRAM	Native CUDA text/TTS execution uses this subset. NVIDIA is optional in the abstraction.

Mobile/embedded SoCs fit the CPU + integrated GPU + NPU composition, but their local controller and model paths still require device-specific qualification. Galaxy S25 AI Hub inference/profiling passed for one Spark attention component; this is component evidence, not full local mobile generation or streaming. AI Hub is a test facility, never a deployment dependency. Intel, Apple and other vendor combinations are not certified by the AMD/NVIDIA/Qualcomm results.

DeviceDescriptor separates physical device identity, execution domain and integration. A CPU, integrated GPU and NPU can all reference machine:ram. An additional discrete GPU references its own machine:vram:<physical-id>. A WSL allocation can consume both machine:ram and wsl:quota constraints; those are two limits on the same allocation, not two amounts of physical RAM. Use the same physical GPU ID when Windows and WSL expose the same adapter.

Package IDs, bandwidth groups and power domains are optional observed relationships. Missing values mean unknown. Fitting memory does not authorize overlap on a shared package: selection and overlap still require model quality and complete workload measurements. Synthetic topology tests cover CPU-only and all four integrated / discrete combinations; they are not hardware certification.

Deploy a graph

Register a PipelineConfig with StageExecutionType.GRAPH, model_stage="graph" and an appropriate final output type. Set async_chunk=False. A graph following another stage must have a model-owned custom_process_input_func that returns {"tensors": {name: numpy_array}}. Names, dtypes and shapes must match the manifest's placement examples. Graph stages do not tokenize prompts or apply model-specific transformations.

Graph-containing v1 pipelines must be a linear chain with exactly one terminal output at the last stage. Fan-out, joins and multiple final outputs are refused before loading; those require reference-counted consumer leases.

Each stage in a graph-containing plan declares the same capacities and its own demands, including native stages:

pipeline: my_registered_pipeline
async_chunk: false
stages:
  - stage_id: 0
    backend:
      name: external.graph.v1
      manifest: /absolute/path/to/bundle/manifest.json
      route:
        name: local-cpu
        interpreter: /absolute/path/to/worker/python
        worker: /absolute/path/to/worker_ort.py
        ep: cpu
        os_domain: posix
      max_io_bytes: 8388608
      min_fraction_on_target: 1.0
      allowed_providers: [CPUExecutionProvider]
    resource_budget:
      capacities: {machine:ram: 2147483648, wsl:quota: 1073741824}
      demands: {machine:ram: 536870912, wsl:quota: 536870912}

Quote pool keys or use block mappings if your YAML producer treats colon-containing keys specially. On native Windows use Windows paths and os_domain: windows. From WSL use controller-visible paths; host services translate worker paths. Named legacy routes still honor VLLM_OMNI_EXTERNAL_PYTHON_<ROUTE>, but explicit routes have no developer-machine defaults.

Capacities are pre-load controller ceilings, below observed free host memory / guest limits with safety margin. They are not continuously sampled available RAM. Demands must include loading peaks, weights, state, activations, workspace, transport copies, retained output and headroom. The ledger does not claim to be an OS memory limiter or to attribute shared vendor allocations from RSS. Driver/cache memory outside the worker remains part of the deployment estimate.

The graph worker environment needs NumPy, ONNX Runtime and ONNX (for checking external weight references against the manifest). Vendor providers need their own qualified environment. The controller need not install their runtimes.

Artifact and placement gate

{
  "schema_version": 1,
  "component": "model-owned-component-name",
  "files": {"graph.onnx": "<sha256>", "weights.bin": "<sha256>", "inputs.npz": "<sha256>"},
  "metadata": {
    "graph_file": "graph.onnx", "example_inputs_file": "inputs.npz",
    "checkpoint_revision": "<revision>", "precision": "<precision>",
    "layout": "<layout>", "exporter": "<version>",
    "calibration": "<record or not applicable>", "validation": "<record>"
  }
}

All payload paths are relative to the manifest. Hashes are checked before startup; ONNX external weight references must also belong to that verified set. Keep bundles immutable during execution. Checkpoint, exporter, precision and validation metadata are model-owned: supplying a manifest does not establish model quality.

Warmup records actual provider node assignments. Unverified placement, absent providers and undeclared CPU partitions refuse startup. The default target fraction is 1.0. A deliberately partitioned NPU graph can lower it and explicitly allow CPU, but reports always retain the real split. execution_plan records the manifest identity, worker generation, route, reservations and placement. It labels this B (backend execution), not complete-model P.

Flow, cancellation and ownership

• Graph pipelines admit one request before the engine ingress queue. Extra work receives ResourceUnavailable; await consumption or cancellation and retry. The current API does not queue a batch of graph requests.
• Each graph client has one in-flight call and one retained result. Delivery does not free the slot. Public synchronous/asynchronous generators acknowledge when iteration advances or closes. Direct engine consumers call OmniRequestOutput.release_stage_buffers() in finally.
• After acknowledgement, retained result arrays belong to the application. Keeping arbitrarily many consumed results is application memory, outside engine admission.
• Intermediate graph handoffs copy their next-stage inputs before acknowledgement. Buffer descriptors carry owner, generation, exact byte size, dtype and shape. Framing rejects unknown required features and malformed lengths before payload allocation.
• Cancellation fences the request epoch immediately. An active non-preemptible graph call retires its worker. It does not replay work or restart transparently; recreate the pipeline to use that backend again. Completed calls can cancel without killing an otherwise reusable worker.
• Windows workers announce PID plus process creation time. Cross-OS termination verifies that identity and retires the native process tree before stopping the WSL launcher. An absent/unverified tree keeps its reservation quarantined. Python call references stay charged until the call unwinds.

Native APIs and their admission logic remain compatible. This change adds a shared explicit reservation transaction for graph-containing plans; it does not yet replace every native memory observation/ledger or qualify mixed-model fan-out and streaming.

Packaging, validation and scope

Build packages/omni-stage-contracts with uv build. Its wheel imports without torch/vLLM; the optional wire extra installs NumPy. The main Omni wheel bundles the same source. Python/native implementers can use conformance/v1.json from the source distribution. A native mobile implementation has not yet run these fixtures.

examples/edge/run_graph_stage.py drives the public API, writes a deployment profile, records the resolved plan and raw request samples, and saves outputs for numerical comparison. It supports explicit CPU, DirectML and VitisAI worker routes.

The implementation has tests for real runtime startup, two-stage handoff on both orchestrator modes, public APIs, ingress/consumer backpressure, malformed framing, placement refusal, shared-pool overcommit, stale acknowledgements, cancellation, worker failure and artifact membership. Native Spark acceptance and small actual Windows/WSL graph probes are recorded separately in the workspace experiment report.

Validation snapshot (2026-09-22, base 00510ce38bbb96ef2e4ce462618d06ab142fae7c):

• WSL focused regression: 415 passed, 1 skipped, 20 deselected. Windows focused regression: 78 passed, 2 skipped; the controller lacks ONNX for those two tests, while actual graph probes use their separate worker environments.
• Spark 4B BF16 CUDA M0: 11 tests passed on each host. These runs check the specified text/state/cancellation workload, not every model or device.
• CPU and DirectML Add graph outputs are exact. AMD NPU versus the same A16W8 CPU graph has maximum absolute error 4.458427429199219e-05, normalized maximum 3.111236521857185e-05 against a 0.002 bound. This is backend validation, not original-checkpoint model-quality validation.
• Real Qwen3-TTS 0.6B two-stage streaming completes on both hosts with checkpoint revision 85e237c12c027371202489a0ec509ded67b5e4b5. One measured utterance per host still has playback stalls; these diagnostic samples do not satisfy the full streaming, interruption, quality or sustained-thermal acceptance gates.
• The built Omni wheel runs 20 public-API CPU graph requests outside the checkout with exact outputs and zero retained reservation after shutdown. The standalone contracts source archive imports in a dependency-free environment.
• Fresh Galaxy S25 inference and NPU profiling passed for the existing W8A16 attention component. Three outputs match the same artifact/input reference exactly; the full1024 artifact label denotes one attention component's cache bucket, not the complete Spark model.

Raw records, machine/package inventories, failed attempts and reproduction commands are preserved in the surrounding edge-infer workspace under analysis/experiments/omni_abstraction_20260922/. They are not bundled as runtime dependencies. Tests of reduced-device configurations are not measurements on a different physical hardware SKU.

Still outside the completed v1 scope: arbitrary device/model certification, mobile C++ control integration, persistent external AR sessions, shared-device zero-copy, state migration/replay, mixed-model streaming/fan-out qualification, joint thermal / bandwidth policy, native Windows CPU text qualification, and the full M1–M4 device matrix. These remain explicit rollout gates, not implied supported features.