vllm-omni/docs/analysis

Six-engine analysis: extending vLLM-Omni to edge TTS and VLA

Source-grounded study of six inference stacks and a practical plan for running vLLM-Omni-style TTS and vision-language-action (VLA) pipelines on edge devices. Produced 2026-09-08 from the local checkouts listed in repository_inventory.md; every engine report cites files and symbols at those exact commits, and every claim carries an evidence label:

• [Verified] read directly in source or repo docs at the analyzed commit
• [Inferred] reasoned from structure, not directly confirmed
• [Proposal] a design suggestion made here
• [Unknown] could not be determined from the tree
• repo-reported a number published by the repository, not reproduced here
• measured a number produced in this session on this host (see experiments.md)

Reading order

1. repository_inventory.md — paths, branches, commits, working-tree state, discrepancies.
2. Engine reports (one per engine, same 13-section structure: device, kernels, model runner, scheduler, I/O, flows, tests, TTS, VLA, edge profile, limitations, references):
   • engines/vllm.md — the serving engine vLLM-Omni builds on
   • engines/vllm_omni.md — multi-stage omni pipeline; what it inherits from and adds to vLLM
   • engines/vllm_omni_models.md — model-level deep dive of the TTS and VLA/world-model families in vLLM-Omni
   • engines/llama_cpp.md
   • engines/ncnn.md
   • engines/mnn.md
   • engines/executorch.md
3. comparison.md — all six across device, kernels, model runner, scheduler, I/O, deployment complexity, conversion, extensibility; serving stacks vs execution runtimes.
4. vllm_omni_edge_feasibility.md — targets (embedded NVIDIA, ARM CPU, Apple Silicon, mobile GPU/NPU), five strategies (direct port, backend substitution, lightweight orchestration, hybrid edge/cloud, edge-native baseline), per-runtime stage placement.
5. tts_edge_design.md and vla_edge_design.md — separate designs with integration points, adapter interfaces, state/buffer ownership, blockers.
6. implementation_roadmap.md — phases, acceptance criteria, reproducible benchmarks.
7. experiments.md — what was actually run (GPU via the shared scheduler, CPU-only llama.cpp), commands, job records, results, cleanup, and what was deferred.
8. edge_branch_plan.md — work packages for an edge/omni-edge branch inside vLLM-Omni itself (deploy profiles, cold start, CPU platform, Jetson, codec-token streaming, slimming, device-direct connector).
9. edge_engine_comparison.md — the branch's vLLM-Omni measured against llama.cpp (native Qwen3-TTS, CPU and CUDA) on the same host, prompts and reference speaker; why MNN/ExecuTorch numbers are still missing.
10. edge_device_simulation.md — what edge devices can be emulated on this H200-class host (MPS SM caps, memory budgets, core pinning) and what cannot (bandwidth, clocks); measured SM-cap sensitivity for both engines, a Jetson projection, and the per-frame attribution that explains the CPU numbers (15 sequential code-predictor passes at ≈6 ms each).
11. mobile_htp_simulation.md — 2026-09-10: Qualcomm AI Runtime SDK (Hexagon HTP simulator) installed without root, the branch's Code2Wav decoder converted (fp16 DLC) and run on the x86 HTP functional simulator with bit-exact CPU-backend parity; the CPU findings it triggered (KV-cached predictor, model-side sampler, fp32 compiled predictor; int8 rejected by in-situ parity) that take the CPU platform from RTF 1.7–1.8 to 0.89–1.27 on 16–32 threads (real time at 32), i.e. 1.6–2.2× faster end-to-end than llama.cpp on the same host, level at 8 threads; the code predictor also runs on the HTP simulator (fp16, 50.7 dB vs fp32); real Galaxy S24/S25 timings via Qualcomm AI Hub (§5): fp16 is faithful on device, and the right placement is heterogeneous — talker+predictor on the Hexagon NPU (49.7 ms/frame, fp16) and Code2Wav on the Adreno GPU (45.8 ms/frame at fp32, numerically exact, vs 182 ms on the NPU) — pipelining to RTF 0.62, i.e. real time on a phone with no quantization and no precision loss; every automatic int8 build is faster but numerically broken.
12. Implementation (2026-09-09): edge_branch_plan.md §5 records what was built on the branch and the measured outcome per work package; experiments.md §8 holds the runs; experiments/edge_branch/results_summary.md the tables.
13. minicpmo_edge_profile.md — 2026-09-11: MiniCPM-o 4.5 (3-stage omni) profiled with the same instruments: H200 batch RTF 0.748, CPU TTS head 11.6 ms/token and vocoder 833 ms/s of audio (91 % of it in the 10-step flow DiT), Galaxy S25 NPU 7.08 ms/token at 48.9 dB — 177 ms of NPU time per second of speech against Qwen3-TTS's 621 ms, because MiniCPM uses one codebook at 25 Hz instead of sixteen at 12.5 Hz; int8 breaks its code head too.
14. spark_edge_support.md — 2026-09-11: Spark-X2.5-1.7B (agentic on-device model, hybrid 3:1 sliding/full attention, head-wise attention gate, per-layer-type partial RoPE) implemented for vLLM-Omni — vLLM 0.28 had no support, and MNN/ExecuTorch/ncnn still have none. Verified exact against the official code on transformers 4.57.1 (fp32, 32/32 greedy, 2.2e-05 logprob). On CPU it beats llama.cpp at matched bf16 by 8.7–15.4× on prefill and 1.19–2.65× on decode; an int8 path adds 1.6× but costs long-context fidelity; llama.cpp Q4_K_M still wins single-stream decode (no dense int4 kernel exists in vLLM's CPU build) while vLLM int8 wins from batch 16 and leads 1.72× at batch 64. GPU vs SGLang is a tie within 4 %. On a real Galaxy S25 a full decode step (output head included, every component verified at 36.9–62.8 dB) is 27.4 ms at 1 k context in calibrated w8a16 — 36.5 tok/s, 11.8% faster than the first working version and the first build whose output is correct at all: reading the per-op device profile showed a fifth of every sliding layer went on handing a recomputed KV window back across the graph boundary, and an on-device parity check showed the uncalibrated w4a16 builds everyone quotes score −1.6 dB, i.e. uncorrelated with the reference. Calibrated w8a16 is the deployable scheme (int8 activations still break it, the third model family here to show that). The hybrid attention beats an all-full-attention model by 1.51× at 4 k but 0.94× at 1 k, where the sliding layers' full-width RoPE outweighs the shorter cache. QNN/TFLite/ORT land within 2 % of each other on the NPU (the export layout sets on-device speed, not the runtime) and the NPU beats the Adreno GPU 2.4x here, the opposite of the TTS vocoder. Android control works with vLLM's stock glm45 parsers; thinking costs 14.6× latency for ~2/10 decisions.

Headline findings

• All six checkouts match the requested branches and commits and are clean; ncnn and ExecuTorch submodules are uninitialized, and the machine's installed vLLM-Omni is a different checkout than the analyzed one (repository_inventory.md).
• vLLM-Omni is a serving stack, not a runtime. It subclasses vLLM's engine process, scheduler, and model runners, patches about ten vLLM symbols at import, runs every AR/generation stage as a vLLM engine process (diffusion stages run as a separate worker or inline via inline_diffusion), and joins stages by connectors whose default, SharedMemoryConnector, stages tensors through host memory (the Mooncake/Mori transfer-engine connectors have tensor fast paths and device pools). It has no CPU platform and no Apple/Android/Jetson code. What is portable is its stage semantics (1-frame first chunk, 25-frame steady chunks, 72-frame decoder context for Qwen3-TTS), its abort rules, its API shapes, and the plain-PyTorch codecs and action heads (engines/vllm_omni.md, engines/vllm_omni_models.md).
• Measured on this host: Qwen3-TTS 1.7B on one L20X gives a median first-audio latency of 27.7 ms, 160 ms per 2 s chunk, RTF 0.08–0.10, but 157–188 s of observed engine initialization in the default configuration (CUDA graphs + torch.compile; a lower bound with enforce_eager or parallel stage init was not measured); llama.cpp on x86 CPU decodes a 409 M-parameter Q4_0 LLM at 135.6 tok/s on 4 threads and spends 0.6–1.2 s per 512 px SigLIP tile on 8 threads (experiments.md).
• Qwen3-TTS already runs natively in llama.cpp and MNN, and ExecuTorch has a streaming TTS runner (Voxtral) to copy from; so edge TTS is mostly an orchestration and parity problem rather than a model-porting problem, subject to a checkpoint/task parity gate (the server benchmark used CustomVoice 1.7B; llama.cpp's README demonstrates 1.7B Base and MNN's demo requires reference-audio cloning) (comparison.md §10).
• No runtime has a VLA policy in tree. ExecuTorch (static memory, zero-copy I/O, in-graph loops, CoreML/QNN delegates) and MNN (DiT runtime, vision exporters, Android NPUs) are the viable hosts; llama.cpp lacks an action head; ncnn is a component runtime only (vla_edge_design.md).
• Recommended strategy: keep vLLM-Omni as the reference/edge-server tier (Jetson-class only), ship a hybrid TTS first (server talker, on-device decoder over a codec-token stream of about 400 B/s raw payload) if the new wire contract and the on-device decoder measurement pass, otherwise start with a C++ orchestrator; in either case build a C++ orchestrator on llama.cpp/MNN/ExecuTorch that mirrors vLLM-Omni's pipeline semantics; VLA goes through ExecuTorch/MNN with simulation-first validation and a control/safety layer independent of inference (vllm_omni_edge_feasibility.md, implementation_roadmap.md).
• Built and measured on 2026-09-09 (branch edge/omni-edge): an edge deploy profile with a ramped chunk schedule removes the ≈80 ms first-chunk stall (playback-start = TTFA = 41–44 ms, RTF unchanged); concurrent stage init cuts warm start from 152 s to 95 s (cold caches 355 s); a CPU platform runs Qwen3-TTS 0.6B end to end (RTF 1.4–2.8 on 8–32 Xeon threads, the talker step alone is 1.7–3× the frame budget); hardware auto-adaptation, codec-token streaming with an on-device decoder, import slimming, a CUDA-IPC connector and per-step overhead attribution (scheduler + transport < 2 % of the frame; no scheduler rewrite or native engine rewrite warranted) are in the tree (edge_branch_plan.md §5).
• Engine comparison (2026-09-09): on one L20X the branch generates a codec frame in 6 ms vs 17–18 ms for llama.cpp's native Qwen3-TTS (RTF 0.10–0.20 vs 0.28–0.31) and streams first audio in 41–54 ms without reference audio, but reference-audio (x-vector) requests cost it 0.4–0.6 s per request and it needs ≈90–150 s of startup and ≈37 GB of reserved GPU memory against llama.cpp's 3–6 s and a few GB; on CPU both engines land at RTF ≈2 for the 0.6B model, llama.cpp with half the RAM (edge_engine_comparison.md).
• Edge simulation and CPU analysis (2026-09-09): the cards are H200-class (804 TFLOPS bf16 measured; the driver labels them L20X). Capping SMs to 6 % with MPS costs vLLM-Omni's talker only +55 % per frame (latency-bound, CUDA graphs) while llama.cpp loses 2.8×; forcing eager mode for an 8 GB budget hurts far more (TTFA 160 ms, RTF 0.5). On CPU, 73 % of every frame is the 15 sequential code-predictor passes (6.4 ms each vs a 1.6 ms bandwidth floor) in vLLM-Omni and ≈85 % in llama.cpp; the fix is fusion/quantization/cheaper sampling for ≈2× and a lighter predictor for real time (edge_device_simulation.md).
• Blockers: no edge hardware reachable from this host; VLA weights not cached (download needs permission); ExecuTorch/MNN toolchains not installed (permission); vLLM-Omni's process model and host-staged default transport are architectural for the direct-port path (cold-start minimum unmeasured).

Conventions

• Links are relative; source links go to ../<repo>/path and were checked for existence by a link-check pass (see the end of experiments.md).
• Diagrams are Mermaid.
• Mobile/ARM/Apple numbers are never measured here (x86 host); they are either repo-reported with a citation or marked Unknown.

Second-model review

On 2026-09-08 the finished study was independently reviewed by a second model (gpt-6-astra via the Codex hook, read-only against the same checkouts; its sandbox could not start because user namespaces are disabled on this host, so it ran unsandboxed under strict read-only instructions and the checkouts were verified clean afterwards). Initial verdict: agree with corrections (9 disagreements, 4 omissions). All were verified against source and applied; the main changes were: the default shared-memory transport is host-staged but the Mooncake/Mori transfer-engine connectors have device fast paths (Mori including intra-node XGMI), unvalidated for edge; only AR/generation stages are vLLM engine processes; the 1→25 chunk schedule leaves an ≈80 ms playback gap and the design now mirrors the YAML ramp/adaptive schedule; the CPU LLM benchmark is backbone-only (code predictor and decoder excluded); the 157–188 s figure is observed initialization, not a lower bound; prompt templates are engine code (Qwen3TTSPromptEmbedsBuilder); MNN's Qwen3-TTS path is single-callback; π0 uses SigLIP-So400m and draws noise inside sample_actions; simulation clients exist for DreamZero; hybrid-first is conditional on a codec-token wire-contract prototype and an on-device decoder measurement; a checkpoint/task parity gate was added. Final verdict after three rounds: AGREE ("agreement covers the source-grounded conclusions and conditional implementation plan; edge performance remains subject to the documented validation gates").