tzhouamandClaude Opus 5
9264ff81ab
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
58 lines
14 KiB
Markdown
58 lines
14 KiB
Markdown
# Six-engine analysis: extending vLLM-Omni to edge TTS and VLA
|
||
|
||
Source-grounded study of six inference stacks and a practical plan for running vLLM-Omni-style TTS and vision-language-action (VLA) pipelines on edge devices. Produced 2026-09-08 from the local checkouts listed in [repository_inventory.md](repository_inventory.md); every engine report cites files and symbols at those exact commits, and every claim carries an evidence label:
|
||
|
||
- **[Verified]** read directly in source or repo docs at the analyzed commit
|
||
- **[Inferred]** reasoned from structure, not directly confirmed
|
||
- **[Proposal]** a design suggestion made here
|
||
- **[Unknown]** could not be determined from the tree
|
||
- **repo-reported** a number published by the repository, not reproduced here
|
||
- **measured** a number produced in this session on this host (see [experiments.md](experiments.md))
|
||
|
||
## Reading order
|
||
|
||
1. [repository_inventory.md](repository_inventory.md) — paths, branches, commits, working-tree state, discrepancies.
|
||
2. Engine reports (one per engine, same 13-section structure: device, kernels, model runner, scheduler, I/O, flows, tests, TTS, VLA, edge profile, limitations, references):
|
||
- [engines/vllm.md](engines/vllm.md) — the serving engine vLLM-Omni builds on
|
||
- [engines/vllm_omni.md](engines/vllm_omni.md) — multi-stage omni pipeline; what it inherits from and adds to vLLM
|
||
- [engines/vllm_omni_models.md](engines/vllm_omni_models.md) — model-level deep dive of the TTS and VLA/world-model families in vLLM-Omni
|
||
- [engines/llama_cpp.md](engines/llama_cpp.md)
|
||
- [engines/ncnn.md](engines/ncnn.md)
|
||
- [engines/mnn.md](engines/mnn.md)
|
||
- [engines/executorch.md](engines/executorch.md)
|
||
3. [comparison.md](comparison.md) — all six across device, kernels, model runner, scheduler, I/O, deployment complexity, conversion, extensibility; serving stacks vs execution runtimes.
|
||
4. [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) — targets (embedded NVIDIA, ARM CPU, Apple Silicon, mobile GPU/NPU), five strategies (direct port, backend substitution, lightweight orchestration, hybrid edge/cloud, edge-native baseline), per-runtime stage placement.
|
||
5. [tts_edge_design.md](tts_edge_design.md) and [vla_edge_design.md](vla_edge_design.md) — separate designs with integration points, adapter interfaces, state/buffer ownership, blockers.
|
||
6. [implementation_roadmap.md](implementation_roadmap.md) — phases, acceptance criteria, reproducible benchmarks.
|
||
7. [experiments.md](experiments.md) — what was actually run (GPU via the shared scheduler, CPU-only llama.cpp), commands, job records, results, cleanup, and what was deferred.
|
||
8. [edge_branch_plan.md](edge_branch_plan.md) — work packages for an `edge/omni-edge` branch inside vLLM-Omni itself (deploy profiles, cold start, CPU platform, Jetson, codec-token streaming, slimming, device-direct connector).
|
||
9. [edge_engine_comparison.md](edge_engine_comparison.md) — the branch's vLLM-Omni measured against llama.cpp (native Qwen3-TTS, CPU and CUDA) on the same host, prompts and reference speaker; why MNN/ExecuTorch numbers are still missing.
|
||
10. [edge_device_simulation.md](edge_device_simulation.md) — what edge devices can be emulated on this H200-class host (MPS SM caps, memory budgets, core pinning) and what cannot (bandwidth, clocks); measured SM-cap sensitivity for both engines, a Jetson projection, and the per-frame attribution that explains the CPU numbers (15 sequential code-predictor passes at ≈6 ms each).
|
||
11. [mobile_htp_simulation.md](mobile_htp_simulation.md) — 2026-09-10: Qualcomm AI Runtime SDK (Hexagon HTP simulator) installed without root, the branch's Code2Wav decoder converted (fp16 DLC) and run on the x86 HTP functional simulator with bit-exact CPU-backend parity; the CPU findings it triggered (KV-cached predictor, model-side sampler, fp32 compiled predictor; int8 rejected by in-situ parity) that take the CPU platform from RTF 1.7–1.8 to 0.89–1.27 on 16–32 threads (real time at 32), i.e. 1.6–2.2× faster end-to-end than llama.cpp on the same host, level at 8 threads; the code predictor also runs on the HTP simulator (fp16, 50.7 dB vs fp32); real Galaxy S24/S25 timings via Qualcomm AI Hub (§5): fp16 is faithful on device, and the right placement is heterogeneous — talker+predictor on the Hexagon NPU (49.7 ms/frame, fp16) and Code2Wav on the Adreno GPU (45.8 ms/frame at fp32, numerically exact, vs 182 ms on the NPU) — pipelining to RTF 0.62, i.e. real time on a phone with no quantization and no precision loss; every automatic int8 build is faster but numerically broken.
|
||
12. **Implementation (2026-09-09):** [edge_branch_plan.md §5](edge_branch_plan.md#5-implementation-status-branch-edgeomni-edge-2026-09-09) records what was built on the branch and the measured outcome per work package; [experiments.md §8](experiments.md#8-edge-branch-experiments-2026-09-09) holds the runs; [experiments/edge_branch/results_summary.md](experiments/edge_branch/results_summary.md) the tables.
|
||
12. [minicpmo_edge_profile.md](minicpmo_edge_profile.md) — 2026-09-11: MiniCPM-o 4.5 (3-stage omni) profiled with the same instruments: H200 batch RTF 0.748, CPU TTS head 11.6 ms/token and vocoder 833 ms/s of audio (91 % of it in the 10-step flow DiT), Galaxy S25 NPU 7.08 ms/token at 48.9 dB — 177 ms of NPU time per second of speech against Qwen3-TTS's 621 ms, because MiniCPM uses one codebook at 25 Hz instead of sixteen at 12.5 Hz; int8 breaks its code head too.
|
||
13. [spark_edge_support.md](spark_edge_support.md) — 2026-09-11: Spark-X2.5-1.7B (agentic on-device model, hybrid 3:1 sliding/full attention, head-wise attention gate, per-layer-type partial RoPE) implemented for vLLM-Omni — vLLM 0.28 had no support, and MNN/ExecuTorch/ncnn still have none. Verified exact against the official code on transformers 4.57.1 (fp32, 32/32 greedy, 2.2e-05 logprob). On CPU it beats llama.cpp at matched bf16 by 8.7–15.4× on prefill and 1.19–2.65× on decode; an int8 path adds 1.6× but costs long-context fidelity; llama.cpp Q4_K_M still wins single-stream decode (no dense int4 kernel exists in vLLM's CPU build) while vLLM int8 wins from batch 16 and leads 1.72× at batch 64. GPU vs SGLang is a tie within 4 %. On a real Galaxy S25 a full decode step (output head included, every component verified at 36.9–62.8 dB) is 27.4 ms at 1 k context in calibrated w8a16 — 36.5 tok/s, 11.8% faster than the first working version and the first build whose output is correct at all: reading the per-op device profile showed a fifth of every sliding layer went on handing a recomputed KV window back across the graph boundary, and an on-device parity check showed the uncalibrated w4a16 builds everyone quotes score −1.6 dB, i.e. uncorrelated with the reference. Calibrated w8a16 is the deployable scheme (int8 activations still break it, the third model family here to show that). The hybrid attention beats an all-full-attention model by 1.51× at 4 k but 0.94× at 1 k, where the sliding layers' full-width RoPE outweighs the shorter cache. QNN/TFLite/ORT land within 2 % of each other on the NPU (the export layout sets on-device speed, not the runtime) and the NPU beats the Adreno GPU 2.4x here, the opposite of the TTS vocoder. Android control works with vLLM's stock glm45 parsers; thinking costs 14.6× latency for ~2/10 decisions.
|
||
|
||
## Headline findings
|
||
|
||
- **All six checkouts match the requested branches and commits and are clean**; ncnn and ExecuTorch submodules are uninitialized, and the machine's installed vLLM-Omni is a different checkout than the analyzed one ([repository_inventory.md](repository_inventory.md)).
|
||
- **vLLM-Omni is a serving stack, not a runtime.** It subclasses vLLM's engine process, scheduler, and model runners, patches about ten vLLM symbols at import, runs every AR/generation stage as a vLLM engine process (diffusion stages run as a separate worker or inline via `inline_diffusion`), and joins stages by connectors whose default, `SharedMemoryConnector`, stages tensors through host memory (the Mooncake/Mori transfer-engine connectors have tensor fast paths and device pools). It has no CPU platform and no Apple/Android/Jetson code. What is portable is its stage semantics (1-frame first chunk, 25-frame steady chunks, 72-frame decoder context for Qwen3-TTS), its abort rules, its API shapes, and the plain-PyTorch codecs and action heads ([engines/vllm_omni.md](engines/vllm_omni.md), [engines/vllm_omni_models.md](engines/vllm_omni_models.md)).
|
||
- **Measured on this host**: Qwen3-TTS 1.7B on one L20X gives a median first-audio latency of 27.7 ms, 160 ms per 2 s chunk, RTF 0.08–0.10, but 157–188 s of observed engine initialization in the default configuration (CUDA graphs + `torch.compile`; a lower bound with `enforce_eager` or parallel stage init was not measured); llama.cpp on x86 CPU decodes a 409 M-parameter Q4_0 LLM at 135.6 tok/s on 4 threads and spends 0.6–1.2 s per 512 px SigLIP tile on 8 threads ([experiments.md](experiments.md)).
|
||
- **Qwen3-TTS already runs natively in llama.cpp and MNN**, and ExecuTorch has a streaming TTS runner (Voxtral) to copy from; so edge TTS is mostly an orchestration and parity problem rather than a model-porting problem, subject to a checkpoint/task parity gate (the server benchmark used CustomVoice 1.7B; llama.cpp's README demonstrates 1.7B Base and MNN's demo requires reference-audio cloning) ([comparison.md](comparison.md) §10).
|
||
- **No runtime has a VLA policy in tree.** ExecuTorch (static memory, zero-copy I/O, in-graph loops, CoreML/QNN delegates) and MNN (DiT runtime, vision exporters, Android NPUs) are the viable hosts; llama.cpp lacks an action head; ncnn is a component runtime only ([vla_edge_design.md](vla_edge_design.md)).
|
||
- **Recommended strategy**: keep vLLM-Omni as the reference/edge-server tier (Jetson-class only), ship a hybrid TTS first (server talker, on-device decoder over a codec-token stream of about 400 B/s raw payload) if the new wire contract and the on-device decoder measurement pass, otherwise start with a C++ orchestrator; in either case build a C++ orchestrator on llama.cpp/MNN/ExecuTorch that mirrors vLLM-Omni's pipeline semantics; VLA goes through ExecuTorch/MNN with simulation-first validation and a control/safety layer independent of inference ([vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md), [implementation_roadmap.md](implementation_roadmap.md)).
|
||
- **Built and measured on 2026-09-09 (branch `edge/omni-edge`)**: an edge deploy profile with a ramped chunk schedule removes the ≈80 ms first-chunk stall (playback-start = TTFA = 41–44 ms, RTF unchanged); concurrent stage init cuts warm start from 152 s to 95 s (cold caches 355 s); a CPU platform runs Qwen3-TTS 0.6B end to end (RTF 1.4–2.8 on 8–32 Xeon threads, the talker step alone is 1.7–3× the frame budget); hardware auto-adaptation, codec-token streaming with an on-device decoder, import slimming, a CUDA-IPC connector and per-step overhead attribution (scheduler + transport < 2 % of the frame; no scheduler rewrite or native engine rewrite warranted) are in the tree ([edge_branch_plan.md §5](edge_branch_plan.md#5-implementation-status-branch-edgeomni-edge-2026-09-09)).
|
||
- **Engine comparison (2026-09-09)**: on one L20X the branch generates a codec frame in 6 ms vs 17–18 ms for llama.cpp's native Qwen3-TTS (RTF 0.10–0.20 vs 0.28–0.31) and streams first audio in 41–54 ms without reference audio, but reference-audio (x-vector) requests cost it 0.4–0.6 s per request and it needs ≈90–150 s of startup and ≈37 GB of reserved GPU memory against llama.cpp's 3–6 s and a few GB; on CPU both engines land at RTF ≈2 for the 0.6B model, llama.cpp with half the RAM ([edge_engine_comparison.md](edge_engine_comparison.md)).
|
||
- **Edge simulation and CPU analysis (2026-09-09)**: the cards are H200-class (804 TFLOPS bf16 measured; the driver labels them L20X). Capping SMs to 6 % with MPS costs vLLM-Omni's talker only +55 % per frame (latency-bound, CUDA graphs) while llama.cpp loses 2.8×; forcing eager mode for an 8 GB budget hurts far more (TTFA 160 ms, RTF 0.5). On CPU, 73 % of every frame is the 15 sequential code-predictor passes (6.4 ms each vs a 1.6 ms bandwidth floor) in vLLM-Omni and ≈85 % in llama.cpp; the fix is fusion/quantization/cheaper sampling for ≈2× and a lighter predictor for real time ([edge_device_simulation.md](edge_device_simulation.md)).
|
||
- **Blockers**: no edge hardware reachable from this host; VLA weights not cached (download needs permission); ExecuTorch/MNN toolchains not installed (permission); vLLM-Omni's process model and host-staged default transport are architectural for the direct-port path (cold-start minimum unmeasured).
|
||
|
||
## Conventions
|
||
|
||
- Links are relative; source links go to `../<repo>/path` and were checked for existence by a link-check pass (see the end of [experiments.md](experiments.md)).
|
||
- Diagrams are Mermaid.
|
||
- Mobile/ARM/Apple numbers are never measured here (x86 host); they are either repo-reported with a citation or marked Unknown.
|
||
|
||
## Second-model review
|
||
|
||
On 2026-09-08 the finished study was independently reviewed by a second model (`gpt-6-astra` via the Codex hook, read-only against the same checkouts; its sandbox could not start because user namespaces are disabled on this host, so it ran unsandboxed under strict read-only instructions and the checkouts were verified clean afterwards). Initial verdict: agree with corrections (9 disagreements, 4 omissions). All were verified against source and applied; the main changes were: the default shared-memory transport is host-staged but the Mooncake/Mori transfer-engine connectors have device fast paths (Mori including intra-node XGMI), unvalidated for edge; only AR/generation stages are vLLM engine processes; the 1→25 chunk schedule leaves an ≈80 ms playback gap and the design now mirrors the YAML ramp/adaptive schedule; the CPU LLM benchmark is backbone-only (code predictor and decoder excluded); the 157–188 s figure is observed initialization, not a lower bound; prompt templates are engine code (`Qwen3TTSPromptEmbedsBuilder`); MNN's Qwen3-TTS path is single-callback; π0 uses SigLIP-So400m and draws noise inside `sample_actions`; simulation clients exist for DreamZero; hybrid-first is conditional on a codec-token wire-contract prototype and an on-device decoder measurement; a checkpoint/task parity gate was added. Final verdict after three rounds: **AGREE** ("agreement covers the source-grounded conclusions and conditional implementation plan; edge performance remains subject to the documented validation gates").
|