Include the baselines, comparisons and study documents, not just the fork

The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ

docs/analysis/README.md

@@ -0,0 +1,57 @@
+# Six-engine analysis: extending vLLM-Omni to edge TTS and VLA
+
+Source-grounded study of six inference stacks and a practical plan for running vLLM-Omni-style TTS and vision-language-action (VLA) pipelines on edge devices. Produced 2026-09-08 from the local checkouts listed in [repository_inventory.md](repository_inventory.md); every engine report cites files and symbols at those exact commits, and every claim carries an evidence label:
+
+- **[Verified]** read directly in source or repo docs at the analyzed commit
+- **[Inferred]** reasoned from structure, not directly confirmed
+- **[Proposal]** a design suggestion made here
+- **[Unknown]** could not be determined from the tree
+- **repo-reported** a number published by the repository, not reproduced here
+- **measured** a number produced in this session on this host (see [experiments.md](experiments.md))
+
+## Reading order
+
+1. [repository_inventory.md](repository_inventory.md) — paths, branches, commits, working-tree state, discrepancies.
+2. Engine reports (one per engine, same 13-section structure: device, kernels, model runner, scheduler, I/O, flows, tests, TTS, VLA, edge profile, limitations, references):
+   - [engines/vllm.md](engines/vllm.md) — the serving engine vLLM-Omni builds on
+   - [engines/vllm_omni.md](engines/vllm_omni.md) — multi-stage omni pipeline; what it inherits from and adds to vLLM
+   - [engines/vllm_omni_models.md](engines/vllm_omni_models.md) — model-level deep dive of the TTS and VLA/world-model families in vLLM-Omni
+   - [engines/llama_cpp.md](engines/llama_cpp.md)
+   - [engines/ncnn.md](engines/ncnn.md)
+   - [engines/mnn.md](engines/mnn.md)
+   - [engines/executorch.md](engines/executorch.md)
+3. [comparison.md](comparison.md) — all six across device, kernels, model runner, scheduler, I/O, deployment complexity, conversion, extensibility; serving stacks vs execution runtimes.
+4. [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) — targets (embedded NVIDIA, ARM CPU, Apple Silicon, mobile GPU/NPU), five strategies (direct port, backend substitution, lightweight orchestration, hybrid edge/cloud, edge-native baseline), per-runtime stage placement.
+5. [tts_edge_design.md](tts_edge_design.md) and [vla_edge_design.md](vla_edge_design.md) — separate designs with integration points, adapter interfaces, state/buffer ownership, blockers.
+6. [implementation_roadmap.md](implementation_roadmap.md) — phases, acceptance criteria, reproducible benchmarks.
+7. [experiments.md](experiments.md) — what was actually run (GPU via the shared scheduler, CPU-only llama.cpp), commands, job records, results, cleanup, and what was deferred.
+8. [edge_branch_plan.md](edge_branch_plan.md) — work packages for an `edge/omni-edge` branch inside vLLM-Omni itself (deploy profiles, cold start, CPU platform, Jetson, codec-token streaming, slimming, device-direct connector).
+9. [edge_engine_comparison.md](edge_engine_comparison.md) — the branch's vLLM-Omni measured against llama.cpp (native Qwen3-TTS, CPU and CUDA) on the same host, prompts and reference speaker; why MNN/ExecuTorch numbers are still missing.
+10. [edge_device_simulation.md](edge_device_simulation.md) — what edge devices can be emulated on this H200-class host (MPS SM caps, memory budgets, core pinning) and what cannot (bandwidth, clocks); measured SM-cap sensitivity for both engines, a Jetson projection, and the per-frame attribution that explains the CPU numbers (15 sequential code-predictor passes at ≈6 ms each).
+11. [mobile_htp_simulation.md](mobile_htp_simulation.md) — 2026-09-10: Qualcomm AI Runtime SDK (Hexagon HTP simulator) installed without root, the branch's Code2Wav decoder converted (fp16 DLC) and run on the x86 HTP functional simulator with bit-exact CPU-backend parity; the CPU findings it triggered (KV-cached predictor, model-side sampler, fp32 compiled predictor; int8 rejected by in-situ parity) that take the CPU platform from RTF 1.7–1.8 to 0.89–1.27 on 16–32 threads (real time at 32), i.e. 1.6–2.2× faster end-to-end than llama.cpp on the same host, level at 8 threads; the code predictor also runs on the HTP simulator (fp16, 50.7 dB vs fp32); real Galaxy S24/S25 timings via Qualcomm AI Hub (§5): fp16 is faithful on device, and the right placement is heterogeneous — talker+predictor on the Hexagon NPU (49.7 ms/frame, fp16) and Code2Wav on the Adreno GPU (45.8 ms/frame at fp32, numerically exact, vs 182 ms on the NPU) — pipelining to RTF 0.62, i.e. real time on a phone with no quantization and no precision loss; every automatic int8 build is faster but numerically broken.
+12. **Implementation (2026-09-09):** [edge_branch_plan.md §5](edge_branch_plan.md#5-implementation-status-branch-edgeomni-edge-2026-09-09) records what was built on the branch and the measured outcome per work package; [experiments.md §8](experiments.md#8-edge-branch-experiments-2026-09-09) holds the runs; [experiments/edge_branch/results_summary.md](experiments/edge_branch/results_summary.md) the tables.
+12. [minicpmo_edge_profile.md](minicpmo_edge_profile.md) — 2026-09-11: MiniCPM-o 4.5 (3-stage omni) profiled with the same instruments: H200 batch RTF 0.748, CPU TTS head 11.6 ms/token and vocoder 833 ms/s of audio (91 % of it in the 10-step flow DiT), Galaxy S25 NPU 7.08 ms/token at 48.9 dB — 177 ms of NPU time per second of speech against Qwen3-TTS's 621 ms, because MiniCPM uses one codebook at 25 Hz instead of sixteen at 12.5 Hz; int8 breaks its code head too.
+13. [spark_edge_support.md](spark_edge_support.md) — 2026-09-11: Spark-X2.5-1.7B (agentic on-device model, hybrid 3:1 sliding/full attention, head-wise attention gate, per-layer-type partial RoPE) implemented for vLLM-Omni — vLLM 0.28 had no support, and MNN/ExecuTorch/ncnn still have none. Verified exact against the official code on transformers 4.57.1 (fp32, 32/32 greedy, 2.2e-05 logprob). On CPU it beats llama.cpp at matched bf16 by 8.7–15.4× on prefill and 1.19–2.65× on decode; an int8 path adds 1.6× but costs long-context fidelity; llama.cpp Q4_K_M still wins single-stream decode (no dense int4 kernel exists in vLLM's CPU build) while vLLM int8 wins from batch 16 and leads 1.72× at batch 64. GPU vs SGLang is a tie within 4 %. On a real Galaxy S25 a full decode step (output head included, every component verified at 36.9–62.8 dB) is 27.4 ms at 1 k context in calibrated w8a16 — 36.5 tok/s, 11.8% faster than the first working version and the first build whose output is correct at all: reading the per-op device profile showed a fifth of every sliding layer went on handing a recomputed KV window back across the graph boundary, and an on-device parity check showed the uncalibrated w4a16 builds everyone quotes score −1.6 dB, i.e. uncorrelated with the reference. Calibrated w8a16 is the deployable scheme (int8 activations still break it, the third model family here to show that). The hybrid attention beats an all-full-attention model by 1.51× at 4 k but 0.94× at 1 k, where the sliding layers' full-width RoPE outweighs the shorter cache. QNN/TFLite/ORT land within 2 % of each other on the NPU (the export layout sets on-device speed, not the runtime) and the NPU beats the Adreno GPU 2.4x here, the opposite of the TTS vocoder. Android control works with vLLM's stock glm45 parsers; thinking costs 14.6× latency for ~2/10 decisions.
+
+## Headline findings
+
+- **All six checkouts match the requested branches and commits and are clean**; ncnn and ExecuTorch submodules are uninitialized, and the machine's installed vLLM-Omni is a different checkout than the analyzed one ([repository_inventory.md](repository_inventory.md)).
+- **vLLM-Omni is a serving stack, not a runtime.** It subclasses vLLM's engine process, scheduler, and model runners, patches about ten vLLM symbols at import, runs every AR/generation stage as a vLLM engine process (diffusion stages run as a separate worker or inline via `inline_diffusion`), and joins stages by connectors whose default, `SharedMemoryConnector`, stages tensors through host memory (the Mooncake/Mori transfer-engine connectors have tensor fast paths and device pools). It has no CPU platform and no Apple/Android/Jetson code. What is portable is its stage semantics (1-frame first chunk, 25-frame steady chunks, 72-frame decoder context for Qwen3-TTS), its abort rules, its API shapes, and the plain-PyTorch codecs and action heads ([engines/vllm_omni.md](engines/vllm_omni.md), [engines/vllm_omni_models.md](engines/vllm_omni_models.md)).
+- **Measured on this host**: Qwen3-TTS 1.7B on one L20X gives a median first-audio latency of 27.7 ms, 160 ms per 2 s chunk, RTF 0.08–0.10, but 157–188 s of observed engine initialization in the default configuration (CUDA graphs + `torch.compile`; a lower bound with `enforce_eager` or parallel stage init was not measured); llama.cpp on x86 CPU decodes a 409 M-parameter Q4_0 LLM at 135.6 tok/s on 4 threads and spends 0.6–1.2 s per 512 px SigLIP tile on 8 threads ([experiments.md](experiments.md)).
+- **Qwen3-TTS already runs natively in llama.cpp and MNN**, and ExecuTorch has a streaming TTS runner (Voxtral) to copy from; so edge TTS is mostly an orchestration and parity problem rather than a model-porting problem, subject to a checkpoint/task parity gate (the server benchmark used CustomVoice 1.7B; llama.cpp's README demonstrates 1.7B Base and MNN's demo requires reference-audio cloning) ([comparison.md](comparison.md) §10).
+- **No runtime has a VLA policy in tree.** ExecuTorch (static memory, zero-copy I/O, in-graph loops, CoreML/QNN delegates) and MNN (DiT runtime, vision exporters, Android NPUs) are the viable hosts; llama.cpp lacks an action head; ncnn is a component runtime only ([vla_edge_design.md](vla_edge_design.md)).
+- **Recommended strategy**: keep vLLM-Omni as the reference/edge-server tier (Jetson-class only), ship a hybrid TTS first (server talker, on-device decoder over a codec-token stream of about 400 B/s raw payload) if the new wire contract and the on-device decoder measurement pass, otherwise start with a C++ orchestrator; in either case build a C++ orchestrator on llama.cpp/MNN/ExecuTorch that mirrors vLLM-Omni's pipeline semantics; VLA goes through ExecuTorch/MNN with simulation-first validation and a control/safety layer independent of inference ([vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md), [implementation_roadmap.md](implementation_roadmap.md)).
+- **Built and measured on 2026-09-09 (branch `edge/omni-edge`)**: an edge deploy profile with a ramped chunk schedule removes the ≈80 ms first-chunk stall (playback-start = TTFA = 41–44 ms, RTF unchanged); concurrent stage init cuts warm start from 152 s to 95 s (cold caches 355 s); a CPU platform runs Qwen3-TTS 0.6B end to end (RTF 1.4–2.8 on 8–32 Xeon threads, the talker step alone is 1.7–3× the frame budget); hardware auto-adaptation, codec-token streaming with an on-device decoder, import slimming, a CUDA-IPC connector and per-step overhead attribution (scheduler + transport < 2 % of the frame; no scheduler rewrite or native engine rewrite warranted) are in the tree ([edge_branch_plan.md §5](edge_branch_plan.md#5-implementation-status-branch-edgeomni-edge-2026-09-09)).
+- **Engine comparison (2026-09-09)**: on one L20X the branch generates a codec frame in 6 ms vs 17–18 ms for llama.cpp's native Qwen3-TTS (RTF 0.10–0.20 vs 0.28–0.31) and streams first audio in 41–54 ms without reference audio, but reference-audio (x-vector) requests cost it 0.4–0.6 s per request and it needs ≈90–150 s of startup and ≈37 GB of reserved GPU memory against llama.cpp's 3–6 s and a few GB; on CPU both engines land at RTF ≈2 for the 0.6B model, llama.cpp with half the RAM ([edge_engine_comparison.md](edge_engine_comparison.md)).
+- **Edge simulation and CPU analysis (2026-09-09)**: the cards are H200-class (804 TFLOPS bf16 measured; the driver labels them L20X). Capping SMs to 6 % with MPS costs vLLM-Omni's talker only +55 % per frame (latency-bound, CUDA graphs) while llama.cpp loses 2.8×; forcing eager mode for an 8 GB budget hurts far more (TTFA 160 ms, RTF 0.5). On CPU, 73 % of every frame is the 15 sequential code-predictor passes (6.4 ms each vs a 1.6 ms bandwidth floor) in vLLM-Omni and ≈85 % in llama.cpp; the fix is fusion/quantization/cheaper sampling for ≈2× and a lighter predictor for real time ([edge_device_simulation.md](edge_device_simulation.md)).
+- **Blockers**: no edge hardware reachable from this host; VLA weights not cached (download needs permission); ExecuTorch/MNN toolchains not installed (permission); vLLM-Omni's process model and host-staged default transport are architectural for the direct-port path (cold-start minimum unmeasured).
+
+## Conventions
+
+- Links are relative; source links go to `../<repo>/path` and were checked for existence by a link-check pass (see the end of [experiments.md](experiments.md)).
+- Diagrams are Mermaid.
+- Mobile/ARM/Apple numbers are never measured here (x86 host); they are either repo-reported with a citation or marked Unknown.
+
+## Second-model review
+
+On 2026-09-08 the finished study was independently reviewed by a second model (`gpt-6-astra` via the Codex hook, read-only against the same checkouts; its sandbox could not start because user namespaces are disabled on this host, so it ran unsandboxed under strict read-only instructions and the checkouts were verified clean afterwards). Initial verdict: agree with corrections (9 disagreements, 4 omissions). All were verified against source and applied; the main changes were: the default shared-memory transport is host-staged but the Mooncake/Mori transfer-engine connectors have device fast paths (Mori including intra-node XGMI), unvalidated for edge; only AR/generation stages are vLLM engine processes; the 1→25 chunk schedule leaves an ≈80 ms playback gap and the design now mirrors the YAML ramp/adaptive schedule; the CPU LLM benchmark is backbone-only (code predictor and decoder excluded); the 157–188 s figure is observed initialization, not a lower bound; prompt templates are engine code (`Qwen3TTSPromptEmbedsBuilder`); MNN's Qwen3-TTS path is single-callback; π0 uses SigLIP-So400m and draws noise inside `sample_actions`; simulation clients exist for DreamZero; hybrid-first is conditional on a codec-token wire-contract prototype and an on-device decoder measurement; a checkpoint/task parity gate was added. Final verdict after three rounds: **AGREE** ("agreement covers the source-grounded conclusions and conditional implementation plan; edge performance remains subject to the documented validation gates").

docs/analysis/edge_branch_plan.md

@@ -0,0 +1,157 @@
+# Plan: an edge branch for vLLM-Omni
+
+Target: a development branch in [`../vllm-omni`](../vllm-omni) (main @ `7be014bc`) that makes vLLM-Omni itself run, and perform acceptably, on edge devices. This is the "S1/S2/S4 inside vLLM-Omni" part of [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md); the C++ orchestrator work on llama.cpp/MNN/ExecuTorch (S3) stays outside this branch and is covered by [implementation_roadmap.md](implementation_roadmap.md). Every touch point below was located in the checkout; nothing has been changed yet.
+
+## 0. Ground rules
+
+- Branch: `edge/omni-edge` off `main@7be014bc`, repo-local identity `tzhouam <tzhouam@connect.ust.hk>`, credential helper via `gh` (per `/data/zhoutaichang/CLAUDE.md`). Work packages land as separate sub-branches (`edge/wp1-deploy-profiles`, ...) so each can become an upstream PR on its own.
+- Environment: a fresh `uv venv` for this checkout (`uv pip install -e .` per the repo's `AGENTS.md`/docs), because the machine's Python imports `/home/zhoutaichang/feature/vllm-omni-main` (`be335a86f`), not this tree. This is a dependency installation and needs your go-ahead.
+- Upstream policy: vLLM-Omni's `.claude/skills` (`precheck-pr`, `review-pr`, `vllm-omni-test`) define PR shape and test levels (`core_model` L1/L2, `advanced_model` L3, `full_model` L4; `cpu` platform marker exists in `pyproject.toml`). Each WP below names the tests it adds at those levels.
+- GPU runs go through `gpu run --timeout --note`; model downloads and installs are permission-gated.
+
+## 1. Work packages
+
+```mermaid
+graph LR
+  WP0[WP0 branch + env + baseline harness] --> WP1[WP1 edge deploy profiles]
+  WP0 --> WP2[WP2 cold-start reduction]
+  WP0 --> WP3[WP3 CPU platform]
+  WP1 --> WP4[WP4 Jetson / aarch64 CUDA]
+  WP2 --> WP4
+  WP1 --> WP5[WP5 codec-token streaming mode]
+  WP3 --> WP6[WP6 dependency slimming]
+  WP1 --> WP7[WP7 device-direct connector]
+  WP5 --> WP8[WP8 benchmarks + CI lanes]
+  WP3 --> WP8
+  WP4 --> WP8
+```
+
+### WP0 — Branch, environment, baseline harness (2–3 days)
+
+- Create the branch and the uv environment; confirm `python -c "import vllm_omni"` resolves to this checkout.
+- Promote the session driver [experiments/vllm_omni_tts/tts_stream_bench.py](experiments/vllm_omni_tts/tts_stream_bench.py) into `benchmarks/tts/stream_latency_bench.py` (reuses the example's `_build_inputs`; records TTFA, per-chunk arrivals, uninterrupted-playback latency, RTF streamed+tail, init time, RSS, `nvidia-smi` peak). JSON schema = the one in [experiments/vllm_omni_tts/bench_results.json](experiments/vllm_omni_tts/bench_results.json).
+- Re-run the Qwen3-TTS 1.7B baseline on this checkout (not the installed one) and add the 0.6B variant (driver flag to override the hard-coded model ids in `_build_inputs`).
+- Acceptance: baseline JSON for 0.6B and 1.7B on L20X committed under `benchmarks/tts/results/`; numbers within noise of [experiments.md](experiments.md) §2 for 1.7B.
+
+### WP1 — Edge deploy profiles (3–5 days)
+
+Touch points: `vllm_omni/deploy/` (90 YAMLs; platform overlays under `platforms:` exist in 23 of them), `vllm_omni/config/stage_config.py` (`enforce_eager` L408, `compilation_config` L418, `gpu_memory_utilization` L402, `devices` L386, `async_chunk` L548, `inline_diffusion` L273), `vllm_omni/model_executor/stage_input_processors/chunk_size_utils.py` (`parse_chunk_ramp` L42, `ramp_chunk_size` L79).
+
+- Add `vllm_omni/deploy/edge/qwen3_tts_edge.yaml` (and one per TTS family that passes Phase 0 parity): both stages on one device, `gpu_memory_utilization` sized from measured need (not 0.3), `codec_chunk_ramp: [4, 4, 8, 16, 25]` on by default (removes the ≈80 ms first-gap measured in [experiments.md](experiments.md)), `decode_batch_max_size: 1`, `max_num_seqs` small, `enforce_eager` documented as a switch per stage, sampling defaults unchanged.
+- Add a `profile: edge` selector so `--deploy-config` is not needed (a `DeployConfig` lookup by `(model, profile)` in `config/config_factory.py`).
+- Tests: L1 `tests/config/test_edge_profiles.py` (YAML resolves, schedule parses, ramp ends at `codec_chunk_frames`); L3 e2e TTFA/playback-gap regression on CUDA using WP0's harness.
+- Acceptance: uninterrupted-playback latency ≤ TTFA + 60 ms on L20X with the ramp (versus ≈190 ms today); no RTF regression > 10 %.
+
+### WP2 — Cold-start reduction (1–2 weeks)
+
+Touch points: `vllm_omni/engine/stage_runtime.py` (parallel replica init exists near L460; sequential today for co-located stages), `vllm_omni/engine/async_omni_engine.py` (`_wait_for_orchestrator_init`, 300 s default that failed run 1), `vllm_omni/worker/gpu_ar_model_runner.py` (talker-MTP graph capture, L546 in the installed log), `model_executor/models/qwen3_tts/{segmented_graph_wrapper.py,cuda_graph_decoder_wrapper.py}`, `common/qwen3_code_predictor.py` (warm-up buckets + `torch.compile`).
+
+- Instrument init with a timeline (process spawn, weight load, profile run, KV alloc, graph capture, compile, predictor warm-up) emitted as JSON; make it a `benchmarks/tts/init_profile.py`.
+- Measure the four configurations: default; `enforce_eager` both stages; compile cache warm; parallel stage init. Keep whichever combination meets the gate and make it the edge profile default.
+- Persist and reuse compile/graph artifacts (`VLLM_CACHE_ROOT` already caches AOT compile; add the code-predictor warm-up buckets and code2wav graph shapes to a versioned cache keyed by checkpoint hash + deploy profile).
+- Raise the default orchestrator timeout when the edge profile is active, or scale it from the measured timeline.
+- Acceptance: ready-to-serve ≤ 30 s on L20X for Qwen3-TTS 0.6B with warm caches, ≤ 90 s cold; init timeline test at L3.
+
+### WP3 — CPU platform (3–5 weeks)
+
+Touch points: `vllm_omni/platforms/__init__.py` (`builtin_omni_platform_plugins` L123: cuda/rocm/npu/xpu/musa), `vllm_omni/platforms/interface.py` (`OmniPlatform` surface: `get_omni_ar_worker_cls` L61, `get_omni_generation_worker_cls` L65, `get_diffusion_worker_cls` L175, `get_device_count` L266, memory/synchronize/autocast/`set_device_control_env_var` L329; `UnspecifiedOmniPlatform` L385), `vllm_omni/platforms/cuda/platform.py` (432 lines, the smallest template), `vllm_omni/worker/{base.py,gpu_ar_worker.py,gpu_ar_model_runner.py,gpu_generation_worker.py,gpu_generation_model_runner.py,mixins.py}`, `requirements/cpu.txt` (today: `common.txt` + `onnxruntime` only), `setup.py` (already accepts device `cpu`).
+
+Design: vLLM's `CPUWorker(Worker)` and `CPUModelRunner(GPUModelRunner)` (`vllm/v1/worker/cpu_worker.py`, `cpu_model_runner.py`) subclass the same GPU classes that vLLM-Omni's `OmniGPUWorkerBase`/`GPUARModelRunner` extend, so the CPU platform is built by composition rather than a rewrite:
+
+- `vllm_omni/platforms/cpu/platform.py`: `CPUOmniPlatform(OmniPlatform)` returning `vllm_omni.worker.cpu_ar_worker.CPUARWorker`, `cpu_generation_worker.CPUGenerationWorker`, a CPU diffusion worker (initially "unsupported" with a clear error), `get_device_count() = 1`, `device_control_env_var` = none, memory functions from `psutil`, `supports_talker_mtp_graph_capture() = False`, `supports_torch_inductor()` = vLLM CPU default, autocast = bf16/fp16 per vLLM CPU dtype rules.
+- `vllm_omni/worker/cpu_ar_worker.py`: `CPUARWorker(CPUWorker, OmniWorkerMixin)` with the omni hooks from `gpu_ar_worker.py` factored into `mixins.py` where they are not already; `cpu_ar_model_runner.py`: `CPUARModelRunner(CPUModelRunner)` + the omni runner mixin (`omni_connector_model_runner_mixin.py`); same pair for the generation stage (code2wav on CPU: no CUDA graphs, eager decoder, fp32).
+- Register `"cpu"` in `builtin_omni_platform_plugins` behind `VLLM_TARGET_DEVICE=cpu` or vLLM's CPU platform detection; keep `UnspecifiedOmniPlatform` for the no-device case.
+- `requirements/cpu.txt`: real runtime deps (soundfile, torchaudio CPU, onnxruntime) and the vLLM CPU wheel line; docs page `docs/getting_started/installation/cpu.md`.
+- Model-level: the Qwen3-TTS decoder's Triton Snake/GroupNorm kernels have torch fallbacks (verified in the model report); add a CPU branch in `CustomOp.dispatch_forward` users where `forward_native` is missing; CosyVoice3's TRT default (`COSYVOICE3_TRT`) must be off on CPU.
+- Tests: L1 `tests/platforms/test_cpu_platform.py` (registration, worker classes, device count) and per-family CPU unit tests already exist under `tests/model_executor/models/<family>/`; L3 e2e Qwen3-TTS 0.6B on CPU (`pytest -m "core_model and cpu"` lane).
+- Validation order: x86 on this host first (the CPU-only path needs no GPU reservation), then aarch64 Linux (vLLM ships aarch64 CPU wheels; KleidiAI W4A8 for the talker).
+- Acceptance: Qwen3-TTS 0.6B end-to-end on 8 x86 threads with RTF ≤ 1.0 and TTFA ≤ 500 ms; same test green on one aarch64 box; RSS ≤ 4 GB.
+
+### WP4 — Jetson / aarch64 CUDA (2–3 weeks, hardware-gated)
+
+Touch points: none in vLLM-Omni beyond the edge profile and docs; the work is environment: vLLM aarch64+CUDA build or wheel (vLLM detects Tegra via `/etc/nv_tegra_release`; CUDA arch 8.7/11.0 are in its `CMakeLists.txt`), torch for JetPack, `recipes/edge/jetson.md`.
+
+- Run WP0's harness and WP2's init profile on Orin; capture memory with unified-memory caveats (`gpu_memory_utilization` semantics differ when the GPU shares RAM with the CPU: add a `max_memory_gb` absolute option to the edge profile in `stage_config.py`).
+- Acceptance: Qwen3-TTS 0.6B RTF ≤ 0.5 and ready-to-serve ≤ 90 s on Orin; documented power mode; blocker if no board is available.
+
+### WP5 — Codec-token streaming mode (S4 wire contract) (2–3 weeks)
+
+Touch points: `vllm_omni/entrypoints/openai/serving_speech_stream.py` (`OmniStreamingSpeechHandler`, `_generate_and_send` L260 consumes PCM chunks), `vllm_omni/entrypoints/openai/tts_adapters/{base.py,capabilities.py}` (`register_tts_adapter`), `vllm_omni/model_executor/models/output_templates.py`, `vllm_omni/data_entry_keys.py` (`codes` key), `vllm_omni/deploy/edge/qwen3_tts_talker_only.yaml` (stage 0 only, `final_output_type` codes).
+
+- Server: a `mode: "codec_tokens"` session option on `/v1/audio/speech/stream` that runs the talker-only profile and emits frames `{seq, n_frames, codes: int16[n_frames×16], eos}` with explicit ordering, EOS, and cancel semantics; abort path unchanged (`AsyncOmni.abort`).
+- Adapter capability flag `supports_codec_stream` per family (Qwen3-TTS first; MOSS next since its codec is transformer-only).
+- Client reference: `examples/online_serving/text_to_speech/edge_decoder_client.py` decoding on device with the exported `Qwen3TTSTokenizerV2Decoder` (plain torch CPU first; ExecuTorch `.pte` second), carrying the 72-frame context and `reset()` on cancel.
+- Tests: L1 protocol tests (ordering, EOS, cancel mid-chunk); L3 e2e PCM parity between server-decoded and client-decoded audio (SNR ≥ 40 dB).
+- Acceptance: client-observed TTFA ≤ server TTFA + RTT + first-chunk decode; bandwidth ≈ 400 B/s payload + framing; interruption stops audio within one chunk.
+
+### WP6 — Dependency and import slimming (1–2 weeks)
+
+Touch points: `pyproject.toml` optional-dependencies, `requirements/common.txt` (diffusers pin, x-transformers, openai-whisper, onnxruntime), `vllm_omni/__init__.py` and `patch.py` import graph, `vllm_omni/diffusion/` imports pulled in by AR-only paths.
+
+- Add an `edge` extra that excludes diffusion/video dependencies; make diffusion, forced-aligner, and VAD imports lazy on the TTS path; measure `import vllm_omni` time and RSS before/after.
+- Acceptance: `python -c "import vllm_omni"` ≤ 3 s and no `diffusers` import for a TTS-only deploy; test at L1 with an import-guard test.
+
+### WP7 — Device-direct connector for co-located stages (2 weeks, optional)
+
+Touch points: `vllm_omni/distributed/omni_connectors/connectors/{base.py,shm_connector.py}` (`OmniConnectorBase.put/get`), `utils/serialization.py` (`OmniMsgpackEncoder`, zero-copy TODO), deploy YAML `connectors:` section.
+
+- `CudaIpcConnector`: `put` exports a CUDA IPC handle (`torch.multiprocessing` reductions) for same-GPU stages instead of D2H + msgpack; `get` maps it; metadata carries handle + shape + dtype; fallback to shared memory for non-tensor payloads. Mori's intra-node `xgmi` path shows the pattern for AMD; this covers single-GPU NVIDIA co-location (Jetson, single-card servers).
+- Acceptance: per-chunk hop latency measured with the WP0 harness drops for the Talker → Code2Wav edge; no change in audio; L1 unit test with a fake two-process exchange.
+
+### WP8 — Benchmarks and CI lanes (ongoing)
+
+- `benchmarks/tts/stream_latency_bench.py` and `init_profile.py` results committed per profile; a `cpu` Buildkite lane running `pytest -m "core_model and cpu"` for WP3; a nightly `edge` job for the L3 e2e tests on CUDA; recipes under `recipes/edge/`.
+
+## 2. Order and effort
+
+| Order | WP | Effort | Depends on | Needs hardware / permission |
+|---|---|---|---|---|
+| 1 | WP0 | 2–3 d | — | uv env install (permission); GPU via scheduler |
+| 2 | WP1 | 3–5 d | WP0 | GPU |
+| 3 | WP2 | 1–2 w | WP0 | GPU |
+| 4 | WP3 | 3–5 w | WP0 | none for x86 CPU; aarch64 box later |
+| 5 | WP5 | 2–3 w | WP1 | GPU; ExecuTorch install for the `.pte` client (permission) |
+| 6 | WP6 | 1–2 w | WP3 | none |
+| 7 | WP4 | 2–3 w | WP1, WP2 | Jetson board (blocker today) |
+| 8 | WP7 | 2 w | WP1 | GPU |
+| 9 | WP8 | ongoing | all | CI access |
+
+Critical path for a demonstrable result: WP0 → WP1 → WP2 gives a measurably better edge-server tier on the hardware we have; WP3 is the first item that reaches devices without a discrete GPU and is the largest.
+
+## 3. Risks
+
+- **vLLM drift**: vLLM-Omni imports 284 `vllm.*` modules and monkeypatches request/output classes; the CPU worker composition in WP3 depends on `CPUWorker`/`CPUModelRunner` internals that changed between the installed 0.28.0 and this checkout's target. Pin the vLLM commit in the branch's `requirements/` and re-validate at each vLLM minor.
+- **Upstream acceptance**: each WP is written as an upstream-shaped PR (tests at L1/L3, precheck skill), but WP3 and WP7 are large; expect design review through the repo's `imdesign`/`review-pr` flow before implementation.
+- **Model coverage**: the edge profiles are validated per family; families relying on TensorRT (CosyVoice3) or external packages (Fish `fish-speech`) stay CUDA-only until their CPU fallbacks are verified.
+- **Hardware**: WP4 and the aarch64 half of WP3 cannot start without boards; WP3-x86 and WP5 can.
+
+## 4. Definition of done for the branch
+
+- Edge profiles, CPU platform, and codec-token streaming merged on `edge/omni-edge`, each with L1 tests green locally and L3 e2e green on the CUDA lane.
+- Published benchmark JSON for: Qwen3-TTS 0.6B/1.7B on L20X (default vs edge profile), 0.6B on x86 CPU and on one aarch64 box, and (if a board arrives) Jetson.
+- Docs: `docs/getting_started/installation/cpu.md`, `recipes/edge/*.md`, and an updated `docs/design/feature/async_chunk.md` section on ramp defaults.
+
+## 5. Implementation status (branch `edge/omni-edge`, 2026-09-09)
+
+The branch lives in `/data/zhoutaichang/embedding_infer/vllm-omni` (from `main@7be014bc`, one commit per work package, not pushed). Everything below was measured on this host (one NVIDIA L20X at a time through `gpu run --gpus 1`, or CPU-only on NUMA node 1); the JSON is committed under `benchmarks/tts/edge_results/` and rendered in [experiments/edge_branch/results_summary.md](experiments/edge_branch/results_summary.md). Section numbers refer to [experiments.md §8](experiments.md#8-edge-branch-experiments-2026-09-09).
+
+| WP | Status | Where | Result / decision |
+|---|---|---|---|
+| WP0 baseline harness | done | `benchmarks/tts/stream_latency_{bench,metrics}.py`, `tests/benchmarks/` | 1.7B default: TTFA p50 32 ms, playback-start 116 ms, stall 85 ms, RTF 0.10; 0.6B: 28 / 95 / 67 ms. Reproduces the study's ≈80 ms first-chunk gap. |
+| WP1 edge profile + `deploy_profile` | done | `vllm_omni/deploy/edge/qwen3_tts.yaml`, `entrypoints/utils.py`, `--deploy-profile` | Ramp `[2,4,8,16,25]` + small batches: stall 0 ms, playback-start = TTFA (44 ms for 1.7B, 41 ms for 0.6B), RTF unchanged. TTFA +12 ms because the first chunk is 2 frames. |
+| WP2 init timeline + cold start | done | `engine/init_timeline.py`, `benchmarks/tts/init_profile.py` | Warm 1.7B: default 152 s, eager stage 1 150 s, eager both 111 s (first TTFA 15 s, rejected), `parallel_stage_init` 94.6 s (adopted via the profile's `orchestrator:` block), cold caches 355 s. Dominant phases: engine-core init 51–75 s per stage (model load 9–13 s, talker-MTP graph capture 27 s, predictor compile 12 s, code2wav graphs 4–6 s); the ≤30 s warm target is not met on this stack, the remainder is vLLM's own per-stage init. |
+| WP3 CPU platform | done (no vLLM change needed) | `vllm_omni/platforms/cpu/`, `platforms: cpu:` overlay, `docs/getting_started/installation/cpu.md` | 0.6B end to end on the `+cpu` wheel. Thread sweep: 8 threads TTFA 426 ms / RTF 2.8; 16: 292 ms / 1.8; 16 + inductor: 234 ms / 1.4; 32: 252 ms / 1.7. Attribution: the talker step is 136–250 ms per frame (170–310 % of the 80 ms budget); scheduler + hop overhead < 1 %. Not real-time on this CPU class. |
+| WP4 hardware adaptation | done (emulated) | `vllm_omni/edge/{hardware_probe,adapt,calibrate,auto_profile,decoder_export}.py`, `deploy/edge/hardware/*.yaml`, `benchmarks/tts/hw_emulation.py` | `deploy_profile="auto"` classifies and sizes every emulated class; Jetson-8 GB emulation (eager, absolute KV budget) TTFA 181 ms / RTF 0.57, Jetson-32 GB 68 ms / 0.09, 8 GB discrete 47 ms / 0.09; CPU classes as in WP3. Decoder export: windowed `torch.export` verified (exact vs eager, 460 MB fp32 per chunk size); ExecuTorch lowering written but not executed (install permission-gated). |
+| WP5 codec-token streaming | done (parity partly) | `deploy/qwen3_tts_talker_only.yaml`, `entrypoints/openai/codec_stream.py`, `serving_speech{,_stream}.py`, client + parity scripts, `docs/serving/codec_token_streaming.md` | Streams per step after three runtime fixes found by measurement (FINAL_ONLY coercion, list payloads, delta rows): client TTFA 152–156 ms, stall 0, ramp 2/4/8/16/25 frames, client RTF 0.5 with CPU decode. Client decoder == exact decoder (72–76 dB); end-to-end SNR vs the two-stage server is not meaningful because the two deployments sample different tokens for the same seed (§8.6). |
+| WP6 import slimming | done | lazy diffusion imports, `requirements/{core,media,*-edge}.txt`, `VLLM_OMNI_EDGE_BUILD` | AR/TTS path no longer imports diffusers/x_transformers/whisper (guard test); the remaining ≈20 s import overhead is transformers itself. |
+| WP7 CUDA-IPC connector | done, not adopted | `connectors/cuda_ipc_connector.py`, `deploy/qwen3_tts_cuda_ipc.yaml` | Talker→code2wav put→get hop: 2.4 ms with shared memory vs 2.9 ms (p95 10.7 ms) with CUDA-IPC (§8.7); below the ≥1 ms gain gate, so the edge profile keeps `SharedMemoryConnector`. |
+| WP8 edge scheduling | attribution done; gated items not triggered | `vllm_omni/edge/step_stats.py`, `vllm_omni/edge/scheduling.py`, `--step-stats` | Per-step counters show scheduler + chunk hop + dispatch < 2 % of the frame budget on GPU and CPU; the model step is the cost. Event-driven hand-off, inline AR stage and deadline co-scheduling stay unimplemented by the plan's 5 % gate; the no-preemption admission rule is implemented (exact KV bytes/token) and tested. |
+
+**Decisions recorded.**
+
+- *Init knobs*: keep CUDA graphs on both stages; enable `parallel_stage_init` in the edge profile (run 20260909-112200). Eager stage 1 buys nothing; eager on both cuts 41 s of init but costs 15 s of first-request latency.
+- *Scheduling*: no scheduler replacement. The dead `connector_get_sleep_s` knob (no consumer in the tree) and the 1 ms receive-loop back-off were confirmed; polling costs ≤1 ms per chunk, so an event-driven hand-off would recover ≤1.3 % of the frame budget and was not built. The admission rule (`vllm_omni/edge/scheduling.py`) is used by the hardware adaptation to size `max_num_seqs` so vLLM never preempts by recompute.
+- *Native rewrite*: not warranted by measurement. On the L20X the whole non-model overhead per frame is a few milliseconds; on CPU the talker forward alone is 2–3× the frame budget. The escape hatch remains the C++ S3 orchestrator on llama.cpp/MNN/ExecuTorch for devices where vLLM cannot run at all (phones), which the codec-token stream and the decoder export now serve.
+- *CUDA-IPC*: measured no gain on the talker → code2wav edge (the chunk tensor is a small host tensor; the IPC export/ack adds 0.5 ms and a worse tail), so the connector only pays off when a hop carries large device tensors (Qwen3-Omni hidden states); it stays available behind `qwen3_tts_cuda_ipc.yaml`.
+
+**Not done / deviations.** ExecuTorch was not installed (permission gate; the export path is unit-tested with a fake decoder and the `.pt2` export with the real one). The torch-profiler attribution on CPU was replaced by the step counters (same question, lower overhead). No real edge hardware was used; Jetson/ARM numbers are emulations of the derivation logic, not of the silicon. The customvoice regression e2e passed; the codec-stream e2e passed after the client fix, and three further fixes were needed before frames actually streamed per step (§8.6) — the e2e test checks frames and decoding, not cadence, which is a gap to close.

docs/analysis/experiments.md

@@ -0,0 +1,294 @@
+# Experiments log
+
+All runs recorded here were executed on 2026-09-08 from this session. Nothing was installed, downloaded, or modified in any checkout. Builds and outputs live in a session scratch directory; logs that matter are copied under [experiments/](experiments/). Numbers below are **measured on this specific host** and are labelled as such; none of them is an edge-device number (the host is an x86 server), so they only calibrate relative costs and validate the execution paths that the design documents rely on.
+
+## 0. GPU scheduling procedure followed
+
+Source: `/home/zhoutaichang/feature/GPU调度工具快速上手.pdf` (read in full, 3 pages) and the copy of its rules in `/data/zhoutaichang/CLAUDE.md`. The relevant rules, all honored:
+
+| Rule (from the guide) | How it was applied |
+|---|---|
+| Use a personal user, never root | All commands ran as `zhoutaichang` (uid 2005). |
+| `gpu status` before starting | Run before submission; all 8 L20X cards were AVAILABLE; a pending booking `84212f67` (another user, GPUs 0-3, 03:00-06:00 daily CI) was noted and avoided by not requesting specific card IDs. |
+| Prefer `gpu run --gpus N -- cmd` (auto reserve, sets `CUDA_VISIBLE_DEVICES`, auto release) | Used for all three GPU submissions (runs 1–3). |
+| Always `--timeout` on long jobs | `--timeout 30m`. |
+| Add `--note` | `--note "tzhouam(claude): qwen3-tts-1.7B streaming TTFA baseline, 12 prompts"`. |
+| Release immediately when done | `gpu run` auto-releases; `gpu release` was additionally invoked after the job and `gpu status` re-checked (see §2.4). |
+| Never bypass the reservation | No process touched a GPU outside `gpu run`; CPU experiments used no GPU (`GGML_CUDA=OFF`). |
+
+No dependency installation, model download (`HF_HUB_OFFLINE=1` was exported for the GPU job), or source change was made. Experiments judged expensive (multi-GPU, large models, long sweeps) were **deferred** and are listed in §5 with what permission they need.
+
+## 1. Hardware and software environment
+
+| Item | Value (verified with `lscpu`, `free`, `nvidia-smi`, `python -c`) |
+|---|---|
+| Host | `dedicated-developjob-8gpu2-a029z-…`, Ubuntu 22.04.5, kernel 5.10.134 |
+| CPU | Intel Xeon Platinum 8480C, 224 hardware threads, x86_64 (AVX-512 class; `-march=native` build) |
+| RAM | 2014 GB total |
+| GPU | 8 × NVIDIA L20X, 143,771 MiB each, driver 570.133.20, CUDA 13.0 (`/usr/local/cuda/bin/nvcc`, not on `PATH`) |
+| Toolchain | gcc 11.4.0, cmake 4.4.0, ninja 1.13 |
+| Python | 3.12.13; `torch 2.13.0+cu129`, `transformers 5.14.1`, `vllm 0.28.0` (wheel), `vllm-omni 0.28.0rc2.dev22+gbe335a86f` (editable, from `/home/zhoutaichang/feature/vllm-omni-main`), `onnxruntime 1.29.0`, `torchao 0.9.0`, `gguf` importable |
+| Analyzed commits | see [repository_inventory.md](repository_inventory.md) |
+
+**Important mismatch.** The GPU experiment runs the *installed* vLLM-Omni (`be335a86f`) against the example script from the *analyzed* checkout (`7be014bc`). The two copies of `examples/offline_inference/text_to_speech/qwen3_tts/end2end.py` were confirmed byte-identical (`diff -q`), and the installed checkout's working tree is clean, but the library code differs by an unknown number of commits. Treat the TTS numbers as "vLLM-Omni around early Sept 2026", not as the exact analyzed commit.
+
+## 2. Experiment A — vLLM-Omni Qwen3-TTS streaming baseline (GPU, scheduled)
+
+**Purpose.** Establish a server-class reference for first-audio latency (TTFA), inter-chunk cadence, and end-to-end time of the two-stage talker → code2wav pipeline that [tts_edge_design.md](tts_edge_design.md) proposes to re-host on edge devices. Also verifies the `async_chunk` shared-memory streaming path described in [engines/vllm_omni.md](engines/vllm_omni.md).
+
+**Model.** `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` (already present in the HF cache, 4.3 GB; the example hard-codes the 1.7B variants, see `_build_inputs` in [`end2end.py`](../vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/end2end.py)). Deploy config: the model's default [`vllm_omni/deploy/qwen3_tts.yaml`](../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml) (`async_chunk: true`, `SharedMemoryConnector` with `codec_streaming: true`, `initial_codec_chunk_frames: 1`, `codec_chunk_frames: 25`, stage 0 `gpu_memory_utilization: 0.3`, both stages on logical device `"0"`).
+
+**Command (run 1, exact; runs 2 and 3 differ as listed in the run-history table below, run 3 invoking [tts_stream_bench.py](experiments/vllm_omni_tts/tts_stream_bench.py) with the same flags plus `--init-timeout 1500 --stage-init-timeout 1500`).**
+```bash
+export HF_HUB_OFFLINE=1
+cd <scratch>/exp_tts
+gpu run --gpus 1 --timeout 30m \
+  --note "tzhouam(claude): qwen3-tts-1.7B streaming TTFA baseline, 12 prompts" -- \
+  python /data/zhoutaichang/embedding_infer/vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
+    --query-type CustomVoice --streaming \
+    --txt-prompts /data/zhoutaichang/embedding_infer/vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt \
+    --output-dir <scratch>/exp_tts/out --log-dir <scratch>/exp_tts/logs
+gpu release
+```
+Prompts: the 12 English sentences shipped in [`benchmark_prompts.txt`](../vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt). Requests are issued sequentially by the example (`main_streaming` loops over prompts one at a time), so this measures single-stream latency, not throughput.
+
+**Job record.**
+
+| Field | Value |
+|---|---|
+| Scheduler | `canhazgpu` via `gpu run`; type RUN |
+| Physical GPU assigned | 6 (log line `Stage 0 logical-to-physical device mapping: 0->6`) |
+| Main PID | 2197539 |
+| Start | see [experiments/vllm_omni_tts/run1_start.txt](experiments/vllm_omni_tts/run1_start.txt) |
+| Full stdout/stderr | [experiments/vllm_omni_tts/run1_stdout.log](experiments/vllm_omni_tts/run1_stdout.log) |
+
+**Run history (three submissions, all through `gpu run`).**
+
+| Run | Command variant | Outcome | Log |
+|---|---|---|---|
+| 1 | example `end2end.py --streaming`, default timeouts | **Failed at startup**: `TimeoutError: Orchestrator did not become ready within 300s`. Stage 0 was still in a cold `torch.compile` of the code predictor (talker CUDA graphs done at 00:13:12, predictor warmup at 00:14:22, timeout at 00:14:34). GPU 6 released by `gpu run`; no stray process. | [run1_stdout.log](experiments/vllm_omni_tts/run1_stdout.log) |
+| 2 | same + `--init-timeout 1500 --stage-init-timeout 1500` | Completed (exit 0), 12 WAVs written, engine ready after 187.7 s (stage 0 startup 00:15:33→00:18:22, stage 1 00:18:22→00:19:22). But the example's per-chunk `logger.info` lines are not emitted (module logger with no handler), and in streaming mode `_save_wav` receives only the final message's audio, so the saved WAVs (0.08–1.92 s) are the last partial chunk, not the utterance. No latency data usable. | [run2_stdout.log](experiments/vllm_omni_tts/run2_stdout.log) |
+| 3 | session driver [tts_stream_bench.py](experiments/vllm_omni_tts/tts_stream_bench.py) (imports the example's `_build_inputs`/`parse_args`, same flags, adds one warm-up request, records every chunk) | Completed (exit 0); engine ready after 156.8 s (compile caches warm). Results below. | [run3_stdout.log](experiments/vllm_omni_tts/run3_stdout.log), [bench_results.json](experiments/vllm_omni_tts/bench_results.json) |
+
+**Results (run 3; 1 × L20X assigned by the scheduler; both stages co-located on it; installed vLLM-Omni `be335a86f` + vLLM 0.28.0; `HF_HUB_OFFLINE=1`).** The warm-up request (`w1`: TTFA 420.8 ms, first request after init) is excluded from the statistics.
+| Metric (12 sequential requests, batch size 1) | p50 | p95 | max |
+|---|---|---|---|
+| TTFA, ms (request submit → first audio chunk received in the client coroutine) | 27.7 | 30.7 | 32.8 |
+| Inter-chunk gap for 25-frame chunks, ms | 159.9 | 165.4 | 221.7 |
+| RTF using streamed chunk audio only | 0.098 | 0.113 | 0.122 |
+| RTF using streamed chunks + final-message tail | 0.083 | 0.085 | 0.086 |
+
+Per-request detail (chunk arrival times in ms after submit; samples at 24 kHz mono):
+
+| Req | Text (truncated) | TTFA (ms) | Chunk arrivals (ms) / samples | Streamed audio (s) | Final-message tail (s) | Wall (s) | RTF (streamed) | RTF (streamed+tail) |
+|---|---|---|---|---|---|---|---|---|
+| 0 | Hello, welcome to the voice synthesis … | 29.7 | 30/1920 190/48000 348/48000 | 4.08 | 0.40 | 0.380 | 0.093 | 0.085 |
+| 1 | She said she would be here by noon, bu… | 30.6 | 31/1920 195/48000 353/48000 | 4.08 | 0.32 | 0.376 | 0.092 | 0.085 |
+| 2 | The quick brown fox jumps over the laz… | 32.8 | 33/1920 190/48000 351/48000 | 4.08 | 0.08 | 0.360 | 0.088 | 0.086 |
+| 3 | I can't believe how beautiful the suns… | 27.8 | 28/1920 187/48000 350/48000 510/48000 | 6.08 | 0.16 | 0.518 | 0.085 | 0.083 |
+| 4 | Please remember to bring your identifi… | 26.5 | 27/1920 190/48000 351/48000 507/48000 | 6.08 | 0.40 | 0.536 | 0.088 | 0.083 |
+| 5 | Have you ever wondered what it would b… | 30.7 | 31/1920 193/48000 353/48000 | 4.08 | 1.92 | 0.497 | 0.122 | 0.083 |
+| 6 | The restaurant on the corner serves th… | 25.5 | 26/1920 191/48000 347/48000 | 4.08 | 1.20 | 0.436 | 0.107 | 0.083 |
+| 7 | After the meeting, we should discuss t… | 27.5 | 28/1920 187/48000 346/48000 | 4.08 | 1.44 | 0.459 | 0.113 | 0.083 |
+| 8 | Learning a new language takes patience… | 27.4 | 27/1920 188/48000 345/48000 503/48000 | 6.08 | 1.60 | 0.623 | 0.103 | 0.081 |
+| 9 | The train leaves at half past seven, s… | 26.4 | 26/1920 189/48000 350/48000 503/48000 | 6.08 | 0.80 | 0.565 | 0.093 | 0.082 |
+| 10 | Could you please turn down the music a… | 28.5 | 28/1920 250/48000 355/48000 | 4.08 | 1.28 | 0.450 | 0.110 | 0.084 |
+| 11 | It was a dark and stormy night when th… | 27.6 | 28/1920 190/48000 348/48000 | 4.08 | 1.20 | 0.437 | 0.107 | 0.083 |
+
+How to read these numbers:
+
+- **Chunk structure matches the deploy config.** Every request delivers a first chunk of 1,920 samples (= 1 codec frame at 12.5 Hz, 80 ms of audio; `initial_codec_chunk_frames: 1`), then 48,000-sample chunks (25 frames = 2.0 s; `codec_chunk_frames: 25`). The streamed chunk count is therefore quantized to 1 + 25k frames, and the remainder arrives in the final message (0.08–1.92 s). The "streamed + tail" column is the best estimate of the true utterance length [Inferred: the tail was not verified to be disjoint from the last streamed chunk, but its RMS envelope is speech-like and its length varies per sentence].
+- **TTFA of about 28 ms** covers talker prefill of the CustomVoice prompt, one talker decode step with the 15-codebook code predictor, code2wav of one frame, shared-memory transfer, and the async return path. This is a single-request, warm-graph number on a data-center GPU and is the floor the edge design has to be compared against, not a target for edge.
+- **Steady-state cadence is about 160 ms per 25 frames**, i.e. roughly 6.4 ms per codec frame for talker decode + code predictor + code2wav together, well under the 80 ms real-time budget per frame (RTF ≈ 0.08–0.10).
+- **Startup dominates**: 157–188 s of engine init (two vLLM engine processes, CUDA-graph capture, `torch.compile`), versus 0.4–0.6 s per utterance. This is the single most important qualitative fact for edge: the vLLM-Omni process model is built for long-running servers, not for cold-start on a device.
+- Memory: stage 0 requested 30 % of the card (41.94 GiB budget by config, not need); actual model footprint was not isolated in this run [Unknown].
+
+Artifacts kept: the JSON above and the logs. The 13 WAV files and the run-2 WAVs stay in the session scratch directory (not copied, ~2 MB).
+
+
+## 3. Experiment B — llama.cpp CPU-only edge-native baseline (no GPU)
+
+**Purpose.** Validate the "edge-native baseline" path that [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) compares against: HF checkpoint → GGUF → quantize → CPU inference, for a small VLM whose shape (SigLIP vision encoder + ~400M LLM) is representative of a VLA perception/language backbone. Also measure how much the vision encoder costs relative to token decode, which drives the VLA stage-placement argument in [vla_edge_design.md](vla_edge_design.md).
+
+**Model.** `HuggingFaceTB/SmolVLM2-500M-Instruct` (cached, 1.9 GB safetensors). Chosen because it was the only cached checkpoint with weights that llama.cpp's converter supports for both text and vision (`conversion/smolvlm.py`, `tools/mtmd/mtmd-image.cpp::mtmd_image_preprocessor_idefics3`).
+
+**Build (out-of-tree, checkout untouched).**
+```bash
+cmake -S llama.cpp -B <scratch>/llama_build -DCMAKE_BUILD_TYPE=Release \
+      -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DGGML_NATIVE=ON
+cmake --build <scratch>/llama_build -j 32 \
+      --target llama-cli llama-bench llama-simple llama-mtmd-cli llama-tts llama-quantize llama-server
+```
+Configure log: [experiments/llama_cpp_cpu/llama_cmake_configure.log](experiments/llama_cpp_cpu/llama_cmake_configure.log) (`GGML_SYSTEM_ARCH: x86`, CPU backend variant `-march=native`). Build finished with exit 0 ([tail](experiments/llama_cpp_cpu/llama_cmake_build.tail.log)). Resulting shared libraries: `libggml-base.so` 0.92 MB, `libggml-cpu.so` 1.53 MB, `libggml.so` 0.06 MB; `llama-cli` executable 1.3 MB (dynamically linked, x86 with AVX-512 code paths, so an ARM/Android build will differ).
+
+**Conversion and quantization (CPU).**
+```bash
+PYTHONPATH=llama.cpp/gguf-py python3 llama.cpp/convert_hf_to_gguf.py <snapshot> --outfile smolvlm2-500m-f16.gguf --outtype f16
+PYTHONPATH=llama.cpp/gguf-py python3 llama.cpp/convert_hf_to_gguf.py <snapshot> --outfile smolvlm2-500m-mmproj-f16.gguf --outtype f16 --mmproj
+llama-quantize smolvlm2-500m-f16.gguf smolvlm2-500m-q8_0.gguf Q8_0
+llama-quantize smolvlm2-500m-f16.gguf smolvlm2-500m-q4_0.gguf Q4_0
+```
+Logs: [convert_text.log](experiments/llama_cpp_cpu/convert_text.log), [convert_mmproj.log](experiments/llama_cpp_cpu/convert_mmproj.log), [quantize_q8_0.log](experiments/llama_cpp_cpu/quantize_q8_0.log), [quantize_q4_0.log](experiments/llama_cpp_cpu/quantize_q4_0.log). All exit 0.
+
+| Artifact | Size (bytes) | Note |
+|---|---|---|
+| `smolvlm2-500m-f16.gguf` | 820,421,728 | text decoder, 409.25 M params as reported by `llama-bench` (the vision tower is excluded) |
+| `smolvlm2-500m-q8_0.gguf` | 436,805,728 | |
+| `smolvlm2-500m-q4_0.gguf` | 255,864,928 | |
+| `smolvlm2-500m-mmproj-f16.gguf` | 199,467,680 | SigLIP vision encoder + projector, always f16 here |
+
+**Text decode throughput (`llama-bench`, `-p 128 -n 64 -r 3`, CPU backend only, build `e71b80510`).** Raw output: [bench_results.md](experiments/llama_cpp_cpu/bench_results.md). Thread counts 4 and 8 were chosen to mimic a mobile big-core cluster; absolute numbers on a Xeon 8480C are **not** transferable to ARM, only the ratios are indicative.
+
+| Quant | Threads | Prompt eval pp128 (tok/s) | Generation tg64 (tok/s) |
+|---|---|---|---|
+| F16 | 4 | 379.6 ± 0.5 | 48.2 ± 4.5 |
+| Q8_0 | 4 | 997.3 ± 6.9 | 74.0 ± 13.3 |
+| Q4_0 | 4 | 935.4 ± 7.7 | 135.6 ± 1.3 |
+| F16 | 8 | 592.7 ± 1.2 | 68.8 ± 7.1 |
+| Q8_0 | 8 | 1684.6 ± 73.1 | 150.1 ± 21.0 |
+| Q4_0 | 8 | 1522.2 ± 165.3 | 220.1 ± 28.3 |
+
+Observations (host-specific): decode is memory-bound, so Q4_0 gives ~2.8-3.2× the F16 decode rate; prompt processing is compute-bound and saturates at Q8_0. This is a backbone-only microbenchmark: a Qwen3-TTS frame additionally runs 15 code-predictor sub-steps (`qwen3_code_predictor.py`) and the speech decoder, neither of which is measured here, so it bounds only the talker backbone's share of an 80 ms frame budget and supports no real-time claim for any device.
+
+**Vision-language path (`llama-mtmd-cli`, Q8_0 text + F16 mmproj, 8 threads, greedy, 48 new tokens).**
+```bash
+llama-mtmd-cli -m smolvlm2-500m-q8_0.gguf --mmproj smolvlm2-500m-mmproj-f16.gguf \
+  --image MNN/resource/images/cat.jpg -p "Describe this image in one sentence." -t 8 -n 48 --temp 0
+```
+Output (verbatim from [mtmd_run.log](experiments/llama_cpp_cpu/mtmd_run.log)): "A small orange kitten is sitting on a rock outdoors, looking directly at the camera." Exit 0. The image is `MNN/resource/images/cat.jpg` (a repo asset, used only as input).
+
+From the timestamps in [mtmd_run.err](experiments/llama_cpp_cpu/mtmd_run.err): the Idefics3 preprocessor split the image into 13 image chunks (27 prompt chunks in total, image and text interleaved); each image chunk was encoded in 596-1184 ms at 8 threads, i.e. roughly **9-11 s of vision encoding per image** with default tiling. With `--image-max-tokens 64` ([mtmd_run_maxtok64.err](experiments/llama_cpp_cpu/mtmd_run_maxtok64.err)) the preprocessor produced 2 image chunks of ~640 ms each (~1.3 s). Implication for VLA (see [vla_edge_design.md](vla_edge_design.md)): the tiling policy of the image preprocessor, not the LLM, dominates per-frame cost on CPU; an edge VLA must pin a single low-resolution tile per camera, and on CPU-only targets even one SigLIP-B/16-class tile at 512 px costs ~0.6 s at 8 x86 threads in llama.cpp's f16 path, which alone exceeds a 10 Hz control budget. This is a measured fact about this host and this engine, not a projection for ARM or GPU.
+
+## 4. Experiments not run and why
+
+| Experiment | Status | Reason / prerequisite |
+|---|---|---|
+| vLLM-Omni VLA (DreamZero / GR00T / InternVLA-A1) control-loop latency on GPU | **Deferred** | Weights are not in the cache (`GEAR-Dreams/DreamZero-DROID` has metadata only); a download of tens of GB requires permission. Also the DreamZero example is a client that expects a running server. |
+| vLLM-Omni Qwen3-TTS 0.6B vs 1.7B TTFA comparison | Deferred | 0.6B weights are cached, but the example hard-codes 1.7B; running 0.6B needs a small script change (source change in an example → permission) or a custom driver. Cheap once approved. |
+| vLLM CPU backend (`VLLM_TARGET_DEVICE=cpu`) on x86 | Deferred | The installed vLLM is the CUDA wheel; a CPU build requires a new environment (dependency installation). |
+| ExecuTorch export of a TTS/VLA sub-module to `.pte` + XNNPACK run | Deferred | ExecuTorch is not installed and its submodules are uninitialized; `./install_executorch.sh` is a dependency installation. |
+| MNN `llmexport.py` + `llm_demo` CPU run | Deferred | Needs an MNN build (cheap) plus its Python export deps; kept for the roadmap's phase 0 gate. |
+| ncnn Vulkan build | Deferred | `glslang` submodule uninitialized; CPU-only build possible but there is no cached model in ncnn format. |
+| Any ARM / Apple Silicon / Android / Jetson measurement | **Blocked** | No such hardware is reachable from this host. |
+
+## 5. Reproducible benchmark protocols proposed for the edge work
+
+These are proposals (not executed) that the roadmap's acceptance criteria refer to; they reuse only artifacts shown to exist above.
+
+**P-TTS-1 (server reference).** Experiment A as-is, plus `--batch-size 1..8` with `--use-batch-sample`, reporting TTFA p50/p95, inter-chunk p95, RTF = generation wall time / audio seconds, and peak `nvidia-smi` memory. Acceptance for any edge port: same text set, same speaker, TTFA and RTF reported with the identical definitions, audio compared with a speech-recognition WER check on the 12 sentences.
+
+**P-TTS-2 (edge talker).** Talker exported alone (see [tts_edge_design.md](tts_edge_design.md) for the split); decode tokens/s measured by the target engine's bench tool (`llama-bench`, MNN `llm_bench`, ExecuTorch `llama_main` timing) at 4 and 8 threads; must exceed 12.5 codec frames/s × (1 + margin) for the "real-time" gate.
+
+**P-VLA-1 (perception cost).** Experiment B's `llama-mtmd-cli` run with `--image-max-tokens` swept over {64, 256, default}, plus the same encoder exported to ExecuTorch/MNN, reporting ms per frame at fixed resolution; gate: encoder + fusion + action head per control step below the chosen control period with p95 jitter below 20 % of it, measured in simulation first.
+
+## 6. Cleanup
+
+- GPU: `gpu run` released the reservation after each of the three runs; `gpu release` was run again after runs 2 and 3 and `gpu status --no-schedule` showed no reservation held by `zhoutaichang` ([cleanup_status.txt](experiments/vllm_omni_tts/cleanup_status.txt), [cleanup_status3.txt](experiments/vllm_omni_tts/cleanup_status3.txt)); `nvidia-smi` showed 0 MiB used on all eight cards afterwards.
+- Build trees, GGUF files (about 1.7 GB), and generated WAV files live only in the session scratch directory and are not part of the repository; the logs copied under [experiments/](experiments/) total a few hundred KB.
+- No checkout was modified (`git status --porcelain` empty for all six after the runs; see [repository_inventory.md](repository_inventory.md)).
+
+## 7. Reference verification
+
+All Markdown files under `analysis/` were scanned with [experiments/linkcheck.py](experiments/linkcheck.py): every relative link target must exist, and every `#Lnnn` anchor must be within the target file's length. Final pass: 2,677 links checked, 0 broken (three stale line anchors in the model-level report were corrected during the pass).
+
+## 8. Edge branch experiments (2026-09-09)
+
+All runs of this section were executed from the `edge/omni-edge` branch of `/data/zhoutaichang/embedding_infer/vllm-omni` with the two approved environments (`../.venvs/omni-cuda`: vllm 0.28.0 CUDA + editable vllm-omni; `../.venvs/omni-cpu`: vllm 0.28.0+cpu wheel). Results are committed JSON under `vllm-omni/benchmarks/tts/edge_results/` and are rendered as tables in [experiments/edge_branch/results_summary.md](experiments/edge_branch/results_summary.md) (regenerate with `python benchmarks/tts/edge_results/summarize.py`). The status per work package is in [edge_branch_plan.md §5](edge_branch_plan.md#5-implementation-status-branch-edgeomni-edge-2026-09-09).
+
+### 8.1 GPU scheduling procedure
+
+The rules of §0 were applied unchanged, with the user's constraint of **at most one GPU at a time**: every GPU job was one `gpu run --gpus 1 --timeout 40m --wait 30m --note "tzhouam(claude): <wp> ..." -- env HF_HUB_OFFLINE=1 VLLM_WORKER_MULTIPROC_METHOD=spawn ...` invocation, executed **sequentially** by three shell chains ([chain1.sh](experiments/edge_branch/chain1.sh), [chain2.sh](experiments/edge_branch/chain2.sh), [chain3.sh](experiments/edge_branch/chain3.sh); each chain waits for the previous one to exit before it starts). Each chain ran from a detached git worktree snapshot of the branch so that code edits during a run could not change a running job. `gpu release` and `gpu status --no-schedule` followed every job; the job list from `gpu report --days 7` is in §8.9. CPU-only runs (CPU platform sweep, x86/ARM emulation, decoder export) used no GPU (`VLLM_TARGET_DEVICE=cpu`, `CUDA_VISIBLE_DEVICES=` empty, pinned to NUMA node 1 with `taskset`).
+
+### 8.2 Streaming latency: default vs edge profile (WP0/WP1)
+
+`benchmarks/tts/stream_latency_bench.py`, 12 prompts × 3 repeats after 1 warm-up, sequential single-stream requests, metrics from `stream_latency_metrics.py` (TTFA; playback-start latency = max over chunks of arrival time minus audio already buffered; stall at TTFA = playback-start − TTFA; RTF = generation wall time / audio seconds).
+
+| model | deploy | run | init s | TTFA p50 / p90 ms | playback-start p50 ms | stall p50 ms | RTF | peak GPU MiB |
+|---|---|---|---|---|---|---|---|---|
+| 1.7B | default | 20260909-103504 | 271 | 32.0 / 35.9 | 116.2 | 84.7 | 0.10 | 46272 |
+| 1.7B | edge | 20260909-145133 | 149 | 43.7 / 48.2 | 43.7 | 0.0 | 0.10 | 40876 |
+| 0.6B | default | 20260909-104204 | 152 | 27.8 / 32.0 | 95.4 | 67.3 | 0.10 | 46846 |
+| 0.6B | edge | 20260909-145500 | 144 | 40.7 / 45.2 | 40.7 | 0.0 | 0.10 | 41248 |
+
+The edge profile removes the first-chunk stall entirely (the ramp's 2-frame first chunk lands before the 4-frame second chunk is needed) for ≈12 ms of TTFA; RTF and peak memory (per-process, both stages on one card) are unchanged within noise. With the profile's `parallel_stage_init` active (chain 2, runs 20260909-152439/152742, `edge_final2`): 1.7B init 87.7 s, TTFA p50 46.4 ms, stall 0; 0.6B init 86.7 s, TTFA p50 54.0 ms (p90 97 ms, one noisy repeat), stall 0 — the init gain (149 → 88 s) with the same steady-state latency.
+
+### 8.3 Init timeline and cold start (WP2)
+
+`benchmarks/tts/init_profile.py` (1.7B, one config per subprocess, `VLLM_OMNI_INIT_TIMELINE` phases from orchestrator and workers; run 20260909-112200 warm caches, 20260909-144054 cold caches):
+
+| config | init s | first-request TTFA ms | engine_core_init s | load_weights s | talker MTP capture s | predictor compile s | code2wav graphs s |
+|---|---|---|---|---|---|---|---|
+| default | 152.0 | 646 | 66.2 | 8.9 | 27.4 | 11.6 | 3.6 |
+| eager_stage1 | 149.9 | 781 | 67.1 | 6.4 | 29.1 | 11.4 | 0 |
+| eager_both | 110.9 | 14956 | 27.8 | 6.2 | – | – | 0 |
+| parallel_stage_init | 94.6 | 520 | 75.4 | 10.5 | 26.9 | 12.3 | 3.7 |
+| edge (profile, serial init) | 145.7 | 512 | 51.4 | 12.7 | 7.2 | 5.1 | 5.8 |
+| cold_cache (fresh VLLM/inductor/triton caches) | 355.5 | 465 | – | – | – | – | – |
+
+Reading: the two stages initialise serially by default and each spends 50–75 s inside vLLM's `EngineCore.__init__` (device init, weight load, KV profiling, graph capture); running them concurrently on the shared card (`parallel_stage_init`) saves 57 s. Disabling graphs on both stages saves 41 s but the first request then pays the talker-MTP and predictor warm-up (15 s), so the edge profile keeps graphs. A cold inductor/Triton cache adds 200 s; the caches must ship with an edge image.
+
+### 8.4 CPU platform (WP3) and CPU thread sweep
+
+`benchmarks/tts/cpu_thread_sweep.py` (0.6B CustomVoice, `+cpu` wheel, bf16 with AMX on the 8480C, stage 0 gets 3/4 of the threads, `--step-stats`), run 20260909-145257 on NUMA node 1 while the GPU chains ran on other cores:
+
+| threads (stage0+stage1) | init s | TTFA p50 ms | playback-start p50 ms | RTF | peak RSS GiB | talker step ms (mean / p95) | scheduler ms per step | chunk hop enqueue→put / get ms |
+|---|---|---|---|---|---|---|---|---|
+| 8 (6+2) | 112.7 | 426 | 9438 | 2.78 | 13.3 | 250.5 / 499.7 | 0.13 | 1.37 / 0.94 |
+| 16 (12+4) | 109.5 | 292 | 5346 | 1.84 | 13.3 | 147.4 / 219.6 | 0.12 | 0.82 / 0.79 |
+| 16 + inductor (`VLLM_OMNI_CPU_INDUCTOR=1`) | 103.4 | 234 | 3642 | 1.43 | 13.3 | 138.9 / 180.0 | 0.12 | 0.83 / 0.91 |
+| 32 (24+8) | 155.7 | 252 | 4995 | 1.73 | 13.5 | 136.2 / 168.9 | 0.11 | 0.86 / 0.55 |
+
+The talker step (one 80 ms codec frame per step, including the 15 code-predictor sub-steps) costs 136–250 ms, i.e. 1.7–3.1× the frame budget, and stops scaling beyond 16 threads; inductor gives 6–22 %. Scheduler work is 0.1 ms per step and the shared-memory chunk hop about 1 ms per chunk, together under 2 % of the budget. The earlier `cpu_smoke` run (12+4 threads from the static overlay) is in the summary file (TTFA 292 ms, RTF 1.7). Conclusion unchanged from the study: this CPU class serves offline synthesis, not real-time streaming; a 4-core/AVX2 box (§8.5) is 12× slower than real time.
+
+### 8.5 Hardware emulation (WP4)
+
+`benchmarks/tts/hw_emulation.py` with `deploy_profile="auto"` and `VLLM_OMNI_HW_PROFILE` overrides (0.6B, 12 prompts × 1). CPU cases ran in the CPU venv pinned to N cores with `ATEN_CPU_CAPABILITY` downgrades; GPU cases ran on the L20X with the derived absolute KV budgets and eager rules of the emulated class (the card's memory is not capped, so peak GPU MiB shows what the budget admits, not a hard limit).
+
+| case | class chosen | dtype / threads or budget | init s | TTFA p50 ms | playback-start p50 ms | RTF | peak RSS GiB / GPU MiB |
+|---|---|---|---|---|---|---|---|
+| x86_cpu_avx2 (4 cores, 8 GiB) | x86_cpu | fp32, 3+1 threads, KV 1.2 GiB | 120 | 2651 | 54419 | 11.9 | 12.8 / – |
+| x86_cpu_avx512 (8 cores, 16 GiB) | x86_cpu | 6+2 threads | 109 | 679 | 7874 | 2.45 | 14.4 / – |
+| x86_cpu_amx (16 cores, 32 GiB) | x86_cpu | bf16, 12+4 threads | 116 | 513 | 5692 | 1.72 | 23.0 / – |
+| arm64_cpu_logic (RK3588-like, logic only) | arm64_cpu | fp32, 3+1 threads, KV 2.2 GiB | 108 | 2183 | 50262 | 11.9 | 13.0 / – |
+| jetson_orin_8g (unified 8 GiB) | jetson | eager both, absolute KV budget | 129 | 181 | 196 | 0.57 | – / 6236 |
+| jetson_orin_32g (unified 32 GiB) | jetson | graphs on | 183 | 68 | 68 | 0.09 | – / 23376 |
+| cuda_discrete_8g (8 GiB VRAM) | cuda_discrete | eager + small KV | 189 | 47 | 47 | 0.09 | – / 9680 |
+
+The derivation never allowed preemption (checked by `tests/core/test_edge_scheduling.py` against the exact talker KV geometry, 112 KiB per token). The Jetson-8 GB emulation shows the cost of running eager on a small budget (RTF 0.57, TTFA 181 ms) and is the case where the S3 native path should be compared on real hardware.
+
+### 8.6 Codec-token streaming: cadence and parity (WP5)
+
+`benchmarks/tts/codec_stream_e2e.py` launches the talker-only server (eager stage 0), streams codec frames to the client decoder (CPU), then launches the full two-stage server (eager stage 0, same seed) for the reference PCM, and reports client TTFA and the length-trimmed SNR. Four runs were needed:
+
+| run | outcome | root cause / fix |
+|---|---|---|
+| chain 1 (20260909-151221) | client crashed in `batched_chunked_decode` | the decoder takes a `[B, Q, F]` tensor, not a list; fixed in `536866b9` and verified against the real 0.6B decoder on CPU |
+| chain 2 (20260909-154659) | frames decoded, but all codec frames arrived in one message ≈2.1 s after the request (client TTFA 2.2–2.7 s = whole generation), SNR −3 dB | see below |
+| chain 4 (20260909-160935) | same, with counters: **1 engine output per request, first codes at output #1** | the speech path forced `FINAL_ONLY` outputs for Qwen3-TTS deployments without async chunks (`qwen3_full_payload`), which a talker-only deployment also is; fixed in `b22dabe2` (talker-only = final stage with latent output keeps DELTA) |
+| chain 5 (20260909-162326) | server now emits per step (2 outputs by step 2) but failed: "inhomogeneous shape" in row alignment | in DELTA mode the payload is a list of per-step tensors; normalized in `380e8022` |
+| chain 6 (20260909-163326) | 53 engine outputs for 52 rows, yet the client still got one message | each per-step payload is the *newest row only*; the alignment treated it as a cumulative snapshot (one yield at row 1, one at the end); fixed in `c893b638` (`accumulate_codec_rows`, unit-tested with per-step deltas) |
+| chain 7 (20260909-164759) | **streams per step**: client TTFA 152–156 ms, playback-start = TTFA, stall 0, chunks 2/4/8/16/25 frames arriving 145 / 285 / 570 / 910 ms apart (≈36 ms per frame from the eager talker-only server + CPU decode), client RTF 0.5–0.6, first utterance 15 s (server warm-up) | – |
+
+Parity was separated into two questions with `benchmarks/tts/codec_parity_diag.py` on the chain-4 codes (client now saves the received rows): (1) client chunked decode vs the exact non-chunked decode of the same codes: **72.7–75.9 dB SNR, correlation 0.99999999** for all four utterances, so the client-side decoder (ramp schedule, 72-frame context, stateful caches) reproduces the server's `Code2Wav` math; (2) the full server's PCM vs the exact decode of the client's codes: −3 dB, correlation ≈ 0 with identical frame counts, i.e. the two server processes sampled different code sequences for the same text and seed (the `seed` request field reaches both paths, so the divergence is numeric between the talker-only and the two-stage deployments; the speech API exposes no temperature, so a greedy comparison was not possible). The ≥40 dB end-to-end acceptance is therefore **not demonstrated**; decoder parity is, and a token-level tap on the two-stage server (or a greedy sampling switch) is the follow-up. The wire cost is unchanged from the design (16 int16 codebooks + 7-byte header per 80 ms frame ≈ 490 B/s). `tests/e2e/online_serving/test_qwen3_tts_codec_stream.py` passed in chain 2 (frames + decode), before the cadence fixes; it does not assert cadence.
+
+### 8.7 Per-step attribution on GPU (WP8) and the CUDA-IPC connector hop (WP7)
+
+`stream_latency_bench.py --step-stats` (0.6B, 12 prompts × 3, chains 2 and 3). Counters per process role; share = mean / 80 ms frame budget (one talker step = one frame).
+
+| counter | default deploy (shm) | edge profile | cuda_ipc deploy |
+|---|---|---|---|
+| stage 0 engine-core step (talker + code predictor) | 6.43 ms (p95 10.8) = 8.0 % | 5.96 ms = 7.4 % | 5.79 ms = 7.2 % |
+| stage 0 `schedule()` + `update_from_output()` | 0.12 + 0.12 ms = 0.3 % | 0.09 + 0.09 ms = 0.2 % | 0.09 + 0.08 ms = 0.2 % |
+| stage 1 engine-core step (code2wav, incl. polling steps) | 0.54 ms | 0.38 ms | 0.35 ms |
+| chunk hop: enqueue→put / put / get | 0.62 / 0.48 / 0.68 ms | 0.33 / 0.23 / 0.34 ms | 0.34 / 0.29 / 0.36 ms |
+| chunk hop put→get (lock-file mtime → get, chain 3) | 2.42 ms (p95 2.58) = 3.0 % | – | 2.90 ms (p95 10.7) = 3.6 % |
+
+Everything outside the model step (scheduler, transport, polling) is below 2 % of the frame budget per stage, and the whole hop from talker put to code2wav get is 2.4 ms with shared memory. CUDA-IPC does not improve it (2.9 ms mean, worse tail): on this edge the chunk tensor is already a small host tensor (codes), so the IPC export/ack adds work without removing a copy. **Decision: `qwen3_tts_cuda_ipc.yaml` stays optional; the edge profile keeps `SharedMemoryConnector`.** The orchestrator dispatch counter only fires under the event-driven orchestrator loop (`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`); the default poll loop was used here, so it is absent from the tables.
+
+### 8.8 Decoder export (WP4, phone/NPU class)
+
+`python -m vllm_omni.edge.decoder_export --model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --ramp 2 25` (CPU, no GPU): two `torch.export` programs (window 74 and 97 frames), exact match against the eager decoder, 460 MB fp32 each (manifest committed as `benchmarks/tts/edge_results/decoder_export_manifest_0.6B_torch_export.json`). ExecuTorch lowering not executed (package not installed; install is permission-gated).
+
+### 8.9 Job records and cleanup
+
+See the `gpu report --days 7` extract in [experiments/edge_branch/gpu_report.txt](experiments/edge_branch/gpu_report.txt) (all jobs of this section are the `tzhouam(claude): WP…` notes, type RUN, one card each, none overlapping). `gpu status --no-schedule` showed no reservation by `zhoutaichang` after the last chain; the worktrees `vllm-omni-run*` were removed; venvs stay outside the tree.

docs/analysis/implementation_roadmap.md

@@ -0,0 +1,90 @@
+# Implementation roadmap
+
+Phased plan for the designs in [tts_edge_design.md](tts_edge_design.md) and [vla_edge_design.md](vla_edge_design.md), following the strategy decision in [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) (S4 hybrid first for TTS, S3 lightweight orchestration as the main path, S5 edge-native numbers at every phase, S1 on Jetson as the reference tier). Each phase has entry conditions, deliverables, acceptance criteria, and the reproducible benchmark it must publish. Nothing below has been executed except the Phase 0 items marked "done" in [experiments.md](experiments.md).
+
+## Phase 0 — Reference numbers and toolchain gates (1–2 weeks)
+
+Entry: this analysis. No device hardware required for the first half.
+
+| Item | Status | Acceptance |
+|---|---|---|
+| Server reference for Qwen3-TTS 1.7B streaming (TTFA, cadence, RTF) | **done** on L20X (p50 TTFA 27.7 ms, RTF ≈ 0.08–0.10) | Numbers and JSON archived; re-runnable with the recorded command |
+| CPU edge-native baseline with llama.cpp (build, convert, quantize, bench, VLM run) | **done** on x86 (Q4_0 decode 135.6 tok/s @ 4 threads; SigLIP tile 0.6–1.2 s @ 8 threads) | Same |
+| Qwen3-TTS 0.6B server reference | pending (needs a driver that overrides the example's hard-coded 1.7B ids; no source change) | TTFA/RTF for 0.6B on the same prompts |
+| llama.cpp `llama-tts` Qwen3-TTS on x86 CPU (needs the GGUF conversion of the cached HF checkpoint; check whether `conversion/` supports it at this commit) | pending | WAV produced; RTF at 4/8 threads; codebook-0 agreement with the server under greedy sampling |
+| MNN build with `MNN_BUILD_LLM` + `llmexport.py` of Qwen3-TTS 0.6B + `qwen3_tts_demo` on x86 CPU | pending (requires its Python export dependencies; permission to install) | Same metrics as above |
+| ExecuTorch install + export of the Qwen3-TTS decoder (`modeling_qwen3_tts_tokenizer_v2.py`) to `.pte` on XNNPACK | pending (dependency installation; submodules) | Decoder PCM SNR vs torch ≥ 40 dB on recorded codes [Proposal threshold] |
+| VLA reference: π0 or GR00T on the L20X through the OpenPI endpoint with recorded observations | pending (weights not cached; download needs permission) | Action chunks for ≥ 100 recorded observations saved as the parity oracle |
+| Device procurement / access: one ARM Linux board, one Android phone with Qualcomm NPU, one Apple Silicon device, one Jetson | pending | Devices enumerated with SoC, RAM, OS, and thermal envelope |
+| **Checkpoint/task parity gate**: pin one Qwen3-TTS checkpoint and task per engine (server benchmark used CustomVoice 1.7B; llama.cpp README shows 1.7B Base; MNN demo needs reference audio and hard-codes `hiddenSize = 1024`) | pending | A table stating, per engine, which size/task variant is confirmed to run, with the exact converted artifact; comparisons in later phases use only matching rows |
+| **Playback-gap measurement**: rerun the server driver with `codec_chunk_ramp` / `codec_chunk_adaptive` enabled and report uninterrupted-playback latency alongside TTFA | pending (cheap, same GPU procedure) | Both metrics in the JSON schema; the default 1→25 schedule's ≈80 ms gap quantified |
+
+Exit criterion: for each runtime kept in scope, a cached-model artifact and a one-command benchmark script exist and run on x86, so device runs in Phase 1 are only a change of host.
+
+## Phase 1 — TTS hybrid (S4): server talker, on-device decoder (2–4 weeks)
+
+Deliverables:
+
+1. vLLM-Omni side: a codec-token streaming mode. This is a new wire contract, not only an adapter: `serving_speech_stream.py`'s `_generate_and_send` consumes PCM from the full pipeline today, so the deliverable is a talker-only deploy profile plus a stream message type carrying `codes.audio` frames (16 × int16 per frame) with defined frame ordering, EOS, cancel, and decoder-state/revision semantics (integration points in [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) §5). This is the only proposed change inside vLLM-Omni and must go through its review process (`review-pr`/`precheck-pr` conventions in the repo). **Gate**: the hybrid-first ordering holds only if this prototype and the on-device decoder measurement (item 2) both pass; otherwise Phase 2 proceeds first.
+2. Device side: `IDecoderAdapter` implementations on llama.cpp (from the mtmd code2wav graph) and ExecuTorch (`.pte` from Phase 0), driven by the same chunk schedule as the server reference (ramp/adaptive from the deploy YAML, 72-frame left context; 1→25 kept as the measured baseline) and `reset()` on abort.
+3. A small client that plays audio from the codec stream with a 2 s ring buffer and supports interruption.
+
+Acceptance:
+
+- TTFA (client-observed, including network) ≤ server TTFA under the same chunk schedule + RTT + first-chunk decode time; **uninterrupted-playback latency** (earliest start with no later underrun, using the ramped chunk schedule) reported alongside; inter-chunk gap p95 ≤ 100 ms on the device decoder for 25-frame chunks (else reduce steady chunk size).
+- Decoder parity: PCM SNR ≥ 40 dB against the server decoder on identical codes; ASR WER on the 12 benchmark prompts within 1 point of server audio.
+- Interruption: audio stops within one chunk length (≤ 2 s worst case, target ≤ 200 ms with a 2-frame ring) and the next utterance starts cleanly (no residual decoder state, verified by bit-identical output to a cold decode).
+
+Benchmark: `P-TTS-1` and `P-TTS-2` from [experiments.md](experiments.md) §5, run on device and on x86, published as JSON in the same schema as `bench_results.json`.
+
+## Phase 2 — TTS fully on device (S3) on llama.cpp and MNN (4–6 weeks)
+
+Deliverables:
+
+1. Edge TTS orchestrator (C++) implementing the loop in [tts_edge_design.md](tts_edge_design.md) §2.2 over `ITalkerAdapter`/`IDecoderAdapter`/`ISpeakerAdapter`.
+2. llama.cpp binding: talker + predictor via the existing Qwen3-TTS mtmd graphs, chunked decode, abort check per step; Android JNI and Apple xcframework builds.
+3. MNN binding: chunked `speech_decoder` calls with carried state (pattern from the Qwen2.5-Omni streaming path), talker at W4; Android app integration (`MnnLlmChat` has the host UI), QNN delegation of the decoder as an experiment.
+4. Parity harness against vLLM-Omni offline outputs (greedy) for talker codes and decoder PCM.
+
+Acceptance:
+
+- 0.6B path: RTF ≤ 0.5 on the ARM board at 4 threads and on the phone's big cores; TTFA ≤ 300 ms; resident memory ≤ 2 GB; 10-minute continuous synthesis without thermal throttling below the RTF gate (log SoC temperature).
+- 1.7B path: RTF ≤ 1.0 on Apple Silicon Metal and on the phone GPU/NPU; report, do not gate, on CPU.
+- Codebook-0 greedy agreement ≥ 95 % on the first 25 frames of each prompt (differences after the first divergence are expected; compare prefix only) [Proposal metric].
+- Interruption and backpressure behave as specified (tests with abort at frame 1, mid-chunk, and after EOS).
+
+Benchmark: `P-TTS-2` plus the interruption test suite; results per device in a table with the same columns as the server run.
+
+## Phase 3 — ExecuTorch TTS path for CoreML/QNN and Jetson reference (4 weeks, parallel with Phase 2)
+
+Deliverables: talker export via `export_llm` (Qwen3 config, 8da4w), predictor and decoder as separate programs, a runner modelled on the Voxtral TTS runner; CoreML and QNN partitioner recipes; S1 reference run of vLLM-Omni on a Jetson (install, cold-start time, TTFA/RTF) for the same prompts.
+
+Acceptance: same gates as Phase 2 on the ExecuTorch path; the Jetson S1 run documents cold start and memory (expected to fail the cold-start gate; recorded as the reference tier, not as a product path).
+
+## Phase 4 — VLA parity and simulation (6–8 weeks)
+
+Deliverables:
+
+1. Exports of π0 (plain torch, CPU parity test exists in the repo) and GR00T N1.7 (HF wrapper) into ExecuTorch programs: `vision_encoder`, `fusion`, `action_head`; MNN exports of the same via ONNX for Android NPU experiments.
+2. `IVlaRunner` implementations and the controller/safety split from [vla_edge_design.md](vla_edge_design.md) §2.1, with the chunk buffer, staleness check, and watchdog.
+3. Parity suite: recorded observations → action chunks compared with the OpenPI server (max-abs, per-dimension RMS) at K = 4 (GR00T) and K = 10 (π0).
+4. Simulation closed loop (external simulator, LIBERO-style tasks as available for the checkpoints) with the edge runner in the loop.
+
+Acceptance:
+
+- Parity: with the flow noise passed explicitly (π0 `sample_actions(noise=...)`) so both sides are deterministic, per-dimension RMS error ≤ 2 % of the action range at fp16 weights, ≤ 5 % at 8da4w [Proposal thresholds, to be revisited after the first measurements].
+- Latency on device: inference wall time p50 ≤ 250 ms and p95 ≤ 400 ms for one 224² camera on the phone NPU or Apple ANE; report the three-camera case; chunk age never exceeds the staleness bound in a 10-minute run [Proposal].
+- Simulation success rate within 5 points of the server policy over ≥ 50 episodes per task.
+- Fault injection tests pass (delayed frames, dropped camera, CPU frequency cap) with the safety layer engaging as specified.
+
+Benchmark: `P-VLA-1` from [experiments.md](experiments.md) §5 for perception cost, plus the parity and simulation suites; publish per-stage ms, jitter, memory, temperature.
+
+## Phase 5 — Hardware-in-the-loop and hybrid re-planning (open-ended)
+
+Only after Phase 4 passes: hardware-in-the-loop with an independent safety layer; optional cloud re-planning via vLLM-Omni for task text updates; long-duration soak (thermal, memory growth) on every target.
+
+## Cross-cutting
+
+- **Repository conventions**: any change proposed to vLLM-Omni or vLLM follows their `AGENTS.md` (uv environments, human-owned PRs, disclosure of AI assistance); llama.cpp explicitly forbids autonomous agent contributions, so llama.cpp work stays in a private fork or is authored by a human maintainer of this project.
+- **Reproducibility**: every benchmark is a script plus a JSON output plus the exact commit of each engine; device runs record SoC, OS, thermal state, and governor settings.
+- **Permission gates** (per the task's rules): dependency installs (ExecuTorch, MNN export deps), model downloads (π0/GR00T/DreamZero weights), and any source change in the six checkouts require explicit approval before they are executed.

docs/analysis/minicpmo_edge_profile.md

@@ -0,0 +1,119 @@
+# MiniCPM-o 4.5 on the edge: measured profile (2026-09-11)
+
+Companion to [mobile_htp_simulation.md](mobile_htp_simulation.md) (Qwen3-TTS) using the same instruments: vLLM-Omni's
+StepStats for per-stage attribution, the Xeon 8480C for CPU, one H200 through `canhazgpu` for GPU, and Qualcomm AI Hub
+for real Snapdragon devices. Scripts and raw results: `experiments/edge_engine_compare/minicpm/` and
+`experiments/edge_engine_compare/results/minicpmo/`.
+
+## 1. What this model is, and why the shape matters
+
+MiniCPM-o 4.5 is a three-stage pipeline in vLLM-Omni (`MINICPMO_4_5_PIPELINE`): an 8 B omni LLM (SigLIP vision tower +
+Whisper audio encoder + text backbone) → a TTS head → Code2Wav. Weights are local (19 GB fp16, ≈9.5 B parameters).
+
+The speech half is much lighter than the thinker, and much lighter than Qwen3-TTS's:
+
+| component | parameters | rate |
+|---|---|---|
+| TTS head (Llama-style decoder, 20 layers × 768, 12 heads, no GQA, no q/k-norm) | 194 M | 1 token per 40 ms (25 Hz) |
+| Code2Wav flow matching (CFM DiT, Step-Audio2 / CosyVoice2 lineage) | 156 M | 10 denoising steps per chunk |
+| Code2Wav HiFT generator | 21 M | 1 chunk = 25 tokens = 1.0 s of audio |
+| thinker (vision + audio + text, 8 B) | ≈9.2 B | prompt-rate |
+
+**`num_vq = 1`**: one codebook, so one decode step per speech token. Qwen3-TTS emits 16 codebooks at 12.5 Hz, which
+costs one talker step *plus fifteen* code-predictor sub-steps per 80 ms frame. That single architectural choice is worth
+3.5× on a phone (§4). The vocoder is also genuinely streaming already — 3 tokens of lookahead, persistent flow and HiFT
+caches — which is the design Qwen3-TTS's 72-frame-context vocoder still lacks.
+
+## 2. GPU (one H200, 6 concurrent requests, text → speech)
+
+Deploy `vllm_omni/deploy/minicpmo_4_5.yaml` (all three stages on one card), 6 prompts from `benchmarks/tts/prompts_12.txt`,
+`max_tokens` 192 for the thinker and the stage-1 deploy defaults for the TTS head.
+
+| metric | value |
+|---|---|
+| engine init (warm caches) | 253 s |
+| first text | p50 393 ms, p90 598 ms |
+| complete audio per request | p50 5.81 s, p90 7.60 s (offline path yields the whole waveform, so this is completion, not first-chunk latency) |
+| batch throughput | 30.2 s of audio in 22.6 s wall → **RTF 0.748** at concurrency 6 |
+
+Per-stage step attribution (mean over the run):
+
+| stage | steps | mean step | mean model forward |
+|---|---|---|---|
+| 0 · thinker | 46 | 9.66 ms | 2.31 ms |
+| 1 · TTS head | 274 | 6.53 ms | 1.72 ms |
+| 2 · Code2Wav | 469 | 14.88 ms | – |
+
+The largest single number in the table is not compute: stage 2's `hop.put_to_get` averages 3.1 s, i.e. codec chunks wait
+in the shared-memory connector while the vocoder works through its backlog. On one card the vocoder is the queue.
+
+## 3. CPU (Xeon 8480C, pinned)
+
+| stage | 8 threads | 16 threads | real-time factor at 16 threads |
+|---|---|---|---|
+| TTS head, per 40 ms token | 17.1 ms | 11.6 ms | **0.29** |
+| Code2Wav, per 1.0 s of audio | 1033 ms | 833 ms | **0.83** |
+| one-time speaker setup (x-vector + prompt mel + caches) | 11.7 s | 9.9 s | cacheable per voice |
+
+Inside the vocoder chunk the split is lopsided, and in the opposite direction from Qwen3-TTS:
+
+| half | per 1.0 s of audio, 16 threads | share |
+|---|---|---|
+| flow matching (10-step CFM DiT) | 652 ms | **91 %** |
+| HiFT generator | 65 ms | 9 % |
+
+Qwen3-TTS's vocoder was elementwise-bound (76 % in Snake activations on sample-rate tensors) and therefore wanted a GPU;
+MiniCPM-o's is matmul-bound in a DiT and therefore suits an NPU. The two models need opposite placements.
+
+The step count is a config field (`tts_config.s3_stream_n_timesteps`), and it trades linearly:
+
+| flow steps | per 1.0 s of audio | mel fidelity vs the 10-step default |
+|---|---|---|
+| 10 (default) | 542 ms | reference |
+| 6 | 341 ms | **30.6 dB** |
+| 4 | 236 ms | 16.5 dB |
+| 2 | 127 ms | 9.6 dB |
+
+Six steps is a 1.6× vocoder speed-up at 30.6 dB on the mel spectrogram — worth a listening test, since it costs nothing
+but a config change.
+
+## 4. Real Snapdragon devices (Qualcomm AI Hub)
+
+The TTS head exported through `vllm_omni/edge/qnn_export.py` (Llama-style path: `qk_norm=False`, weight-normalised code
+head), 256-token KV cache, QNN context binary, every operator on the Hexagon NPU.
+
+| build | Galaxy S25 (8 Elite) | Galaxy S24 (8 Gen 3) |
+|---|---|---|
+| fp16, per 40 ms token | **7.08 ms** | **8.18 ms** |
+| fp16 fidelity vs torch fp32 | 48.9 dB, top-1 token matches | 49.2 dB, top-1 matches |
+| int8 weights (w8a16), per token | 4.88 ms | 6.15 ms |
+| int8 fidelity | **2.3 dB, top-1 differs** | **0.1 dB, top-1 differs** |
+
+Per second of speech, the comparison with Qwen3-TTS on the same phone is the headline:
+
+| model, S25 NPU fp16 | per second of audio |
+|---|---|
+| **MiniCPM-o 4.5** (1 codebook @ 25 Hz, 194 M) | **177 ms** |
+| Qwen3-TTS 0.6B (16 codebooks @ 12.5 Hz: talker 15.8 + predictor 33.9 per 80 ms frame) | 621 ms |
+
+**Int8 fails here too.** Per-tensor int8 weights destroy MiniCPM-o's code head exactly as they destroy Qwen3-TTS's code
+predictor (3.1 dB there, 2.3 dB here) — two unrelated families, same failure, so this is a property of speech-token heads
+rather than of one model or one runtime. The faster int8 rows stay timing-only.
+
+Not yet measured on device: the Code2Wav flow + HiFT (its streaming caches need the same flattening the KV cache needed),
+and the 8 B thinker, which at fp16 is 16 GB and does not fit a phone at all — int4 would be mandatory, and §4 is the
+reason to expect trouble there.
+
+## 5. Reading
+
+- **For a phone, MiniCPM-o's speech stack is the better architecture** and Qwen3-TTS's is the better size. One codebook at
+  25 Hz costs 177 ms of NPU time per second of audio against 621 ms; but the thinker in front of it is 8 B, where
+  Qwen3-TTS's whole pipeline is 0.6 B.
+- **Placement is model-specific.** Qwen3-TTS wants transformer-on-NPU + vocoder-on-GPU. MiniCPM-o's vocoder is a DiT, so
+  the NPU is plausible for both halves; that is the next measurement.
+- **The cheapest real win is the flow step count**, not quantization: 10 → 6 steps is 1.6× at 30.6 dB, while every int8
+  path measured in this study has been numerically broken.
+- Two bugs found while profiling, both in the offline path rather than the model: `examples/offline_inference/minicpmo/end2end.py`
+  builds two sampling-parameter sets and fails on this three-stage pipeline with `Expected 3 sampling params, got 2`, and
+  its talker parameters cap the codec LM at `max_tokens=1`, which silently yields zero audio. The driver in
+  `experiments/edge_engine_compare/minicpm/bench_minicpmo.py` passes three sets and the stage-1 deploy defaults.

docs/analysis/optimization_plan.md

@@ -0,0 +1,585 @@
+# Optimization plan: inference and harness
+
+Written 2026-09-12 against [spark_edge_support.md](spark_edge_support.md), then
+revised after an independent review by `gpt-6-astra` through the Codex hook
+(read-only against these checkouts; the sandbox could not start because user
+namespaces are disabled on this host, so it ran unsandboxed under explicit
+read-only instructions and the checkouts were verified clean afterwards — the
+same procedure recorded in [README.md](README.md) for the 2026-09-08 study).
+
+The review rejected the first draft's ordering and several of its expected
+gains. Most of that was correct and is applied below; §"Review record" lists
+what I accepted, what I corrected with new measurements, and why.
+
+Every item starts from a measurement in this tree, states the expected gain,
+and says how it would be verified — because on this host a claim without a
+verification method is how three earlier conclusions turned out to be wrong.
+**A fourth is now on the list**: the first draft of this plan mis-described the
+740 MB it proposed to remove.
+
+## Where the time actually goes
+
+W4A16 g128 decode, 24 threads, per-op profile (reproducible to ~5%, where
+throughput swings ~30%). The first draft built this table from two different
+builds — 5.91 ms is the **W4A8** GEMM, 6.85 ms the W4A16 one
+([support:193](spark_edge_support.md)) — so the step it described (11.8 ms) was
+neither configuration. The W4A16 step is **12.51 ms**:
+
+| | ms | share of 12.51 |
+|---|---|---|
+| int4 GEMM (tinygemm) | 6.85 | 55% |
+| inductor elementwise (norms, rope, GEGLU, residuals) | 2.20 | 18% |
+| vLLM attention ops (4 per layer × 28) | 1.31 | 10% |
+| custom-op dispatch boundary (W4A16 only) | 0.55 | 4% |
+| unattributed remainder (sampling, view/slice churn) | 1.60 | 13% |
+
+Two corrections to how this table was read. The GEMM runs near this socket's
+achievable bandwidth for *its own* shape, which says the GEMM is memory-bound;
+it does **not** say the other 45% is framework overhead — attention, softmax,
+norms and sampling contain real arithmetic. And the per-step savings of W4A8
+over W4A16 do not add: 0.94 ms (GEMM) + 0.48 ms (dispatch) = 1.42 ms predicted
+against 12.51 − 11.55 = **0.96 ms** measured. Something offsets them, and no
+item below may assume its saving reaches the step time unchallenged.
+
+## Where the memory actually goes
+
+Anonymous RSS by phase ([support:324](spark_edge_support.md)):
+
+| phase | W4A8 (GPTQ) | W4A16 (tinygemm) |
+|---|---|---|
+| before import | 12 MB | 12 MB |
+| after `import vllm_omni` | 513 MB | 515 MB |
+| after engine construction | **7338 MB** | 5023 MB |
+| after releasing free arenas | 3510 MB | 3411 MB |
+
+The first draft treated the post-release figure as the number to reduce. That
+is the wrong target for an edge device: **7338 MB is the peak that has to fit**,
+and `release_after_load()` runs only after `load_model` and warm-up have both
+returned ([patch.py:730](../vllm-omni/vllm_omni/patch.py)). A device that cannot
+survive the load never reaches the release. This reordered the plan.
+
+The draft's weight breakdown ("0.86 GB quantized + 0.54 GB bf16 embedding") is
+also wrong for the checkpoint actually measured: `spark2_5-int4-g128/config.json`
+sets `quantize_embedding: true` with `tie_word_embeddings: true`, so there is no
+bf16 embedding, and the persistent prefill duplicate was left out of the
+breakdown entirely. There is therefore no trustworthy ledger to subtract from —
+which is why the "unexplained 1.6 GB" item is now a prerequisite, not a
+long-shot.
+
+---
+
+## Inference
+
+### I0. Peak load-time memory — *now first*
+
+7338 MB peak vs 3510 MB steady is the gap that decides whether a device can run
+this at all. The peak comes from holding the checkpoint, the dequantized
+intermediate and the repacked layouts alive at once, per layer, with freed
+arena pages not yet returned.
+
+**Do**: (a) release arenas *during* the load, per layer or per N-layer block,
+rather than only after `load_model` returns; (b) repack blockwise so no
+whole-tensor bf16 intermediate exists; (c) as the durable fix, write the packed
+layout to a deployment artifact at build time so the device loads what it runs.
+
+**Expect**: peak toward the steady 3.4–3.5 GB. Unquantified deliberately — the
+mechanism is understood, the size is not.
+
+**Verify**: sample `RssAnon` on a timer through load (0.2% reproducible), report
+the maximum, not the endpoint. The harness measures endpoints today and would
+not have caught this.
+
+### I1. One packed weight layout, not two — *replaces the draft's I2*
+
+The W4A16 build keeps a second copy costing **740 MB** (3542 → 2802 MB) and
+buying **7.3× prefill** (3252 → 448 tok/s). The draft described this as "prefill
+unpacks to bf16" and proposed a tiled dequant kernel to avoid the bf16 copy.
+That was wrong about the artefact. `_finalize_int4` keeps `weight_halves` — a
+**second packed 4-bit copy** — plus cloned `weight_group_scale` and
+`weight_group_zero`; the bf16 matrix is already a per-call temporary and
+`weight_packed` is deleted
+([cpu_int4.py:344,375](../vllm-omni/vllm_omni/model_executor/layers/quantization/cpu_int4.py)).
+
+So the cheap fix is not a new GEMM. It is to **dequantize directly from the
+tinygemm layout** and delete `weight_halves` and the duplicated scale/zero,
+keeping the existing single `F.linear`. The only reason that was not the
+original design is that the tinygemm layout was believed opaque. It is not —
+measured on this host (`torch._convert_weight_to_int4pack_for_cpu`, AVX-512):
+
+- The packed buffer is `[N/64][K][32 bytes]`: **K is the middle axis**, so a
+  K-range is contiguous *within each n-block of 64*.
+- Reassembling a K-tile from those per-n-block segments is **byte-identical** to
+  packing that K-slice directly (verified exactly at N=2048, K=2048 for tile
+  widths 128/256/512), and split-K through tinygemm reconstructs the single-call
+  result at **51.5 dB**.
+- The lane order inside a 32-byte run is a **fixed 64-entry interleave** — low
+  nibble = output channel *d*, high nibble = *d+32* — identical across every k
+  and every n-block. So decoding is a deinterleave, not a gather or a
+  reverse-engineered ISA table.
+
+**Do**: point the existing dequantizer at the tinygemm layout via that
+deinterleave; drop `weight_halves`, `weight_group_scale`, `weight_group_zero`.
+
+**Expect**: most of the 740 MB back **at unchanged prefill**, because the GEMM
+shape and reduction order do not change — only where the nibbles are read from.
+The largest single temporary stays one dequantized `gate_up` (N=13312, K=2048,
+bf16 = **52 MiB**), which already exists today.
+
+**Verify**: `RssAnon` (0.2%) plus pp512, three interleaved passes, and an SNR
+assertion against the current dequant path — the check that caught the original
+layout bug.
+
+**Risk**: tails. N not divisible by 64 or K odd need the scalar path; the
+deinterleave rule above is verified only for full blocks.
+
+**Not doing first**: the draft's K-tiled dequant-and-accumulate. Even granting
+the layout, calling oneDNN once per K-tile means the M×N accumulator is re-read
+and re-written per tile — for `gate_up` at M=512 that is ~26 MiB of fp32
+accumulator × 15 extra tiles ≈ 780 MiB per layer per prefill, which is not the
+"few MB of scratch" the draft claimed. Avoiding that traffic means keeping the
+accumulator panel in registers across the K loop, i.e. writing a blocked GEMM
+microkernel rather than calling a library per tile. If temporaries still need
+cutting after the item above, **N-blocking is the better experiment** — disjoint
+output columns need no cross-tile accumulation at all.
+
+### I2. Measure the native non-AMX 4-bit path before building anything
+
+`weight_path.py:14` calls tinygemm "the only option" without AMX. That is false:
+when `layer.use_w4a8` is off, vLLM 0.28 falls through to `ops.cpu_gemm_wna16`
+([cpu.py:202](../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py)),
+a vector-ISA W4A16 kernel that has never been measured here. Both the ARM story
+and the "fidelity" weight path rest on an untested assertion.
+
+**Do**: benchmark `cpu_gemm_wna16` against tinygemm at Spark's shapes, decode
+and prefill. **Verify**: `bench_int4_kernels.py`. **Then** fix `weight_path.py`.
+
+This is a day's work that could invalidate a kernel project, so it comes first
+among the kernel items.
+
+### I3. Asymmetric W4A8 — a fidelity experiment already supported by the kernel
+
+The report treats W4A8's zero point pinned at 8 as a kernel limitation. It is
+not: `_process_gptq_weights_w4a8` uses the checkpoint's zero points whenever
+`config.zero_points` is set, and synthesizes the constant only otherwise
+([cpu.py:98](../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py)).
+Our exporter chooses `"sym": true`
+([quantize_gptq_int4.py:102](experiments/spark_edge/quantize_gptq_int4.py)).
+W4A8 is the fastest path and the worst on task completion (1/12 greedy; the
+agent navigates correctly then fails to stop). An asymmetric export is a
+config change, not a kernel project.
+
+**Do**: re-export with `sym: false` and zero points, re-run fidelity and the
+agent suite. **Expect**: unknown, but it is the cheapest fidelity experiment
+available and it tests the *stated* cause of W4A8's error.
+
+### I4. A better quantizer — only after I3, and on an honest bit budget
+
+Ours reproduces 2/12 greedy prompts; llama.cpp Q4_K_M manages 4/12. The draft
+called this "the same bit budget". It is not. Ours is **4.25 bits/weight**
+(4 + two bf16 per 128-weight group; `weight_path.py` says 4.16, which is wrong);
+plain Q4_K is **4.5 bits** over groups of 32, and Q4_K_M additionally promotes
+some tensors to higher-bit types. Group size, per-tensor type selection and
+rounding are all confounded, so "6-bit sub-block scales cause the gap" is an
+attribution the experiment never isolated.
+
+**Do**: first close the budget gap cheaply (g64 and/or fp16 scale+zero, which is
+a config change) and re-measure; only then implement error-compensated rounding
+(GPTQ-style, calibration data exists) or AWQ-style scaling.
+
+**Verify**: greedy divergence *and* the scored agent suite — but only after H1
+below, because today's suite cannot rank builds.
+
+**Risk**: may produce nothing. Timebox it; a null result is worth recording.
+
+### I5. Attention op-chain fusion — worthwhile, but not "highest confidence"
+
+Four ops per layer, 112 dispatches, **1.3085 ms/step** measured
+([profile_w4a16_g128_p2.json](experiments/spark_edge/profile_w4a16_g128_p2.json)):
+`unified_attention_with_output` 0.4379, `unified_kv_cache_update` 0.3948,
+`cpu_attention_with_kv_cache` 0.3471, `cpu_attn_reshape_and_cache` 0.1287.
+
+These are **two wrapper/kernel pairs**, not four independent kernels. Wrapper
+self time is 0.833 ms — 64% of the total — so overhead does dominate at this
+shape: at batch 1 the cache-update OpenMP loop runs over tokens × KV heads, i.e.
+**two iterations**, while both paths independently rebuild cache views and the
+wrapper allocates a dummy tensor for compile ordering.
+
+But the draft's supporting arithmetic was about a workload it never profiled.
+"28 MB moved" assumes every layer attends 512 tokens; 7 of 28 layers are full
+attention, and `profile_cpu_step.py` profiles a short prompt generating 64
+tokens — nowhere near a filled window.
+
+**Expect**: revised down. Eliminating *all* wrapper time saves 0.833 ms, and
+some wrapper is required, so the draft's 0.6–0.9 ms demanded savings from the
+C++ bodies too. Target **0.3–0.5 ms** (2–4% of step) from consolidating context
+and cache-view handling through the existing `forward_includes_kv_cache_update`
+interface, plus a serial path for the two-iteration cache update.
+
+**Verify**: per-op profile **and** unprofiled step latency — one outer call that
+still launches two threaded kernels removes neither thread launch, and fewer
+profiler events alone make this look better than it is.
+
+### I6. Import cost — an upper bound, not a saving
+
+`import vllm_omni` costs 19.6 s against `import vllm`'s 5.2 s; auto
+image-processing (4.6 s) and the `openai` client (1.3 s) are identifiable inside
+it. The draft promised 4–6 s per process. That 5.9 s is an **upper bound on
+removable import work**, not a demonstrated reduction in time-to-ready: deferred
+imports often just move the cost later. The init timer starts *after* imports
+([bench_init_memory.py:44](experiments/spark_edge/bench_init_memory.py)), so the
+current harness cannot measure this at all. And `patch.py` uses `ModelConfig` at
+import time to install a descriptor patch
+([patch.py:54](../vllm-omni/vllm_omni/patch.py)) — deferring it means changing
+how that patch installs, not moving a statement.
+
+**Do**: first make the harness measure **process start → ready**; then defer.
+**Verify**: that end-to-end number, with `-X importtime` as the diagnostic.
+
+### I7. A real memory ledger
+
+Prerequisite for any claim about a remainder. The draft's tools do not do the
+job: `smaps_rollup` is an aggregate (use `smaps` per mapping), and `tracemalloc`
+does not attribute native tensor storage or allocator arenas.
+
+**Do**: count unique tensor storages (both embedding layouts included), then
+`smaps` by mapping for the rest. **Expect**: a decision about whether the
+remainder is reducible — the draft's "unexplained 1.6 GB" was a subtraction from
+a breakdown now known to be wrong.
+
+### I8. Resolve the `prefill_dequant: false` correctness hazard
+
+`cpu_int4.py:55` records that running tinygemm on a multi-row batch and then
+`cpu_attention_with_kv_cache` in the same forward dies with SIGILL — an AMX tile
+configuration bug — and sets `dequant_threshold: 2` to avoid it. But
+`prefill_dequant: false`, the 2802 MB build the report calls the smallest
+resident configuration, routes multi-row batches to exactly that path. Either
+the 448 tok/s measurement and the crash report disagree about build or shape, or
+that configuration is not deployable. **Do**: reconcile before quoting 2802 MB
+as a build anyone can use.
+
+### I10. Megakernel / persistent thread pool -- the biggest measured headroom
+
+Raised after the review. On GPU a megakernel (Hazy Research's low-latency
+Llama, Mirage's MPK) fuses a whole forward pass into one launch, removing
+per-kernel launch cost and the pipeline bubble where the next layer's weights
+cannot start loading. There are no kernel launches on CPU, so the analogues
+are **OpenMP fork/join per op**, **ATen dispatch**, and the same cross-op
+prefetch bubble.
+
+This is not a hypothetical for us: **llama.cpp already is one.**
+`ggml_threadpool` starts its threads once and they walk the entire graph
+together, synchronising at `ggml_barrier` through an atomic spin
+(`ggml-cpu.c:490,576`). vLLM enters a separate `omp parallel` region inside
+each kernel. So the comparison this whole study is built on is, in part, a
+comparison of execution models.
+
+Measured here, so the prize is sized before anything is built:
+
+* **Parallel-region cost: ~1.2 us** at 24 threads (a just-above-grain-size op,
+  24 threads vs 1). Spin-waiting does not help -- `OMP_WAIT_POLICY=ACTIVE` plus
+  `GOMP_SPINCOUNT=infinite` moved it 1.23 -> 1.18 us, inside noise -- so this
+  is barrier and scheduling cost, not thread wake-up, and the one-env-var
+  approximation of a persistent pool buys nothing.
+* **Regions per decode step: a few hundred.** 113 quantized GEMMs and 112
+  attention ops certainly parallelise; most elementwise tensors at batch 1 are
+  below PyTorch's grain size and run serially, so the 764 candidate calls in
+  the profile overstate it. At ~250-350 regions this is **0.3-0.4 ms, 2-3%** --
+  real, but the same money as I5, which is far cheaper.
+* **The bubble is the prize, not the barriers.** The GEMM sustains 60% of the
+  measured read ceiling and the whole step 33%. A persistent kernel that keeps
+  the weight stream running across op boundaries is the mechanism that could
+  close part of a **2.76 ms** gap.
+
+One of our own decisions works against this: `cpu_int4_linear` is deliberately
+opaque to dynamo so the batch-size branch never becomes a guard on vLLM's
+traced graph. That also stops inductor fusing anything across it. The guard
+stability was worth it, but it is a cost worth naming.
+
+**Do**: *not* build a megakernel next. The honest read is that the fork/join
+half is small and already covered by I5, and the streaming half needs the
+execution model replaced -- a persistent pool plus cross-op pipelining inside
+vLLM's CPU backend, fighting its attention backends, KV cache management and
+torch.compile integration. That is weeks, and it reimplements ggml.
+
+What *is* worth doing, in order, because each one tests the same thesis for a
+fraction of the cost:
+
+1. **Find out why the GEMM is at 60%.** It is not obviously bandwidth. The
+   tinygemm kernel dequantizes once per activation row, so part of that time is
+   arithmetic, and each of the 113 calls streams only a few MB -- too little for
+   the prefetcher to reach steady state. Attribute it before optimising it.
+2. **Measure `cpu_gemm_wna16`** (I2) and W4A8's 69% against the same ceiling.
+   If one kernel is materially closer, the answer is a better kernel, not a
+   different execution model.
+3. **I5**, which is the cheap slice of the fork/join half.
+
+**Verify**: per-op profile plus the streaming-bandwidth figure above, which is
+now the denominator any kernel claim should be quoted against.
+
+### I9. Explicitly not planned
+
+- **More kernel fusion.** Four attempts, four losses: vLLM's fused CPU RMSNorm
+  is 3.6–4.5× faster in isolation and slower in situ (inductor already fuses the
+  native form with the residual add); fused `gelu_and_mul` 2.7× slower at
+  Spark's shape; `cpp.min_chunk_size` −6% decode/−19% prefill; an inline gate
+  reduction −44%. I5 is op-*count* reduction, a different mechanism.
+- **MiniCPM-o CPU overlay.** Dropped. The 8 B thinker at bf16 is ~19 GB, so it
+  buys no edge deployability; and the draft's "three stages should beat the
+  two-stage 2.1×" is not how overlap works — the bound is ΣT/max(T), so stages
+  of 90/5/5 s give 1.11×. (That the measured two-stage ratio is 64.57/30.54 =
+  2.114× when the bound for two tasks is lower means something else changed too,
+  which the init work should explain before anyone quotes it.)
+- **Thread count and warm-up batch size as memory levers** — 64 MB and 2 MB.
+- **KV-cache sizing is *not* on this list any more.** The 188 MB that retired it
+  was measured generating 8 tokens. At 32 k context `kv_budget.py`'s own formula
+  gives 1792 MiB flat vs 469 MiB hybrid — a 1323 MiB difference. It was measured
+  at a context where no cache is touched, which is not evidence that it never
+  matters. (`CpuPlatform.support_hybrid_kv_cache()` returns `True` in vLLM 0.28,
+  so the report's flat accounting needs a configuration explanation too.)
+- **Speculative decoding** for tg128: it would inflate a fixed-prompt
+  `ignore_eos` benchmark without representing anything. Measure it on the agent
+  suite or not at all.
+
+---
+
+## Harness
+
+The review's sharpest finding is that the harness cannot currently support the
+comparisons the inference work needs. Correctness first, statistics second.
+
+### H1. Fix the scored suite before running it more
+
+- **The alarm oracle is wrong.** `add_alarm` passes if any typed string contains
+  `"7"` ([tasks.py:101](experiments/spark_edge/android_agent/tasks.py)) — not
+  7:30, not AM, not that an alarm was saved. Typing `7` anywhere scores a pass.
+- **The toggle task scores final state only**, so turning a setting off and back
+  on counts as success.
+- **The scored suite is simulator-only**: task builders and predicates are typed
+  to `SimulatedDevice`. The real-device *episode* path exists; real-device
+  scoring and reset do not.
+- **Repeats at temperature 0 over fixed fixtures measure repeatability, not
+  generalization** ([agent.py:48](experiments/spark_edge/android_agent/agent.py)).
+
+**Do**: correct the oracles; add paired task/layout/state variations so N runs
+are N trials; implement real-device scoring and reset.
+
+This supersedes the draft's "N≥5 runs per task" — more repetitions of a wrong
+oracle produce a confident wrong answer, which is the failure mode this project
+already has three instances of.
+
+### H2. Fix measurement semantics
+
+- `run_grid`'s docstring says it rotates arms; it iterates `for pass: for arm in
+  arms`, i.e. the **same order every pass**
+  ([bench_harness.py:160](experiments/spark_edge/bench_harness.py)). It
+  interleaves but does not rotate, so position bias survives.
+- `summarize` reports `best = max(values)` for every metric
+  ([bench_harness.py:123](experiments/spark_edge/bench_harness.py)) — wrong for
+  memory and init, where lower is better. Published "best" memory figures are
+  the worst pass.
+- `profile_cpu_step.py` wraps all of `generate` (prefill + sampling) and divides
+  by `tg`, while the wall-clock path subtracts independent best-of-three
+  measurements. These are different quantities presented as one.
+- Profile JSONs record no model path, context length or build; saved W4A8 and
+  W4A16 tags both read `bfloat16/None/eager=False`, and some saved W4A8 profiles
+  still contain the retracted bf16-head build with nothing marking them invalid.
+
+**Do**: rotate arm order per pass; take `best` by metric direction; separate
+prefill from decode in the profiler; stamp provenance (model path, config hash,
+commit, thread set) into every artifact and mark the retracted ones.
+
+### H3. Measure what the inference items need
+
+- **Process start → ready**, not engine-construction time (required by I6).
+- **Peak `RssAnon` through load**, sampled, not the endpoint (required by I0).
+- **Memory at realistic occupied context**, not 8 generated tokens (required to
+  re-open KV sizing).
+
+### H4. Contention guard
+
+Two sessions on the same cores wrecked several measurements and one grid's
+control drifted 30% mid-run. A cooperative lock file only excludes cooperating
+benchmarks, and load average > 8 identifies neither *which* cores are busy nor
+whether the load is ours.
+
+**Do**: lock file plus a pre-flight check on per-core utilization for the
+*target* core set, recorded in the artifact.
+
+### H5. Regression gates on measured bands
+
+The draft proposed an 8% throughput band. That 8% is a configured constant in
+the harness, not an estimated interval, and it sits against acknowledged ~30%
+run-to-run variation — it would fire constantly or never.
+
+**Do**: a checked-in expectations file whose bands come from the observed
+distribution per metric (`RssAnon` ~0.2%, per-op self time ~5%, end-to-end
+throughput wide enough to be meaningless, which is itself the argument for
+gating on per-op time and memory instead).
+
+### H6. A denser fidelity metric
+
+Greedy divergence over 12 prompts saturates and cannot separate "slightly worse"
+from "much worse". Top-1 agreement and KL against the bf16 reference over a few
+thousand teacher-forced tokens, symmetric with `llama-perplexity
+--kl-divergence` so llama.cpp stays comparable. Needs `llama-perplexity` built.
+
+Bring this forward: I3 and I4 are fidelity experiments and cannot be judged on a
+saturated metric.
+
+### H7. Make profiling the default for small differences
+
+Per-op self time resolves ~5%; throughput on this host ~30%. Have
+`bench_harness.py` say so when an arm difference falls inside the noise band,
+and point at `profile_cpu_step.py`.
+
+### H8. Purge retracted numbers from executable source
+
+Retractions are still being shipped as documentation:
+`kv_budget.py:9` advertises the withdrawn **8329 → 5254 MB**;
+`weight_path.py` reports W4A16 as **4.16 bits** (it is 4.25), sets
+`fused_cpu_norms=True` in three profiles against the recorded negative result
+and the `"0"` default, and calls tinygemm "the only option" (see I2).
+
+---
+
+## Order
+
+1. **H1, H2, H8** — the suite and harness cannot currently rank builds, and two
+   modules still assert retracted numbers. Everything downstream depends on this.
+2. **H3** — add the three measurements the inference items are defined in terms
+   of (start→ready, peak RSS, memory at context). Then **H4**, **H5**, **H6**.
+3. **I0** (peak load memory) — the actual edge blocker.
+4. **I7** (memory ledger), **I6** (imports) — now measurable.
+5. **I2** (measure `cpu_gemm_wna16`) — may invalidate a kernel project; cheap.
+6. **I1** (single packed layout) — largest confirmed memory win at unchanged
+   prefill, and the layout facts it needs are now measured.
+7. **I8** (SIGILL reconciliation), then **I3** (asymmetric W4A8) and **I4**
+   (quantizer) on a fixed fidelity metric.
+8. **I5** (attention wrappers) — bounded overhead experiment, revised target.
+
+The draft ordered by "whose verification method already exists". The revision
+orders by **whether the verification method is correct**, which is not the same
+question and is the one this project keeps getting wrong.
+
+---
+
+## Progress
+
+Started 2026-09-13, in the plan's own order. Items are struck through here only
+when measured, not when coded.
+
+**Step 1 -- H1, H2, H8: done.** Commits `6424c69` (oracles and measurement
+semantics), `27b94007` (retracted numbers out of `vllm-omni` source), `28aa63c`,
+`b713d6e`, `4285f92`. Agent tests 27 -> 38, harness tests 9 -> 34, plus 7 for
+the new fidelity metric.
+
+**Step 2 -- H3 to H7: done.** And it found a fifth measurement problem, of the
+same kind as the first four:
+
+> **The memory benchmark was reading the wrong process.** It took
+> `/proc/self/status`, which is the whole story only when the model is
+> in-process. That happens when `VLLM_ENABLE_V1_MULTIPROCESSING=0` keeps the
+> executor at `uni`; `vllm/platforms/cpu.py:273` promotes it to `mp` otherwise.
+> Every earlier run in this study set that variable, so **every published
+> memory figure is an in-process figure**. At vLLM's defaults the weights load
+> into a worker the benchmark never looked at: a smoke run reported 796 MB
+> resident for a model whose weights alone are 1.4 GB. Memory is now summed
+> over the process tree.
+
+First numbers from the rebuilt benchmark (W4A16 g128, 24 threads, 2048 ctx,
+`VLLM_CPU_KVCACHE_SPACE=1`, 2 interleaved passes):
+
+| | in-process (`uni`) | default (`mp`) |
+|---|---|---|
+| peak RssAnon, tree | 5722-5837 MB | 6068-6078 MB |
+| after arena release | ~4480 MB | ~5544 MB |
+| processes | 1 | 4 |
+| `ready_s` (process start to serving) | 78.5-111 s | 88-153 s |
+| of which imports | ~29-30 s | ~30 s |
+
+Three things follow. **The peak is 1.2-1.3 GB above the settled figure**, which
+is I0's premise and is now measured rather than asserted. **vLLM's default
+process layout costs ~240 MB of peak and ~1.1 GB of settled memory** over the
+in-process one, which no published number reflected. And **`ready_s` is far
+above `init_s`** -- imports are ~30 s, not the 19.6 s the plan quotes, so I6 is
+worth more than it claimed and was previously unmeasurable.
+
+These are not comparable to the report's 3411 MB: that row was taken
+in-process, with a sized KV budget, generating 8 tokens. Which is the argument
+for the provenance stamping added in the same step.
+
+**Step 3 -- I0 (peak load memory): done.** The driver was in our own repack
+path, not in vLLM: `unpack_nibbles` expands 4-bit codes to int32, and doing
+that a whole matrix at a time makes the 131072-row tied embedding a **1024 MiB
+transient**. Repacking in n-blocks (the layout makes them independent) is
+byte-identical and cuts the load-time transient from 550 to 105 MB, **489 MB
+off the peak**, with settled memory and init time unchanged -- chunked
+repacking is in fact faster than whole-matrix (79 ms vs 208 ms for the
+embedding). Chunk size tuned from a sweep rather than guessed.
+
+**Step 5 -- I2 (measure `cpu_gemm_wna16`): done, and it did not invalidate
+anything.** The fallback is real, so `weight_path.py`'s "only option" was
+false, but at 32.3 tok/s it is **2.2x slower than tinygemm** (71.6) and 2.3x
+slower than W4A8 (74.4), all on the same cores with only the env var
+differing. The routing was right; it now cites a measurement. Caveat recorded:
+`_get_isa_hint` returns `"amx"` whenever AMX is present, so this measured the
+kernel's best path on this host, not the `"vec"` path an AMX-less machine takes.
+
+**Step 6 -- I1 (single packed layout): done.** The layout
+facts the item needed are now established on this host rather than assumed
+(see I1 above), and the reconstruction is **bit-exact** against the existing
+dequantizer at 2048x2048, 3072x2048 and 256x512 -- including `gate_up`'s real
+13312x2048. `weight_halves` and the duplicated scale/zero are no longer
+allocated where N tiles the 64-channel block, which every Spark linear does
+(2048, 3072, 13312, 131072); shapes like N=80 keep the copy rather than
+mis-read the layout. `VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT=0` restores the old
+path, which is how the two are being A/B'd with identical code.
+
+*Process note, since this study keeps being bitten by measurement: I edited
+`cpu_int4.py` while an unrelated grid was running, so that grid's later passes
+used different code from its earlier ones. Those rows were discarded and the
+comparison re-run under the escape hatch instead.*
+
+---
+
+## Review record
+
+**Accepted and applied** (each verified in source before applying): the step
+table conflated W4A8 and W4A16 (W4A16 is 12.51 ms, and the remainder is 1.60 ms
+not ~1.9); bandwidth saturation of the GEMM does not make the rest overhead;
+per-step savings were added when the measured total contradicts the sum; the
+740 MB is a second *packed* copy plus duplicated scale/zero, not a bf16 matrix,
+so the cheap fix is direct dequant rather than a tiled kernel; per-tile oneDNN
+calls incur accumulator traffic the draft ignored; 4.25 ≠ 4.16 bits and Q4_K_M
+is 4.5 bits with mixed types, so the fidelity comparison was never
+budget-matched; the import figure is an upper bound and the init timer starts
+after imports; the MiniCPM-o stage-count extrapolation is invalid and the item
+does not serve edge deployability; peak load memory, not steady memory, is the
+edge blocker; KV sizing was retired on an 8-token measurement; W4A8 supports
+asymmetric zero points, so the report mistook our export choice for a kernel
+limitation; `cpu_gemm_wna16` is a real non-AMX path and `weight_path.py`'s "only
+option" is false; `smaps_rollup`/`tracemalloc` cannot deliver the attribution
+I7 needs; `prefill_dequant: false` bypasses the documented SIGILL guard; the
+harness does not rotate arm order, takes `max` as "best" for every metric, mixes
+prefill into per-step timing and stamps no provenance; the alarm oracle accepts
+a bare `"7"`, the toggle task scores final state only, and the scored suite is
+simulator-only.
+
+**Corrected with new measurements.** The review held that the tinygemm layout
+requires implementing "the ISA-specific lane mapping and tail handling" and that
+a K interval "is not a plain column slice", concluding a tiled decoder is
+expensive to get right. The first half is understated and the second is true but
+not the obstacle it implies. Measured here: a K-tile is contiguous *within each
+n-block of 64* and reassembling those segments is byte-identical to packing the
+K-slice (N=2048, K=2048, tile widths 128/256/512), split-K reconstructs the
+single call at 51.5 dB, and the lane order inside each 32-byte run is a fixed
+interleave (low nibble → channel *d*, high → *d+32*) that is constant across
+every k and n-block. So decoding needs a deinterleave, not a reverse-engineered
+table — which is what makes I1's direct-dequant fix small enough to be the first
+thing tried. The review's conclusion (do direct dequant, not the tiled kernel)
+is unchanged and adopted; its reasoning about why is superseded.
+
+The review's accumulator-traffic objection assumes K-outer with a full M×N
+accumulator, which a register-blocked GEMM would not pay — but since the draft
+proposed calling a library per tile, the objection lands against the draft as
+written, and I restructured rather than defended it.

docs/analysis/spark_cpu_int4.md

@@ -0,0 +1,237 @@
+# Closing the CPU decode gap: 4-bit weights for Spark-X2.5 on vLLM
+
+`analysis/spark_edge_support.md` ended with one open item. At matched bf16,
+vLLM beat llama.cpp on Spark-X2.5-1.7B everywhere -- 8.7-15.4x on prefill,
+1.19-2.65x on decode -- but llama.cpp's **Q4_K_M** build won single-stream
+decode outright. Decode at batch 1 reads every weight to emit one token, so
+bits per weight, not flops, set the token rate, and vLLM had no 4-bit CPU
+path in use. This note closes that and re-runs the comparison at matched
+precision.
+
+Two corrections to earlier findings come first, because both changed the
+answer.
+
+Code: `analysis/experiments/spark_edge/{export_gptq_cpu,quantize_int4,ppl_vllm,profile_cpu_step,bench_vllm_w4a8,cmp_llamacpp,dequantize_int4}.py`,
+`vllm_omni/model_executor/layers/quantization/cpu_int4.py` (+ tests).
+
+## 0a. Correction: the llama.cpp numbers were measured cold
+
+The Q4_K_M column in `spark_edge_support.md` was taken on a cold page cache.
+The GGUF lives on a FUSE network filesystem and llama.cpp mmaps it, so the
+first grid was partly timing storage. Warm, same pinned cores:
+
+| llama.cpp Q4_K_M, 32 threads | pp512 | tg128 |
+|---|---|---|
+| as previously published (cold) | 1199 | 82.81 |
+| warm | **1356** | **98.02** |
+
+The bar this work has to clear is 98 tok/s, not 82.8.
+
+## 0b. Correction: vLLM's CPU backend *does* ship a 4-bit kernel
+
+`spark_edge_support.md` reported that vLLM 0.28's CPU build has no int4 dense
+kernel, on the evidence that `woq_int4_linear`, `int4_scaled_mm_with_quant`
+and `da8w4_linear` are all absent. They are -- but the kernel exists under a
+different name. `torch.ops._C.int4_scaled_mm_cpu` is present in the released
+CPU wheel: an **AMX W4A8** GEMM, reached by loading a GPTQ checkpoint with
+`VLLM_CPU_INT4_W4A8=1`, with `cpu_gemm_wna16` as the W4A16 fallback. The
+search missed it because it searched for names, not for the capability.
+
+This matters, because it is the fastest option by a wide margin. Measured at
+Spark's shapes against PyTorch's `aten::_weight_int4pack_mm_for_cpu`
+(`bench_vllm_w4a8.py`, 28 layers + head, group 128):
+
+| batch | tinygemm | vLLM W4A8 (AMX) | speedup |
+|---|---|---|---|
+| 1 row (decode) | 4.45 ms | **2.66 ms** | 1.67x |
+| 8 rows | 14.25 ms | **6.76 ms** | 2.11x |
+| 512 rows (prefill) | 878 ms | **104 ms** | 8.44x |
+
+## 1. What was actually slow
+
+`profile_cpu_step.py` attributes one decode step. At bf16, 32 threads:
+
+| bucket | ms/step | share |
+|---|---|---|
+| `_C::onednn_mm` (113 GEMMs) | 21.24 | 76% |
+| inductor-generated kernels (norms, rope, GEGLU) | 2.47 | 9% |
+| attention + KV cache ops (4 x 28) | 1.63 | 6% |
+| head-gate `aten::mm` (28) | 1.08 | 4% |
+| sampling (`argmax` over 131072) | 0.24 | 1% |
+| ~900 small view/slice ops | 0.55 | 2% |
+| **total** | **27.96** | 35.8 tok/s |
+
+3.42 GB of bf16 weights at ~161 GB/s effective. Only fewer bits move it.
+
+## 2. Results
+
+Xeon Platinum 8480C, one socket, 32 physical cores pinned with `taskset`,
+warm page cache, engines run strictly one at a time, and every run records the
+busy fraction of its own cores before it starts (`cpu_busy.py`) -- a
+shared-core benchmark measures the scheduler, not the engine. Perplexity is
+teacher-forced over 512-token windows, 126 217 scored tokens, each engine
+scored against **its own** bf16 run so tokenizer and chunking differences
+cancel.
+
+| build | bits/weight | pp512 | tg128 | ppl vs own bf16 |
+|---|---|---|---|---|
+| vLLM bf16 | 16 | 4455 | 36.35 | -- |
+| vLLM GPTQ W4A16 (sym, g128) | 4.16 | **4659** | 37.53 | +25.5% |
+| **vLLM GPTQ W4A8 (sym, g128)** | 4.16 | **4355** | **88.7** | +27.2% |
+| vLLM `cpu_int4` (asym, g128) | 4.25 | 3589 | 85.5 | +18.2% |
+| vLLM `cpu_int4` (asym, g64) | 4.50 | 3689 | 81.1 | +12.5% |
+| llama.cpp Q4_K_M | 5.15 | 1356 | **98.02** | **+7.6%** |
+
+Against llama.cpp at 4-bit: **prefill 3.21x faster, decode 0.90x.** Decode
+went 36.35 -> 88.7, a 2.44x gain, which turns a 2.70x deficit into a 1.11x
+one without quite closing it.
+
+Quantizing the head matters more than anything else in that table: Spark ties
+its 131072 x 2048 vocab table to the output head, and that one GEMM is the
+largest read in a decode step. Leaving it in bf16 costs 12 tok/s (76.0 vs
+88.7), so `export_gptq_cpu.py --quantize-lm-head` unties it, keeping the
+table in bf16 for the lookup (a few hundred bytes a token) and giving the
+head its own 4-bit copy.
+
+Note the W4A16 row: best prefill of anything measured, worst decode of the
+4-bit builds. It dequantizes to bf16 per call, which amortizes over 512 rows
+and is pure overhead on one.
+
+### Thread scaling (cores 56-87, both engines, default OpenMP settings)
+
+Measured with `cpu_int4`, before the W4A8 path was found, but the shape of it
+is the point:
+
+| threads | vLLM int4 pp512 | llama.cpp pp512 | vLLM int4 tg128 | llama.cpp tg128 |
+|---|---|---|---|---|
+| 4 | **800.1** | 210.2 | **27.08** | 24.20 |
+| 8 | **1476.2** | 409.8 | 44.02 | 44.00 |
+| 16 | **2599.1** | 799.2 | 66.72 | **73.19** |
+| 32 | **3524.9** | 1348.0 | 80.20 | **99.55** |
+
+Decode wins at 4 threads, ties at 8, loses from 16 up. That is the signature
+of a fixed per-step cost that does not shrink when the GEMMs do.
+
+### What else moved the needle, and what didn't
+
+| change | tg128 | |
+|---|---|---|
+| `cpu_int4` g64 baseline | 81.10 | |
+| `KMP_BLOCKTIME=infinite OMP_WAIT_POLICY=ACTIVE KMP_LIBRARY=turnaround` | 83.61 | ~250 OpenMP parallel regions per step; how threads park between them is not free |
+| group 128 instead of 64 | 85.48 | 5.5% fewer weight bytes |
+| `detokenize=False` (what `llama-bench` measures) | 82.13 | |
+
+Tried and rejected: `compile_sizes=[1]`, to give inductor static shapes for
+the decode step (81.77, no change); `enforce_eager` (58.88, much worse);
+inductor `cpp_wrapper` (fails outright -- vLLM's own ops take a `str`
+argument `custom_op_wrapper` cannot box).
+
+## 3. The remaining decode gap is framework, not kernel
+
+Re-profiling the `cpu_int4` step at 12.50 ms:
+
+| bucket | ms/step |
+|---|---|
+| int4 GEMMs (113) | 5.49 |
+| inductor-generated kernels | 2.29 |
+| attention + KV cache ops | 1.41 |
+| head-gate `aten::mm` (28) | 0.92 |
+| custom-op dispatch (113) | 0.58 |
+| sampling + small ops | 0.80 |
+| **total** | **12.50** |
+
+Those GEMMs move 0.95 GB in 5.49 ms, **173 GB/s**; llama.cpp does its whole
+10.2 ms step against 1.1 GB, ~108 GB/s all-in. The 4-bit kernel is not what
+is losing. **7.0 ms of the 12.5 ms step is per-step framework cost**, and
+llama.cpp has almost none of it. With the faster W4A8 kernel the GEMM share
+falls further and the framework share rises, which is why the gap closes to
+0.90x and then stops.
+
+Closing the rest is vLLM CPU engine work -- the attention wrappers, the ~900
+small view ops, inductor launch overhead -- and it is out of reach here
+because vLLM itself is off-limits to modification. The one piece inside reach
+is the head gate: 28 `aten::mm` calls, 0.92 ms, almost all thread barrier for
+a 2048x8 matmul. Rewriting it as a fused broadcast-and-reduce was tried and is
+much worse (44.5 tok/s; inductor does not vectorize the reduction). Folding
+those eight rows into the fused QKV GEMM would remove it, and is the obvious
+next step.
+
+## 4. Quality: the grid, not the runtime
+
+Q4_K_M spends **5.15 bits per weight** (1.1 GB for 1.7 B parameters) against
+4.16-4.50 here, and still loses less: +7.6% perplexity against +12.5% for the
+best of ours. Two separate things cause that.
+
+**vLLM's GPTQ path cannot express an asymmetric grid at 4 bits.** It rejects
+`sym=False` outright ("Unsupported quantization config: bits=4, sym=False"),
+so the fast W4A8 kernel can only be fed a symmetric grid -- which wastes a
+level on every group whose weights are off-centre. That is most of why the
+W4A8 rows cost +27% while `cpu_int4`, which is asymmetric, costs +12.5% at a
+comparable bit budget. The AMX kernel itself takes real zero points
+(`convert_weight_packed_scale_zp(..., CPUQuantAlgo.AWQ)`); it is the GPTQ
+*config* that refuses. Exporting AWQ format instead should get an asymmetric
+grid through the same kernel, and is the highest-value thing left undone
+here.
+
+**W4A8 quantizes activations to int8, and that is nearly free.** W4A16 on the
+same checkpoint scores +25.5% against W4A8's +27.2% -- a 1.7 point difference
+for a 2.4x decode speedup. That is a much better outcome than int8
+activations have had elsewhere in this repo (Qwen3-TTS, MiniCPM-o, Spark on
+the NPU), and worth recording as the counter-example.
+
+**Even asymmetric, our grid is coarser than K-quants.** `cpu_int4` does
+round-to-nearest on each group's min/max with a small shrink search; the
+search is worth about what it costs (at g64: 48.0 perplexity with it, 59.7
+without, on the smaller corpus) but it optimizes per-group squared error,
+which is the wrong objective. Q4_K_M uses an importance-weighted scale search
+per 32-weight sub-block and spends more bits on the tensors that need them.
+
+This is a quantizer gap, not a runtime gap, and the control proves it:
+expanding a quantized checkpoint back to plain bf16 offline
+(`dequantize_int4.py`) and scoring that reproduces the quantized run to
+**within 0.2%** (48.08 vs 48.01 at g64, 56.80 vs 56.89 at g32). Both GEMM
+paths, the nibble packing and the quantized tied embedding are all faithful.
+
+A caution on measuring this: on a smaller, harder corpus (34 k words of this
+repo's own notes, bf16 perplexity 43.7) the ordering is not even stable --
+group 32 lands worse than group 128 there, and reverses on the larger corpus.
+Perplexity differences between 4-bit variants of a 1.7 B model need a corpus
+big enough to be in a sane regime before they mean anything.
+
+## 5. An AMX hazard worth recording
+
+Running PyTorch's tinygemm kernel on a multi-row batch and then vLLM's
+`cpu_attention_with_kv_cache` in the same forward kills the process with
+**SIGILL inside the attention kernel** -- the signature of an AMX instruction
+issued with no valid tile configuration loaded. The identical prompt is fine
+in bf16, and fine at any batch size once every multi-row GEMM takes a
+different path, so the two kernels disagree about who owns the AMX tile state.
+`cpu_int4` therefore routes only single-row steps to tinygemm
+(`dequant_threshold` defaults to 2). vLLM's own W4A8 path does not have the
+problem.
+
+## 6. Where this leaves it
+
+| | before | after |
+|---|---|---|
+| decode vs llama.cpp Q4_K_M | 0.37x (36.35 vs 98.02) | **0.90x** (88.7) |
+| prefill vs llama.cpp Q4_K_M | no 4-bit path in use | **3.21x** (4355 vs 1356) |
+| quality cost of 4-bit | -- | +27.2% ppl (W4A8) / +12.5% (`cpu_int4` g64) vs Q4_K_M's +7.6% |
+| model on disk | 3.4 GB | 0.9-1.4 GB |
+
+Honest summary: 4-bit weights turn a 2.7x single-stream decode deficit into a
+1.11x one and put prefill 3.2x ahead, but vLLM still does not beat llama.cpp
+at single-stream decode on 32 cores, and our 4-bit grids are measurably
+coarser than K-quants. Three specific, measured things stand between here and
+a win, in order of expected value:
+
+1. **An AWQ-format export**, to get an asymmetric grid through the AMX W4A8
+   kernel -- worth roughly the +27.2% -> +12.5% perplexity difference at no
+   speed cost.
+2. **Folding the head gate into the QKV GEMM** -- 0.92 ms of a ~11 ms step.
+3. **vLLM CPU per-step overhead** -- 7.0 ms at batch 1, the whole remaining
+   gap, and not addressable without changing vLLM.
+
+`cpu_int4` keeps a narrower role than intended: it is the more accurate of the
+two (asymmetric grid, +12.5% against +27.2%) and it needs no AMX, but on an
+AMX host vLLM's own W4A8 path is faster on both axes. Its docstring says so.

docs/analysis/spark_edge_support.md

@@ -0,0 +1,994 @@
+# Spark-X2.5-1.7B on CPU and Galaxy S25
+
+Spark-X2.5 (XHToken/iFlytek, Apache 2.0, released 2026-09-01) is a 1.7 B
+on-device model built for agentic tool use: BFCL-V4 46.9, tau-squared-bench
+65.3, a native 1 M-token context and 200+ languages. This note covers adding
+it to vLLM-Omni, measuring it against every engine that can run it, putting it
+on a real Snapdragon 8 Elite, and driving an Android phone with it.
+
+Reproduction scripts: `analysis/experiments/spark_edge/`.
+Model code: `vllm_omni/model_executor/models/spark2_5/`, exporter
+`vllm_omni/edge/spark_export.py`.
+
+## 1. What had to be built
+
+Three traits separate Spark from a Llama-style decoder, and all three are why
+no generic converter picks it up:
+
+| trait | detail | why it matters |
+|---|---|---|
+| hybrid attention | `layer_types` = 3x `sliding_attention` (window 512) then 1x `full_attention`, 28 layers -> 21 sliding + 7 full | bounds KV growth; this is what makes 1 M context affordable on a phone |
+| per-layer-type RoPE | sliding: all 256 head dims, theta 1e4. full: first 64 dims only (`partial_rotary_factor` 0.25), theta 5e6 | two rotary tables, not one |
+| head-wise output gate | `g_proj` [2048 -> 8] reads the same post-norm input as QKV; `sigmoid` scales each head's attention output before `out_proj` | an extra projection with no analogue in Llama/Qwen |
+
+Also: QKV ships pre-fused as one `q_k_v_proj`, the MLP is GEGLU with *exact*
+gelu, embeddings are tied, and `head_dim` is 256 with only 2 KV heads.
+
+### Engine support before this work
+
+| engine | Spark-X2.5 support |
+|---|---|
+| llama.cpp | yes, native (`src/models/spark2-5.cpp`) |
+| SGLang | yes, native (`srt/models/spark2_5.py`) |
+| vLLM 0.28 | **no** -- added here |
+| MNN | no. `llmexport.py` dispatches on a fixed `model_type` list; `spark2_5` is absent |
+| ExecuTorch | no |
+| ncnn | no |
+
+MNN, ExecuTorch and ncnn would each need the architecture written from
+scratch. That is the honest reason they do not appear in the benchmark tables
+below: they cannot load the model at all, not that they are slow at it.
+
+### Correctness
+
+Verified against the official remote-code implementation pinned to
+transformers 4.57.1 (the version `config.json` targets; it does not import on
+the 5.x in our venvs, so a dedicated venv runs ground truth).
+
+| scenario | prompt | fp32 | bf16 |
+|---|---|---|---|
+| short | 28 tok | 32/32 greedy, max logprob diff 2.2e-05 | 24/24 greedy |
+| long (exercises sliding window) | 2465 tok | -- | 32/32 greedy |
+| tool calling | 137 tok | 32/32 greedy | diverges at token 23 |
+
+The bf16 tool-calling divergence is a near-tie under greedy decoding, not a
+defect: in fp32 the same prompt matches exactly with a 2.2e-05 logprob
+difference.
+
+One vLLM bug had to be worked around. The head-wise gate is a [2048, 8]
+projection, and vLLM's CPU backend routes any narrow linear to the sgl-kernel
+packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim
+0. VNNI packing does not for N that small, so it reports K instead of N and
+`torch.compile` dies tracing the gate multiply. The gate is kept as a plain
+parameter with `F.linear` -- a 2048x8 matmul gains nothing from the packed AMX
+kernel anyway.
+
+## 2. CPU: vLLM-Omni vs llama.cpp
+
+Xeon Platinum 8480C (Sapphire Rapids, AMX-BF16/INT8). Both engines pinned with
+`taskset` to the same physical cores and run **strictly one at a time**;
+prefix caching disabled on vLLM so a replayed prompt is really recomputed
+(llama-bench has no such cache). pp512 = prefill throughput, tg128 = decode.
+
+### Same precision (bf16)
+
+| threads | vLLM pp512 | llama.cpp pp512 | prefill | vLLM tg128 | llama.cpp tg128 | decode |
+|---|---|---|---|---|---|---|
+| 4 | **1113.4** | 72.4 | 15.4x | **11.11** | 4.20 | 2.65x |
+| 8 | **2113.7** | 173.8 | 12.2x | **19.88** | 16.15 | 1.23x |
+| 16 | **3639.4** | 341.7 | 10.7x | **30.16** | 25.31 | 1.19x |
+| 32 | **4454.9** | 509.5 | 8.7x | **36.35** | 28.57 | 1.27x |
+
+vLLM wins both, at every thread count. The prefill gap is AMX: llama.cpp's
+bf16 path runs at AVX-512 speed, vLLM's goes through oneDNN/AMX tiles.
+
+### Against llama.cpp's quantized builds
+
+| threads | vLLM bf16 pp | llama.cpp Q8_0 pp | llama.cpp Q4_K_M pp | vLLM bf16 tg | llama.cpp Q8_0 tg | llama.cpp Q4_K_M tg |
+|---|---|---|---|---|---|---|
+| 4 | **1113.4** | 118.1 | 103.6 | 11.11 | 9.85 | **10.36** |
+| 8 | **2113.7** | 490.2 | 434.2 | 19.88 | 45.01 | **42.40** |
+| 16 | **3639.4** | 914.7 | 806.3 | 30.16 | 66.23 | **71.50** |
+| 32 | **4454.9** | 1493.1 | 1199.0 | 36.35 | 76.54 | **82.81** |
+
+Prefill: vLLM at bf16 still beats llama.cpp's *best quantized* build by 3.0x
+at 32 threads. Prefill is compute-bound, and AMX beats 4-bit weight savings.
+
+Decode: llama.cpp Q4_K_M wins from 8 threads up, by 2.3x at 32. Decode is
+memory-bound -- 4-bit weights move a quarter of the bytes -- and this is a
+precision difference, not an engine difference. Closing it needs a quantized
+vLLM CPU path for Spark, which is the open item in section 6.
+
+### Quantized: int8 on CPU
+
+vLLM 0.28's CPU build has int8 dense kernels (`int8_scaled_mm_with_quant`,
+`dynamic_scaled_int8_quant`), and `static_scaled_fp8_quant` is registered with
+an empty schema, so on-the-fly fp8 fails outright.
+
+An earlier draft of this note said the backend had no dense int4 kernel
+either, on the strength of `woq_int4_linear`, `int4_scaled_mm_with_quant` and
+`da8w4_linear` all being absent. That was wrong -- they are simply not the
+names it uses. See section 2c: there are two four-bit paths, and the one
+vLLM ships is `int4_scaled_mm_cpu`, an AMX W4A8 GEMM reached through the
+GPTQ/AWQ kernel selector.
+
+Int8 is also available. `analysis/experiments/spark_edge/quantize_int8.py` writes a
+compressed-tensors W8A8 checkpoint straight from the safetensors (symmetric
+per-output-channel round-to-nearest; no calibration needed, and llmcompressor
+is avoided because it pulls transformers 5.x, on which Spark's remote code
+does not import). The head-wise gate is left in full precision: its eight
+outputs each scale a whole attention head, and it is 0.001% of the weights.
+Weight SNR 36.9-40.9 dB across 140 tensors.
+
+| threads | pp512 bf16 -> int8 | tg128 bf16 -> int8 |
+|---|---|---|
+| 16 | 3639 -> **5135** (1.41x) | 30.16 -> **48.30** (1.60x) |
+| 32 | 4455 -> **6978** (1.57x) | 36.35 -> **58.44** (1.61x) |
+
+Quality against the fp32 reference, though, is not free:
+
+| scenario | greedy match | first-token top-1 | top-20 overlap | max logprob diff |
+|---|---|---|---|---|
+| short (28 tok) | 23/24 | agrees | 18/20 | 0.83 |
+| long (2465 tok) | **12/32** | agrees | 19/20 | 1.56 |
+| tool calling | 31/32 | agrees | 18/20 | 0.58 |
+
+bf16 matched 32/32 on all three with a max difference of 0.24. Int8 keeps the
+first token everywhere, but at long context the distribution has moved enough
+that greedy output diverges after a dozen tokens. Usable, not free -- the same
+caution the Qwen3-TTS and MiniCPM-o work in this repo reached about automatic
+int8.
+
+## 2b. CPU under concurrency
+
+Single-stream decode is memory-bound, which is a precision contest llama.cpp
+wins with 4-bit weights. Concurrent decode is compute-bound, which is an AMX
+and continuous-batching contest. Total generated tokens/s, 32 pinned cores,
+128-token prompts generating 128 tokens each:
+
+| batch | vLLM bf16 | vLLM int8 | llama.cpp Q4_K_M | int8 vs llama.cpp |
+|---|---|---|---|---|
+| 1 | 34.6 | 54.9 | **91.1** | 0.60x |
+| 2 | 69.9 | 105.2 | **129.6** | 0.81x |
+| 4 | 139.8 | 210.3 | **251.6** | 0.84x |
+| 8 | 266.1 | 387.8 | **406.5** | 0.95x |
+| 16 | 495.7 | **694.7** | 613.9 | 1.13x |
+| 32 | -- | **1080.6** | 778.3 | 1.39x |
+| 64 | -- | **1557.9** | 904.5 | 1.72x |
+
+The crossover is around batch 12. Below it llama.cpp's 4-bit weights win;
+above it vLLM's AMX-INT8 GEMMs and continuous batching pull away, reaching
+1.72x at batch 64 and still climbing. A phone agent is a batch-of-one
+workload, so llama.cpp is the right single-device choice today; an edge
+gateway serving a dozen sessions is vLLM's.
+
+## 2c. Four bits on CPU: two paths, and which one a device should take
+
+vLLM 0.28's CPU build has a 4-bit GEMM. `CPUWNA16LinearKernel`
+(`vllm/model_executor/kernels/linear/mixed_precision/cpu.py`) routes GPTQ and
+AWQ checkpoints to **`int4_scaled_mm_cpu`**, an AMX W4A8 kernel, whenever the
+activations are bf16, the ISA has AMX tiles and the checkpoint carries no
+activation reordering. Four-bit weights times eight-bit activations is exactly
+what llama.cpp's Q4_K_M does (Q4_K weights, Q8_K activations), so this is the
+honest precision match rather than a handicap.
+
+Two paths were built, because they are not the same trade:
+
+| | weights | activations | kernel | bits/weight | reached by |
+|---|---|---|---|---|---|
+| **W4A8** | 4-bit symmetric, group-wise | int8 (per the kernel) | `_C::int4_scaled_mm_cpu` (AMX) | 4.16 @ g128 | GPTQ checkpoint, vLLM's own dispatch |
+| **W4A16** | 4-bit asymmetric min/max, group-wise | bf16 | `aten::_weight_int4pack_mm_for_cpu` | 4.16 @ g128 | `cpu_int4`, registered out-of-tree |
+
+W4A8 is forced symmetric: `_process_gptq_weights_w4a8` builds its own zero
+point (a constant 8) and ignores the checkpoint's `qzeros`, so the grid must be
+`w = (code - 8) * scale`. W4A16 is free to use an asymmetric min/max grid, and
+a per-group scale search recovers roughly a dB in either case.
+
+### Which is faster
+
+Per-op profiles, taken back to back on the same 24 cores. Throughput on this
+host swings ~30% run to run and the gap here is ~8%, so the GEMM self time is
+the measurement that can actually rank them:
+
+| path | GEMM/step | step | tok/s |
+|---|---|---|---|
+| W4A8 g128 | **5.91 ms** | **11.55 ms** | **86.6** |
+| W4A16 g128 | 6.85 ms | 12.51 ms | 80.0 |
+| llama.cpp Q4_K_M | -- | 11.11 ms | 90.0 |
+| vLLM bf16 | -- | -- | 41.0 |
+
+W4A8 wins twice: the kernel is 0.94 ms cheaper per step, and because it is
+reached through vLLM's own dispatch it also avoids the 0.48 ms/step custom-op
+boundary the tinygemm path needs (that boundary exists because tinygemm's cost
+is linear in batch size, so prefill has to be routed to a dequantize-then-AMX
+path instead -- see below).
+
+**The trap that cost a day: a GPTQ checkpoint must quantize the output head.**
+Spark ties its embedding to its head, so a converter that leaves the tie in
+place emits no `lm_head` tensors and that one GEMM runs bf16 through
+`_C::onednn_mm` -- **2.97 ms per step**, more than the entire 4-bit saving. The
+profile is what caught it: `int4_scaled_mm_cpu` ran 112 times (28 layers x 4
+linears) and `onednn_mm` once. Every "W4A8" number measured before that was on
+a handicapped build, which is why the W4A8/W4A16 ordering appeared to flip
+between grids.
+
+### Which is more faithful
+
+Greedy divergence from each build's own bf16 output, 12 prompts, measured the
+same way on both engines:
+
+| build | exact match | mean prefix kept |
+|---|---|---|
+| llama.cpp Q4_K_M | **4/12** | **0.51** |
+| vLLM W4A16 | 2/12 | 0.30 |
+| vLLM W4A8 | 1/12 | 0.13 |
+
+Q4_K_M's 6-bit sub-block scales buy real granularity at the same bit budget;
+round-to-nearest with a per-group search does not match it. So the two paths
+are a genuine choice, not a ranking: **W4A8 for speed, W4A16 for fidelity**,
+and `vllm_omni/edge/weight_path.py` makes that choice from a hardware probe
+rather than a flag, because the fast kernel on each device needs a different
+and mutually exclusive weight layout:
+
+| device class | path | why |
+|---|---|---|
+| x86 + AMX | W4A8 / GPTQ | `int4_scaled_mm_cpu` applies |
+| x86 without AMX | W4A16 / tinygemm | no AMX, no int4 kernel |
+| x86 + AMX, fidelity asked | W4A16 / tinygemm | keeps bf16 activations |
+| ARM64 | W4A16, **unverified** | PyTorch's 4-bit CPU GEMM dispatches a different kernel there |
+| phone NPU | exported graph, calibrated w8a16 | vLLM does not run there at all |
+
+### Prefill needs the other kernel
+
+The tinygemm kernel dequantizes once per row of activations, so its cost is
+linear in batch size: it beats AMX bf16 below ~8 rows and loses badly above.
+Measured at Spark's `gate_up` shape, per call: 0.070 ms at 1 token, 14.2 ms at
+512. Left alone, pp512 collapsed from 4455 tok/s (bf16) to 557. Unpacking the
+weight to bf16 once per call and handing it to the ordinary oneDNN/AMX GEMM
+brings it back to **3347 tok/s**, 3.0x llama.cpp's 1121. W4A8 needs none of
+this -- the AMX kernel is the right one at every batch size, which is the
+second reason it is the default on an AMX machine.
+
+## 2d. Init time and resident memory, which is what decides edge at all
+
+Speed is not the binding constraint on an edge box; a 90-second start and 8 GB
+resident is. Both turned out to be fixable, and one of them was a bug.
+
+### vLLM recompiled the model on every single start
+
+`_try_load_aot_compiled_fn` finishes a cache load with
+`loaded_fn._artifacts.compiled_fn.finalize_loading(cfg)`. On CUDA that artifact
+is a wrapper object carrying the method; with the inductor CPU backend it is a
+plain function, so the call raises `AttributeError`, the handler treats the
+whole load as a cache miss ("Compiling model again due to a load failure"), and
+the model is compiled again from scratch. Every restart paid it.
+
+Attaching the missing no-op (a plain Python function accepts attribute
+assignment; the CPU artifact loads its generated `.so` lazily on first call,
+which is the warm-up run) fixes it. Same cache directory, one variable changed:
+
+| | init |
+|---|---|
+| cold cache (first ever compile) | 93.8 s |
+| warm cache, vLLM as shipped | 49.8 s |
+| **warm cache, shim installed** | **14.9 s** |
+
+**3.3x faster warm start**, reproduced across passes (14.9 / 14.7 / 15.0, and
+12.8-13.0 on the W4A16 path). The shim is self-disabling: if the artifact
+already has `finalize_loading`, as it does on GPU and will if upstream gives
+the CPU path one, it changes nothing.
+
+### Measure anonymous memory, not RSS
+
+A first pass concluded that sizing the KV cache saved 3.1 GB. It does not, and
+the error is worth recording because it is easy to repeat: that comparison
+read `VmRSS`, which counts the mmap'd checkpoint's file pages, and those come
+and go with page-cache pressure. **Identical configurations measured 1.2-2.0 GB
+apart.** `RssAnon` -- allocations only -- repeats to within **0.5 MB**, and is
+what a memory budget is actually about.
+
+### The KV cache is not a memory lever on CPU
+
+Re-measured properly, with only the budget changing:
+
+| KV budget | anonymous RSS |
+|---|---|
+| 4 GiB (env default) | 7601 MB |
+| 512 MiB | 7405 MB |
+| 128 MiB | 7413 MB |
+
+Shrinking the cache 32x moves resident memory by **188 MB**, and 512 MiB is
+indistinguishable from 128 MiB. The cache is allocated but its pages are never
+touched, so it costs address space, not memory.
+
+It still has to be *correct*, which is a separate problem: `--kv-cache-memory-bytes`
+was silently discarded on CPU. `CpuPlatform.check_and_update_config` ends with
+a block its own comment calls "Lagecy setting" that overwrites
+`cache_config.kv_cache_memory_bytes` from `VLLM_CPU_KVCACHE_SPACE`
+unconditionally, so the budget could only be set in whole GiB and vllm-omni's
+per-stage budgets had no effect at all. Verified by allocation rather than by
+memory, because allocation cannot drift: with the env at 1 GiB, asking for
+128 MiB produced the same **17 881** cache tokens as asking for nothing, and
+with the env at 4 GiB it produced **71 527**. After restoring the obvious
+precedence the same request yields **2 231** tokens. That matters for
+admission -- vLLM preempts by recompute when the cache cannot hold
+`max_num_seqs x max_model_len`, which on a single-stream deployment is a stall
+-- not for footprint. `vllm_omni/edge/kv_budget.py` prices the cache from the
+config, reporting the flat figure (what is safe to pass, since vLLM's CPU
+accounting is flat) beside the hybrid one a layer-type-aware backend could use.
+
+### Where the memory actually goes: the weight load
+
+Split by phase, on anonymous RSS:
+
+| phase | W4A8 (GPTQ) | W4A16 (tinygemm) |
+|---|---|---|
+| before import | 12 MB | 12 MB |
+| after `import vllm_omni` | 513 MB | 515 MB |
+| after engine construction | 7338 MB | 5023 MB |
+| **after releasing free arenas** | **3510 MB** | **3411 MB** |
+
+Imports cost half a gigabyte -- they are the init-*time* problem, not the
+memory one. The engine adds 4.5-6.8 GB for a model whose weights are under
+1.4 GB, and **most of it is not live**: loading a quantized checkpoint unpacks
+4-bit codes to int32 -- four bytes per weight -- to permute and repack them,
+and although the intermediates are freed the allocator keeps the arenas. One
+release call returns **3828 MB** on the GPTQ path and 1612 MB on tinygemm, a
+**52%** cut on the configuration we actually want to deploy.
+
+The second reading of that table matters more than the first: after the
+release both paths land in the same place. W4A8's apparent 2.3 GB penalty over
+W4A16 was never a footprint difference, it was the repack. So W4A8 wins on
+speed *and* ties on memory, and fidelity is the only remaining reason to
+choose W4A16.
+
+`vllm_omni/edge/memory.py` does the release and `patch.py` hooks it to the CPU
+worker's load and warm-up (`VLLM_OMNI_CPU_RELEASE_MEMORY=0` to disable). The
+~3.5 GB that remains, against 0.86 GB of weights, is the next thing to chase.
+
+### Two bugs that made CPU deployments take the slow path silently
+
+Init is not one number per model; it has a different cause in each pipeline,
+and in two places the slow path was taken without saying so.
+
+**Parallel stage init could never run on CPU.**
+`resolve_stage_physical_devices` returns `None` for `devices: "cpu"`, and the
+admission resolver fails closed on `None` -- right for a GPU replica whose
+devices cannot be accounted, wrong here -- so every multi-stage CPU pipeline
+raised `StageAdmissionError` and fell back to sequential startup. The edge
+profile *sets* `parallel_stage_init: true`, so on CPU that profile was broken
+rather than merely slower.
+
+The fix is not to skip admission. The per-device ledger is vacuous on CPU, but
+the resource concurrent init contends for is host memory, and two stages
+allocating KV caches at once exhaust it exactly as two GPU stages exhaust a
+card. `check_host_admission` applies the same discipline to that resource: sum
+the declared budgets, compare with available RAM and a margin, refuse to start
+otherwise, and fail closed on a stage that declares no budget.
+
+| Qwen3-TTS 0.6B, init only | |
+|---|---|
+| sequential | 64.57 s |
+| **parallel** | **30.54 s** |
+
+**A deploy YAML meant different things depending on which flag loaded it.**
+The `orchestrator:` block carrying `parallel_stage_init` was applied only on
+the `deploy_profile` path, so pointing `--deploy-config` at the very same file
+loaded its stages and dropped the flag: 63.76 s against 30.76 s, with nothing
+in the log to explain the difference. The block now applies either way.
+
+### What is not a memory lever
+
+Two candidates tested and closed, so they are not re-tried:
+
+| lever | effect |
+|---|---|
+| OMP thread count (8 / 24 / 48) | 3754 / 3775 / 3818 MB -- 64 MB across a 6x range |
+
+**Retracted (2026-09-14): KV cache size was on this list, and it is the single
+largest lever there is.** The row read "4 GiB -> 128 MiB saves 188 MB, the
+pages are never touched". Re-measured now that the budget is actually
+honoured -- 3 interleaved passes per arm, both filling a 1536-token context:
+
+| KV budget | resident |
+|---|---|
+| 1 GiB (the smallest `VLLM_CPU_KVCACHE_SPACE` can express) | 3264 MB |
+| 192 MiB, explicit | **2421 MB** |
+
+**843 MB.** And `mincore(2)` says why the old reasoning was wrong: the cache is
+not allocated-and-untouched, it is fully in core -- 1023.8 MB allocated against
+1024.1 MB resident, and 190.8 against 191.1 when sized down.
+
+The explanation for the original 188 MB is two sections below, in this report's
+own words: `--kv-cache-memory-bytes` *was silently discarded on CPU*, because
+`CpuPlatform.check_and_update_config` overwrote it from
+`VLLM_CPU_KVCACHE_SPACE` unconditionally. So the "32x shrink" never happened;
+both arms ran the same cache and the 188 MB was measuring something else. The
+lesson is the one this study keeps relearning: a lever tested through a path
+that silently ignores it measures nothing, and "no effect" is the most
+dangerous result to accept without checking that the knob was connected.
+
+The dual weight layout the W4A16 path keeps (tinygemm for decode, a row-major
+copy for prefill) *is* one. Median of three interleaved passes, anonymous RSS,
+spread under 1.5%:
+
+| build | resident | prefill | decode | init |
+|---|---|---|---|---|
+| W4A8 g128 | 3640 MB | -- | 86.6 tok/s | 15.0 s |
+| W4A16 g128 | 3542 MB | 3252 tok/s | 76.9 tok/s | 12.4 s |
+| **W4A16, `prefill_dequant: false`** | **2802 MB** | 448 tok/s | 78.5 tok/s | 16.0 s |
+
+The second layout costs **740 MB** and buys **7.3x prefill**, because the
+tinygemm kernel dequantizes once per row of activations and its cost is linear
+in the batch.
+
+**That trade-off is smaller than it looked, but it is real** (2026-09-13). It
+rested partly on treating the tinygemm layout as opaque, so a row-major
+duplicate was kept beside it to have something to dequantize from -- but the
+duplicate was also doing work: `to_split_halves` performs the nibble
+de-interleave *once at load*, so every prefill call reads two contiguous
+planes. It is a precomputed permutation, not just a copy. Measured on this host instead of
+assumed -- by packing an index pattern and reading back where each nibble
+landed -- the layout is `[N/64][K][32 bytes]`: K is the middle axis, so a K
+range is contiguous within each n-block, and inside every 32-byte run byte `d`
+carries output channel `d` in its low nibble and `d + 32` in its high one, the
+same permutation for every `k` and every block. Decoding it is a deinterleave,
+not a reverse-engineered ISA table.
+
+So the prefill path can dequantize from the kernel's own layout, and then the
+duplicate is not allocated -- at the cost of redoing that de-interleave on
+every call. The reconstruction is **bit-exact** against the previous
+dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048.
+Shapes where N does not tile the 64-channel block (N=80 is 64+16) keep the
+copy rather than mis-read the layout; every Spark linear tiles cleanly.
+
+Measured with `VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT` toggling the two paths in
+otherwise identical code, 3 arms x 2 interleaved passes, anonymous RSS summed
+over the process tree, 1536-token context:
+
+| build | resident (median) | peak | spread |
+|---|---|---|---|
+| W4A16, second copy (old) | 4591 MB | 5508 MB | 0.2% |
+| **W4A16, one layout** | **3976 MB** | **4593 MB** | 5.4% |
+| W4A16, `prefill_dequant: false` | 3902 MB | 4522 MB | 2.6% |
+
+**615 MB of resident memory and ~915 MB of peak**, and the one-layout build is
+statistically indistinguishable from `prefill_dequant: false` -- the harness
+refused to rank them (0.4% apart, inside the measurement's own spread). The
+arithmetic predicts 764 MB from the config; ~150 MB of that is allocator
+retention `malloc_trim` does not return.
+
+It is not free. Prefill, pp512, pooled over 7 interleaved passes per arm on
+idle node-1 cores with memory bound to the same node:
+
+| build | resident | pp512 median | range | spread |
+|---|---|---|---|---|
+| two layouts (old default) | 4591 MB | 2190 tok/s | 2117-2345 | 9.7% |
+| **one layout, direct dequant** | **3976 MB** | **1344 tok/s** | 1271-1555 | 18.3% |
+| `prefill_dequant: false` | 3902 MB | 453 tok/s | 451-453 | 0.3% |
+
+So it costs **1.63x prefill** on medians -- and the ratio itself is only good
+to about 1.4-1.8x, because both dequantizing arms swing ~10-18% on this shared
+host while the tinygemm-only arm reproduces to 0.3%. The *existence* and rough
+size of the cost is solid (a 63% gap is far outside that band); the third digit
+is not. An earlier draft of this section said 1.9x, taken from the single
+surviving pass of a grid whose other passes had crashed -- a number the harness
+itself had labelled "single pass -- not a result".
+
+The mechanism is that undoing the kernel's interleave per call costs ~1.6x
+reading the pre-permuted planes (14.4 ms vs 9.2 ms per `gate_up` dequantize at
+8 threads). Two variants were tried and rejected: dequantizing to `[K, N]` and
+using `matmul` instead of `F.linear`, which avoids the element-wise transpose,
+measured *worse* at 20.3 ms, and the blocked form needs either 208 tiny GEMMs
+or the same permutation.
+
+What this changes is the shape of the choice, not its existence:
+
+* against `prefill_dequant: false` it is **strictly better** -- the same memory
+  to within the noise, **2.97x** the prefill, and it does not route multi-row
+  batches into the tinygemm/AMX SIGILL hazard documented in `cpu_int4.py`. That
+  setting is now dominated and should not be used. This comparison is the solid
+  one: the tinygemm-only arm reproduces to 0.3%.
+* against the old default it is a real dial: 615 MB for ~1.6x prefill. On a
+  device chosen for its memory ceiling that is the right side of the trade;
+  on one with headroom it is not. `direct_dequant: false` in the checkpoint's
+  `quantization_config` selects the old behaviour -- a config field rather than
+  only an env var, because the two paths build different graphs and must not
+  share an AOT compile artifact (see below).
+
+Recovering the prefill would take a C++ de-interleave rather than an ATen
+expression chain, which is a real kernel and not in scope here.
+
+One hazard found while measuring this, worth recording because it presented as
+a crash rather than a wrong number. The two paths build different graphs -- one
+passes `weight_halves` into the custom op, the other passes `None` -- but vLLM
+keys its AOT compile cache on the *config*, so an env-var-only switch shares
+one cached artifact between them. Three passes died with
+`KeyError: 'weight_halves'` inside an AOT-loaded forward after the first pass
+saved an artifact for the other path. The choice is therefore a
+`quantization_config` field, which `ModelConfig.compute_hash` does hash; the
+env override remains for experiments and needs its own `VLLM_CACHE_ROOT`.
+
+### The largest memory item was a RoPE table for a context nobody asks for
+
+The report used to account for memory by subtraction -- weights, imports, and
+an "unexplained ~1.6 GB" left over. A remainder computed from two uncertain
+terms is not a measurement, so `memory_ledger.py` counts instead: unique tensor
+storages keyed by `data_ptr` (so tied weights are counted once), the KV cache
+found by attribute, and every `/proc/self/smaps` mapping bucketed. Allocated is
+separated from resident with `mincore(2)`, because a cache that was allocated
+and never written costs address space, not memory, and counting it would hide
+the remainder rather than explain it.
+
+The first run put 640 MB in an "other" bucket across 114 tensors. Naming the
+largest storages showed what it was:
+
+    model.layers.0.self_attn.rotary_emb.cos_sin_cache   512.0 MB   [1048576, 256]
+    model.layers.3.self_attn.rotary_emb.cos_sin_cache   128.0 MB   [1048576, 64]
+
+Spark advertises a **1 048 576-token context**, and `spark2_5.py` passed
+`max_position=config.max_position_embeddings` straight into `get_rope`, so vLLM
+built `torch.arange(1048576)` cos/sin tables -- two of them, because the
+sliding and full layers are trained with different theta and different partial
+rotary factors, giving one 256-wide table and one 64-wide one. Both fully
+resident by `mincore`. For a deployment whose `max_model_len` is 2048 they need
+**1.25 MB between them**.
+
+vLLM's scheduler cannot produce a position beyond `max_model_len`, so the table
+only has to cover that. Capping it, 3 interleaved passes per arm with
+`VLLM_OMNI_SPARK_ROPE_FULL` toggling the two:
+
+| | settled | peak | init |
+|---|---|---|---|
+| 1M-position tables | 3891 MB | 4019 MB | 13.5 s |
+| **capped to `max_model_len`** | **3273 MB** | **3370 MB** | 13.7 s |
+
+**618 MB settled and 649 MB of peak, for one line and no measurable init
+cost** -- the largest single item found in this study, larger than the single
+weight layout (615 MB) or the blocked repack (489 MB), and unlike those two it
+costs nothing at all. The ledger's storage accounting agrees independently: the
+"other" bucket falls from 640.0 MB to 1.3 MB and unique tensor storage drops
+638.8 MB.
+
+Two things the same ledger exposes, not yet acted on:
+
+* **The KV cache is fully resident** -- 1023.8 MB allocated, 1024.1 MB resident
+  by `mincore`. The "not a memory lever" verdict below was measured while
+  generating 8 tokens; at page level the cache is entirely in core, and it is
+  27% of total memory. That conclusion needs re-opening on this evidence.
+* **`embed_tokens.weight_packed` (128 MB) is kept only because the tinygemm
+  layout was believed un-addressable for a single-row lookup.** That premise is
+  now known to be false -- the deinterleave above reaches any row -- so the
+  row-major copy is probably removable too.
+
+### Peak load memory: the number an edge device actually has to survive
+
+Everything above reports memory *after* the allocator is asked for its arenas
+back. That is the wrong target for a device: `release_after_load()` runs only
+once `load_model` and warm-up have both returned, so a box that cannot hold the
+peak never reaches the release. Sampling `RssAnon` on a timer through the load
+(rather than reading the endpoint) put the W4A16 peak at **5722-5837 MB**
+against ~4480 settled -- a 1.2-1.3 GB spike nothing in the tree had measured.
+
+Most of it was in our own repack path. `unpack_nibbles` expands 4-bit codes to
+int32 -- 4 bytes per weight, eight times the packed form -- because the
+kernel's packer wants int32, and doing that for a whole matrix makes the
+transient the largest object in the process:
+
+| tensor | shape | int32 expansion |
+|---|---|---|
+| `gate_up` | 13312 x 2048 | 104 MiB |
+| `down_proj` | 2048 x 6656 | 52 MiB |
+| `q_k_v_proj` | 3072 x 2048 | 24 MiB |
+| **embedding (tied, so also the head)** | **131072 x 2048** | **1024 MiB** |
+
+against 24 MiB of packed weights for an entire layer.
+
+The packed layout is `[N/64][K][32 bytes]`, so n-blocks are independent and the
+expansion can cover a few thousand rows at a time -- verified byte-identical to
+packing the whole matrix at four shapes plus a ragged last chunk. Measured with
+`VLLM_OMNI_CPU_INT4_REPACK_ROWS` toggling the two in identical code, 3
+interleaved passes, node-local cores:
+
+| | peak | settled | load-time transient | init |
+|---|---|---|---|---|
+| whole-matrix repack | 4502 MB | 3905 MB | 550 MB | 17.7 s |
+| **blocked repack** | **4013 MB** | 3892 MB | **105 MB** | 16.6 s |
+
+**489 MB of peak, for nothing.** Settled memory and init time are unchanged,
+and the repack is in fact *faster* chunked than whole -- 79 ms against 208 ms
+for the embedding at 24 threads, because the working set stays in cache. The
+chunk size is tuned from that sweep: 2048 rows is both the fastest and nearly
+the leanest (16 MiB transient), where 512 costs 116 ms and 32768 costs 129 ms.
+
+Two things this nearly got wrong, both caught by re-measuring rather than
+reasoning. The first blocked pass showed init at 50.2 s against 17.7 -- but it
+is the one pass the harness flagged `[busy]` at load 34.8, and the two clean
+passes read 16.6 and 16.2. And its settled memory looked 239 MB worse, which
+was pass-1 scatter: the medians are 3892 against 3905.
+
+The absolute peak improves less than the 1 GB the arithmetic predicts, because
+the single-layout change above had already removed `to_split_halves`, which was
+doing its own whole-matrix expansion. The two items overlap and their savings
+must not be added.
+
+### Kernel fusion: a negative result
+
+vLLM's CPU build ships `_C::rms_norm` and `_C::fused_add_rms_norm`, and
+`vllm/kernels/vllm_c.py` already wraps both as IR-op implementations -- but
+gates them on `GPGPU_DEVICE = is_cuda_alike() or is_xpu()`, so on CPU every
+norm falls through to the PyTorch path. In isolation the fused kernels are
+**3.6x and 4.5x faster** at Spark's decode shape (5.76 vs 20.99 us; 9.24 vs
+41.14 us). End to end they are **slightly slower**: 72.7 vs 74.0 tok/s.
+
+The reason is that the native norm was never running alone -- inductor fuses it
+into the residual add and the surrounding elementwise work, and an opaque `_C`
+call breaks that fusion for more than the norm itself saves.
+`vllm_omni/model_executor/layers/cpu_ir_ops.py` keeps the registration, off by
+default (`VLLM_OMNI_CPU_FUSED_NORMS=1` to try it on another machine), with the
+measurement recorded. `gelu_and_mul` is deliberately not registered at all: its
+fused CPU kernel is 2.7x *slower* than native at Spark's GEGLU shape (64.11 vs
+23.36 us), and `GeluAndMul.forward_cpu` in vLLM only reaches for it on POWERPC
+anyway.
+
+Two more fusion attempts that lost, recorded so they are not repeated:
+`cpp.min_chunk_size` raised to keep inductor's small elementwise kernels
+single-threaded cost 6% of decode and 19% of prefill; and the head gate written
+as an inline reduction (rather than behind an opaque op) made the whole step
+**1.8x slower** -- 78.8 to 44.5 tok/s -- because inductor's generated loop is
+worse than the matmul it replaced.
+
+## 3. GPU: vLLM-Omni vs SGLang
+
+One L20X (140 GiB HBM), bf16, single sequence, caches disabled on both
+(`enable_prefix_caching=False` / `disable_radix_cache=True`), 2048-token
+prefill so the measurement is not dominated by dispatch overhead.
+
+| engine | init | pp2048 | tg128 |
+|---|---|---|---|
+| vLLM-Omni | 111.3 s | 137 102 tok/s | **451.4 tok/s** |
+| SGLang 0.5.19 | 97.1 s | **142 253 tok/s** | 434.7 tok/s |
+
+A tie within 4% either way: SGLang +3.8% on prefill, vLLM +3.8% on decode.
+
+SGLang has an Intel AMX CPU backend, but the released `sgl-kernel` wheel does
+not ship the CPU ops it needs (`convert_weight_packed` is absent), so SGLang
+could not be measured on CPU without a source build. On the two targets this
+goal is actually about -- CPU and a phone -- SGLang does not run at all today.
+
+## 4. Galaxy S25 (Snapdragon 8 Elite), real device
+
+`vllm_omni/edge/spark_export.py` exports one decode step with the KV cache as
+explicit tensors, sized per layer type: sliding layers get a fixed window at
+any context, full layers grow. Parity against the HF reference is **140.3 dB**
+with identical top-1 and top-5.
+
+### What the device profile said to fix
+
+AI Hub returns per-op cycle counts, and they made the first export's mistake
+obvious. Of a sliding layer at w4a16:
+
+| op | share | what it was |
+|---|---|---|
+| `Output` | 9.9% | handing the recomputed 512-entry window back |
+| `output_..._converted` | 3.1% | quantising that window on the way out |
+| `Slice` x3 | 7.4% | rolling the window to drop the oldest entry |
+| **subtotal** | **20.4%** | **moving a cache-sized tensor across the graph boundary, once per token, in 21 of 28 layers** |
+
+Full-attention layers spent a similar 9.5% on `Concat`, appending the new key
+to the cache.
+
+The fix is a ring buffer: the runtime owns a fixed-size cache and writes the
+new entry into the slot for this position, so the graph returns one K/V entry
+instead of a window and never rolls anything. Ring order needs no fixing up --
+softmax over keys is order-independent and each cached key already carries its
+own rotary phase, so only slots not yet written need masking. The sliding
+cache correspondingly holds `window - 1` entries, because a 512-token window
+counts the current token (transformers' own sliding cache holds 511 for
+exactly this reason).
+
+| layer | roll | ring | |
+|---|---|---|---|
+| sliding, fp16 | 1.689 ms | **1.486 ms** | −12.0% |
+| sliding, w4a16 | 0.917 ms | **0.838 ms** | −8.6% |
+| full, fp16 | 1.559 ms | 1.567 ms | no gain |
+
+Full layers gain nothing, so the default now picks per layer type: ring for
+sliding, roll for full.
+
+### Three things that did not work, and one that changed the answer
+
+**The concat was not the problem.** Joining in score space instead of key
+space removes the concat entirely and is numerically identical (140.3 dB), but
+it needs an explicit max/exp/sum/div in place of a fused `Softmax`, and that
+cannot be quantised -- QNN's context-binary converter exits 14 at w4a16 while
+every fp16 build compiles. Two other shapes failed the same way: a broadcast
+multiply plus `ReduceSum` for the new token's score, and a two-axis broadcast
+multiply for its value contribution. Keeping roll's fused-softmax attention and
+changing only what the layer *returns* captures the win and compiles.
+
+**The cheaper GELU is not cheaper here.** The erf form is 11–13% of every
+layer. The tanh approximation holds 65.2 dB with top-1/top-5 intact, so it was
+adoptable -- but on this NPU it is *slower* (1.535 vs 1.483 ms). The exact form
+stays, which is also the better-fidelity choice.
+
+**Per-graph overhead is negligible, so the per-layer method is sound.**
+Profiling one layer at a time pays the graph's Input/Output boundary once per
+layer, where a real step pays it once for 28. Exporting the architecture's
+repeating 4-layer block (3 sliding + 1 full) measured 6.103 ms against 6.016 ms
+summed from single layers — a 1.4% gap. There is no hidden win in fusing
+layers, and extrapolating from per-layer numbers is legitimate.
+
+**Uncalibrated quantisation produces garbage, which invalidates the earlier
+numbers.** Running the real layer on the device and comparing against the fp32
+reference:
+
+| build | SNR vs reference |
+|---|---|
+| fp16 | 60.6 dB |
+| w4a16, no calibration data | **−1.6 dB** |
+| w4a8, no calibration data | **−1.28 dB** |
+
+Negative SNR means the output is uncorrelated with the truth. Every w4a16
+figure in the first version of this document came from a build in that state.
+Quantisation ranges have to be inferred from activations the model actually
+sees: real ones have `x` at absmax 0.12 and the KV cache at 7.4, where the
+random tensors used for tracing had 0.4 and 2.5.
+
+Re-quantising with activations captured from the reference model on real text
+(4 samples, English, code and Chinese) fixes it:
+
+| scheme | SNR | sliding layer |
+|---|---|---|
+| **w8a16, calibrated** | **36.9 dB** | **0.803 ms** |
+| w16a16, calibrated | 44.7 dB | 1.656 ms |
+| w8a8, calibrated | 5.1 dB — broken | 0.749 ms |
+| w4a16 / w4a8, calibrated | converter fails | — |
+
+int8 *activations* destroy the layer even with calibration, which is the third
+model family in this repo to show it after Qwen3-TTS and MiniCPM-o. int4
+weights do not survive the calibrated path at all. **w8a16 with real
+calibration data is the deployable configuration**, and it is both faster and
+verified where the old w4a16 was neither.
+
+### What is left, and why int4 is not it
+
+Profiling the optimised layer shows the shape of the problem has changed
+completely. `Input` and `Output` -- 26% of the original layer -- are no longer
+in the top ten ops at all. What remains, of 0.803 ms:
+
+| op group | share | can it move? |
+|---|---|---|
+| MLP up + down matmuls | 36.3% | no, this is the model |
+| GELU (fused elementwise) | 17.5% | no, tanh is slower here (measured) |
+| attention matmuls | 10.5% | no |
+| RMSNorm, decomposed | 8.5% | **yes, see below** |
+| cache slice | 4.1% | marginal |
+
+**int4 weights are not the next lever, on two independent counts.** They do not
+compile: AI Hub's calibrated quantise path fails with `QAIRT converter failed
+with exit code 1` for w4a16 and w4a8, with and without `--quantize_io`. And
+they would not help anyway -- on this NPU 4-bit and 8-bit weights time
+identically, for the layer (0.917 vs 0.912 ms) and for the output head (3.868
+vs 3.862 ms). The calibrated *8-bit* build is faster than the uncalibrated
+4-bit one (0.803 vs 0.838). 4-bit buys memory here, not speed.
+
+**The real remaining item is RMSNorm fusion, which calibration costs us.** The
+uncalibrated build contains two fused `rms` ops worth 1.0% of the layer; the
+calibrated build contains eight decomposed `Pow`/`ReduceMean`/`Div`/`Mul` ops
+worth 8.5%. Inserting QDQ nodes around each op breaks QNN's RMSNorm pattern
+match. The calibrated build is still faster overall and is the only correct
+one, so this is a trade worth making today -- but roughly 7.5% of every layer
+is sitting there for whoever can keep the fusion under quantisation.
+(`F.rms_norm` is not a route to it: the TorchScript ONNX exporter has no
+`aten::rms_norm` at any opset.)
+
+### The decode step, end to end
+
+Every component measured on a real S25 at calibrated w8a16, 1k context:
+
+| component | ms each | ms total | SNR |
+|---|---|---|---|
+| sliding layer x21 (ring) | 0.803 | 16.86 | 36.9 dB |
+| full-attention layer x7 (roll) | 0.942 | 6.59 | 37.4 dB |
+| output head x1 | 3.907 | 3.91 | 62.8 dB |
+| **decode step** | | **27.36** | **36.5 tok/s** |
+
+Against the previous best (roll layout, uncalibrated w4a16, plus the same
+head) at 31.01 ms: **11.8% faster, and numerically valid for the first time.**
+
+The output head is 14% of the step and is 98.6% a single 2048x131072 matmul
+tied to the embedding — there is no overhead in it to remove. It was missing
+from the first version of this document entirely.
+
+At 4k context the full-attention layers grow (2.96 ms each at fp16 vs 1.56 at
+1k) while the sliding layers do not move at all, which is what the hybrid
+architecture buys:
+
+| quant | context | Spark hybrid | all-full counterfactual | gain |
+|---|---|---|---|---|
+| fp16 | 1 k | 46.4 ms | 43.7 ms | 0.94x |
+| fp16 | 4 k | 56.2 ms | 83.0 ms | 1.48x |
+
+At 1k in fp16 the hybrid is *slightly slower*: sliding layers rotate all 256
+head dims where full layers rotate 64, and below roughly 2k that RoPE cost
+outweighs the shorter cache. It pays from ~2k onward and keeps paying, because
+the full-layer term grows with context and the sliding term never does.
+
+### Which phone runtime, and which processor
+
+The same exported sliding layer, one Galaxy S25:
+
+| runtime | processor | precision | ms/layer |
+|---|---|---|---|
+| **QNN context binary** | **Hexagon NPU** | **w8a16 calibrated** | **0.803** |
+| TFLite | CPU | int8 | 0.877 |
+| QNN context binary | Hexagon NPU | fp16 | 1.479 |
+| TFLite | Hexagon NPU | fp16 | 1.491 |
+| ONNX Runtime | Hexagon NPU | fp16 | 1.500 |
+| TFLite | Adreno GPU | fp16 | 2.259 |
+| TFLite | CPU | fp16 | 2.958 |
+
+Two readings. At matched fp16 the three NPU runtimes land within 1.4% of each
+other, so the export layout -- static shapes, explicit caches, GQA without
+expanding K/V -- is what sets on-device speed, not the engine; that reproduces
+what the Qwen3-TTS talker step showed on the same device. And the placement is
+the opposite of that TTS result: Spark's decoder wants the NPU (2.0x the CPU,
+1.5x the GPU) because it is matmul-bound, where the TTS vocoder wanted the GPU
+because it was elementwise-bound.
+
+The row that matters for competition is the int8 CPU one. llama.cpp cannot be
+run on an AI Hub device, and on a phone it would use 4-bit weights on the CPU,
+so comparing our NPU path against an fp16 CPU run would flatter us. The
+closest honest proxy AI Hub offers is the same graph, int8, on the S25's own
+CPU: 0.877 ms against our 0.803 ms, so the NPU path is **9.2% ahead of the
+processor and precision class llama.cpp would actually run in**. That is a
+proxy, not llama.cpp itself.
+
+Two independent checks say the proxy is not flattering us either. A 1.11 GiB
+Q4_K_M build on the S25's ~137 GB/s LPDDR5X has a hard bandwidth ceiling of
+7.9 ms/token at 100% of peak and 13.2 ms at a more realistic 60% — against our
+measured 27.4 ms for a step that includes the output head the bandwidth model
+ignores. Per layer, the bandwidth bound gives roughly 0.28-0.47 ms where the
+measured int8 CPU run takes 0.877 ms, i.e. real CPU execution lands at about a
+third of the bandwidth ceiling, which is the usual outcome on a phone.
+
+## 5. Driving an Android phone
+
+Spark-X2.5 is text-only, so any harness must drive from the accessibility tree
+rather than screenshots. Surveyed:
+
+| harness | transport | text-only a11y mode | local OpenAI endpoint | role |
+|---|---|---|---|---|
+| droidrun/mobilerun | ADB + Portal accessibility service | yes | yes, OpenAI-compatible | control |
+| ghost-in-the-droid/android-agent | ADB + uiautomator, no app on device | yes | lists vLLM, base_url undocumented | control |
+| mobileClaw | on-device app, accessibility | vision-oriented | n/a | control |
+| AndroidWorld (DeepMind) | Android emulator | yes | via custom agent | **evaluation**, 116 scored tasks |
+
+Recommendation: droidrun/mobilerun for control (it is the only one whose
+OpenAI-compatible base URL is documented, so it points at a local vLLM server
+unchanged), AndroidWorld for measuring whether the agent actually succeeds.
+
+`analysis/experiments/spark_edge/android_agent/` implements the action space
+those harnesses share -- `tap`, `type_text`, `swipe`, `press_key`, `open_app`,
+`task_complete`, all indexed against accessibility-tree elements, with the
+`adb shell input` command each maps to -- and runs Spark against ten Android
+screens through vLLM's OpenAI server.
+
+Serving works with vLLM's stock parsers: Spark emits
+`<tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call>`,
+which is GLM-4.5's format, so `--tool-call-parser glm45 --reasoning-parser
+glm45 --enable-auto-tool-choice` parses it as-is. No new parser needed.
+
+| config | correct tool | fully correct action | median decision |
+|---|---|---|---|
+| thinking on | 7-8 / 10 | 5-6 / 10 | 8.9 s |
+| thinking off | 5 / 10 | 4 / 10 | **0.61 s** |
+
+(16 CPU cores, bf16, measured under concurrent benchmark load, so latency is
+pessimistic.)
+
+Thinking is worth roughly two of ten decisions and costs 14.6x the latency.
+With thinking off the model collapses toward answering `tap` for everything.
+With it on, the failure mode is the opposite: on the one genuinely ambiguous
+screen (reach Developer options, which is not visible and needs scrolling) it
+reasoned past a 1200-token budget without ever closing `<think>`, so vLLM's
+reasoning parser -- which only emits reasoning once the block closes --
+returned empty content and no tool call at all. A phone agent on this model
+needs a step budget and a fallback, not just a bigger cap.
+
+The clearest wrong action was tapping an already-on Wi-Fi switch instead of
+calling `task_complete`, i.e. undoing the task it had completed.
+
+No Android device or emulator is attached to this host (`/dev/kvm` absent), so
+these numbers measure the model's decisions over real accessibility-tree
+layouts, not a live device.
+
+### 5b. Driving it in a loop, not one decision at a time
+
+The numbers above measure single decisions over fixed screens. A phone agent
+is a loop, and a loop fails differently. `analysis/experiments/spark_edge/
+android_agent/` is now a working toolkit -- uiautomator XML to an indexed
+element list, an ADB device and a scripted simulator that emits the same XML,
+a tool executor, a tolerant tool-call parser and the loop itself (21 tests,
+no phone and no server required).
+
+**Stock `glm45` does not always parse Spark.** Earlier this note said Spark's
+tool format is GLM-4.5's, so vLLM's parsers work as-is. True on 4 of 5 screens.
+On the Wi-Fi toggle screen -- the one where a switch's state decides the
+action -- the model left the format and emitted
+
+    [tap]
+    {"tool": "tap", "parameters": {"index": 1}
+
+unterminated, with `tool_calls` empty. An agent reading only `tool_calls` sees
+a step with no action. `tool_parse.py` recovers the bracketed form, GLM blocks
+and bare JSON, repairing unbalanced braces, and reports every recovery rather
+than silently repairing.
+
+**Navigation survives quantization; recognising completion does not.** Same
+harness, same simulated phone, one run per cell:
+
+| build | decode | "Turn on Wi-Fi." | "Set an alarm for 7:30 AM." |
+|---|---|---|---|
+| bf16 | 41.0 tok/s | done in 4 steps | wrong app (Settings, not Clock), toggles Airplane mode, stuck |
+| **W4A16 g128** | 80.0 tok/s | done in 4 steps | right app, full navigation, then no tool call |
+| W4A8 g128 | 86.6 tok/s | turns Wi-Fi on, then taps it off again (2 runs) | reaches Alarm, then loops |
+
+Every build walks home -> Settings -> Network -> Wi-Fi correctly. Only some
+notice the switch now reads `checked=true` and stop. W4A8 -- the fastest build,
+and the one a speed-first selector picks -- ends with the task undone.
+
+This is what section 2c's fidelity numbers meant. W4A8 reproducing 1/12 greedy
+prompts against its own bf16, and W4A16 2/12, looked like a small difference on
+a proxy metric; it is the difference between finishing a task and reversing it.
+**For agentic work the 4-bit choice is W4A16**, which `weight_path.py` selects
+under `priority="fidelity"` -- 8% of decode for the difference.
+
+Adding an explicit "if the screen already shows what was asked, call
+task_complete" rule to the system prompt did not rescue W4A8. Nor did feeding
+back "the screen did not change", which bf16 was told five times on the alarm
+task before swiping again.
+
+Limits: two tasks, one run each, on a simulated phone -- enough to show a
+failure mode reproduces, not to rank builds by success rate. The simulator
+emits real uiautomator XML, so the parser, the index-to-coordinate mapping and
+the stopping rules run on the path hardware drives; whether a real Settings app
+is laid out the way these fixtures assume is still untested, because this host
+has no `adb`, no emulator and no `/dev/kvm`.
+
+## 6. Where it stands against the goal
+
+- **Runs on CPU**: yes. Faster than llama.cpp at matched precision on both
+  prefill (8.7-15.4x) and decode (1.19-2.65x), and an int8 path adds a further
+  1.6x.
+- **Runs on Galaxy S25**: yes, measured on real hardware -- a full decode
+  step, output head included, is 27.4 ms at 1 k context (36.5 tok/s) in
+  calibrated w8a16, with every component verified at 36.9-62.8 dB against the
+  reference. That is 11.8% faster than the first working version and the first
+  configuration whose output is actually correct: the uncalibrated w4a16 builds
+  quoted earlier score -1.6 dB, i.e. uncorrelated with the truth.
+- **Beats every edge engine**: on prefill, yes, against everything that can
+  run the model, at any precision -- 4.7x over llama.cpp's best quantized
+  build at 32 threads. On decode it depends on load: llama.cpp Q4_K_M wins
+  single-stream by 1.6x, vLLM int8 wins from batch 16 and leads 1.72x at batch
+  64. The single-stream gap is not closable on vLLM 0.28's CPU backend, which
+  ships no dense int4 kernel.
+- **Beats SGLang**: on decode by 3.8% on GPU (prefill 3.8% the other way), and
+  by default on CPU and phone, where SGLang does not run -- its released
+  `sgl-kernel` wheel has no CPU ops and it has no phone backend at all.
+- **Fastest on the phone**: of every runtime and processor combination
+  measurable on the S25, our QNN NPU path is quickest, including a 9.2% margin
+  over the int8-on-CPU class llama.cpp would run in. llama.cpp itself cannot be
+  run on an AI Hub device, so that specific comparison is a proxy.
+- **Phone control**: the action space, an agent loop and measured decision
+  accuracy are in `analysis/experiments/spark_edge/android_agent/`; serving
+  needs no new parser, just `--tool-call-parser glm45 --reasoning-parser
+  glm45`.
+
+Open items: no dense int4 on CPU (upstream kernel gap); int8 long-context
+fidelity would benefit from a smarter scheme than round-to-nearest; on the NPU
+the next ~7.5% per layer is RMSNorm fusion, which calibration currently breaks
+(int4 weights are neither available nor useful there -- 4-bit and 8-bit time
+identically on this hardware); and none of the phone-agent numbers have been run against a live device, since
+this host has no emulator (`/dev/kvm` absent).

docs/analysis/tts_edge_design.md

@@ -0,0 +1,148 @@
+# Edge TTS design (vLLM-Omni-compatible)
+
+Scope: run vLLM-Omni's autoregressive-plus-codec TTS pipelines on edge devices with the same chunking, state, and interruption semantics as the server, using llama.cpp, MNN, or ExecuTorch as the execution runtime. Reference model: **Qwen3-TTS-12Hz (0.6B / 1.7B)**, because it is the only family implemented natively in vLLM-Omni, llama.cpp, and MNN at the analyzed commits [Verified: [engines/vllm_omni_models.md](engines/vllm_omni_models.md) §2, [engines/llama_cpp.md](engines/llama_cpp.md) §9, [engines/mnn.md](engines/mnn.md) §9]. Secondary references: MOSS-TTS (transformer-only codec with ring KV), CosyVoice3 (flow-matching code2wav), Voxtral TTS (ExecuTorch streaming runner).
+
+## 1. Analysis of the pipeline being ported
+
+### 1.1 Conditioning
+
+| Input | Qwen3-TTS (server) | Edge implication |
+|---|---|---|
+| Text | Tokenized by the HF tokenizer; the task template (CustomVoice / VoiceDesign / Base) is built inside the model by `Qwen3TTSPromptEmbedsBuilder.build_prompt_embeds` in `prompt_embeds_builder.py`, called from the talker; the example only supplies `additional_information` and estimates prompt length [Verified] | Tokenizer must ship on device (llama.cpp vocab, MNN `.mtok`, ExecuTorch tokenizers submodule). The prompt-embeds builder is engine code and the parity source for the edge orchestrator's template construction. |
+| Speaker | ECAPA speaker encoder on reference audio (`qwen3_tts_talker.py#L202`) or predefined speaker id; optional in-context codes from a Mimi encoder [Verified] | llama.cpp and MNN both run the ECAPA encoder on device (`qwen3tts-spkenc.cpp`; MNN requires `--ref_audio`, zero-embedding path unsupported [Verified]). Speaker embeddings should be **computed once and cached** as a small vector; the encoder does not need to be in the hot path. |
+| Instruction / style | Text tokens in the template [Verified] | Same as text. |
+
+### 1.2 Generation type
+
+Autoregressive over codec frames at **12.5 Hz** (80 ms per frame): the talker predicts codebook 0 per step and a 5-layer code predictor fills the remaining 15 codebooks, re-prefilling per codebook without a KV cache [Verified, `common/qwen3_code_predictor.py#L996`]. In vLLM-Omni the code predictor is captured in dedicated CUDA graphs (`talker_mtp`); in llama.cpp it runs with on-graph top-k/top-p [Verified]. There is no diffusion in this family; CosyVoice3 (CFM, 10 Euler steps, CFG batch 2) and IndexTTS-2 (S2Mel DiT) are the diffusion-flow variants and cost proportionally more per chunk [Verified].
+
+### 1.3 Codec / vocoder
+
+`Qwen3TTSTokenizerV2Decoder`: RVQ embedding sum → causal conv → 8-layer sliding-window (72 frames) transformer → 2×2 upsampling → BigVGAN/Snake stack, fp32, 24 kHz, 1920 samples per frame [Verified]. Cross-chunk state: conv context (2 + 12 frames), a reference-prefix `DynamicCache`, and the 72-frame attention window (`modeling_qwen3_tts_tokenizer_v2.py#L803`) [Verified]. On the server this runs as an `LLM_GENERATION` stage with stateful CUDA graphs per reachable chunk length.
+
+### 1.4 Streaming and first-audio latency structure
+
+Server schedule (from `deploy/qwen3_tts.yaml` and measured in [experiments.md](experiments.md)):
+
+```
+talker step 1 ──► 1 frame ──► code2wav ──► first chunk (1,920 samples)      TTFA ≈ 28 ms on L20X
+talker steps 2..26 ──► 25 frames ──► code2wav with 72-frame left ctx ──► chunk 2 (48,000 samples)  +160 ms
+...
+EOS ──► remaining frames ──► final message carries the tail
+```
+
+TTFA is bounded below by: text prefill + one talker decode + code predictor (15 sub-steps) + decoder on 1 frame with empty context + transfer. On the server the transfer is a shared-memory hop; on device it is a function call.
+
+### 1.5 Real-time factor, quality, interruption, memory, thermals
+
+| Concern | Server fact | Edge design consequence |
+|---|---|---|
+| RTF | 0.08–0.10 on L20X (measured); repo-reported 0.16 on H200 at concurrency 1 | Per-frame budget on device is 80 ms for talker + predictor + decoder amortized. From the CPU baseline (409 M LLM at 135 tok/s Q4_0 on 4 x86 threads, measured), the AR part of a 0.6B talker is plausible on ARM big cores; the decoder (BigVGAN + 8-layer transformer per 25-frame window) is the unknown and must be measured first [Unknown]. |
+| Quality | fp32 decoder, BF16 talker; quantization "scoped to the AR LM stage" in repo docs | Quantize the talker (Q4/Q8, W4 block-64); keep the decoder fp16 at most; validate with the parity harness (§5). |
+| Interruption | `AsyncOmni.abort` → Orchestrator → per-stage abort; decoder state released in `on_requests_finished` [Verified] | Edge orchestrator needs the same three-part abort: stop talker loop at the next frame, drop queued codec frames, flush/reset decoder state; playback ring buffer must be cleared by the app. |
+| Memory | Stage 0 asked for 30 % of a 140 GB card by config, not need [Unknown actual] | Weights: 0.6B talker ≈ 0.4 GB at Q4, ≈ 1.2 GB at fp16 [Inferred from parameter count]; decoder ≈ 200 M params (repo doc for Qwen3-Omni code2wav) ≈ 0.4 GB fp16 [Inferred]; KV for ≤4096 positions small. Target under 2 GB resident for the 0.6B path. |
+| Thermals | N/A on server | Sustained decode on phone big cores throttles within minutes; design must allow moving the decoder to GPU/NPU (MNN OpenCL/QNN, ExecuTorch CoreML/QNN) and keep the talker on CPU, or vice versa, without changing the orchestrator. |
+
+## 2. Design
+
+### 2.1 Architecture
+
+```mermaid
+graph TB
+  subgraph app["Application (Android/iOS/Linux)"]
+    UI["Text + speaker request"] --> ORCH
+    ORCH["Edge TTS orchestrator (C++)<br/>mirrors PipelineConfig: talker → code2wav<br/>chunk schedule 1 / 25 / ctx 72<br/>abort + backpressure"]
+    ORCH --> RING["Audio ring buffer<br/>(owner: app)"]
+    RING --> OUT["Audio sink"]
+  end
+  subgraph adapters["Adapter interfaces"]
+    TA["ITalkerAdapter<br/>prefill(tokens, spk) / step() → 16 codes"]
+    DA["IDecoderAdapter<br/>decode(frames, is_first, is_last) → PCM<br/>reset()"]
+    SA["ISpeakerAdapter<br/>embed(wav) → vector (cached)"]
+  end
+  ORCH --> TA
+  ORCH --> DA
+  ORCH --> SA
+  subgraph rt["Runtime bindings (choose per target)"]
+    L1["llama.cpp: libmtmd Qwen3-TTS graphs<br/>(CPU/Metal/OpenCL/Hexagon/CUDA)"]
+    M1["MNN: Llm talker + code_predictor + speech_decoder modules<br/>(CPU/OpenCL/QNN/Metal/CoreML)"]
+    E1["ExecuTorch: talker.pte + predictor.pte + decoder.pte<br/>(XNNPACK/CoreML/QNN/CUDA)"]
+  end
+  TA --> L1
+  TA --> M1
+  TA --> E1
+  DA --> L1
+  DA --> M1
+  DA --> E1
+```
+
+### 2.2 Adapter interfaces [Proposal]
+
+```cpp
+// Talker: one AR frame per step; returns 16 codebook ids (codebook 0 from the LLM, 1..15 from the predictor)
+struct ITalkerAdapter {
+  virtual bool prefill(const std::vector<int32_t>& text_tokens, const SpeakerCond& spk) = 0;
+  virtual bool step(std::array<int32_t,16>& codes, bool& eos) = 0;      // ≤ 80 ms budget
+  virtual void set_sampling(const Sampling& s) = 0;                     // temp 0.9, top_k 50 (deploy default)
+  virtual void reset() = 0;                                             // drops KV, ready for next utterance
+};
+// Decoder: stateful across chunks; caller decides chunking (1, then 25) and supplies left context policy
+struct IDecoderAdapter {
+  virtual bool decode(const int32_t* codes, int n_frames, bool first, bool last, PcmBuffer& out) = 0;
+  virtual void reset() = 0;                                             // clears conv ctx, window, prefix cache
+  virtual int left_context_frames() const = 0;                          // 72 for Qwen3-TTS
+};
+struct ISpeakerAdapter { virtual bool embed(const float* pcm, int n, int sr, std::vector<float>& out) = 0; };
+```
+
+Orchestrator loop (per utterance). The chunk schedule is a parameter mirrored from the deploy YAML, not a constant: the measured default (1 frame, then 25) gives an ≈80 ms first chunk followed by a ≈160 ms wait, i.e. a playback underrun if playback starts on the first chunk; the YAML's `codec_chunk_ramp` (for example 4, 4, 8, 16, 25) or `codec_chunk_adaptive` remove that cliff and should be the edge default. Gate on uninterrupted-playback latency (time until playback can start without a later gap), measured separately from first-chunk arrival.
+
+1. `spk = cache.get_or(ISpeakerAdapter::embed)`; `talker.prefill(tokens, spk)`.
+2. Run `talker.step` until `schedule.size(0)` frames are collected (1 frame in the measured baseline; 4 with the example ramp) → `decoder.decode(frames, first=true)` → push PCM (this is the TTFA path).
+3. For chunk index i ≥ 1, accumulate `schedule.size(i)` frames (ramp or adaptive, converging to 25; or until EOS) → `decoder.decode(frames, first=false, last=eos)` → push. `schedule` mirrors `codec_chunk_ramp`/`codec_chunk_adaptive`/`codec_chunk_frames` from the deploy YAML (`chunk_size_utils.py`, `ramp_chunk_size`), and TTFA is compared against a server reference run with the same schedule; 1→25 stays the measured baseline.
+4. On abort: set flag checked before every `talker.step`; discard accumulated frames; `decoder.reset()`; `talker.reset()`; app clears the ring buffer.
+5. Backpressure: if the ring buffer holds more than N seconds (default 2 s, one chunk), pause the talker loop rather than running ahead; this bounds memory and makes interruption cheap.
+
+### 2.3 State and buffer ownership
+
+| State | Owner | Lifetime | Notes |
+|---|---|---|---|
+| Talker KV cache | Runtime (llama.cpp `llama_kv_cache`, MNN `CPUKVCacheManager`, ExecuTorch in-graph buffer) | Utterance | Sized for prompt + max frames (deploy `max_model_len 4096`); reset per utterance. No paging needed at batch 1. |
+| Code predictor scratch | Runtime | Step | No KV (re-prefill per codebook); llama.cpp/MNN implement this already. |
+| Decoder conv/window/prefix state | Decoder adapter | Utterance | Must survive between `decode` calls; `reset()` on abort or EOS. |
+| Frame accumulator (≤25 × 16 ints) | Orchestrator | Chunk | Tiny; discarded on abort. |
+| PCM ring buffer | Application | Session | Sized to 2–4 s; cleared on abort; the only buffer the audio thread touches. |
+| Speaker embedding cache | Application | Session | Keyed by reference audio hash. |
+
+### 2.4 Stage placement per target [Proposal]
+
+| Target | Talker | Decoder | Runtime | Rationale |
+|---|---|---|---|---|
+| ARM Linux board / phone CPU | CPU Q4 (KleidiAI) | CPU fp16 | llama.cpp or MNN | Both have the full path today; MNN adds NNAPI/QNN escape hatch. |
+| Apple Silicon | Metal (llama.cpp) or CPU | Metal (llama.cpp) / CoreML (ExecuTorch) | llama.cpp first; ExecuTorch for ANE | llama.cpp ships xcframework; ExecuTorch has CoreML/MLX delegates. |
+| Android with Qualcomm NPU | CPU or Hexagon HTP (llama.cpp) | QNN (MNN or ExecuTorch) | MNN (single engine) | MNN has QNN online/offline compile and an Android app to host it. |
+| Jetson | CUDA (llama.cpp) or vLLM-Omni as-is | CUDA | vLLM-Omni (S1) or llama.cpp | S1 gives the reference numbers on the same box. |
+
+## 3. Hybrid (edge/cloud) variant
+
+Talker in vLLM-Omni (cloud or Jetson server), decoder on device. The wire payload is the codec token stream: 16 × int16 per frame at 12.5 Hz = 400 B/s of raw payload before framing and transport overhead, far below any audio stream; the on-device decoder needs only the 72-frame context it already keeps [Inferred]. Integration is more than an adapter: `serving_speech_stream.py`'s `_generate_and_send` consumes PCM from the full pipeline, so a codec-token mode needs a defined wire contract (stage termination after the talker, frame ordering, decoder state/revision, EOS and cancel), most naturally as a talker-only deploy profile plus a new stream message type; the `OmniPayload` key `codes` exists for the intra-pipeline representation [Verified]. S4-first is conditional on prototyping this contract and measuring the on-device decoder. This variant gives server-quality talker sampling, device-local audio with no PCM over the network, and a clean interruption story (client stops consuming; server abort as today).
+
+## 4. Runtime-specific integration notes
+
+- **llama.cpp**: the audio-gen helper (`mtmd-helper-gen.cpp`, `step_prompt/step_gen/get_output`) is already frame-stepped with caller-owned state; `tools/tts/tts.cpp` shows the loop. Work: expose chunked decode (1/25 frames) instead of one WAV, add abort check per step, add a server endpoint later. The decoder window (72 frames) is hard-coded in `clip.cpp` per the report; keep it in sync with the deploy YAML.
+- **MNN**: `Llm::generateTTS` runs the decoder once per utterance; work: split `speech_decoder` invocation into chunked calls carrying conv/window state (the Qwen2.5-Omni path already streams with a callback and worker threads, so the pattern exists in `omni.cpp`). Export via `llmexport.py` (W4 default) and verify the fp32 requirement of the decoder on GPU (the Omni DiT is forced fp32 on GPU).
+- **ExecuTorch**: model the runner on `examples/models/voxtral_tts/voxtral_tts_runner.cpp` (`AudioChunkCallback`, `streaming_chunk_frames_`, left context). Export three programs: talker (`export_llm` with a Qwen3 config; the code-predictor as a separate method), decoder (from `modeling_qwen3_tts_tokenizer_v2.py`, with explicit state tensors as method I/O in the static-attention style so QNN/CoreML can run it). Upper-bound dims: 25 + 72 frames for the decoder, 4096 for the talker.
+
+## 5. Parity and validation
+
+- Oracle: vLLM-Omni offline run of the same text/speaker with `temperature 0` (or a fixed seed) via `Omni.generate`, saving codec tokens and PCM; the session driver in [experiments/vllm_omni_tts/tts_stream_bench.py](experiments/vllm_omni_tts/tts_stream_bench.py) already records chunk boundaries.
+- Metrics: token-level agreement of codebook 0 under greedy decoding (talker parity), PCM SNR of the decoder given identical codes (decoder parity), then perceptual checks (ASR WER on the 12 benchmark prompts, speaker similarity via the same ECAPA encoder).
+- Streaming metrics identical to the server definitions in [experiments.md](experiments.md): TTFA, inter-chunk gap p95, RTF (streamed + tail), plus device-only metrics: resident memory, CPU/GPU utilization, and SoC temperature over a 10-minute continuous synthesis run.
+
+## 6. Blockers and unknowns
+
+- Decoder cost on ARM CPU/GPU is unmeasured; if a 25-frame window exceeds 2 s of compute on the target, use smaller steady chunks (the server supports `codec_chunk_ramp` and adaptive chunking, so the schedule is a parameter, not a redesign) [Verified server feature; edge cost Unknown].
+- llama.cpp and MNN Qwen3-TTS conversions come from their own converters (GGUF, `.mnn`); numerical parity with vLLM-Omni was not measured here.
+- Voice cloning (Base variant) requires the ECAPA encoder and reference-audio codes on device; only speaker-embedding mode is confirmed in MNN [Verified].
+- Checkpoint/task parity is not established across engines: the server benchmark used CustomVoice 1.7B, llama.cpp's README demonstrates 1.7B Base, and MNN's demo requires reference-audio cloning and hard-codes `hiddenSize = 1024` in `Talker::generateQwen3TTS` (`omni.cpp`). Family support does not imply every size/task variant runs everywhere; Phase 0 must pin one checkpoint and task per engine before comparing.
+- No edge hardware was available; every device number above is a projection to be replaced by Phase 0 measurements in [implementation_roadmap.md](implementation_roadmap.md).

docs/analysis/vla_edge_design.md

@@ -0,0 +1,131 @@
+# Edge VLA design (vLLM-Omni-compatible)
+
+Scope: run vision-language-action policies of the kind vLLM-Omni serves (GR00T N1.7, π0, and by extension InternVLA-A1) on edge devices, with vLLM-Omni's OpenPI endpoint as the behavioural oracle. Inference is separated from control and safety throughout; nothing here assumes hard real-time guarantees. Evidence: [engines/vllm_omni_models.md](engines/vllm_omni_models.md) §3, [engines/vllm_omni.md](engines/vllm_omni.md), the four runtime reports, and the CPU vision measurement in [experiments.md](experiments.md).
+
+## 1. Analysis of the pipelines being ported
+
+### 1.1 Perception
+
+| Model | Vision path (server) | Edge implication |
+|---|---|---|
+| GR00T N1.7 | Qwen3-VL encoder from Cosmos-Reason2-2B, backbone truncated to 12 layers, SDPA, eager, batch 1 [Verified] | ViT + 12-layer LLM per control step; exportable (HF `AutoModel` wrapper). |
+| π0 | PaliGemma (SigLIP-So400m producing 256 × 1152 tokens per 224² image, per `modeling_pi0.py`; + Gemma-2B) with up to 3 cameras, prompt ≤ 48 tokens, fp32 [Verified] | Three So400m encoder passes per step. The CPU measurement (0.6–1.2 s per 512 px tile of SmolVLM's smaller SigLIP encoder at 8 x86 threads in llama.cpp) shows only that vision encoding dominates on CPU; it is not an estimate for π0's encoder, which must be measured from the actual graph. |
+| DreamZero | 14B DiT world model, 16 UniPC steps, 3 cameras 352×640, per-session paged KV of 603 MiB [Verified] | Not an edge candidate at this size; kept as a cloud tier only. |
+
+### 1.2 Structured inputs and fusion
+
+Proprioception enters GR00T through a per-embodiment `CategorySpecificMLP` (32 embodiments, padded to 132 dims) and π0 through its state projection; language is tokenized once per task; fusion happens in the backbone (GR00T: `AlternateVLDiT` cross-attends to vision/text tokens; π0: Gemma expert attends to PaliGemma KV) [Verified]. In vLLM-Omni the observation dict travels in `OmniDiffusionSamplingParams.extra_args["robot_obs"]` set by `entrypoints/openpi/serving.py`, and never through `OmniPayload` [Verified]. Edge design: the observation struct is defined by the application and passed as plain tensors; there is no connector layer to preserve.
+
+### 1.3 Action generation and chunking
+
+| Model | Head | Steps | Horizon × dim | Output |
+|---|---|---|---|---|
+| GR00T N1.7 | `AlternateVLDiT` 16 blocks × 1536 | 4 Euler | 40 × per-embodiment (DROID: eef_9d 9, gripper 1, joint 7) | dict of action tensors [Verified] |
+| π0 | Gemma-300M expert, flow matching | 10 Euler | 50 × 32 | (50, 32) [Verified] |
+| InternVLA-A1 | flow head | 10 | 50 × 32 | offline only [Verified] |
+
+All are **stateless across inferences**: one observation → one action chunk; no KV reuse across control steps in the served code (eager, batch 1) [Verified]. Within one inference π0's `sample_actions` builds a prefix KV that the denoise loop reuses and draws Gaussian noise unless `noise` is passed [Verified]. For edge this means no persistent KV between steps, only weights and per-inference scratch, and that parity tests must pass the noise explicitly to make a step a deterministic function of the observation.
+
+### 1.4 Temporal state, latency, jitter, stale inputs
+
+- Temporal state lives in the **controller**, not the model: the action chunk (40–50 steps) is executed over several control ticks while the next inference runs [Inferred from the OpenPI one-request/one-chunk protocol]. The edge design keeps an explicit chunk buffer with timestamps.
+- Latency per inference = preprocessing + encoder(s) + backbone + K flow steps. On the server there is no published per-step number in tree; InternVLA docs mention 38 s for its offline example (unmeasured claim) [Unknown].
+- Jitter sources on device: thermal throttling, GPU/NPU queue contention with the camera pipeline, garbage collection if any managed runtime is involved, and dynamic shapes (never allow them in the hot path).
+- Stale inputs: the observation must carry a capture timestamp; if inference finishes later than a configured staleness bound, the controller keeps executing the previous chunk's remaining actions and the new chunk is time-aligned (skip the steps already elapsed) or discarded.
+
+### 1.5 Task success
+
+Success is measured in simulation first. vLLM-Omni ships simulation evaluation clients for DreamZero (`examples/online_serving/dreamzero/droid_sim_eval_client.py`, `molmospace_dreamzero_eval_demo.py`) that drive an external simulator through the server [Verified]; no simulator itself is in any of the six trees. The plan reuses those clients where the policy matches, still requires an external simulator, and defines success only relative to the server policy running the same episodes.
+
+## 2. Design
+
+### 2.1 Architecture: inference isolated from control and safety
+
+```mermaid
+graph TB
+  subgraph sense["Sensors"]
+    CAM["Cameras (1–3)"] --> PRE
+    PROP["Joint/eef state"] --> OBS
+  end
+  PRE["Preprocess: fixed resize/normalize<br/>(one tile per camera)"] --> OBS["Observation struct<br/>images, state, task tokens, t_capture"]
+  OBS --> INF
+  subgraph inf["Inference process (best-effort, no RT guarantees)"]
+    INF["VLA edge runner<br/>encoder → backbone → K-step flow head"]
+    INF --> CHUNK["Action chunk (H × D) + t_capture"]
+  end
+  CHUNK --> CTRL
+  subgraph ctrl["Control process (fixed rate, higher priority)"]
+    CTRL["Chunk buffer + time alignment<br/>stale check, blending"] --> SAFE["Safety layer<br/>limits, watchdog, stop"]
+    SAFE --> ACT["Actuators"]
+  end
+  CLOUD["vLLM-Omni OpenPI server<br/>(parity oracle / re-planning tier, never in loop)"] -.-> CTRL
+```
+
+Rules [Proposal]:
+
+- Inference and control are separate OS processes (or at least threads with different priorities) sharing a lock-free single-slot mailbox for the latest chunk; the control loop never blocks on inference.
+- Safety (joint limits, velocity limits, watchdog on chunk age, e-stop) is enforced after the controller and does not depend on the model runtime.
+- The cloud endpoint is used only for parity tests and optional slow re-planning (task text updates), never for per-tick actions.
+
+### 2.2 Adapter interfaces [Proposal]
+
+```cpp
+struct IVisionAdapter   { virtual bool encode(const ImageTensor& img, int cam, TokenTensor& out) = 0; };   // fixed shape per camera
+struct IFusionAdapter   { virtual bool fuse(const TokenTensor* vis, int n_cam, const StateVec& s, const TaskTokens& t, Cond& out) = 0; };
+struct IActionHead      { virtual bool sample(const Cond& c, int k_steps, ActionChunk& out) = 0; };        // K Euler steps inside
+struct IVlaRunner {                                                                                        // composes the three, owns scratch
+  virtual bool infer(const Observation& o, ActionChunk& out, InferStats& st) = 0;                         // returns wall time, per-stage ms
+};
+```
+
+`infer` is stateless across calls; `Observation` carries `t_capture`. The runner may keep encoder outputs for cameras whose frames did not change (frame-hash check) as an optimization, which is safe because the model is stateless [Inferred].
+
+### 2.3 State and buffer ownership
+
+| State | Owner | Notes |
+|---|---|---|
+| Weights, runtime scratch | Runner | Static allocation at load (ExecuTorch plans it at export; MNN/llama.cpp reserve at init). |
+| Observation buffers | Sensor process | Double-buffered; the runner reads the latest complete frame. |
+| Action chunk buffer | Controller | Ring of two chunks with timestamps; blending policy at chunk boundaries is the controller's. |
+| Task text tokens | Application | Re-tokenized only when the instruction changes. |
+| Per-embodiment tables (GR00T) | Runner | Constant tensors selected by embodiment id at load. |
+
+### 2.4 Stage placement per runtime [Proposal]
+
+| Runtime | Encoder | Backbone | Action head | Assessment |
+|---|---|---|---|---|
+| **ExecuTorch (primary)** | `vision_encoder` method (CoreML/QNN/XNNPACK) with fixed 224² input | `text_decoder`-style method with static shapes (prompt ≤ 48 tokens, no KV persistence) | Flow loop as a `while_loop`/unrolled K steps inside one method, or K calls from the runner | Static memory bound and zero-copy I/O are available mechanisms, not automatic: outputs must be left unplanned at export for `set_output_data_ptr` to accept them, and every op must be covered by the chosen delegate or it falls to portable CPU kernels; PT2E quantizer recipes per delegate; runner pattern exists in `MultimodalRunner`. Export sources: GR00T `diffusion/models/gr00t/` (HF wrapper), π0 `diffusion/models/pi0/modeling_pi0.py` (plain torch, parity test already runs on CPU [Verified]). |
+| **MNN (secondary)** | `utils/vision.py` exporters (Qwen3-VL/SigLIP families) | `llmexport.py` path with `input_embeds` overloads, or ONNX | DiT via the diffusion engine or ONNX with a host loop | Broadest Android NPU coverage (QNN/Hexagon/NNAPI) and existing apps; batch 1 and static shapes are already the norm. |
+| **llama.cpp** | `clip.cpp` projector families (SigLIP present) | `libllama` with `llama_batch.embd` for proprio/vision embeddings | Not available; would be a new C++ graph (sidecar proposal in [engines/llama_cpp.md](engines/llama_cpp.md)) | Viable for VLM-only reasoning tiers; not for the action head without new engine code. |
+| **ncnn** | ViT/CNN encoder via pnnx | No | Small MLP head only | Component runtime for Vulkan-only devices. |
+
+Hardware placement: keep the encoder on the NPU/GPU and the flow head wherever K small matmuls run with the least launch overhead (often CPU for a 300 M expert at K = 4–10 [Inferred]); measure both.
+
+### 2.5 Latency budget template [Proposal]
+
+For a target control period T (for example 100 ms at 10 Hz):
+
+| Stage | Budget | How enforced |
+|---|---|---|
+| Preprocess | ≤ 5 % T | Fixed resize on the ISP/GPU |
+| Encoders (n cameras) | ≤ 40 % T | One tile per camera; NPU delegate; skip unchanged frames |
+| Backbone + fusion | ≤ 30 % T | Static shapes; quantized weights |
+| Action head (K steps) | ≤ 20 % T | K fixed; unrolled |
+| Slack for jitter | ≥ 5 % T | Monitored p95; if exceeded for N consecutive steps, drop to a lower-rate mode |
+
+Chunking makes the effective requirement softer: with a 40-step chunk executed at 10 Hz, inference may take up to several periods as long as chunk age stays within the staleness bound; the design treats T as the *desired* inference cadence, not a deadline.
+
+## 3. Validation in simulation first
+
+1. **Numerical parity** (no robot): export each module, feed recorded observations from an episode, compare action chunks with vLLM-Omni's OpenPI server (`examples/online_serving/pi0/openpi_client.py`) at fixed seeds; report max-abs and per-dimension RMS error per flow step count.
+2. **Closed-loop in simulation**: the same episodes in a simulator with the edge runner in the loop; success rate versus the server policy on ≥ 50 episodes per task; report jitter (p95 of inference wall time) and chunk-age distribution.
+3. **Fault injection**: delayed frames, dropped cameras, thermal throttling emulation (CPU frequency cap); verify the controller's stale-input path and the safety layer engage as specified.
+4. Only after 1–3 pass on the target device class does hardware-in-the-loop start, with the safety layer active and independent of the model runtime.
+
+## 4. Blockers and unknowns
+
+- No VLA model runs in any of the four runtimes today; every path is an export project [Verified absence].
+- Model sizes: GR00T ≈ 6 GiB BF16 per recipe; π0 backbone is a 2B PaliGemma; both need weight quantization to fit under 4 GB on phones, and the quality impact on actions is unmeasured [Unknown].
+- Vision cost on CPU is the dominant risk (measured on x86 with llama.cpp: 0.6–1.2 s per 512 px SigLIP tile at 8 threads); NPU delegation of the encoder is required for anything above 1–2 Hz on CPU-only boards [Inferred].
+- Weights for DreamZero/π0/GR00T are not in the local cache and their download requires permission; reference numbers on the L20X could not be produced in this session.
+- Simulation evaluation clients exist in vLLM-Omni (`examples/online_serving/dreamzero/droid_sim_eval_client.py`, `molmospace_dreamzero_eval_demo.py`), but the simulator itself and any π0/GR00T-specific harness remain external dependencies.

docs/analysis/vllm_omni_edge_feasibility.md

@@ -0,0 +1,130 @@
+# Feasibility: extending vLLM-Omni to edge devices
+
+Question answered here: can vLLM-Omni's TTS and VLA pipelines run on edge devices, and if so, which pieces of vLLM-Omni are re-used, replaced, or re-implemented? Evidence comes from [engines/vllm_omni.md](engines/vllm_omni.md), [engines/vllm_omni_models.md](engines/vllm_omni_models.md), [engines/vllm.md](engines/vllm.md), the four runtime reports, and [experiments.md](experiments.md). Labels as in [README.md](README.md).
+
+## 1. What "vLLM-Omni on edge" can mean
+
+vLLM-Omni is three things at once [Verified]:
+
+1. **A process/serving model**: `AsyncOmniEngine` → `Orchestrator` → per-stage vLLM engine processes (`StageEngineCoreProc`) or diffusion workers, wired by connectors (default: host-staged shared memory), exposed over HTTP/WebSocket. Initialization observed at 157–188 s for a 2-stage TTS on an L20X in the default configuration; the minimum with `enforce_eager`/parallel stage init was not measured.
+2. **Pipeline semantics**: stage graph per model (`PipelineConfig`), deployment overlay (`DeployConfig`), async-chunk hand-off (1-frame first chunk, 25-frame steady chunks, 72-frame decoder context for Qwen3-TTS), abort propagation, output templates, OpenPI robot protocol.
+3. **Model code**: talkers built on vLLM layers; codecs/vocoders/action heads that are plain PyTorch modules.
+
+Only (2) and parts of (3) are portable. The plan below therefore distinguishes: *re-host* (run the real vLLM-Omni on an edge server), *substitute* (keep vLLM-Omni's control plane, swap execution), *orchestrate lightly* (re-implement the control plane in a small runtime), *split* (edge/cloud hybrid), and *edge-native baseline* (use a runtime's own pipeline and compare).
+
+## 2. Target hardware classes and assumptions
+
+No edge hardware was available in this session (x86 host only). Statements about targets are drawn from what each repository ships for that target and are labelled.
+
+| Target | Assumption made here | vLLM / vLLM-Omni status | Runtime coverage |
+|---|---|---|---|
+| **Embedded NVIDIA (Jetson Orin / Thor class)** | aarch64 Linux, CUDA, 8–64 GB unified memory, JetPack; power caps 15–60 W | vLLM: Tegra detection path and CUDA arch 8.7/11.0 in the build list, no wheels/CI/docs beyond an sm_110a note [Verified]. vLLM-Omni: CUDA platform code is arch-agnostic; aarch64+CUDA is exercised in vLLM CI on GH200 only [Verified]. **Feasible in principle, unproven** [Inferred]. | llama.cpp CUDA (needs explicit sm_87), MNN CUDA (sm87 in list), ExecuTorch CUDA (AOTInductor; Jetson unknown), ncnn (aarch64 CPU + Tegra Vulkan) [Verified] |
+| **ARM CPU (Linux SBC, Graviton-class, phone big cores)** | ARMv8.2+ with fp16/dotprod, often i8mm; 4–8 usable cores; 4–16 GB | vLLM CPU platform is first-class on aarch64 Linux (wheels, ARM CI, oneDNN+ACL, KleidiAI W4A8) [Verified]; vLLM-Omni **has no CPU platform** [Verified]. | All four runtimes; KleidiAI in llama.cpp, MNN, ExecuTorch(XNNPACK) [Verified] |
+| **Apple Silicon (M-series Mac, iPhone/iPad A-series)** | macOS/iOS, Metal, ANE via CoreML, unified memory | vLLM: CPU-only experimental on macOS; Metal via out-of-tree `vllm-metal` (MLX) [Verified]; vLLM-Omni: nothing [Verified]. **Not feasible as-is** on iOS (no Python/torch runtime model). | llama.cpp Metal + xcframework; MNN Metal/CoreML + iOS framework; ExecuTorch CoreML/MLX/Metal via SwiftPM; ncnn via MoltenVK [Verified] |
+| **Mobile GPU/NPU (Android: Adreno/Mali GPU, Hexagon/QNN, MediaTek, Samsung)** | Android 10+, 8–16 GB, thermal throttling within minutes | vLLM/vLLM-Omni: nothing [Verified]. **Not feasible.** | llama.cpp OpenCL (Adreno), Vulkan, Hexagon HTP; MNN OpenCL/Vulkan/QNN/Hexagon/NNAPI; ExecuTorch Vulkan/QNN/MediaTek/Samsung; ncnn Vulkan [Verified] |
+
+## 3. Strategy comparison
+
+```mermaid
+graph LR
+  A[Cloud / edge-server tier<br/>vLLM-Omni as-is] --> B{Strategy}
+  B --> S1["S1 Direct port<br/>run vLLM-Omni on Jetson/ARM"]
+  B --> S2["S2 Backend substitution<br/>keep Orchestrator, swap stage workers"]
+  B --> S3["S3 Lightweight orchestration<br/>re-implement stage graph in C++ on a runtime"]
+  B --> S4["S4 Hybrid edge/cloud<br/>split stages across the network"]
+  B --> S5["S5 Edge-native baseline<br/>runtime's own pipeline, compare"]
+```
+
+| Strategy | What it keeps from vLLM-Omni | Targets it can reach | Blocking facts | Verdict |
+|---|---|---|---|---|
+| **S1 Direct port** (install vLLM + vLLM-Omni on the device) | Everything | Jetson-class CUDA only (aarch64 Linux + CUDA); ARM CPU only if a CPU platform is added to vLLM-Omni | Initialization 157–188 s observed on L20X with default compile/graph capture, likely worse on Jetson [Inferred]; Python + torch 2.13 + transformers + diffusers footprint; two engine processes for two AR/generation stages; no CPU platform in vLLM-Omni (missing implementation, not architectural) [Verified] | Viable for an **edge server** (Orin 64 GB, Thor) where long-lived processes are fine; not viable for phones, MCUs, or cold-start use. Recommended as the *reference tier* only. |
+| **S2 Backend substitution** (vLLM platform plugin whose worker calls llama.cpp/MNN/ExecuTorch) | Orchestrator, schedulers, connectors, APIs | Same as S1 (still Python + vLLM process model) | The vLLM platform plugin contract requires a `WorkerBase`, an `AttentionBackend`, a device communicator and torch tensors flowing through the model runner [Verified]; a worker that delegates a whole stage to a foreign runtime bypasses paged attention, sampling and CUDA-graph logic, so most of what vLLM contributes is lost while its process overhead stays. `OmniPayload` has no action keys; the default shared-memory connector serializes via CPU; the Mooncake/Mori transfer-engine connectors have device fast paths (Mori's `xgmi` backend is intra-node), but none is validated for an edge target [Verified]. | Not recommended as the main path. A narrow variant is useful: **replace only the vocoder/codec stage** with an exported artifact executed by torch (or ExecuTorch's Python runtime) inside the existing `LLM_GENERATION` worker on Jetson. |
+| **S3 Lightweight orchestration** (re-implement `PipelineConfig` semantics in a small C++ runtime on top of llama.cpp / MNN / ExecuTorch) | Stage graph, chunk schedule, state-ownership rules, abort semantics, API shapes, tests as parity oracle | All four target classes | Requires exporting every stage to the runtime's format; talkers are vLLM-bound modules and must be re-exported from the HF checkpoint, not from vLLM-Omni code [Verified]; no shared connector abstraction exists across the runtimes (all in-process, app-owned buffers) | **Recommended primary path** for phones, Apple devices, ARM boards. Two of the runtimes already implement the Qwen3-TTS graph natively (llama.cpp `MTMD_GEN_AUDIO_TYPE_QWEN3TTS`, MNN `Llm::generateTTS`), so the TTS orchestration layer mostly adds streaming, interruption and parity, not model porting. |
+| **S4 Hybrid edge/cloud** (e.g. thinker or talker in the cloud, codec/vocoder on device; or perception on device, policy in cloud) | Cloud stages unchanged; device runs one exported stage | All targets for the device stage | vLLM-Omni's connectors are process/host transports (shared memory, RDMA) not WAN-ready [Verified]; the WebSocket endpoints (`/v1/audio/speech/stream`, `/v1/realtime`, OpenPI) are the practical boundary. Network jitter dominates for VLA; for TTS a codec-token stream is tiny (16 codebooks × 12.5 Hz) which makes talker-in-cloud / vocoder-on-device attractive [Inferred]. | Recommended as the **first deliverable** for TTS (codec tokens over WebSocket, on-device vocoder), **conditional on** a prototype of the new wire contract (the speech stream endpoint's `_generate_and_send` consumes PCM today, so codec-token streaming needs stage termination, token ordering, decoder state/revision, EOS and cancel semantics defined) and on a measured on-device decoder; and as a fallback tier for VLA (never in the control loop, only for re-planning). |
+| **S5 Edge-native baseline** (llama.cpp `llama-tts`, MNN `qwen3_tts_demo`, ExecuTorch Voxtral runner; VLM via `llama-mtmd-cli`) | Nothing but the model weights and the benchmark protocol | All targets | These paths are single-request, mostly non-streaming (llama.cpp CLI, MNN Qwen3-TTS single callback) [Verified] | Required as the **measurement floor** in every phase; also the fastest way to get audio on a device today. |
+
+Assessment of the "must keep vLLM-Omni" premise: an edge deployment that keeps the vLLM-Omni *code* is only sensible on Jetson-class servers (S1). For every other target, the deliverable is "vLLM-Omni-compatible edge pipelines": same models, same chunking and state rules, same API surface, validated against vLLM-Omni outputs. That is S3 with S4 and S5 as bookends.
+
+## 4. Per-runtime assessment for stage placement
+
+Stages considered (from [engines/vllm_omni_models.md](engines/vllm_omni_models.md)): text front-end + speaker conditioning (ECAPA), **AR talker** (Qwen3 LLM with 15/16-codebook code predictor), **codec decoder / vocoder** (RVQ decode + sliding-window transformer + BigVGAN), and for VLA: **vision encoder** (SigLIP/Qwen3-VL ViT), **state/language fusion** (LLM backbone), **action head** (flow/diffusion DiT, 4–10 Euler steps).
+
+### 4.1 llama.cpp
+
+| Aspect | Finding |
+|---|---|
+| Stage placement | Talker + code predictor + code2wav + speaker encoder all inside `libmtmd` for Qwen3-TTS (`tools/mtmd/models/qwen3tts-gen.cpp`, `qwen3tts-spkenc.cpp`; enum in `tools/mtmd/mtmd.h`) [Verified]. Vision encoder for a VLM in `clip.cpp` with ~60 projector types; LLM backbone in `libllama`; **no action head** [Verified]. |
+| Operator gaps | For TTS: none for Qwen3-TTS and Pocket-TTS; other vLLM-Omni families (CosyVoice3 CFM+HiFT, Fish DAC, MOSS codec) would need new graph code in C++ [Verified absence]. For VLA: flow head = new graph; proprio input has to enter through `llama_batch.embd`; `while`-style denoise loop must be driven from the app. `ggml` has conv/pool/1-D ops and flash attention, so the DiT itself is expressible [Inferred]. |
+| State | KV in `llama_kv_cache` (per-seq, ring, quantizable, save/load); code2wav 72-frame window state inside the mtmd helper; caller-owned state blobs in the step API [Verified]. |
+| Copies | Zero-copy mmap weights; scheduler copies at backend splits; audio helper returns float buffers [Verified]. |
+| Integration effort | TTS: low (already runs); work is streaming callback + interruption + parity harness. VLA: medium-high (C++ graph for the action head + app loop). |
+| Where it fits | ARM CPU (KleidiAI), Apple Metal, Android via OpenCL/Adreno, Hexagon HTP, Jetson CUDA; static single binary. |
+
+### 4.2 ncnn
+
+| Aspect | Finding |
+|---|---|
+| Stage placement | Vocoder/VITS-class nets and CNN/ViT vision encoders; the Piper example shows a 5-net app-driven TTS [Verified]. LLM talker only via the batch-1 KV cache path with host-provided masks and RoPE, no tokenizer, no sampler, seq-len-1 export [Verified]. |
+| Operator gaps | No Vulkan for Embed/MatMul/GLU/RNN; no tokenizer; no audio I/O; block-quant LLM only on CPU [Verified]. |
+| State | App-owned `Mat` KV blobs (consume-and-replace); no paging [Verified]. |
+| Copies | Explicit upload/download per extract on Vulkan [Verified]. |
+| Integration effort | High for anything LLM-shaped; low for a stand-alone vocoder or vision encoder. |
+| Where it fits | Only as a **component runtime** (vocoder, image encoder, small MLP/CNN action head) inside an S3 orchestrator on Vulkan-only devices; not recommended as the primary TTS/VLA runtime. |
+
+### 4.3 MNN
+
+| Aspect | Finding |
+|---|---|
+| Stage placement | Full Qwen3-TTS path in the LLM engine (talker → 16-group code predictor → codec embedder → speech decoder; `Llm::generateTTS`, `demo/qwen3_tts_demo.cpp`) and Qwen2.5-Omni talker → DiT → BigVGAN with chunked callback [Verified]. VLA: vision exporters for 16 encoder families, DiT/flow runtime in the diffusion engine, `input_embeds` overloads; no policy model [Verified]. |
+| Operator gaps | Qwen3-TTS decoder runs as **one call per utterance** (single callback) rather than chunk-streamed; GPU backends ignore `attention_mode` (KV quant/flash only on CPU/Metal) [Verified]. Action head: DiT ops exist; proprio MLP + embodiment lookup is ordinary graph code [Inferred]. |
+| State | `KVMeta` per forward, contiguous KV with quant and mmap-to-disk; codec state in engine worker threads; `Module::clone` shares weights [Verified]. |
+| Copies | Per-op CPU fallback wraps; NPU subgraph compile is all-or-nothing [Verified]. |
+| Integration effort | TTS: low-medium (add chunked streaming to the Qwen3-TTS path, mirror vLLM-Omni's 1/25/72 schedule). VLA: medium (export encoder + backbone + head via `llmexport.py`/ONNX, write the control loop in C++ against `Module`). |
+| Where it fits | Android (CPU/OpenCL/QNN/Hexagon/NNAPI), iOS/macOS (Metal/CoreML), OpenHarmony, Jetson CUDA; the only runtime with in-tree Android/iOS chat apps and an Omni engine. |
+
+### 4.4 ExecuTorch
+
+| Aspect | Finding |
+|---|---|
+| Stage placement | LLM runner (`TextLLMRunner`, `MultimodalRunner`) for talker/backbone; Voxtral TTS example shows LM + flow head + codec decoder as two `.pte`s with a chunked streaming callback (25 frames, 5 initial, 25 left context) [Verified]. VLA: any sub-module exportable with `torch.export`; `while_loop` supported inside graphs; static planning and zero-copy I/O suit fixed-rate loops [Verified]. |
+| Operator gaps | Per delegate: QNN/Ethos-U static shapes only; CoreML/XNNPACK coverage decided by partitioner; unsupported ops fall to portable CPU kernels, which is correct but slow [Verified]. The Qwen3-TTS decoder's Snake activations and sliding-window attention have no delegate-specific kernels [Unknown until exported]. |
+| State | KV as in-graph mutable buffers with upper-bound length (or static-attention I/O for NPUs); runner tracks position; no paging [Verified]. |
+| Copies | Inputs aliased; outputs zero-copy only when the export leaves them unplanned (`Method::set_output_data_ptr` rejects memory-planned outputs; see `docs/source/compiler-memory-planning.md`); `_h2d/_d2h` only on CUDA delegates [Verified]. |
+| Integration effort | TTS: medium (export talker + code predictor + decoder; write a Qwen3-TTS runner modelled on the Voxtral runner). VLA: medium (export three modules; runner loop; QNN/CoreML quantizer recipes). Highest AOT rigor (memory bound known at build time), which is what a safety-reviewed control path wants. |
+| Where it fits | Apple (CoreML/MLX), Qualcomm QNN, MediaTek/Samsung NPUs, XNNPACK ARM CPU, Cortex-M/Ethos-U for tiny heads; CUDA on Linux (Jetson unknown). |
+
+## 5. Source-level integration points in vLLM-Omni
+
+These are the places an edge implementation reads from (as specification) or hooks into (for hybrid/parity), all [Verified] in the engine reports:
+
+| Purpose | Path | Use |
+|---|---|---|
+| Stage graph per model | `vllm_omni/model_executor/models/<family>/pipeline.py`, registry `vllm_omni/config/pipeline_registry.py` | Source of truth for stage order, types, and output modality; the edge orchestrator mirrors it |
+| Chunk schedule and decoder context | [`vllm_omni/deploy/qwen3_tts.yaml`](../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml) (`initial_codec_chunk_frames`, `codec_chunk_frames`, `codec_left_context_frames`), `chunk_size_utils.py` | Same numbers on device so audio is bit-comparable |
+| Payload contract | `vllm_omni/data_entry_keys.py` (`OmniPayload`: hidden_states, embed, ids, codes, meta) | Defines what crosses a stage boundary; edge adapters carry the same keys |
+| Streaming/abort semantics | `Orchestrator._handle_abort`, `OmniChunkTransferAdapter`, `WAITING_FOR_CHUNK` | Interruption rules to replicate |
+| TTS output formatting | `vllm_omni/model_executor/models/output_templates.py`, TTS adapter registry | Output shape (`audio`, `sr`, chunk list) the edge API should match |
+| Robot API | `vllm_omni/entrypoints/openpi/` (`serving.py` puts observations into `robot_obs`), `diffusion/output_formatter.py` (`actions`) | Wire protocol for hybrid VLA and for parity tests |
+| Parity oracles | `tests/e2e/offline_inference/test_qwen3_tts_*.py`, `tests/e2e/online_serving/test_{pi0,gr00t_openpi,dreamzero}_expansion.py`, CPU unit tests per family | Reference outputs for edge validation |
+| Model code that is plain torch | Qwen3-TTS `tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py` (decoder), `common/qwen3_code_predictor.py`, GR00T `diffusion/models/gr00t/`, π0 `diffusion/models/pi0/` | Export sources for ExecuTorch/MNN/ONNX |
+| Model code that is vLLM-bound | Talker classes using `Qwen2Model`/`Qwen3Model`, `ParallelLMHead` | Must be re-exported from the HF checkpoint, not from these classes |
+
+## 6. Blockers
+
+| Blocker | Type | Affects |
+|---|---|---|
+| No CPU platform in vLLM-Omni | Missing implementation | S1/S2 on ARM CPU |
+| Initialization of 2.5–3 min for two stages (observed, default config; lower bound unmeasured) | Process model is architectural; compile/graph capture at start is configurable (`enforce_eager`, `compilation_config`) | S1 on any device with cold-start needs |
+| Default inter-stage transport (`SharedMemoryConnector`) is host-staged; device fast paths exist in the Mooncake/Mori transfer-engine connectors (including Mori's intra-node `xgmi` backend on AMD), none validated for an edge target | Missing validation/implementation for edge co-location (repo docs list D2D as roadmap) | S1/S2 latency; irrelevant to S3 |
+| Talkers depend on vLLM layers | Architectural for reuse; not for re-export | S3 must export from HF checkpoints |
+| `OmniPayload` lacks action/video keys; VLA bypasses connectors | Missing implementation | S2/S4 for VLA |
+| Qwen3-TTS decoder streaming in MNN is single-callback; llama.cpp CLI non-streaming | Missing implementation | S3 on MNN/llama.cpp |
+| No VLA model in any runtime | Missing implementation | S3/S5 for VLA |
+| No edge hardware in this environment | Environmental | All measurements beyond x86/L20X |
+| Model weights for VLA (DreamZero, π0, GR00T) not cached; downloads need permission | Environmental | Reference measurements for VLA |
+
+## 7. Decision
+
+- **TTS**: S4 first (talker in vLLM-Omni, on-device vocoder) **only if** the codec-token wire-contract prototype and the on-device decoder measurement pass (the speech stream endpoint's `_generate_and_send` consumes PCM today); otherwise S3 first. Then S3 on llama.cpp (ARM CPU, Apple) and MNN (Android NPU/GPU, iOS) with ExecuTorch for CoreML/QNN targets, S5 numbers reported alongside at every phase. Design in [tts_edge_design.md](tts_edge_design.md).
+- **VLA**: S3 only, on ExecuTorch (primary, for its static memory model) and MNN (secondary, Android/NPU breadth), with vLLM-Omni's OpenPI endpoint as the parity oracle and as an S4 re-planning tier that is never inside the control loop. Design in [vla_edge_design.md](vla_edge_design.md).
+- **S1** is retained for Jetson-class edge servers as the way to run the unmodified reference and to validate the other tiers.