The Code2Wav decoder already had an on-device export (WP4); the talker and its
code predictor did not, so the codec-token path stopped at the vocoder. This
adds the missing half: a one-token decode step with an explicit per-layer KV
cache, exportable to ONNX and from there to a QNN context binary, TFLite or
ONNX Runtime.
Two layout choices carry the performance, both measured on real Snapdragon
devices through Qualcomm AI Hub (Galaxy S24 = 8 Gen 3, S25 = 8 Elite):
* per-layer cache tensors instead of one stacked [num_layers, ...] tensor that
every layer slices, and
* grouped-query attention by reshaping the query rather than expanding keys and
values.
The stacked/expanded form spent ~70% of its NPU cycles in cache select,
reshape and fp32->fp16 conversion; the layout here is bandwidth-bound and runs
the 0.6B talker step in 15.8 ms instead of 36.9 ms on an 8 Elite (fp16,
256-token cache). The code predictor step is 2.25 ms, so talker plus its 15
sub-steps cost about 50 ms of an 80 ms frame.
sanitize_onnx() fixes the three things the dynamo exporter emits that
qairt-converter and AI Hub reject: IsNaN/Where guards, Reshape allowzero=1, and
rank-0 axes initializers on Unsqueeze/Squeeze.
The module docstring records the measured placement table: this graph belongs
on the NPU, the vocoder on the GPU (21.7 ms vs 182 ms per frame), which is 3.3x
better than sending everything to the NPU and makes the pipeline real time on
the phone in fp16. Integer quantization is not the lever it looks like: every
automatic int8 build measured was numerically broken.
Full write-up, job ids and profiles: analysis/mobile_htp_simulation.md section 5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dzr1Jusv4s6g765gqS9L92
vllm_omni/edge/decoder_export.py exports a stateless windowed decoder (context
72 frames + chunk, one program per ramp chunk size) via torch.export (.pt2,
verified here on CPU for chunks 2 and 25 with the real 0.6B decoder: exact
match vs eager, 460 MB fp32 per program) or ExecuTorch XNNPACK (.pte; written
against the executorch 1.x API, not executed: package not installed, install
is permission gated; QNN partitioner export-only/unverified). The WP5 client
gains --decoder-backend executorch --pte-manifest for phone/NPU targets.
benchmarks/tts/edge_results/summarize.py renders all committed JSON as tables.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- vllm_omni/edge/step_stats.py: process-local counters enabled by
VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core
step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped
per process and merged into an attribution table (share of the 80 ms frame).
Hooks: stage_engine_core_proc (instrument after construction, dump at exit),
chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime),
orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats.
- vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and
the no-preemption KV admission rule (exact bytes/token from the talker KV
geometry); adapt.derive_overrides now sizes max_num_seqs with it and
auto_profile passes the geometry read from the model config.
- Engine re-reads parallel_stage_init after apply_deploy_profile so the edge
profile's orchestrator default takes effect (was read before the profile).
- benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor)
with per-step attribution.
- Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py,
tests/benchmarks/test_cpu_thread_sweep.py, profile default test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch
(ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores,
big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU
via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu /
npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests).
* vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from
available memory minus weights, max_num_seqs, per-stage disjoint CPU thread
ranges on the big cluster, eager/graph switches for small or unified memory,
adaptive-chunk parameters, timeouts from a load cost model.
* vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk
arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT.
* vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml ->
edge/hardware/<class>.yaml -> derived overrides -> cached calibration,
materialized as a temp deploy YAML (platform overlays are superseded).
* benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY,
taskset, profile overrides) driving stream_latency_bench with auto profiles.
* CPU-only tests for probe snapshots, derivation rules, calibration and auto.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
