tzhouamandClaude Opus 5
9264ff81ab
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
11 KiB
Simulating edge devices on this host, and why the CPU path is slow
Companion to edge_engine_comparison.md. Measured 2026-09-09; raw data under experiments/edge_engine_compare/results/gpu_emu and results/cpu_attrib, driver scripts gpu_emu_chain2.sh (MPS-capped runs) and the branch's hw_emulation.py.
1. What this GPU is, and what can be emulated without root
nvidia-smi reports the eight cards as "NVIDIA L20X, Ada Lovelace, 143,771 MiB", but the measured dense bf16 matmul throughput is 804 TFLOPS (8192³, 20 iterations) with 132 SMs at 1980 MHz and 140 GiB, which is H200-class silicon, not an Ada L20 (≈120 TFLOPS). The comparison and the branch's results should therefore be read as H200-class numbers; the label in earlier documents is the driver's.
| Edge property | Emulable here? | How | Not covered |
|---|---|---|---|
| Fewer SMs (Orin AGX 16 SMs, Orin NX 8, Thor 20 vs 132) | yes | user-level MPS daemon + CUDA_MPS_ACTIVE_THREAD_PERCENTAGE (verified: 12 % cap → 100.6 TFLOPS, ratio 0.125). Two traps: the Unix-socket path of CUDA_MPS_PIPE_DIRECTORY must be < 108 bytes, and clients inside the MPS namespace must use CUDA_VISIBLE_DEVICES=0 even when gpu run assigned another index |
lower SM clocks (Orin 1.3 GHz vs 1.98 GHz) |
| Small VRAM / unified memory budgets | yes | branch deploy_profile="auto" with VLLM_OMNI_HW_PROFILE overrides: absolute KV budgets, eager mode, small batches (hw_emulation.py jetson_orin_8g / 32g / cuda_discrete_8g) |
unified-memory bandwidth sharing with the CPU |
| Weak host CPU (Orin: 8–12 Cortex-A78AE at 2.2 GHz) | partly | taskset to 4–6 Xeon cores for the whole engine; x86 cores are still ≈2× faster per core |
ARM ISA, big.LITTLE |
| Memory bandwidth (HBM 4.8 TB/s vs LPDDR5 102–273 GB/s) | no | analytic projection only (§4) | – |
| Clock / power caps, MIG partitions | no | nvidia-smi -lgc/-pl and MIG need root; MIG is disabled on these cards |
– |
| Thermal throttling, CPU/GPU memory contention | no | – | – |
2. GPU: sensitivity to SM count (MPS caps, H200, 0.6B CustomVoice, edge profile)
Same 12 prompts, one utterance at a time, deploy_profile="edge", 0.6B CustomVoice; MPS CUDA_MPS_ACTIVE_THREAD_PERCENTAGE = share of the 132 SMs available to every process of the deployment (both stages). llama.cpp rows: llama-tts CUDA build, all layers offloaded, same cap. The "Jetson-like" row combines a 12 % SM cap (16 SMs ≈ Orin AGX), 6 host cores for the whole engine, and the branch's jetson_orin_8g hardware profile (eager mode, absolute 8 GB budget).
| SM share (≈ SMs) | vLLM-Omni talker step ms/frame | vLLM-Omni TTFA p50 / p90 | vLLM-Omni RTF | llama.cpp talker ms/frame | llama.cpp vocoder ms/frame | llama.cpp RTF |
|---|---|---|---|---|---|---|
| 100 % (132) | 6.0 | 41–46 / 49 ms | 0.10 | 17.4 | 3.8 | 0.28 |
| 25 % (33) | 6.5–7.0 | 43 / 49–53 ms | 0.08–0.09 | 23.1 | 2.1 | 0.33 |
| 12 % (16, Orin AGX class) | 9.1 | 52 / 58 ms | 0.12 | 33.7 | 4.7 | 0.50 |
| 6 % (8, Orin NX class) | 9.3 | 58 / 66 ms | 0.13 | 48.0 | 4.1 | 0.68 |
| Jetson-like combined: 12 % + 6 host cores + 8 GB profile (eager) | ≈13 (from RTF) | 160 / – ms | 0.50 | – | – | – |
| (reference) 8 GB profile alone, uncapped, from §8.5 | – | 181 ms | 0.57 | – | – | – |
Reading: with CUDA graphs and one utterance in flight, the talker step is latency-bound: cutting the SMs to 6 % of an H200 costs vLLM-Omni only +55 % per frame (6.0 → 9.3 ms) and +15 ms of TTFA, and RTF stays ≈0.13. llama.cpp's kernels are launch-bound per layer and lose 2.8× at the same cap (17 → 48 ms per frame). The combined Jetson-like case shows where the real cost sits: forcing eager mode (the 8 GB profile disables CUDA graphs so the small budget is not consumed by graph pools) raises the frame time to ≈13 ms and TTFA to 160 ms regardless of the SM cap, i.e. on a small-memory Jetson the branch should keep graph capture for the talker/predictor and spend the memory elsewhere, which is the next tuning item for deploy/edge/hardware/jetson.yaml. What SM caps cannot show is the memory-bandwidth term, projected in §4.
3. Where a GPU frame goes (synchronized attribution, uncapped)
VLLM_OMNI_STEP_STATS_SYNC=1 synchronizes the device around the model-level timers (adds ≈3 ms per step of sync overhead, so core.step reads 9.6 ms here instead of the 6.0 ms measured without sync):
| per talker step (= one 80 ms frame) | ms |
|---|---|
| model forward (28-layer talker, CUDA graph) incl. sampling hook | 1.8 |
| sampler | 0.6 |
| talker-MTP graph (15 code-predictor sub-steps + embeddings, one CUDA graph) | inside the forward's 1.8 ms (graph replay; the separate model.mtp_graph_ms timer measures 0.04–0.43 ms of launch time without sync) |
scheduler (schedule + update_from_output) |
0.2 |
| remainder: input preparation, D2H of codes/hidden, Python bookkeeping | ≈3–4 |
The GPU compute for a frame is a couple of milliseconds; more than half of the 6 ms step is host-side runner work, which is what a slow ARM host CPU would stretch first.
4. Analytic projection to Jetson-class devices
The talker step is memory-bound at batch 1: every frame streams the talker weights once (0.6B bf16: 1.2 GB) and the code-predictor weights fifteen times (5 layers × 1024 hidden ≈ 160 MB each pass, 2.4 GB per frame). Using the measured H200 split and datasheet bandwidths (weights from the HF config; bandwidths: H200 4.8 TB/s, Orin AGX 204.8 GB/s, Orin NX 102.4 GB/s, Thor 273 GB/s):
| device | weight streaming per frame (talker + 15 predictor passes) | host-side runner work (≈4 ms on Xeon, scaled by single-core speed) | projected ms/frame | projected RTF (80 ms frame) |
|---|---|---|---|---|
| H200 (measured) | 0.75 ms | ≈4 ms | 6.0 (measured) | 0.10 (measured) |
| Jetson AGX Orin 64 GB | 17.6 ms | ≈8 ms | ≈26 | ≈0.33 |
| Jetson Orin NX 16 GB | 35 ms | ≈8 ms | ≈43 | ≈0.55 |
| Jetson Thor | 13 ms | ≈6 ms | ≈19 | ≈0.24 |
These are floors from bandwidth plus measured host overhead, not measurements: kernel launch latency at small SM counts, unified-memory contention with the code2wav stage on the same memory bus, and thermal limits push them up. The MPS 12 % run (§2) is the compute-side check of the Orin AGX case (16 of 132 SMs); a 4-bit talker (llama.cpp Q4_0 runs at 3.1 ms/token on 16 Xeon threads for the same backbone) would cut the streaming term by ≈3× on all rows. Conclusion: the branch's talker stays real-time on Orin AGX / Thor with 2–4× headroom by this model, is marginal on Orin NX, and its 15-pass code predictor is the term to attack (see §5).
5. Why the CPU path is slow
5.1 Measured attribution (vLLM-Omni CPU platform, 0.6B CustomVoice, 12+4 threads on one socket, bf16 with AMX)
| per frame (one talker step) | ms | share of the 80 ms frame |
|---|---|---|
| code predictor loop (15 sub-steps × 6.4 ms, 5-layer 1024-hidden model, batch 1) | 97.1 | 121 % |
| talker forward (28 layers, 0.45 B params) | 25.4 | 32 % |
| sampler (top-k/top-p on a 3072-way vocabulary, penalties, CPU) | 9.9 | 12 % |
| scheduler + bookkeeping | ≈2 | 2 % |
| engine-core step | 133.9 | 167 % |
(core.step 134 ms here vs 147 ms in the earlier sweep: the attribution run had the host to itself.) The code2wav stage runs concurrently in its own process and does not add to this cadence.
5.2 The same split in llama.cpp (native C++, same host, 16 threads)
llama-bench on the converted talker GGUF gives the backbone alone at 126.7 tok/s f16 (7.9 ms/token) and 325.8 tok/s Q4_0 (3.1 ms/token), yet llama-tts needs 85–97 ms per frame: ≈80 ms of every frame is the fifteen code-predictor passes (≈5.3 ms each) plus sampling. llama.cpp's predictor pass is only ≈20 % cheaper than vLLM-Omni's 6.4 ms even though it is hand-written C++ with no Python — so the cost is not "Python overhead"; it is the structure of the work.
5.3 Root causes, in order
- Sixteen sequential forward passes per 80 ms frame. One talker pass plus fifteen residual-codebook passes, each depending on the previous sample. Real time needs the whole chain under 80 ms, i.e. < 5 ms per pass including sampling. Both engines sit at 5.3–6.4 ms per predictor pass on 16 Xeon threads.
- Each pass is latency-bound, not bandwidth-bound. A predictor pass reads ≈160 MB of bf16 weights, which is ≈1.6 ms at the ≈100 GB/s one socket-half delivers; the measured 6.4 ms is 4× that. The rest is per-operator cost: ≈100 small kernels per pass (5 layers × QKV/attention/RoPE/norms/MLP/gate) each paying OpenMP fork-join across 12 threads, cache-cold tiny GEMVs, and (in PyTorch) operator dispatch. The talker's 28-layer pass is at 2× its bandwidth floor (25 ms vs 12 ms) for the same reason.
- Sampling costs 10 ms on CPU. vLLM's sampler applies penalties, top-k and top-p over the padded vocabulary per step with several full-vocabulary passes; on GPU this is 0.6 ms. The fifteen residual codebooks are also sampled (Gumbel top-k) inside the predictor loop.
- Threads stop helping at 16. 16 → 32 threads changed the frame time by 8 % (147 → 136 ms) because the work is latency-bound (item 2) and OpenMP synchronization grows with thread count; the two stages also compete for the same memory channels.
- bf16/AMX is the only CPU precision on the branch. There is no int8/int4 talker path in vLLM's CPU backend for this model; llama.cpp's Q4_0 backbone shows the bandwidth term can shrink 2.5×, but that only helps the part that is bandwidth-bound (talker pass, ≈25 → ≈12 ms), not the predictor's overhead floor.
- Vocoder: not on the vLLM-Omni critical path (concurrent stage), but in llama.cpp the BigVGAN decoder costs a further 55 ms per frame serially, which is why its end-to-end RTF matches vLLM-Omni's despite the faster talker.
5.4 What would make CPU real-time, with expected effect
| change | where | expected per-frame effect (0.6B, 16 threads) | evidence |
|---|---|---|---|
| Fuse each predictor pass into one kernel / one OpenMP region (C++ or a compiled graph), drop per-op dispatch | predictor | 6.4 → ≈2–3 ms per pass, i.e. −50…−65 ms per frame | bandwidth floor 1.6 ms; llama.cpp's eager C++ is already at 5.3 ms, so fusion (not language) is the lever |
| Greedy or cheap top-k for the 15 residual codebooks; skip penalties for codec ids | sampler + predictor | −8 ms (sampler) and part of each pass | GPU sampler is 0.6 ms for the same work |
| Int8/int4 talker weights (AMX int8 or a GGUF-style kernel) | talker pass | 25 → ≈10–12 ms | llama-bench Q4_0 3.1 ms vs f16 7.9 ms per token |
torch.compile (inductor) on the CPU runner |
whole step | −6 % measured (cpu16_inductor in §8.4) |
measured |
| Architectural: fewer residual codebooks per pass (predict 2–4 codebooks per pass) or a parallel/non-autoregressive predictor | model | 15 → 4–8 passes per frame | requires retraining or a different checkpoint; out of scope for the branch |
Adding the first three rows gives ≈134 → ≈45–60 ms per frame, i.e. real time with little margin on 16 Xeon threads; on a 4–8-core ARM box (≈2–3× slower per core, see the emulated x86_cpu_avx2 case at RTF 12) the architectural row is the only one that reaches real time. That matches the study's original conclusion: CPU-only real-time TTS with this model family needs a lighter talker/predictor, not a faster runtime.