tzhouamandClaude Opus 5
9264ff81ab
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
98 lines
11 KiB
Markdown
98 lines
11 KiB
Markdown
# Simulating edge devices on this host, and why the CPU path is slow
|
||
|
||
Companion to [edge_engine_comparison.md](edge_engine_comparison.md). Measured 2026-09-09; raw data under [experiments/edge_engine_compare/results/gpu_emu](experiments/edge_engine_compare/results/gpu_emu/) and [results/cpu_attrib](experiments/edge_engine_compare/results/cpu_attrib/), driver scripts `gpu_emu_chain2.sh` (MPS-capped runs) and the branch's `hw_emulation.py`.
|
||
|
||
## 1. What this GPU is, and what can be emulated without root
|
||
|
||
`nvidia-smi` reports the eight cards as "NVIDIA L20X, Ada Lovelace, 143,771 MiB", but the measured dense bf16 matmul throughput is **804 TFLOPS** (8192³, 20 iterations) with 132 SMs at 1980 MHz and 140 GiB, which is H200-class silicon, not an Ada L20 (≈120 TFLOPS). The comparison and the branch's results should therefore be read as **H200-class** numbers; the label in earlier documents is the driver's.
|
||
|
||
| Edge property | Emulable here? | How | Not covered |
|
||
|---|---|---|---|
|
||
| Fewer SMs (Orin AGX 16 SMs, Orin NX 8, Thor 20 vs 132) | **yes** | user-level MPS daemon + `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` (verified: 12 % cap → 100.6 TFLOPS, ratio 0.125). Two traps: the Unix-socket path of `CUDA_MPS_PIPE_DIRECTORY` must be < 108 bytes, and clients inside the MPS namespace must use `CUDA_VISIBLE_DEVICES=0` even when `gpu run` assigned another index | lower SM clocks (Orin 1.3 GHz vs 1.98 GHz) |
|
||
| Small VRAM / unified memory budgets | yes | branch `deploy_profile="auto"` with `VLLM_OMNI_HW_PROFILE` overrides: absolute KV budgets, eager mode, small batches (`hw_emulation.py` jetson_orin_8g / 32g / cuda_discrete_8g) | unified-memory bandwidth sharing with the CPU |
|
||
| Weak host CPU (Orin: 8–12 Cortex-A78AE at 2.2 GHz) | partly | `taskset` to 4–6 Xeon cores for the whole engine; x86 cores are still ≈2× faster per core | ARM ISA, big.LITTLE |
|
||
| Memory bandwidth (HBM 4.8 TB/s vs LPDDR5 102–273 GB/s) | **no** | analytic projection only (§4) | – |
|
||
| Clock / power caps, MIG partitions | no | `nvidia-smi -lgc/-pl` and MIG need root; MIG is disabled on these cards | – |
|
||
| Thermal throttling, CPU/GPU memory contention | no | – | – |
|
||
|
||
## 2. GPU: sensitivity to SM count (MPS caps, H200, 0.6B CustomVoice, edge profile)
|
||
|
||
Same 12 prompts, one utterance at a time, `deploy_profile="edge"`, 0.6B CustomVoice; MPS `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` = share of the 132 SMs available to every process of the deployment (both stages). llama.cpp rows: `llama-tts` CUDA build, all layers offloaded, same cap. The "Jetson-like" row combines a 12 % SM cap (16 SMs ≈ Orin AGX), 6 host cores for the whole engine, and the branch's `jetson_orin_8g` hardware profile (eager mode, absolute 8 GB budget).
|
||
|
||
| SM share (≈ SMs) | vLLM-Omni talker step ms/frame | vLLM-Omni TTFA p50 / p90 | vLLM-Omni RTF | llama.cpp talker ms/frame | llama.cpp vocoder ms/frame | llama.cpp RTF |
|
||
|---|---|---|---|---|---|---|
|
||
| 100 % (132) | 6.0 | 41–46 / 49 ms | 0.10 | 17.4 | 3.8 | 0.28 |
|
||
| 25 % (33) | 6.5–7.0 | 43 / 49–53 ms | 0.08–0.09 | 23.1 | 2.1 | 0.33 |
|
||
| 12 % (16, Orin AGX class) | 9.1 | 52 / 58 ms | 0.12 | 33.7 | 4.7 | 0.50 |
|
||
| 6 % (8, Orin NX class) | 9.3 | 58 / 66 ms | 0.13 | 48.0 | 4.1 | 0.68 |
|
||
| Jetson-like combined: 12 % + 6 host cores + 8 GB profile (eager) | ≈13 (from RTF) | 160 / – ms | 0.50 | – | – | – |
|
||
| (reference) 8 GB profile alone, uncapped, from §8.5 | – | 181 ms | 0.57 | – | – | – |
|
||
|
||
Reading: with CUDA graphs and one utterance in flight, the talker step is latency-bound: cutting the SMs to 6 % of an H200 costs vLLM-Omni only +55 % per frame (6.0 → 9.3 ms) and +15 ms of TTFA, and RTF stays ≈0.13. llama.cpp's kernels are launch-bound per layer and lose 2.8× at the same cap (17 → 48 ms per frame). The combined Jetson-like case shows where the real cost sits: forcing eager mode (the 8 GB profile disables CUDA graphs so the small budget is not consumed by graph pools) raises the frame time to ≈13 ms and TTFA to 160 ms regardless of the SM cap, i.e. **on a small-memory Jetson the branch should keep graph capture for the talker/predictor and spend the memory elsewhere**, which is the next tuning item for `deploy/edge/hardware/jetson.yaml`. What SM caps cannot show is the memory-bandwidth term, projected in §4.
|
||
|
||
## 3. Where a GPU frame goes (synchronized attribution, uncapped)
|
||
|
||
`VLLM_OMNI_STEP_STATS_SYNC=1` synchronizes the device around the model-level timers (adds ≈3 ms per step of sync overhead, so `core.step` reads 9.6 ms here instead of the 6.0 ms measured without sync):
|
||
|
||
| per talker step (= one 80 ms frame) | ms |
|
||
|---|---|
|
||
| model forward (28-layer talker, CUDA graph) incl. sampling hook | 1.8 |
|
||
| sampler | 0.6 |
|
||
| talker-MTP graph (15 code-predictor sub-steps + embeddings, one CUDA graph) | inside the forward's 1.8 ms (graph replay; the separate `model.mtp_graph_ms` timer measures 0.04–0.43 ms of launch time without sync) |
|
||
| scheduler (`schedule` + `update_from_output`) | 0.2 |
|
||
| remainder: input preparation, D2H of codes/hidden, Python bookkeeping | ≈3–4 |
|
||
|
||
The GPU compute for a frame is a couple of milliseconds; more than half of the 6 ms step is host-side runner work, which is what a slow ARM host CPU would stretch first.
|
||
|
||
## 4. Analytic projection to Jetson-class devices
|
||
|
||
The talker step is memory-bound at batch 1: every frame streams the talker weights once (0.6B bf16: 1.2 GB) and the code-predictor weights fifteen times (5 layers × 1024 hidden ≈ 160 MB each pass, 2.4 GB per frame). Using the measured H200 split and datasheet bandwidths (weights from the HF config; bandwidths: H200 4.8 TB/s, Orin AGX 204.8 GB/s, Orin NX 102.4 GB/s, Thor 273 GB/s):
|
||
|
||
| device | weight streaming per frame (talker + 15 predictor passes) | host-side runner work (≈4 ms on Xeon, scaled by single-core speed) | projected ms/frame | projected RTF (80 ms frame) |
|
||
|---|---|---|---|---|
|
||
| H200 (measured) | 0.75 ms | ≈4 ms | 6.0 (measured) | 0.10 (measured) |
|
||
| Jetson AGX Orin 64 GB | 17.6 ms | ≈8 ms | ≈26 | ≈0.33 |
|
||
| Jetson Orin NX 16 GB | 35 ms | ≈8 ms | ≈43 | ≈0.55 |
|
||
| Jetson Thor | 13 ms | ≈6 ms | ≈19 | ≈0.24 |
|
||
|
||
These are floors from bandwidth plus measured host overhead, not measurements: kernel launch latency at small SM counts, unified-memory contention with the code2wav stage on the same memory bus, and thermal limits push them up. The MPS 12 % run (§2) is the compute-side check of the Orin AGX case (16 of 132 SMs); a 4-bit talker (llama.cpp Q4_0 runs at 3.1 ms/token on 16 Xeon threads for the same backbone) would cut the streaming term by ≈3× on all rows. **Conclusion:** the branch's talker stays real-time on Orin AGX / Thor with 2–4× headroom by this model, is marginal on Orin NX, and its 15-pass code predictor is the term to attack (see §5).
|
||
|
||
## 5. Why the CPU path is slow
|
||
|
||
### 5.1 Measured attribution (vLLM-Omni CPU platform, 0.6B CustomVoice, 12+4 threads on one socket, bf16 with AMX)
|
||
|
||
| per frame (one talker step) | ms | share of the 80 ms frame |
|
||
|---|---|---|
|
||
| **code predictor loop** (15 sub-steps × 6.4 ms, 5-layer 1024-hidden model, batch 1) | **97.1** | 121 % |
|
||
| talker forward (28 layers, 0.45 B params) | 25.4 | 32 % |
|
||
| sampler (top-k/top-p on a 3072-way vocabulary, penalties, CPU) | 9.9 | 12 % |
|
||
| scheduler + bookkeeping | ≈2 | 2 % |
|
||
| **engine-core step** | **133.9** | **167 %** |
|
||
|
||
(`core.step` 134 ms here vs 147 ms in the earlier sweep: the attribution run had the host to itself.) The code2wav stage runs concurrently in its own process and does not add to this cadence.
|
||
|
||
### 5.2 The same split in llama.cpp (native C++, same host, 16 threads)
|
||
|
||
`llama-bench` on the converted talker GGUF gives the backbone alone at **126.7 tok/s f16 (7.9 ms/token)** and **325.8 tok/s Q4_0 (3.1 ms/token)**, yet `llama-tts` needs **85–97 ms per frame**: ≈80 ms of every frame is the fifteen code-predictor passes (≈5.3 ms each) plus sampling. llama.cpp's predictor pass is only ≈20 % cheaper than vLLM-Omni's 6.4 ms even though it is hand-written C++ with no Python — so the cost is not "Python overhead"; it is the structure of the work.
|
||
|
||
### 5.3 Root causes, in order
|
||
|
||
1. **Sixteen sequential forward passes per 80 ms frame.** One talker pass plus fifteen residual-codebook passes, each depending on the previous sample. Real time needs the whole chain under 80 ms, i.e. **< 5 ms per pass including sampling**. Both engines sit at 5.3–6.4 ms per predictor pass on 16 Xeon threads.
|
||
2. **Each pass is latency-bound, not bandwidth-bound.** A predictor pass reads ≈160 MB of bf16 weights, which is ≈1.6 ms at the ≈100 GB/s one socket-half delivers; the measured 6.4 ms is 4× that. The rest is per-operator cost: ≈100 small kernels per pass (5 layers × QKV/attention/RoPE/norms/MLP/gate) each paying OpenMP fork-join across 12 threads, cache-cold tiny GEMVs, and (in PyTorch) operator dispatch. The talker's 28-layer pass is at 2× its bandwidth floor (25 ms vs 12 ms) for the same reason.
|
||
3. **Sampling costs 10 ms on CPU.** vLLM's sampler applies penalties, top-k and top-p over the padded vocabulary per step with several full-vocabulary passes; on GPU this is 0.6 ms. The fifteen residual codebooks are also sampled (Gumbel top-k) inside the predictor loop.
|
||
4. **Threads stop helping at 16.** 16 → 32 threads changed the frame time by 8 % (147 → 136 ms) because the work is latency-bound (item 2) and OpenMP synchronization grows with thread count; the two stages also compete for the same memory channels.
|
||
5. **bf16/AMX is the only CPU precision on the branch.** There is no int8/int4 talker path in vLLM's CPU backend for this model; llama.cpp's Q4_0 backbone shows the bandwidth term can shrink 2.5×, but that only helps the part that is bandwidth-bound (talker pass, ≈25 → ≈12 ms), not the predictor's overhead floor.
|
||
6. **Vocoder**: not on the vLLM-Omni critical path (concurrent stage), but in llama.cpp the BigVGAN decoder costs a further 55 ms per frame serially, which is why its end-to-end RTF matches vLLM-Omni's despite the faster talker.
|
||
|
||
### 5.4 What would make CPU real-time, with expected effect
|
||
|
||
| change | where | expected per-frame effect (0.6B, 16 threads) | evidence |
|
||
|---|---|---|---|
|
||
| Fuse each predictor pass into one kernel / one OpenMP region (C++ or a compiled graph), drop per-op dispatch | predictor | 6.4 → ≈2–3 ms per pass, i.e. −50…−65 ms per frame | bandwidth floor 1.6 ms; llama.cpp's eager C++ is already at 5.3 ms, so fusion (not language) is the lever |
|
||
| Greedy or cheap top-k for the 15 residual codebooks; skip penalties for codec ids | sampler + predictor | −8 ms (sampler) and part of each pass | GPU sampler is 0.6 ms for the same work |
|
||
| Int8/int4 talker weights (AMX int8 or a GGUF-style kernel) | talker pass | 25 → ≈10–12 ms | llama-bench Q4_0 3.1 ms vs f16 7.9 ms per token |
|
||
| `torch.compile` (inductor) on the CPU runner | whole step | −6 % measured (`cpu16_inductor` in §8.4) | measured |
|
||
| Architectural: fewer residual codebooks per pass (predict 2–4 codebooks per pass) or a parallel/non-autoregressive predictor | model | 15 → 4–8 passes per frame | requires retraining or a different checkpoint; out of scope for the branch |
|
||
|
||
Adding the first three rows gives ≈134 → ≈45–60 ms per frame, i.e. real time with little margin on 16 Xeon threads; on a 4–8-core ARM box (≈2–3× slower per core, see the emulated `x86_cpu_avx2` case at RTF 12) the architectural row is the only one that reaches real time. That matches the study's original conclusion: CPU-only real-time TTS with this model family needs a lighter talker/predictor, not a faster runtime.
|