tzhouamandClaude Opus 5
3834de3105
Renamed from vllm-embedding: nothing here is about embeddings, and the name had come from the parent working directory. edge-infer says what it does. The GitHub repository is renamed to match; nothing had been pushed yet. The audit found the packaging was incomplete in two ways. Scripts: 56 of them produced results that are published in this repo but were not included, so the numbers could be read and not checked. Added under benchmarks/edge_harness/experiments/ (one-off drivers: the AI Hub on-device runs, the 4-bit GEMM microbenchmark, fidelity and parity checks, checkpoint build and round-trip, activation capture, and the load generator used to prove the contention guard actually refuses) and benchmarks/edge_harness/runs/ (the 24 shell drivers behind published grids). Both directories carry a README saying plainly that they are a record of what was run rather than a supported interface -- most hardcode absolute paths from the machine they ran on. The supported entry points remain the tools one level up. One broken relative link fixed (comparison.md is under docs/engines/, not docs/analysis/). All other links and every script reference in the docs now resolve inside the repo. Secret scan: the token in ~/.qai_hub/client.ini is absent from the worktree and from all 61 commits. Worth noting that an earlier version of my own check reported a false positive, because `grep -rl ... | head` takes its exit status from head, which always succeeds -- the same class of mistake as the measurement errors this project keeps recording. The AI Hub scripts authenticate by reading that file at runtime and contain no credential. README reframed for the new name and the wider scope: the repository is the engine changes, the harness, the raw results, the baselines and the analyses, not just a patch set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
4.4 KiB
Baselines and cross-engine comparison
Where this fork's numbers sit against other engines, and what each comparison does and does not establish.
The baseline engine: llama.cpp
llama.cpp is the reference for CPU inference of a quantized model, so it is what
"good" is measured against. Both engines are pinned to the same physical cores
with taskset and run strictly one at a time; vLLM's prefix caching is
disabled so a replayed prompt is really recomputed, because llama-bench has no
such cache and leaving it on would not be a comparison.
Spark-X2.5-1.7B, Xeon Platinum 8480C:
| decode (tg128) | prefill (pp512) | |
|---|---|---|
| llama.cpp Q4_K_M | 90.0 tok/s | 1121 tok/s |
| this fork, W4A8 AMX | 74.4 tok/s | — |
| this fork, W4A16 tinygemm | 71.6 tok/s | 2190 tok/s |
Decode is llama.cpp's. Prefill is ours by ~2×, which is the shape one expects: vLLM's prefill goes through oneDNN/AMX bf16 GEMMs at large batch, where llama.cpp's advantage in per-step overhead does not apply.
Not a matched-precision comparison in the strict sense: Q4_K_M stores 4.5 bits/weight over groups of 32 with higher-bit types for some tensors, against our 4.25 over groups of 128. It buys fidelity with those bits (below).
Why llama.cpp wins decode
Both engines are far from the hardware limit. A node-local streaming read on the same 24 cores sustains 222 GB/s:
| GB/s | of ceiling | |
|---|---|---|
| llama.cpp whole step | 86 | 39% |
| this fork, whole step | 73 | 33% |
So the gap is not bandwidth, and it is not primarily kernel quality — it is the
execution model. ggml_threadpool starts threads once and they walk the entire
graph together, synchronising at ggml_barrier through an atomic spin. vLLM
enters a separate omp parallel region inside each kernel, and a region costs
~1.2 µs at 24 threads. See ROADMAP.md item I10 for the
measurements that size this and the reason it is not the next thing to build.
Fidelity
Against each build's own bf16 output, 12 greedy prompts, exact reproductions:
| bits/weight | ||
|---|---|---|
| llama.cpp Q4_K_M | 4/12 | 4.5 |
| W4A16 | 2/12 | 4.25 |
| W4A8 | 1/12 | 4.25 |
Some of that gap is simply the extra quarter-bit and the finer grouping. How much is unknown, which is why closing the budget gap (g64, or fp16 scale+zero — both config changes) comes before implementing a better quantizer.
This metric also saturates: it discards everything after the first divergent
token. benchmarks/edge_harness/fidelity_kl.py replaces it with top-1 agreement
and KL over a corpus, deliberately symmetric with llama-perplexity --kl-divergence so llama.cpp stays comparable on our metric rather than being
judged on a different one.
Other engines considered
Full analyses in docs/engines/. In brief, for this workload:
| engine | verdict |
|---|---|
| llama.cpp | the CPU baseline; best decode, mature quantization, no multimodal pipeline |
| ExecuTorch | the phone path — exported artifacts, XNNPACK on CPU, QNN for NPU; not a server engine |
| MNN | strong mobile CPU/GPU, own model format, another export toolchain |
| ncnn | mobile-first, no LLM serving story of the kind needed here |
| vLLM / vLLM-Omni | the only one with a multi-stage multimodal pipeline, continuous batching and an OpenAI-compatible server — which is why the edge work is a fork of it rather than a port away from it |
The decision recorded in docs/analysis/comparison.md is that a phone NPU cannot run vLLM at all: there the deliverable is an exported graph, and the quantization scheme is decided by the export toolchain. On device (Snapdragon via QNN) only calibrated w8a16 was measured correct — 36.9 dB; int8 activations break it at 5.1 dB, and uncalibrated w4a16 scores −1.6 dB, i.e. uncorrelated with the reference.
What the comparisons do not establish
- ARM. Every number here is x86 with AMX.
cpu_gemm_wna16was measured on its AMX path because_get_isa_hintreturns"amx"whenever AMX is present; the"vec"path a genuinely AMX-less machine takes is untested. - Task-level quality. One run per cell on the agent suite showed that navigation survives quantization and recognising completion does not. That is enough to show a failure reproduces, not to rank builds.
- Other models. Qwen3-TTS and MiniCPM-o have edge profiles and init work in this tree, but the 4-bit CPU results above are Spark-X2.5 only.