tzhouamandClaude Opus 5
3834de3105
Renamed from vllm-embedding: nothing here is about embeddings, and the name had come from the parent working directory. edge-infer says what it does. The GitHub repository is renamed to match; nothing had been pushed yet. The audit found the packaging was incomplete in two ways. Scripts: 56 of them produced results that are published in this repo but were not included, so the numbers could be read and not checked. Added under benchmarks/edge_harness/experiments/ (one-off drivers: the AI Hub on-device runs, the 4-bit GEMM microbenchmark, fidelity and parity checks, checkpoint build and round-trip, activation capture, and the load generator used to prove the contention guard actually refuses) and benchmarks/edge_harness/runs/ (the 24 shell drivers behind published grids). Both directories carry a README saying plainly that they are a record of what was run rather than a supported interface -- most hardcode absolute paths from the machine they ran on. The supported entry points remain the tools one level up. One broken relative link fixed (comparison.md is under docs/engines/, not docs/analysis/). All other links and every script reference in the docs now resolve inside the repo. Secret scan: the token in ~/.qai_hub/client.ini is absent from the worktree and from all 61 commits. Worth noting that an earlier version of my own check reported a false positive, because `grep -rl ... | head` takes its exit status from head, which always succeeds -- the same class of mistake as the measurement errors this project keeps recording. The AI Hub scripts authenticate by reading that file at runtime and contain no credential. README reframed for the new name and the wider scope: the repository is the engine changes, the harness, the raw results, the baselines and the analyses, not just a patch set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
96 lines
4.4 KiB
Markdown
96 lines
4.4 KiB
Markdown
# Baselines and cross-engine comparison
|
||
|
||
Where this fork's numbers sit against other engines, and what each comparison
|
||
does and does not establish.
|
||
|
||
## The baseline engine: llama.cpp
|
||
|
||
llama.cpp is the reference for CPU inference of a quantized model, so it is what
|
||
"good" is measured against. Both engines are pinned to the same physical cores
|
||
with `taskset` and run **strictly one at a time**; vLLM's prefix caching is
|
||
disabled so a replayed prompt is really recomputed, because `llama-bench` has no
|
||
such cache and leaving it on would not be a comparison.
|
||
|
||
Spark-X2.5-1.7B, Xeon Platinum 8480C:
|
||
|
||
| | decode (tg128) | prefill (pp512) |
|
||
|---|---|---|
|
||
| llama.cpp Q4_K_M | **90.0 tok/s** | 1121 tok/s |
|
||
| this fork, W4A8 AMX | 74.4 tok/s | — |
|
||
| this fork, W4A16 tinygemm | 71.6 tok/s | **2190 tok/s** |
|
||
|
||
**Decode is llama.cpp's.** Prefill is ours by ~2×, which is the shape one expects:
|
||
vLLM's prefill goes through oneDNN/AMX bf16 GEMMs at large batch, where
|
||
llama.cpp's advantage in per-step overhead does not apply.
|
||
|
||
Not a matched-precision comparison in the strict sense: Q4_K_M stores **4.5
|
||
bits/weight** over groups of 32 with higher-bit types for some tensors, against
|
||
our **4.25** over groups of 128. It buys fidelity with those bits (below).
|
||
|
||
### Why llama.cpp wins decode
|
||
|
||
Both engines are far from the hardware limit. A node-local streaming read on the
|
||
same 24 cores sustains 222 GB/s:
|
||
|
||
| | GB/s | of ceiling |
|
||
|---|---|---|
|
||
| llama.cpp whole step | 86 | 39% |
|
||
| this fork, whole step | 73 | 33% |
|
||
|
||
So the gap is not bandwidth, and it is not primarily kernel quality — it is the
|
||
execution model. `ggml_threadpool` starts threads once and they walk the entire
|
||
graph together, synchronising at `ggml_barrier` through an atomic spin. vLLM
|
||
enters a separate `omp parallel` region inside each kernel, and a region costs
|
||
~1.2 µs at 24 threads. See [ROADMAP.md](../../ROADMAP.md) item I10 for the
|
||
measurements that size this and the reason it is not the next thing to build.
|
||
|
||
### Fidelity
|
||
|
||
Against each build's own bf16 output, 12 greedy prompts, exact reproductions:
|
||
|
||
| | | bits/weight |
|
||
|---|---|---|
|
||
| llama.cpp Q4_K_M | 4/12 | 4.5 |
|
||
| W4A16 | 2/12 | 4.25 |
|
||
| W4A8 | 1/12 | 4.25 |
|
||
|
||
Some of that gap is simply the extra quarter-bit and the finer grouping. How
|
||
much is unknown, which is why closing the budget gap (g64, or fp16 scale+zero —
|
||
both config changes) comes before implementing a better quantizer.
|
||
|
||
This metric also saturates: it discards everything after the first divergent
|
||
token. `benchmarks/edge_harness/fidelity_kl.py` replaces it with top-1 agreement
|
||
and KL over a corpus, deliberately symmetric with `llama-perplexity
|
||
--kl-divergence` so llama.cpp stays comparable on our metric rather than being
|
||
judged on a different one.
|
||
|
||
## Other engines considered
|
||
|
||
Full analyses in [docs/engines/](../engines/). In brief, for this workload:
|
||
|
||
| engine | verdict |
|
||
|---|---|
|
||
| **llama.cpp** | the CPU baseline; best decode, mature quantization, no multimodal pipeline |
|
||
| **ExecuTorch** | the phone path — exported artifacts, XNNPACK on CPU, QNN for NPU; not a server engine |
|
||
| **MNN** | strong mobile CPU/GPU, own model format, another export toolchain |
|
||
| **ncnn** | mobile-first, no LLM serving story of the kind needed here |
|
||
| **vLLM / vLLM-Omni** | the only one with a multi-stage multimodal pipeline, continuous batching and an OpenAI-compatible server — which is why the edge work is a fork of it rather than a port away from it |
|
||
|
||
The decision recorded in [docs/analysis/comparison.md](../engines/comparison.md)
|
||
is that a phone NPU cannot run vLLM at all: there the deliverable is an exported
|
||
graph, and the quantization scheme is decided by the export toolchain. On device
|
||
(Snapdragon via QNN) only calibrated **w8a16** was measured correct — 36.9 dB;
|
||
int8 activations break it at 5.1 dB, and uncalibrated w4a16 scores −1.6 dB, i.e.
|
||
uncorrelated with the reference.
|
||
|
||
## What the comparisons do not establish
|
||
|
||
- **ARM.** Every number here is x86 with AMX. `cpu_gemm_wna16` was measured on
|
||
its AMX path because `_get_isa_hint` returns `"amx"` whenever AMX is present;
|
||
the `"vec"` path a genuinely AMX-less machine takes is untested.
|
||
- **Task-level quality.** One run per cell on the agent suite showed that
|
||
navigation survives quantization and recognising completion does not. That is
|
||
enough to show a failure reproduces, not to rank builds.
|
||
- **Other models.** Qwen3-TTS and MiniCPM-o have edge profiles and init work in
|
||
this tree, but the 4-bit CPU results above are Spark-X2.5 only.
|