0 Fork 0

Files

T

tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork

The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ

2026-09-14 14:01:12 +08:00

4.4 KiB

Raw Blame History

Baselines and cross-engine comparison

Where this fork's numbers sit against other engines, and what each comparison does and does not establish.

The baseline engine: llama.cpp

llama.cpp is the reference for CPU inference of a quantized model, so it is what "good" is measured against. Both engines are pinned to the same physical cores with taskset and run strictly one at a time; vLLM's prefix caching is disabled so a replayed prompt is really recomputed, because llama-bench has no such cache and leaving it on would not be a comparison.

Spark-X2.5-1.7B, Xeon Platinum 8480C:

	decode (tg128)	prefill (pp512)
llama.cpp Q4_K_M	90.0 tok/s	1121 tok/s
this fork, W4A8 AMX	74.4 tok/s	—
this fork, W4A16 tinygemm	71.6 tok/s	2190 tok/s

Decode is llama.cpp's. Prefill is ours by ~2×, which is the shape one expects: vLLM's prefill goes through oneDNN/AMX bf16 GEMMs at large batch, where llama.cpp's advantage in per-step overhead does not apply.

Not a matched-precision comparison in the strict sense: Q4_K_M stores 4.5 bits/weight over groups of 32 with higher-bit types for some tensors, against our 4.25 over groups of 128. It buys fidelity with those bits (below).

Why llama.cpp wins decode

Both engines are far from the hardware limit. A node-local streaming read on the same 24 cores sustains 222 GB/s:

	GB/s	of ceiling
llama.cpp whole step	86	39%
this fork, whole step	73	33%

So the gap is not bandwidth, and it is not primarily kernel quality — it is the execution model. ggml_threadpool starts threads once and they walk the entire graph together, synchronising at ggml_barrier through an atomic spin. vLLM enters a separate omp parallel region inside each kernel, and a region costs ~1.2 µs at 24 threads. See ROADMAP.md item I10 for the measurements that size this and the reason it is not the next thing to build.

Fidelity

Against each build's own bf16 output, 12 greedy prompts, exact reproductions:

		bits/weight
llama.cpp Q4_K_M	4/12	4.5
W4A16	2/12	4.25
W4A8	1/12	4.25

Some of that gap is simply the extra quarter-bit and the finer grouping. How much is unknown, which is why closing the budget gap (g64, or fp16 scale+zero — both config changes) comes before implementing a better quantizer.

This metric also saturates: it discards everything after the first divergent token. benchmarks/edge_harness/fidelity_kl.py replaces it with top-1 agreement and KL over a corpus, deliberately symmetric with llama-perplexity --kl-divergence so llama.cpp stays comparable on our metric rather than being judged on a different one.

Other engines considered

Full analyses in docs/engines/. In brief, for this workload:

engine	verdict
llama.cpp	the CPU baseline; best decode, mature quantization, no multimodal pipeline
ExecuTorch	the phone path — exported artifacts, XNNPACK on CPU, QNN for NPU; not a server engine
MNN	strong mobile CPU/GPU, own model format, another export toolchain
ncnn	mobile-first, no LLM serving story of the kind needed here
vLLM / vLLM-Omni	the only one with a multi-stage multimodal pipeline, continuous batching and an OpenAI-compatible server — which is why the edge work is a fork of it rather than a port away from it

The decision recorded in docs/analysis/comparison.md is that a phone NPU cannot run vLLM at all: there the deliverable is an exported graph, and the quantization scheme is decided by the export toolchain. On device (Snapdragon via QNN) only calibrated w8a16 was measured correct — 36.9 dB; int8 activations break it at 5.1 dB, and uncalibrated w4a16 scores −1.6 dB, i.e. uncorrelated with the reference.

What the comparisons do not establish

ARM. Every number here is x86 with AMX. cpu_gemm_wna16 was measured on its AMX path because _get_isa_hint returns "amx" whenever AMX is present; the "vec" path a genuinely AMX-less machine takes is untested.
Task-level quality. One run per cell on the agent suite showed that navigation survives quantization and recognising completion does not. That is enough to show a failure reproduces, not to rank builds.
Other models. Qwen3-TTS and MiniCPM-o have edge profiles and init work in this tree, but the 4-bit CPU results above are Spark-X2.5 only.

4.4 KiB Raw Blame History Unescape Escape