tzhouamandClaude Opus 5
9264ff81ab
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
238 lines
11 KiB
Markdown
238 lines
11 KiB
Markdown
# Closing the CPU decode gap: 4-bit weights for Spark-X2.5 on vLLM
|
|
|
|
`analysis/spark_edge_support.md` ended with one open item. At matched bf16,
|
|
vLLM beat llama.cpp on Spark-X2.5-1.7B everywhere -- 8.7-15.4x on prefill,
|
|
1.19-2.65x on decode -- but llama.cpp's **Q4_K_M** build won single-stream
|
|
decode outright. Decode at batch 1 reads every weight to emit one token, so
|
|
bits per weight, not flops, set the token rate, and vLLM had no 4-bit CPU
|
|
path in use. This note closes that and re-runs the comparison at matched
|
|
precision.
|
|
|
|
Two corrections to earlier findings come first, because both changed the
|
|
answer.
|
|
|
|
Code: `analysis/experiments/spark_edge/{export_gptq_cpu,quantize_int4,ppl_vllm,profile_cpu_step,bench_vllm_w4a8,cmp_llamacpp,dequantize_int4}.py`,
|
|
`vllm_omni/model_executor/layers/quantization/cpu_int4.py` (+ tests).
|
|
|
|
## 0a. Correction: the llama.cpp numbers were measured cold
|
|
|
|
The Q4_K_M column in `spark_edge_support.md` was taken on a cold page cache.
|
|
The GGUF lives on a FUSE network filesystem and llama.cpp mmaps it, so the
|
|
first grid was partly timing storage. Warm, same pinned cores:
|
|
|
|
| llama.cpp Q4_K_M, 32 threads | pp512 | tg128 |
|
|
|---|---|---|
|
|
| as previously published (cold) | 1199 | 82.81 |
|
|
| warm | **1356** | **98.02** |
|
|
|
|
The bar this work has to clear is 98 tok/s, not 82.8.
|
|
|
|
## 0b. Correction: vLLM's CPU backend *does* ship a 4-bit kernel
|
|
|
|
`spark_edge_support.md` reported that vLLM 0.28's CPU build has no int4 dense
|
|
kernel, on the evidence that `woq_int4_linear`, `int4_scaled_mm_with_quant`
|
|
and `da8w4_linear` are all absent. They are -- but the kernel exists under a
|
|
different name. `torch.ops._C.int4_scaled_mm_cpu` is present in the released
|
|
CPU wheel: an **AMX W4A8** GEMM, reached by loading a GPTQ checkpoint with
|
|
`VLLM_CPU_INT4_W4A8=1`, with `cpu_gemm_wna16` as the W4A16 fallback. The
|
|
search missed it because it searched for names, not for the capability.
|
|
|
|
This matters, because it is the fastest option by a wide margin. Measured at
|
|
Spark's shapes against PyTorch's `aten::_weight_int4pack_mm_for_cpu`
|
|
(`bench_vllm_w4a8.py`, 28 layers + head, group 128):
|
|
|
|
| batch | tinygemm | vLLM W4A8 (AMX) | speedup |
|
|
|---|---|---|---|
|
|
| 1 row (decode) | 4.45 ms | **2.66 ms** | 1.67x |
|
|
| 8 rows | 14.25 ms | **6.76 ms** | 2.11x |
|
|
| 512 rows (prefill) | 878 ms | **104 ms** | 8.44x |
|
|
|
|
## 1. What was actually slow
|
|
|
|
`profile_cpu_step.py` attributes one decode step. At bf16, 32 threads:
|
|
|
|
| bucket | ms/step | share |
|
|
|---|---|---|
|
|
| `_C::onednn_mm` (113 GEMMs) | 21.24 | 76% |
|
|
| inductor-generated kernels (norms, rope, GEGLU) | 2.47 | 9% |
|
|
| attention + KV cache ops (4 x 28) | 1.63 | 6% |
|
|
| head-gate `aten::mm` (28) | 1.08 | 4% |
|
|
| sampling (`argmax` over 131072) | 0.24 | 1% |
|
|
| ~900 small view/slice ops | 0.55 | 2% |
|
|
| **total** | **27.96** | 35.8 tok/s |
|
|
|
|
3.42 GB of bf16 weights at ~161 GB/s effective. Only fewer bits move it.
|
|
|
|
## 2. Results
|
|
|
|
Xeon Platinum 8480C, one socket, 32 physical cores pinned with `taskset`,
|
|
warm page cache, engines run strictly one at a time, and every run records the
|
|
busy fraction of its own cores before it starts (`cpu_busy.py`) -- a
|
|
shared-core benchmark measures the scheduler, not the engine. Perplexity is
|
|
teacher-forced over 512-token windows, 126 217 scored tokens, each engine
|
|
scored against **its own** bf16 run so tokenizer and chunking differences
|
|
cancel.
|
|
|
|
| build | bits/weight | pp512 | tg128 | ppl vs own bf16 |
|
|
|---|---|---|---|---|
|
|
| vLLM bf16 | 16 | 4455 | 36.35 | -- |
|
|
| vLLM GPTQ W4A16 (sym, g128) | 4.16 | **4659** | 37.53 | +25.5% |
|
|
| **vLLM GPTQ W4A8 (sym, g128)** | 4.16 | **4355** | **88.7** | +27.2% |
|
|
| vLLM `cpu_int4` (asym, g128) | 4.25 | 3589 | 85.5 | +18.2% |
|
|
| vLLM `cpu_int4` (asym, g64) | 4.50 | 3689 | 81.1 | +12.5% |
|
|
| llama.cpp Q4_K_M | 5.15 | 1356 | **98.02** | **+7.6%** |
|
|
|
|
Against llama.cpp at 4-bit: **prefill 3.21x faster, decode 0.90x.** Decode
|
|
went 36.35 -> 88.7, a 2.44x gain, which turns a 2.70x deficit into a 1.11x
|
|
one without quite closing it.
|
|
|
|
Quantizing the head matters more than anything else in that table: Spark ties
|
|
its 131072 x 2048 vocab table to the output head, and that one GEMM is the
|
|
largest read in a decode step. Leaving it in bf16 costs 12 tok/s (76.0 vs
|
|
88.7), so `export_gptq_cpu.py --quantize-lm-head` unties it, keeping the
|
|
table in bf16 for the lookup (a few hundred bytes a token) and giving the
|
|
head its own 4-bit copy.
|
|
|
|
Note the W4A16 row: best prefill of anything measured, worst decode of the
|
|
4-bit builds. It dequantizes to bf16 per call, which amortizes over 512 rows
|
|
and is pure overhead on one.
|
|
|
|
### Thread scaling (cores 56-87, both engines, default OpenMP settings)
|
|
|
|
Measured with `cpu_int4`, before the W4A8 path was found, but the shape of it
|
|
is the point:
|
|
|
|
| threads | vLLM int4 pp512 | llama.cpp pp512 | vLLM int4 tg128 | llama.cpp tg128 |
|
|
|---|---|---|---|---|
|
|
| 4 | **800.1** | 210.2 | **27.08** | 24.20 |
|
|
| 8 | **1476.2** | 409.8 | 44.02 | 44.00 |
|
|
| 16 | **2599.1** | 799.2 | 66.72 | **73.19** |
|
|
| 32 | **3524.9** | 1348.0 | 80.20 | **99.55** |
|
|
|
|
Decode wins at 4 threads, ties at 8, loses from 16 up. That is the signature
|
|
of a fixed per-step cost that does not shrink when the GEMMs do.
|
|
|
|
### What else moved the needle, and what didn't
|
|
|
|
| change | tg128 | |
|
|
|---|---|---|
|
|
| `cpu_int4` g64 baseline | 81.10 | |
|
|
| `KMP_BLOCKTIME=infinite OMP_WAIT_POLICY=ACTIVE KMP_LIBRARY=turnaround` | 83.61 | ~250 OpenMP parallel regions per step; how threads park between them is not free |
|
|
| group 128 instead of 64 | 85.48 | 5.5% fewer weight bytes |
|
|
| `detokenize=False` (what `llama-bench` measures) | 82.13 | |
|
|
|
|
Tried and rejected: `compile_sizes=[1]`, to give inductor static shapes for
|
|
the decode step (81.77, no change); `enforce_eager` (58.88, much worse);
|
|
inductor `cpp_wrapper` (fails outright -- vLLM's own ops take a `str`
|
|
argument `custom_op_wrapper` cannot box).
|
|
|
|
## 3. The remaining decode gap is framework, not kernel
|
|
|
|
Re-profiling the `cpu_int4` step at 12.50 ms:
|
|
|
|
| bucket | ms/step |
|
|
|---|---|
|
|
| int4 GEMMs (113) | 5.49 |
|
|
| inductor-generated kernels | 2.29 |
|
|
| attention + KV cache ops | 1.41 |
|
|
| head-gate `aten::mm` (28) | 0.92 |
|
|
| custom-op dispatch (113) | 0.58 |
|
|
| sampling + small ops | 0.80 |
|
|
| **total** | **12.50** |
|
|
|
|
Those GEMMs move 0.95 GB in 5.49 ms, **173 GB/s**; llama.cpp does its whole
|
|
10.2 ms step against 1.1 GB, ~108 GB/s all-in. The 4-bit kernel is not what
|
|
is losing. **7.0 ms of the 12.5 ms step is per-step framework cost**, and
|
|
llama.cpp has almost none of it. With the faster W4A8 kernel the GEMM share
|
|
falls further and the framework share rises, which is why the gap closes to
|
|
0.90x and then stops.
|
|
|
|
Closing the rest is vLLM CPU engine work -- the attention wrappers, the ~900
|
|
small view ops, inductor launch overhead -- and it is out of reach here
|
|
because vLLM itself is off-limits to modification. The one piece inside reach
|
|
is the head gate: 28 `aten::mm` calls, 0.92 ms, almost all thread barrier for
|
|
a 2048x8 matmul. Rewriting it as a fused broadcast-and-reduce was tried and is
|
|
much worse (44.5 tok/s; inductor does not vectorize the reduction). Folding
|
|
those eight rows into the fused QKV GEMM would remove it, and is the obvious
|
|
next step.
|
|
|
|
## 4. Quality: the grid, not the runtime
|
|
|
|
Q4_K_M spends **5.15 bits per weight** (1.1 GB for 1.7 B parameters) against
|
|
4.16-4.50 here, and still loses less: +7.6% perplexity against +12.5% for the
|
|
best of ours. Two separate things cause that.
|
|
|
|
**vLLM's GPTQ path cannot express an asymmetric grid at 4 bits.** It rejects
|
|
`sym=False` outright ("Unsupported quantization config: bits=4, sym=False"),
|
|
so the fast W4A8 kernel can only be fed a symmetric grid -- which wastes a
|
|
level on every group whose weights are off-centre. That is most of why the
|
|
W4A8 rows cost +27% while `cpu_int4`, which is asymmetric, costs +12.5% at a
|
|
comparable bit budget. The AMX kernel itself takes real zero points
|
|
(`convert_weight_packed_scale_zp(..., CPUQuantAlgo.AWQ)`); it is the GPTQ
|
|
*config* that refuses. Exporting AWQ format instead should get an asymmetric
|
|
grid through the same kernel, and is the highest-value thing left undone
|
|
here.
|
|
|
|
**W4A8 quantizes activations to int8, and that is nearly free.** W4A16 on the
|
|
same checkpoint scores +25.5% against W4A8's +27.2% -- a 1.7 point difference
|
|
for a 2.4x decode speedup. That is a much better outcome than int8
|
|
activations have had elsewhere in this repo (Qwen3-TTS, MiniCPM-o, Spark on
|
|
the NPU), and worth recording as the counter-example.
|
|
|
|
**Even asymmetric, our grid is coarser than K-quants.** `cpu_int4` does
|
|
round-to-nearest on each group's min/max with a small shrink search; the
|
|
search is worth about what it costs (at g64: 48.0 perplexity with it, 59.7
|
|
without, on the smaller corpus) but it optimizes per-group squared error,
|
|
which is the wrong objective. Q4_K_M uses an importance-weighted scale search
|
|
per 32-weight sub-block and spends more bits on the tensors that need them.
|
|
|
|
This is a quantizer gap, not a runtime gap, and the control proves it:
|
|
expanding a quantized checkpoint back to plain bf16 offline
|
|
(`dequantize_int4.py`) and scoring that reproduces the quantized run to
|
|
**within 0.2%** (48.08 vs 48.01 at g64, 56.80 vs 56.89 at g32). Both GEMM
|
|
paths, the nibble packing and the quantized tied embedding are all faithful.
|
|
|
|
A caution on measuring this: on a smaller, harder corpus (34 k words of this
|
|
repo's own notes, bf16 perplexity 43.7) the ordering is not even stable --
|
|
group 32 lands worse than group 128 there, and reverses on the larger corpus.
|
|
Perplexity differences between 4-bit variants of a 1.7 B model need a corpus
|
|
big enough to be in a sane regime before they mean anything.
|
|
|
|
## 5. An AMX hazard worth recording
|
|
|
|
Running PyTorch's tinygemm kernel on a multi-row batch and then vLLM's
|
|
`cpu_attention_with_kv_cache` in the same forward kills the process with
|
|
**SIGILL inside the attention kernel** -- the signature of an AMX instruction
|
|
issued with no valid tile configuration loaded. The identical prompt is fine
|
|
in bf16, and fine at any batch size once every multi-row GEMM takes a
|
|
different path, so the two kernels disagree about who owns the AMX tile state.
|
|
`cpu_int4` therefore routes only single-row steps to tinygemm
|
|
(`dequant_threshold` defaults to 2). vLLM's own W4A8 path does not have the
|
|
problem.
|
|
|
|
## 6. Where this leaves it
|
|
|
|
| | before | after |
|
|
|---|---|---|
|
|
| decode vs llama.cpp Q4_K_M | 0.37x (36.35 vs 98.02) | **0.90x** (88.7) |
|
|
| prefill vs llama.cpp Q4_K_M | no 4-bit path in use | **3.21x** (4355 vs 1356) |
|
|
| quality cost of 4-bit | -- | +27.2% ppl (W4A8) / +12.5% (`cpu_int4` g64) vs Q4_K_M's +7.6% |
|
|
| model on disk | 3.4 GB | 0.9-1.4 GB |
|
|
|
|
Honest summary: 4-bit weights turn a 2.7x single-stream decode deficit into a
|
|
1.11x one and put prefill 3.2x ahead, but vLLM still does not beat llama.cpp
|
|
at single-stream decode on 32 cores, and our 4-bit grids are measurably
|
|
coarser than K-quants. Three specific, measured things stand between here and
|
|
a win, in order of expected value:
|
|
|
|
1. **An AWQ-format export**, to get an asymmetric grid through the AMX W4A8
|
|
kernel -- worth roughly the +27.2% -> +12.5% perplexity difference at no
|
|
speed cost.
|
|
2. **Folding the head gate into the QKV GEMM** -- 0.92 ms of a ~11 ms step.
|
|
3. **vLLM CPU per-step overhead** -- 7.0 ms at batch 1, the whole remaining
|
|
gap, and not addressable without changing vLLM.
|
|
|
|
`cpu_int4` keeps a narrower role than intended: it is the more accurate of the
|
|
two (asymmetric grid, +12.5% against +27.2%) and it needs no AMX, but on an
|
|
AMX host vLLM's own W4A8 path is faster on both axes. Its docstring says so.
|