vllm-omni/docs/analysis/spark_cpu_int4.md

Closing the CPU decode gap: 4-bit weights for Spark-X2.5 on vLLM

analysis/spark_edge_support.md ended with one open item. At matched bf16, vLLM beat llama.cpp on Spark-X2.5-1.7B everywhere -- 8.7-15.4x on prefill, 1.19-2.65x on decode -- but llama.cpp's Q4_K_M build won single-stream decode outright. Decode at batch 1 reads every weight to emit one token, so bits per weight, not flops, set the token rate, and vLLM had no 4-bit CPU path in use. This note closes that and re-runs the comparison at matched precision.

Two corrections to earlier findings come first, because both changed the answer.

Code: analysis/experiments/spark_edge/{export_gptq_cpu,quantize_int4,ppl_vllm,profile_cpu_step,bench_vllm_w4a8,cmp_llamacpp,dequantize_int4}.py, vllm_omni/model_executor/layers/quantization/cpu_int4.py (+ tests).

0a. Correction: the llama.cpp numbers were measured cold

The Q4_K_M column in spark_edge_support.md was taken on a cold page cache. The GGUF lives on a FUSE network filesystem and llama.cpp mmaps it, so the first grid was partly timing storage. Warm, same pinned cores:

llama.cpp Q4_K_M, 32 threads	pp512	tg128
as previously published (cold)	1199	82.81
warm	1356	98.02

The bar this work has to clear is 98 tok/s, not 82.8.

0b. Correction: vLLM's CPU backend does ship a 4-bit kernel

spark_edge_support.md reported that vLLM 0.28's CPU build has no int4 dense kernel, on the evidence that woq_int4_linear, int4_scaled_mm_with_quant and da8w4_linear are all absent. They are -- but the kernel exists under a different name. torch.ops._C.int4_scaled_mm_cpu is present in the released CPU wheel: an AMX W4A8 GEMM, reached by loading a GPTQ checkpoint with VLLM_CPU_INT4_W4A8=1, with cpu_gemm_wna16 as the W4A16 fallback. The search missed it because it searched for names, not for the capability.

This matters, because it is the fastest option by a wide margin. Measured at Spark's shapes against PyTorch's aten::_weight_int4pack_mm_for_cpu (bench_vllm_w4a8.py, 28 layers + head, group 128):

batch	tinygemm	vLLM W4A8 (AMX)	speedup
1 row (decode)	4.45 ms	2.66 ms	1.67x
8 rows	14.25 ms	6.76 ms	2.11x
512 rows (prefill)	878 ms	104 ms	8.44x

1. What was actually slow

profile_cpu_step.py attributes one decode step. At bf16, 32 threads:

bucket	ms/step	share
_C::onednn_mm (113 GEMMs)	21.24	76%
inductor-generated kernels (norms, rope, GEGLU)	2.47	9%
attention + KV cache ops (4 x 28)	1.63	6%
head-gate aten::mm (28)	1.08	4%
sampling (argmax over 131072)	0.24	1%
~900 small view/slice ops	0.55	2%
total	27.96	35.8 tok/s

3.42 GB of bf16 weights at ~161 GB/s effective. Only fewer bits move it.

2. Results

Xeon Platinum 8480C, one socket, 32 physical cores pinned with taskset, warm page cache, engines run strictly one at a time, and every run records the busy fraction of its own cores before it starts (cpu_busy.py) -- a shared-core benchmark measures the scheduler, not the engine. Perplexity is teacher-forced over 512-token windows, 126 217 scored tokens, each engine scored against its own bf16 run so tokenizer and chunking differences cancel.

build	bits/weight	pp512	tg128	ppl vs own bf16
vLLM bf16	16	4455	36.35	--
vLLM GPTQ W4A16 (sym, g128)	4.16	4659	37.53	+25.5%
vLLM GPTQ W4A8 (sym, g128)	4.16	4355	88.7	+27.2%
vLLM cpu_int4 (asym, g128)	4.25	3589	85.5	+18.2%
vLLM cpu_int4 (asym, g64)	4.50	3689	81.1	+12.5%
llama.cpp Q4_K_M	5.15	1356	98.02	+7.6%

Against llama.cpp at 4-bit: prefill 3.21x faster, decode 0.90x. Decode went 36.35 -> 88.7, a 2.44x gain, which turns a 2.70x deficit into a 1.11x one without quite closing it.

Quantizing the head matters more than anything else in that table: Spark ties its 131072 x 2048 vocab table to the output head, and that one GEMM is the largest read in a decode step. Leaving it in bf16 costs 12 tok/s (76.0 vs 88.7), so export_gptq_cpu.py --quantize-lm-head unties it, keeping the table in bf16 for the lookup (a few hundred bytes a token) and giving the head its own 4-bit copy.

Note the W4A16 row: best prefill of anything measured, worst decode of the 4-bit builds. It dequantizes to bf16 per call, which amortizes over 512 rows and is pure overhead on one.

Thread scaling (cores 56-87, both engines, default OpenMP settings)

Measured with cpu_int4, before the W4A8 path was found, but the shape of it is the point:

threads	vLLM int4 pp512	llama.cpp pp512	vLLM int4 tg128	llama.cpp tg128
4	800.1	210.2	27.08	24.20
8	1476.2	409.8	44.02	44.00
16	2599.1	799.2	66.72	73.19
32	3524.9	1348.0	80.20	99.55

Decode wins at 4 threads, ties at 8, loses from 16 up. That is the signature of a fixed per-step cost that does not shrink when the GEMMs do.

What else moved the needle, and what didn't

change	tg128
cpu_int4 g64 baseline	81.10
KMP_BLOCKTIME=infinite OMP_WAIT_POLICY=ACTIVE KMP_LIBRARY=turnaround	83.61	~250 OpenMP parallel regions per step; how threads park between them is not free
group 128 instead of 64	85.48	5.5% fewer weight bytes
detokenize=False (what llama-bench measures)	82.13

Tried and rejected: compile_sizes=[1], to give inductor static shapes for the decode step (81.77, no change); enforce_eager (58.88, much worse); inductor cpp_wrapper (fails outright -- vLLM's own ops take a str argument custom_op_wrapper cannot box).

3. The remaining decode gap is framework, not kernel

Re-profiling the cpu_int4 step at 12.50 ms:

bucket	ms/step
int4 GEMMs (113)	5.49
inductor-generated kernels	2.29
attention + KV cache ops	1.41
head-gate aten::mm (28)	0.92
custom-op dispatch (113)	0.58
sampling + small ops	0.80
total	12.50

Those GEMMs move 0.95 GB in 5.49 ms, 173 GB/s; llama.cpp does its whole 10.2 ms step against 1.1 GB, ~108 GB/s all-in. The 4-bit kernel is not what is losing. 7.0 ms of the 12.5 ms step is per-step framework cost, and llama.cpp has almost none of it. With the faster W4A8 kernel the GEMM share falls further and the framework share rises, which is why the gap closes to 0.90x and then stops.

Closing the rest is vLLM CPU engine work -- the attention wrappers, the ~900 small view ops, inductor launch overhead -- and it is out of reach here because vLLM itself is off-limits to modification. The one piece inside reach is the head gate: 28 aten::mm calls, 0.92 ms, almost all thread barrier for a 2048x8 matmul. Rewriting it as a fused broadcast-and-reduce was tried and is much worse (44.5 tok/s; inductor does not vectorize the reduction). Folding those eight rows into the fused QKV GEMM would remove it, and is the obvious next step.

4. Quality: the grid, not the runtime

Q4_K_M spends 5.15 bits per weight (1.1 GB for 1.7 B parameters) against 4.16-4.50 here, and still loses less: +7.6% perplexity against +12.5% for the best of ours. Two separate things cause that.

vLLM's GPTQ path cannot express an asymmetric grid at 4 bits. It rejects sym=False outright ("Unsupported quantization config: bits=4, sym=False"), so the fast W4A8 kernel can only be fed a symmetric grid -- which wastes a level on every group whose weights are off-centre. That is most of why the W4A8 rows cost +27% while cpu_int4, which is asymmetric, costs +12.5% at a comparable bit budget. The AMX kernel itself takes real zero points (convert_weight_packed_scale_zp(..., CPUQuantAlgo.AWQ)); it is the GPTQ config that refuses. Exporting AWQ format instead should get an asymmetric grid through the same kernel, and is the highest-value thing left undone here.

W4A8 quantizes activations to int8, and that is nearly free. W4A16 on the same checkpoint scores +25.5% against W4A8's +27.2% -- a 1.7 point difference for a 2.4x decode speedup. That is a much better outcome than int8 activations have had elsewhere in this repo (Qwen3-TTS, MiniCPM-o, Spark on the NPU), and worth recording as the counter-example.

Even asymmetric, our grid is coarser than K-quants. cpu_int4 does round-to-nearest on each group's min/max with a small shrink search; the search is worth about what it costs (at g64: 48.0 perplexity with it, 59.7 without, on the smaller corpus) but it optimizes per-group squared error, which is the wrong objective. Q4_K_M uses an importance-weighted scale search per 32-weight sub-block and spends more bits on the tensors that need them.

This is a quantizer gap, not a runtime gap, and the control proves it: expanding a quantized checkpoint back to plain bf16 offline (dequantize_int4.py) and scoring that reproduces the quantized run to within 0.2% (48.08 vs 48.01 at g64, 56.80 vs 56.89 at g32). Both GEMM paths, the nibble packing and the quantized tied embedding are all faithful.

A caution on measuring this: on a smaller, harder corpus (34 k words of this repo's own notes, bf16 perplexity 43.7) the ordering is not even stable -- group 32 lands worse than group 128 there, and reverses on the larger corpus. Perplexity differences between 4-bit variants of a 1.7 B model need a corpus big enough to be in a sane regime before they mean anything.

5. An AMX hazard worth recording

Running PyTorch's tinygemm kernel on a multi-row batch and then vLLM's cpu_attention_with_kv_cache in the same forward kills the process with SIGILL inside the attention kernel -- the signature of an AMX instruction issued with no valid tile configuration loaded. The identical prompt is fine in bf16, and fine at any batch size once every multi-row GEMM takes a different path, so the two kernels disagree about who owns the AMX tile state. cpu_int4 therefore routes only single-row steps to tinygemm (dequant_threshold defaults to 2). vLLM's own W4A8 path does not have the problem.

6. Where this leaves it

	before	after
decode vs llama.cpp Q4_K_M	0.37x (36.35 vs 98.02)	0.90x (88.7)
prefill vs llama.cpp Q4_K_M	no 4-bit path in use	3.21x (4355 vs 1356)
quality cost of 4-bit	--	+27.2% ppl (W4A8) / +12.5% (cpu_int4 g64) vs Q4_K_M's +7.6%
model on disk	3.4 GB	0.9-1.4 GB

Honest summary: 4-bit weights turn a 2.7x single-stream decode deficit into a 1.11x one and put prefill 3.2x ahead, but vLLM still does not beat llama.cpp at single-stream decode on 32 cores, and our 4-bit grids are measurably coarser than K-quants. Three specific, measured things stand between here and a win, in order of expected value:

1. An AWQ-format export, to get an asymmetric grid through the AMX W4A8 kernel -- worth roughly the +27.2% -> +12.5% perplexity difference at no speed cost.
2. Folding the head gate into the QKV GEMM -- 0.92 ms of a ~11 ms step.
3. vLLM CPU per-step overhead -- 7.0 ms at batch 1, the whole remaining gap, and not addressable without changing vLLM.

cpu_int4 keeps a narrower role than intended: it is the more accurate of the two (asymmetric grid, +12.5% against +27.2%) and it needs no AMX, but on an AMX host vLLM's own W4A8 path is faster on both axes. Its docstring says so.