The GPTQ checkpoint the comparison had been using left Spark's output head
tied and unquantized, so that one GEMM ran bf16 through `_C::onednn_mm` at
2.97 ms/step -- more than the whole 4-bit saving, and enough to make the
W4A8/W4A16 ordering flip between grids. The per-op profile is what caught it:
`int4_scaled_mm_cpu` ran 112 times (28 layers x 4 linears) and `onednn_mm`
once.
Rebuilt with the head quantized and re-profiled back to back on the same 24
cores, because this host's throughput swings ~30% and the gap is ~8%:
w4a8 AMX g128 GEMM 5.91 ms step 11.55 ms 86.6 tok/s
w4a16 tinygemm GEMM 6.85 ms step 12.51 ms 80.0 tok/s
llama.cpp Q4_K_M step 11.11 ms 90.0 tok/s
W4A8 wins twice over: the kernel is 0.94 ms cheaper per step, and it is
reached through vLLM's own dispatch, so it also avoids the 0.48 ms/step
custom-op boundary the tinygemm path needs to send prefill to a different
kernel. The selector's recommendation is unchanged; only its numbers were
wrong, and the lm_head trap is now documented where someone building a
checkpoint will read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
