From 5f27f60bdefd53eb0f6f7aabc6a252c479062f3a Mon Sep 17 00:00:00 2001 From: tzhouam Date: Sat, 12 Sep 2026 13:50:35 +0800 Subject: [PATCH] [Edge] Correct the W4A8/W4A16 figures: the earlier ones had an unquantized head The GPTQ checkpoint the comparison had been using left Spark's output head tied and unquantized, so that one GEMM ran bf16 through `_C::onednn_mm` at 2.97 ms/step -- more than the whole 4-bit saving, and enough to make the W4A8/W4A16 ordering flip between grids. The per-op profile is what caught it: `int4_scaled_mm_cpu` ran 112 times (28 layers x 4 linears) and `onednn_mm` once. Rebuilt with the head quantized and re-profiled back to back on the same 24 cores, because this host's throughput swings ~30% and the gap is ~8%: w4a8 AMX g128 GEMM 5.91 ms step 11.55 ms 86.6 tok/s w4a16 tinygemm GEMM 6.85 ms step 12.51 ms 80.0 tok/s llama.cpp Q4_K_M step 11.11 ms 90.0 tok/s W4A8 wins twice over: the kernel is 0.94 ms cheaper per step, and it is reached through vLLM's own dispatch, so it also avoids the 0.48 ms/step custom-op boundary the tinygemm path needs to send prefill to a different kernel. The selector's recommendation is unchanged; only its numbers were wrong, and the lm_head trap is now documented where someone building a checkpoint will read it. Co-Authored-By: Claude Opus 5 (1M context) --- vllm_omni/edge/weight_path.py | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/vllm_omni/edge/weight_path.py b/vllm_omni/edge/weight_path.py index 3cf7e64b0..9b44dca63 100644 --- a/vllm_omni/edge/weight_path.py +++ b/vllm_omni/edge/weight_path.py @@ -17,14 +17,26 @@ one needs a different weight layout, and the layouts are mutually exclusive: So the choice is a property of the device, and picking it at deploy time from a probe is the difference between "4-bit is 22% faster" and "4-bit is broken -here". Measured on a Xeon 8480C (24 threads, Spark-X2.5-1.7B, best of three -interleaved passes, tok/s decode / prefill): +here". Measured on a Xeon 8480C, 24 threads, Spark-X2.5-1.7B. The two 4-bit +rows are from per-op profiles taken back to back on the same cores, because +this host's throughput swings ~30% and the gap between them is ~8%: - bf16 41.0 / 4796 no 4-bit kernel used - w4a16 tinygemm g64 75.1 / 3207 - w4a16 tinygemm g128 78.7 / 3346 more faithful of the two - w4a8 AMX g128 84.5 / 3347 fastest; least faithful - llama.cpp Q4_K_M 90.0 / 1121 the bar + path GEMM/step step tok/s + w4a8 AMX g128 5.91 ms 11.55 86.6 fastest; least faithful + w4a16 tinygemm g128 6.85 ms 12.51 80.0 more faithful + llama.cpp Q4_K_M -- 11.11 90.0 the bar + bf16 (no 4-bit kernel) -- -- 41.0 + +W4A8 wins twice over: `int4_scaled_mm_cpu` is 0.94 ms cheaper than tinygemm +per step, and it is reached through vLLM's own kernel dispatch, so it also +avoids the 0.48 ms/step custom-op boundary the tinygemm path needs to route +prefill and decode to different kernels. + +One trap worth naming: a GPTQ checkpoint must quantize the output head +(`lm_head`), or that one GEMM runs in bf16 through `onednn_mm` and costs +2.97 ms/step by itself -- more than the whole 4-bit saving. Spark ties its +embedding to its head, so the head has to be untied and emitted separately; +`quantize_gptq_int4.py` does that by default. Fidelity is the reason both 4-bit rows stay: against each build's own bf16 output over 12 greedy prompts, llama.cpp's Q4_K_M reproduced 4/12 exactly,