[Edge] Correct the W4A8/W4A16 figures: the earlier ones had an unquantized head
The GPTQ checkpoint the comparison had been using left Spark's output head
tied and unquantized, so that one GEMM ran bf16 through `_C::onednn_mm` at
2.97 ms/step -- more than the whole 4-bit saving, and enough to make the
W4A8/W4A16 ordering flip between grids. The per-op profile is what caught it:
`int4_scaled_mm_cpu` ran 112 times (28 layers x 4 linears) and `onednn_mm`
once.
Rebuilt with the head quantized and re-profiled back to back on the same 24
cores, because this host's throughput swings ~30% and the gap is ~8%:
w4a8 AMX g128 GEMM 5.91 ms step 11.55 ms 86.6 tok/s
w4a16 tinygemm GEMM 6.85 ms step 12.51 ms 80.0 tok/s
llama.cpp Q4_K_M step 11.11 ms 90.0 tok/s
W4A8 wins twice over: the kernel is 0.94 ms cheaper per step, and it is
reached through vLLM's own dispatch, so it also avoids the 0.48 ms/step
custom-op boundary the tinygemm path needs to send prefill to a different
kernel. The selector's recommendation is unchanged; only its numbers were
wrong, and the lm_head trap is now documented where someone building a
checkpoint will read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f5d3bfcf93
commit
5f27f60bde
1 file changed
+19
-7
@@ -17,14 +17,26 @@ one needs a different weight layout, and the layouts are mutually exclusive:
|
||||
|
||||
So the choice is a property of the device, and picking it at deploy time from
|
||||
a probe is the difference between "4-bit is 22% faster" and "4-bit is broken
|
||||
here". Measured on a Xeon 8480C (24 threads, Spark-X2.5-1.7B, best of three
|
||||
interleaved passes, tok/s decode / prefill):
|
||||
here". Measured on a Xeon 8480C, 24 threads, Spark-X2.5-1.7B. The two 4-bit
|
||||
rows are from per-op profiles taken back to back on the same cores, because
|
||||
this host's throughput swings ~30% and the gap between them is ~8%:
|
||||
|
||||
bf16 41.0 / 4796 no 4-bit kernel used
|
||||
w4a16 tinygemm g64 75.1 / 3207
|
||||
w4a16 tinygemm g128 78.7 / 3346 more faithful of the two
|
||||
w4a8 AMX g128 84.5 / 3347 fastest; least faithful
|
||||
llama.cpp Q4_K_M 90.0 / 1121 the bar
|
||||
path GEMM/step step tok/s
|
||||
w4a8 AMX g128 5.91 ms 11.55 86.6 fastest; least faithful
|
||||
w4a16 tinygemm g128 6.85 ms 12.51 80.0 more faithful
|
||||
llama.cpp Q4_K_M -- 11.11 90.0 the bar
|
||||
bf16 (no 4-bit kernel) -- -- 41.0
|
||||
|
||||
W4A8 wins twice over: `int4_scaled_mm_cpu` is 0.94 ms cheaper than tinygemm
|
||||
per step, and it is reached through vLLM's own kernel dispatch, so it also
|
||||
avoids the 0.48 ms/step custom-op boundary the tinygemm path needs to route
|
||||
prefill and decode to different kernels.
|
||||
|
||||
One trap worth naming: a GPTQ checkpoint must quantize the output head
|
||||
(`lm_head`), or that one GEMM runs in bf16 through `onednn_mm` and costs
|
||||
2.97 ms/step by itself -- more than the whole 4-bit saving. Spark ties its
|
||||
embedding to its head, so the head has to be untied and emitted separately;
|
||||
`quantize_gptq_int4.py` does that by default.
|
||||
|
||||
Fidelity is the reason both 4-bit rows stay: against each build's own bf16
|
||||
output over 12 greedy prompts, llama.cpp's Q4_K_M reproduced 4/12 exactly,
|
||||
|
||||
Reference in new issue
Block a user