[Edge][CPU] Price the dual weight layout properly: 740 MB for 7.3x prefill
The W4A16 path keeps two weight layouts -- the tinygemm one decode reads and a
row-major copy prefill dequantizes -- and the docstring said only that it cost
"8 bits per weight resident". Measured, median of three interleaved passes on
anonymous RSS (spread under 1.5%, where VmRSS moved 1-2 GB between identical
runs):
prefill_dequant resident prefill decode
true (default) 3542 MB 3252 tok/s 76.9 tok/s
false 2802 MB 448 tok/s 78.5 tok/s
So it is 740 MB for 7.3x prefill. Default stays on, because losing 7x of
prefill hurts more than 740 MB on anything with memory to spare -- but on a
device where it does not, this is the smallest resident build in the tree:
2.8 GB against the 7.3 GB the W4A8 path started at before the load-time
allocator release.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a7e6c412ab
commit
cab5ef7010
1 file changed
+13
-2
@@ -38,8 +38,19 @@ oneDNN/AMX GEMM, which costs one extra pass over the weights and wins back
|
||||
prefill. That path reads a row-major copy, so with ``prefill_dequant`` on
|
||||
(the default) a layer holds both layouts -- 8 bits per weight resident, but
|
||||
still 4.5 bits per weight *read* in a decode step, which is what sets the
|
||||
token rate. Set ``prefill_dequant: false`` to trade prefill for the smaller
|
||||
footprint.
|
||||
token rate.
|
||||
|
||||
Measured on Spark-X2.5-1.7B (Xeon 8480C, anonymous RSS, median of three
|
||||
interleaved passes, spread under 1.5%):
|
||||
|
||||
prefill_dequant resident prefill decode
|
||||
true (default) 3542 MB 3252 tok/s 76.9 tok/s
|
||||
false 2802 MB 448 tok/s 78.5 tok/s
|
||||
|
||||
So the second layout costs **740 MB** and buys **7.3x prefill**, because the
|
||||
tinygemm kernel dequantizes once per row of activations and its cost is linear
|
||||
in the batch. Keep it on unless memory is tighter than prefill -- on a device
|
||||
where it is, ``prefill_dequant: false`` is the smallest resident build here.
|
||||
|
||||
``dequant_threshold`` defaults to 2, i.e. only a single-row step takes the
|
||||
tinygemm path. That is not just a speed choice. On a Sapphire Rapids host,
|
||||
|
||||
Reference in new issue
Block a user