The W4A16 path keeps two weight layouts -- the tinygemm one decode reads and a
row-major copy prefill dequantizes -- and the docstring said only that it cost
"8 bits per weight resident". Measured, median of three interleaved passes on
anonymous RSS (spread under 1.5%, where VmRSS moved 1-2 GB between identical
runs):
prefill_dequant resident prefill decode
true (default) 3542 MB 3252 tok/s 76.9 tok/s
false 2802 MB 448 tok/s 78.5 tok/s
So it is 740 MB for 7.3x prefill. Default stays on, because losing 7x of
prefill hurts more than 740 MB on anything with memory to spare -- but on a
device where it does not, this is the smallest resident build in the tree:
2.8 GB against the 7.3 GB the W4A8 path started at before the load-time
allocator release.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
