From cab5ef701049cfed48a5cd0034e3104a46576f16 Mon Sep 17 00:00:00 2001 From: tzhouam Date: Sat, 12 Sep 2026 21:43:19 +0800 Subject: [PATCH] [Edge][CPU] Price the dual weight layout properly: 740 MB for 7.3x prefill The W4A16 path keeps two weight layouts -- the tinygemm one decode reads and a row-major copy prefill dequantizes -- and the docstring said only that it cost "8 bits per weight resident". Measured, median of three interleaved passes on anonymous RSS (spread under 1.5%, where VmRSS moved 1-2 GB between identical runs): prefill_dequant resident prefill decode true (default) 3542 MB 3252 tok/s 76.9 tok/s false 2802 MB 448 tok/s 78.5 tok/s So it is 740 MB for 7.3x prefill. Default stays on, because losing 7x of prefill hurts more than 740 MB on anything with memory to spare -- but on a device where it does not, this is the smallest resident build in the tree: 2.8 GB against the 7.3 GB the W4A8 path started at before the load-time allocator release. Co-Authored-By: Claude Opus 5 (1M context) --- .../layers/quantization/cpu_int4.py | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/vllm_omni/model_executor/layers/quantization/cpu_int4.py b/vllm_omni/model_executor/layers/quantization/cpu_int4.py index 2b2ee6470..f9fa21457 100644 --- a/vllm_omni/model_executor/layers/quantization/cpu_int4.py +++ b/vllm_omni/model_executor/layers/quantization/cpu_int4.py @@ -38,8 +38,19 @@ oneDNN/AMX GEMM, which costs one extra pass over the weights and wins back prefill. That path reads a row-major copy, so with ``prefill_dequant`` on (the default) a layer holds both layouts -- 8 bits per weight resident, but still 4.5 bits per weight *read* in a decode step, which is what sets the -token rate. Set ``prefill_dequant: false`` to trade prefill for the smaller -footprint. +token rate. + +Measured on Spark-X2.5-1.7B (Xeon 8480C, anonymous RSS, median of three +interleaved passes, spread under 1.5%): + + prefill_dequant resident prefill decode + true (default) 3542 MB 3252 tok/s 76.9 tok/s + false 2802 MB 448 tok/s 78.5 tok/s + +So the second layout costs **740 MB** and buys **7.3x prefill**, because the +tinygemm kernel dequantizes once per row of activations and its cost is linear +in the batch. Keep it on unless memory is tighter than prefill -- on a device +where it is, ``prefill_dequant: false`` is the smallest resident build here. ``dequant_threshold`` defaults to 2, i.e. only a single-row step takes the tinygemm path. That is not just a speed choice. On a Sapphire Rapids host,