From 6bad5a7cd46e9941aa0e3a6449a814bbffe0a43e Mon Sep 17 00:00:00 2001 From: tzhouam Date: Sun, 13 Sep 2026 11:06:22 +0800 Subject: [PATCH] cpu_int4: settle the prefill cost at 1.63x, correcting an overstated 1.9x The docstring said one layout costs 1.9x prefill. That came from the single surviving pass of a grid whose other passes had crashed on the AOT cache collision -- a number the harness itself had labelled "single pass, not a result", which I then published. Re-measured properly: caches isolated per arm so all passes complete, on idle node-1 cores with memory bound to the same node, pooled over 7 interleaved passes per arm. pp512 goes 2190 -> 1344 tok/s, i.e. 1.63x, not 1.9x. The ratio is only good to about 1.4-1.8x (worst pairing 1.84x, best 1.36x) because both dequantizing arms swing 10-18% here, while the tinygemm-only arm repeats to 0.3%. The existence and rough size of the cost is solid -- a 63% gap is far outside that band -- but the third digit is not, and the docstring now says so rather than implying precision it does not have. The comparison that is reliable is against prefill_dequant: false, whose arm is the steady one: 2.97x its prefill at the same memory, without routing multi-row batches into the tinygemm/AMX SIGILL hazard. That setting stays dominated. Also corrected a hypothesis of my own. The first three direct passes read 1344 -> 1421 -> 1555 and I called it a warm-up trend; four more passes gave 1271, 1335, 1443, 1309, so it was noise and three ascending points out of three at 12% spread meant nothing. Co-Authored-By: Claude Opus 5 (1M context) --- .../layers/quantization/cpu_int4.py | 23 +++++++++++-------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/vllm_omni/model_executor/layers/quantization/cpu_int4.py b/vllm_omni/model_executor/layers/quantization/cpu_int4.py index 1e19f469c..630aa878f 100644 --- a/vllm_omni/model_executor/layers/quantization/cpu_int4.py +++ b/vllm_omni/model_executor/layers/quantization/cpu_int4.py @@ -53,20 +53,25 @@ interleaved passes, RssAnon over the process tree). It is not free, because the duplicate was not only a copy. ``to_split_halves`` de-interleaves the nibbles *once at load*, so each prefill call reads two contiguous planes; reading the kernel layout redoes that -de-interleave every call, and prefill drops **2272 -> 1210 tok/s** (pp512, same -grid). Dequantizing to ``[K, N]`` and using ``matmul`` to dodge the transpose -was tried and is worse still (20.3 ms vs 14.4 per ``gate_up`` at 8 threads). -Recovering it wants a C++ de-interleave, not an ATen expression chain. +de-interleave every call. pp512 drops **2190 -> 1344 tok/s**, i.e. **1.63x**, +pooled over 7 interleaved passes per arm -- and the ratio is only good to about +1.4-1.8x, because both dequantizing arms swing 10-18% on a shared host while +the tinygemm-only arm reproduces to 0.3%. Dequantizing to ``[K, N]`` and using +``matmul`` to dodge the transpose was tried and is worse still (20.3 ms vs 14.4 +per ``gate_up`` at 8 threads). Recovering it wants a C++ de-interleave, not an +ATen expression chain. So this is a dial, not a free win: * versus ``prefill_dequant: false`` it is strictly better -- same memory to - within the noise, 2.7x the prefill (1210 vs ~450), and it keeps + within the noise, **2.97x** the prefill (1344 vs 453), and it keeps ``dequant_threshold`` in force so multi-row batches never reach the tinygemm - kernel, avoiding the SIGILL hazard below. That setting is dominated. -* versus keeping both layouts it trades 615 MB for 1.9x prefill. Default on, - because this path exists for devices picked for their memory ceiling; - ``VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT=0`` restores the old behaviour. + kernel, avoiding the SIGILL hazard below. That setting is dominated, and + this is the reliable comparison: the tinygemm-only arm repeats to 0.3%. +* versus keeping both layouts it trades 615 MB for ~1.6x prefill. Default on, + because this path exists for devices picked for their memory ceiling; set + ``direct_dequant: false`` in the checkpoint's ``quantization_config`` to + restore the old behaviour. ``supports_direct_dequant`` gates it: N has to tile the 64-channel block, which every Spark linear does (2048, 3072, 13312, 131072). An N like 80 packs as one