The docstring said one layout costs 1.9x prefill. That came from the single
surviving pass of a grid whose other passes had crashed on the AOT cache
collision -- a number the harness itself had labelled "single pass, not a
result", which I then published.
Re-measured properly: caches isolated per arm so all passes complete, on idle
node-1 cores with memory bound to the same node, pooled over 7 interleaved
passes per arm. pp512 goes 2190 -> 1344 tok/s, i.e. 1.63x, not 1.9x.
The ratio is only good to about 1.4-1.8x (worst pairing 1.84x, best 1.36x)
because both dequantizing arms swing 10-18% here, while the tinygemm-only arm
repeats to 0.3%. The existence and rough size of the cost is solid -- a 63%
gap is far outside that band -- but the third digit is not, and the docstring
now says so rather than implying precision it does not have.
The comparison that is reliable is against prefill_dequant: false, whose arm is
the steady one: 2.97x its prefill at the same memory, without routing multi-row
batches into the tinygemm/AMX SIGILL hazard. That setting stays dominated.
Also corrected a hypothesis of my own. The first three direct passes read
1344 -> 1421 -> 1555 and I called it a warm-up trend; four more passes gave
1271, 1335, 1443, 1309, so it was noise and three ascending points out of three
at 12% spread meant nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
