edge: KV sizing is the largest memory lever, not a dead end
This module carried a retraction saying the measured saving from right-sizing
the cache was 188 MB. That retraction was itself measured through a path that
ignored the knob: CpuPlatform.check_and_update_config overwrote
kv_cache_memory_bytes from VLLM_CPU_KVCACHE_SPACE unconditionally, so the
"32x shrink" never happened and both arms ran the same cache.
With the budget actually honoured, 3 interleaved passes per arm, both filling
a 1536-token context:
1 GiB budget 3264 MB resident
192 MiB budget 2421 MB resident -> 843 MB
And the reasoning behind "the pages are never touched" is wrong too. mincore(2)
reports the cache fully in core: 1024.1 MB resident of 1023.8 allocated, and
191.1 of 190.8 when sized down. It is not address space, it is memory.
843 MB is the largest single lever measured on this model -- more than the
single weight layout (615 MB), the blocked repack (489 MB peak), or the RoPE
table (618 MB).
The general lesson, recorded in the report: a lever tested through a path that
silently ignores it measures nothing, and "no effect" is the most dangerous
result to accept without first checking the knob was connected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
This commit is contained in:
1 parent
df3bbfe675
commit
ee2129e2fa
1 file changed
+13
-7
@@ -13,13 +13,19 @@ pages and moved 1-2 GB between identical runs, and the two arms were not the
|
||||
same model file. Re-measured on ``RssAnon``, which repeats to 0.2%, the saving
|
||||
is **188 MB**.
|
||||
|
||||
That is small because of *when* it was measured, not because the budget does
|
||||
not matter: the benchmark generated 8 tokens, so almost no cache was ever
|
||||
touched. Allocation is what this module sizes, and the two coincide only at
|
||||
long context -- see the arithmetic below, where the flat and hybrid budgets at
|
||||
32 k differ by 1323 MiB. Sizing the cache is worth re-opening on a benchmark
|
||||
that actually fills it (harness item H3), and worth nothing on one that does
|
||||
not.
|
||||
**Re-measured 2026-09-14, and the retraction needs its own retraction.** The
|
||||
188 MB was itself measured through a path that ignored the knob: this platform
|
||||
silently overwrote ``kv_cache_memory_bytes`` from ``VLLM_CPU_KVCACHE_SPACE``,
|
||||
so both arms ran the same cache. With the budget actually honoured, and both
|
||||
arms filling a 1536-token context, 3 interleaved passes each:
|
||||
|
||||
1 GiB budget 3264 MB resident
|
||||
192 MiB budget 2421 MB resident -> 843 MB
|
||||
|
||||
The cache is **fully resident**, not allocated-and-untouched: ``mincore(2)``
|
||||
reports 1024.1 MB in core of 1023.8 MB allocated, and 191.1 of 190.8 when
|
||||
sized down. Sizing this correctly is the largest single memory lever measured
|
||||
on this model -- larger than the weight layout, the repack, or the RoPE table.
|
||||
|
||||
A hybrid-attention model does not *need* a full-length cache in every layer.
|
||||
Spark-X2.5 is 3 sliding layers (window 512) to 1 full layer over 28 layers, so
|
||||
|
||||
Reference in new issue
Block a user