`VLLM_CPU_KVCACHE_SPACE` is a server-shaped default and is parsed with
`int()`, so it cannot express an edge-sized cache at all -- the smallest
non-zero value it accepts is 1 GiB. Measured on Spark-X2.5-1.7B (Xeon 8480C,
2048-token single stream), passing `kv_cache_memory_bytes` sized from the
model instead of the 4 GiB default takes resident memory from 8329 MB to
5254 MB, a 3.1 GB saving with no effect on output.
`kv_budget_for` reads the config and prices the cache. It reports two numbers
because the model and the backend disagree: a hybrid-attention model does not
need a full-length cache in every layer -- Spark is 21 sliding (window 512)
to 7 full layers, so at 32 k context its real cache is 3.8x smaller than
`layers x context` -- but vLLM's CPU path does not take that discount today
(for a 4 GiB budget it reported 71 527 tokens, i.e. the flat 56 KiB/token).
So `bytes_total` is the flat figure, which is what is safe to pass now, and
`bytes_total_hybrid` is reported beside it for a backend that accounts per
layer type.
Deliberately conservative in three places, because a budget that is too small
is not a graceful degradation -- vLLM preempts by recompute, which on a
single-stream edge deployment is a stall: the window is rounded up to the
block size, a 1.25 headroom factor is applied, and a config that has a
`sliding_window` but no `layer_types` map is charged the flat rate rather than
assumed to apply the window everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
