82 lines
4.7 KiB
Markdown
82 lines
4.7 KiB
Markdown
|
# How to measure on a shared machine without fooling yourself
|
|||
|
|
|
|||
|
|
Five conclusions in this project were retracted. None was a coding error; every
|
|||
|
|
one was a measurement error that looked like a result. This page is what was
|
|||
|
|
left after each of them, and it is the part of this repository most likely to be
|
|||
|
|
useful somewhere else.
|
|||
|
|
|
|||
|
|
## The five
|
|||
|
|
|
|||
|
|
**1. `VmRSS` counts pages you do not own.** A "3.1 GB saving" was the page cache
|
|||
|
|
holding the mmap'd checkpoint. `VmRSS` moved 1–2 GB between *identical* runs;
|
|||
|
|
`RssAnon` repeats to 0.2%. Measure allocation, not residency-of-everything.
|
|||
|
|
|
|||
|
|
**2. The build under test was not the build you think.** Every W4A8 timing was
|
|||
|
|
taken on a checkpoint with `tie_word_embeddings: true` and no `lm_head` tensors,
|
|||
|
|
so the output projection ran in bf16 and cost 2.97 ms/step by itself — more than
|
|||
|
|
the entire 4-bit saving being measured. Artifacts now carry provenance: model
|
|||
|
|
path, quantization, thread set, build SHAs. Two saved profiles in this tree were
|
|||
|
|
indistinguishable from each other before that.
|
|||
|
|
|
|||
|
|
**3. You measured the wrong process.** The memory benchmark read
|
|||
|
|
`/proc/self/status`, which is the whole story only when the model is in-process.
|
|||
|
|
vLLM's CPU default promotes the executor to `mp`, putting the weights in a
|
|||
|
|
worker: the benchmark reported 796 MB resident for a model whose weights alone
|
|||
|
|
are 1.4 GB. Memory is summed over the process tree now.
|
|||
|
|
|
|||
|
|
**4. The knob was not connected.** "KV cache sizing saves only 188 MB, the pages
|
|||
|
|
are never touched" sat on a *closed, do not retry* list. The comparison ran
|
|||
|
|
through `CpuPlatform.check_and_update_config`, which overwrote
|
|||
|
|
`kv_cache_memory_bytes` from an environment variable unconditionally — so both
|
|||
|
|
arms ran the same cache. With the setting honoured the saving is **843 MB**, and
|
|||
|
|
`mincore(2)` shows the cache fully in core rather than untouched. *A lever tested
|
|||
|
|
through a path that silently ignores it measures nothing, and "no effect" is the
|
|||
|
|
most dangerous result to accept without first checking the knob was connected.*
|
|||
|
|
|
|||
|
|
**5. One surviving pass is not a result.** A 1.9× prefill cost was quoted from
|
|||
|
|
the single pass of a grid whose other passes had crashed — a number the harness
|
|||
|
|
had already labelled `single pass — not a result`. With every pass completing it
|
|||
|
|
is 1.63×, and only good to ±0.2× at that.
|
|||
|
|
|
|||
|
|
## What the harness enforces
|
|||
|
|
|
|||
|
|
- **Rotate arm order every pass.** Interleaving alone leaves arm 0 permanently
|
|||
|
|
first. `run_grid` rotates; a test pins it, because the previous test asserted
|
|||
|
|
the *un-rotated* order and that is how the bug survived.
|
|||
|
|
- **Know which direction is better.** `best` is the maximum for a rate and the
|
|||
|
|
minimum for memory or latency. Taking the maximum for everything published the
|
|||
|
|
*worst* memory pass as the headline.
|
|||
|
|
- **Refuse to rank inside the noise.** When the top two arms differ by less than
|
|||
|
|
their own spread, the summary prints `NOT A RANKING` and points at the per-op
|
|||
|
|
profiler. Per-op self time reproduces to ~5% here; end-to-end throughput swings
|
|||
|
|
~30%, which is why throughput is explicitly ungateable in the regression check.
|
|||
|
|
- **Guard the cores, not the host.** Load average is a whole-host number and this
|
|||
|
|
machine has 224 cores, so it says nothing about the 24 a run is pinned to. The
|
|||
|
|
pre-flight samples per-core utilisation over two windows, takes the minimum
|
|||
|
|
(so a blip is not load), and refuses on either >5% aggregate stolen capacity or
|
|||
|
|
any core sustained above 50% — one slow core gates every OpenMP barrier, so it
|
|||
|
|
cannot be averaged away.
|
|||
|
|
- **Record failures.** Three passes of one grid died with
|
|||
|
|
`KeyError: 'weight_halves'`; the harness recorded the exit status and stderr
|
|||
|
|
instead of dropping the runs, which is the only reason the bug was found.
|
|||
|
|
- **Stamp provenance.** Build SHAs, host, thread affinity, environment. A number
|
|||
|
|
whose origin was not recorded cannot be retracted later, only doubted.
|
|||
|
|
|
|||
|
|
## Two traps specific to this stack
|
|||
|
|
|
|||
|
|
**Compile-cache collisions.** vLLM keys its AOT compile cache on the *config*, so
|
|||
|
|
two arms differing only by an environment variable share one cached artifact. If
|
|||
|
|
that variable changes the traced graph, later passes load the wrong one. Give
|
|||
|
|
each arm its own `VLLM_CACHE_ROOT`, and prefer expressing a graph-affecting
|
|||
|
|
choice as a config field — `ModelConfig.compute_hash` hashes those.
|
|||
|
|
|
|||
|
|
**Allocated is not resident.** An untouched buffer costs address space. Use
|
|||
|
|
`mincore(2)` before attributing it to memory: `memory_ledger.py` does, and that
|
|||
|
|
is what proved the KV cache was real rather than reserved.
|
|||
|
|
|
|||
|
|
## The rule that generalises
|
|||
|
|
|
|||
|
|
Attribute, do not subtract. The "unexplained ~1.6 GB" in an earlier report was a
|
|||
|
|
remainder computed from two uncertain terms, and it hid a 640 MB RoPE table in
|
|||
|
|
plain sight. Counting unique tensor storages by `data_ptr`, naming the largest,
|
|||
|
|
and checking residency per page turned "unexplained" into a one-line fix.
|