Include the baselines, comparisons and study documents, not just the fork

The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ

docs/analysis/optimization_plan.md

@@ -0,0 +1,585 @@
+# Optimization plan: inference and harness
+
+Written 2026-09-12 against [spark_edge_support.md](spark_edge_support.md), then
+revised after an independent review by `gpt-6-astra` through the Codex hook
+(read-only against these checkouts; the sandbox could not start because user
+namespaces are disabled on this host, so it ran unsandboxed under explicit
+read-only instructions and the checkouts were verified clean afterwards — the
+same procedure recorded in [README.md](README.md) for the 2026-09-08 study).
+
+The review rejected the first draft's ordering and several of its expected
+gains. Most of that was correct and is applied below; §"Review record" lists
+what I accepted, what I corrected with new measurements, and why.
+
+Every item starts from a measurement in this tree, states the expected gain,
+and says how it would be verified — because on this host a claim without a
+verification method is how three earlier conclusions turned out to be wrong.
+**A fourth is now on the list**: the first draft of this plan mis-described the
+740 MB it proposed to remove.
+
+## Where the time actually goes
+
+W4A16 g128 decode, 24 threads, per-op profile (reproducible to ~5%, where
+throughput swings ~30%). The first draft built this table from two different
+builds — 5.91 ms is the **W4A8** GEMM, 6.85 ms the W4A16 one
+([support:193](spark_edge_support.md)) — so the step it described (11.8 ms) was
+neither configuration. The W4A16 step is **12.51 ms**:
+
+| | ms | share of 12.51 |
+|---|---|---|
+| int4 GEMM (tinygemm) | 6.85 | 55% |
+| inductor elementwise (norms, rope, GEGLU, residuals) | 2.20 | 18% |
+| vLLM attention ops (4 per layer × 28) | 1.31 | 10% |
+| custom-op dispatch boundary (W4A16 only) | 0.55 | 4% |
+| unattributed remainder (sampling, view/slice churn) | 1.60 | 13% |
+
+Two corrections to how this table was read. The GEMM runs near this socket's
+achievable bandwidth for *its own* shape, which says the GEMM is memory-bound;
+it does **not** say the other 45% is framework overhead — attention, softmax,
+norms and sampling contain real arithmetic. And the per-step savings of W4A8
+over W4A16 do not add: 0.94 ms (GEMM) + 0.48 ms (dispatch) = 1.42 ms predicted
+against 12.51 − 11.55 = **0.96 ms** measured. Something offsets them, and no
+item below may assume its saving reaches the step time unchallenged.
+
+## Where the memory actually goes
+
+Anonymous RSS by phase ([support:324](spark_edge_support.md)):
+
+| phase | W4A8 (GPTQ) | W4A16 (tinygemm) |
+|---|---|---|
+| before import | 12 MB | 12 MB |
+| after `import vllm_omni` | 513 MB | 515 MB |
+| after engine construction | **7338 MB** | 5023 MB |
+| after releasing free arenas | 3510 MB | 3411 MB |
+
+The first draft treated the post-release figure as the number to reduce. That
+is the wrong target for an edge device: **7338 MB is the peak that has to fit**,
+and `release_after_load()` runs only after `load_model` and warm-up have both
+returned ([patch.py:730](../vllm-omni/vllm_omni/patch.py)). A device that cannot
+survive the load never reaches the release. This reordered the plan.
+
+The draft's weight breakdown ("0.86 GB quantized + 0.54 GB bf16 embedding") is
+also wrong for the checkpoint actually measured: `spark2_5-int4-g128/config.json`
+sets `quantize_embedding: true` with `tie_word_embeddings: true`, so there is no
+bf16 embedding, and the persistent prefill duplicate was left out of the
+breakdown entirely. There is therefore no trustworthy ledger to subtract from —
+which is why the "unexplained 1.6 GB" item is now a prerequisite, not a
+long-shot.
+
+---
+
+## Inference
+
+### I0. Peak load-time memory — *now first*
+
+7338 MB peak vs 3510 MB steady is the gap that decides whether a device can run
+this at all. The peak comes from holding the checkpoint, the dequantized
+intermediate and the repacked layouts alive at once, per layer, with freed
+arena pages not yet returned.
+
+**Do**: (a) release arenas *during* the load, per layer or per N-layer block,
+rather than only after `load_model` returns; (b) repack blockwise so no
+whole-tensor bf16 intermediate exists; (c) as the durable fix, write the packed
+layout to a deployment artifact at build time so the device loads what it runs.
+
+**Expect**: peak toward the steady 3.4–3.5 GB. Unquantified deliberately — the
+mechanism is understood, the size is not.
+
+**Verify**: sample `RssAnon` on a timer through load (0.2% reproducible), report
+the maximum, not the endpoint. The harness measures endpoints today and would
+not have caught this.
+
+### I1. One packed weight layout, not two — *replaces the draft's I2*
+
+The W4A16 build keeps a second copy costing **740 MB** (3542 → 2802 MB) and
+buying **7.3× prefill** (3252 → 448 tok/s). The draft described this as "prefill
+unpacks to bf16" and proposed a tiled dequant kernel to avoid the bf16 copy.
+That was wrong about the artefact. `_finalize_int4` keeps `weight_halves` — a
+**second packed 4-bit copy** — plus cloned `weight_group_scale` and
+`weight_group_zero`; the bf16 matrix is already a per-call temporary and
+`weight_packed` is deleted
+([cpu_int4.py:344,375](../vllm-omni/vllm_omni/model_executor/layers/quantization/cpu_int4.py)).
+
+So the cheap fix is not a new GEMM. It is to **dequantize directly from the
+tinygemm layout** and delete `weight_halves` and the duplicated scale/zero,
+keeping the existing single `F.linear`. The only reason that was not the
+original design is that the tinygemm layout was believed opaque. It is not —
+measured on this host (`torch._convert_weight_to_int4pack_for_cpu`, AVX-512):
+
+- The packed buffer is `[N/64][K][32 bytes]`: **K is the middle axis**, so a
+  K-range is contiguous *within each n-block of 64*.
+- Reassembling a K-tile from those per-n-block segments is **byte-identical** to
+  packing that K-slice directly (verified exactly at N=2048, K=2048 for tile
+  widths 128/256/512), and split-K through tinygemm reconstructs the single-call
+  result at **51.5 dB**.
+- The lane order inside a 32-byte run is a **fixed 64-entry interleave** — low
+  nibble = output channel *d*, high nibble = *d+32* — identical across every k
+  and every n-block. So decoding is a deinterleave, not a gather or a
+  reverse-engineered ISA table.
+
+**Do**: point the existing dequantizer at the tinygemm layout via that
+deinterleave; drop `weight_halves`, `weight_group_scale`, `weight_group_zero`.
+
+**Expect**: most of the 740 MB back **at unchanged prefill**, because the GEMM
+shape and reduction order do not change — only where the nibbles are read from.
+The largest single temporary stays one dequantized `gate_up` (N=13312, K=2048,
+bf16 = **52 MiB**), which already exists today.
+
+**Verify**: `RssAnon` (0.2%) plus pp512, three interleaved passes, and an SNR
+assertion against the current dequant path — the check that caught the original
+layout bug.
+
+**Risk**: tails. N not divisible by 64 or K odd need the scalar path; the
+deinterleave rule above is verified only for full blocks.
+
+**Not doing first**: the draft's K-tiled dequant-and-accumulate. Even granting
+the layout, calling oneDNN once per K-tile means the M×N accumulator is re-read
+and re-written per tile — for `gate_up` at M=512 that is ~26 MiB of fp32
+accumulator × 15 extra tiles ≈ 780 MiB per layer per prefill, which is not the
+"few MB of scratch" the draft claimed. Avoiding that traffic means keeping the
+accumulator panel in registers across the K loop, i.e. writing a blocked GEMM
+microkernel rather than calling a library per tile. If temporaries still need
+cutting after the item above, **N-blocking is the better experiment** — disjoint
+output columns need no cross-tile accumulation at all.
+
+### I2. Measure the native non-AMX 4-bit path before building anything
+
+`weight_path.py:14` calls tinygemm "the only option" without AMX. That is false:
+when `layer.use_w4a8` is off, vLLM 0.28 falls through to `ops.cpu_gemm_wna16`
+([cpu.py:202](../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py)),
+a vector-ISA W4A16 kernel that has never been measured here. Both the ARM story
+and the "fidelity" weight path rest on an untested assertion.
+
+**Do**: benchmark `cpu_gemm_wna16` against tinygemm at Spark's shapes, decode
+and prefill. **Verify**: `bench_int4_kernels.py`. **Then** fix `weight_path.py`.
+
+This is a day's work that could invalidate a kernel project, so it comes first
+among the kernel items.
+
+### I3. Asymmetric W4A8 — a fidelity experiment already supported by the kernel
+
+The report treats W4A8's zero point pinned at 8 as a kernel limitation. It is
+not: `_process_gptq_weights_w4a8` uses the checkpoint's zero points whenever
+`config.zero_points` is set, and synthesizes the constant only otherwise
+([cpu.py:98](../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py)).
+Our exporter chooses `"sym": true`
+([quantize_gptq_int4.py:102](experiments/spark_edge/quantize_gptq_int4.py)).
+W4A8 is the fastest path and the worst on task completion (1/12 greedy; the
+agent navigates correctly then fails to stop). An asymmetric export is a
+config change, not a kernel project.
+
+**Do**: re-export with `sym: false` and zero points, re-run fidelity and the
+agent suite. **Expect**: unknown, but it is the cheapest fidelity experiment
+available and it tests the *stated* cause of W4A8's error.
+
+### I4. A better quantizer — only after I3, and on an honest bit budget
+
+Ours reproduces 2/12 greedy prompts; llama.cpp Q4_K_M manages 4/12. The draft
+called this "the same bit budget". It is not. Ours is **4.25 bits/weight**
+(4 + two bf16 per 128-weight group; `weight_path.py` says 4.16, which is wrong);
+plain Q4_K is **4.5 bits** over groups of 32, and Q4_K_M additionally promotes
+some tensors to higher-bit types. Group size, per-tensor type selection and
+rounding are all confounded, so "6-bit sub-block scales cause the gap" is an
+attribution the experiment never isolated.
+
+**Do**: first close the budget gap cheaply (g64 and/or fp16 scale+zero, which is
+a config change) and re-measure; only then implement error-compensated rounding
+(GPTQ-style, calibration data exists) or AWQ-style scaling.
+
+**Verify**: greedy divergence *and* the scored agent suite — but only after H1
+below, because today's suite cannot rank builds.
+
+**Risk**: may produce nothing. Timebox it; a null result is worth recording.
+
+### I5. Attention op-chain fusion — worthwhile, but not "highest confidence"
+
+Four ops per layer, 112 dispatches, **1.3085 ms/step** measured
+([profile_w4a16_g128_p2.json](experiments/spark_edge/profile_w4a16_g128_p2.json)):
+`unified_attention_with_output` 0.4379, `unified_kv_cache_update` 0.3948,
+`cpu_attention_with_kv_cache` 0.3471, `cpu_attn_reshape_and_cache` 0.1287.
+
+These are **two wrapper/kernel pairs**, not four independent kernels. Wrapper
+self time is 0.833 ms — 64% of the total — so overhead does dominate at this
+shape: at batch 1 the cache-update OpenMP loop runs over tokens × KV heads, i.e.
+**two iterations**, while both paths independently rebuild cache views and the
+wrapper allocates a dummy tensor for compile ordering.
+
+But the draft's supporting arithmetic was about a workload it never profiled.
+"28 MB moved" assumes every layer attends 512 tokens; 7 of 28 layers are full
+attention, and `profile_cpu_step.py` profiles a short prompt generating 64
+tokens — nowhere near a filled window.
+
+**Expect**: revised down. Eliminating *all* wrapper time saves 0.833 ms, and
+some wrapper is required, so the draft's 0.6–0.9 ms demanded savings from the
+C++ bodies too. Target **0.3–0.5 ms** (2–4% of step) from consolidating context
+and cache-view handling through the existing `forward_includes_kv_cache_update`
+interface, plus a serial path for the two-iteration cache update.
+
+**Verify**: per-op profile **and** unprofiled step latency — one outer call that
+still launches two threaded kernels removes neither thread launch, and fewer
+profiler events alone make this look better than it is.
+
+### I6. Import cost — an upper bound, not a saving
+
+`import vllm_omni` costs 19.6 s against `import vllm`'s 5.2 s; auto
+image-processing (4.6 s) and the `openai` client (1.3 s) are identifiable inside
+it. The draft promised 4–6 s per process. That 5.9 s is an **upper bound on
+removable import work**, not a demonstrated reduction in time-to-ready: deferred
+imports often just move the cost later. The init timer starts *after* imports
+([bench_init_memory.py:44](experiments/spark_edge/bench_init_memory.py)), so the
+current harness cannot measure this at all. And `patch.py` uses `ModelConfig` at
+import time to install a descriptor patch
+([patch.py:54](../vllm-omni/vllm_omni/patch.py)) — deferring it means changing
+how that patch installs, not moving a statement.
+
+**Do**: first make the harness measure **process start → ready**; then defer.
+**Verify**: that end-to-end number, with `-X importtime` as the diagnostic.
+
+### I7. A real memory ledger
+
+Prerequisite for any claim about a remainder. The draft's tools do not do the
+job: `smaps_rollup` is an aggregate (use `smaps` per mapping), and `tracemalloc`
+does not attribute native tensor storage or allocator arenas.
+
+**Do**: count unique tensor storages (both embedding layouts included), then
+`smaps` by mapping for the rest. **Expect**: a decision about whether the
+remainder is reducible — the draft's "unexplained 1.6 GB" was a subtraction from
+a breakdown now known to be wrong.
+
+### I8. Resolve the `prefill_dequant: false` correctness hazard
+
+`cpu_int4.py:55` records that running tinygemm on a multi-row batch and then
+`cpu_attention_with_kv_cache` in the same forward dies with SIGILL — an AMX tile
+configuration bug — and sets `dequant_threshold: 2` to avoid it. But
+`prefill_dequant: false`, the 2802 MB build the report calls the smallest
+resident configuration, routes multi-row batches to exactly that path. Either
+the 448 tok/s measurement and the crash report disagree about build or shape, or
+that configuration is not deployable. **Do**: reconcile before quoting 2802 MB
+as a build anyone can use.
+
+### I10. Megakernel / persistent thread pool -- the biggest measured headroom
+
+Raised after the review. On GPU a megakernel (Hazy Research's low-latency
+Llama, Mirage's MPK) fuses a whole forward pass into one launch, removing
+per-kernel launch cost and the pipeline bubble where the next layer's weights
+cannot start loading. There are no kernel launches on CPU, so the analogues
+are **OpenMP fork/join per op**, **ATen dispatch**, and the same cross-op
+prefetch bubble.
+
+This is not a hypothetical for us: **llama.cpp already is one.**
+`ggml_threadpool` starts its threads once and they walk the entire graph
+together, synchronising at `ggml_barrier` through an atomic spin
+(`ggml-cpu.c:490,576`). vLLM enters a separate `omp parallel` region inside
+each kernel. So the comparison this whole study is built on is, in part, a
+comparison of execution models.
+
+Measured here, so the prize is sized before anything is built:
+
+* **Parallel-region cost: ~1.2 us** at 24 threads (a just-above-grain-size op,
+  24 threads vs 1). Spin-waiting does not help -- `OMP_WAIT_POLICY=ACTIVE` plus
+  `GOMP_SPINCOUNT=infinite` moved it 1.23 -> 1.18 us, inside noise -- so this
+  is barrier and scheduling cost, not thread wake-up, and the one-env-var
+  approximation of a persistent pool buys nothing.
+* **Regions per decode step: a few hundred.** 113 quantized GEMMs and 112
+  attention ops certainly parallelise; most elementwise tensors at batch 1 are
+  below PyTorch's grain size and run serially, so the 764 candidate calls in
+  the profile overstate it. At ~250-350 regions this is **0.3-0.4 ms, 2-3%** --
+  real, but the same money as I5, which is far cheaper.
+* **The bubble is the prize, not the barriers.** The GEMM sustains 60% of the
+  measured read ceiling and the whole step 33%. A persistent kernel that keeps
+  the weight stream running across op boundaries is the mechanism that could
+  close part of a **2.76 ms** gap.
+
+One of our own decisions works against this: `cpu_int4_linear` is deliberately
+opaque to dynamo so the batch-size branch never becomes a guard on vLLM's
+traced graph. That also stops inductor fusing anything across it. The guard
+stability was worth it, but it is a cost worth naming.
+
+**Do**: *not* build a megakernel next. The honest read is that the fork/join
+half is small and already covered by I5, and the streaming half needs the
+execution model replaced -- a persistent pool plus cross-op pipelining inside
+vLLM's CPU backend, fighting its attention backends, KV cache management and
+torch.compile integration. That is weeks, and it reimplements ggml.
+
+What *is* worth doing, in order, because each one tests the same thesis for a
+fraction of the cost:
+
+1. **Find out why the GEMM is at 60%.** It is not obviously bandwidth. The
+   tinygemm kernel dequantizes once per activation row, so part of that time is
+   arithmetic, and each of the 113 calls streams only a few MB -- too little for
+   the prefetcher to reach steady state. Attribute it before optimising it.
+2. **Measure `cpu_gemm_wna16`** (I2) and W4A8's 69% against the same ceiling.
+   If one kernel is materially closer, the answer is a better kernel, not a
+   different execution model.
+3. **I5**, which is the cheap slice of the fork/join half.
+
+**Verify**: per-op profile plus the streaming-bandwidth figure above, which is
+now the denominator any kernel claim should be quoted against.
+
+### I9. Explicitly not planned
+
+- **More kernel fusion.** Four attempts, four losses: vLLM's fused CPU RMSNorm
+  is 3.6–4.5× faster in isolation and slower in situ (inductor already fuses the
+  native form with the residual add); fused `gelu_and_mul` 2.7× slower at
+  Spark's shape; `cpp.min_chunk_size` −6% decode/−19% prefill; an inline gate
+  reduction −44%. I5 is op-*count* reduction, a different mechanism.
+- **MiniCPM-o CPU overlay.** Dropped. The 8 B thinker at bf16 is ~19 GB, so it
+  buys no edge deployability; and the draft's "three stages should beat the
+  two-stage 2.1×" is not how overlap works — the bound is ΣT/max(T), so stages
+  of 90/5/5 s give 1.11×. (That the measured two-stage ratio is 64.57/30.54 =
+  2.114× when the bound for two tasks is lower means something else changed too,
+  which the init work should explain before anyone quotes it.)
+- **Thread count and warm-up batch size as memory levers** — 64 MB and 2 MB.
+- **KV-cache sizing is *not* on this list any more.** The 188 MB that retired it
+  was measured generating 8 tokens. At 32 k context `kv_budget.py`'s own formula
+  gives 1792 MiB flat vs 469 MiB hybrid — a 1323 MiB difference. It was measured
+  at a context where no cache is touched, which is not evidence that it never
+  matters. (`CpuPlatform.support_hybrid_kv_cache()` returns `True` in vLLM 0.28,
+  so the report's flat accounting needs a configuration explanation too.)
+- **Speculative decoding** for tg128: it would inflate a fixed-prompt
+  `ignore_eos` benchmark without representing anything. Measure it on the agent
+  suite or not at all.
+
+---
+
+## Harness
+
+The review's sharpest finding is that the harness cannot currently support the
+comparisons the inference work needs. Correctness first, statistics second.
+
+### H1. Fix the scored suite before running it more
+
+- **The alarm oracle is wrong.** `add_alarm` passes if any typed string contains
+  `"7"` ([tasks.py:101](experiments/spark_edge/android_agent/tasks.py)) — not
+  7:30, not AM, not that an alarm was saved. Typing `7` anywhere scores a pass.
+- **The toggle task scores final state only**, so turning a setting off and back
+  on counts as success.
+- **The scored suite is simulator-only**: task builders and predicates are typed
+  to `SimulatedDevice`. The real-device *episode* path exists; real-device
+  scoring and reset do not.
+- **Repeats at temperature 0 over fixed fixtures measure repeatability, not
+  generalization** ([agent.py:48](experiments/spark_edge/android_agent/agent.py)).
+
+**Do**: correct the oracles; add paired task/layout/state variations so N runs
+are N trials; implement real-device scoring and reset.
+
+This supersedes the draft's "N≥5 runs per task" — more repetitions of a wrong
+oracle produce a confident wrong answer, which is the failure mode this project
+already has three instances of.
+
+### H2. Fix measurement semantics
+
+- `run_grid`'s docstring says it rotates arms; it iterates `for pass: for arm in
+  arms`, i.e. the **same order every pass**
+  ([bench_harness.py:160](experiments/spark_edge/bench_harness.py)). It
+  interleaves but does not rotate, so position bias survives.
+- `summarize` reports `best = max(values)` for every metric
+  ([bench_harness.py:123](experiments/spark_edge/bench_harness.py)) — wrong for
+  memory and init, where lower is better. Published "best" memory figures are
+  the worst pass.
+- `profile_cpu_step.py` wraps all of `generate` (prefill + sampling) and divides
+  by `tg`, while the wall-clock path subtracts independent best-of-three
+  measurements. These are different quantities presented as one.
+- Profile JSONs record no model path, context length or build; saved W4A8 and
+  W4A16 tags both read `bfloat16/None/eager=False`, and some saved W4A8 profiles
+  still contain the retracted bf16-head build with nothing marking them invalid.
+
+**Do**: rotate arm order per pass; take `best` by metric direction; separate
+prefill from decode in the profiler; stamp provenance (model path, config hash,
+commit, thread set) into every artifact and mark the retracted ones.
+
+### H3. Measure what the inference items need
+
+- **Process start → ready**, not engine-construction time (required by I6).
+- **Peak `RssAnon` through load**, sampled, not the endpoint (required by I0).
+- **Memory at realistic occupied context**, not 8 generated tokens (required to
+  re-open KV sizing).
+
+### H4. Contention guard
+
+Two sessions on the same cores wrecked several measurements and one grid's
+control drifted 30% mid-run. A cooperative lock file only excludes cooperating
+benchmarks, and load average > 8 identifies neither *which* cores are busy nor
+whether the load is ours.
+
+**Do**: lock file plus a pre-flight check on per-core utilization for the
+*target* core set, recorded in the artifact.
+
+### H5. Regression gates on measured bands
+
+The draft proposed an 8% throughput band. That 8% is a configured constant in
+the harness, not an estimated interval, and it sits against acknowledged ~30%
+run-to-run variation — it would fire constantly or never.
+
+**Do**: a checked-in expectations file whose bands come from the observed
+distribution per metric (`RssAnon` ~0.2%, per-op self time ~5%, end-to-end
+throughput wide enough to be meaningless, which is itself the argument for
+gating on per-op time and memory instead).
+
+### H6. A denser fidelity metric
+
+Greedy divergence over 12 prompts saturates and cannot separate "slightly worse"
+from "much worse". Top-1 agreement and KL against the bf16 reference over a few
+thousand teacher-forced tokens, symmetric with `llama-perplexity
+--kl-divergence` so llama.cpp stays comparable. Needs `llama-perplexity` built.
+
+Bring this forward: I3 and I4 are fidelity experiments and cannot be judged on a
+saturated metric.
+
+### H7. Make profiling the default for small differences
+
+Per-op self time resolves ~5%; throughput on this host ~30%. Have
+`bench_harness.py` say so when an arm difference falls inside the noise band,
+and point at `profile_cpu_step.py`.
+
+### H8. Purge retracted numbers from executable source
+
+Retractions are still being shipped as documentation:
+`kv_budget.py:9` advertises the withdrawn **8329 → 5254 MB**;
+`weight_path.py` reports W4A16 as **4.16 bits** (it is 4.25), sets
+`fused_cpu_norms=True` in three profiles against the recorded negative result
+and the `"0"` default, and calls tinygemm "the only option" (see I2).
+
+---
+
+## Order
+
+1. **H1, H2, H8** — the suite and harness cannot currently rank builds, and two
+   modules still assert retracted numbers. Everything downstream depends on this.
+2. **H3** — add the three measurements the inference items are defined in terms
+   of (start→ready, peak RSS, memory at context). Then **H4**, **H5**, **H6**.
+3. **I0** (peak load memory) — the actual edge blocker.
+4. **I7** (memory ledger), **I6** (imports) — now measurable.
+5. **I2** (measure `cpu_gemm_wna16`) — may invalidate a kernel project; cheap.
+6. **I1** (single packed layout) — largest confirmed memory win at unchanged
+   prefill, and the layout facts it needs are now measured.
+7. **I8** (SIGILL reconciliation), then **I3** (asymmetric W4A8) and **I4**
+   (quantizer) on a fixed fidelity metric.
+8. **I5** (attention wrappers) — bounded overhead experiment, revised target.
+
+The draft ordered by "whose verification method already exists". The revision
+orders by **whether the verification method is correct**, which is not the same
+question and is the one this project keeps getting wrong.
+
+---
+
+## Progress
+
+Started 2026-09-13, in the plan's own order. Items are struck through here only
+when measured, not when coded.
+
+**Step 1 -- H1, H2, H8: done.** Commits `6424c69` (oracles and measurement
+semantics), `27b94007` (retracted numbers out of `vllm-omni` source), `28aa63c`,
+`b713d6e`, `4285f92`. Agent tests 27 -> 38, harness tests 9 -> 34, plus 7 for
+the new fidelity metric.
+
+**Step 2 -- H3 to H7: done.** And it found a fifth measurement problem, of the
+same kind as the first four:
+
+> **The memory benchmark was reading the wrong process.** It took
+> `/proc/self/status`, which is the whole story only when the model is
+> in-process. That happens when `VLLM_ENABLE_V1_MULTIPROCESSING=0` keeps the
+> executor at `uni`; `vllm/platforms/cpu.py:273` promotes it to `mp` otherwise.
+> Every earlier run in this study set that variable, so **every published
+> memory figure is an in-process figure**. At vLLM's defaults the weights load
+> into a worker the benchmark never looked at: a smoke run reported 796 MB
+> resident for a model whose weights alone are 1.4 GB. Memory is now summed
+> over the process tree.
+
+First numbers from the rebuilt benchmark (W4A16 g128, 24 threads, 2048 ctx,
+`VLLM_CPU_KVCACHE_SPACE=1`, 2 interleaved passes):
+
+| | in-process (`uni`) | default (`mp`) |
+|---|---|---|
+| peak RssAnon, tree | 5722-5837 MB | 6068-6078 MB |
+| after arena release | ~4480 MB | ~5544 MB |
+| processes | 1 | 4 |
+| `ready_s` (process start to serving) | 78.5-111 s | 88-153 s |
+| of which imports | ~29-30 s | ~30 s |
+
+Three things follow. **The peak is 1.2-1.3 GB above the settled figure**, which
+is I0's premise and is now measured rather than asserted. **vLLM's default
+process layout costs ~240 MB of peak and ~1.1 GB of settled memory** over the
+in-process one, which no published number reflected. And **`ready_s` is far
+above `init_s`** -- imports are ~30 s, not the 19.6 s the plan quotes, so I6 is
+worth more than it claimed and was previously unmeasurable.
+
+These are not comparable to the report's 3411 MB: that row was taken
+in-process, with a sized KV budget, generating 8 tokens. Which is the argument
+for the provenance stamping added in the same step.
+
+**Step 3 -- I0 (peak load memory): done.** The driver was in our own repack
+path, not in vLLM: `unpack_nibbles` expands 4-bit codes to int32, and doing
+that a whole matrix at a time makes the 131072-row tied embedding a **1024 MiB
+transient**. Repacking in n-blocks (the layout makes them independent) is
+byte-identical and cuts the load-time transient from 550 to 105 MB, **489 MB
+off the peak**, with settled memory and init time unchanged -- chunked
+repacking is in fact faster than whole-matrix (79 ms vs 208 ms for the
+embedding). Chunk size tuned from a sweep rather than guessed.
+
+**Step 5 -- I2 (measure `cpu_gemm_wna16`): done, and it did not invalidate
+anything.** The fallback is real, so `weight_path.py`'s "only option" was
+false, but at 32.3 tok/s it is **2.2x slower than tinygemm** (71.6) and 2.3x
+slower than W4A8 (74.4), all on the same cores with only the env var
+differing. The routing was right; it now cites a measurement. Caveat recorded:
+`_get_isa_hint` returns `"amx"` whenever AMX is present, so this measured the
+kernel's best path on this host, not the `"vec"` path an AMX-less machine takes.
+
+**Step 6 -- I1 (single packed layout): done.** The layout
+facts the item needed are now established on this host rather than assumed
+(see I1 above), and the reconstruction is **bit-exact** against the existing
+dequantizer at 2048x2048, 3072x2048 and 256x512 -- including `gate_up`'s real
+13312x2048. `weight_halves` and the duplicated scale/zero are no longer
+allocated where N tiles the 64-channel block, which every Spark linear does
+(2048, 3072, 13312, 131072); shapes like N=80 keep the copy rather than
+mis-read the layout. `VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT=0` restores the old
+path, which is how the two are being A/B'd with identical code.
+
+*Process note, since this study keeps being bitten by measurement: I edited
+`cpu_int4.py` while an unrelated grid was running, so that grid's later passes
+used different code from its earlier ones. Those rows were discarded and the
+comparison re-run under the escape hatch instead.*
+
+---
+
+## Review record
+
+**Accepted and applied** (each verified in source before applying): the step
+table conflated W4A8 and W4A16 (W4A16 is 12.51 ms, and the remainder is 1.60 ms
+not ~1.9); bandwidth saturation of the GEMM does not make the rest overhead;
+per-step savings were added when the measured total contradicts the sum; the
+740 MB is a second *packed* copy plus duplicated scale/zero, not a bf16 matrix,
+so the cheap fix is direct dequant rather than a tiled kernel; per-tile oneDNN
+calls incur accumulator traffic the draft ignored; 4.25 ≠ 4.16 bits and Q4_K_M
+is 4.5 bits with mixed types, so the fidelity comparison was never
+budget-matched; the import figure is an upper bound and the init timer starts
+after imports; the MiniCPM-o stage-count extrapolation is invalid and the item
+does not serve edge deployability; peak load memory, not steady memory, is the
+edge blocker; KV sizing was retired on an 8-token measurement; W4A8 supports
+asymmetric zero points, so the report mistook our export choice for a kernel
+limitation; `cpu_gemm_wna16` is a real non-AMX path and `weight_path.py`'s "only
+option" is false; `smaps_rollup`/`tracemalloc` cannot deliver the attribution
+I7 needs; `prefill_dequant: false` bypasses the documented SIGILL guard; the
+harness does not rotate arm order, takes `max` as "best" for every metric, mixes
+prefill into per-step timing and stamps no provenance; the alarm oracle accepts
+a bare `"7"`, the toggle task scores final state only, and the scored suite is
+simulator-only.
+
+**Corrected with new measurements.** The review held that the tinygemm layout
+requires implementing "the ISA-specific lane mapping and tail handling" and that
+a K interval "is not a plain column slice", concluding a tiled decoder is
+expensive to get right. The first half is understated and the second is true but
+not the obstacle it implies. Measured here: a K-tile is contiguous *within each
+n-block of 64* and reassembling those segments is byte-identical to packing the
+K-slice (N=2048, K=2048, tile widths 128/256/512), split-K reconstructs the
+single call at 51.5 dB, and the lane order inside each 32-byte run is a fixed
+interleave (low nibble → channel *d*, high → *d+32*) that is constant across
+every k and n-block. So decoding needs a deinterleave, not a reverse-engineered
+table — which is what makes I1's direct-dequant fix small enough to be the first
+thing tried. The review's conclusion (do direct dequant, not the tiled kernel)
+is unchanged and adopted; its reasoning about why is superseded.
+
+The review's accumulator-traffic objection assumes K-outer with a full M×N
+accumulator, which a register-blocked GEMM would not pay — but since the draft
+proposed calling a library per tile, the objection lands against the draft as
+written, and I restructured rather than defended it.