Package this fork as vllm-embedding: README, docs, roadmap, harness

Turns a branch of accumulated edge/CPU work into something someone else can read, reproduce and continue. README.md states what the fork is for (multimodal models on edge CPUs rather than servers), the headline memory and speed results with their costs, and why the numbers are worth reading -- five retracted conclusions, all measurement errors rather than coding errors. Upstream's README is preserved at docs/README.vllm-omni.md and NOTICE records the Apache-2.0 derivation and the files this fork changes. docs/edge/: methodology.md the measurement discipline, which is the most transferable thing here: what each of the five retractions was, and the rule it left behind. Includes the two traps specific to this stack -- vLLM's AOT compile cache keying on config so two arms sharing an artifact load the wrong graph, and allocated not being resident until mincore(2) says so. results.md every measured figure with its spread and baseline, and an explicit note that the four memory levers overlap and must not be added. cpu-4bit.md the two 4-bit paths, the packed layout as measured rather than assumed, why the path choice is a config field, and the GPTQ output-head trap that invalidated a whole round of W4A8 numbers. harness.md how to run a comparison, and the two arguments that matter more than they look (--context-len, --prompt-len), each of which has produced a wrong conclusion by being left at its default. ROADMAP.md orders by whether the verification method is correct rather than by expected size, lists what is done with results, what is next with expected gains and how each would be verified, and -- at equal length -- what was considered and rejected, including the CPU megakernel with the measurements that rule it out for now. benchmarks/edge_harness/ brings the tooling into the repo so the numbers are reproducible: the grid runner, the single-run benchmarks, the memory ledger, the fidelity metric, the Android agent suite, the quantizers, and the tests that pin the harness's judgement. 82 tests pass from the new location. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-14 13:25:32 +08:00
1 parent 289c13df68
commit b36a3431bd
@@ -0,0 +1,109 @@
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/logos/vllm-omni-logo.png">
+    <img alt="vllm-omni" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/logos/vllm-omni-logo.png" width=55%>
+  </picture>
+</p>
+<h3 align="center">
+Easy, fast, and cheap omni-modality model serving for everyone
+</h3>
+
+<p align="center">
+| <a href="https://vllm-omni.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href="https://deepwiki.com/vllm-project/vllm-omni"><b>DeepWiki</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> | <a href="docs/assets/WeChat.jpg"><b>WeChat</b></a> | <a href="https://arxiv.org/abs/2602.02204"><b>Paper</b></a> | <a href="https://docs.google.com/presentation/d/1aPj0OGl_-ZVoib-Qne5dGDAlrRFB-PdHl6E-EE99g8E/edit?usp=sharing"><b>Slides</b></a> |
+</p>
+
+---
+
+*Latest News* 🔥
+
+- [2026/08] We released [0.28.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.28.0), featuring production-ready [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) serving on GPU and NPU, a unified AR/DiT paged KV cache runtime, and enhanced realtime full-duplex serving for the [MiniCPM-o series](recipes/OpenBMB/MiniCPM-o-4_5.md).
+- [2026/08] [VeRL-Omni](https://github.com/verl-project/verl-omni) `v0.2.0` is released: faster diffusion RL powered by vLLM-Omni (request-level/step-wise batching with FA3), rebuilt Qwen3-Omni multimodal training (DPO & GSPO), plus LTX-2.3, Qwen-Image-Edit support and more. See the [release notes](https://github.com/verl-project/verl-omni/releases/tag/v0.2.0).
+- [2026/08] We released [0.26.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.26.0) - aligned with the vLLM 0.26 release line, featuring [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) joint video/audio generation, an experimental full-duplex realtime runtime for [MiniCPM-o 4.5](recipes/OpenBMB/MiniCPM-o-4_5.md), distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support.
+- [2026/07] We released [0.24.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.24.0) - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
+- [2026/06] Starting with [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0), vLLM-Omni publishes a stable release aligned with every even-numbered upstream vLLM minor version. [0.16.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.16.0), [0.18.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.18.0), [0.20.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.20.0), and [0.22.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.22.0) continued this cadence, expanding omni and world-model support with [NVIDIA Cosmos3](recipes/cosmos3/Cosmos3-Nano.md) and DreamZero, adding models such as MiniCPM-o 4.5, MOSS-TTS, and Lance, and advancing TTS, diffusion, distributed execution, quantization, RL integration through [VeRL-Omni](https://github.com/verl-project/verl-omni), and CUDA/ROCm/MUSA/NPU/XPU coverage.
+- [2026/03] Check out our first public [project deepdive](https://youtu.be/sgwNfsNnR9I) at the vLLM Hong Kong Meetup!
+- [2025/11] vLLM community officially released [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) in order to support omni-modality models serving.
+
+---
+
+## About
+
+[vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:
+
+- **Omni-modality**: Text, image, audio, video, and action data processing
+- **Non-autoregressive Architectures**: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
+- **Heterogeneous outputs**: from traditional text generation to multimodal and action outputs
+
+<p align="center">
+  <picture>
+    <img alt="vllm-omni" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/omni-modality-model-architecture.png" width=55%>
+  </picture>
+</p>
+
+vLLM-Omni is fast with:
+
+- State-of-the-art AR support by leveraging efficient KV cache management from vLLM
+- Pipelined stage execution overlapping for high throughput performance
+- Fully disaggregation based on OmniConnector and dynamic resource allocation across stages
+
+vLLM-Omni is flexible and easy to use with:
+
+- Heterogeneous pipeline abstraction to manage complex model workflows
+- Seamless integration with popular Hugging Face models
+- Tensor, pipeline, data and expert parallelism support for distributed inference
+- Streaming outputs
+- OpenAI-compatible API server
+- [Full-duplex realtime serving](docs/serving/realtime_duplex_api.md) with streaming audio input and output
+
+vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
+
+- **Omni-modality models** (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
+- **TTS models** (e.g. Qwen3-TTS, IndexTTS 2.5, dots.tts, CosyVoice3)
+- **Diffusion models** — image, video, and audio generation (e.g. MiniMax H3, LTX-2.5, SANA-Video, Wan2.2)
+- **Robot-policy and action models** (e.g. π0, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
+
+## Getting Started
+
+Visit our [documentation](https://vllm-omni.readthedocs.io/en/latest/) to learn more.
+
+- [Installation](https://vllm-omni.readthedocs.io/en/latest/getting_started/installation/)
+- [Quickstart](https://vllm-omni.readthedocs.io/en/latest/getting_started/quickstart/)
+- [Nightly CUDA Docker images](https://hub.docker.com/r/vllm/vllm-omni/tags?name=nightly) built from the latest `main`
+- [List of Supported Models](https://vllm-omni.readthedocs.io/en/latest/models/supported_models/)
+- [Deployment Recipes](https://recipes.vllm.ai) for vLLM-Omni model serving
+
+## Contributing
+
+We welcome and value any contributions and collaborations.
+Please check out [Contributing to vLLM-Omni](https://vllm-omni.readthedocs.io/en/latest/contributing/) for how to get involved.
+
+## Citation
+
+If you use vLLM-Omni for your research, please cite our [paper](https://arxiv.org/abs/2602.02204):
+
+```bibtex
+@article{yin2026vllmomni,
+  title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models},
+  author={Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu},
+  journal={arXiv preprint arXiv:2602.02204},
+  year={2026}
+}
+```
+
+## Join the Community
+
+Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in `#sig-omni` slack channel at [slack.vllm.ai](https://slack.vllm.ai) or vLLM user forum at [discuss.vllm.ai](https://discuss.vllm.ai).
+
+## Star History
+
+<a href="https://www.star-history.com/?repos=vllm-project%2Fvllm-omni&type=date&legend=top-left">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=vllm-project/vllm-omni&type=date&theme=dark&legend=top-left&sealed_token=ExgLDZJoQEg27Zfhhut2LqN0GYO6Fw2PWLwPE6JYBUp2BgM3hmsYlwaIVopnUEfbRXidQ4nisumrTdKYydiKhy1SZXipw47qY2_tiUDhCpsPXeXtPuEVKVzBwKs3pw0tiHsJgtSfwXx5yjHXck0Y2SblzFWeJYCkTe1WLGTbUAOIETjXXQJjyCGZvKz5" />
+    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=vllm-project/vllm-omni&type=date&legend=top-left&sealed_token=ExgLDZJoQEg27Zfhhut2LqN0GYO6Fw2PWLwPE6JYBUp2BgM3hmsYlwaIVopnUEfbRXidQ4nisumrTdKYydiKhy1SZXipw47qY2_tiUDhCpsPXeXtPuEVKVzBwKs3pw0tiHsJgtSfwXx5yjHXck0Y2SblzFWeJYCkTe1WLGTbUAOIETjXXQJjyCGZvKz5" />
+    <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=vllm-project/vllm-omni&type=date&legend=top-left&sealed_token=ExgLDZJoQEg27Zfhhut2LqN0GYO6Fw2PWLwPE6JYBUp2BgM3hmsYlwaIVopnUEfbRXidQ4nisumrTdKYydiKhy1SZXipw47qY2_tiUDhCpsPXeXtPuEVKVzBwKs3pw0tiHsJgtSfwXx5yjHXck0Y2SblzFWeJYCkTe1WLGTbUAOIETjXXQJjyCGZvKz5" />
+  </picture>
+</a>
+
+## License
+
+Apache License 2.0, as found in the [LICENSE](./LICENSE) file.
@@ -0,0 +1,100 @@
+# The 4-bit CPU path
+
+Two 4-bit paths exist on an x86 CPU and they are not interchangeable. Which one
+a device should use is a property of the device, and `vllm_omni/edge/weight_path.py`
+picks it from a hardware probe.
+
+## W4A8 — `_C::int4_scaled_mm_cpu`
+
+Needs AMX tiles. Wants GPTQ-packed weights and quantizes activations to int8.
+Fastest measured here (74.4 tok/s) and least faithful (1/12 greedy prompts
+reproduced exactly).
+
+One trap: a GPTQ checkpoint **must** quantize the output head. Spark ties its
+embedding to its head, so the head has to be untied and emitted separately; if
+it is not, that one projection runs in bf16 through `onednn_mm` and costs
+2.97 ms/step by itself — more than the entire 4-bit saving. Every W4A8 number in
+this project's first round was invalid for exactly this reason.
+
+The symmetric grid is **our exporter's choice, not the kernel's limit**:
+`_process_gptq_weights_w4a8` uses the checkpoint's zero points whenever
+`config.zero_points` is set and synthesises the constant 8 only otherwise. An
+asymmetric export is a config change and is the cheapest untried fidelity
+experiment.
+
+## W4A16 — PyTorch tinygemm
+
+No AMX requirement, activations stay bf16, more faithful (2/12). 71.6 tok/s.
+This is `quantization="cpu_int4"`.
+
+The kernel dequantizes once per row of activations, so its cost is linear in
+batch size: fine for decode, ruinous for prefill (453 tok/s if used for
+everything). Large batches therefore take a second path — dequantize to bf16
+once and hand it to the ordinary oneDNN/AMX GEMM.
+
+### The weight layout, measured
+
+That prefill path used to read a row-major *duplicate* of every weight, because
+the tinygemm layout was treated as opaque. It is not. Packing an index pattern
+and reading back where each nibble landed shows, on AVX-512:
+
+```
+[N/64][K][32 bytes]
+```
+
+K is the middle axis, so a K range is contiguous inside each n-block; and within
+every 32-byte run, byte `d` carries output channel `d` in its low nibble and
+`d + 32` in its high one — the same permutation for every `k` and every block.
+Decoding it is a deinterleave, not a reverse-engineered ISA table.
+
+So prefill can dequantize from the kernel's own layout and the duplicate is
+never allocated: **615 MB**, bit-exact against the old path at 2048×2048,
+3072×2048 and `gate_up`'s real 13312×2048.
+
+It is not free. `to_split_halves` performed that deinterleave *once at load*, so
+each prefill call read two contiguous planes; doing it per call costs **1.63×
+prefill**. It is a dial, not a win:
+
+- against `prefill_dequant: false` it is **strictly better** — same memory, 2.97×
+  the prefill, and multi-row batches never reach the tinygemm kernel, avoiding
+  the AMX tile-configuration SIGILL documented in `cpu_int4.py`. That setting is
+  dominated.
+- against keeping both layouts it trades 615 MB for ~1.6× prefill. Default on,
+  because this path exists for devices chosen for their memory ceiling. Set
+  `direct_dequant: false` in the checkpoint's `quantization_config` to revert.
+
+`supports_direct_dequant` gates the reshape on N tiling the 64-channel block,
+which every Spark linear does. An N like 80 packs as 64 + a 16 tail the reshape
+would mis-read, so those layers keep the duplicate rather than being quietly
+wrong.
+
+### Why the path choice is a config field
+
+The two prefill paths build *different graphs* — one passes `weight_halves` into
+the custom op, the other passes `None`. vLLM keys its AOT compile cache on the
+config, so an env-var-only switch shares one cached artifact between them, and a
+warm cache loads the wrong graph: three benchmark passes died with
+`KeyError: 'weight_halves'` inside an AOT-loaded forward. `direct_dequant` is a
+`quantization_config` field because `ModelConfig.compute_hash` hashes those.
+
+## `cpu_gemm_wna16` — the fallback
+
+vLLM selects this when `use_w4a8` is off. Measured at 32.3 tok/s, **2.2× slower
+than tinygemm**, so the selector does not route to it. It is documented because
+`weight_path.py` previously asserted it did not exist.
+
+## Building a checkpoint
+
+```bash
+# W4A16 (cpu_int4)
+python benchmarks/edge_harness/quantize_int4.py \
+    --model <bf16-model> --group-size 128 --out models/<name>
+
+# W4A8 (GPTQ) — note the head must be untied and quantized
+python benchmarks/edge_harness/quantize_gptq_int4.py \
+    --model <bf16-model> --group-size 128 --out models/<name>
+```
+
+`quantization_config` keys the `cpu_int4` path understands: `group_size`,
+`quantize_embedding`, `prefill_dequant`, `direct_dequant`, `dequant_threshold`,
+`use_custom_op`.
@@ -0,0 +1,79 @@
+# The measurement harness
+
+`benchmarks/edge_harness/` — the tooling every number in this fork was produced
+with. It exists because five earlier conclusions were wrong; see
+[methodology.md](methodology.md).
+
+## Running a comparison
+
+Arms are subprocess invocations of a single-run benchmark, so the measurement
+code stays the one that was verified; the harness decides what to run, when, and
+how to read it.
+
+```bash
+python benchmarks/edge_harness/bench_harness.py \
+    --spec arms.json --passes 3 --metric rss_anon_tree_mb \
+    --keys ready_s,init_s,rss_anon_tree_mb,peak_rss_anon_mb \
+    --cores 56-79 --out summary.json
+```
+
+`arms.json` is `{"arms": [{"tag": ..., "command": [...], "env": {...}}]}`.
+Give each arm its own `VLLM_CACHE_ROOT` if the arms differ in anything that
+changes the traced graph.
+
+It rotates arm order every pass, reports spread, labels contended passes, refuses to
+start when the target cores are busy, prints `NOT A RANKING` when the top two
+arms differ by less than their own spread, and stamps provenance into the
+output. `--check expectations.json` turns it into a regression gate.
+
+## The single-run benchmarks
+
+| script | measures |
+|---|---|
+| `bench_init_memory.py` | `ready_s` (process start → serving), peak `RssAnon` sampled through load, resident memory over the **process tree**, memory at a filled context |
+| `bench_cpu_vllm.py` | pp512 / tg128, mirroring `llama-bench` semantics |
+| `profile_cpu_step.py` | per-op decode cost, measured by difference so it matches the wall clock |
+| `memory_ledger.py` | attributes resident memory to tensor storages, KV cache and mappings, with `mincore(2)` residency |
+| `fidelity_kl.py` | top-1 agreement and KL against a bf16 reference, comparable with `llama-perplexity --kl-divergence` |
+
+Two arguments matter more than they look:
+
+- `--context-len` on `bench_init_memory.py`. Without it the KV cache is
+  allocated and never touched, and a memory number taken that way says nothing
+  about cache sizing. That is how "KV sizing is not a lever" happened.
+- `--prompt-len` on `profile_cpu_step.py`. The default of 1 measures an empty
+  cache; Spark's sliding layers have a 512-token window, so attention cost at a
+  1-token prompt is not attention cost at context.
+
+## The agent harness
+
+`benchmarks/edge_harness/android_agent/` drives a phone (real, over `adb`, or a
+scripted simulator that emits real uiautomator XML) and scores tasks by the
+phone's own settings and screen rather than by the transcript.
+
+Scoring reads the *setting* where one exists and the UI dump otherwise, so the
+same predicate works on both backends; an agent that turns Wi-Fi on and then
+presses Home has still turned Wi-Fi on. Tasks carry variants — a renamed
+control, reordered rows, a plausible decoy — because repeats at temperature 0
+against one fixture measure repeatability, not whether the agent can do the task.
+
+```bash
+python benchmarks/edge_harness/android_agent/run_suite.py \
+    --base-url http://127.0.0.1:8077/v1 --repeat 5          # simulator
+python benchmarks/edge_harness/android_agent/run_suite.py --device adb --repeat 5
+```
+
+Tasks needing flip history (a phone cannot report which switches were touched)
+are skipped with a reason on a real device rather than scored from final state.
+
+## Tests
+
+```bash
+pytest benchmarks/edge_harness/test_bench_harness.py \
+       benchmarks/edge_harness/test_fidelity_kl.py \
+       benchmarks/edge_harness/android_agent/test_android_agent.py
+```
+
+They pin judgement, not plumbing: that arm order rotates, that `best` is the
+minimum for memory, that a difference inside the noise is not reported as a
+ranking, that the alarm oracle needs a saved alarm rather than a typed digit.
@@ -0,0 +1,82 @@
+# How to measure on a shared machine without fooling yourself
+
+Five conclusions in this project were retracted. None was a coding error; every
+one was a measurement error that looked like a result. This page is what was
+left after each of them, and it is the part of this repository most likely to be
+useful somewhere else.
+
+## The five
+
+**1. `VmRSS` counts pages you do not own.** A "3.1 GB saving" was the page cache
+holding the mmap'd checkpoint. `VmRSS` moved 1–2 GB between *identical* runs;
+`RssAnon` repeats to 0.2%. Measure allocation, not residency-of-everything.
+
+**2. The build under test was not the build you think.** Every W4A8 timing was
+taken on a checkpoint with `tie_word_embeddings: true` and no `lm_head` tensors,
+so the output projection ran in bf16 and cost 2.97 ms/step by itself — more than
+the entire 4-bit saving being measured. Artifacts now carry provenance: model
+path, quantization, thread set, build SHAs. Two saved profiles in this tree were
+indistinguishable from each other before that.
+
+**3. You measured the wrong process.** The memory benchmark read
+`/proc/self/status`, which is the whole story only when the model is in-process.
+vLLM's CPU default promotes the executor to `mp`, putting the weights in a
+worker: the benchmark reported 796 MB resident for a model whose weights alone
+are 1.4 GB. Memory is summed over the process tree now.
+
+**4. The knob was not connected.** "KV cache sizing saves only 188 MB, the pages
+are never touched" sat on a *closed, do not retry* list. The comparison ran
+through `CpuPlatform.check_and_update_config`, which overwrote
+`kv_cache_memory_bytes` from an environment variable unconditionally — so both
+arms ran the same cache. With the setting honoured the saving is **843 MB**, and
+`mincore(2)` shows the cache fully in core rather than untouched. *A lever tested
+through a path that silently ignores it measures nothing, and "no effect" is the
+most dangerous result to accept without first checking the knob was connected.*
+
+**5. One surviving pass is not a result.** A 1.9× prefill cost was quoted from
+the single pass of a grid whose other passes had crashed — a number the harness
+had already labelled `single pass — not a result`. With every pass completing it
+is 1.63×, and only good to ±0.2× at that.
+
+## What the harness enforces
+
+- **Rotate arm order every pass.** Interleaving alone leaves arm 0 permanently
+  first. `run_grid` rotates; a test pins it, because the previous test asserted
+  the *un-rotated* order and that is how the bug survived.
+- **Know which direction is better.** `best` is the maximum for a rate and the
+  minimum for memory or latency. Taking the maximum for everything published the
+  *worst* memory pass as the headline.
+- **Refuse to rank inside the noise.** When the top two arms differ by less than
+  their own spread, the summary prints `NOT A RANKING` and points at the per-op
+  profiler. Per-op self time reproduces to ~5% here; end-to-end throughput swings
+  ~30%, which is why throughput is explicitly ungateable in the regression check.
+- **Guard the cores, not the host.** Load average is a whole-host number and this
+  machine has 224 cores, so it says nothing about the 24 a run is pinned to. The
+  pre-flight samples per-core utilisation over two windows, takes the minimum
+  (so a blip is not load), and refuses on either >5% aggregate stolen capacity or
+  any core sustained above 50% — one slow core gates every OpenMP barrier, so it
+  cannot be averaged away.
+- **Record failures.** Three passes of one grid died with
+  `KeyError: 'weight_halves'`; the harness recorded the exit status and stderr
+  instead of dropping the runs, which is the only reason the bug was found.
+- **Stamp provenance.** Build SHAs, host, thread affinity, environment. A number
+  whose origin was not recorded cannot be retracted later, only doubted.
+
+## Two traps specific to this stack
+
+**Compile-cache collisions.** vLLM keys its AOT compile cache on the *config*, so
+two arms differing only by an environment variable share one cached artifact. If
+that variable changes the traced graph, later passes load the wrong one. Give
+each arm its own `VLLM_CACHE_ROOT`, and prefer expressing a graph-affecting
+choice as a config field — `ModelConfig.compute_hash` hashes those.
+
+**Allocated is not resident.** An untouched buffer costs address space. Use
+`mincore(2)` before attributing it to memory: `memory_ledger.py` does, and that
+is what proved the KV cache was real rather than reserved.
+
+## The rule that generalises
+
+Attribute, do not subtract. The "unexplained ~1.6 GB" in an earlier report was a
+remainder computed from two uncertain terms, and it hid a 640 MB RoPE table in
+plain sight. Counting unique tensor storages by `data_ptr`, naming the largest,
+and checking residency per page turned "unexplained" into a one-line fix.
@@ -0,0 +1,116 @@
+# Measured results
+
+Spark-X2.5-1.7B unless stated. Xeon Platinum 8480C, 24 pinned physical cores,
+memory bound to the same NUMA node, W4A16 g128, `max_model_len` 2048.
+Each row is a median over at least three interleaved passes against a control
+arm running in the same grid.
+
+## Memory
+
+Every lever below was measured as its own A/B, with the alternative reachable by
+a flag so both arms run identical code.
+
+| lever | before | after | saving | cost |
+|---|---|---|---|---|
+| KV budget 1 GiB → 192 MiB | 3264 MB | 2421 MB | **843 MB** | fewer cached tokens |
+| RoPE table → `max_model_len` | 3891 MB | 3273 MB | **618 MB** | none |
+| One weight layout, not two | 4591 MB | 3976 MB | **615 MB** | ~1.63× prefill |
+| Blocked repack (peak) | 4502 MB | 4013 MB | **489 MB** peak | none |
+
+**These do not add.** The single-layout change removed `to_split_halves`, which
+was doing its own whole-matrix expansion, so part of the blocked-repack saving
+was already gone by the time it was measured. The report each was taken from
+states its baseline; quote them individually.
+
+### Where the memory actually is
+
+From `memory_ledger.py`, which counts unique tensor storages by `data_ptr` (so
+tied weights are counted once) and checks residency with `mincore(2)`:
+
+| | MB | note |
+|---|---|---|
+| KV cache | 1024 | fully resident — 1024.1 of 1023.8 allocated |
+| int4 packed weights | 686 | |
+| RoPE cos/sin tables | 640 | **before the cap** — sized for 1 048 576 positions |
+| embedding / head | 408 | tied; includes a 128 MB row-major copy |
+| imports (`import vllm_omni`) | 501 | |
+
+The 640 MB RoPE entry is the clearest single finding in the project. Spark
+advertises a 1 048 576-token context and the model passed
+`max_position_embeddings` straight to `get_rope`, so vLLM built
+`torch.arange(1048576)` cos/sin tables — two of them, because the sliding and
+full layers use different theta and partial-rotary factors. At `max_model_len`
+2048 they need 1.25 MB between them. It had been sitting inside an
+"unexplained ~1.6 GB" remainder.
+
+### Peak vs settled
+
+`release_after_load()` runs only once loading and warm-up return, so a device
+that cannot hold the *peak* never reaches the release. Sampling `RssAnon` on a
+timer through the load, rather than reading the endpoint:
+
+| | peak | settled | transient |
+|---|---|---|---|
+| whole-matrix repack | 4502 MB | 3905 MB | 550 MB |
+| blocked repack | 4013 MB | 3892 MB | 105 MB |
+
+`unpack_nibbles` expands 4-bit codes to int32 — eight times the packed form —
+because the kernel's packer wants int32. Whole-matrix that is 104 MiB for
+`gate_up` and **1024 MiB for the tied embedding**, against 24 MiB of packed
+weights for a whole layer. Chunking is also *faster* (79 ms vs 208 ms for the
+embedding) because the working set stays in cache.
+
+## Speed
+
+| path | decode | prefill (pp512) | spread |
+|---|---|---|---|
+| W4A8, AMX `int4_scaled_mm_cpu` | 74.4 tok/s | — | 1.1% |
+| W4A16, tinygemm | 71.6 tok/s | 2190 tok/s | 9.7% |
+| W4A16, one layout | 71.6 tok/s | 1344 tok/s | 18.3% |
+| W4A16, `prefill_dequant: false` | 73.0 tok/s | 453 tok/s | 0.3% |
+| `cpu_gemm_wna16` fallback | 32.3 tok/s | — | 1.5% |
+| llama.cpp Q4_K_M | 90.0 tok/s | 1121 tok/s | — |
+
+`cpu_gemm_wna16` is the kernel vLLM selects when `use_w4a8` is off. It exists —
+`weight_path.py` used to call tinygemm "the only option" without AMX, which was
+false — but at 2.2× slower than tinygemm the routing was right anyway. Caveat:
+`_get_isa_hint` returns `"amx"` whenever AMX is present regardless of the W4A8
+flag, so this measured its *best* path on this host, not the `"vec"` path a
+genuinely AMX-less machine takes.
+
+## Bandwidth headroom
+
+A node-local streaming read on the same 24 cores sustains **222 GB/s**.
+
+| | GB/s | of ceiling |
+|---|---|---|
+| W4A16 GEMM only | 132 | 60% |
+| W4A8 GEMM only | 153 | 69% |
+| W4A16 whole step | 73 | 33% |
+| llama.cpp whole step | 86 | 39% |
+
+Neither engine is bandwidth-bound. An earlier draft asserted the GEMM ran at
+"85% of what this socket delivers" and concluded the remainder was framework
+overhead; at 60% of the measured ceiling, kernel and scheduling efficiency are
+still on the table. See [ROADMAP.md](../../ROADMAP.md) item I10.
+
+## Fidelity
+
+Against each build's own bf16 output over 12 greedy prompts:
+
+| build | exact reproductions | bits/weight |
+|---|---|---|
+| llama.cpp Q4_K_M | 4/12 | 4.5 (groups of 32, mixed types) |
+| W4A16 | 2/12 | 4.25 |
+| W4A8 | 1/12 | 4.25 |
+
+**Not a matched-budget comparison** — Q4_K_M stores more. And greedy divergence
+saturates: it discards everything after the first divergent token, so one
+unlucky token scores the same as a build that is wrong everywhere.
+`fidelity_kl.py` replaces it with top-1 agreement and KL over a corpus.
+
+On the Android agent harness, every build navigates a multi-screen task
+correctly and only some notice they have arrived: W4A8 turned Wi-Fi on and then
+tapped the switch off again. **Navigation survives quantization; recognising
+completion does not.** That is one run per cell — enough to show a failure
+reproduces, not to rank success rates.