From 99ebfbbd04465151d49d70c1a5ffe1017b5b0f96 Mon Sep 17 00:00:00 2001 From: tzhouam Date: Mon, 14 Sep 2026 14:07:31 +0800 Subject: [PATCH] docs: we do patch vLLM, at runtime -- correct two overstatements The previous commit said "vLLM itself is not modified" and described the changes as going "through vLLM's own extension points". Challenged on it, and the challenge was right: those are two different claims and only one of them holds as written. What is true: no file in the vLLM tree or in site-packages/vllm is edited, the checkout is clean upstream and the wheel is stock 0.28.0+cpu, so there is no patch series to apply and no vLLM fork to maintain. What was wrong: 1. vllm_omni/patch.py does nine _patch_* functions performing ten direct attribute replacements -- CpuPlatform.check_and_update_config, CPUWorker.load_model, torch.compiler.load_compiled_function. That is monkey-patching, not an extension point, and calling it one made the arrangement sound more robust than it is. Only the registries are genuine hooks: register_quantization_config, ir.ops.*.register_impl, platform plugins, direct_register_custom_op. 2. "The changes survive a wheel upgrade rather than being lost on it" is true of the registrations and overstated for the patches. Each patch is keyed to a specific method on a specific vLLM class; an upstream rename or behaviour change can break it or silently turn it into a no-op wrapper. The failure mode is a missing optimisation, not a crash, which is worse -- so the doc now says to re-run the harness after upgrading. The section is rewritten to separate the two mechanisms, list what each patch works around, and note that two of the three exist for upstream bugs and should disappear when those are fixed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ --- README.md | 2 +- docs/edge/environment.md | 63 ++++++++++++++++++++++++++++++---------- 2 files changed, 49 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index c7ffd599b..eecc65877 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ Upstream's own README is preserved at [docs/README.vllm-omni.md](docs/README.vll |---|---| | 4-bit CPU quantization | `vllm_omni/model_executor/layers/quantization/cpu_int4.py` — W4A16 on PyTorch's tinygemm kernel, prefill reading the kernel's own weight layout instead of a duplicate | | Memory | RoPE tables sized to the servable context; weights repacked in blocks; a KV budget that is actually honoured on CPU | -| CPU platform + vLLM patches | `vllm_omni/platforms/cpu/`, `vllm_omni/patch.py` — vLLM itself is **unmodified**; see [docs/edge/environment.md](docs/edge/environment.md) | +| CPU platform + vLLM patches | `vllm_omni/platforms/cpu/`, `vllm_omni/patch.py` — vLLM is monkey-patched at import; its *source* is unmodified and no vLLM is redistributed. See [docs/edge/environment.md](docs/edge/environment.md) | | Device routing | `vllm_omni/edge/weight_path.py` picks the weight format per device class, from measurements rather than assumption | | Measurement harness | `benchmarks/edge_harness/` — the grid runner, benchmarks, memory ledger, fidelity metric and agent suite every number here came from | | Raw results | `benchmarks/edge_harness/results/` — 100+ artifacts backing the published figures, each with provenance | diff --git a/docs/edge/environment.md b/docs/edge/environment.md index 67561b236..e415fa218 100644 --- a/docs/edge/environment.md +++ b/docs/edge/environment.md @@ -5,27 +5,60 @@ whose environment was not recorded cannot be retracted later, only doubted, so the harness stamps these into each artifact automatically — see the `provenance` block in any file under `benchmarks/edge_harness/results/`. -## vLLM is **not** modified +## Do we patch vLLM? Yes — at runtime. Is vLLM *modified*? No. -Worth stating plainly, because it is the obvious assumption and it is wrong. +Those are two different questions and this document previously blurred them. -This fork changes vLLM's *behaviour* on CPU, but the vLLM tree and the installed -wheel are stock. The changes are runtime patches applied at import through -vLLM's own extension points, so they survive a wheel upgrade instead of being -lost on it: +**We do patch vLLM.** `vllm_omni/patch.py` has nine `_patch_*` functions that +perform ten direct attribute replacements on vLLM classes and one on a torch +function. That is monkey-patching by any honest reading: -| patch (`vllm_omni/patch.py`) | what it changes | +```python +CpuPlatform.check_and_update_config = classmethod(check_and_update_config) +CPUWorker.load_model = load_model +torch.compiler.load_compiled_function = load_compiled_function +``` + +**vLLM's source is not modified.** No file in the vLLM tree or in +`site-packages/vllm` is edited; the checkout is clean upstream (0 commits ahead, +0 uncommitted) and the installed wheel is stock `0.28.0+cpu`. There is no patch +series to apply and no vLLM fork to maintain — `pip install vllm==0.28.0+cpu` +gives you exactly upstream, and this package changes its behaviour at import. + +### The three CPU monkey-patches + +| patch | what it changes | why it is a patch and not a hook | +|---|---|---| +| `_patch_cpu_aot_compile_cache_load` | vLLM's CPU path fails to load its own AOT artifact (`finalize_loading` missing on a plain function), treats it as a miss, and recompiles on every start | replaces `torch.compiler.load_compiled_function`; torch offers no hook | +| `_patch_cpu_explicit_kv_cache_memory` | `CpuPlatform.check_and_update_config` overwrites `kv_cache_memory_bytes` from `VLLM_CPU_KVCACHE_SPACE` unconditionally, so a per-stage budget had no effect — this is what made "KV sizing is not a memory lever" look true | wraps a classmethod on a platform class | +| `_patch_cpu_release_memory_after_load` | returns freed allocator arenas to the OS after weight load | wraps `CPUWorker.load_model` and `compile_or_warm_up_model` | + +They are guarded with `getattr`, made idempotent by a marker attribute, and +gated on `current_platform.is_cpu()`. They are **not** covered by any stability +contract: each is keyed to a specific method on a specific vLLM class, and an +upstream rename or behaviour change can break them or silently turn them into +no-op wrappers. Two of the three exist to work around upstream bugs and should +disappear when those are fixed. + +### The genuine extension points + +These are registries vLLM publishes, and they are stable across upgrades in a +way the patches above are not: + +| mechanism | used for | |---|---| -| `_patch_cpu_aot_compile_cache_load` | vLLM's CPU path fails to load its own AOT artifact (`finalize_loading` missing on a plain function), treats it as a miss, and recompiles on every start | -| `_patch_cpu_explicit_kv_cache_memory` | `CpuPlatform.check_and_update_config` overwrites `kv_cache_memory_bytes` from `VLLM_CPU_KVCACHE_SPACE` unconditionally, so a per-stage budget had no effect at all — this is what made "KV sizing is not a memory lever" look true | -| `_patch_cpu_release_memory_after_load` | returns freed allocator arenas to the OS after weight load | +| `register_quantization_config` | the `cpu_int4` quantization method | +| `ir.ops.*.register_impl` | CPU providers for `rms_norm` / `fused_add_rms_norm` (off by default — fused is slower in situ) | +| platform plugins | the CPU `OmniPlatform` | +| `direct_register_custom_op` | `cpu_int4_linear`, `spark_head_gate` | -Other extension points used, rather than edits: the quantization-config registry -(`register_quantization_config` for `cpu_int4`), the IR op registry, platform -hooks, and worker method wrapping. +### What this means for you -So there is no vLLM patch series to apply. Pin the version and let `patch.py` -do the rest. +Pin the vLLM version (below). Do not expect the monkey-patches to survive an +arbitrary upgrade — re-run `benchmarks/edge_harness/` after one, because the +failure mode of a stale patch is a silently missing optimisation, not a crash. +An earlier version of this page claimed these changes "survive a wheel upgrade"; +that is true of the registrations and overstated for the patches. ## Pins