docs: we do patch vLLM, at runtime -- correct two overstatements

The previous commit said "vLLM itself is not modified" and described the
changes as going "through vLLM's own extension points". Challenged on it, and
the challenge was right: those are two different claims and only one of them
holds as written.

What is true: no file in the vLLM tree or in site-packages/vllm is edited, the
checkout is clean upstream and the wheel is stock 0.28.0+cpu, so there is no
patch series to apply and no vLLM fork to maintain.

What was wrong:

1. vllm_omni/patch.py does nine _patch_* functions performing ten direct
   attribute replacements -- CpuPlatform.check_and_update_config,
   CPUWorker.load_model, torch.compiler.load_compiled_function. That is
   monkey-patching, not an extension point, and calling it one made the
   arrangement sound more robust than it is. Only the registries are genuine
   hooks: register_quantization_config, ir.ops.*.register_impl, platform
   plugins, direct_register_custom_op.

2. "The changes survive a wheel upgrade rather than being lost on it" is true of
   the registrations and overstated for the patches. Each patch is keyed to a
   specific method on a specific vLLM class; an upstream rename or behaviour
   change can break it or silently turn it into a no-op wrapper. The failure
   mode is a missing optimisation, not a crash, which is worse -- so the doc now
   says to re-run the harness after upgrading.

The section is rewritten to separate the two mechanisms, list what each patch
works around, and note that two of the three exist for upstream bugs and should
disappear when those are fixed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
This commit is contained in:
tzhouamandClaude Opus 5 committed 2026-09-14 14:07:31 +08:00
1 parent 9264ff81ab
commit 99ebfbbd04
2 files changed
+49 -16

No files matched your search

+1 -1
View File
@@ -18,7 +18,7 @@ Upstream's own README is preserved at [docs/README.vllm-omni.md](docs/README.vll
|---|---|
| 4-bit CPU quantization | `vllm_omni/model_executor/layers/quantization/cpu_int4.py` — W4A16 on PyTorch's tinygemm kernel, prefill reading the kernel's own weight layout instead of a duplicate |
| Memory | RoPE tables sized to the servable context; weights repacked in blocks; a KV budget that is actually honoured on CPU |
| CPU platform + vLLM patches | `vllm_omni/platforms/cpu/`, `vllm_omni/patch.py` — vLLM itself is **unmodified**; see [docs/edge/environment.md](docs/edge/environment.md) |
| CPU platform + vLLM patches | `vllm_omni/platforms/cpu/`, `vllm_omni/patch.py` — vLLM is monkey-patched at import; its *source* is unmodified and no vLLM is redistributed. See [docs/edge/environment.md](docs/edge/environment.md) |
| Device routing | `vllm_omni/edge/weight_path.py` picks the weight format per device class, from measurements rather than assumption |
| Measurement harness | `benchmarks/edge_harness/` — the grid runner, benchmarks, memory ledger, fidelity metric and agent suite every number here came from |
| Raw results | `benchmarks/edge_harness/results/` — 100+ artifacts backing the published figures, each with provenance |