99ebfbbd04465151d49d70c1a5ffe1017b5b0f96
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 tzhouamandClaude Opus 5
|
99ebfbbd04 |
docs: we do patch vLLM, at runtime -- correct two overstatements
The previous commit said "vLLM itself is not modified" and described the changes as going "through vLLM's own extension points". Challenged on it, and the challenge was right: those are two different claims and only one of them holds as written. What is true: no file in the vLLM tree or in site-packages/vllm is edited, the checkout is clean upstream and the wheel is stock 0.28.0+cpu, so there is no patch series to apply and no vLLM fork to maintain. What was wrong: 1. vllm_omni/patch.py does nine _patch_* functions performing ten direct attribute replacements -- CpuPlatform.check_and_update_config, CPUWorker.load_model, torch.compiler.load_compiled_function. That is monkey-patching, not an extension point, and calling it one made the arrangement sound more robust than it is. Only the registries are genuine hooks: register_quantization_config, ir.ops.*.register_impl, platform plugins, direct_register_custom_op. 2. "The changes survive a wheel upgrade rather than being lost on it" is true of the registrations and overstated for the patches. Each patch is keyed to a specific method on a specific vLLM class; an upstream rename or behaviour change can break it or silently turn it into a no-op wrapper. The failure mode is a missing optimisation, not a crash, which is worse -- so the doc now says to re-run the harness after upgrading. The section is rewritten to separate the two mechanisms, list what each patch works around, and note that two of the three exist for upstream bugs and should disappear when those are fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ |
||
|
tzhouamandClaude Opus 5
|
9264ff81ab |
Include the baselines, comparisons and study documents, not just the fork
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
|
||
|
tzhouamandClaude Opus 5
|
b36a3431bd |
Package this fork as vllm-embedding: README, docs, roadmap, harness
Turns a branch of accumulated edge/CPU work into something someone else can
read, reproduce and continue.
README.md states what the fork is for (multimodal models on edge CPUs rather
than servers), the headline memory and speed results with their costs, and why
the numbers are worth reading -- five retracted conclusions, all measurement
errors rather than coding errors. Upstream's README is preserved at
docs/README.vllm-omni.md and NOTICE records the Apache-2.0 derivation and the
files this fork changes.
docs/edge/:
methodology.md the measurement discipline, which is the most transferable
thing here: what each of the five retractions was, and the
rule it left behind. Includes the two traps specific to this
stack -- vLLM's AOT compile cache keying on config so two
arms sharing an artifact load the wrong graph, and allocated
not being resident until mincore(2) says so.
results.md every measured figure with its spread and baseline, and an
explicit note that the four memory levers overlap and must
not be added.
cpu-4bit.md the two 4-bit paths, the packed layout as measured rather
than assumed, why the path choice is a config field, and the
GPTQ output-head trap that invalidated a whole round of W4A8
numbers.
harness.md how to run a comparison, and the two arguments that matter
more than they look (--context-len, --prompt-len), each of
which has produced a wrong conclusion by being left at its
default.
ROADMAP.md orders by whether the verification method is correct rather than by
expected size, lists what is done with results, what is next with expected
gains and how each would be verified, and -- at equal length -- what was
considered and rejected, including the CPU megakernel with the measurements
that rule it out for now.
benchmarks/edge_harness/ brings the tooling into the repo so the numbers are
reproducible: the grid runner, the single-run benchmarks, the memory ledger,
the fidelity metric, the Android agent suite, the quantizers, and the tests
that pin the harness's judgement. 82 tests pass from the new location.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
|