Renamed from vllm-embedding: nothing here is about embeddings, and the name had
come from the parent working directory. edge-infer says what it does. The
GitHub repository is renamed to match; nothing had been pushed yet.
The audit found the packaging was incomplete in two ways.
Scripts: 56 of them produced results that are published in this repo but were
not included, so the numbers could be read and not checked. Added under
benchmarks/edge_harness/experiments/ (one-off drivers: the AI Hub on-device
runs, the 4-bit GEMM microbenchmark, fidelity and parity checks, checkpoint
build and round-trip, activation capture, and the load generator used to prove
the contention guard actually refuses) and benchmarks/edge_harness/runs/ (the
24 shell drivers behind published grids). Both directories carry a README
saying plainly that they are a record of what was run rather than a supported
interface -- most hardcode absolute paths from the machine they ran on. The
supported entry points remain the tools one level up.
One broken relative link fixed (comparison.md is under docs/engines/, not
docs/analysis/). All other links and every script reference in the docs now
resolve inside the repo.
Secret scan: the token in ~/.qai_hub/client.ini is absent from the worktree and
from all 61 commits. Worth noting that an earlier version of my own check
reported a false positive, because `grep -rl ... | head` takes its exit status
from head, which always succeeds -- the same class of mistake as the measurement
errors this project keeps recording. The AI Hub scripts authenticate by reading
that file at runtime and contain no credential.
README reframed for the new name and the wider scope: the repository is the
engine changes, the harness, the raw results, the baselines and the analyses,
not just a patch set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
