3 Commits

Include renames

Author	SHA1	Message	Date
tzhouamandClaude Opus 5	3834de3105	Rename the project to edge-infer, and complete the packaging audit ... Renamed from vllm-embedding: nothing here is about embeddings, and the name had come from the parent working directory. edge-infer says what it does. The GitHub repository is renamed to match; nothing had been pushed yet. The audit found the packaging was incomplete in two ways. Scripts: 56 of them produced results that are published in this repo but were not included, so the numbers could be read and not checked. Added under benchmarks/edge_harness/experiments/ (one-off drivers: the AI Hub on-device runs, the 4-bit GEMM microbenchmark, fidelity and parity checks, checkpoint build and round-trip, activation capture, and the load generator used to prove the contention guard actually refuses) and benchmarks/edge_harness/runs/ (the 24 shell drivers behind published grids). Both directories carry a README saying plainly that they are a record of what was run rather than a supported interface -- most hardcode absolute paths from the machine they ran on. The supported entry points remain the tools one level up. One broken relative link fixed (comparison.md is under docs/engines/, not docs/analysis/). All other links and every script reference in the docs now resolve inside the repo. Secret scan: the token in ~/.qai_hub/client.ini is absent from the worktree and from all 61 commits. Worth noting that an earlier version of my own check reported a false positive, because `grep -rl ... | head` takes its exit status from head, which always succeeds -- the same class of mistake as the measurement errors this project keeps recording. The AI Hub scripts authenticate by reading that file at runtime and contain no credential. README reframed for the new name and the wider scope: the repository is the engine changes, the harness, the raw results, the baselines and the analyses, not just a patch set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ	2026-09-14 14:21:07 +08:00
tzhouamandClaude Opus 5	9264ff81ab	Include the baselines, comparisons and study documents, not just the fork ... The packaging commit shipped the code and a summary of the numbers. It left out everything the numbers are measured *against*, which is most of what makes them checkable. Added: baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the interleaved A/B passes behind the headline decode figure) and the quality references: each build against its own bf16 over 12 greedy prompts. benchmarks/.../results/ 102 artifacts backing the published figures, each with a provenance block naming the model, quantization, thread set and build SHAs. docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN, ncnn, vLLM, vLLM-Omni -- and the cross-engine and device-simulation comparisons. docs/analysis/ the full study documents the summaries distil. docs/edge/comparisons.md where this fork sits against llama.cpp and why: decode is llama.cpp's, prefill is ours by ~2x, and since neither engine exceeds 39% of the measured 222 GB/s ceiling the decode gap is the execution model rather than the kernels. docs/edge/environment.md pins and machine, and the correction below. On "the modified vllm": there isn't one, and the doc says so plainly because it is the obvious assumption. The vLLM checkout is clean upstream (0 commits ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork changes vLLM's CPU *behaviour* through vLLM's own extension points at import time -- three CPU patches in vllm_omni/patch.py, the quantization-config registry, the IR op registry, platform hooks -- specifically so the changes survive a wheel upgrade rather than being lost on it. One of those patches is why "KV sizing is not a memory lever" looked true for months: CpuPlatform overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect. There is no patch series to apply; pin the version and let patch.py do the rest. .gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/` rules were silently eating the measurement artifacts, which in this fork are the substance rather than build output. Binary captures stay ignored. No vLLM or llama.cpp source is redistributed -- only output produced with them. NOTICE records both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ	2026-09-14 14:01:12 +08:00
tzhouamandClaude Opus 5	b36a3431bd	Package this fork as vllm-embedding: README, docs, roadmap, harness ... Turns a branch of accumulated edge/CPU work into something someone else can read, reproduce and continue. README.md states what the fork is for (multimodal models on edge CPUs rather than servers), the headline memory and speed results with their costs, and why the numbers are worth reading -- five retracted conclusions, all measurement errors rather than coding errors. Upstream's README is preserved at docs/README.vllm-omni.md and NOTICE records the Apache-2.0 derivation and the files this fork changes. docs/edge/: methodology.md the measurement discipline, which is the most transferable thing here: what each of the five retractions was, and the rule it left behind. Includes the two traps specific to this stack -- vLLM's AOT compile cache keying on config so two arms sharing an artifact load the wrong graph, and allocated not being resident until mincore(2) says so. results.md every measured figure with its spread and baseline, and an explicit note that the four memory levers overlap and must not be added. cpu-4bit.md the two 4-bit paths, the packed layout as measured rather than assumed, why the path choice is a config field, and the GPTQ output-head trap that invalidated a whole round of W4A8 numbers. harness.md how to run a comparison, and the two arguments that matter more than they look (--context-len, --prompt-len), each of which has produced a wrong conclusion by being left at its default. ROADMAP.md orders by whether the verification method is correct rather than by expected size, lists what is done with results, what is next with expected gains and how each would be verified, and -- at equal length -- what was considered and rejected, including the CPU megakernel with the measurements that rule it out for now. benchmarks/edge_harness/ brings the tooling into the repo so the numbers are reproducible: the grid runner, the single-run benchmarks, the memory ledger, the fidelity metric, the Android agent suite, the quantizers, and the tests that pin the harness's judgement. 82 tests pass from the new location. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ	2026-09-14 13:25:32 +08:00