tzhouamandClaude Opus 5
b36a3431bd
Turns a branch of accumulated edge/CPU work into something someone else can
read, reproduce and continue.
README.md states what the fork is for (multimodal models on edge CPUs rather
than servers), the headline memory and speed results with their costs, and why
the numbers are worth reading -- five retracted conclusions, all measurement
errors rather than coding errors. Upstream's README is preserved at
docs/README.vllm-omni.md and NOTICE records the Apache-2.0 derivation and the
files this fork changes.
docs/edge/:
methodology.md the measurement discipline, which is the most transferable
thing here: what each of the five retractions was, and the
rule it left behind. Includes the two traps specific to this
stack -- vLLM's AOT compile cache keying on config so two
arms sharing an artifact load the wrong graph, and allocated
not being resident until mincore(2) says so.
results.md every measured figure with its spread and baseline, and an
explicit note that the four memory levers overlap and must
not be added.
cpu-4bit.md the two 4-bit paths, the packed layout as measured rather
than assumed, why the path choice is a config field, and the
GPTQ output-head trap that invalidated a whole round of W4A8
numbers.
harness.md how to run a comparison, and the two arguments that matter
more than they look (--context-len, --prompt-len), each of
which has produced a wrong conclusion by being left at its
default.
ROADMAP.md orders by whether the verification method is correct rather than by
expected size, lists what is done with results, what is next with expected
gains and how each would be verified, and -- at equal length -- what was
considered and rejected, including the CPU megakernel with the measurements
that rule it out for now.
benchmarks/edge_harness/ brings the tooling into the repo so the numbers are
reproducible: the grid runner, the single-run benchmarks, the memory ledger,
the fidelity metric, the Android agent suite, the quantizers, and the tests
that pin the harness's judgement. 82 tests pass from the new location.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
117 lines
5.0 KiB
Markdown
117 lines
5.0 KiB
Markdown
# Measured results
|
||
|
||
Spark-X2.5-1.7B unless stated. Xeon Platinum 8480C, 24 pinned physical cores,
|
||
memory bound to the same NUMA node, W4A16 g128, `max_model_len` 2048.
|
||
Each row is a median over at least three interleaved passes against a control
|
||
arm running in the same grid.
|
||
|
||
## Memory
|
||
|
||
Every lever below was measured as its own A/B, with the alternative reachable by
|
||
a flag so both arms run identical code.
|
||
|
||
| lever | before | after | saving | cost |
|
||
|---|---|---|---|---|
|
||
| KV budget 1 GiB → 192 MiB | 3264 MB | 2421 MB | **843 MB** | fewer cached tokens |
|
||
| RoPE table → `max_model_len` | 3891 MB | 3273 MB | **618 MB** | none |
|
||
| One weight layout, not two | 4591 MB | 3976 MB | **615 MB** | ~1.63× prefill |
|
||
| Blocked repack (peak) | 4502 MB | 4013 MB | **489 MB** peak | none |
|
||
|
||
**These do not add.** The single-layout change removed `to_split_halves`, which
|
||
was doing its own whole-matrix expansion, so part of the blocked-repack saving
|
||
was already gone by the time it was measured. The report each was taken from
|
||
states its baseline; quote them individually.
|
||
|
||
### Where the memory actually is
|
||
|
||
From `memory_ledger.py`, which counts unique tensor storages by `data_ptr` (so
|
||
tied weights are counted once) and checks residency with `mincore(2)`:
|
||
|
||
| | MB | note |
|
||
|---|---|---|
|
||
| KV cache | 1024 | fully resident — 1024.1 of 1023.8 allocated |
|
||
| int4 packed weights | 686 | |
|
||
| RoPE cos/sin tables | 640 | **before the cap** — sized for 1 048 576 positions |
|
||
| embedding / head | 408 | tied; includes a 128 MB row-major copy |
|
||
| imports (`import vllm_omni`) | 501 | |
|
||
|
||
The 640 MB RoPE entry is the clearest single finding in the project. Spark
|
||
advertises a 1 048 576-token context and the model passed
|
||
`max_position_embeddings` straight to `get_rope`, so vLLM built
|
||
`torch.arange(1048576)` cos/sin tables — two of them, because the sliding and
|
||
full layers use different theta and partial-rotary factors. At `max_model_len`
|
||
2048 they need 1.25 MB between them. It had been sitting inside an
|
||
"unexplained ~1.6 GB" remainder.
|
||
|
||
### Peak vs settled
|
||
|
||
`release_after_load()` runs only once loading and warm-up return, so a device
|
||
that cannot hold the *peak* never reaches the release. Sampling `RssAnon` on a
|
||
timer through the load, rather than reading the endpoint:
|
||
|
||
| | peak | settled | transient |
|
||
|---|---|---|---|
|
||
| whole-matrix repack | 4502 MB | 3905 MB | 550 MB |
|
||
| blocked repack | 4013 MB | 3892 MB | 105 MB |
|
||
|
||
`unpack_nibbles` expands 4-bit codes to int32 — eight times the packed form —
|
||
because the kernel's packer wants int32. Whole-matrix that is 104 MiB for
|
||
`gate_up` and **1024 MiB for the tied embedding**, against 24 MiB of packed
|
||
weights for a whole layer. Chunking is also *faster* (79 ms vs 208 ms for the
|
||
embedding) because the working set stays in cache.
|
||
|
||
## Speed
|
||
|
||
| path | decode | prefill (pp512) | spread |
|
||
|---|---|---|---|
|
||
| W4A8, AMX `int4_scaled_mm_cpu` | 74.4 tok/s | — | 1.1% |
|
||
| W4A16, tinygemm | 71.6 tok/s | 2190 tok/s | 9.7% |
|
||
| W4A16, one layout | 71.6 tok/s | 1344 tok/s | 18.3% |
|
||
| W4A16, `prefill_dequant: false` | 73.0 tok/s | 453 tok/s | 0.3% |
|
||
| `cpu_gemm_wna16` fallback | 32.3 tok/s | — | 1.5% |
|
||
| llama.cpp Q4_K_M | 90.0 tok/s | 1121 tok/s | — |
|
||
|
||
`cpu_gemm_wna16` is the kernel vLLM selects when `use_w4a8` is off. It exists —
|
||
`weight_path.py` used to call tinygemm "the only option" without AMX, which was
|
||
false — but at 2.2× slower than tinygemm the routing was right anyway. Caveat:
|
||
`_get_isa_hint` returns `"amx"` whenever AMX is present regardless of the W4A8
|
||
flag, so this measured its *best* path on this host, not the `"vec"` path a
|
||
genuinely AMX-less machine takes.
|
||
|
||
## Bandwidth headroom
|
||
|
||
A node-local streaming read on the same 24 cores sustains **222 GB/s**.
|
||
|
||
| | GB/s | of ceiling |
|
||
|---|---|---|
|
||
| W4A16 GEMM only | 132 | 60% |
|
||
| W4A8 GEMM only | 153 | 69% |
|
||
| W4A16 whole step | 73 | 33% |
|
||
| llama.cpp whole step | 86 | 39% |
|
||
|
||
Neither engine is bandwidth-bound. An earlier draft asserted the GEMM ran at
|
||
"85% of what this socket delivers" and concluded the remainder was framework
|
||
overhead; at 60% of the measured ceiling, kernel and scheduling efficiency are
|
||
still on the table. See [ROADMAP.md](../../ROADMAP.md) item I10.
|
||
|
||
## Fidelity
|
||
|
||
Against each build's own bf16 output over 12 greedy prompts:
|
||
|
||
| build | exact reproductions | bits/weight |
|
||
|---|---|---|
|
||
| llama.cpp Q4_K_M | 4/12 | 4.5 (groups of 32, mixed types) |
|
||
| W4A16 | 2/12 | 4.25 |
|
||
| W4A8 | 1/12 | 4.25 |
|
||
|
||
**Not a matched-budget comparison** — Q4_K_M stores more. And greedy divergence
|
||
saturates: it discards everything after the first divergent token, so one
|
||
unlucky token scores the same as a build that is wrong everywhere.
|
||
`fidelity_kl.py` replaces it with top-1 agreement and KL over a corpus.
|
||
|
||
On the Android agent harness, every build navigates a multi-screen task
|
||
correctly and only some notice they have arrived: W4A8 turned Wi-Fi on and then
|
||
tapped the switch off again. **Navigation survives quantization; recognising
|
||
completion does not.** That is one run per cell — enough to show a failure
|
||
reproduces, not to rank success rates.
|