ee2129e2fa73ca1714e8eb3cb457a7185bf73dcb
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 tzhouamandClaude Opus 5
|
bc865385a8 |
edge: measure cpu_gemm_wna16 instead of asserting it away (I2)
The review caught weight_path.py claiming tinygemm is "the only option" for an
x86 core without AMX. That was false -- vLLM 0.28 selects ops.cpu_gemm_wna16
whenever layer.use_w4a8 is off -- and the honest response was to measure the
kernel, not to reword the claim.
Same GPTQ checkpoint, only VLLM_CPU_INT4_W4A8 differing, isolated compile
caches, 3 interleaved passes on 24 pinned node-local cores:
int4_scaled_mm_cpu (W4A8, AMX) 74.4 tok/s spread 1.1%
tinygemm W4A16 (same cores) 71.6 tok/s
cpu_gemm_wna16 32.3 tok/s spread 1.5%
The fallback is real but 2.2x slower than tinygemm, so the selector's routing
was right for a reason nobody had checked. It stays as it is; what changes is
that the module now cites a measurement rather than an assumption.
Recorded with the caveat that matters for reading it off-x86: _get_isa_hint
returns "amx" whenever AMX is present and activations are bf16, regardless of
the W4A8 flag, so this measured cpu_gemm_wna16 on its AMX path -- its best case
on this host. The "vec" path a genuinely AMX-less machine takes is untested and
presumably no faster, which only strengthens the conclusion, but no number is
claimed for it.
Also adds VLLM_OMNI_CPU_INT4_REPACK_ROWS so the blocked repack can be A/B'd
against whole-matrix packing in identical code. Unlike the prefill-path switch
this is load-time only and yields identical tensors, so it does not perturb the
compiled graph or its cache key.
tests/edge: 8 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
tzhouamandClaude Opus 5
|
27b9400750 |
edge: remove retracted and wrong numbers from weight_path and kv_budget
An independent review (gpt-6-astra via the Codex hook) found that two modules still ship claims this study had already withdrawn or that the vLLM source contradicts. Documentation in executable files is how a retracted number gets quoted again, so these are corrected at the source: kv_budget.py claimed right-sizing the cache saved 8329 -> 5254 MB. That was read from VmRSS (which counts the mmap'd checkpoint and moved 1-2 GB between identical runs) and the two arms were different model files. On RssAnon the saving is 188 MB. The retraction is now stated in the docstring, together with why the number is small -- the benchmark generated 8 tokens, so no cache was filled -- and the 1323 MiB flat-vs-hybrid difference at 32k that makes this worth re-measuring on a benchmark that does fill it. weight_path.py: - bits_per_weight 4.16 -> 4.25 in all three 4-bit paths. 4 bits plus a bf16 scale and zero per 128-weight group is 4 + 32/128; 4.16 matched no arithmetic and made the llama.cpp fidelity comparison look budget-matched when it is not (Q4_K is 4.5 bits over groups of 32, Q4_K_M promotes some tensors further). - fused_cpu_norms True -> False in all three. Fused CPU RMSNorm is 3.6-4.5x faster in isolation and slower in situ; VLLM_OMNI_CPU_FUSED_NORMS already defaults to "0" for that reason, so the metadata contradicted the default. - "tinygemm is the only option without AMX" is false: vLLM 0.28 selects ops.cpu_gemm_wna16 whenever use_w4a8 is off (mixed_precision/cpu.py:202). Unmeasured here, so the selector is unchanged, but the claim is not. - W4A8's symmetric grid is our exporter's "sym": true, not a kernel limit -- _process_gptq_weights_w4a8 uses checkpoint zero points when config.zero_points is set (same file, :98). An asymmetric export is a config change and the cheapest untried fidelity experiment. - kv_budget's flat-accounting observation is flagged for re-checking, since CpuPlatform.support_hybrid_kv_cache() returns True (platforms/cpu.py:545). Both fields now carry docstrings saying what they mean, so they cannot drift back silently. tests/edge: 18 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
tzhouamandClaude Opus 5
|
bc7f8237e3 |
[Edge] Agentic evidence for the 4-bit choice: W4A16, not W4A8
Driving a phone through the Android agent harness turns section 2c's fidelity
numbers into a task outcome. Same simulated device, one run per cell:
build decode "Turn on Wi-Fi." "Set an alarm"
bf16 41.0 tok/s done in 4 steps wrong app, stuck
W4A16 g128 80.0 tok/s done in 4 steps right app, no completion
W4A8 g128 86.6 tok/s turns it on, then off (x2) reaches Alarm, loops
Every build navigates home -> Settings -> Network -> Wi-Fi correctly. Only
some notice the switch now reads checked=true and call task_complete. W4A8 --
the fastest build, and the one the speed-first path recommends -- ends with
the task undone, on both runs.
So "W4A16 reproduces 2/12 greedy prompts and W4A8 1/12" was not a small
difference on a proxy metric: it is finishing versus reversing. The selector
now says so where someone choosing a build will read it, and points at
priority="fidelity" for agentic work.
Two tasks, one run per cell: enough to show the failure reproduces, not to
rank success rates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
tzhouamandClaude Opus 5
|
5f27f60bde |
[Edge] Correct the W4A8/W4A16 figures: the earlier ones had an unquantized head
The GPTQ checkpoint the comparison had been using left Spark's output head
tied and unquantized, so that one GEMM ran bf16 through `_C::onednn_mm` at
2.97 ms/step -- more than the whole 4-bit saving, and enough to make the
W4A8/W4A16 ordering flip between grids. The per-op profile is what caught it:
`int4_scaled_mm_cpu` ran 112 times (28 layers x 4 linears) and `onednn_mm`
once.
Rebuilt with the head quantized and re-profiled back to back on the same 24
cores, because this host's throughput swings ~30% and the gap is ~8%:
w4a8 AMX g128 GEMM 5.91 ms step 11.55 ms 86.6 tok/s
w4a16 tinygemm GEMM 6.85 ms step 12.51 ms 80.0 tok/s
llama.cpp Q4_K_M step 11.11 ms 90.0 tok/s
W4A8 wins twice over: the kernel is 0.94 ms cheaper per step, and it is
reached through vLLM's own dispatch, so it also avoids the 0.48 ms/step
custom-op boundary the tinygemm path needs to send prefill to a different
kernel. The selector's recommendation is unchanged; only its numbers were
wrong, and the lm_head trap is now documented where someone building a
checkpoint will read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
tzhouamandClaude Opus 5
|
3dac97e0bd |
[Edge][CPU] W4A8 via vLLM's own AMX kernel, a per-device weight path, and stop recompiling on every start
Four changes, each measured on a Xeon 8480C with Spark-X2.5-1.7B at 24 threads. Numbers are best-of-interleaved-passes; this host is shared and a single sequential grid swung the llama.cpp control 30%, so anything under ~8% is not treated as a result. **vLLM's CPU backend does have a 4-bit kernel.** An earlier note in this tree said it did not, on the strength of `woq_int4_linear`, `int4_scaled_mm_with_quant` and `da8w4_linear` all being absent. Those are simply not the names it uses: `CPUWNA16LinearKernel` dispatches GPTQ/AWQ checkpoints to `int4_scaled_mm_cpu`, an AMX W4A8 GEMM, whenever activations are bf16 and the checkpoint carries no act-order. Four-bit weights times eight-bit activations is the arithmetic llama.cpp's Q4_K_M already does. `gptq_cpu_export` builds checkpoints for it; the kernel pins the zero point at 8, so the grid must be symmetric, and a per-group scale search recovers most of what that costs. **A per-device weight path** (`edge/weight_path.py`). The fast kernel on each device needs a different, mutually exclusive weight layout, so the choice has to come from a probe rather than a flag: AMX x86 -> W4A8/GPTQ; x86 without AMX, or priority="fidelity" -> W4A16/tinygemm (bf16 activations, the more faithful of the two); ARM -> W4A16 but flagged unverified, because PyTorch's 4-bit CPU GEMM dispatches a different kernel there than the AVX-512 one measured here; phone -> runs_in_vllm=False, the deliverable is an exported graph with calibrated w8a16. **The CPU compile cache never loaded.** `_try_load_aot_compiled_fn` finishes with `compiled_fn.finalize_loading(cfg)`, which the CUDA wrapper object has and the inductor CPU backend's plain function does not. The AttributeError was swallowed as a cache miss, so every engine start recompiled the model: warm start 49.8 s against 14.9 s with the missing no-op attached. Self- disabling if upstream gives the CPU path a real one. **The head gate stops being a GEMM.** A 2048x8 projection cannot amortize a thread fan-out: dispatched as `aten::mm` it costs 1.155 ms of a decode step, nearly all barrier. As an eager reduction behind an opaque op it costs 0.444 ms. It has to be opaque -- written inline the reduction is traced, and inductor's loop made the whole step 1.8x slower (78.8 -> 44.5 tok/s). Also included and *off by default*: `cpu_ir_ops` registers the fused `_C::rms_norm` / `_C::fused_add_rms_norm` kernels that vLLM's CPU build ships but gates on `GPGPU_DEVICE`. In isolation they are 3.6x and 4.5x faster than the native path; end to end they are slightly slower (72.7 vs 74.0 tok/s), because inductor was already fusing the native norm into its neighbours and an opaque call breaks that fusion. Kept with the measurement recorded so the next person does not repeat it. `gelu_and_mul` is deliberately not registered: its fused CPU kernel is 2.7x slower at Spark's GEGLU shape. Decode, W4A16 g64 -> W4A8 g128 with the gate fix: 69.1 -> 84.5 tok/s (+22%), against llama.cpp Q4_K_M's 90.0; prefill 3347 against 1121 (3.0x). The two 4-bit paths are not equally faithful: against each build's own bf16 over 12 greedy prompts, Q4_K_M reproduces 4/12, W4A16 2/12 and W4A8 1/12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |