c4e61133fbb5587e0cf8972ce32ffac4e37fa739
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 tzhouamandClaude Opus 5
|
c4e61133fb |
[Edge][CPU] Fused norms off by default, and register the switches this branch added
Two corrections and the paperwork they imply.
The fused-RMSNorm registration defaulted to *on*, which contradicts the
measurement it was committed with: those kernels are 3.6x and 4.5x faster in
isolation but measured 72.7 tok/s against 74.0 end to end, because inductor
already fuses the native norm into the residual add and an opaque call breaks
that fusion. The implementations stay registered, so the provider remains
selectable through `kernel_config.ir_op_priority` on a machine where the trade
lands differently; only the default ranking is left alone now, and the test
asserts both states rather than just the opt-in one.
The four switches this branch added (VLLM_OMNI_AOT_CACHE_SHIM,
_CPU_FUSED_NORMS, _CPU_RELEASE_MEMORY, _SPARK_GATE_REDUCTION) were not in the
reviewed environment-variable inventory, so
`test_statically_resolvable_environment_accesses_are_classified` failed. They
are classified now, the reviewed count moves 28 -> 32 with a note saying why,
and each has a row in docs/configuration/environment_variables.md carrying the
measurement that justifies its default.
`VLLM_OMNI_CPU_FAST_SAMPLER` and `VLLM_OMNI_STEP_STATS_SYNC` remain
unclassified and that test still fails on them. They arrived in
|
||
|
tzhouamandClaude Opus 5
|
f5d3bfcf93 |
[Edge] Size the KV cache from the model, not from a server default
`VLLM_CPU_KVCACHE_SPACE` is a server-shaped default and is parsed with `int()`, so it cannot express an edge-sized cache at all -- the smallest non-zero value it accepts is 1 GiB. Measured on Spark-X2.5-1.7B (Xeon 8480C, 2048-token single stream), passing `kv_cache_memory_bytes` sized from the model instead of the 4 GiB default takes resident memory from 8329 MB to 5254 MB, a 3.1 GB saving with no effect on output. `kv_budget_for` reads the config and prices the cache. It reports two numbers because the model and the backend disagree: a hybrid-attention model does not need a full-length cache in every layer -- Spark is 21 sliding (window 512) to 7 full layers, so at 32 k context its real cache is 3.8x smaller than `layers x context` -- but vLLM's CPU path does not take that discount today (for a 4 GiB budget it reported 71 527 tokens, i.e. the flat 56 KiB/token). So `bytes_total` is the flat figure, which is what is safe to pass now, and `bytes_total_hybrid` is reported beside it for a backend that accounts per layer type. Deliberately conservative in three places, because a budget that is too small is not a graceful degradation -- vLLM preempts by recompute, which on a single-stream edge deployment is a stall: the window is rounded up to the block size, a 1.25 headroom factor is applied, and a config that has a `sliding_window` but no `layer_types` map is charged the flat rate rather than assumed to apply the window everywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
tzhouamandClaude Opus 5
|
3dac97e0bd |
[Edge][CPU] W4A8 via vLLM's own AMX kernel, a per-device weight path, and stop recompiling on every start
Four changes, each measured on a Xeon 8480C with Spark-X2.5-1.7B at 24 threads. Numbers are best-of-interleaved-passes; this host is shared and a single sequential grid swung the llama.cpp control 30%, so anything under ~8% is not treated as a result. **vLLM's CPU backend does have a 4-bit kernel.** An earlier note in this tree said it did not, on the strength of `woq_int4_linear`, `int4_scaled_mm_with_quant` and `da8w4_linear` all being absent. Those are simply not the names it uses: `CPUWNA16LinearKernel` dispatches GPTQ/AWQ checkpoints to `int4_scaled_mm_cpu`, an AMX W4A8 GEMM, whenever activations are bf16 and the checkpoint carries no act-order. Four-bit weights times eight-bit activations is the arithmetic llama.cpp's Q4_K_M already does. `gptq_cpu_export` builds checkpoints for it; the kernel pins the zero point at 8, so the grid must be symmetric, and a per-group scale search recovers most of what that costs. **A per-device weight path** (`edge/weight_path.py`). The fast kernel on each device needs a different, mutually exclusive weight layout, so the choice has to come from a probe rather than a flag: AMX x86 -> W4A8/GPTQ; x86 without AMX, or priority="fidelity" -> W4A16/tinygemm (bf16 activations, the more faithful of the two); ARM -> W4A16 but flagged unverified, because PyTorch's 4-bit CPU GEMM dispatches a different kernel there than the AVX-512 one measured here; phone -> runs_in_vllm=False, the deliverable is an exported graph with calibrated w8a16. **The CPU compile cache never loaded.** `_try_load_aot_compiled_fn` finishes with `compiled_fn.finalize_loading(cfg)`, which the CUDA wrapper object has and the inductor CPU backend's plain function does not. The AttributeError was swallowed as a cache miss, so every engine start recompiled the model: warm start 49.8 s against 14.9 s with the missing no-op attached. Self- disabling if upstream gives the CPU path a real one. **The head gate stops being a GEMM.** A 2048x8 projection cannot amortize a thread fan-out: dispatched as `aten::mm` it costs 1.155 ms of a decode step, nearly all barrier. As an eager reduction behind an opaque op it costs 0.444 ms. It has to be opaque -- written inline the reduction is traced, and inductor's loop made the whole step 1.8x slower (78.8 -> 44.5 tok/s). Also included and *off by default*: `cpu_ir_ops` registers the fused `_C::rms_norm` / `_C::fused_add_rms_norm` kernels that vLLM's CPU build ships but gates on `GPGPU_DEVICE`. In isolation they are 3.6x and 4.5x faster than the native path; end to end they are slightly slower (72.7 vs 74.0 tok/s), because inductor was already fusing the native norm into its neighbours and an opaque call breaks that fusion. Kept with the measurement recorded so the next person does not repeat it. `gelu_and_mul` is deliberately not registered: its fused CPU kernel is 2.7x slower at Spark's GEGLU shape. Decode, W4A16 g64 -> W4A8 g128 with the gate fix: 69.1 -> 84.5 tok/s (+22%), against llama.cpp Q4_K_M's 90.0; prefill 3347 against 1121 (3.0x). The two 4-bit paths are not equally faithful: against each build's own bf16 over 12 greedy prompts, Q4_K_M reproduces 4/12, W4A16 2/12 and W4A8 1/12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
tzhouamandClaude Opus 5
|
9a4f295625 |
[Edge][Mobile] Ring cache keeps the fused softmax so it survives w4a16
The score-space join removed the concat as intended and is numerically identical (140.3 dB), but it cannot be quantized: replacing torch.softmax with an explicit max/exp/sum/div made QNN's context-binary converter exit 14 under w4a16, while every fp16 build compiled. Two earlier shapes failed the same way -- a broadcast multiply plus ReduceSum for the new token's score, and a two-axis broadcast multiply for its value contribution. The device profile says the win was never the concat anyway. Of a sliding layer, Output was 9.9%, its quantise conversion 3.1% and the window Slice 7.4% -- all of it the cost of handing a 512-entry window back across the graph boundary once per token. So keep roll's attention exactly as it was, one fused Softmax over concatenated keys, and change only what the layer returns: the single new K/V entry, which the runtime drops into the ring slot. Ring order needs no fixing up, because softmax over keys is order-independent and each cached key already carries its own rotary phase. Measured on a real Galaxy S25, sliding layer, ms: w4a16 0.917 -> 0.838 (-8.6%) <- now compiles fp16 1.689 -> 1.486 (-12.0%) Full-attention layers keep roll: ring gains them nothing (1.567 vs 1.559 at fp16, the concat saved being spent on the extra score matmul), so the default layout is now per layer type. Parity against the reference is 140.3 dB for the mixed default with identical top-1/top-5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ |
||
|
tzhouamandClaude Opus 5
|
67bee59fda |
[Edge][Mobile] Ring-buffer KV and score-space attention for the decode step
Reading the device profile of the first export showed a fifth of every
sliding layer went nowhere useful: Output 9.9%, its quantise conversion
3.1%, and the window Slice 7.4%. The layer took a 512-entry window in,
appended the new key, sliced the oldest off and handed the whole recomputed
window back out -- 21 of 28 layers paying to move a cache-sized tensor
across the graph boundary once per token. Full layers paid a similar 9.5%
concatenating the new key onto the cache.
Both are the same mistake, and both go away by joining in score space
instead of key space: score the cache and the new token separately, then
concatenate two vectors of length n and 1 rather than an n x 256 key matrix.
Nothing cache-sized is copied, and each layer returns just the new K/V entry
for the runtime to drop into its ring slot. Softmax over keys does not care
what order the slots are in and each cached key already carries its own
rotary phase, so a ring buffer needs no reordering -- only a mask for slots
not yet written. The sliding cache correspondingly holds window-1 entries,
since a 512-token window counts the current token (transformers' own sliding
cache holds 511 for the same reason).
Measured on a real Galaxy S25, fp16, ms per layer:
sliding 1.689 -> 1.483 (-12.2%)
full 1.559 -> 1.567 (unchanged: the concat saved is spent on the
extra score matmul)
Two shapes had to be written as matmuls rather than the obvious elementwise
forms -- the new token's score (broadcast multiply + ReduceSum) and its
value contribution (a two-axis broadcast multiply). Both compile at fp16 and
both make QNN's context-binary converter exit 14 under w4a16; as matmuls
they compile and are the cheaper op on HTP anyway.
Also adds a selectable GELU approximation, and the measurement that closes
the question: erf GELU is 11-13% of every layer, but tanh is *slower* on
this NPU (1.535 vs 1.483 ms) despite holding 65.2 dB. The exact form stays,
which is also the better-fidelity choice.
Parity against the reference implementation is 136.8 dB with identical
top-1/top-5, and a test pins ring and roll to the same output on a shared
history so the device numbers cannot silently measure a different model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
|
||
|
tzhouamandClaude Opus 5
|
658341531a |
[Edge][Mobile] Spark-X2.5 decode-step export, and a compile-safe head gate
Exports one Spark decode step with the KV cache as explicit tensors, sized per layer type: the 21 sliding layers hold a fixed 512 entries at any context, the 7 full-attention layers grow. That asymmetry is the whole reason the architecture is interesting on a phone, so the two types get separately sized caches, masks and rotary angles. Verified against the HF reference at 1024 context: 138.5 dB logits SNR, identical top-1 and top-5. Profiled on a real Galaxy S25 (Snapdragon 8 Elite) through AI Hub, QNN context binary on the NPU, ms per layer: sliding (512 KV, any ctx) fp16 1.689 w8a16 0.912 w4a16 0.917 full @ 1k fp16 1.559 w8a16 1.129 w4a16 1.127 full @ 4k fp16 2.964 w8a16 1.681 w4a16 1.662 Composed into a step (21 sliding + 7 full), w4a16 gives 27.2 ms at 1k and 30.9 ms at 4k, against 31.6 and 46.5 ms for the same model with every layer full-attention -- 1.16x and 1.51x. Worth noting the hybrid is 0.94x at 1k in fp16: sliding layers rotate all 256 head dims where full layers rotate 64, and below roughly 2k that RoPE cost outweighs the shorter cache. Also moves the head-wise attention gate off vLLM's CPU linear dispatch. The gate is [2048, 8], and any linear that narrow is routed to the sgl-kernel packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim 0; VNNI packing does not at that size, so it reports K and torch.compile fails tracing the gate multiply. A plain parameter with F.linear gains nothing less -- a 2048x8 matmul has no use for packed AMX tiles -- and it keeps the model compilable on CPU and GPU alike. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ |
||
|
tzhouamandClaude Opus 5
|
73cdc7395a |
[Edge][Mobile] QNN/TFLite decode-step export for phone accelerators
The Code2Wav decoder already had an on-device export (WP4); the talker and its code predictor did not, so the codec-token path stopped at the vocoder. This adds the missing half: a one-token decode step with an explicit per-layer KV cache, exportable to ONNX and from there to a QNN context binary, TFLite or ONNX Runtime. Two layout choices carry the performance, both measured on real Snapdragon devices through Qualcomm AI Hub (Galaxy S24 = 8 Gen 3, S25 = 8 Elite): * per-layer cache tensors instead of one stacked [num_layers, ...] tensor that every layer slices, and * grouped-query attention by reshaping the query rather than expanding keys and values. The stacked/expanded form spent ~70% of its NPU cycles in cache select, reshape and fp32->fp16 conversion; the layout here is bandwidth-bound and runs the 0.6B talker step in 15.8 ms instead of 36.9 ms on an 8 Elite (fp16, 256-token cache). The code predictor step is 2.25 ms, so talker plus its 15 sub-steps cost about 50 ms of an 80 ms frame. sanitize_onnx() fixes the three things the dynamo exporter emits that qairt-converter and AI Hub reject: IsNaN/Where guards, Reshape allowzero=1, and rank-0 axes initializers on Unsqueeze/Squeeze. The module docstring records the measured placement table: this graph belongs on the NPU, the vocoder on the GPU (21.7 ms vs 182 ms per frame), which is 3.3x better than sending everything to the NPU and makes the pipeline real time on the phone in fp16. Integer quantization is not the lever it looks like: every automatic int8 build measured was numerically broken. Full write-up, job ids and profiles: analysis/mobile_htp_simulation.md section 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzr1Jusv4s6g765gqS9L92 |
||
|
tzhouamandClaude Fable 5.1
|
580a201631 |
[Edge][WP4] Windowed Code2Wav decoder export (torch.export / ExecuTorch) + client backend; results summarizer
vllm_omni/edge/decoder_export.py exports a stateless windowed decoder (context 72 frames + chunk, one program per ramp chunk size) via torch.export (.pt2, verified here on CPU for chunks 2 and 25 with the real 0.6B decoder: exact match vs eager, 460 MB fp32 per program) or ExecuTorch XNNPACK (.pte; written against the executorch 1.x API, not executed: package not installed, install is permission gated; QNN partitioner export-only/unverified). The WP5 client gains --decoder-backend executorch --pte-manifest for phone/NPU targets. benchmarks/tts/edge_results/summarize.py renders all committed JSON as tables. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
tzhouamandClaude Fable 5.1
|
37c0ea9a6b |
[Edge][WP8] Per-step overhead attribution, edge scheduling helpers, CPU thread sweep
- vllm_omni/edge/step_stats.py: process-local counters enabled by VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped per process and merged into an attribution table (share of the 80 ms frame). Hooks: stage_engine_core_proc (instrument after construction, dump at exit), chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime), orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats. - vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and the no-preemption KV admission rule (exact bytes/token from the talker KV geometry); adapt.derive_overrides now sizes max_num_seqs with it and auto_profile passes the geometry read from the model config. - Engine re-reads parallel_stage_init after apply_deploy_profile so the edge profile's orchestrator default takes effect (was read before the profile). - benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor) with per-step attribution. - Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py, tests/benchmarks/test_cpu_thread_sweep.py, profile default test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
tzhouamandClaude Fable 5.1
|
55192de984 |
[Edge][WP4] Add hardware adaptation: probe, derived overrides, auto profile, calibration
* vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch (ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores, big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu / npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests). * vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from available memory minus weights, max_num_seqs, per-stage disjoint CPU thread ranges on the big cluster, eager/graph switches for small or unified memory, adaptive-chunk parameters, timeouts from a load cost model. * vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT. * vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml -> edge/hardware/<class>.yaml -> derived overrides -> cached calibration, materialized as a temp deploy YAML (platform overlays are superseded). * benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY, taskset, profile overrides) driving stream_latency_bench with auto profiles. * CPU-only tests for probe snapshots, derivation rules, calibration and auto. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |