ee2129e2fa73ca1714e8eb3cb457a7185bf73dcb
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 tzhouamandClaude Opus 5
|
ee2129e2fa |
edge: KV sizing is the largest memory lever, not a dead end
This module carried a retraction saying the measured saving from right-sizing
the cache was 188 MB. That retraction was itself measured through a path that
ignored the knob: CpuPlatform.check_and_update_config overwrote
kv_cache_memory_bytes from VLLM_CPU_KVCACHE_SPACE unconditionally, so the
"32x shrink" never happened and both arms ran the same cache.
With the budget actually honoured, 3 interleaved passes per arm, both filling
a 1536-token context:
1 GiB budget 3264 MB resident
192 MiB budget 2421 MB resident -> 843 MB
And the reasoning behind "the pages are never touched" is wrong too. mincore(2)
reports the cache fully in core: 1024.1 MB resident of 1023.8 allocated, and
191.1 of 190.8 when sized down. It is not address space, it is memory.
843 MB is the largest single lever measured on this model -- more than the
single weight layout (615 MB), the blocked repack (489 MB peak), or the RoPE
table (618 MB).
The general lesson, recorded in the report: a lever tested through a path that
silently ignores it measures nothing, and "no effect" is the most dangerous
result to accept without first checking the knob was connected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
|
||
|
tzhouamandClaude Opus 5
|
bc865385a8 |
edge: measure cpu_gemm_wna16 instead of asserting it away (I2)
The review caught weight_path.py claiming tinygemm is "the only option" for an
x86 core without AMX. That was false -- vLLM 0.28 selects ops.cpu_gemm_wna16
whenever layer.use_w4a8 is off -- and the honest response was to measure the
kernel, not to reword the claim.
Same GPTQ checkpoint, only VLLM_CPU_INT4_W4A8 differing, isolated compile
caches, 3 interleaved passes on 24 pinned node-local cores:
int4_scaled_mm_cpu (W4A8, AMX) 74.4 tok/s spread 1.1%
tinygemm W4A16 (same cores) 71.6 tok/s
cpu_gemm_wna16 32.3 tok/s spread 1.5%
The fallback is real but 2.2x slower than tinygemm, so the selector's routing
was right for a reason nobody had checked. It stays as it is; what changes is
that the module now cites a measurement rather than an assumption.
Recorded with the caveat that matters for reading it off-x86: _get_isa_hint
returns "amx" whenever AMX is present and activations are bf16, regardless of
the W4A8 flag, so this measured cpu_gemm_wna16 on its AMX path -- its best case
on this host. The "vec" path a genuinely AMX-less machine takes is untested and
presumably no faster, which only strengthens the conclusion, but no number is
claimed for it.
Also adds VLLM_OMNI_CPU_INT4_REPACK_ROWS so the blocked repack can be A/B'd
against whole-matrix packing in identical code. Unlike the prefill-path switch
this is load-time only and yields identical tensors, so it does not perturb the
compiled graph or its cache key.
tests/edge: 8 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
tzhouamandClaude Opus 5
|
27b9400750 |
edge: remove retracted and wrong numbers from weight_path and kv_budget
An independent review (gpt-6-astra via the Codex hook) found that two modules still ship claims this study had already withdrawn or that the vLLM source contradicts. Documentation in executable files is how a retracted number gets quoted again, so these are corrected at the source: kv_budget.py claimed right-sizing the cache saved 8329 -> 5254 MB. That was read from VmRSS (which counts the mmap'd checkpoint and moved 1-2 GB between identical runs) and the two arms were different model files. On RssAnon the saving is 188 MB. The retraction is now stated in the docstring, together with why the number is small -- the benchmark generated 8 tokens, so no cache was filled -- and the 1323 MiB flat-vs-hybrid difference at 32k that makes this worth re-measuring on a benchmark that does fill it. weight_path.py: - bits_per_weight 4.16 -> 4.25 in all three 4-bit paths. 4 bits plus a bf16 scale and zero per 128-weight group is 4 + 32/128; 4.16 matched no arithmetic and made the llama.cpp fidelity comparison look budget-matched when it is not (Q4_K is 4.5 bits over groups of 32, Q4_K_M promotes some tensors further). - fused_cpu_norms True -> False in all three. Fused CPU RMSNorm is 3.6-4.5x faster in isolation and slower in situ; VLLM_OMNI_CPU_FUSED_NORMS already defaults to "0" for that reason, so the metadata contradicted the default. - "tinygemm is the only option without AMX" is false: vLLM 0.28 selects ops.cpu_gemm_wna16 whenever use_w4a8 is off (mixed_precision/cpu.py:202). Unmeasured here, so the selector is unchanged, but the claim is not. - W4A8's symmetric grid is our exporter's "sym": true, not a kernel limit -- _process_gptq_weights_w4a8 uses checkpoint zero points when config.zero_points is set (same file, :98). An asymmetric export is a config change and the cheapest untried fidelity experiment. - kv_budget's flat-accounting observation is flagged for re-checking, since CpuPlatform.support_hybrid_kv_cache() returns True (platforms/cpu.py:545). Both fields now carry docstrings saying what they mean, so they cannot drift back silently. tests/edge: 18 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
tzhouamandClaude Opus 5
|
bc7f8237e3 |
[Edge] Agentic evidence for the 4-bit choice: W4A16, not W4A8
Driving a phone through the Android agent harness turns section 2c's fidelity
numbers into a task outcome. Same simulated device, one run per cell:
build decode "Turn on Wi-Fi." "Set an alarm"
bf16 41.0 tok/s done in 4 steps wrong app, stuck
W4A16 g128 80.0 tok/s done in 4 steps right app, no completion
W4A8 g128 86.6 tok/s turns it on, then off (x2) reaches Alarm, loops
Every build navigates home -> Settings -> Network -> Wi-Fi correctly. Only
some notice the switch now reads checked=true and call task_complete. W4A8 --
the fastest build, and the one the speed-first path recommends -- ends with
the task undone, on both runs.
So "W4A16 reproduces 2/12 greedy prompts and W4A8 1/12" was not a small
difference on a proxy metric: it is finishing versus reversing. The selector
now says so where someone choosing a build will read it, and points at
priority="fidelity" for agentic work.
Two tasks, one run per cell: enough to show the failure reproduces, not to
rank success rates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
tzhouamandClaude Opus 5
|
c440616c8b |
[Edge][CPU] Return the weight-loading churn to the OS: 7338 -> 3510 MB
Loading a quantized checkpoint allocates far more than the weights occupy.
`_process_gptq_weights_w4a8` unpacks 4-bit codes to int32 -- four bytes per
weight -- permutes, repacks, and drops the intermediates. For a 1.7 B model
that is gigabytes of short-lived allocation, correctly freed, and still
resident afterwards because a caching allocator keeps the arenas.
Measured on Spark-X2.5-1.7B, anonymous RSS:
path after load after release freed
W4A8 (GPTQ) 7338 MB 3510 MB 3828 MB (-52%)
W4A16 (tinygemm) 5023 MB 3411 MB 1612 MB
The second column is the more useful finding: after the release both paths
land in the same place, so W4A8's apparent 2.3 GB penalty over W4A16 was never
a footprint difference -- it was the repack. W4A8 is therefore faster *and*
level on memory, and fidelity is the only remaining reason to pick W4A16.
Hooked to CPUWorker.load_model and compile_or_warm_up_model, since those churn
separately; `VLLM_OMNI_CPU_RELEASE_MEMORY=0` disables it. CPU only: on GPU the
weights are in device memory and host churn constrains nothing.
`anon_rss_bytes()` ships alongside because the metric matters as much as the
fix. VmRSS counts the mmap'd checkpoint's file pages and moved 1.2-2.0 GB
between *identical* runs on this host; RssAnon repeats to within 0.5 MB. An
earlier claim in this tree that KV sizing saved 3.1 GB came from reading VmRSS
across a comparison that also changed the model file; re-measured on RssAnon,
shrinking the KV cache 32x is worth 188 MB, because its pages are never
touched. The analysis note is corrected accordingly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
tzhouamandClaude Opus 5
|
5f27f60bde |
[Edge] Correct the W4A8/W4A16 figures: the earlier ones had an unquantized head
The GPTQ checkpoint the comparison had been using left Spark's output head
tied and unquantized, so that one GEMM ran bf16 through `_C::onednn_mm` at
2.97 ms/step -- more than the whole 4-bit saving, and enough to make the
W4A8/W4A16 ordering flip between grids. The per-op profile is what caught it:
`int4_scaled_mm_cpu` ran 112 times (28 layers x 4 linears) and `onednn_mm`
once.
Rebuilt with the head quantized and re-profiled back to back on the same 24
cores, because this host's throughput swings ~30% and the gap is ~8%:
w4a8 AMX g128 GEMM 5.91 ms step 11.55 ms 86.6 tok/s
w4a16 tinygemm GEMM 6.85 ms step 12.51 ms 80.0 tok/s
llama.cpp Q4_K_M step 11.11 ms 90.0 tok/s
W4A8 wins twice over: the kernel is 0.94 ms cheaper per step, and it is
reached through vLLM's own dispatch, so it also avoids the 0.48 ms/step
custom-op boundary the tinygemm path needs to send prefill to a different
kernel. The selector's recommendation is unchanged; only its numbers were
wrong, and the lm_head trap is now documented where someone building a
checkpoint will read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
tzhouamandClaude Opus 5
|
f5d3bfcf93 |
[Edge] Size the KV cache from the model, not from a server default
`VLLM_CPU_KVCACHE_SPACE` is a server-shaped default and is parsed with `int()`, so it cannot express an edge-sized cache at all -- the smallest non-zero value it accepts is 1 GiB. Measured on Spark-X2.5-1.7B (Xeon 8480C, 2048-token single stream), passing `kv_cache_memory_bytes` sized from the model instead of the 4 GiB default takes resident memory from 8329 MB to 5254 MB, a 3.1 GB saving with no effect on output. `kv_budget_for` reads the config and prices the cache. It reports two numbers because the model and the backend disagree: a hybrid-attention model does not need a full-length cache in every layer -- Spark is 21 sliding (window 512) to 7 full layers, so at 32 k context its real cache is 3.8x smaller than `layers x context` -- but vLLM's CPU path does not take that discount today (for a 4 GiB budget it reported 71 527 tokens, i.e. the flat 56 KiB/token). So `bytes_total` is the flat figure, which is what is safe to pass now, and `bytes_total_hybrid` is reported beside it for a backend that accounts per layer type. Deliberately conservative in three places, because a budget that is too small is not a graceful degradation -- vLLM preempts by recompute, which on a single-stream edge deployment is a stall: the window is rounded up to the block size, a 1.25 headroom factor is applied, and a config that has a `sliding_window` but no `layer_types` map is charged the flat rate rather than assumed to apply the window everywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
tzhouamandClaude Opus 5
|
3dac97e0bd |
[Edge][CPU] W4A8 via vLLM's own AMX kernel, a per-device weight path, and stop recompiling on every start
Four changes, each measured on a Xeon 8480C with Spark-X2.5-1.7B at 24 threads. Numbers are best-of-interleaved-passes; this host is shared and a single sequential grid swung the llama.cpp control 30%, so anything under ~8% is not treated as a result. **vLLM's CPU backend does have a 4-bit kernel.** An earlier note in this tree said it did not, on the strength of `woq_int4_linear`, `int4_scaled_mm_with_quant` and `da8w4_linear` all being absent. Those are simply not the names it uses: `CPUWNA16LinearKernel` dispatches GPTQ/AWQ checkpoints to `int4_scaled_mm_cpu`, an AMX W4A8 GEMM, whenever activations are bf16 and the checkpoint carries no act-order. Four-bit weights times eight-bit activations is the arithmetic llama.cpp's Q4_K_M already does. `gptq_cpu_export` builds checkpoints for it; the kernel pins the zero point at 8, so the grid must be symmetric, and a per-group scale search recovers most of what that costs. **A per-device weight path** (`edge/weight_path.py`). The fast kernel on each device needs a different, mutually exclusive weight layout, so the choice has to come from a probe rather than a flag: AMX x86 -> W4A8/GPTQ; x86 without AMX, or priority="fidelity" -> W4A16/tinygemm (bf16 activations, the more faithful of the two); ARM -> W4A16 but flagged unverified, because PyTorch's 4-bit CPU GEMM dispatches a different kernel there than the AVX-512 one measured here; phone -> runs_in_vllm=False, the deliverable is an exported graph with calibrated w8a16. **The CPU compile cache never loaded.** `_try_load_aot_compiled_fn` finishes with `compiled_fn.finalize_loading(cfg)`, which the CUDA wrapper object has and the inductor CPU backend's plain function does not. The AttributeError was swallowed as a cache miss, so every engine start recompiled the model: warm start 49.8 s against 14.9 s with the missing no-op attached. Self- disabling if upstream gives the CPU path a real one. **The head gate stops being a GEMM.** A 2048x8 projection cannot amortize a thread fan-out: dispatched as `aten::mm` it costs 1.155 ms of a decode step, nearly all barrier. As an eager reduction behind an opaque op it costs 0.444 ms. It has to be opaque -- written inline the reduction is traced, and inductor's loop made the whole step 1.8x slower (78.8 -> 44.5 tok/s). Also included and *off by default*: `cpu_ir_ops` registers the fused `_C::rms_norm` / `_C::fused_add_rms_norm` kernels that vLLM's CPU build ships but gates on `GPGPU_DEVICE`. In isolation they are 3.6x and 4.5x faster than the native path; end to end they are slightly slower (72.7 vs 74.0 tok/s), because inductor was already fusing the native norm into its neighbours and an opaque call breaks that fusion. Kept with the measurement recorded so the next person does not repeat it. `gelu_and_mul` is deliberately not registered: its fused CPU kernel is 2.7x slower at Spark's GEGLU shape. Decode, W4A16 g64 -> W4A8 g128 with the gate fix: 69.1 -> 84.5 tok/s (+22%), against llama.cpp Q4_K_M's 90.0; prefill 3347 against 1121 (3.0x). The two 4-bit paths are not equally faithful: against each build's own bf16 over 12 greedy prompts, Q4_K_M reproduces 4/12, W4A16 2/12 and W4A8 1/12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
tzhouamandClaude Opus 5
|
9a4f295625 |
[Edge][Mobile] Ring cache keeps the fused softmax so it survives w4a16
The score-space join removed the concat as intended and is numerically identical (140.3 dB), but it cannot be quantized: replacing torch.softmax with an explicit max/exp/sum/div made QNN's context-binary converter exit 14 under w4a16, while every fp16 build compiled. Two earlier shapes failed the same way -- a broadcast multiply plus ReduceSum for the new token's score, and a two-axis broadcast multiply for its value contribution. The device profile says the win was never the concat anyway. Of a sliding layer, Output was 9.9%, its quantise conversion 3.1% and the window Slice 7.4% -- all of it the cost of handing a 512-entry window back across the graph boundary once per token. So keep roll's attention exactly as it was, one fused Softmax over concatenated keys, and change only what the layer returns: the single new K/V entry, which the runtime drops into the ring slot. Ring order needs no fixing up, because softmax over keys is order-independent and each cached key already carries its own rotary phase. Measured on a real Galaxy S25, sliding layer, ms: w4a16 0.917 -> 0.838 (-8.6%) <- now compiles fp16 1.689 -> 1.486 (-12.0%) Full-attention layers keep roll: ring gains them nothing (1.567 vs 1.559 at fp16, the concat saved being spent on the extra score matmul), so the default layout is now per layer type. Parity against the reference is 140.3 dB for the mixed default with identical top-1/top-5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ |
||
|
tzhouamandClaude Opus 5
|
67bee59fda |
[Edge][Mobile] Ring-buffer KV and score-space attention for the decode step
Reading the device profile of the first export showed a fifth of every
sliding layer went nowhere useful: Output 9.9%, its quantise conversion
3.1%, and the window Slice 7.4%. The layer took a 512-entry window in,
appended the new key, sliced the oldest off and handed the whole recomputed
window back out -- 21 of 28 layers paying to move a cache-sized tensor
across the graph boundary once per token. Full layers paid a similar 9.5%
concatenating the new key onto the cache.
Both are the same mistake, and both go away by joining in score space
instead of key space: score the cache and the new token separately, then
concatenate two vectors of length n and 1 rather than an n x 256 key matrix.
Nothing cache-sized is copied, and each layer returns just the new K/V entry
for the runtime to drop into its ring slot. Softmax over keys does not care
what order the slots are in and each cached key already carries its own
rotary phase, so a ring buffer needs no reordering -- only a mask for slots
not yet written. The sliding cache correspondingly holds window-1 entries,
since a 512-token window counts the current token (transformers' own sliding
cache holds 511 for the same reason).
Measured on a real Galaxy S25, fp16, ms per layer:
sliding 1.689 -> 1.483 (-12.2%)
full 1.559 -> 1.567 (unchanged: the concat saved is spent on the
extra score matmul)
Two shapes had to be written as matmuls rather than the obvious elementwise
forms -- the new token's score (broadcast multiply + ReduceSum) and its
value contribution (a two-axis broadcast multiply). Both compile at fp16 and
both make QNN's context-binary converter exit 14 under w4a16; as matmuls
they compile and are the cheaper op on HTP anyway.
Also adds a selectable GELU approximation, and the measurement that closes
the question: erf GELU is 11-13% of every layer, but tanh is *slower* on
this NPU (1.535 vs 1.483 ms) despite holding 65.2 dB. The exact form stays,
which is also the better-fidelity choice.
Parity against the reference implementation is 136.8 dB with identical
top-1/top-5, and a test pins ring and roll to the same output on a shared
history so the device numbers cannot silently measure a different model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
|
||
|
tzhouamandClaude Opus 5
|
658341531a |
[Edge][Mobile] Spark-X2.5 decode-step export, and a compile-safe head gate
Exports one Spark decode step with the KV cache as explicit tensors, sized per layer type: the 21 sliding layers hold a fixed 512 entries at any context, the 7 full-attention layers grow. That asymmetry is the whole reason the architecture is interesting on a phone, so the two types get separately sized caches, masks and rotary angles. Verified against the HF reference at 1024 context: 138.5 dB logits SNR, identical top-1 and top-5. Profiled on a real Galaxy S25 (Snapdragon 8 Elite) through AI Hub, QNN context binary on the NPU, ms per layer: sliding (512 KV, any ctx) fp16 1.689 w8a16 0.912 w4a16 0.917 full @ 1k fp16 1.559 w8a16 1.129 w4a16 1.127 full @ 4k fp16 2.964 w8a16 1.681 w4a16 1.662 Composed into a step (21 sliding + 7 full), w4a16 gives 27.2 ms at 1k and 30.9 ms at 4k, against 31.6 and 46.5 ms for the same model with every layer full-attention -- 1.16x and 1.51x. Worth noting the hybrid is 0.94x at 1k in fp16: sliding layers rotate all 256 head dims where full layers rotate 64, and below roughly 2k that RoPE cost outweighs the shorter cache. Also moves the head-wise attention gate off vLLM's CPU linear dispatch. The gate is [2048, 8], and any linear that narrow is routed to the sgl-kernel packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim 0; VNNI packing does not at that size, so it reports K and torch.compile fails tracing the gate multiply. A plain parameter with F.linear gains nothing less -- a 2048x8 matmul has no use for packed AMX tiles -- and it keeps the model compilable on CPU and GPU alike. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ |
||
|
tzhouamandClaude Opus 5
|
73cdc7395a |
[Edge][Mobile] QNN/TFLite decode-step export for phone accelerators
The Code2Wav decoder already had an on-device export (WP4); the talker and its code predictor did not, so the codec-token path stopped at the vocoder. This adds the missing half: a one-token decode step with an explicit per-layer KV cache, exportable to ONNX and from there to a QNN context binary, TFLite or ONNX Runtime. Two layout choices carry the performance, both measured on real Snapdragon devices through Qualcomm AI Hub (Galaxy S24 = 8 Gen 3, S25 = 8 Elite): * per-layer cache tensors instead of one stacked [num_layers, ...] tensor that every layer slices, and * grouped-query attention by reshaping the query rather than expanding keys and values. The stacked/expanded form spent ~70% of its NPU cycles in cache select, reshape and fp32->fp16 conversion; the layout here is bandwidth-bound and runs the 0.6B talker step in 15.8 ms instead of 36.9 ms on an 8 Elite (fp16, 256-token cache). The code predictor step is 2.25 ms, so talker plus its 15 sub-steps cost about 50 ms of an 80 ms frame. sanitize_onnx() fixes the three things the dynamo exporter emits that qairt-converter and AI Hub reject: IsNaN/Where guards, Reshape allowzero=1, and rank-0 axes initializers on Unsqueeze/Squeeze. The module docstring records the measured placement table: this graph belongs on the NPU, the vocoder on the GPU (21.7 ms vs 182 ms per frame), which is 3.3x better than sending everything to the NPU and makes the pipeline real time on the phone in fp16. Integer quantization is not the lever it looks like: every automatic int8 build measured was numerically broken. Full write-up, job ids and profiles: analysis/mobile_htp_simulation.md section 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzr1Jusv4s6g765gqS9L92 |
||
|
tzhouamandClaude Fable 5.1
|
6ce8de1b28 |
[Edge][CPU] KV-cached code predictor, int8-dynamic CPU fast path, model-side CPU sampler
Measured on a Xeon 8480C (12+4 threads, 0.6B): the 15 residual-codebook sub-steps re-ran the 5-layer predictor over the whole prefix each time (97 ms per frame of the 134 ms talker step) and vLLM's stock sampler spent 9 ms per step in its torch.compile'd random-sample kernel on CPU. - code predictor: forward_cached (KV cache per layer, 2-token prefill then one token per sub-step) and a shape-static forward_static_step (tensor position + boolean mask) that torch.compile handles as one graph; incremental decoding is the default on CPU (VLLM_OMNI_PREDICTOR_INCREMENTAL), optional compile (VLLM_OMNI_PREDICTOR_COMPILE=step), optional thread cap. - CPU fast path: dynamic int8 Linear layers (fp32 activations) for the predictor and its lm heads (VLLM_OMNI_PREDICTOR_INT8, default on; micro-benchmark 70 -> 36 ms per frame for the 15 sub-steps); VLLM_OMNI_PREDICTOR_PARITY_CHECK=1 records top-1 agreement / max logit diff against the bf16 model in situ. - Qwen3-TTS talker: prefer_model_sampler on CPU with a sample() that mirrors vLLM's Sampler (logits processors + penalties, temperature, top-k/top-p, exponential-noise sampling with per-request generators) without the compiled kernel (VLLM_OMNI_CPU_FAST_SAMPLER, default on). - step_stats: sampler phase timers and sample-metadata timers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
tzhouamandClaude Fable 5.1
|
abf81f4114 |
[Edge][WP8] Model-level step timers: talker forward, sampling, talker-MTP, code-predictor sub-steps
Counters model.forward_ms / model.sample_ms / model.talker_mtp_ms and predictor.substep_ms / predictor.loop_ms (VLLM_OMNI_STEP_STATS_DIR); with VLLM_OMNI_STEP_STATS_SYNC=1 the accelerator is synchronized around the model timers so async kernels are attributed (attribution runs only). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ |
||
|
tzhouamandClaude Fable 5.1
|
580a201631 |
[Edge][WP4] Windowed Code2Wav decoder export (torch.export / ExecuTorch) + client backend; results summarizer
vllm_omni/edge/decoder_export.py exports a stateless windowed decoder (context 72 frames + chunk, one program per ramp chunk size) via torch.export (.pt2, verified here on CPU for chunks 2 and 25 with the real 0.6B decoder: exact match vs eager, 460 MB fp32 per program) or ExecuTorch XNNPACK (.pte; written against the executorch 1.x API, not executed: package not installed, install is permission gated; QNN partitioner export-only/unverified). The WP5 client gains --decoder-backend executorch --pte-manifest for phone/NPU targets. benchmarks/tts/edge_results/summarize.py renders all committed JSON as tables. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
tzhouamandClaude Fable 5.1
|
37c0ea9a6b |
[Edge][WP8] Per-step overhead attribution, edge scheduling helpers, CPU thread sweep
- vllm_omni/edge/step_stats.py: process-local counters enabled by VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped per process and merged into an attribution table (share of the 80 ms frame). Hooks: stage_engine_core_proc (instrument after construction, dump at exit), chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime), orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats. - vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and the no-preemption KV admission rule (exact bytes/token from the talker KV geometry); adapt.derive_overrides now sizes max_num_seqs with it and auto_profile passes the geometry read from the model config. - Engine re-reads parallel_stage_init after apply_deploy_profile so the edge profile's orchestrator default takes effect (was read before the profile). - benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor) with per-step attribution. - Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py, tests/benchmarks/test_cpu_thread_sweep.py, profile default test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
tzhouamandClaude Fable 5.1
|
55192de984 |
[Edge][WP4] Add hardware adaptation: probe, derived overrides, auto profile, calibration
* vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch (ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores, big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu / npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests). * vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from available memory minus weights, max_num_seqs, per-stage disjoint CPU thread ranges on the big cluster, eager/graph switches for small or unified memory, adaptive-chunk parameters, timeouts from a load cost model. * vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT. * vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml -> edge/hardware/<class>.yaml -> derived overrides -> cached calibration, materialized as a temp deploy YAML (platform overlays are superseded). * benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY, taskset, profile overrides) driving stream_latency_bench with auto profiles. * CPU-only tests for probe snapshots, derivation rules, calibration and auto. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |