This module carried a retraction saying the measured saving from right-sizing
the cache was 188 MB. That retraction was itself measured through a path that
ignored the knob: CpuPlatform.check_and_update_config overwrote
kv_cache_memory_bytes from VLLM_CPU_KVCACHE_SPACE unconditionally, so the
"32x shrink" never happened and both arms ran the same cache.
With the budget actually honoured, 3 interleaved passes per arm, both filling
a 1536-token context:
1 GiB budget 3264 MB resident
192 MiB budget 2421 MB resident -> 843 MB
And the reasoning behind "the pages are never touched" is wrong too. mincore(2)
reports the cache fully in core: 1024.1 MB resident of 1023.8 allocated, and
191.1 of 190.8 when sized down. It is not address space, it is memory.
843 MB is the largest single lever measured on this model -- more than the
single weight layout (615 MB), the blocked repack (489 MB peak), or the RoPE
table (618 MB).
The general lesson, recorded in the report: a lever tested through a path that
silently ignores it measures nothing, and "no effect" is the most dangerous
result to accept without first checking the knob was connected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
618 MB of resident memory and 649 MB of peak, for one line -- the largest
single memory item found in this study, and unlike the others it costs nothing.
Spark advertises a 1,048,576-token context. This file passed
max_position=config.max_position_embeddings straight into get_rope, and vLLM
builds its cos/sin table with torch.arange(max_position), so a 2048-token
deployment carried tables for a million positions. Because the sliding and full
layers are trained with different theta and different partial rotary factors
there are two of them:
layers.0 (full, head_dim 256) [1048576, 256] 512 MB
layers.3 (sliding, partial 0.25) [1048576, 64] 128 MB
Both fully resident -- confirmed with mincore(2), not inferred. At
max_model_len 2048 they need 1.25 MB between them.
Found by memory_ledger.py, which attributes memory instead of subtracting
toward a remainder: 640 MB had been sitting in an unnamed "other" bucket across
114 tensors, and naming the largest storages showed what it was. After the cap
that bucket is 1.3 MB and unique tensor storage falls 638.8 MB, which agrees
independently with the end-to-end measurement.
vLLM's scheduler cannot produce a position beyond max_model_len, so covering
that is sufficient. Measured with VLLM_OMNI_SPARK_ROPE_FULL toggling the two in
identical code, 3 interleaved passes per arm on node-local cores:
1M tables settled 3891 MB peak 4019 MB init 13.5 s
capped settled 3273 MB peak 3370 MB init 13.7 s
The escape hatch restores the old behaviour and is covered by a test, as is the
cap itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The review caught weight_path.py claiming tinygemm is "the only option" for an
x86 core without AMX. That was false -- vLLM 0.28 selects ops.cpu_gemm_wna16
whenever layer.use_w4a8 is off -- and the honest response was to measure the
kernel, not to reword the claim.
Same GPTQ checkpoint, only VLLM_CPU_INT4_W4A8 differing, isolated compile
caches, 3 interleaved passes on 24 pinned node-local cores:
int4_scaled_mm_cpu (W4A8, AMX) 74.4 tok/s spread 1.1%
tinygemm W4A16 (same cores) 71.6 tok/s
cpu_gemm_wna16 32.3 tok/s spread 1.5%
The fallback is real but 2.2x slower than tinygemm, so the selector's routing
was right for a reason nobody had checked. It stays as it is; what changes is
that the module now cites a measurement rather than an assumption.
Recorded with the caveat that matters for reading it off-x86: _get_isa_hint
returns "amx" whenever AMX is present and activations are bf16, regardless of
the W4A8 flag, so this measured cpu_gemm_wna16 on its AMX path -- its best case
on this host. The "vec" path a genuinely AMX-less machine takes is untested and
presumably no faster, which only strengthens the conclusion, but no number is
claimed for it.
Also adds VLLM_OMNI_CPU_INT4_REPACK_ROWS so the blocked repack can be A/B'd
against whole-matrix packing in identical code. Unlike the prefill-path switch
this is load-time only and yields identical tensors, so it does not perturb the
compiled graph or its cache key.
tests/edge: 8 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
I0 of the reviewed plan: the peak, not the settled figure, is what an edge
device has to survive. Measured earlier this session, W4A16 loads at 5722-5837
MB and settles near 4480 after the allocator is asked for its arenas back -- a
1.2-1.3 GB gap, and a device that cannot hold the peak never reaches the
release.
A large part of that gap is in our own repack path. unpack_nibbles expands
4-bit codes to int32, which is 4 bytes per weight and eight times the packed
form, and the kernel's packer wants int32. Done for a whole matrix that
transient is the largest object in the process:
gate_up 13312 x 2048 104 MiB
down 2048 x 6656 52 MiB
q_k_v 3072 x 2048 24 MiB
embedding 131072 x 2048 1024 MiB <- tied, so it is also the head
against 24 MiB of packed weights for an entire layer. The embedding alone is a
~1 GB spike, which is about the size of the whole peak-vs-settled gap.
The packed layout is [N/64][K][32 bytes], so n-blocks are independent and the
expansion can cover one chunk at a time. Verified byte-identical to packing the
whole matrix at 13312x2048, 2048x6656, 3072x2048 and 131072x2048, plus a ragged
last chunk. The int32 transient drops from 1024 MiB to 4 MiB for the embedding
and from 104 to 4 MiB for gate_up. to_split_halves is chunked the same way for
the fallback path.
Not yet claimed: the end-to-end peak reduction. The arithmetic says ~1 GB, but
this study has been wrong often enough about memory that the number goes in
only after RssAnon is sampled through a real load. Tests: 21 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docstring said one layout costs 1.9x prefill. That came from the single
surviving pass of a grid whose other passes had crashed on the AOT cache
collision -- a number the harness itself had labelled "single pass, not a
result", which I then published.
Re-measured properly: caches isolated per arm so all passes complete, on idle
node-1 cores with memory bound to the same node, pooled over 7 interleaved
passes per arm. pp512 goes 2190 -> 1344 tok/s, i.e. 1.63x, not 1.9x.
The ratio is only good to about 1.4-1.8x (worst pairing 1.84x, best 1.36x)
because both dequantizing arms swing 10-18% here, while the tinygemm-only arm
repeats to 0.3%. The existence and rough size of the cost is solid -- a 63%
gap is far outside that band -- but the third digit is not, and the docstring
now says so rather than implying precision it does not have.
The comparison that is reliable is against prefill_dequant: false, whose arm is
the steady one: 2.97x its prefill at the same memory, without routing multi-row
batches into the tinygemm/AMX SIGILL hazard. That setting stays dominated.
Also corrected a hypothesis of my own. The first three direct passes read
1344 -> 1421 -> 1555 and I called it a warm-up trend; four more passes gave
1271, 1335, 1443, 1309, so it was noise and three ascending points out of three
at 12% spread meant nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The W4A16 path kept two packed weight layouts because the tinygemm layout was
treated as opaque: the prefill path needed something row-major to dequantize
from, so `_finalize_int4` built `weight_halves` plus cloned scale/zero beside
the kernel's own buffer. That duplicate is the 740 MB the report attributes to
"the dual weight layout".
The layout is not opaque. Measured on this host by packing an index pattern and
reading back where each nibble landed: it is [N/64][K][32 bytes] -- K is the
middle axis, so a K range is contiguous inside each n-block -- and within every
32-byte run, byte d carries output channel d in its low nibble and d+32 in its
high one, the same permutation for every k and every block. Decoding it is a
deinterleave. _dequantize_from_int4pack does that, bit-exact against the old
dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048, and the
duplicate is no longer allocated.
Measured with the two paths toggled in otherwise identical code, 3 arms x 2
interleaved passes, RssAnon summed over the process tree, 1536-token context:
615 MB of resident memory and ~915 MB of peak, landing statistically
indistinguishable from prefill_dequant: false (the harness refused to rank
them, 0.4% apart inside the spread). Config arithmetic predicts 764 MB; ~150 MB
is allocator retention malloc_trim does not return.
It is not free, and the docstring says so. The duplicate was not only a copy:
to_split_halves de-interleaves once at load, so each prefill call read two
contiguous planes, while this redoes the de-interleave every call. pp512 drops
2272 -> 1210 tok/s. Dequantizing to [K, N] and using matmul to dodge the
transpose was tried and is worse (20.3 ms vs 14.4 per gate_up at 8 threads);
recovering the prefill wants a C++ de-interleave, not an ATen expression chain.
So it is a dial. Against prefill_dequant: false it is strictly better -- same
memory, 2.7x the prefill, and dequant_threshold stays in force so multi-row
batches never reach the tinygemm kernel and the documented SIGILL hazard. That
setting is now dominated. Against keeping both layouts it trades 615 MB for
1.9x prefill; default on, because this path exists for devices chosen for
their memory ceiling.
supports_direct_dequant gates the reshape on N tiling the 64-channel block,
which every Spark linear does (2048, 3072, 13312, 131072). An N like 80 packs
as 64 + a 16 tail the reshape would mis-read, so those layers keep the copy
rather than being quietly wrong.
The choice is a `direct_dequant` field of the checkpoint's quantization_config,
not only an env var, and that matters: the two paths build different graphs
(one passes weight_halves into the custom op, the other passes None), so they
must not share a compiled artifact. ModelConfig.compute_hash hashes config
fields, so a config field reaches vLLM's compile-cache key; an env var does
not. Toggling the env var against a warm cache loaded the wrong graph and
killed three benchmark passes with `KeyError: 'weight_halves'` inside the
AOT-loaded forward -- found because the harness records failures instead of
dropping them. VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT remains as an experiment
override, documented as needing its own VLLM_CACHE_ROOT.
Tests: 18 passed, including bit-exactness at three shapes, a pin on the packed
layout itself (so a torch release that repacks differently fails loudly rather
than returning wrong weights), the N=80 fallback, and both paths producing
identical output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An independent review (gpt-6-astra via the Codex hook) found that two modules
still ship claims this study had already withdrawn or that the vLLM source
contradicts. Documentation in executable files is how a retracted number gets
quoted again, so these are corrected at the source:
kv_budget.py claimed right-sizing the cache saved 8329 -> 5254 MB. That was
read from VmRSS (which counts the mmap'd checkpoint and moved 1-2 GB between
identical runs) and the two arms were different model files. On RssAnon the
saving is 188 MB. The retraction is now stated in the docstring, together with
why the number is small -- the benchmark generated 8 tokens, so no cache was
filled -- and the 1323 MiB flat-vs-hybrid difference at 32k that makes this
worth re-measuring on a benchmark that does fill it.
weight_path.py:
- bits_per_weight 4.16 -> 4.25 in all three 4-bit paths. 4 bits plus a bf16
scale and zero per 128-weight group is 4 + 32/128; 4.16 matched no
arithmetic and made the llama.cpp fidelity comparison look budget-matched
when it is not (Q4_K is 4.5 bits over groups of 32, Q4_K_M promotes some
tensors further).
- fused_cpu_norms True -> False in all three. Fused CPU RMSNorm is 3.6-4.5x
faster in isolation and slower in situ; VLLM_OMNI_CPU_FUSED_NORMS already
defaults to "0" for that reason, so the metadata contradicted the default.
- "tinygemm is the only option without AMX" is false: vLLM 0.28 selects
ops.cpu_gemm_wna16 whenever use_w4a8 is off (mixed_precision/cpu.py:202).
Unmeasured here, so the selector is unchanged, but the claim is not.
- W4A8's symmetric grid is our exporter's "sym": true, not a kernel limit --
_process_gptq_weights_w4a8 uses checkpoint zero points when
config.zero_points is set (same file, :98). An asymmetric export is a config
change and the cheapest untried fidelity experiment.
- kv_budget's flat-accounting observation is flagged for re-checking, since
CpuPlatform.support_hybrid_kv_cache() returns True (platforms/cpu.py:545).
Both fields now carry docstrings saying what they mean, so they cannot drift
back silently. tests/edge: 18 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The W4A16 path keeps two weight layouts -- the tinygemm one decode reads and a
row-major copy prefill dequantizes -- and the docstring said only that it cost
"8 bits per weight resident". Measured, median of three interleaved passes on
anonymous RSS (spread under 1.5%, where VmRSS moved 1-2 GB between identical
runs):
prefill_dequant resident prefill decode
true (default) 3542 MB 3252 tok/s 76.9 tok/s
false 2802 MB 448 tok/s 78.5 tok/s
So it is 740 MB for 7.3x prefill. Default stays on, because losing 7x of
prefill hurts more than 740 MB on anything with memory to spare -- but on a
device where it does not, this is the smallest resident build in the tree:
2.8 GB against the 7.3 GB the W4A8 path started at before the load-time
allocator release.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`vllm_omni/deploy/edge/qwen3_tts.yaml` carries `orchestrator:
parallel_stage_init: true`, but `_apply_profile_orchestrator_defaults` ran
only on the `deploy_profile` path. Pointing `--deploy-config` at the same file
loaded its stages and silently dropped the block, so the edge deployment
started sequentially:
--deploy-profile edge 30.76 s
--deploy-config .../edge/qwen3_tts.yaml 63.76 s (same file)
with nothing in the log to say why. The block now applies however the config
was reached. Explicit arguments still win, and the allow-list stays narrow --
a test checks an unrecognised key cannot be injected through it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`parallel_stage_init` was structurally impossible on a CPU deployment.
`resolve_stage_physical_devices` returns None for `devices: "cpu"`, and the
admission resolver fails closed on None -- correctly, for a GPU replica whose
devices cannot be accounted -- so every CPU pipeline raised
StageAdmissionError and fell back to sequential stage startup. The edge
profile sets parallel_stage_init: true, so that profile was broken on CPU
rather than merely slower.
Measured on Qwen3-TTS 0.6B (Xeon 8480C, 24 threads, init only):
sequential 64.57 s
parallel 30.54 s 2.1x
The fix is not to skip admission. The per-device ledger is vacuous on CPU --
there is no device to overcommit -- but the resource concurrent CPU init
contends for is host memory, and two stages allocating KV caches at once can
exhaust it exactly as two GPU stages exhaust a card. So there is now a host
ledger with the same discipline: sum the declared KV budgets, compare against
available RAM with a margin, refuse to start if it does not fit, and fail
closed on a stage that declares no budget at all. The arithmetic is pure
(`evaluate_host`) so it is testable without a device, matching the existing
design.
Which ledger applies is decided by where the stages are *placed*, not by what
platform the process reports: a first version keyed on
`current_platform.is_cpu()` and broke five existing GPU-path tests, because a
CPU-only test host still has to admit GPU-shaped plans correctly. It now
routes on the replicas' own `devices`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Driving a phone through the Android agent harness turns section 2c's fidelity
numbers into a task outcome. Same simulated device, one run per cell:
build decode "Turn on Wi-Fi." "Set an alarm"
bf16 41.0 tok/s done in 4 steps wrong app, stuck
W4A16 g128 80.0 tok/s done in 4 steps right app, no completion
W4A8 g128 86.6 tok/s turns it on, then off (x2) reaches Alarm, loops
Every build navigates home -> Settings -> Network -> Wi-Fi correctly. Only
some notice the switch now reads checked=true and call task_complete. W4A8 --
the fastest build, and the one the speed-first path recommends -- ends with
the task undone, on both runs.
So "W4A16 reproduces 2/12 greedy prompts and W4A8 1/12" was not a small
difference on a proxy metric: it is finishing versus reversing. The selector
now says so where someone choosing a build will read it, and points at
priority="fidelity" for agentic work.
Two tasks, one run per cell: enough to show the failure reproduces, not to
rank success rates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two corrections and the paperwork they imply.
The fused-RMSNorm registration defaulted to *on*, which contradicts the
measurement it was committed with: those kernels are 3.6x and 4.5x faster in
isolation but measured 72.7 tok/s against 74.0 end to end, because inductor
already fuses the native norm into the residual add and an opaque call breaks
that fusion. The implementations stay registered, so the provider remains
selectable through `kernel_config.ir_op_priority` on a machine where the trade
lands differently; only the default ranking is left alone now, and the test
asserts both states rather than just the opt-in one.
The four switches this branch added (VLLM_OMNI_AOT_CACHE_SHIM,
_CPU_FUSED_NORMS, _CPU_RELEASE_MEMORY, _SPARK_GATE_REDUCTION) were not in the
reviewed environment-variable inventory, so
`test_statically_resolvable_environment_accesses_are_classified` failed. They
are classified now, the reviewed count moves 28 -> 32 with a note saying why,
and each has a row in docs/configuration/environment_variables.md carrying the
measurement that justifies its default.
`VLLM_OMNI_CPU_FAST_SAMPLER` and `VLLM_OMNI_STEP_STATS_SYNC` remain
unclassified and that test still fails on them. They arrived in 6ce8de1b, in
files this branch does not touch, so they are left to whoever owns that work
rather than documented from a guess.
Under `-m "core_model and cpu"`: 2593 passed before these fixes with 9
failures, of which two were mine. The rest are CUDA-only tests in a CPU venv
("Torch not compiled with CUDA enabled") plus test_qnn_export.py, which
another session is actively editing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Loading a quantized checkpoint allocates far more than the weights occupy.
`_process_gptq_weights_w4a8` unpacks 4-bit codes to int32 -- four bytes per
weight -- permutes, repacks, and drops the intermediates. For a 1.7 B model
that is gigabytes of short-lived allocation, correctly freed, and still
resident afterwards because a caching allocator keeps the arenas.
Measured on Spark-X2.5-1.7B, anonymous RSS:
path after load after release freed
W4A8 (GPTQ) 7338 MB 3510 MB 3828 MB (-52%)
W4A16 (tinygemm) 5023 MB 3411 MB 1612 MB
The second column is the more useful finding: after the release both paths
land in the same place, so W4A8's apparent 2.3 GB penalty over W4A16 was never
a footprint difference -- it was the repack. W4A8 is therefore faster *and*
level on memory, and fidelity is the only remaining reason to pick W4A16.
Hooked to CPUWorker.load_model and compile_or_warm_up_model, since those churn
separately; `VLLM_OMNI_CPU_RELEASE_MEMORY=0` disables it. CPU only: on GPU the
weights are in device memory and host churn constrains nothing.
`anon_rss_bytes()` ships alongside because the metric matters as much as the
fix. VmRSS counts the mmap'd checkpoint's file pages and moved 1.2-2.0 GB
between *identical* runs on this host; RssAnon repeats to within 0.5 MB. An
earlier claim in this tree that KV sizing saved 3.1 GB came from reading VmRSS
across a comparison that also changed the model file; re-measured on RssAnon,
shrinking the KV cache 32x is worth 188 MB, because its pages are never
touched. The analysis note is corrected accordingly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`CpuPlatform.check_and_update_config` ends with a block whose own comment
reads "Lagecy setting":
env_key = "VLLM_CPU_KVCACHE_SPACE"
if env_key in os.environ and os.environ[env_key] != "":
cache_config.kv_cache_memory_bytes = int(os.environ[env_key]) * GiB
It runs unconditionally, so `--kv-cache-memory-bytes` is silently discarded
whenever the env var is set -- which it has to be on CPU, or the engine tries
to claim most of the machine. Two consequences for edge serving: the budget
can only be expressed in whole GiB (parsed with `int()`, so the smallest
non-zero cache is 1 GiB), and vllm-omni's per-stage `kv_cache_memory_bytes`,
which is how the deploy YAMLs express per-stage budgets, has no effect at all
on CPU.
Verified by allocation rather than by memory, because allocation cannot drift:
with the env at 1 GiB, asking for 128 MiB produced the same 17 881 cache
tokens as asking for nothing, and with the env at 4 GiB it produced 71 527 --
the explicit value never applied. After restoring the obvious precedence
(explicit wins, env stays the default for callers that set nothing) the same
request yields 2 231 tokens, the 128 MiB it asked for.
The Qwen3-TTS CPU overlay now sizes stage 0 from the model -- 112 KiB/token,
so 2 seqs at 2048 tokens needs 560 MiB, not 4 GiB. Stage 1 is left alone on
purpose: Code2Wav is a generation stage whose cache is not a plain attention
KV, and under-sizing makes vLLM preempt by recompute, which on a single-stream
edge deployment is a stall.
What this saves in resident memory is deliberately *not* claimed. VmRSS on
this host moves 1.2-2.0 GB between identical runs, because it counts the
mmap'd checkpoint's file pages; an earlier 3.1 GB figure came from a
comparison that changed the model file as well as the budget and does not
stand. RssAnon is the metric to use and that measurement is still in flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The GPTQ checkpoint the comparison had been using left Spark's output head
tied and unquantized, so that one GEMM ran bf16 through `_C::onednn_mm` at
2.97 ms/step -- more than the whole 4-bit saving, and enough to make the
W4A8/W4A16 ordering flip between grids. The per-op profile is what caught it:
`int4_scaled_mm_cpu` ran 112 times (28 layers x 4 linears) and `onednn_mm`
once.
Rebuilt with the head quantized and re-profiled back to back on the same 24
cores, because this host's throughput swings ~30% and the gap is ~8%:
w4a8 AMX g128 GEMM 5.91 ms step 11.55 ms 86.6 tok/s
w4a16 tinygemm GEMM 6.85 ms step 12.51 ms 80.0 tok/s
llama.cpp Q4_K_M step 11.11 ms 90.0 tok/s
W4A8 wins twice over: the kernel is 0.94 ms cheaper per step, and it is
reached through vLLM's own dispatch, so it also avoids the 0.48 ms/step
custom-op boundary the tinygemm path needs to send prefill to a different
kernel. The selector's recommendation is unchanged; only its numbers were
wrong, and the lm_head trap is now documented where someone building a
checkpoint will read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`VLLM_CPU_KVCACHE_SPACE` is a server-shaped default and is parsed with
`int()`, so it cannot express an edge-sized cache at all -- the smallest
non-zero value it accepts is 1 GiB. Measured on Spark-X2.5-1.7B (Xeon 8480C,
2048-token single stream), passing `kv_cache_memory_bytes` sized from the
model instead of the 4 GiB default takes resident memory from 8329 MB to
5254 MB, a 3.1 GB saving with no effect on output.
`kv_budget_for` reads the config and prices the cache. It reports two numbers
because the model and the backend disagree: a hybrid-attention model does not
need a full-length cache in every layer -- Spark is 21 sliding (window 512)
to 7 full layers, so at 32 k context its real cache is 3.8x smaller than
`layers x context` -- but vLLM's CPU path does not take that discount today
(for a 4 GiB budget it reported 71 527 tokens, i.e. the flat 56 KiB/token).
So `bytes_total` is the flat figure, which is what is safe to pass now, and
`bytes_total_hybrid` is reported beside it for a backend that accounts per
layer type.
Deliberately conservative in three places, because a budget that is too small
is not a graceful degradation -- vLLM preempts by recompute, which on a
single-stream edge deployment is a stall: the window is rounded up to the
block size, a 1.25 headroom factor is applied, and a config that has a
`sliding_window` but no `layer_types` map is charged the flat rate rather than
assumed to apply the window everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four changes, each measured on a Xeon 8480C with Spark-X2.5-1.7B at 24
threads. Numbers are best-of-interleaved-passes; this host is shared and a
single sequential grid swung the llama.cpp control 30%, so anything under
~8% is not treated as a result.
**vLLM's CPU backend does have a 4-bit kernel.** An earlier note in this
tree said it did not, on the strength of `woq_int4_linear`,
`int4_scaled_mm_with_quant` and `da8w4_linear` all being absent. Those are
simply not the names it uses: `CPUWNA16LinearKernel` dispatches GPTQ/AWQ
checkpoints to `int4_scaled_mm_cpu`, an AMX W4A8 GEMM, whenever activations
are bf16 and the checkpoint carries no act-order. Four-bit weights times
eight-bit activations is the arithmetic llama.cpp's Q4_K_M already does.
`gptq_cpu_export` builds checkpoints for it; the kernel pins the zero point
at 8, so the grid must be symmetric, and a per-group scale search recovers
most of what that costs.
**A per-device weight path** (`edge/weight_path.py`). The fast kernel on each
device needs a different, mutually exclusive weight layout, so the choice has
to come from a probe rather than a flag: AMX x86 -> W4A8/GPTQ; x86 without
AMX, or priority="fidelity" -> W4A16/tinygemm (bf16 activations, the more
faithful of the two); ARM -> W4A16 but flagged unverified, because PyTorch's
4-bit CPU GEMM dispatches a different kernel there than the AVX-512 one
measured here; phone -> runs_in_vllm=False, the deliverable is an exported
graph with calibrated w8a16.
**The CPU compile cache never loaded.** `_try_load_aot_compiled_fn` finishes
with `compiled_fn.finalize_loading(cfg)`, which the CUDA wrapper object has
and the inductor CPU backend's plain function does not. The AttributeError
was swallowed as a cache miss, so every engine start recompiled the model:
warm start 49.8 s against 14.9 s with the missing no-op attached. Self-
disabling if upstream gives the CPU path a real one.
**The head gate stops being a GEMM.** A 2048x8 projection cannot amortize a
thread fan-out: dispatched as `aten::mm` it costs 1.155 ms of a decode step,
nearly all barrier. As an eager reduction behind an opaque op it costs 0.444
ms. It has to be opaque -- written inline the reduction is traced, and
inductor's loop made the whole step 1.8x slower (78.8 -> 44.5 tok/s).
Also included and *off by default*: `cpu_ir_ops` registers the fused
`_C::rms_norm` / `_C::fused_add_rms_norm` kernels that vLLM's CPU build ships
but gates on `GPGPU_DEVICE`. In isolation they are 3.6x and 4.5x faster than
the native path; end to end they are slightly slower (72.7 vs 74.0 tok/s),
because inductor was already fusing the native norm into its neighbours and
an opaque call breaks that fusion. Kept with the measurement recorded so the
next person does not repeat it. `gelu_and_mul` is deliberately not registered:
its fused CPU kernel is 2.7x slower at Spark's GEGLU shape.
Decode, W4A16 g64 -> W4A8 g128 with the gate fix: 69.1 -> 84.5 tok/s (+22%),
against llama.cpp Q4_K_M's 90.0; prefill 3347 against 1121 (3.0x). The two
4-bit paths are not equally faithful: against each build's own bf16 over 12
greedy prompts, Q4_K_M reproduces 4/12, W4A16 2/12 and W4A8 1/12.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Single-stream CPU decode reads every weight to emit one token, so bits per
weight set the token rate. This registers a W4A16 group-wise method
out-of-tree through register_quantization_config -- vLLM itself is untouched
-- using PyTorch's _weight_int4pack_mm_for_cpu tinygemm kernel.
Two paths over one set of weights, because that kernel dequantizes once per
row of activations and so is linear in the batch (12.9x slower than bf16 at
512 rows). Single-row steps take tinygemm; wider batches unpack the weight to
bf16 once and use the ordinary oneDNN/AMX GEMM. Shipping only the first
measured pp512 = 557 tok/s against bf16's 4455; with both, 3589.
The batch threshold is 2 rather than the ~8 rows where the unpack path starts
winning on speed, because running tinygemm on a multi-row batch and then
vLLM's cpu_attention_with_kv_cache in the same forward dies with SIGILL
inside the attention kernel -- an AMX instruction with no valid tile config
loaded. The same prompt is fine in bf16 and fine once every multi-row GEMM
takes the unpack path.
Positioning, measured on a Xeon 8480C with 32 pinned cores and a warm cache
(analysis/spark_cpu_int4.md): vLLM's own GPTQ W4A8 AMX path is faster --
88.7 tok/s decode and 4355 prefill against 85.5 and 3589 here -- but its
config rejects sym=False at 4 bits, so it can only be fed a symmetric grid.
This one is asymmetric and costs +12.5% perplexity against that path's
+27.2%, and needs no AMX. llama.cpp Q4_K_M, for reference, is 98.0 tok/s
decode and 1356 prefill at +7.6% and 5.15 bits per weight.
Expanding a quantized checkpoint back to bf16 offline and scoring it
reproduces the quantized run to within 0.2% perplexity, so both GEMM paths,
the nibble packing and the quantized tied embedding are faithful.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXhbAfkSNRaeqZrus5Z2fx
The score-space join removed the concat as intended and is numerically
identical (140.3 dB), but it cannot be quantized: replacing torch.softmax
with an explicit max/exp/sum/div made QNN's context-binary converter exit 14
under w4a16, while every fp16 build compiled. Two earlier shapes failed the
same way -- a broadcast multiply plus ReduceSum for the new token's score,
and a two-axis broadcast multiply for its value contribution.
The device profile says the win was never the concat anyway. Of a sliding
layer, Output was 9.9%, its quantise conversion 3.1% and the window Slice
7.4% -- all of it the cost of handing a 512-entry window back across the
graph boundary once per token. So keep roll's attention exactly as it was,
one fused Softmax over concatenated keys, and change only what the layer
returns: the single new K/V entry, which the runtime drops into the ring
slot. Ring order needs no fixing up, because softmax over keys is
order-independent and each cached key already carries its own rotary phase.
Measured on a real Galaxy S25, sliding layer, ms:
w4a16 0.917 -> 0.838 (-8.6%) <- now compiles
fp16 1.689 -> 1.486 (-12.0%)
Full-attention layers keep roll: ring gains them nothing (1.567 vs 1.559 at
fp16, the concat saved being spent on the extra score matmul), so the
default layout is now per layer type.
Parity against the reference is 140.3 dB for the mixed default with
identical top-1/top-5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Reading the device profile of the first export showed a fifth of every
sliding layer went nowhere useful: Output 9.9%, its quantise conversion
3.1%, and the window Slice 7.4%. The layer took a 512-entry window in,
appended the new key, sliced the oldest off and handed the whole recomputed
window back out -- 21 of 28 layers paying to move a cache-sized tensor
across the graph boundary once per token. Full layers paid a similar 9.5%
concatenating the new key onto the cache.
Both are the same mistake, and both go away by joining in score space
instead of key space: score the cache and the new token separately, then
concatenate two vectors of length n and 1 rather than an n x 256 key matrix.
Nothing cache-sized is copied, and each layer returns just the new K/V entry
for the runtime to drop into its ring slot. Softmax over keys does not care
what order the slots are in and each cached key already carries its own
rotary phase, so a ring buffer needs no reordering -- only a mask for slots
not yet written. The sliding cache correspondingly holds window-1 entries,
since a 512-token window counts the current token (transformers' own sliding
cache holds 511 for the same reason).
Measured on a real Galaxy S25, fp16, ms per layer:
sliding 1.689 -> 1.483 (-12.2%)
full 1.559 -> 1.567 (unchanged: the concat saved is spent on the
extra score matmul)
Two shapes had to be written as matmuls rather than the obvious elementwise
forms -- the new token's score (broadcast multiply + ReduceSum) and its
value contribution (a two-axis broadcast multiply). Both compile at fp16 and
both make QNN's context-binary converter exit 14 under w4a16; as matmuls
they compile and are the cheaper op on HTP anyway.
Also adds a selectable GELU approximation, and the measurement that closes
the question: erf GELU is 11-13% of every layer, but tanh is *slower* on
this NPU (1.535 vs 1.483 ms) despite holding 65.2 dB. The exact form stays,
which is also the better-fidelity choice.
Parity against the reference implementation is 136.8 dB with identical
top-1/top-5, and a test pins ring and roll to the same output on a shared
history so the device numbers cannot silently measure a different model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Exports one Spark decode step with the KV cache as explicit tensors, sized
per layer type: the 21 sliding layers hold a fixed 512 entries at any
context, the 7 full-attention layers grow. That asymmetry is the whole
reason the architecture is interesting on a phone, so the two types get
separately sized caches, masks and rotary angles.
Verified against the HF reference at 1024 context: 138.5 dB logits SNR,
identical top-1 and top-5.
Profiled on a real Galaxy S25 (Snapdragon 8 Elite) through AI Hub, QNN
context binary on the NPU, ms per layer:
sliding (512 KV, any ctx) fp16 1.689 w8a16 0.912 w4a16 0.917
full @ 1k fp16 1.559 w8a16 1.129 w4a16 1.127
full @ 4k fp16 2.964 w8a16 1.681 w4a16 1.662
Composed into a step (21 sliding + 7 full), w4a16 gives 27.2 ms at 1k and
30.9 ms at 4k, against 31.6 and 46.5 ms for the same model with every layer
full-attention -- 1.16x and 1.51x. Worth noting the hybrid is 0.94x at 1k in
fp16: sliding layers rotate all 256 head dims where full layers rotate 64,
and below roughly 2k that RoPE cost outweighs the shorter cache.
Also moves the head-wise attention gate off vLLM's CPU linear dispatch. The
gate is [2048, 8], and any linear that narrow is routed to the sgl-kernel
packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim
0; VNNI packing does not at that size, so it reports K and torch.compile
fails tracing the gate multiply. A plain parameter with F.linear gains
nothing less -- a 2048x8 matmul has no use for packed AMX tiles -- and it
keeps the model compilable on CPU and GPU alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Spark-X2.5-1.7B is an Apache-2.0 on-device model built for agentic tool
use. vLLM 0.28 has no implementation for it, so it could not be served at
all; llama.cpp already ships one, which made it the only edge engine able
to run the model.
Three traits separate it from a Llama-style decoder:
* Hybrid attention -- layer_types interleaves three 512-token sliding
layers with one full-attention layer, which is what makes the 1M-token
context affordable on a phone.
* Per-layer-type RoPE -- sliding layers rotate all 256 head dims with
theta 1e4, full layers only the first quarter with theta 5e6.
* A head-wise attention output gate -- g_proj reads the same post-norm
input as QKV and scales each head's attention output by a sigmoid.
The checkpoint also stores QKV pre-fused as a single q_k_v_proj, so
load_weights splits it before vLLM's sharded QKVParallelLinear sees it.
Verified against the official remote-code implementation on transformers
4.57.1 (the version config.json targets; it does not import on the 5.x in
our venvs). In fp32 all three scenarios match exactly -- 32/32 greedy
tokens with a max top-20 logprob difference of 2.2e-05. In bf16 the short
and the 2465-token long-context prompts still match 32/32, which is what
exercises the sliding layers; the tool-calling prompt diverges at token 23
on a near-tie, and fp32 confirms that is rounding rather than a defect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
The Code2Wav decoder already had an on-device export (WP4); the talker and its
code predictor did not, so the codec-token path stopped at the vocoder. This
adds the missing half: a one-token decode step with an explicit per-layer KV
cache, exportable to ONNX and from there to a QNN context binary, TFLite or
ONNX Runtime.
Two layout choices carry the performance, both measured on real Snapdragon
devices through Qualcomm AI Hub (Galaxy S24 = 8 Gen 3, S25 = 8 Elite):
* per-layer cache tensors instead of one stacked [num_layers, ...] tensor that
every layer slices, and
* grouped-query attention by reshaping the query rather than expanding keys and
values.
The stacked/expanded form spent ~70% of its NPU cycles in cache select,
reshape and fp32->fp16 conversion; the layout here is bandwidth-bound and runs
the 0.6B talker step in 15.8 ms instead of 36.9 ms on an 8 Elite (fp16,
256-token cache). The code predictor step is 2.25 ms, so talker plus its 15
sub-steps cost about 50 ms of an 80 ms frame.
sanitize_onnx() fixes the three things the dynamo exporter emits that
qairt-converter and AI Hub reject: IsNaN/Where guards, Reshape allowzero=1, and
rank-0 axes initializers on Unsqueeze/Squeeze.
The module docstring records the measured placement table: this graph belongs
on the NPU, the vocoder on the GPU (21.7 ms vs 182 ms per frame), which is 3.3x
better than sending everything to the NPU and makes the pipeline real time on
the phone in fp16. Integer quantization is not the lever it looks like: every
automatic int8 build measured was numerically broken.
Full write-up, job ids and profiles: analysis/mobile_htp_simulation.md section 5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dzr1Jusv4s6g765gqS9L92
Noise is drawn eagerly with the request generators (same RNG consumption as the
eager loop) and passed in, so the compiled post-step is generator-free and
shape-static; used on the fp32/torchao CPU path outside stored (top-p) mode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
In-situ parity against the bf16 model (VLLM_OMNI_PREDICTOR_PARITY_CHECK=1):
fp32 control keeps 93.8 % top-1 / 95 % top-5 / KL 0.0015; torch.ao dynamic
int8 falls to 22 % top-1 (max logit diff 7), so per-tensor activation
quantization is not acceptable as a default. Adds the ao-nohead variant
(int8 transformer body, fp32 lm heads) for measurement.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
torchao's dynamic-activation int8 is numerically wrong under torch.compile on
CPU (cosine 0.02 vs fp32; caused runaway generations), so it is not offered;
weight-only + compile is faithful but slower (78 ms vs 36 ms per frame for the
15 sub-steps), so torch.ao dynamic int8 stays the default. Parity mode now also
records top-5 overlap and KL vs the bf16 model; compiled-step equivalence test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Measured on a Xeon 8480C (12+4 threads, 0.6B): the 15 residual-codebook sub-steps
re-ran the 5-layer predictor over the whole prefix each time (97 ms per frame of
the 134 ms talker step) and vLLM's stock sampler spent 9 ms per step in its
torch.compile'd random-sample kernel on CPU.
- code predictor: forward_cached (KV cache per layer, 2-token prefill then one
token per sub-step) and a shape-static forward_static_step (tensor position +
boolean mask) that torch.compile handles as one graph; incremental decoding
is the default on CPU (VLLM_OMNI_PREDICTOR_INCREMENTAL), optional compile
(VLLM_OMNI_PREDICTOR_COMPILE=step), optional thread cap.
- CPU fast path: dynamic int8 Linear layers (fp32 activations) for the predictor
and its lm heads (VLLM_OMNI_PREDICTOR_INT8, default on; micro-benchmark 70 ->
36 ms per frame for the 15 sub-steps); VLLM_OMNI_PREDICTOR_PARITY_CHECK=1
records top-1 agreement / max logit diff against the bf16 model in situ.
- Qwen3-TTS talker: prefer_model_sampler on CPU with a sample() that mirrors
vLLM's Sampler (logits processors + penalties, temperature, top-k/top-p,
exponential-noise sampling with per-request generators) without the compiled
kernel (VLLM_OMNI_CPU_FAST_SAMPLER, default on).
- step_stats: sampler phase timers and sample-metadata timers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Counters model.forward_ms / model.sample_ms / model.talker_mtp_ms and
predictor.substep_ms / predictor.loop_ms (VLLM_OMNI_STEP_STATS_DIR); with
VLLM_OMNI_STEP_STATS_SYNC=1 the accelerator is synchronized around the model
timers so async kernels are attributed (attribution runs only).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Measured after the DELTA fix: 53 engine outputs for 52 rows, but the client
still received everything at once because each per-step payload is the newest
row only and the row alignment treated it as a cumulative snapshot (one yield
at the first row, one at the end). accumulate_codec_rows folds delta rows and
the final consolidated matrix into one growing matrix; unit test drives the
generator with per-step deltas and asserts one yield per step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The speech path forced FINAL_ONLY for Qwen3-TTS deployments without async
chunks, so a talker-only server emitted one engine output per request and all
codec frames arrived at once (measured: 1 engine output per request, first
codes at output #1). A talker-only stage (final_output + latent) now keeps
DELTA outputs so codec frames stream per step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- environment_variable_inventory: VLLM_OMNI_CPU_INDUCTOR / HW_PROFILE /
INIT_TIMELINE / STEP_STATS_DIR (public), STAGE_ID / INIT_TIMELINE_DIR
(internal); reference page rows added.
- edge_decoder_client.LocalDecoder: batched_chunked_decode takes a [B, Q, F]
tensor, not a list (the WP5 e2e run failed with a TypeError); the decoder
now also carries the ramp schedule attributes the Code2Wav stage sets.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
vllm_omni/edge/decoder_export.py exports a stateless windowed decoder (context
72 frames + chunk, one program per ramp chunk size) via torch.export (.pt2,
verified here on CPU for chunks 2 and 25 with the real 0.6B decoder: exact
match vs eager, 460 MB fp32 per program) or ExecuTorch XNNPACK (.pte; written
against the executorch 1.x API, not executed: package not installed, install
is permission gated; QNN partitioner export-only/unverified). The WP5 client
gains --decoder-backend executorch --pte-manifest for phone/NPU targets.
benchmarks/tts/edge_results/summarize.py renders all committed JSON as tables.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- vllm_omni/edge/step_stats.py: process-local counters enabled by
VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core
step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped
per process and merged into an attribution table (share of the 80 ms frame).
Hooks: stage_engine_core_proc (instrument after construction, dump at exit),
chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime),
orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats.
- vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and
the no-preemption KV admission rule (exact bytes/token from the talker KV
geometry); adapt.derive_overrides now sizes max_num_seqs with it and
auto_profile passes the geometry read from the model config.
- Engine re-reads parallel_stage_init after apply_deploy_profile so the edge
profile's orchestrator default takes effect (was read before the profile).
- benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor)
with per-step attribution.
- Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py,
tests/benchmarks/test_cpu_thread_sweep.py, profile default test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wraps SharedMemoryConnector: CUDA tensor leaves are exported with
torch.multiprocessing reduce_tensor + an inter-process CUDA event, rebuilt and
cloned by the consumer, released by the producer after an ack file appears.
CPU leaves and metadata keep the shm path. Registered as "CudaIpcConnector";
deploy/qwen3_tts_cuda_ipc.yaml selects it for the talker->code2wav hop.
Adoption in the edge profile is measurement-gated (hop latency bench).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* diffusers is no longer imported by vllm_omni.diffusion.data (annotation only);
vllm_omni.diffusion.io_support (diffusion registry + distributed state) is
imported lazily by config_factory and async_omni_engine, so constructing a
Qwen3-TTS pipeline config / AsyncOmni no longer loads diffusers,
x_transformers or whisper (guarded by tests/config/test_import_guard.py).
* requirements/common.txt = core.txt (AR/speech runtime) + media.txt
(diffusion/video/extra audio); VLLM_OMNI_EDGE_BUILD=1 makes setup.py use
requirements/<device>-edge.txt (core only); `edge` extra adds websockets and
sounddevice for on-device clients.
Measured on this host (CUDA venv): `import vllm` 7.0-7.9 s, `import vllm_omni`
adds ~20 s of which ~17 s is transformers' own import (transformers.utils.import_utils)
pulled in by vllm_omni.patch / vllm_omni.version; that floor is not
vLLM-Omni's and is unchanged. Guard test: no diffusion module loaded on the TTS path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
