One package that makes the POSIX assumptions measured in W0-W8 hold in
every process vLLM / vLLM-Omni starts, and reports what it did:
- provides fcntl (flock/lockf on msvcrt.locking; LOCK_SH served as
exclusive) and uvloop (asyncio.run on a selector loop) when absent;
- installs WindowsSelectorEventLoopPolicy and a signal.signal fallback
for loop.add_signal_handler on the Windows loop classes;
- replaces whole vLLM's get_open_zmq_ipc_path / find_loaded_library when
the wheel lacks the branch patch and the upstream body hash is known;
- verifies the wheel-only sites (sentinel poll, int64 seed) and reports
wheel_required at the exact site instead of failing downstream;
- shm_dir() replaces the /dev/shm literals; os.geteuid provided.
Activated from vllm_omni/patch.py and as vllm.general_plugins /
vllm_omni.general_plugins entry points (idempotent; no-op off Windows).
prints the audit; the M0 placement
report embeds it. 16 Linux tests through injected primitives.
Design: analysis/design_windows_compat_plugin_20260921.md
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
test_e2e_serve_text.py no longer imports tests.helpers.runtime (it pulls
the media/diffusion test helpers and therefore diffusers, absent from the
Windows venv); a small Popen launcher with a psutil process-tree kill
replaces OmniServer for this one test.
vllm-omni serve died at import on Windows: entrypoints/cli imports
benchmarks.patch -> benchmarks/omniinteract.py, which did 'import fcntl'
at module scope. Its output lock is rebuilt on the fork's
_filelock_compat primitive (fcntl on POSIX, msvcrt on Windows).
engine/stage_phase_lock.py (parallel_stage_init's SH/EX device locks)
gets the same treatment: fcntl optional, modes spelled locally, and on
Windows both modes map to the exclusive byte lock -- more serialising,
never less safe. Measured: the serving e2e's server exited with
ModuleNotFoundError: No module named 'fcntl' on both Windows venvs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pytest-asyncio builds its loops from the current policy; on win32 that is
the Proactor loop, which zmq.asyncio cannot use, so the engine/serving e2e
hung at the first async socket (30 min, no output, GPU idle) while the
accept CLI -- which installs WindowsSelectorEventLoopPolicy itself --
passed on the same venv. The conftest now installs the same policy and
provides it as pytest-asyncio's event_loop_policy fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Drives `vllm-omni serve <model> --omni` as a subprocess through the
online-serving launcher and checks /health, /v1/models, non-streaming
and streaming chat completions, and that a client disconnect mid-stream
leaves the server answering. Sets VLLM_USE_FLASHINFER_SAMPLER=0 like
LocalTextEngine does: the flashinfer sampler JITs with nvcc at runtime,
which a venv without a CUDA toolkit does not have (measured: the WSL
reference venv died in profile_run with 'Could not find nvcc').
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of
rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while
our 0.28-based tree needs seven modules v0.29.0 no longer has
(vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args,
openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant,
models.hunyuan_v1).
Ten files conflicted. Upstream's structure is the base everywhere; the
fork's features are re-attached to it:
- deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config
pops deploy_config (async_omni_engine, cli/serve); the deploy-profile
selector and the resolve_model_config_path chain it needs stay in
entrypoints/utils.py; stage_overrides takes rc1's dict type.
- config_factory keeps the lazy diffusion output-type import; the diffusion
registry import in rc1's new config/resolver.py is made lazy for the same
WP6 reason (TTS path must not load diffusion).
- requirements: our core/media split kept; rc1's transformers>=5.13 floor
and kernels==0.16.1 applied in core.txt / media.txt.
- TTS: rc1's split_granularity and bounded seed plus our output_mode=
codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation;
our private _conditioning_cache_salt is superseded by rc1's public
tts_adapters/base.py::conditioning_cache_salt.
- stage_runtime: rc1's single ExitStack G2 path (lock released before READY)
replaces our two-path block, with the four worker_phase names the init
profiler consumes still emitted; _cpu_placed / host admission kept.
Not yet run against vLLM 0.29.0: the tests listed in the W6 record.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Verbatim copy of the edge/omni-edge working tree (tracked diffs + untracked
edge/local, tests/edge/local, docs/edge, pipeline files). Taken so that
vLLM-Omni v0.29.0rc1 can be merged on a branch without touching the
user's working tree. logs_edge/ and benchmark .npy outputs excluded.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
618 MB of resident memory and 649 MB of peak, for one line -- the largest
single memory item found in this study, and unlike the others it costs nothing.
Spark advertises a 1,048,576-token context. This file passed
max_position=config.max_position_embeddings straight into get_rope, and vLLM
builds its cos/sin table with torch.arange(max_position), so a 2048-token
deployment carried tables for a million positions. Because the sliding and full
layers are trained with different theta and different partial rotary factors
there are two of them:
layers.0 (full, head_dim 256) [1048576, 256] 512 MB
layers.3 (sliding, partial 0.25) [1048576, 64] 128 MB
Both fully resident -- confirmed with mincore(2), not inferred. At
max_model_len 2048 they need 1.25 MB between them.
Found by memory_ledger.py, which attributes memory instead of subtracting
toward a remainder: 640 MB had been sitting in an unnamed "other" bucket across
114 tensors, and naming the largest storages showed what it was. After the cap
that bucket is 1.3 MB and unique tensor storage falls 638.8 MB, which agrees
independently with the end-to-end measurement.
vLLM's scheduler cannot produce a position beyond max_model_len, so covering
that is sufficient. Measured with VLLM_OMNI_SPARK_ROPE_FULL toggling the two in
identical code, 3 interleaved passes per arm on node-local cores:
1M tables settled 3891 MB peak 4019 MB init 13.5 s
capped settled 3273 MB peak 3370 MB init 13.7 s
The escape hatch restores the old behaviour and is covered by a test, as is the
cap itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
I0 of the reviewed plan: the peak, not the settled figure, is what an edge
device has to survive. Measured earlier this session, W4A16 loads at 5722-5837
MB and settles near 4480 after the allocator is asked for its arenas back -- a
1.2-1.3 GB gap, and a device that cannot hold the peak never reaches the
release.
A large part of that gap is in our own repack path. unpack_nibbles expands
4-bit codes to int32, which is 4 bytes per weight and eight times the packed
form, and the kernel's packer wants int32. Done for a whole matrix that
transient is the largest object in the process:
gate_up 13312 x 2048 104 MiB
down 2048 x 6656 52 MiB
q_k_v 3072 x 2048 24 MiB
embedding 131072 x 2048 1024 MiB <- tied, so it is also the head
against 24 MiB of packed weights for an entire layer. The embedding alone is a
~1 GB spike, which is about the size of the whole peak-vs-settled gap.
The packed layout is [N/64][K][32 bytes], so n-blocks are independent and the
expansion can cover one chunk at a time. Verified byte-identical to packing the
whole matrix at 13312x2048, 2048x6656, 3072x2048 and 131072x2048, plus a ragged
last chunk. The int32 transient drops from 1024 MiB to 4 MiB for the embedding
and from 104 to 4 MiB for gate_up. to_split_halves is chunked the same way for
the fallback path.
Not yet claimed: the end-to-end peak reduction. The arithmetic says ~1 GB, but
this study has been wrong often enough about memory that the number goes in
only after RssAnon is sampled through a real load. Tests: 21 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The W4A16 path kept two packed weight layouts because the tinygemm layout was
treated as opaque: the prefill path needed something row-major to dequantize
from, so `_finalize_int4` built `weight_halves` plus cloned scale/zero beside
the kernel's own buffer. That duplicate is the 740 MB the report attributes to
"the dual weight layout".
The layout is not opaque. Measured on this host by packing an index pattern and
reading back where each nibble landed: it is [N/64][K][32 bytes] -- K is the
middle axis, so a K range is contiguous inside each n-block -- and within every
32-byte run, byte d carries output channel d in its low nibble and d+32 in its
high one, the same permutation for every k and every block. Decoding it is a
deinterleave. _dequantize_from_int4pack does that, bit-exact against the old
dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048, and the
duplicate is no longer allocated.
Measured with the two paths toggled in otherwise identical code, 3 arms x 2
interleaved passes, RssAnon summed over the process tree, 1536-token context:
615 MB of resident memory and ~915 MB of peak, landing statistically
indistinguishable from prefill_dequant: false (the harness refused to rank
them, 0.4% apart inside the spread). Config arithmetic predicts 764 MB; ~150 MB
is allocator retention malloc_trim does not return.
It is not free, and the docstring says so. The duplicate was not only a copy:
to_split_halves de-interleaves once at load, so each prefill call read two
contiguous planes, while this redoes the de-interleave every call. pp512 drops
2272 -> 1210 tok/s. Dequantizing to [K, N] and using matmul to dodge the
transpose was tried and is worse (20.3 ms vs 14.4 per gate_up at 8 threads);
recovering the prefill wants a C++ de-interleave, not an ATen expression chain.
So it is a dial. Against prefill_dequant: false it is strictly better -- same
memory, 2.7x the prefill, and dequant_threshold stays in force so multi-row
batches never reach the tinygemm kernel and the documented SIGILL hazard. That
setting is now dominated. Against keeping both layouts it trades 615 MB for
1.9x prefill; default on, because this path exists for devices chosen for
their memory ceiling.
supports_direct_dequant gates the reshape on N tiling the 64-channel block,
which every Spark linear does (2048, 3072, 13312, 131072). An N like 80 packs
as 64 + a 16 tail the reshape would mis-read, so those layers keep the copy
rather than being quietly wrong.
The choice is a `direct_dequant` field of the checkpoint's quantization_config,
not only an env var, and that matters: the two paths build different graphs
(one passes weight_halves into the custom op, the other passes None), so they
must not share a compiled artifact. ModelConfig.compute_hash hashes config
fields, so a config field reaches vLLM's compile-cache key; an env var does
not. Toggling the env var against a warm cache loaded the wrong graph and
killed three benchmark passes with `KeyError: 'weight_halves'` inside the
AOT-loaded forward -- found because the harness records failures instead of
dropping them. VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT remains as an experiment
override, documented as needing its own VLLM_CACHE_ROOT.
Tests: 18 passed, including bit-exactness at three shapes, a pin on the packed
layout itself (so a torch release that repacks differently fails loudly rather
than returning wrong weights), the N=80 fallback, and both paths producing
identical output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`vllm_omni/deploy/edge/qwen3_tts.yaml` carries `orchestrator:
parallel_stage_init: true`, but `_apply_profile_orchestrator_defaults` ran
only on the `deploy_profile` path. Pointing `--deploy-config` at the same file
loaded its stages and silently dropped the block, so the edge deployment
started sequentially:
--deploy-profile edge 30.76 s
--deploy-config .../edge/qwen3_tts.yaml 63.76 s (same file)
with nothing in the log to say why. The block now applies however the config
was reached. Explicit arguments still win, and the allow-list stays narrow --
a test checks an unrecognised key cannot be injected through it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`parallel_stage_init` was structurally impossible on a CPU deployment.
`resolve_stage_physical_devices` returns None for `devices: "cpu"`, and the
admission resolver fails closed on None -- correctly, for a GPU replica whose
devices cannot be accounted -- so every CPU pipeline raised
StageAdmissionError and fell back to sequential stage startup. The edge
profile sets parallel_stage_init: true, so that profile was broken on CPU
rather than merely slower.
Measured on Qwen3-TTS 0.6B (Xeon 8480C, 24 threads, init only):
sequential 64.57 s
parallel 30.54 s 2.1x
The fix is not to skip admission. The per-device ledger is vacuous on CPU --
there is no device to overcommit -- but the resource concurrent CPU init
contends for is host memory, and two stages allocating KV caches at once can
exhaust it exactly as two GPU stages exhaust a card. So there is now a host
ledger with the same discipline: sum the declared KV budgets, compare against
available RAM with a margin, refuse to start if it does not fit, and fail
closed on a stage that declares no budget at all. The arithmetic is pure
(`evaluate_host`) so it is testable without a device, matching the existing
design.
Which ledger applies is decided by where the stages are *placed*, not by what
platform the process reports: a first version keyed on
`current_platform.is_cpu()` and broke five existing GPU-path tests, because a
CPU-only test host still has to admit GPU-shaped plans correctly. It now
routes on the replicas' own `devices`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two corrections and the paperwork they imply.
The fused-RMSNorm registration defaulted to *on*, which contradicts the
measurement it was committed with: those kernels are 3.6x and 4.5x faster in
isolation but measured 72.7 tok/s against 74.0 end to end, because inductor
already fuses the native norm into the residual add and an opaque call breaks
that fusion. The implementations stay registered, so the provider remains
selectable through `kernel_config.ir_op_priority` on a machine where the trade
lands differently; only the default ranking is left alone now, and the test
asserts both states rather than just the opt-in one.
The four switches this branch added (VLLM_OMNI_AOT_CACHE_SHIM,
_CPU_FUSED_NORMS, _CPU_RELEASE_MEMORY, _SPARK_GATE_REDUCTION) were not in the
reviewed environment-variable inventory, so
`test_statically_resolvable_environment_accesses_are_classified` failed. They
are classified now, the reviewed count moves 28 -> 32 with a note saying why,
and each has a row in docs/configuration/environment_variables.md carrying the
measurement that justifies its default.
`VLLM_OMNI_CPU_FAST_SAMPLER` and `VLLM_OMNI_STEP_STATS_SYNC` remain
unclassified and that test still fails on them. They arrived in 6ce8de1b, in
files this branch does not touch, so they are left to whoever owns that work
rather than documented from a guess.
Under `-m "core_model and cpu"`: 2593 passed before these fixes with 9
failures, of which two were mine. The rest are CUDA-only tests in a CPU venv
("Torch not compiled with CUDA enabled") plus test_qnn_export.py, which
another session is actively editing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`VLLM_CPU_KVCACHE_SPACE` is a server-shaped default and is parsed with
`int()`, so it cannot express an edge-sized cache at all -- the smallest
non-zero value it accepts is 1 GiB. Measured on Spark-X2.5-1.7B (Xeon 8480C,
2048-token single stream), passing `kv_cache_memory_bytes` sized from the
model instead of the 4 GiB default takes resident memory from 8329 MB to
5254 MB, a 3.1 GB saving with no effect on output.
`kv_budget_for` reads the config and prices the cache. It reports two numbers
because the model and the backend disagree: a hybrid-attention model does not
need a full-length cache in every layer -- Spark is 21 sliding (window 512)
to 7 full layers, so at 32 k context its real cache is 3.8x smaller than
`layers x context` -- but vLLM's CPU path does not take that discount today
(for a 4 GiB budget it reported 71 527 tokens, i.e. the flat 56 KiB/token).
So `bytes_total` is the flat figure, which is what is safe to pass now, and
`bytes_total_hybrid` is reported beside it for a backend that accounts per
layer type.
Deliberately conservative in three places, because a budget that is too small
is not a graceful degradation -- vLLM preempts by recompute, which on a
single-stream edge deployment is a stall: the window is rounded up to the
block size, a 1.25 headroom factor is applied, and a config that has a
`sliding_window` but no `layer_types` map is charged the flat rate rather than
assumed to apply the window everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four changes, each measured on a Xeon 8480C with Spark-X2.5-1.7B at 24
threads. Numbers are best-of-interleaved-passes; this host is shared and a
single sequential grid swung the llama.cpp control 30%, so anything under
~8% is not treated as a result.
**vLLM's CPU backend does have a 4-bit kernel.** An earlier note in this
tree said it did not, on the strength of `woq_int4_linear`,
`int4_scaled_mm_with_quant` and `da8w4_linear` all being absent. Those are
simply not the names it uses: `CPUWNA16LinearKernel` dispatches GPTQ/AWQ
checkpoints to `int4_scaled_mm_cpu`, an AMX W4A8 GEMM, whenever activations
are bf16 and the checkpoint carries no act-order. Four-bit weights times
eight-bit activations is the arithmetic llama.cpp's Q4_K_M already does.
`gptq_cpu_export` builds checkpoints for it; the kernel pins the zero point
at 8, so the grid must be symmetric, and a per-group scale search recovers
most of what that costs.
**A per-device weight path** (`edge/weight_path.py`). The fast kernel on each
device needs a different, mutually exclusive weight layout, so the choice has
to come from a probe rather than a flag: AMX x86 -> W4A8/GPTQ; x86 without
AMX, or priority="fidelity" -> W4A16/tinygemm (bf16 activations, the more
faithful of the two); ARM -> W4A16 but flagged unverified, because PyTorch's
4-bit CPU GEMM dispatches a different kernel there than the AVX-512 one
measured here; phone -> runs_in_vllm=False, the deliverable is an exported
graph with calibrated w8a16.
**The CPU compile cache never loaded.** `_try_load_aot_compiled_fn` finishes
with `compiled_fn.finalize_loading(cfg)`, which the CUDA wrapper object has
and the inductor CPU backend's plain function does not. The AttributeError
was swallowed as a cache miss, so every engine start recompiled the model:
warm start 49.8 s against 14.9 s with the missing no-op attached. Self-
disabling if upstream gives the CPU path a real one.
**The head gate stops being a GEMM.** A 2048x8 projection cannot amortize a
thread fan-out: dispatched as `aten::mm` it costs 1.155 ms of a decode step,
nearly all barrier. As an eager reduction behind an opaque op it costs 0.444
ms. It has to be opaque -- written inline the reduction is traced, and
inductor's loop made the whole step 1.8x slower (78.8 -> 44.5 tok/s).
Also included and *off by default*: `cpu_ir_ops` registers the fused
`_C::rms_norm` / `_C::fused_add_rms_norm` kernels that vLLM's CPU build ships
but gates on `GPGPU_DEVICE`. In isolation they are 3.6x and 4.5x faster than
the native path; end to end they are slightly slower (72.7 vs 74.0 tok/s),
because inductor was already fusing the native norm into its neighbours and
an opaque call breaks that fusion. Kept with the measurement recorded so the
next person does not repeat it. `gelu_and_mul` is deliberately not registered:
its fused CPU kernel is 2.7x slower at Spark's GEGLU shape.
Decode, W4A16 g64 -> W4A8 g128 with the gate fix: 69.1 -> 84.5 tok/s (+22%),
against llama.cpp Q4_K_M's 90.0; prefill 3347 against 1121 (3.0x). The two
4-bit paths are not equally faithful: against each build's own bf16 over 12
greedy prompts, Q4_K_M reproduces 4/12, W4A16 2/12 and W4A8 1/12.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Single-stream CPU decode reads every weight to emit one token, so bits per
weight set the token rate. This registers a W4A16 group-wise method
out-of-tree through register_quantization_config -- vLLM itself is untouched
-- using PyTorch's _weight_int4pack_mm_for_cpu tinygemm kernel.
Two paths over one set of weights, because that kernel dequantizes once per
row of activations and so is linear in the batch (12.9x slower than bf16 at
512 rows). Single-row steps take tinygemm; wider batches unpack the weight to
bf16 once and use the ordinary oneDNN/AMX GEMM. Shipping only the first
measured pp512 = 557 tok/s against bf16's 4455; with both, 3589.
The batch threshold is 2 rather than the ~8 rows where the unpack path starts
winning on speed, because running tinygemm on a multi-row batch and then
vLLM's cpu_attention_with_kv_cache in the same forward dies with SIGILL
inside the attention kernel -- an AMX instruction with no valid tile config
loaded. The same prompt is fine in bf16 and fine once every multi-row GEMM
takes the unpack path.
Positioning, measured on a Xeon 8480C with 32 pinned cores and a warm cache
(analysis/spark_cpu_int4.md): vLLM's own GPTQ W4A8 AMX path is faster --
88.7 tok/s decode and 4355 prefill against 85.5 and 3589 here -- but its
config rejects sym=False at 4 bits, so it can only be fed a symmetric grid.
This one is asymmetric and costs +12.5% perplexity against that path's
+27.2%, and needs no AMX. llama.cpp Q4_K_M, for reference, is 98.0 tok/s
decode and 1356 prefill at +7.6% and 5.15 bits per weight.
Expanding a quantized checkpoint back to bf16 offline and scoring it
reproduces the quantized run to within 0.2% perplexity, so both GEMM paths,
the nibble packing and the quantized tied embedding are faithful.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXhbAfkSNRaeqZrus5Z2fx
The score-space join removed the concat as intended and is numerically
identical (140.3 dB), but it cannot be quantized: replacing torch.softmax
with an explicit max/exp/sum/div made QNN's context-binary converter exit 14
under w4a16, while every fp16 build compiled. Two earlier shapes failed the
same way -- a broadcast multiply plus ReduceSum for the new token's score,
and a two-axis broadcast multiply for its value contribution.
The device profile says the win was never the concat anyway. Of a sliding
layer, Output was 9.9%, its quantise conversion 3.1% and the window Slice
7.4% -- all of it the cost of handing a 512-entry window back across the
graph boundary once per token. So keep roll's attention exactly as it was,
one fused Softmax over concatenated keys, and change only what the layer
returns: the single new K/V entry, which the runtime drops into the ring
slot. Ring order needs no fixing up, because softmax over keys is
order-independent and each cached key already carries its own rotary phase.
Measured on a real Galaxy S25, sliding layer, ms:
w4a16 0.917 -> 0.838 (-8.6%) <- now compiles
fp16 1.689 -> 1.486 (-12.0%)
Full-attention layers keep roll: ring gains them nothing (1.567 vs 1.559 at
fp16, the concat saved being spent on the extra score matmul), so the
default layout is now per layer type.
Parity against the reference is 140.3 dB for the mixed default with
identical top-1/top-5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Reading the device profile of the first export showed a fifth of every
sliding layer went nowhere useful: Output 9.9%, its quantise conversion
3.1%, and the window Slice 7.4%. The layer took a 512-entry window in,
appended the new key, sliced the oldest off and handed the whole recomputed
window back out -- 21 of 28 layers paying to move a cache-sized tensor
across the graph boundary once per token. Full layers paid a similar 9.5%
concatenating the new key onto the cache.
Both are the same mistake, and both go away by joining in score space
instead of key space: score the cache and the new token separately, then
concatenate two vectors of length n and 1 rather than an n x 256 key matrix.
Nothing cache-sized is copied, and each layer returns just the new K/V entry
for the runtime to drop into its ring slot. Softmax over keys does not care
what order the slots are in and each cached key already carries its own
rotary phase, so a ring buffer needs no reordering -- only a mask for slots
not yet written. The sliding cache correspondingly holds window-1 entries,
since a 512-token window counts the current token (transformers' own sliding
cache holds 511 for the same reason).
Measured on a real Galaxy S25, fp16, ms per layer:
sliding 1.689 -> 1.483 (-12.2%)
full 1.559 -> 1.567 (unchanged: the concat saved is spent on the
extra score matmul)
Two shapes had to be written as matmuls rather than the obvious elementwise
forms -- the new token's score (broadcast multiply + ReduceSum) and its
value contribution (a two-axis broadcast multiply). Both compile at fp16 and
both make QNN's context-binary converter exit 14 under w4a16; as matmuls
they compile and are the cheaper op on HTP anyway.
Also adds a selectable GELU approximation, and the measurement that closes
the question: erf GELU is 11-13% of every layer, but tanh is *slower* on
this NPU (1.535 vs 1.483 ms) despite holding 65.2 dB. The exact form stays,
which is also the better-fidelity choice.
Parity against the reference implementation is 136.8 dB with identical
top-1/top-5, and a test pins ring and roll to the same output on a shared
history so the device numbers cannot silently measure a different model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Exports one Spark decode step with the KV cache as explicit tensors, sized
per layer type: the 21 sliding layers hold a fixed 512 entries at any
context, the 7 full-attention layers grow. That asymmetry is the whole
reason the architecture is interesting on a phone, so the two types get
separately sized caches, masks and rotary angles.
Verified against the HF reference at 1024 context: 138.5 dB logits SNR,
identical top-1 and top-5.
Profiled on a real Galaxy S25 (Snapdragon 8 Elite) through AI Hub, QNN
context binary on the NPU, ms per layer:
sliding (512 KV, any ctx) fp16 1.689 w8a16 0.912 w4a16 0.917
full @ 1k fp16 1.559 w8a16 1.129 w4a16 1.127
full @ 4k fp16 2.964 w8a16 1.681 w4a16 1.662
Composed into a step (21 sliding + 7 full), w4a16 gives 27.2 ms at 1k and
30.9 ms at 4k, against 31.6 and 46.5 ms for the same model with every layer
full-attention -- 1.16x and 1.51x. Worth noting the hybrid is 0.94x at 1k in
fp16: sliding layers rotate all 256 head dims where full layers rotate 64,
and below roughly 2k that RoPE cost outweighs the shorter cache.
Also moves the head-wise attention gate off vLLM's CPU linear dispatch. The
gate is [2048, 8], and any linear that narrow is routed to the sgl-kernel
packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim
0; VNNI packing does not at that size, so it reports K and torch.compile
fails tracing the gate multiply. A plain parameter with F.linear gains
nothing less -- a 2048x8 matmul has no use for packed AMX tiles -- and it
keeps the model compilable on CPU and GPU alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Spark-X2.5-1.7B is an Apache-2.0 on-device model built for agentic tool
use. vLLM 0.28 has no implementation for it, so it could not be served at
all; llama.cpp already ships one, which made it the only edge engine able
to run the model.
Three traits separate it from a Llama-style decoder:
* Hybrid attention -- layer_types interleaves three 512-token sliding
layers with one full-attention layer, which is what makes the 1M-token
context affordable on a phone.
* Per-layer-type RoPE -- sliding layers rotate all 256 head dims with
theta 1e4, full layers only the first quarter with theta 5e6.
* A head-wise attention output gate -- g_proj reads the same post-norm
input as QKV and scales each head's attention output by a sigmoid.
The checkpoint also stores QKV pre-fused as a single q_k_v_proj, so
load_weights splits it before vLLM's sharded QKVParallelLinear sees it.
Verified against the official remote-code implementation on transformers
4.57.1 (the version config.json targets; it does not import on the 5.x in
our venvs). In fp32 all three scenarios match exactly -- 32/32 greedy
tokens with a max top-20 logprob difference of 2.2e-05. In bf16 the short
and the 2465-token long-context prompts still match 32/32, which is what
exercises the sliding layers; the tool-calling prompt diverges at token 23
on a near-tie, and fp32 confirms that is rounding rather than a defect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
The Code2Wav decoder already had an on-device export (WP4); the talker and its
code predictor did not, so the codec-token path stopped at the vocoder. This
adds the missing half: a one-token decode step with an explicit per-layer KV
cache, exportable to ONNX and from there to a QNN context binary, TFLite or
ONNX Runtime.
Two layout choices carry the performance, both measured on real Snapdragon
devices through Qualcomm AI Hub (Galaxy S24 = 8 Gen 3, S25 = 8 Elite):
* per-layer cache tensors instead of one stacked [num_layers, ...] tensor that
every layer slices, and
* grouped-query attention by reshaping the query rather than expanding keys and
values.
The stacked/expanded form spent ~70% of its NPU cycles in cache select,
reshape and fp32->fp16 conversion; the layout here is bandwidth-bound and runs
the 0.6B talker step in 15.8 ms instead of 36.9 ms on an 8 Elite (fp16,
256-token cache). The code predictor step is 2.25 ms, so talker plus its 15
sub-steps cost about 50 ms of an 80 ms frame.
sanitize_onnx() fixes the three things the dynamo exporter emits that
qairt-converter and AI Hub reject: IsNaN/Where guards, Reshape allowzero=1, and
rank-0 axes initializers on Unsqueeze/Squeeze.
The module docstring records the measured placement table: this graph belongs
on the NPU, the vocoder on the GPU (21.7 ms vs 182 ms per frame), which is 3.3x
better than sending everything to the NPU and makes the pipeline real time on
the phone in fp16. Integer quantization is not the lever it looks like: every
automatic int8 build measured was numerically broken.
Full write-up, job ids and profiles: analysis/mobile_htp_simulation.md section 5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dzr1Jusv4s6g765gqS9L92
Noise is drawn eagerly with the request generators (same RNG consumption as the
eager loop) and passed in, so the compiled post-step is generator-free and
shape-static; used on the fp32/torchao CPU path outside stored (top-p) mode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
torchao's dynamic-activation int8 is numerically wrong under torch.compile on
CPU (cosine 0.02 vs fp32; caused runaway generations), so it is not offered;
weight-only + compile is faithful but slower (78 ms vs 36 ms per frame for the
15 sub-steps), so torch.ao dynamic int8 stays the default. Parity mode now also
records top-5 overlap and KL vs the bf16 model; compiled-step equivalence test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Measured on a Xeon 8480C (12+4 threads, 0.6B): the 15 residual-codebook sub-steps
re-ran the 5-layer predictor over the whole prefix each time (97 ms per frame of
the 134 ms talker step) and vLLM's stock sampler spent 9 ms per step in its
torch.compile'd random-sample kernel on CPU.
- code predictor: forward_cached (KV cache per layer, 2-token prefill then one
token per sub-step) and a shape-static forward_static_step (tensor position +
boolean mask) that torch.compile handles as one graph; incremental decoding
is the default on CPU (VLLM_OMNI_PREDICTOR_INCREMENTAL), optional compile
(VLLM_OMNI_PREDICTOR_COMPILE=step), optional thread cap.
- CPU fast path: dynamic int8 Linear layers (fp32 activations) for the predictor
and its lm heads (VLLM_OMNI_PREDICTOR_INT8, default on; micro-benchmark 70 ->
36 ms per frame for the 15 sub-steps); VLLM_OMNI_PREDICTOR_PARITY_CHECK=1
records top-1 agreement / max logit diff against the bf16 model in situ.
- Qwen3-TTS talker: prefer_model_sampler on CPU with a sample() that mirrors
vLLM's Sampler (logits processors + penalties, temperature, top-k/top-p,
exponential-noise sampling with per-request generators) without the compiled
kernel (VLLM_OMNI_CPU_FAST_SAMPLER, default on).
- step_stats: sampler phase timers and sample-metadata timers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Measured after the DELTA fix: 53 engine outputs for 52 rows, but the client
still received everything at once because each per-step payload is the newest
row only and the row alignment treated it as a cumulative snapshot (one yield
at the first row, one at the end). accumulate_codec_rows folds delta rows and
the final consolidated matrix into one growing matrix; unit test drives the
generator with per-step deltas and asserts one yield per step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The speech path forced FINAL_ONLY for Qwen3-TTS deployments without async
chunks, so a talker-only server emitted one engine output per request and all
codec frames arrived at once (measured: 1 engine output per request, first
codes at output #1). A talker-only stage (final_output + latent) now keeps
DELTA outputs so codec frames stream per step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- environment_variable_inventory: VLLM_OMNI_CPU_INDUCTOR / HW_PROFILE /
INIT_TIMELINE / STEP_STATS_DIR (public), STAGE_ID / INIT_TIMELINE_DIR
(internal); reference page rows added.
- edge_decoder_client.LocalDecoder: batched_chunked_decode takes a [B, Q, F]
tensor, not a list (the WP5 e2e run failed with a TypeError); the decoder
now also carries the ramp schedule attributes the Code2Wav stage sets.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
vllm_omni/edge/decoder_export.py exports a stateless windowed decoder (context
72 frames + chunk, one program per ramp chunk size) via torch.export (.pt2,
verified here on CPU for chunks 2 and 25 with the real 0.6B decoder: exact
match vs eager, 460 MB fp32 per program) or ExecuTorch XNNPACK (.pte; written
against the executorch 1.x API, not executed: package not installed, install
is permission gated; QNN partitioner export-only/unverified). The WP5 client
gains --decoder-backend executorch --pte-manifest for phone/NPU targets.
benchmarks/tts/edge_results/summarize.py renders all committed JSON as tables.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- vllm_omni/edge/step_stats.py: process-local counters enabled by
VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core
step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped
per process and merged into an attribution table (share of the 80 ms frame).
Hooks: stage_engine_core_proc (instrument after construction, dump at exit),
chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime),
orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats.
- vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and
the no-preemption KV admission rule (exact bytes/token from the talker KV
geometry); adapt.derive_overrides now sizes max_num_seqs with it and
auto_profile passes the geometry read from the model config.
- Engine re-reads parallel_stage_init after apply_deploy_profile so the edge
profile's orchestrator default takes effect (was read before the profile).
- benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor)
with per-step attribution.
- Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py,
tests/benchmarks/test_cpu_thread_sweep.py, profile default test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wraps SharedMemoryConnector: CUDA tensor leaves are exported with
torch.multiprocessing reduce_tensor + an inter-process CUDA event, rebuilt and
cloned by the consumer, released by the producer after an ack file appears.
CPU leaves and metadata keep the shm path. Registered as "CudaIpcConnector";
deploy/qwen3_tts_cuda_ipc.yaml selects it for the talker->code2wav hop.
Adoption in the edge profile is measurement-gated (hop latency bench).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
tzhouam