One package that makes the POSIX assumptions measured in W0-W8 hold in
every process vLLM / vLLM-Omni starts, and reports what it did:
- provides fcntl (flock/lockf on msvcrt.locking; LOCK_SH served as
exclusive) and uvloop (asyncio.run on a selector loop) when absent;
- installs WindowsSelectorEventLoopPolicy and a signal.signal fallback
for loop.add_signal_handler on the Windows loop classes;
- replaces whole vLLM's get_open_zmq_ipc_path / find_loaded_library when
the wheel lacks the branch patch and the upstream body hash is known;
- verifies the wheel-only sites (sentinel poll, int64 seed) and reports
wheel_required at the exact site instead of failing downstream;
- shm_dir() replaces the /dev/shm literals; os.geteuid provided.
Activated from vllm_omni/patch.py and as vllm.general_plugins /
vllm_omni.general_plugins entry points (idempotent; no-op off Windows).
prints the audit; the M0 placement
report embeds it. 16 Linux tests through injected primitives.
Design: analysis/design_windows_compat_plugin_20260921.md
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
uvloop ships no Windows wheels, so 'import uvloop' at module scope in
entrypoints/cli/serve.py killed vllm-omni serve on Windows at import
(measured in the serving e2e after the fcntl fix). _run_forever() uses
uvloop where present and asyncio.run on WindowsSelectorEventLoopPolicy
otherwise, which zmq.asyncio needs. The remaining serve-import blocker
on the Windows venv was a missing dependency, torchaudio (vLLM's own
requirement), installed into the venvs rather than patched.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
test_e2e_serve_text.py no longer imports tests.helpers.runtime (it pulls
the media/diffusion test helpers and therefore diffusers, absent from the
Windows venv); a small Popen launcher with a psutil process-tree kill
replaces OmniServer for this one test.
vllm-omni serve died at import on Windows: entrypoints/cli imports
benchmarks.patch -> benchmarks/omniinteract.py, which did 'import fcntl'
at module scope. Its output lock is rebuilt on the fork's
_filelock_compat primitive (fcntl on POSIX, msvcrt on Windows).
engine/stage_phase_lock.py (parallel_stage_init's SH/EX device locks)
gets the same treatment: fcntl optional, modes spelled locally, and on
Windows both modes map to the exclusive byte lock -- more serialising,
never less safe. Measured: the serving e2e's server exited with
ModuleNotFoundError: No module named 'fcntl' on both Windows venvs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pytest-asyncio builds its loops from the current policy; on win32 that is
the Proactor loop, which zmq.asyncio cannot use, so the engine/serving e2e
hung at the first async socket (30 min, no output, GPU idle) while the
accept CLI -- which installs WindowsSelectorEventLoopPolicy itself --
passed on the same venv. The conftest now installs the same policy and
provides it as pytest-asyncio's event_loop_policy fixture.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Drives `vllm-omni serve <model> --omni` as a subprocess through the
online-serving launcher and checks /health, /v1/models, non-streaming
and streaming chat completions, and that a client disconnect mid-stream
leaves the server answering. Sets VLLM_USE_FLASHINFER_SAMPLER=0 like
LocalTextEngine does: the flashinfer sampler JITs with nvcc at runtime,
which a venv without a CUDA toolkit does not have (measured: the WSL
reference venv died in profile_run with 'Could not find nvcc').
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 0.29.0 loader de-duplicates tied parameters itself, so the explicit
lm_head. skip for tie_word_embeddings is no longer needed (and its keyword
no longer exists: engine init failed with TypeError on the first 0.29.0
parity run).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
vllm_omni/edge/local/acceptance_chat_prompts.json was invisible to the WIP
snapshot because .gitignore ignores *.json; without it the acceptance CLI
and the parity runner cannot start on a fresh export.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of
rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while
our 0.28-based tree needs seven modules v0.29.0 no longer has
(vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args,
openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant,
models.hunyuan_v1).
Ten files conflicted. Upstream's structure is the base everywhere; the
fork's features are re-attached to it:
- deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config
pops deploy_config (async_omni_engine, cli/serve); the deploy-profile
selector and the resolve_model_config_path chain it needs stay in
entrypoints/utils.py; stage_overrides takes rc1's dict type.
- config_factory keeps the lazy diffusion output-type import; the diffusion
registry import in rc1's new config/resolver.py is made lazy for the same
WP6 reason (TTS path must not load diffusion).
- requirements: our core/media split kept; rc1's transformers>=5.13 floor
and kernels==0.16.1 applied in core.txt / media.txt.
- TTS: rc1's split_granularity and bounded seed plus our output_mode=
codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation;
our private _conditioning_cache_salt is superseded by rc1's public
tts_adapters/base.py::conditioning_cache_salt.
- stage_runtime: rc1's single ExitStack G2 path (lock released before READY)
replaces our two-path block, with the four worker_phase names the init
profiler consumes still emitted; _cpu_placed / host admission kept.
Not yet run against vLLM 0.29.0: the tests listed in the W6 record.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Verbatim copy of the edge/omni-edge working tree (tracked diffs + untracked
edge/local, tests/edge/local, docs/edge, pipeline files). Taken so that
vLLM-Omni v0.29.0rc1 can be merged on a branch without touching the
user's working tree. logs_edge/ and benchmark .npy outputs excluded.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Renamed from vllm-embedding: nothing here is about embeddings, and the name had
come from the parent working directory. edge-infer says what it does. The
GitHub repository is renamed to match; nothing had been pushed yet.
The audit found the packaging was incomplete in two ways.
Scripts: 56 of them produced results that are published in this repo but were
not included, so the numbers could be read and not checked. Added under
benchmarks/edge_harness/experiments/ (one-off drivers: the AI Hub on-device
runs, the 4-bit GEMM microbenchmark, fidelity and parity checks, checkpoint
build and round-trip, activation capture, and the load generator used to prove
the contention guard actually refuses) and benchmarks/edge_harness/runs/ (the
24 shell drivers behind published grids). Both directories carry a README
saying plainly that they are a record of what was run rather than a supported
interface -- most hardcode absolute paths from the machine they ran on. The
supported entry points remain the tools one level up.
One broken relative link fixed (comparison.md is under docs/engines/, not
docs/analysis/). All other links and every script reference in the docs now
resolve inside the repo.
Secret scan: the token in ~/.qai_hub/client.ini is absent from the worktree and
from all 61 commits. Worth noting that an earlier version of my own check
reported a false positive, because `grep -rl ... | head` takes its exit status
from head, which always succeeds -- the same class of mistake as the measurement
errors this project keeps recording. The AI Hub scripts authenticate by reading
that file at runtime and contain no credential.
README reframed for the new name and the wider scope: the repository is the
engine changes, the harness, the raw results, the baselines and the analyses,
not just a patch set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
The previous commit said "vLLM itself is not modified" and described the
changes as going "through vLLM's own extension points". Challenged on it, and
the challenge was right: those are two different claims and only one of them
holds as written.
What is true: no file in the vLLM tree or in site-packages/vllm is edited, the
checkout is clean upstream and the wheel is stock 0.28.0+cpu, so there is no
patch series to apply and no vLLM fork to maintain.
What was wrong:
1. vllm_omni/patch.py does nine _patch_* functions performing ten direct
attribute replacements -- CpuPlatform.check_and_update_config,
CPUWorker.load_model, torch.compiler.load_compiled_function. That is
monkey-patching, not an extension point, and calling it one made the
arrangement sound more robust than it is. Only the registries are genuine
hooks: register_quantization_config, ir.ops.*.register_impl, platform
plugins, direct_register_custom_op.
2. "The changes survive a wheel upgrade rather than being lost on it" is true of
the registrations and overstated for the patches. Each patch is keyed to a
specific method on a specific vLLM class; an upstream rename or behaviour
change can break it or silently turn it into a no-op wrapper. The failure
mode is a missing optimisation, not a crash, which is worse -- so the doc now
says to re-run the harness after upgrading.
The section is rewritten to separate the two mechanisms, list what each patch
works around, and note that two of the three exist for upstream bugs and should
disappear when those are fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Turns a branch of accumulated edge/CPU work into something someone else can
read, reproduce and continue.
README.md states what the fork is for (multimodal models on edge CPUs rather
than servers), the headline memory and speed results with their costs, and why
the numbers are worth reading -- five retracted conclusions, all measurement
errors rather than coding errors. Upstream's README is preserved at
docs/README.vllm-omni.md and NOTICE records the Apache-2.0 derivation and the
files this fork changes.
docs/edge/:
methodology.md the measurement discipline, which is the most transferable
thing here: what each of the five retractions was, and the
rule it left behind. Includes the two traps specific to this
stack -- vLLM's AOT compile cache keying on config so two
arms sharing an artifact load the wrong graph, and allocated
not being resident until mincore(2) says so.
results.md every measured figure with its spread and baseline, and an
explicit note that the four memory levers overlap and must
not be added.
cpu-4bit.md the two 4-bit paths, the packed layout as measured rather
than assumed, why the path choice is a config field, and the
GPTQ output-head trap that invalidated a whole round of W4A8
numbers.
harness.md how to run a comparison, and the two arguments that matter
more than they look (--context-len, --prompt-len), each of
which has produced a wrong conclusion by being left at its
default.
ROADMAP.md orders by whether the verification method is correct rather than by
expected size, lists what is done with results, what is next with expected
gains and how each would be verified, and -- at equal length -- what was
considered and rejected, including the CPU megakernel with the measurements
that rule it out for now.
benchmarks/edge_harness/ brings the tooling into the repo so the numbers are
reproducible: the grid runner, the single-run benchmarks, the memory ledger,
the fidelity metric, the Android agent suite, the quantizers, and the tests
that pin the harness's judgement. 82 tests pass from the new location.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
512 was a first guess. Repacking the 131072-row tied embedding at 24 threads:
rows chunks time int32 transient
whole 1 208 ms 1024 MiB
512 256 116 ms 4 MiB
2048 64 79 ms 16 MiB
8192 16 89 ms 64 MiB
32768 4 129 ms 256 MiB
2048 is both the fastest and nearly the leanest. The sweep also settles a
suspicion: chunking is faster than packing the whole matrix, because the
working set stays in cache, so this costs nothing at load. The one pass that
appeared to show a 30 s init regression was the pass the harness had flagged
as contended.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
This module carried a retraction saying the measured saving from right-sizing
the cache was 188 MB. That retraction was itself measured through a path that
ignored the knob: CpuPlatform.check_and_update_config overwrote
kv_cache_memory_bytes from VLLM_CPU_KVCACHE_SPACE unconditionally, so the
"32x shrink" never happened and both arms ran the same cache.
With the budget actually honoured, 3 interleaved passes per arm, both filling
a 1536-token context:
1 GiB budget 3264 MB resident
192 MiB budget 2421 MB resident -> 843 MB
And the reasoning behind "the pages are never touched" is wrong too. mincore(2)
reports the cache fully in core: 1024.1 MB resident of 1023.8 allocated, and
191.1 of 190.8 when sized down. It is not address space, it is memory.
843 MB is the largest single lever measured on this model -- more than the
single weight layout (615 MB), the blocked repack (489 MB peak), or the RoPE
table (618 MB).
The general lesson, recorded in the report: a lever tested through a path that
silently ignores it measures nothing, and "no effect" is the most dangerous
result to accept without first checking the knob was connected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
618 MB of resident memory and 649 MB of peak, for one line -- the largest
single memory item found in this study, and unlike the others it costs nothing.
Spark advertises a 1,048,576-token context. This file passed
max_position=config.max_position_embeddings straight into get_rope, and vLLM
builds its cos/sin table with torch.arange(max_position), so a 2048-token
deployment carried tables for a million positions. Because the sliding and full
layers are trained with different theta and different partial rotary factors
there are two of them:
layers.0 (full, head_dim 256) [1048576, 256] 512 MB
layers.3 (sliding, partial 0.25) [1048576, 64] 128 MB
Both fully resident -- confirmed with mincore(2), not inferred. At
max_model_len 2048 they need 1.25 MB between them.
Found by memory_ledger.py, which attributes memory instead of subtracting
toward a remainder: 640 MB had been sitting in an unnamed "other" bucket across
114 tensors, and naming the largest storages showed what it was. After the cap
that bucket is 1.3 MB and unique tensor storage falls 638.8 MB, which agrees
independently with the end-to-end measurement.
vLLM's scheduler cannot produce a position beyond max_model_len, so covering
that is sufficient. Measured with VLLM_OMNI_SPARK_ROPE_FULL toggling the two in
identical code, 3 interleaved passes per arm on node-local cores:
1M tables settled 3891 MB peak 4019 MB init 13.5 s
capped settled 3273 MB peak 3370 MB init 13.7 s
The escape hatch restores the old behaviour and is covered by a test, as is the
cap itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The review caught weight_path.py claiming tinygemm is "the only option" for an
x86 core without AMX. That was false -- vLLM 0.28 selects ops.cpu_gemm_wna16
whenever layer.use_w4a8 is off -- and the honest response was to measure the
kernel, not to reword the claim.
Same GPTQ checkpoint, only VLLM_CPU_INT4_W4A8 differing, isolated compile
caches, 3 interleaved passes on 24 pinned node-local cores:
int4_scaled_mm_cpu (W4A8, AMX) 74.4 tok/s spread 1.1%
tinygemm W4A16 (same cores) 71.6 tok/s
cpu_gemm_wna16 32.3 tok/s spread 1.5%
The fallback is real but 2.2x slower than tinygemm, so the selector's routing
was right for a reason nobody had checked. It stays as it is; what changes is
that the module now cites a measurement rather than an assumption.
Recorded with the caveat that matters for reading it off-x86: _get_isa_hint
returns "amx" whenever AMX is present and activations are bf16, regardless of
the W4A8 flag, so this measured cpu_gemm_wna16 on its AMX path -- its best case
on this host. The "vec" path a genuinely AMX-less machine takes is untested and
presumably no faster, which only strengthens the conclusion, but no number is
claimed for it.
Also adds VLLM_OMNI_CPU_INT4_REPACK_ROWS so the blocked repack can be A/B'd
against whole-matrix packing in identical code. Unlike the prefill-path switch
this is load-time only and yields identical tensors, so it does not perturb the
compiled graph or its cache key.
tests/edge: 8 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
I0 of the reviewed plan: the peak, not the settled figure, is what an edge
device has to survive. Measured earlier this session, W4A16 loads at 5722-5837
MB and settles near 4480 after the allocator is asked for its arenas back -- a
1.2-1.3 GB gap, and a device that cannot hold the peak never reaches the
release.
A large part of that gap is in our own repack path. unpack_nibbles expands
4-bit codes to int32, which is 4 bytes per weight and eight times the packed
form, and the kernel's packer wants int32. Done for a whole matrix that
transient is the largest object in the process:
gate_up 13312 x 2048 104 MiB
down 2048 x 6656 52 MiB
q_k_v 3072 x 2048 24 MiB
embedding 131072 x 2048 1024 MiB <- tied, so it is also the head
against 24 MiB of packed weights for an entire layer. The embedding alone is a
~1 GB spike, which is about the size of the whole peak-vs-settled gap.
The packed layout is [N/64][K][32 bytes], so n-blocks are independent and the
expansion can cover one chunk at a time. Verified byte-identical to packing the
whole matrix at 13312x2048, 2048x6656, 3072x2048 and 131072x2048, plus a ragged
last chunk. The int32 transient drops from 1024 MiB to 4 MiB for the embedding
and from 104 to 4 MiB for gate_up. to_split_halves is chunked the same way for
the fallback path.
Not yet claimed: the end-to-end peak reduction. The arithmetic says ~1 GB, but
this study has been wrong often enough about memory that the number goes in
only after RssAnon is sampled through a real load. Tests: 21 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docstring said one layout costs 1.9x prefill. That came from the single
surviving pass of a grid whose other passes had crashed on the AOT cache
collision -- a number the harness itself had labelled "single pass, not a
result", which I then published.
Re-measured properly: caches isolated per arm so all passes complete, on idle
node-1 cores with memory bound to the same node, pooled over 7 interleaved
passes per arm. pp512 goes 2190 -> 1344 tok/s, i.e. 1.63x, not 1.9x.
The ratio is only good to about 1.4-1.8x (worst pairing 1.84x, best 1.36x)
because both dequantizing arms swing 10-18% here, while the tinygemm-only arm
repeats to 0.3%. The existence and rough size of the cost is solid -- a 63%
gap is far outside that band -- but the third digit is not, and the docstring
now says so rather than implying precision it does not have.
The comparison that is reliable is against prefill_dequant: false, whose arm is
the steady one: 2.97x its prefill at the same memory, without routing multi-row
batches into the tinygemm/AMX SIGILL hazard. That setting stays dominated.
Also corrected a hypothesis of my own. The first three direct passes read
1344 -> 1421 -> 1555 and I called it a warm-up trend; four more passes gave
1271, 1335, 1443, 1309, so it was noise and three ascending points out of three
at 12% spread meant nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The W4A16 path kept two packed weight layouts because the tinygemm layout was
treated as opaque: the prefill path needed something row-major to dequantize
from, so `_finalize_int4` built `weight_halves` plus cloned scale/zero beside
the kernel's own buffer. That duplicate is the 740 MB the report attributes to
"the dual weight layout".
The layout is not opaque. Measured on this host by packing an index pattern and
reading back where each nibble landed: it is [N/64][K][32 bytes] -- K is the
middle axis, so a K range is contiguous inside each n-block -- and within every
32-byte run, byte d carries output channel d in its low nibble and d+32 in its
high one, the same permutation for every k and every block. Decoding it is a
deinterleave. _dequantize_from_int4pack does that, bit-exact against the old
dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048, and the
duplicate is no longer allocated.
Measured with the two paths toggled in otherwise identical code, 3 arms x 2
interleaved passes, RssAnon summed over the process tree, 1536-token context:
615 MB of resident memory and ~915 MB of peak, landing statistically
indistinguishable from prefill_dequant: false (the harness refused to rank
them, 0.4% apart inside the spread). Config arithmetic predicts 764 MB; ~150 MB
is allocator retention malloc_trim does not return.
It is not free, and the docstring says so. The duplicate was not only a copy:
to_split_halves de-interleaves once at load, so each prefill call read two
contiguous planes, while this redoes the de-interleave every call. pp512 drops
2272 -> 1210 tok/s. Dequantizing to [K, N] and using matmul to dodge the
transpose was tried and is worse (20.3 ms vs 14.4 per gate_up at 8 threads);
recovering the prefill wants a C++ de-interleave, not an ATen expression chain.
So it is a dial. Against prefill_dequant: false it is strictly better -- same
memory, 2.7x the prefill, and dequant_threshold stays in force so multi-row
batches never reach the tinygemm kernel and the documented SIGILL hazard. That
setting is now dominated. Against keeping both layouts it trades 615 MB for
1.9x prefill; default on, because this path exists for devices chosen for
their memory ceiling.
supports_direct_dequant gates the reshape on N tiling the 64-channel block,
which every Spark linear does (2048, 3072, 13312, 131072). An N like 80 packs
as 64 + a 16 tail the reshape would mis-read, so those layers keep the copy
rather than being quietly wrong.
The choice is a `direct_dequant` field of the checkpoint's quantization_config,
not only an env var, and that matters: the two paths build different graphs
(one passes weight_halves into the custom op, the other passes None), so they
must not share a compiled artifact. ModelConfig.compute_hash hashes config
fields, so a config field reaches vLLM's compile-cache key; an env var does
not. Toggling the env var against a warm cache loaded the wrong graph and
killed three benchmark passes with `KeyError: 'weight_halves'` inside the
AOT-loaded forward -- found because the harness records failures instead of
dropping them. VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT remains as an experiment
override, documented as needing its own VLLM_CACHE_ROOT.
Tests: 18 passed, including bit-exactness at three shapes, a pin on the packed
layout itself (so a torch release that repacks differently fails loudly rather
than returning wrong weights), the N=80 fallback, and both paths producing
identical output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An independent review (gpt-6-astra via the Codex hook) found that two modules
still ship claims this study had already withdrawn or that the vLLM source
contradicts. Documentation in executable files is how a retracted number gets
quoted again, so these are corrected at the source:
kv_budget.py claimed right-sizing the cache saved 8329 -> 5254 MB. That was
read from VmRSS (which counts the mmap'd checkpoint and moved 1-2 GB between
identical runs) and the two arms were different model files. On RssAnon the
saving is 188 MB. The retraction is now stated in the docstring, together with
why the number is small -- the benchmark generated 8 tokens, so no cache was
filled -- and the 1323 MiB flat-vs-hybrid difference at 32k that makes this
worth re-measuring on a benchmark that does fill it.
weight_path.py:
- bits_per_weight 4.16 -> 4.25 in all three 4-bit paths. 4 bits plus a bf16
scale and zero per 128-weight group is 4 + 32/128; 4.16 matched no
arithmetic and made the llama.cpp fidelity comparison look budget-matched
when it is not (Q4_K is 4.5 bits over groups of 32, Q4_K_M promotes some
tensors further).
- fused_cpu_norms True -> False in all three. Fused CPU RMSNorm is 3.6-4.5x
faster in isolation and slower in situ; VLLM_OMNI_CPU_FUSED_NORMS already
defaults to "0" for that reason, so the metadata contradicted the default.
- "tinygemm is the only option without AMX" is false: vLLM 0.28 selects
ops.cpu_gemm_wna16 whenever use_w4a8 is off (mixed_precision/cpu.py:202).
Unmeasured here, so the selector is unchanged, but the claim is not.
- W4A8's symmetric grid is our exporter's "sym": true, not a kernel limit --
_process_gptq_weights_w4a8 uses checkpoint zero points when
config.zero_points is set (same file, :98). An asymmetric export is a config
change and the cheapest untried fidelity experiment.
- kv_budget's flat-accounting observation is flagged for re-checking, since
CpuPlatform.support_hybrid_kv_cache() returns True (platforms/cpu.py:545).
Both fields now carry docstrings saying what they mean, so they cannot drift
back silently. tests/edge: 18 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The W4A16 path keeps two weight layouts -- the tinygemm one decode reads and a
row-major copy prefill dequantizes -- and the docstring said only that it cost
"8 bits per weight resident". Measured, median of three interleaved passes on
anonymous RSS (spread under 1.5%, where VmRSS moved 1-2 GB between identical
runs):
prefill_dequant resident prefill decode
true (default) 3542 MB 3252 tok/s 76.9 tok/s
false 2802 MB 448 tok/s 78.5 tok/s
So it is 740 MB for 7.3x prefill. Default stays on, because losing 7x of
prefill hurts more than 740 MB on anything with memory to spare -- but on a
device where it does not, this is the smallest resident build in the tree:
2.8 GB against the 7.3 GB the W4A8 path started at before the load-time
allocator release.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`vllm_omni/deploy/edge/qwen3_tts.yaml` carries `orchestrator:
parallel_stage_init: true`, but `_apply_profile_orchestrator_defaults` ran
only on the `deploy_profile` path. Pointing `--deploy-config` at the same file
loaded its stages and silently dropped the block, so the edge deployment
started sequentially:
--deploy-profile edge 30.76 s
--deploy-config .../edge/qwen3_tts.yaml 63.76 s (same file)
with nothing in the log to say why. The block now applies however the config
was reached. Explicit arguments still win, and the allow-list stays narrow --
a test checks an unrecognised key cannot be injected through it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`parallel_stage_init` was structurally impossible on a CPU deployment.
`resolve_stage_physical_devices` returns None for `devices: "cpu"`, and the
admission resolver fails closed on None -- correctly, for a GPU replica whose
devices cannot be accounted -- so every CPU pipeline raised
StageAdmissionError and fell back to sequential stage startup. The edge
profile sets parallel_stage_init: true, so that profile was broken on CPU
rather than merely slower.
Measured on Qwen3-TTS 0.6B (Xeon 8480C, 24 threads, init only):
sequential 64.57 s
parallel 30.54 s 2.1x
The fix is not to skip admission. The per-device ledger is vacuous on CPU --
there is no device to overcommit -- but the resource concurrent CPU init
contends for is host memory, and two stages allocating KV caches at once can
exhaust it exactly as two GPU stages exhaust a card. So there is now a host
ledger with the same discipline: sum the declared KV budgets, compare against
available RAM with a margin, refuse to start if it does not fit, and fail
closed on a stage that declares no budget at all. The arithmetic is pure
(`evaluate_host`) so it is testable without a device, matching the existing
design.
Which ledger applies is decided by where the stages are *placed*, not by what
platform the process reports: a first version keyed on
`current_platform.is_cpu()` and broke five existing GPU-path tests, because a
CPU-only test host still has to admit GPU-shaped plans correctly. It now
routes on the replicas' own `devices`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Driving a phone through the Android agent harness turns section 2c's fidelity
numbers into a task outcome. Same simulated device, one run per cell:
build decode "Turn on Wi-Fi." "Set an alarm"
bf16 41.0 tok/s done in 4 steps wrong app, stuck
W4A16 g128 80.0 tok/s done in 4 steps right app, no completion
W4A8 g128 86.6 tok/s turns it on, then off (x2) reaches Alarm, loops
Every build navigates home -> Settings -> Network -> Wi-Fi correctly. Only
some notice the switch now reads checked=true and call task_complete. W4A8 --
the fastest build, and the one the speed-first path recommends -- ends with
the task undone, on both runs.
So "W4A16 reproduces 2/12 greedy prompts and W4A8 1/12" was not a small
difference on a proxy metric: it is finishing versus reversing. The selector
now says so where someone choosing a build will read it, and points at
priority="fidelity" for agentic work.
Two tasks, one run per cell: enough to show the failure reproduces, not to
rank success rates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two corrections and the paperwork they imply.
The fused-RMSNorm registration defaulted to *on*, which contradicts the
measurement it was committed with: those kernels are 3.6x and 4.5x faster in
isolation but measured 72.7 tok/s against 74.0 end to end, because inductor
already fuses the native norm into the residual add and an opaque call breaks
that fusion. The implementations stay registered, so the provider remains
selectable through `kernel_config.ir_op_priority` on a machine where the trade
lands differently; only the default ranking is left alone now, and the test
asserts both states rather than just the opt-in one.
The four switches this branch added (VLLM_OMNI_AOT_CACHE_SHIM,
_CPU_FUSED_NORMS, _CPU_RELEASE_MEMORY, _SPARK_GATE_REDUCTION) were not in the
reviewed environment-variable inventory, so
`test_statically_resolvable_environment_accesses_are_classified` failed. They
are classified now, the reviewed count moves 28 -> 32 with a note saying why,
and each has a row in docs/configuration/environment_variables.md carrying the
measurement that justifies its default.
`VLLM_OMNI_CPU_FAST_SAMPLER` and `VLLM_OMNI_STEP_STATS_SYNC` remain
unclassified and that test still fails on them. They arrived in 6ce8de1b, in
files this branch does not touch, so they are left to whoever owns that work
rather than documented from a guess.
Under `-m "core_model and cpu"`: 2593 passed before these fixes with 9
failures, of which two were mine. The rest are CUDA-only tests in a CPU venv
("Torch not compiled with CUDA enabled") plus test_qnn_export.py, which
another session is actively editing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Loading a quantized checkpoint allocates far more than the weights occupy.
`_process_gptq_weights_w4a8` unpacks 4-bit codes to int32 -- four bytes per
weight -- permutes, repacks, and drops the intermediates. For a 1.7 B model
that is gigabytes of short-lived allocation, correctly freed, and still
resident afterwards because a caching allocator keeps the arenas.
Measured on Spark-X2.5-1.7B, anonymous RSS:
path after load after release freed
W4A8 (GPTQ) 7338 MB 3510 MB 3828 MB (-52%)
W4A16 (tinygemm) 5023 MB 3411 MB 1612 MB
The second column is the more useful finding: after the release both paths
land in the same place, so W4A8's apparent 2.3 GB penalty over W4A16 was never
a footprint difference -- it was the repack. W4A8 is therefore faster *and*
level on memory, and fidelity is the only remaining reason to pick W4A16.
Hooked to CPUWorker.load_model and compile_or_warm_up_model, since those churn
separately; `VLLM_OMNI_CPU_RELEASE_MEMORY=0` disables it. CPU only: on GPU the
weights are in device memory and host churn constrains nothing.
`anon_rss_bytes()` ships alongside because the metric matters as much as the
fix. VmRSS counts the mmap'd checkpoint's file pages and moved 1.2-2.0 GB
between *identical* runs on this host; RssAnon repeats to within 0.5 MB. An
earlier claim in this tree that KV sizing saved 3.1 GB came from reading VmRSS
across a comparison that also changed the model file; re-measured on RssAnon,
shrinking the KV cache 32x is worth 188 MB, because its pages are never
touched. The analysis note is corrected accordingly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`CpuPlatform.check_and_update_config` ends with a block whose own comment
reads "Lagecy setting":
env_key = "VLLM_CPU_KVCACHE_SPACE"
if env_key in os.environ and os.environ[env_key] != "":
cache_config.kv_cache_memory_bytes = int(os.environ[env_key]) * GiB
It runs unconditionally, so `--kv-cache-memory-bytes` is silently discarded
whenever the env var is set -- which it has to be on CPU, or the engine tries
to claim most of the machine. Two consequences for edge serving: the budget
can only be expressed in whole GiB (parsed with `int()`, so the smallest
non-zero cache is 1 GiB), and vllm-omni's per-stage `kv_cache_memory_bytes`,
which is how the deploy YAMLs express per-stage budgets, has no effect at all
on CPU.
Verified by allocation rather than by memory, because allocation cannot drift:
with the env at 1 GiB, asking for 128 MiB produced the same 17 881 cache
tokens as asking for nothing, and with the env at 4 GiB it produced 71 527 --
the explicit value never applied. After restoring the obvious precedence
(explicit wins, env stays the default for callers that set nothing) the same
request yields 2 231 tokens, the 128 MiB it asked for.
The Qwen3-TTS CPU overlay now sizes stage 0 from the model -- 112 KiB/token,
so 2 seqs at 2048 tokens needs 560 MiB, not 4 GiB. Stage 1 is left alone on
purpose: Code2Wav is a generation stage whose cache is not a plain attention
KV, and under-sizing makes vLLM preempt by recompute, which on a single-stream
edge deployment is a stall.
What this saves in resident memory is deliberately *not* claimed. VmRSS on
this host moves 1.2-2.0 GB between identical runs, because it counts the
mmap'd checkpoint's file pages; an earlier 3.1 GB figure came from a
comparison that changed the model file as well as the budget and does not
stand. RssAnon is the metric to use and that measurement is still in flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The GPTQ checkpoint the comparison had been using left Spark's output head
tied and unquantized, so that one GEMM ran bf16 through `_C::onednn_mm` at
2.97 ms/step -- more than the whole 4-bit saving, and enough to make the
W4A8/W4A16 ordering flip between grids. The per-op profile is what caught it:
`int4_scaled_mm_cpu` ran 112 times (28 layers x 4 linears) and `onednn_mm`
once.
Rebuilt with the head quantized and re-profiled back to back on the same 24
cores, because this host's throughput swings ~30% and the gap is ~8%:
w4a8 AMX g128 GEMM 5.91 ms step 11.55 ms 86.6 tok/s
w4a16 tinygemm GEMM 6.85 ms step 12.51 ms 80.0 tok/s
llama.cpp Q4_K_M step 11.11 ms 90.0 tok/s
W4A8 wins twice over: the kernel is 0.94 ms cheaper per step, and it is
reached through vLLM's own dispatch, so it also avoids the 0.48 ms/step
custom-op boundary the tinygemm path needs to send prefill to a different
kernel. The selector's recommendation is unchanged; only its numbers were
wrong, and the lm_head trap is now documented where someone building a
checkpoint will read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`VLLM_CPU_KVCACHE_SPACE` is a server-shaped default and is parsed with
`int()`, so it cannot express an edge-sized cache at all -- the smallest
non-zero value it accepts is 1 GiB. Measured on Spark-X2.5-1.7B (Xeon 8480C,
2048-token single stream), passing `kv_cache_memory_bytes` sized from the
model instead of the 4 GiB default takes resident memory from 8329 MB to
5254 MB, a 3.1 GB saving with no effect on output.
`kv_budget_for` reads the config and prices the cache. It reports two numbers
because the model and the backend disagree: a hybrid-attention model does not
need a full-length cache in every layer -- Spark is 21 sliding (window 512)
to 7 full layers, so at 32 k context its real cache is 3.8x smaller than
`layers x context` -- but vLLM's CPU path does not take that discount today
(for a 4 GiB budget it reported 71 527 tokens, i.e. the flat 56 KiB/token).
So `bytes_total` is the flat figure, which is what is safe to pass now, and
`bytes_total_hybrid` is reported beside it for a backend that accounts per
layer type.
Deliberately conservative in three places, because a budget that is too small
is not a graceful degradation -- vLLM preempts by recompute, which on a
single-stream edge deployment is a stall: the window is rounded up to the
block size, a 1.25 headroom factor is applied, and a config that has a
`sliding_window` but no `layer_types` map is charged the flat rate rather than
assumed to apply the window everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four changes, each measured on a Xeon 8480C with Spark-X2.5-1.7B at 24
threads. Numbers are best-of-interleaved-passes; this host is shared and a
single sequential grid swung the llama.cpp control 30%, so anything under
~8% is not treated as a result.
**vLLM's CPU backend does have a 4-bit kernel.** An earlier note in this
tree said it did not, on the strength of `woq_int4_linear`,
`int4_scaled_mm_with_quant` and `da8w4_linear` all being absent. Those are
simply not the names it uses: `CPUWNA16LinearKernel` dispatches GPTQ/AWQ
checkpoints to `int4_scaled_mm_cpu`, an AMX W4A8 GEMM, whenever activations
are bf16 and the checkpoint carries no act-order. Four-bit weights times
eight-bit activations is the arithmetic llama.cpp's Q4_K_M already does.
`gptq_cpu_export` builds checkpoints for it; the kernel pins the zero point
at 8, so the grid must be symmetric, and a per-group scale search recovers
most of what that costs.
**A per-device weight path** (`edge/weight_path.py`). The fast kernel on each
device needs a different, mutually exclusive weight layout, so the choice has
to come from a probe rather than a flag: AMX x86 -> W4A8/GPTQ; x86 without
AMX, or priority="fidelity" -> W4A16/tinygemm (bf16 activations, the more
faithful of the two); ARM -> W4A16 but flagged unverified, because PyTorch's
4-bit CPU GEMM dispatches a different kernel there than the AVX-512 one
measured here; phone -> runs_in_vllm=False, the deliverable is an exported
graph with calibrated w8a16.
**The CPU compile cache never loaded.** `_try_load_aot_compiled_fn` finishes
with `compiled_fn.finalize_loading(cfg)`, which the CUDA wrapper object has
and the inductor CPU backend's plain function does not. The AttributeError
was swallowed as a cache miss, so every engine start recompiled the model:
warm start 49.8 s against 14.9 s with the missing no-op attached. Self-
disabling if upstream gives the CPU path a real one.
**The head gate stops being a GEMM.** A 2048x8 projection cannot amortize a
thread fan-out: dispatched as `aten::mm` it costs 1.155 ms of a decode step,
nearly all barrier. As an eager reduction behind an opaque op it costs 0.444
ms. It has to be opaque -- written inline the reduction is traced, and
inductor's loop made the whole step 1.8x slower (78.8 -> 44.5 tok/s).
Also included and *off by default*: `cpu_ir_ops` registers the fused
`_C::rms_norm` / `_C::fused_add_rms_norm` kernels that vLLM's CPU build ships
but gates on `GPGPU_DEVICE`. In isolation they are 3.6x and 4.5x faster than
the native path; end to end they are slightly slower (72.7 vs 74.0 tok/s),
because inductor was already fusing the native norm into its neighbours and
an opaque call breaks that fusion. Kept with the measurement recorded so the
next person does not repeat it. `gelu_and_mul` is deliberately not registered:
its fused CPU kernel is 2.7x slower at Spark's GEGLU shape.
Decode, W4A16 g64 -> W4A8 g128 with the gate fix: 69.1 -> 84.5 tok/s (+22%),
against llama.cpp Q4_K_M's 90.0; prefill 3347 against 1121 (3.0x). The two
4-bit paths are not equally faithful: against each build's own bf16 over 12
greedy prompts, Q4_K_M reproduces 4/12, W4A16 2/12 and W4A8 1/12.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Single-stream CPU decode reads every weight to emit one token, so bits per
weight set the token rate. This registers a W4A16 group-wise method
out-of-tree through register_quantization_config -- vLLM itself is untouched
-- using PyTorch's _weight_int4pack_mm_for_cpu tinygemm kernel.
Two paths over one set of weights, because that kernel dequantizes once per
row of activations and so is linear in the batch (12.9x slower than bf16 at
512 rows). Single-row steps take tinygemm; wider batches unpack the weight to
bf16 once and use the ordinary oneDNN/AMX GEMM. Shipping only the first
measured pp512 = 557 tok/s against bf16's 4455; with both, 3589.
The batch threshold is 2 rather than the ~8 rows where the unpack path starts
winning on speed, because running tinygemm on a multi-row batch and then
vLLM's cpu_attention_with_kv_cache in the same forward dies with SIGILL
inside the attention kernel -- an AMX instruction with no valid tile config
loaded. The same prompt is fine in bf16 and fine once every multi-row GEMM
takes the unpack path.
Positioning, measured on a Xeon 8480C with 32 pinned cores and a warm cache
(analysis/spark_cpu_int4.md): vLLM's own GPTQ W4A8 AMX path is faster --
88.7 tok/s decode and 4355 prefill against 85.5 and 3589 here -- but its
config rejects sym=False at 4 bits, so it can only be fed a symmetric grid.
This one is asymmetric and costs +12.5% perplexity against that path's
+27.2%, and needs no AMX. llama.cpp Q4_K_M, for reference, is 98.0 tok/s
decode and 1356 prefill at +7.6% and 5.15 bits per weight.
Expanding a quantized checkpoint back to bf16 offline and scoring it
reproduces the quantized run to within 0.2% perplexity, so both GEMM paths,
the nibble packing and the quantized tied embedding are faithful.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXhbAfkSNRaeqZrus5Z2fx
The score-space join removed the concat as intended and is numerically
identical (140.3 dB), but it cannot be quantized: replacing torch.softmax
with an explicit max/exp/sum/div made QNN's context-binary converter exit 14
under w4a16, while every fp16 build compiled. Two earlier shapes failed the
same way -- a broadcast multiply plus ReduceSum for the new token's score,
and a two-axis broadcast multiply for its value contribution.
The device profile says the win was never the concat anyway. Of a sliding
layer, Output was 9.9%, its quantise conversion 3.1% and the window Slice
7.4% -- all of it the cost of handing a 512-entry window back across the
graph boundary once per token. So keep roll's attention exactly as it was,
one fused Softmax over concatenated keys, and change only what the layer
returns: the single new K/V entry, which the runtime drops into the ring
slot. Ring order needs no fixing up, because softmax over keys is
order-independent and each cached key already carries its own rotary phase.
Measured on a real Galaxy S25, sliding layer, ms:
w4a16 0.917 -> 0.838 (-8.6%) <- now compiles
fp16 1.689 -> 1.486 (-12.0%)
Full-attention layers keep roll: ring gains them nothing (1.567 vs 1.559 at
fp16, the concat saved being spent on the extra score matmul), so the
default layout is now per layer type.
Parity against the reference is 140.3 dB for the mixed default with
identical top-1/top-5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Reading the device profile of the first export showed a fifth of every
sliding layer went nowhere useful: Output 9.9%, its quantise conversion
3.1%, and the window Slice 7.4%. The layer took a 512-entry window in,
appended the new key, sliced the oldest off and handed the whole recomputed
window back out -- 21 of 28 layers paying to move a cache-sized tensor
across the graph boundary once per token. Full layers paid a similar 9.5%
concatenating the new key onto the cache.
Both are the same mistake, and both go away by joining in score space
instead of key space: score the cache and the new token separately, then
concatenate two vectors of length n and 1 rather than an n x 256 key matrix.
Nothing cache-sized is copied, and each layer returns just the new K/V entry
for the runtime to drop into its ring slot. Softmax over keys does not care
what order the slots are in and each cached key already carries its own
rotary phase, so a ring buffer needs no reordering -- only a mask for slots
not yet written. The sliding cache correspondingly holds window-1 entries,
since a 512-token window counts the current token (transformers' own sliding
cache holds 511 for the same reason).
Measured on a real Galaxy S25, fp16, ms per layer:
sliding 1.689 -> 1.483 (-12.2%)
full 1.559 -> 1.567 (unchanged: the concat saved is spent on the
extra score matmul)
Two shapes had to be written as matmuls rather than the obvious elementwise
forms -- the new token's score (broadcast multiply + ReduceSum) and its
value contribution (a two-axis broadcast multiply). Both compile at fp16 and
both make QNN's context-binary converter exit 14 under w4a16; as matmuls
they compile and are the cheaper op on HTP anyway.
Also adds a selectable GELU approximation, and the measurement that closes
the question: erf GELU is 11-13% of every layer, but tanh is *slower* on
this NPU (1.535 vs 1.483 ms) despite holding 65.2 dB. The exact form stays,
which is also the better-fidelity choice.
Parity against the reference implementation is 136.8 dB with identical
top-1/top-5, and a test pins ring and roll to the same output on a shared
history so the device numbers cannot silently measure a different model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Exports one Spark decode step with the KV cache as explicit tensors, sized
per layer type: the 21 sliding layers hold a fixed 512 entries at any
context, the 7 full-attention layers grow. That asymmetry is the whole
reason the architecture is interesting on a phone, so the two types get
separately sized caches, masks and rotary angles.
Verified against the HF reference at 1024 context: 138.5 dB logits SNR,
identical top-1 and top-5.
Profiled on a real Galaxy S25 (Snapdragon 8 Elite) through AI Hub, QNN
context binary on the NPU, ms per layer:
sliding (512 KV, any ctx) fp16 1.689 w8a16 0.912 w4a16 0.917
full @ 1k fp16 1.559 w8a16 1.129 w4a16 1.127
full @ 4k fp16 2.964 w8a16 1.681 w4a16 1.662
Composed into a step (21 sliding + 7 full), w4a16 gives 27.2 ms at 1k and
30.9 ms at 4k, against 31.6 and 46.5 ms for the same model with every layer
full-attention -- 1.16x and 1.51x. Worth noting the hybrid is 0.94x at 1k in
fp16: sliding layers rotate all 256 head dims where full layers rotate 64,
and below roughly 2k that RoPE cost outweighs the shorter cache.
Also moves the head-wise attention gate off vLLM's CPU linear dispatch. The
gate is [2048, 8], and any linear that narrow is routed to the sgl-kernel
packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim
0; VNNI packing does not at that size, so it reports K and torch.compile
fails tracing the gate multiply. A plain parameter with F.linear gains
nothing less -- a 2048x8 matmul has no use for packed AMX tiles -- and it
keeps the model compilable on CPU and GPU alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Spark-X2.5-1.7B is an Apache-2.0 on-device model built for agentic tool
use. vLLM 0.28 has no implementation for it, so it could not be served at
all; llama.cpp already ships one, which made it the only edge engine able
to run the model.
Three traits separate it from a Llama-style decoder:
* Hybrid attention -- layer_types interleaves three 512-token sliding
layers with one full-attention layer, which is what makes the 1M-token
context affordable on a phone.
* Per-layer-type RoPE -- sliding layers rotate all 256 head dims with
theta 1e4, full layers only the first quarter with theta 5e6.
* A head-wise attention output gate -- g_proj reads the same post-norm
input as QKV and scales each head's attention output by a sigmoid.
The checkpoint also stores QKV pre-fused as a single q_k_v_proj, so
load_weights splits it before vLLM's sharded QKVParallelLinear sees it.
Verified against the official remote-code implementation on transformers
4.57.1 (the version config.json targets; it does not import on the 5.x in
our venvs). In fp32 all three scenarios match exactly -- 32/32 greedy
tokens with a max top-20 logprob difference of 2.2e-05. In bf16 the short
and the 2465-token long-context prompts still match 32/32, which is what
exercises the sliding layers; the tool-calling prompt diverges at token 23
on a near-tie, and fp32 confirms that is rounding rather than a defect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
The Code2Wav decoder already had an on-device export (WP4); the talker and its
code predictor did not, so the codec-token path stopped at the vocoder. This
adds the missing half: a one-token decode step with an explicit per-layer KV
cache, exportable to ONNX and from there to a QNN context binary, TFLite or
ONNX Runtime.
Two layout choices carry the performance, both measured on real Snapdragon
devices through Qualcomm AI Hub (Galaxy S24 = 8 Gen 3, S25 = 8 Elite):
* per-layer cache tensors instead of one stacked [num_layers, ...] tensor that
every layer slices, and
* grouped-query attention by reshaping the query rather than expanding keys and
values.
The stacked/expanded form spent ~70% of its NPU cycles in cache select,
reshape and fp32->fp16 conversion; the layout here is bandwidth-bound and runs
the 0.6B talker step in 15.8 ms instead of 36.9 ms on an 8 Elite (fp16,
256-token cache). The code predictor step is 2.25 ms, so talker plus its 15
sub-steps cost about 50 ms of an 80 ms frame.
sanitize_onnx() fixes the three things the dynamo exporter emits that
qairt-converter and AI Hub reject: IsNaN/Where guards, Reshape allowzero=1, and
rank-0 axes initializers on Unsqueeze/Squeeze.
The module docstring records the measured placement table: this graph belongs
on the NPU, the vocoder on the GPU (21.7 ms vs 182 ms per frame), which is 3.3x
better than sending everything to the NPU and makes the pipeline real time on
the phone in fp16. Integer quantization is not the lever it looks like: every
automatic int8 build measured was numerically broken.
Full write-up, job ids and profiles: analysis/mobile_htp_simulation.md section 5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dzr1Jusv4s6g765gqS9L92
tzhouam