vllm-omni/docs/analysis/spark_edge_support.md

Spark-X2.5-1.7B on CPU and Galaxy S25

Spark-X2.5 (XHToken/iFlytek, Apache 2.0, released 2026-09-01) is a 1.7 B on-device model built for agentic tool use: BFCL-V4 46.9, tau-squared-bench 65.3, a native 1 M-token context and 200+ languages. This note covers adding it to vLLM-Omni, measuring it against every engine that can run it, putting it on a real Snapdragon 8 Elite, and driving an Android phone with it.

Reproduction scripts: analysis/experiments/spark_edge/. Model code: vllm_omni/model_executor/models/spark2_5/, exporter vllm_omni/edge/spark_export.py.

1. What had to be built

Three traits separate Spark from a Llama-style decoder, and all three are why no generic converter picks it up:

trait	detail	why it matters
hybrid attention	layer_types = 3x sliding_attention (window 512) then 1x full_attention, 28 layers -> 21 sliding + 7 full	bounds KV growth; this is what makes 1 M context affordable on a phone
per-layer-type RoPE	sliding: all 256 head dims, theta 1e4. full: first 64 dims only (partial_rotary_factor 0.25), theta 5e6	two rotary tables, not one
head-wise output gate	g_proj [2048 -> 8] reads the same post-norm input as QKV; sigmoid scales each head's attention output before out_proj	an extra projection with no analogue in Llama/Qwen

Also: QKV ships pre-fused as one q_k_v_proj, the MLP is GEGLU with exact gelu, embeddings are tied, and head_dim is 256 with only 2 KV heads.

Engine support before this work

engine	Spark-X2.5 support
llama.cpp	yes, native (src/models/spark2-5.cpp)
SGLang	yes, native (srt/models/spark2_5.py)
vLLM 0.28	no -- added here
MNN	no. llmexport.py dispatches on a fixed model_type list; spark2_5 is absent
ExecuTorch	no
ncnn	no

MNN, ExecuTorch and ncnn would each need the architecture written from scratch. That is the honest reason they do not appear in the benchmark tables below: they cannot load the model at all, not that they are slow at it.

Correctness

Verified against the official remote-code implementation pinned to transformers 4.57.1 (the version config.json targets; it does not import on the 5.x in our venvs, so a dedicated venv runs ground truth).

scenario	prompt	fp32	bf16
short	28 tok	32/32 greedy, max logprob diff 2.2e-05	24/24 greedy
long (exercises sliding window)	2465 tok	--	32/32 greedy
tool calling	137 tok	32/32 greedy	diverges at token 23

The bf16 tool-calling divergence is a near-tie under greedy decoding, not a defect: in fp32 the same prompt matches exactly with a 2.2e-05 logprob difference.

One vLLM bug had to be worked around. The head-wise gate is a [2048, 8] projection, and vLLM's CPU backend routes any narrow linear to the sgl-kernel packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim 0. VNNI packing does not for N that small, so it reports K instead of N and torch.compile dies tracing the gate multiply. The gate is kept as a plain parameter with F.linear -- a 2048x8 matmul gains nothing from the packed AMX kernel anyway.

2. CPU: vLLM-Omni vs llama.cpp

Xeon Platinum 8480C (Sapphire Rapids, AMX-BF16/INT8). Both engines pinned with taskset to the same physical cores and run strictly one at a time; prefix caching disabled on vLLM so a replayed prompt is really recomputed (llama-bench has no such cache). pp512 = prefill throughput, tg128 = decode.

Same precision (bf16)

threads	vLLM pp512	llama.cpp pp512	prefill	vLLM tg128	llama.cpp tg128	decode
4	1113.4	72.4	15.4x	11.11	4.20	2.65x
8	2113.7	173.8	12.2x	19.88	16.15	1.23x
16	3639.4	341.7	10.7x	30.16	25.31	1.19x
32	4454.9	509.5	8.7x	36.35	28.57	1.27x

vLLM wins both, at every thread count. The prefill gap is AMX: llama.cpp's bf16 path runs at AVX-512 speed, vLLM's goes through oneDNN/AMX tiles.

Against llama.cpp's quantized builds

threads	vLLM bf16 pp	llama.cpp Q8_0 pp	llama.cpp Q4_K_M pp	vLLM bf16 tg	llama.cpp Q8_0 tg	llama.cpp Q4_K_M tg
4	1113.4	118.1	103.6	11.11	9.85	10.36
8	2113.7	490.2	434.2	19.88	45.01	42.40
16	3639.4	914.7	806.3	30.16	66.23	71.50
32	4454.9	1493.1	1199.0	36.35	76.54	82.81

Prefill: vLLM at bf16 still beats llama.cpp's best quantized build by 3.0x at 32 threads. Prefill is compute-bound, and AMX beats 4-bit weight savings.

Decode: llama.cpp Q4_K_M wins from 8 threads up, by 2.3x at 32. Decode is memory-bound -- 4-bit weights move a quarter of the bytes -- and this is a precision difference, not an engine difference. Closing it needs a quantized vLLM CPU path for Spark, which is the open item in section 6.

Quantized: int8 on CPU

vLLM 0.28's CPU build has int8 dense kernels (int8_scaled_mm_with_quant, dynamic_scaled_int8_quant), and static_scaled_fp8_quant is registered with an empty schema, so on-the-fly fp8 fails outright.

An earlier draft of this note said the backend had no dense int4 kernel either, on the strength of woq_int4_linear, int4_scaled_mm_with_quant and da8w4_linear all being absent. That was wrong -- they are simply not the names it uses. See section 2c: there are two four-bit paths, and the one vLLM ships is int4_scaled_mm_cpu, an AMX W4A8 GEMM reached through the GPTQ/AWQ kernel selector.

Int8 is also available. analysis/experiments/spark_edge/quantize_int8.py writes a compressed-tensors W8A8 checkpoint straight from the safetensors (symmetric per-output-channel round-to-nearest; no calibration needed, and llmcompressor is avoided because it pulls transformers 5.x, on which Spark's remote code does not import). The head-wise gate is left in full precision: its eight outputs each scale a whole attention head, and it is 0.001% of the weights. Weight SNR 36.9-40.9 dB across 140 tensors.

threads	pp512 bf16 -> int8	tg128 bf16 -> int8
16	3639 -> 5135 (1.41x)	30.16 -> 48.30 (1.60x)
32	4455 -> 6978 (1.57x)	36.35 -> 58.44 (1.61x)

Quality against the fp32 reference, though, is not free:

scenario	greedy match	first-token top-1	top-20 overlap	max logprob diff
short (28 tok)	23/24	agrees	18/20	0.83
long (2465 tok)	12/32	agrees	19/20	1.56
tool calling	31/32	agrees	18/20	0.58

bf16 matched 32/32 on all three with a max difference of 0.24. Int8 keeps the first token everywhere, but at long context the distribution has moved enough that greedy output diverges after a dozen tokens. Usable, not free -- the same caution the Qwen3-TTS and MiniCPM-o work in this repo reached about automatic int8.

2b. CPU under concurrency

Single-stream decode is memory-bound, which is a precision contest llama.cpp wins with 4-bit weights. Concurrent decode is compute-bound, which is an AMX and continuous-batching contest. Total generated tokens/s, 32 pinned cores, 128-token prompts generating 128 tokens each:

batch	vLLM bf16	vLLM int8	llama.cpp Q4_K_M	int8 vs llama.cpp
1	34.6	54.9	91.1	0.60x
2	69.9	105.2	129.6	0.81x
4	139.8	210.3	251.6	0.84x
8	266.1	387.8	406.5	0.95x
16	495.7	694.7	613.9	1.13x
32	--	1080.6	778.3	1.39x
64	--	1557.9	904.5	1.72x

The crossover is around batch 12. Below it llama.cpp's 4-bit weights win; above it vLLM's AMX-INT8 GEMMs and continuous batching pull away, reaching 1.72x at batch 64 and still climbing. A phone agent is a batch-of-one workload, so llama.cpp is the right single-device choice today; an edge gateway serving a dozen sessions is vLLM's.

2c. Four bits on CPU: two paths, and which one a device should take

vLLM 0.28's CPU build has a 4-bit GEMM. CPUWNA16LinearKernel (vllm/model_executor/kernels/linear/mixed_precision/cpu.py) routes GPTQ and AWQ checkpoints to int4_scaled_mm_cpu, an AMX W4A8 kernel, whenever the activations are bf16, the ISA has AMX tiles and the checkpoint carries no activation reordering. Four-bit weights times eight-bit activations is exactly what llama.cpp's Q4_K_M does (Q4_K weights, Q8_K activations), so this is the honest precision match rather than a handicap.

Two paths were built, because they are not the same trade:

	weights	activations	kernel	bits/weight	reached by
W4A8	4-bit symmetric, group-wise	int8 (per the kernel)	_C::int4_scaled_mm_cpu (AMX)	4.16 @ g128	GPTQ checkpoint, vLLM's own dispatch
W4A16	4-bit asymmetric min/max, group-wise	bf16	aten::_weight_int4pack_mm_for_cpu	4.16 @ g128	cpu_int4, registered out-of-tree

W4A8 is forced symmetric: _process_gptq_weights_w4a8 builds its own zero point (a constant 8) and ignores the checkpoint's qzeros, so the grid must be w = (code - 8) * scale. W4A16 is free to use an asymmetric min/max grid, and a per-group scale search recovers roughly a dB in either case.

Which is faster

Per-op profiles, taken back to back on the same 24 cores. Throughput on this host swings ~30% run to run and the gap here is ~8%, so the GEMM self time is the measurement that can actually rank them:

path	GEMM/step	step	tok/s
W4A8 g128	5.91 ms	11.55 ms	86.6
W4A16 g128	6.85 ms	12.51 ms	80.0
llama.cpp Q4_K_M	--	11.11 ms	90.0
vLLM bf16	--	--	41.0

W4A8 wins twice: the kernel is 0.94 ms cheaper per step, and because it is reached through vLLM's own dispatch it also avoids the 0.48 ms/step custom-op boundary the tinygemm path needs (that boundary exists because tinygemm's cost is linear in batch size, so prefill has to be routed to a dequantize-then-AMX path instead -- see below).

The trap that cost a day: a GPTQ checkpoint must quantize the output head. Spark ties its embedding to its head, so a converter that leaves the tie in place emits no lm_head tensors and that one GEMM runs bf16 through _C::onednn_mm -- 2.97 ms per step, more than the entire 4-bit saving. The profile is what caught it: int4_scaled_mm_cpu ran 112 times (28 layers x 4 linears) and onednn_mm once. Every "W4A8" number measured before that was on a handicapped build, which is why the W4A8/W4A16 ordering appeared to flip between grids.

Which is more faithful

Greedy divergence from each build's own bf16 output, 12 prompts, measured the same way on both engines:

build	exact match	mean prefix kept
llama.cpp Q4_K_M	4/12	0.51
vLLM W4A16	2/12	0.30
vLLM W4A8	1/12	0.13

Q4_K_M's 6-bit sub-block scales buy real granularity at the same bit budget; round-to-nearest with a per-group search does not match it. So the two paths are a genuine choice, not a ranking: W4A8 for speed, W4A16 for fidelity, and vllm_omni/edge/weight_path.py makes that choice from a hardware probe rather than a flag, because the fast kernel on each device needs a different and mutually exclusive weight layout:

device class	path	why
x86 + AMX	W4A8 / GPTQ	int4_scaled_mm_cpu applies
x86 without AMX	W4A16 / tinygemm	no AMX, no int4 kernel
x86 + AMX, fidelity asked	W4A16 / tinygemm	keeps bf16 activations
ARM64	W4A16, unverified	PyTorch's 4-bit CPU GEMM dispatches a different kernel there
phone NPU	exported graph, calibrated w8a16	vLLM does not run there at all

Prefill needs the other kernel

The tinygemm kernel dequantizes once per row of activations, so its cost is linear in batch size: it beats AMX bf16 below ~8 rows and loses badly above. Measured at Spark's gate_up shape, per call: 0.070 ms at 1 token, 14.2 ms at 512. Left alone, pp512 collapsed from 4455 tok/s (bf16) to 557. Unpacking the weight to bf16 once per call and handing it to the ordinary oneDNN/AMX GEMM brings it back to 3347 tok/s, 3.0x llama.cpp's 1121. W4A8 needs none of this -- the AMX kernel is the right one at every batch size, which is the second reason it is the default on an AMX machine.

2d. Init time and resident memory, which is what decides edge at all

Speed is not the binding constraint on an edge box; a 90-second start and 8 GB resident is. Both turned out to be fixable, and one of them was a bug.

vLLM recompiled the model on every single start

_try_load_aot_compiled_fn finishes a cache load with loaded_fn._artifacts.compiled_fn.finalize_loading(cfg). On CUDA that artifact is a wrapper object carrying the method; with the inductor CPU backend it is a plain function, so the call raises AttributeError, the handler treats the whole load as a cache miss ("Compiling model again due to a load failure"), and the model is compiled again from scratch. Every restart paid it.

Attaching the missing no-op (a plain Python function accepts attribute assignment; the CPU artifact loads its generated .so lazily on first call, which is the warm-up run) fixes it. Same cache directory, one variable changed:

	init
cold cache (first ever compile)	93.8 s
warm cache, vLLM as shipped	49.8 s
warm cache, shim installed	14.9 s

3.3x faster warm start, reproduced across passes (14.9 / 14.7 / 15.0, and 12.8-13.0 on the W4A16 path). The shim is self-disabling: if the artifact already has finalize_loading, as it does on GPU and will if upstream gives the CPU path one, it changes nothing.

Measure anonymous memory, not RSS

A first pass concluded that sizing the KV cache saved 3.1 GB. It does not, and the error is worth recording because it is easy to repeat: that comparison read VmRSS, which counts the mmap'd checkpoint's file pages, and those come and go with page-cache pressure. Identical configurations measured 1.2-2.0 GB apart. RssAnon -- allocations only -- repeats to within 0.5 MB, and is what a memory budget is actually about.

The KV cache is not a memory lever on CPU

Re-measured properly, with only the budget changing:

KV budget	anonymous RSS
4 GiB (env default)	7601 MB
512 MiB	7405 MB
128 MiB	7413 MB

Shrinking the cache 32x moves resident memory by 188 MB, and 512 MiB is indistinguishable from 128 MiB. The cache is allocated but its pages are never touched, so it costs address space, not memory.

It still has to be correct, which is a separate problem: --kv-cache-memory-bytes was silently discarded on CPU. CpuPlatform.check_and_update_config ends with a block its own comment calls "Lagecy setting" that overwrites cache_config.kv_cache_memory_bytes from VLLM_CPU_KVCACHE_SPACE unconditionally, so the budget could only be set in whole GiB and vllm-omni's per-stage budgets had no effect at all. Verified by allocation rather than by memory, because allocation cannot drift: with the env at 1 GiB, asking for 128 MiB produced the same 17 881 cache tokens as asking for nothing, and with the env at 4 GiB it produced 71 527. After restoring the obvious precedence the same request yields 2 231 tokens. That matters for admission -- vLLM preempts by recompute when the cache cannot hold max_num_seqs x max_model_len, which on a single-stream deployment is a stall -- not for footprint. vllm_omni/edge/kv_budget.py prices the cache from the config, reporting the flat figure (what is safe to pass, since vLLM's CPU accounting is flat) beside the hybrid one a layer-type-aware backend could use.

Where the memory actually goes: the weight load

Split by phase, on anonymous RSS:

phase	W4A8 (GPTQ)	W4A16 (tinygemm)
before import	12 MB	12 MB
after import vllm_omni	513 MB	515 MB
after engine construction	7338 MB	5023 MB
after releasing free arenas	3510 MB	3411 MB

Imports cost half a gigabyte -- they are the init-time problem, not the memory one. The engine adds 4.5-6.8 GB for a model whose weights are under 1.4 GB, and most of it is not live: loading a quantized checkpoint unpacks 4-bit codes to int32 -- four bytes per weight -- to permute and repack them, and although the intermediates are freed the allocator keeps the arenas. One release call returns 3828 MB on the GPTQ path and 1612 MB on tinygemm, a 52% cut on the configuration we actually want to deploy.

The second reading of that table matters more than the first: after the release both paths land in the same place. W4A8's apparent 2.3 GB penalty over W4A16 was never a footprint difference, it was the repack. So W4A8 wins on speed and ties on memory, and fidelity is the only remaining reason to choose W4A16.

vllm_omni/edge/memory.py does the release and patch.py hooks it to the CPU worker's load and warm-up (VLLM_OMNI_CPU_RELEASE_MEMORY=0 to disable). The ~3.5 GB that remains, against 0.86 GB of weights, is the next thing to chase.

Two bugs that made CPU deployments take the slow path silently

Init is not one number per model; it has a different cause in each pipeline, and in two places the slow path was taken without saying so.

Parallel stage init could never run on CPU. resolve_stage_physical_devices returns None for devices: "cpu", and the admission resolver fails closed on None -- right for a GPU replica whose devices cannot be accounted, wrong here -- so every multi-stage CPU pipeline raised StageAdmissionError and fell back to sequential startup. The edge profile sets parallel_stage_init: true, so on CPU that profile was broken rather than merely slower.

The fix is not to skip admission. The per-device ledger is vacuous on CPU, but the resource concurrent init contends for is host memory, and two stages allocating KV caches at once exhaust it exactly as two GPU stages exhaust a card. check_host_admission applies the same discipline to that resource: sum the declared budgets, compare with available RAM and a margin, refuse to start otherwise, and fail closed on a stage that declares no budget.

Qwen3-TTS 0.6B, init only
sequential	64.57 s
parallel	30.54 s

A deploy YAML meant different things depending on which flag loaded it. The orchestrator: block carrying parallel_stage_init was applied only on the deploy_profile path, so pointing --deploy-config at the very same file loaded its stages and dropped the flag: 63.76 s against 30.76 s, with nothing in the log to explain the difference. The block now applies either way.

What is not a memory lever

Two candidates tested and closed, so they are not re-tried:

lever	effect
OMP thread count (8 / 24 / 48)	3754 / 3775 / 3818 MB -- 64 MB across a 6x range

Retracted (2026-09-14): KV cache size was on this list, and it is the single largest lever there is. The row read "4 GiB -> 128 MiB saves 188 MB, the pages are never touched". Re-measured now that the budget is actually honoured -- 3 interleaved passes per arm, both filling a 1536-token context:

KV budget	resident
1 GiB (the smallest VLLM_CPU_KVCACHE_SPACE can express)	3264 MB
192 MiB, explicit	2421 MB

843 MB. And mincore(2) says why the old reasoning was wrong: the cache is not allocated-and-untouched, it is fully in core -- 1023.8 MB allocated against 1024.1 MB resident, and 190.8 against 191.1 when sized down.

The explanation for the original 188 MB is two sections below, in this report's own words: --kv-cache-memory-bytes was silently discarded on CPU, because CpuPlatform.check_and_update_config overwrote it from VLLM_CPU_KVCACHE_SPACE unconditionally. So the "32x shrink" never happened; both arms ran the same cache and the 188 MB was measuring something else. The lesson is the one this study keeps relearning: a lever tested through a path that silently ignores it measures nothing, and "no effect" is the most dangerous result to accept without checking that the knob was connected.

The dual weight layout the W4A16 path keeps (tinygemm for decode, a row-major copy for prefill) is one. Median of three interleaved passes, anonymous RSS, spread under 1.5%:

build	resident	prefill	decode	init
W4A8 g128	3640 MB	--	86.6 tok/s	15.0 s
W4A16 g128	3542 MB	3252 tok/s	76.9 tok/s	12.4 s
W4A16, prefill_dequant: false	2802 MB	448 tok/s	78.5 tok/s	16.0 s

The second layout costs 740 MB and buys 7.3x prefill, because the tinygemm kernel dequantizes once per row of activations and its cost is linear in the batch.

That trade-off is smaller than it looked, but it is real (2026-09-13). It rested partly on treating the tinygemm layout as opaque, so a row-major duplicate was kept beside it to have something to dequantize from -- but the duplicate was also doing work: to_split_halves performs the nibble de-interleave once at load, so every prefill call reads two contiguous planes. It is a precomputed permutation, not just a copy. Measured on this host instead of assumed -- by packing an index pattern and reading back where each nibble landed -- the layout is [N/64][K][32 bytes]: K is the middle axis, so a K range is contiguous within each n-block, and inside every 32-byte run byte d carries output channel d in its low nibble and d + 32 in its high one, the same permutation for every k and every block. Decoding it is a deinterleave, not a reverse-engineered ISA table.

So the prefill path can dequantize from the kernel's own layout, and then the duplicate is not allocated -- at the cost of redoing that de-interleave on every call. The reconstruction is bit-exact against the previous dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048. Shapes where N does not tile the 64-channel block (N=80 is 64+16) keep the copy rather than mis-read the layout; every Spark linear tiles cleanly.

Measured with VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT toggling the two paths in otherwise identical code, 3 arms x 2 interleaved passes, anonymous RSS summed over the process tree, 1536-token context:

build	resident (median)	peak	spread
W4A16, second copy (old)	4591 MB	5508 MB	0.2%
W4A16, one layout	3976 MB	4593 MB	5.4%
W4A16, prefill_dequant: false	3902 MB	4522 MB	2.6%

615 MB of resident memory and ~915 MB of peak, and the one-layout build is statistically indistinguishable from prefill_dequant: false -- the harness refused to rank them (0.4% apart, inside the measurement's own spread). The arithmetic predicts 764 MB from the config; ~150 MB of that is allocator retention malloc_trim does not return.

It is not free. Prefill, pp512, pooled over 7 interleaved passes per arm on idle node-1 cores with memory bound to the same node:

build	resident	pp512 median	range	spread
two layouts (old default)	4591 MB	2190 tok/s	2117-2345	9.7%
one layout, direct dequant	3976 MB	1344 tok/s	1271-1555	18.3%
prefill_dequant: false	3902 MB	453 tok/s	451-453	0.3%

So it costs 1.63x prefill on medians -- and the ratio itself is only good to about 1.4-1.8x, because both dequantizing arms swing ~10-18% on this shared host while the tinygemm-only arm reproduces to 0.3%. The existence and rough size of the cost is solid (a 63% gap is far outside that band); the third digit is not. An earlier draft of this section said 1.9x, taken from the single surviving pass of a grid whose other passes had crashed -- a number the harness itself had labelled "single pass -- not a result".

The mechanism is that undoing the kernel's interleave per call costs ~1.6x reading the pre-permuted planes (14.4 ms vs 9.2 ms per gate_up dequantize at 8 threads). Two variants were tried and rejected: dequantizing to [K, N] and using matmul instead of F.linear, which avoids the element-wise transpose, measured worse at 20.3 ms, and the blocked form needs either 208 tiny GEMMs or the same permutation.

What this changes is the shape of the choice, not its existence:

• against prefill_dequant: false it is strictly better -- the same memory to within the noise, 2.97x the prefill, and it does not route multi-row batches into the tinygemm/AMX SIGILL hazard documented in cpu_int4.py. That setting is now dominated and should not be used. This comparison is the solid one: the tinygemm-only arm reproduces to 0.3%.
• against the old default it is a real dial: 615 MB for ~1.6x prefill. On a device chosen for its memory ceiling that is the right side of the trade; on one with headroom it is not. direct_dequant: false in the checkpoint's quantization_config selects the old behaviour -- a config field rather than only an env var, because the two paths build different graphs and must not share an AOT compile artifact (see below).

Recovering the prefill would take a C++ de-interleave rather than an ATen expression chain, which is a real kernel and not in scope here.

One hazard found while measuring this, worth recording because it presented as a crash rather than a wrong number. The two paths build different graphs -- one passes weight_halves into the custom op, the other passes None -- but vLLM keys its AOT compile cache on the config, so an env-var-only switch shares one cached artifact between them. Three passes died with KeyError: 'weight_halves' inside an AOT-loaded forward after the first pass saved an artifact for the other path. The choice is therefore a quantization_config field, which ModelConfig.compute_hash does hash; the env override remains for experiments and needs its own VLLM_CACHE_ROOT.

The largest memory item was a RoPE table for a context nobody asks for

The report used to account for memory by subtraction -- weights, imports, and an "unexplained ~1.6 GB" left over. A remainder computed from two uncertain terms is not a measurement, so memory_ledger.py counts instead: unique tensor storages keyed by data_ptr (so tied weights are counted once), the KV cache found by attribute, and every /proc/self/smaps mapping bucketed. Allocated is separated from resident with mincore(2), because a cache that was allocated and never written costs address space, not memory, and counting it would hide the remainder rather than explain it.

The first run put 640 MB in an "other" bucket across 114 tensors. Naming the largest storages showed what it was:

model.layers.0.self_attn.rotary_emb.cos_sin_cache   512.0 MB   [1048576, 256]
model.layers.3.self_attn.rotary_emb.cos_sin_cache   128.0 MB   [1048576, 64]

Spark advertises a 1 048 576-token context, and spark2_5.py passed max_position=config.max_position_embeddings straight into get_rope, so vLLM built torch.arange(1048576) cos/sin tables -- two of them, because the sliding and full layers are trained with different theta and different partial rotary factors, giving one 256-wide table and one 64-wide one. Both fully resident by mincore. For a deployment whose max_model_len is 2048 they need 1.25 MB between them.

vLLM's scheduler cannot produce a position beyond max_model_len, so the table only has to cover that. Capping it, 3 interleaved passes per arm with VLLM_OMNI_SPARK_ROPE_FULL toggling the two:

	settled	peak	init
1M-position tables	3891 MB	4019 MB	13.5 s
capped to max_model_len	3273 MB	3370 MB	13.7 s

618 MB settled and 649 MB of peak, for one line and no measurable init cost -- the largest single item found in this study, larger than the single weight layout (615 MB) or the blocked repack (489 MB), and unlike those two it costs nothing at all. The ledger's storage accounting agrees independently: the "other" bucket falls from 640.0 MB to 1.3 MB and unique tensor storage drops 638.8 MB.

Two things the same ledger exposes, not yet acted on:

• The KV cache is fully resident -- 1023.8 MB allocated, 1024.1 MB resident by mincore. The "not a memory lever" verdict below was measured while generating 8 tokens; at page level the cache is entirely in core, and it is 27% of total memory. That conclusion needs re-opening on this evidence.
• embed_tokens.weight_packed (128 MB) is kept only because the tinygemm layout was believed un-addressable for a single-row lookup. That premise is now known to be false -- the deinterleave above reaches any row -- so the row-major copy is probably removable too.

Peak load memory: the number an edge device actually has to survive

Everything above reports memory after the allocator is asked for its arenas back. That is the wrong target for a device: release_after_load() runs only once load_model and warm-up have both returned, so a box that cannot hold the peak never reaches the release. Sampling RssAnon on a timer through the load (rather than reading the endpoint) put the W4A16 peak at 5722-5837 MB against ~4480 settled -- a 1.2-1.3 GB spike nothing in the tree had measured.

Most of it was in our own repack path. unpack_nibbles expands 4-bit codes to int32 -- 4 bytes per weight, eight times the packed form -- because the kernel's packer wants int32, and doing that for a whole matrix makes the transient the largest object in the process:

tensor	shape	int32 expansion
gate_up	13312 x 2048	104 MiB
down_proj	2048 x 6656	52 MiB
q_k_v_proj	3072 x 2048	24 MiB
embedding (tied, so also the head)	131072 x 2048	1024 MiB

against 24 MiB of packed weights for an entire layer.

The packed layout is [N/64][K][32 bytes], so n-blocks are independent and the expansion can cover a few thousand rows at a time -- verified byte-identical to packing the whole matrix at four shapes plus a ragged last chunk. Measured with VLLM_OMNI_CPU_INT4_REPACK_ROWS toggling the two in identical code, 3 interleaved passes, node-local cores:

	peak	settled	load-time transient	init
whole-matrix repack	4502 MB	3905 MB	550 MB	17.7 s
blocked repack	4013 MB	3892 MB	105 MB	16.6 s

489 MB of peak, for nothing. Settled memory and init time are unchanged, and the repack is in fact faster chunked than whole -- 79 ms against 208 ms for the embedding at 24 threads, because the working set stays in cache. The chunk size is tuned from that sweep: 2048 rows is both the fastest and nearly the leanest (16 MiB transient), where 512 costs 116 ms and 32768 costs 129 ms.

Two things this nearly got wrong, both caught by re-measuring rather than reasoning. The first blocked pass showed init at 50.2 s against 17.7 -- but it is the one pass the harness flagged [busy] at load 34.8, and the two clean passes read 16.6 and 16.2. And its settled memory looked 239 MB worse, which was pass-1 scatter: the medians are 3892 against 3905.

The absolute peak improves less than the 1 GB the arithmetic predicts, because the single-layout change above had already removed to_split_halves, which was doing its own whole-matrix expansion. The two items overlap and their savings must not be added.

Kernel fusion: a negative result

vLLM's CPU build ships _C::rms_norm and _C::fused_add_rms_norm, and vllm/kernels/vllm_c.py already wraps both as IR-op implementations -- but gates them on GPGPU_DEVICE = is_cuda_alike() or is_xpu(), so on CPU every norm falls through to the PyTorch path. In isolation the fused kernels are 3.6x and 4.5x faster at Spark's decode shape (5.76 vs 20.99 us; 9.24 vs 41.14 us). End to end they are slightly slower: 72.7 vs 74.0 tok/s.

The reason is that the native norm was never running alone -- inductor fuses it into the residual add and the surrounding elementwise work, and an opaque _C call breaks that fusion for more than the norm itself saves. vllm_omni/model_executor/layers/cpu_ir_ops.py keeps the registration, off by default (VLLM_OMNI_CPU_FUSED_NORMS=1 to try it on another machine), with the measurement recorded. gelu_and_mul is deliberately not registered at all: its fused CPU kernel is 2.7x slower than native at Spark's GEGLU shape (64.11 vs 23.36 us), and GeluAndMul.forward_cpu in vLLM only reaches for it on POWERPC anyway.

Two more fusion attempts that lost, recorded so they are not repeated: cpp.min_chunk_size raised to keep inductor's small elementwise kernels single-threaded cost 6% of decode and 19% of prefill; and the head gate written as an inline reduction (rather than behind an opaque op) made the whole step 1.8x slower -- 78.8 to 44.5 tok/s -- because inductor's generated loop is worse than the matmul it replaced.

3. GPU: vLLM-Omni vs SGLang

One L20X (140 GiB HBM), bf16, single sequence, caches disabled on both (enable_prefix_caching=False / disable_radix_cache=True), 2048-token prefill so the measurement is not dominated by dispatch overhead.

engine	init	pp2048	tg128
vLLM-Omni	111.3 s	137 102 tok/s	451.4 tok/s
SGLang 0.5.19	97.1 s	142 253 tok/s	434.7 tok/s

A tie within 4% either way: SGLang +3.8% on prefill, vLLM +3.8% on decode.

SGLang has an Intel AMX CPU backend, but the released sgl-kernel wheel does not ship the CPU ops it needs (convert_weight_packed is absent), so SGLang could not be measured on CPU without a source build. On the two targets this goal is actually about -- CPU and a phone -- SGLang does not run at all today.

4. Galaxy S25 (Snapdragon 8 Elite), real device

vllm_omni/edge/spark_export.py exports one decode step with the KV cache as explicit tensors, sized per layer type: sliding layers get a fixed window at any context, full layers grow. Parity against the HF reference is 140.3 dB with identical top-1 and top-5.

What the device profile said to fix

AI Hub returns per-op cycle counts, and they made the first export's mistake obvious. Of a sliding layer at w4a16:

op	share	what it was
Output	9.9%	handing the recomputed 512-entry window back
output_..._converted	3.1%	quantising that window on the way out
Slice x3	7.4%	rolling the window to drop the oldest entry
subtotal	20.4%	moving a cache-sized tensor across the graph boundary, once per token, in 21 of 28 layers

Full-attention layers spent a similar 9.5% on Concat, appending the new key to the cache.

The fix is a ring buffer: the runtime owns a fixed-size cache and writes the new entry into the slot for this position, so the graph returns one K/V entry instead of a window and never rolls anything. Ring order needs no fixing up -- softmax over keys is order-independent and each cached key already carries its own rotary phase, so only slots not yet written need masking. The sliding cache correspondingly holds window - 1 entries, because a 512-token window counts the current token (transformers' own sliding cache holds 511 for exactly this reason).

layer	roll	ring
sliding, fp16	1.689 ms	1.486 ms	−12.0%
sliding, w4a16	0.917 ms	0.838 ms	−8.6%
full, fp16	1.559 ms	1.567 ms	no gain

Full layers gain nothing, so the default now picks per layer type: ring for sliding, roll for full.

Three things that did not work, and one that changed the answer

The concat was not the problem. Joining in score space instead of key space removes the concat entirely and is numerically identical (140.3 dB), but it needs an explicit max/exp/sum/div in place of a fused Softmax, and that cannot be quantised -- QNN's context-binary converter exits 14 at w4a16 while every fp16 build compiles. Two other shapes failed the same way: a broadcast multiply plus ReduceSum for the new token's score, and a two-axis broadcast multiply for its value contribution. Keeping roll's fused-softmax attention and changing only what the layer returns captures the win and compiles.

The cheaper GELU is not cheaper here. The erf form is 11–13% of every layer. The tanh approximation holds 65.2 dB with top-1/top-5 intact, so it was adoptable -- but on this NPU it is slower (1.535 vs 1.483 ms). The exact form stays, which is also the better-fidelity choice.

Per-graph overhead is negligible, so the per-layer method is sound. Profiling one layer at a time pays the graph's Input/Output boundary once per layer, where a real step pays it once for 28. Exporting the architecture's repeating 4-layer block (3 sliding + 1 full) measured 6.103 ms against 6.016 ms summed from single layers — a 1.4% gap. There is no hidden win in fusing layers, and extrapolating from per-layer numbers is legitimate.

Uncalibrated quantisation produces garbage, which invalidates the earlier numbers. Running the real layer on the device and comparing against the fp32 reference:

build	SNR vs reference
fp16	60.6 dB
w4a16, no calibration data	−1.6 dB
w4a8, no calibration data	−1.28 dB

Negative SNR means the output is uncorrelated with the truth. Every w4a16 figure in the first version of this document came from a build in that state. Quantisation ranges have to be inferred from activations the model actually sees: real ones have x at absmax 0.12 and the KV cache at 7.4, where the random tensors used for tracing had 0.4 and 2.5.

Re-quantising with activations captured from the reference model on real text (4 samples, English, code and Chinese) fixes it:

scheme	SNR	sliding layer
w8a16, calibrated	36.9 dB	0.803 ms
w16a16, calibrated	44.7 dB	1.656 ms
w8a8, calibrated	5.1 dB — broken	0.749 ms
w4a16 / w4a8, calibrated	converter fails	—

int8 activations destroy the layer even with calibration, which is the third model family in this repo to show it after Qwen3-TTS and MiniCPM-o. int4 weights do not survive the calibrated path at all. w8a16 with real calibration data is the deployable configuration, and it is both faster and verified where the old w4a16 was neither.

What is left, and why int4 is not it

Profiling the optimised layer shows the shape of the problem has changed completely. Input and Output -- 26% of the original layer -- are no longer in the top ten ops at all. What remains, of 0.803 ms:

op group	share	can it move?
MLP up + down matmuls	36.3%	no, this is the model
GELU (fused elementwise)	17.5%	no, tanh is slower here (measured)
attention matmuls	10.5%	no
RMSNorm, decomposed	8.5%	yes, see below
cache slice	4.1%	marginal

int4 weights are not the next lever, on two independent counts. They do not compile: AI Hub's calibrated quantise path fails with QAIRT converter failed with exit code 1 for w4a16 and w4a8, with and without --quantize_io. And they would not help anyway -- on this NPU 4-bit and 8-bit weights time identically, for the layer (0.917 vs 0.912 ms) and for the output head (3.868 vs 3.862 ms). The calibrated 8-bit build is faster than the uncalibrated 4-bit one (0.803 vs 0.838). 4-bit buys memory here, not speed.

The real remaining item is RMSNorm fusion, which calibration costs us. The uncalibrated build contains two fused rms ops worth 1.0% of the layer; the calibrated build contains eight decomposed Pow/ReduceMean/Div/Mul ops worth 8.5%. Inserting QDQ nodes around each op breaks QNN's RMSNorm pattern match. The calibrated build is still faster overall and is the only correct one, so this is a trade worth making today -- but roughly 7.5% of every layer is sitting there for whoever can keep the fusion under quantisation. (F.rms_norm is not a route to it: the TorchScript ONNX exporter has no aten::rms_norm at any opset.)

The decode step, end to end

Every component measured on a real S25 at calibrated w8a16, 1k context:

component	ms each	ms total	SNR
sliding layer x21 (ring)	0.803	16.86	36.9 dB
full-attention layer x7 (roll)	0.942	6.59	37.4 dB
output head x1	3.907	3.91	62.8 dB
decode step		27.36	36.5 tok/s

Against the previous best (roll layout, uncalibrated w4a16, plus the same head) at 31.01 ms: 11.8% faster, and numerically valid for the first time.

The output head is 14% of the step and is 98.6% a single 2048x131072 matmul tied to the embedding — there is no overhead in it to remove. It was missing from the first version of this document entirely.

At 4k context the full-attention layers grow (2.96 ms each at fp16 vs 1.56 at 1k) while the sliding layers do not move at all, which is what the hybrid architecture buys:

quant	context	Spark hybrid	all-full counterfactual	gain
fp16	1 k	46.4 ms	43.7 ms	0.94x
fp16	4 k	56.2 ms	83.0 ms	1.48x

At 1k in fp16 the hybrid is slightly slower: sliding layers rotate all 256 head dims where full layers rotate 64, and below roughly 2k that RoPE cost outweighs the shorter cache. It pays from ~2k onward and keeps paying, because the full-layer term grows with context and the sliding term never does.

Which phone runtime, and which processor

The same exported sliding layer, one Galaxy S25:

runtime	processor	precision	ms/layer
QNN context binary	Hexagon NPU	w8a16 calibrated	0.803
TFLite	CPU	int8	0.877
QNN context binary	Hexagon NPU	fp16	1.479
TFLite	Hexagon NPU	fp16	1.491
ONNX Runtime	Hexagon NPU	fp16	1.500
TFLite	Adreno GPU	fp16	2.259
TFLite	CPU	fp16	2.958

Two readings. At matched fp16 the three NPU runtimes land within 1.4% of each other, so the export layout -- static shapes, explicit caches, GQA without expanding K/V -- is what sets on-device speed, not the engine; that reproduces what the Qwen3-TTS talker step showed on the same device. And the placement is the opposite of that TTS result: Spark's decoder wants the NPU (2.0x the CPU, 1.5x the GPU) because it is matmul-bound, where the TTS vocoder wanted the GPU because it was elementwise-bound.

The row that matters for competition is the int8 CPU one. llama.cpp cannot be run on an AI Hub device, and on a phone it would use 4-bit weights on the CPU, so comparing our NPU path against an fp16 CPU run would flatter us. The closest honest proxy AI Hub offers is the same graph, int8, on the S25's own CPU: 0.877 ms against our 0.803 ms, so the NPU path is 9.2% ahead of the processor and precision class llama.cpp would actually run in. That is a proxy, not llama.cpp itself.

Two independent checks say the proxy is not flattering us either. A 1.11 GiB Q4_K_M build on the S25's ~137 GB/s LPDDR5X has a hard bandwidth ceiling of 7.9 ms/token at 100% of peak and 13.2 ms at a more realistic 60% — against our measured 27.4 ms for a step that includes the output head the bandwidth model ignores. Per layer, the bandwidth bound gives roughly 0.28-0.47 ms where the measured int8 CPU run takes 0.877 ms, i.e. real CPU execution lands at about a third of the bandwidth ceiling, which is the usual outcome on a phone.

5. Driving an Android phone

Spark-X2.5 is text-only, so any harness must drive from the accessibility tree rather than screenshots. Surveyed:

harness	transport	text-only a11y mode	local OpenAI endpoint	role
droidrun/mobilerun	ADB + Portal accessibility service	yes	yes, OpenAI-compatible	control
ghost-in-the-droid/android-agent	ADB + uiautomator, no app on device	yes	lists vLLM, base_url undocumented	control
mobileClaw	on-device app, accessibility	vision-oriented	n/a	control
AndroidWorld (DeepMind)	Android emulator	yes	via custom agent	evaluation, 116 scored tasks

Recommendation: droidrun/mobilerun for control (it is the only one whose OpenAI-compatible base URL is documented, so it points at a local vLLM server unchanged), AndroidWorld for measuring whether the agent actually succeeds.

analysis/experiments/spark_edge/android_agent/ implements the action space those harnesses share -- tap, type_text, swipe, press_key, open_app, task_complete, all indexed against accessibility-tree elements, with the adb shell input command each maps to -- and runs Spark against ten Android screens through vLLM's OpenAI server.

Serving works with vLLM's stock parsers: Spark emits <tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call>, which is GLM-4.5's format, so --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice parses it as-is. No new parser needed.

config	correct tool	fully correct action	median decision
thinking on	7-8 / 10	5-6 / 10	8.9 s
thinking off	5 / 10	4 / 10	0.61 s

(16 CPU cores, bf16, measured under concurrent benchmark load, so latency is pessimistic.)

Thinking is worth roughly two of ten decisions and costs 14.6x the latency. With thinking off the model collapses toward answering tap for everything. With it on, the failure mode is the opposite: on the one genuinely ambiguous screen (reach Developer options, which is not visible and needs scrolling) it reasoned past a 1200-token budget without ever closing <think>, so vLLM's reasoning parser -- which only emits reasoning once the block closes -- returned empty content and no tool call at all. A phone agent on this model needs a step budget and a fallback, not just a bigger cap.

The clearest wrong action was tapping an already-on Wi-Fi switch instead of calling task_complete, i.e. undoing the task it had completed.

No Android device or emulator is attached to this host (/dev/kvm absent), so these numbers measure the model's decisions over real accessibility-tree layouts, not a live device.

5b. Driving it in a loop, not one decision at a time

The numbers above measure single decisions over fixed screens. A phone agent is a loop, and a loop fails differently. analysis/experiments/spark_edge/ android_agent/ is now a working toolkit -- uiautomator XML to an indexed element list, an ADB device and a scripted simulator that emits the same XML, a tool executor, a tolerant tool-call parser and the loop itself (21 tests, no phone and no server required).

Stock glm45 does not always parse Spark. Earlier this note said Spark's tool format is GLM-4.5's, so vLLM's parsers work as-is. True on 4 of 5 screens. On the Wi-Fi toggle screen -- the one where a switch's state decides the action -- the model left the format and emitted

[tap]
{"tool": "tap", "parameters": {"index": 1}

unterminated, with tool_calls empty. An agent reading only tool_calls sees a step with no action. tool_parse.py recovers the bracketed form, GLM blocks and bare JSON, repairing unbalanced braces, and reports every recovery rather than silently repairing.

Navigation survives quantization; recognising completion does not. Same harness, same simulated phone, one run per cell:

build	decode	"Turn on Wi-Fi."	"Set an alarm for 7:30 AM."
bf16	41.0 tok/s	done in 4 steps	wrong app (Settings, not Clock), toggles Airplane mode, stuck
W4A16 g128	80.0 tok/s	done in 4 steps	right app, full navigation, then no tool call
W4A8 g128	86.6 tok/s	turns Wi-Fi on, then taps it off again (2 runs)	reaches Alarm, then loops

Every build walks home -> Settings -> Network -> Wi-Fi correctly. Only some notice the switch now reads checked=true and stop. W4A8 -- the fastest build, and the one a speed-first selector picks -- ends with the task undone.

This is what section 2c's fidelity numbers meant. W4A8 reproducing 1/12 greedy prompts against its own bf16, and W4A16 2/12, looked like a small difference on a proxy metric; it is the difference between finishing a task and reversing it. For agentic work the 4-bit choice is W4A16, which weight_path.py selects under priority="fidelity" -- 8% of decode for the difference.

Adding an explicit "if the screen already shows what was asked, call task_complete" rule to the system prompt did not rescue W4A8. Nor did feeding back "the screen did not change", which bf16 was told five times on the alarm task before swiping again.

Limits: two tasks, one run each, on a simulated phone -- enough to show a failure mode reproduces, not to rank builds by success rate. The simulator emits real uiautomator XML, so the parser, the index-to-coordinate mapping and the stopping rules run on the path hardware drives; whether a real Settings app is laid out the way these fixtures assume is still untested, because this host has no adb, no emulator and no /dev/kvm.

6. Where it stands against the goal

• Runs on CPU: yes. Faster than llama.cpp at matched precision on both prefill (8.7-15.4x) and decode (1.19-2.65x), and an int8 path adds a further 1.6x.
• Runs on Galaxy S25: yes, measured on real hardware -- a full decode step, output head included, is 27.4 ms at 1 k context (36.5 tok/s) in calibrated w8a16, with every component verified at 36.9-62.8 dB against the reference. That is 11.8% faster than the first working version and the first configuration whose output is actually correct: the uncalibrated w4a16 builds quoted earlier score -1.6 dB, i.e. uncorrelated with the truth.
• Beats every edge engine: on prefill, yes, against everything that can run the model, at any precision -- 4.7x over llama.cpp's best quantized build at 32 threads. On decode it depends on load: llama.cpp Q4_K_M wins single-stream by 1.6x, vLLM int8 wins from batch 16 and leads 1.72x at batch 64. The single-stream gap is not closable on vLLM 0.28's CPU backend, which ships no dense int4 kernel.
• Beats SGLang: on decode by 3.8% on GPU (prefill 3.8% the other way), and by default on CPU and phone, where SGLang does not run -- its released sgl-kernel wheel has no CPU ops and it has no phone backend at all.
• Fastest on the phone: of every runtime and processor combination measurable on the S25, our QNN NPU path is quickest, including a 9.2% margin over the int8-on-CPU class llama.cpp would run in. llama.cpp itself cannot be run on an AI Hub device, so that specific comparison is a proxy.
• Phone control: the action space, an agent loop and measured decision accuracy are in analysis/experiments/spark_edge/android_agent/; serving needs no new parser, just --tool-call-parser glm45 --reasoning-parser glm45.

Open items: no dense int4 on CPU (upstream kernel gap); int8 long-context fidelity would benefit from a smarter scheme than round-to-nearest; on the NPU the next ~7.5% per layer is RMSNorm fusion, which calibration currently breaks (int4 weights are neither available nor useful there -- 4-bit and 8-bit time identically on this hardware); and none of the phone-agent numbers have been run against a live device, since this host has no emulator (/dev/kvm absent).