Include the baselines, comparisons and study documents, not just the fork

The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ

docs/analysis/spark_edge_support.md

@@ -0,0 +1,994 @@
+# Spark-X2.5-1.7B on CPU and Galaxy S25
+
+Spark-X2.5 (XHToken/iFlytek, Apache 2.0, released 2026-09-01) is a 1.7 B
+on-device model built for agentic tool use: BFCL-V4 46.9, tau-squared-bench
+65.3, a native 1 M-token context and 200+ languages. This note covers adding
+it to vLLM-Omni, measuring it against every engine that can run it, putting it
+on a real Snapdragon 8 Elite, and driving an Android phone with it.
+
+Reproduction scripts: `analysis/experiments/spark_edge/`.
+Model code: `vllm_omni/model_executor/models/spark2_5/`, exporter
+`vllm_omni/edge/spark_export.py`.
+
+## 1. What had to be built
+
+Three traits separate Spark from a Llama-style decoder, and all three are why
+no generic converter picks it up:
+
+| trait | detail | why it matters |
+|---|---|---|
+| hybrid attention | `layer_types` = 3x `sliding_attention` (window 512) then 1x `full_attention`, 28 layers -> 21 sliding + 7 full | bounds KV growth; this is what makes 1 M context affordable on a phone |
+| per-layer-type RoPE | sliding: all 256 head dims, theta 1e4. full: first 64 dims only (`partial_rotary_factor` 0.25), theta 5e6 | two rotary tables, not one |
+| head-wise output gate | `g_proj` [2048 -> 8] reads the same post-norm input as QKV; `sigmoid` scales each head's attention output before `out_proj` | an extra projection with no analogue in Llama/Qwen |
+
+Also: QKV ships pre-fused as one `q_k_v_proj`, the MLP is GEGLU with *exact*
+gelu, embeddings are tied, and `head_dim` is 256 with only 2 KV heads.
+
+### Engine support before this work
+
+| engine | Spark-X2.5 support |
+|---|---|
+| llama.cpp | yes, native (`src/models/spark2-5.cpp`) |
+| SGLang | yes, native (`srt/models/spark2_5.py`) |
+| vLLM 0.28 | **no** -- added here |
+| MNN | no. `llmexport.py` dispatches on a fixed `model_type` list; `spark2_5` is absent |
+| ExecuTorch | no |
+| ncnn | no |
+
+MNN, ExecuTorch and ncnn would each need the architecture written from
+scratch. That is the honest reason they do not appear in the benchmark tables
+below: they cannot load the model at all, not that they are slow at it.
+
+### Correctness
+
+Verified against the official remote-code implementation pinned to
+transformers 4.57.1 (the version `config.json` targets; it does not import on
+the 5.x in our venvs, so a dedicated venv runs ground truth).
+
+| scenario | prompt | fp32 | bf16 |
+|---|---|---|---|
+| short | 28 tok | 32/32 greedy, max logprob diff 2.2e-05 | 24/24 greedy |
+| long (exercises sliding window) | 2465 tok | -- | 32/32 greedy |
+| tool calling | 137 tok | 32/32 greedy | diverges at token 23 |
+
+The bf16 tool-calling divergence is a near-tie under greedy decoding, not a
+defect: in fp32 the same prompt matches exactly with a 2.2e-05 logprob
+difference.
+
+One vLLM bug had to be worked around. The head-wise gate is a [2048, 8]
+projection, and vLLM's CPU backend routes any narrow linear to the sgl-kernel
+packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim
+0. VNNI packing does not for N that small, so it reports K instead of N and
+`torch.compile` dies tracing the gate multiply. The gate is kept as a plain
+parameter with `F.linear` -- a 2048x8 matmul gains nothing from the packed AMX
+kernel anyway.
+
+## 2. CPU: vLLM-Omni vs llama.cpp
+
+Xeon Platinum 8480C (Sapphire Rapids, AMX-BF16/INT8). Both engines pinned with
+`taskset` to the same physical cores and run **strictly one at a time**;
+prefix caching disabled on vLLM so a replayed prompt is really recomputed
+(llama-bench has no such cache). pp512 = prefill throughput, tg128 = decode.
+
+### Same precision (bf16)
+
+| threads | vLLM pp512 | llama.cpp pp512 | prefill | vLLM tg128 | llama.cpp tg128 | decode |
+|---|---|---|---|---|---|---|
+| 4 | **1113.4** | 72.4 | 15.4x | **11.11** | 4.20 | 2.65x |
+| 8 | **2113.7** | 173.8 | 12.2x | **19.88** | 16.15 | 1.23x |
+| 16 | **3639.4** | 341.7 | 10.7x | **30.16** | 25.31 | 1.19x |
+| 32 | **4454.9** | 509.5 | 8.7x | **36.35** | 28.57 | 1.27x |
+
+vLLM wins both, at every thread count. The prefill gap is AMX: llama.cpp's
+bf16 path runs at AVX-512 speed, vLLM's goes through oneDNN/AMX tiles.
+
+### Against llama.cpp's quantized builds
+
+| threads | vLLM bf16 pp | llama.cpp Q8_0 pp | llama.cpp Q4_K_M pp | vLLM bf16 tg | llama.cpp Q8_0 tg | llama.cpp Q4_K_M tg |
+|---|---|---|---|---|---|---|
+| 4 | **1113.4** | 118.1 | 103.6 | 11.11 | 9.85 | **10.36** |
+| 8 | **2113.7** | 490.2 | 434.2 | 19.88 | 45.01 | **42.40** |
+| 16 | **3639.4** | 914.7 | 806.3 | 30.16 | 66.23 | **71.50** |
+| 32 | **4454.9** | 1493.1 | 1199.0 | 36.35 | 76.54 | **82.81** |
+
+Prefill: vLLM at bf16 still beats llama.cpp's *best quantized* build by 3.0x
+at 32 threads. Prefill is compute-bound, and AMX beats 4-bit weight savings.
+
+Decode: llama.cpp Q4_K_M wins from 8 threads up, by 2.3x at 32. Decode is
+memory-bound -- 4-bit weights move a quarter of the bytes -- and this is a
+precision difference, not an engine difference. Closing it needs a quantized
+vLLM CPU path for Spark, which is the open item in section 6.
+
+### Quantized: int8 on CPU
+
+vLLM 0.28's CPU build has int8 dense kernels (`int8_scaled_mm_with_quant`,
+`dynamic_scaled_int8_quant`), and `static_scaled_fp8_quant` is registered with
+an empty schema, so on-the-fly fp8 fails outright.
+
+An earlier draft of this note said the backend had no dense int4 kernel
+either, on the strength of `woq_int4_linear`, `int4_scaled_mm_with_quant` and
+`da8w4_linear` all being absent. That was wrong -- they are simply not the
+names it uses. See section 2c: there are two four-bit paths, and the one
+vLLM ships is `int4_scaled_mm_cpu`, an AMX W4A8 GEMM reached through the
+GPTQ/AWQ kernel selector.
+
+Int8 is also available. `analysis/experiments/spark_edge/quantize_int8.py` writes a
+compressed-tensors W8A8 checkpoint straight from the safetensors (symmetric
+per-output-channel round-to-nearest; no calibration needed, and llmcompressor
+is avoided because it pulls transformers 5.x, on which Spark's remote code
+does not import). The head-wise gate is left in full precision: its eight
+outputs each scale a whole attention head, and it is 0.001% of the weights.
+Weight SNR 36.9-40.9 dB across 140 tensors.
+
+| threads | pp512 bf16 -> int8 | tg128 bf16 -> int8 |
+|---|---|---|
+| 16 | 3639 -> **5135** (1.41x) | 30.16 -> **48.30** (1.60x) |
+| 32 | 4455 -> **6978** (1.57x) | 36.35 -> **58.44** (1.61x) |
+
+Quality against the fp32 reference, though, is not free:
+
+| scenario | greedy match | first-token top-1 | top-20 overlap | max logprob diff |
+|---|---|---|---|---|
+| short (28 tok) | 23/24 | agrees | 18/20 | 0.83 |
+| long (2465 tok) | **12/32** | agrees | 19/20 | 1.56 |
+| tool calling | 31/32 | agrees | 18/20 | 0.58 |
+
+bf16 matched 32/32 on all three with a max difference of 0.24. Int8 keeps the
+first token everywhere, but at long context the distribution has moved enough
+that greedy output diverges after a dozen tokens. Usable, not free -- the same
+caution the Qwen3-TTS and MiniCPM-o work in this repo reached about automatic
+int8.
+
+## 2b. CPU under concurrency
+
+Single-stream decode is memory-bound, which is a precision contest llama.cpp
+wins with 4-bit weights. Concurrent decode is compute-bound, which is an AMX
+and continuous-batching contest. Total generated tokens/s, 32 pinned cores,
+128-token prompts generating 128 tokens each:
+
+| batch | vLLM bf16 | vLLM int8 | llama.cpp Q4_K_M | int8 vs llama.cpp |
+|---|---|---|---|---|
+| 1 | 34.6 | 54.9 | **91.1** | 0.60x |
+| 2 | 69.9 | 105.2 | **129.6** | 0.81x |
+| 4 | 139.8 | 210.3 | **251.6** | 0.84x |
+| 8 | 266.1 | 387.8 | **406.5** | 0.95x |
+| 16 | 495.7 | **694.7** | 613.9 | 1.13x |
+| 32 | -- | **1080.6** | 778.3 | 1.39x |
+| 64 | -- | **1557.9** | 904.5 | 1.72x |
+
+The crossover is around batch 12. Below it llama.cpp's 4-bit weights win;
+above it vLLM's AMX-INT8 GEMMs and continuous batching pull away, reaching
+1.72x at batch 64 and still climbing. A phone agent is a batch-of-one
+workload, so llama.cpp is the right single-device choice today; an edge
+gateway serving a dozen sessions is vLLM's.
+
+## 2c. Four bits on CPU: two paths, and which one a device should take
+
+vLLM 0.28's CPU build has a 4-bit GEMM. `CPUWNA16LinearKernel`
+(`vllm/model_executor/kernels/linear/mixed_precision/cpu.py`) routes GPTQ and
+AWQ checkpoints to **`int4_scaled_mm_cpu`**, an AMX W4A8 kernel, whenever the
+activations are bf16, the ISA has AMX tiles and the checkpoint carries no
+activation reordering. Four-bit weights times eight-bit activations is exactly
+what llama.cpp's Q4_K_M does (Q4_K weights, Q8_K activations), so this is the
+honest precision match rather than a handicap.
+
+Two paths were built, because they are not the same trade:
+
+| | weights | activations | kernel | bits/weight | reached by |
+|---|---|---|---|---|---|
+| **W4A8** | 4-bit symmetric, group-wise | int8 (per the kernel) | `_C::int4_scaled_mm_cpu` (AMX) | 4.16 @ g128 | GPTQ checkpoint, vLLM's own dispatch |
+| **W4A16** | 4-bit asymmetric min/max, group-wise | bf16 | `aten::_weight_int4pack_mm_for_cpu` | 4.16 @ g128 | `cpu_int4`, registered out-of-tree |
+
+W4A8 is forced symmetric: `_process_gptq_weights_w4a8` builds its own zero
+point (a constant 8) and ignores the checkpoint's `qzeros`, so the grid must be
+`w = (code - 8) * scale`. W4A16 is free to use an asymmetric min/max grid, and
+a per-group scale search recovers roughly a dB in either case.
+
+### Which is faster
+
+Per-op profiles, taken back to back on the same 24 cores. Throughput on this
+host swings ~30% run to run and the gap here is ~8%, so the GEMM self time is
+the measurement that can actually rank them:
+
+| path | GEMM/step | step | tok/s |
+|---|---|---|---|
+| W4A8 g128 | **5.91 ms** | **11.55 ms** | **86.6** |
+| W4A16 g128 | 6.85 ms | 12.51 ms | 80.0 |
+| llama.cpp Q4_K_M | -- | 11.11 ms | 90.0 |
+| vLLM bf16 | -- | -- | 41.0 |
+
+W4A8 wins twice: the kernel is 0.94 ms cheaper per step, and because it is
+reached through vLLM's own dispatch it also avoids the 0.48 ms/step custom-op
+boundary the tinygemm path needs (that boundary exists because tinygemm's cost
+is linear in batch size, so prefill has to be routed to a dequantize-then-AMX
+path instead -- see below).
+
+**The trap that cost a day: a GPTQ checkpoint must quantize the output head.**
+Spark ties its embedding to its head, so a converter that leaves the tie in
+place emits no `lm_head` tensors and that one GEMM runs bf16 through
+`_C::onednn_mm` -- **2.97 ms per step**, more than the entire 4-bit saving. The
+profile is what caught it: `int4_scaled_mm_cpu` ran 112 times (28 layers x 4
+linears) and `onednn_mm` once. Every "W4A8" number measured before that was on
+a handicapped build, which is why the W4A8/W4A16 ordering appeared to flip
+between grids.
+
+### Which is more faithful
+
+Greedy divergence from each build's own bf16 output, 12 prompts, measured the
+same way on both engines:
+
+| build | exact match | mean prefix kept |
+|---|---|---|
+| llama.cpp Q4_K_M | **4/12** | **0.51** |
+| vLLM W4A16 | 2/12 | 0.30 |
+| vLLM W4A8 | 1/12 | 0.13 |
+
+Q4_K_M's 6-bit sub-block scales buy real granularity at the same bit budget;
+round-to-nearest with a per-group search does not match it. So the two paths
+are a genuine choice, not a ranking: **W4A8 for speed, W4A16 for fidelity**,
+and `vllm_omni/edge/weight_path.py` makes that choice from a hardware probe
+rather than a flag, because the fast kernel on each device needs a different
+and mutually exclusive weight layout:
+
+| device class | path | why |
+|---|---|---|
+| x86 + AMX | W4A8 / GPTQ | `int4_scaled_mm_cpu` applies |
+| x86 without AMX | W4A16 / tinygemm | no AMX, no int4 kernel |
+| x86 + AMX, fidelity asked | W4A16 / tinygemm | keeps bf16 activations |
+| ARM64 | W4A16, **unverified** | PyTorch's 4-bit CPU GEMM dispatches a different kernel there |
+| phone NPU | exported graph, calibrated w8a16 | vLLM does not run there at all |
+
+### Prefill needs the other kernel
+
+The tinygemm kernel dequantizes once per row of activations, so its cost is
+linear in batch size: it beats AMX bf16 below ~8 rows and loses badly above.
+Measured at Spark's `gate_up` shape, per call: 0.070 ms at 1 token, 14.2 ms at
+512. Left alone, pp512 collapsed from 4455 tok/s (bf16) to 557. Unpacking the
+weight to bf16 once per call and handing it to the ordinary oneDNN/AMX GEMM
+brings it back to **3347 tok/s**, 3.0x llama.cpp's 1121. W4A8 needs none of
+this -- the AMX kernel is the right one at every batch size, which is the
+second reason it is the default on an AMX machine.
+
+## 2d. Init time and resident memory, which is what decides edge at all
+
+Speed is not the binding constraint on an edge box; a 90-second start and 8 GB
+resident is. Both turned out to be fixable, and one of them was a bug.
+
+### vLLM recompiled the model on every single start
+
+`_try_load_aot_compiled_fn` finishes a cache load with
+`loaded_fn._artifacts.compiled_fn.finalize_loading(cfg)`. On CUDA that artifact
+is a wrapper object carrying the method; with the inductor CPU backend it is a
+plain function, so the call raises `AttributeError`, the handler treats the
+whole load as a cache miss ("Compiling model again due to a load failure"), and
+the model is compiled again from scratch. Every restart paid it.
+
+Attaching the missing no-op (a plain Python function accepts attribute
+assignment; the CPU artifact loads its generated `.so` lazily on first call,
+which is the warm-up run) fixes it. Same cache directory, one variable changed:
+
+| | init |
+|---|---|
+| cold cache (first ever compile) | 93.8 s |
+| warm cache, vLLM as shipped | 49.8 s |
+| **warm cache, shim installed** | **14.9 s** |
+
+**3.3x faster warm start**, reproduced across passes (14.9 / 14.7 / 15.0, and
+12.8-13.0 on the W4A16 path). The shim is self-disabling: if the artifact
+already has `finalize_loading`, as it does on GPU and will if upstream gives
+the CPU path one, it changes nothing.
+
+### Measure anonymous memory, not RSS
+
+A first pass concluded that sizing the KV cache saved 3.1 GB. It does not, and
+the error is worth recording because it is easy to repeat: that comparison
+read `VmRSS`, which counts the mmap'd checkpoint's file pages, and those come
+and go with page-cache pressure. **Identical configurations measured 1.2-2.0 GB
+apart.** `RssAnon` -- allocations only -- repeats to within **0.5 MB**, and is
+what a memory budget is actually about.
+
+### The KV cache is not a memory lever on CPU
+
+Re-measured properly, with only the budget changing:
+
+| KV budget | anonymous RSS |
+|---|---|
+| 4 GiB (env default) | 7601 MB |
+| 512 MiB | 7405 MB |
+| 128 MiB | 7413 MB |
+
+Shrinking the cache 32x moves resident memory by **188 MB**, and 512 MiB is
+indistinguishable from 128 MiB. The cache is allocated but its pages are never
+touched, so it costs address space, not memory.
+
+It still has to be *correct*, which is a separate problem: `--kv-cache-memory-bytes`
+was silently discarded on CPU. `CpuPlatform.check_and_update_config` ends with
+a block its own comment calls "Lagecy setting" that overwrites
+`cache_config.kv_cache_memory_bytes` from `VLLM_CPU_KVCACHE_SPACE`
+unconditionally, so the budget could only be set in whole GiB and vllm-omni's
+per-stage budgets had no effect at all. Verified by allocation rather than by
+memory, because allocation cannot drift: with the env at 1 GiB, asking for
+128 MiB produced the same **17 881** cache tokens as asking for nothing, and
+with the env at 4 GiB it produced **71 527**. After restoring the obvious
+precedence the same request yields **2 231** tokens. That matters for
+admission -- vLLM preempts by recompute when the cache cannot hold
+`max_num_seqs x max_model_len`, which on a single-stream deployment is a stall
+-- not for footprint. `vllm_omni/edge/kv_budget.py` prices the cache from the
+config, reporting the flat figure (what is safe to pass, since vLLM's CPU
+accounting is flat) beside the hybrid one a layer-type-aware backend could use.
+
+### Where the memory actually goes: the weight load
+
+Split by phase, on anonymous RSS:
+
+| phase | W4A8 (GPTQ) | W4A16 (tinygemm) |
+|---|---|---|
+| before import | 12 MB | 12 MB |
+| after `import vllm_omni` | 513 MB | 515 MB |
+| after engine construction | 7338 MB | 5023 MB |
+| **after releasing free arenas** | **3510 MB** | **3411 MB** |
+
+Imports cost half a gigabyte -- they are the init-*time* problem, not the
+memory one. The engine adds 4.5-6.8 GB for a model whose weights are under
+1.4 GB, and **most of it is not live**: loading a quantized checkpoint unpacks
+4-bit codes to int32 -- four bytes per weight -- to permute and repack them,
+and although the intermediates are freed the allocator keeps the arenas. One
+release call returns **3828 MB** on the GPTQ path and 1612 MB on tinygemm, a
+**52%** cut on the configuration we actually want to deploy.
+
+The second reading of that table matters more than the first: after the
+release both paths land in the same place. W4A8's apparent 2.3 GB penalty over
+W4A16 was never a footprint difference, it was the repack. So W4A8 wins on
+speed *and* ties on memory, and fidelity is the only remaining reason to
+choose W4A16.
+
+`vllm_omni/edge/memory.py` does the release and `patch.py` hooks it to the CPU
+worker's load and warm-up (`VLLM_OMNI_CPU_RELEASE_MEMORY=0` to disable). The
+~3.5 GB that remains, against 0.86 GB of weights, is the next thing to chase.
+
+### Two bugs that made CPU deployments take the slow path silently
+
+Init is not one number per model; it has a different cause in each pipeline,
+and in two places the slow path was taken without saying so.
+
+**Parallel stage init could never run on CPU.**
+`resolve_stage_physical_devices` returns `None` for `devices: "cpu"`, and the
+admission resolver fails closed on `None` -- right for a GPU replica whose
+devices cannot be accounted, wrong here -- so every multi-stage CPU pipeline
+raised `StageAdmissionError` and fell back to sequential startup. The edge
+profile *sets* `parallel_stage_init: true`, so on CPU that profile was broken
+rather than merely slower.
+
+The fix is not to skip admission. The per-device ledger is vacuous on CPU, but
+the resource concurrent init contends for is host memory, and two stages
+allocating KV caches at once exhaust it exactly as two GPU stages exhaust a
+card. `check_host_admission` applies the same discipline to that resource: sum
+the declared budgets, compare with available RAM and a margin, refuse to start
+otherwise, and fail closed on a stage that declares no budget.
+
+| Qwen3-TTS 0.6B, init only | |
+|---|---|
+| sequential | 64.57 s |
+| **parallel** | **30.54 s** |
+
+**A deploy YAML meant different things depending on which flag loaded it.**
+The `orchestrator:` block carrying `parallel_stage_init` was applied only on
+the `deploy_profile` path, so pointing `--deploy-config` at the very same file
+loaded its stages and dropped the flag: 63.76 s against 30.76 s, with nothing
+in the log to explain the difference. The block now applies either way.
+
+### What is not a memory lever
+
+Two candidates tested and closed, so they are not re-tried:
+
+| lever | effect |
+|---|---|
+| OMP thread count (8 / 24 / 48) | 3754 / 3775 / 3818 MB -- 64 MB across a 6x range |
+
+**Retracted (2026-09-14): KV cache size was on this list, and it is the single
+largest lever there is.** The row read "4 GiB -> 128 MiB saves 188 MB, the
+pages are never touched". Re-measured now that the budget is actually
+honoured -- 3 interleaved passes per arm, both filling a 1536-token context:
+
+| KV budget | resident |
+|---|---|
+| 1 GiB (the smallest `VLLM_CPU_KVCACHE_SPACE` can express) | 3264 MB |
+| 192 MiB, explicit | **2421 MB** |
+
+**843 MB.** And `mincore(2)` says why the old reasoning was wrong: the cache is
+not allocated-and-untouched, it is fully in core -- 1023.8 MB allocated against
+1024.1 MB resident, and 190.8 against 191.1 when sized down.
+
+The explanation for the original 188 MB is two sections below, in this report's
+own words: `--kv-cache-memory-bytes` *was silently discarded on CPU*, because
+`CpuPlatform.check_and_update_config` overwrote it from
+`VLLM_CPU_KVCACHE_SPACE` unconditionally. So the "32x shrink" never happened;
+both arms ran the same cache and the 188 MB was measuring something else. The
+lesson is the one this study keeps relearning: a lever tested through a path
+that silently ignores it measures nothing, and "no effect" is the most
+dangerous result to accept without checking that the knob was connected.
+
+The dual weight layout the W4A16 path keeps (tinygemm for decode, a row-major
+copy for prefill) *is* one. Median of three interleaved passes, anonymous RSS,
+spread under 1.5%:
+
+| build | resident | prefill | decode | init |
+|---|---|---|---|---|
+| W4A8 g128 | 3640 MB | -- | 86.6 tok/s | 15.0 s |
+| W4A16 g128 | 3542 MB | 3252 tok/s | 76.9 tok/s | 12.4 s |
+| **W4A16, `prefill_dequant: false`** | **2802 MB** | 448 tok/s | 78.5 tok/s | 16.0 s |
+
+The second layout costs **740 MB** and buys **7.3x prefill**, because the
+tinygemm kernel dequantizes once per row of activations and its cost is linear
+in the batch.
+
+**That trade-off is smaller than it looked, but it is real** (2026-09-13). It
+rested partly on treating the tinygemm layout as opaque, so a row-major
+duplicate was kept beside it to have something to dequantize from -- but the
+duplicate was also doing work: `to_split_halves` performs the nibble
+de-interleave *once at load*, so every prefill call reads two contiguous
+planes. It is a precomputed permutation, not just a copy. Measured on this host instead of
+assumed -- by packing an index pattern and reading back where each nibble
+landed -- the layout is `[N/64][K][32 bytes]`: K is the middle axis, so a K
+range is contiguous within each n-block, and inside every 32-byte run byte `d`
+carries output channel `d` in its low nibble and `d + 32` in its high one, the
+same permutation for every `k` and every block. Decoding it is a deinterleave,
+not a reverse-engineered ISA table.
+
+So the prefill path can dequantize from the kernel's own layout, and then the
+duplicate is not allocated -- at the cost of redoing that de-interleave on
+every call. The reconstruction is **bit-exact** against the previous
+dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048.
+Shapes where N does not tile the 64-channel block (N=80 is 64+16) keep the
+copy rather than mis-read the layout; every Spark linear tiles cleanly.
+
+Measured with `VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT` toggling the two paths in
+otherwise identical code, 3 arms x 2 interleaved passes, anonymous RSS summed
+over the process tree, 1536-token context:
+
+| build | resident (median) | peak | spread |
+|---|---|---|---|
+| W4A16, second copy (old) | 4591 MB | 5508 MB | 0.2% |
+| **W4A16, one layout** | **3976 MB** | **4593 MB** | 5.4% |
+| W4A16, `prefill_dequant: false` | 3902 MB | 4522 MB | 2.6% |
+
+**615 MB of resident memory and ~915 MB of peak**, and the one-layout build is
+statistically indistinguishable from `prefill_dequant: false` -- the harness
+refused to rank them (0.4% apart, inside the measurement's own spread). The
+arithmetic predicts 764 MB from the config; ~150 MB of that is allocator
+retention `malloc_trim` does not return.
+
+It is not free. Prefill, pp512, pooled over 7 interleaved passes per arm on
+idle node-1 cores with memory bound to the same node:
+
+| build | resident | pp512 median | range | spread |
+|---|---|---|---|---|
+| two layouts (old default) | 4591 MB | 2190 tok/s | 2117-2345 | 9.7% |
+| **one layout, direct dequant** | **3976 MB** | **1344 tok/s** | 1271-1555 | 18.3% |
+| `prefill_dequant: false` | 3902 MB | 453 tok/s | 451-453 | 0.3% |
+
+So it costs **1.63x prefill** on medians -- and the ratio itself is only good
+to about 1.4-1.8x, because both dequantizing arms swing ~10-18% on this shared
+host while the tinygemm-only arm reproduces to 0.3%. The *existence* and rough
+size of the cost is solid (a 63% gap is far outside that band); the third digit
+is not. An earlier draft of this section said 1.9x, taken from the single
+surviving pass of a grid whose other passes had crashed -- a number the harness
+itself had labelled "single pass -- not a result".
+
+The mechanism is that undoing the kernel's interleave per call costs ~1.6x
+reading the pre-permuted planes (14.4 ms vs 9.2 ms per `gate_up` dequantize at
+8 threads). Two variants were tried and rejected: dequantizing to `[K, N]` and
+using `matmul` instead of `F.linear`, which avoids the element-wise transpose,
+measured *worse* at 20.3 ms, and the blocked form needs either 208 tiny GEMMs
+or the same permutation.
+
+What this changes is the shape of the choice, not its existence:
+
+* against `prefill_dequant: false` it is **strictly better** -- the same memory
+  to within the noise, **2.97x** the prefill, and it does not route multi-row
+  batches into the tinygemm/AMX SIGILL hazard documented in `cpu_int4.py`. That
+  setting is now dominated and should not be used. This comparison is the solid
+  one: the tinygemm-only arm reproduces to 0.3%.
+* against the old default it is a real dial: 615 MB for ~1.6x prefill. On a
+  device chosen for its memory ceiling that is the right side of the trade;
+  on one with headroom it is not. `direct_dequant: false` in the checkpoint's
+  `quantization_config` selects the old behaviour -- a config field rather than
+  only an env var, because the two paths build different graphs and must not
+  share an AOT compile artifact (see below).
+
+Recovering the prefill would take a C++ de-interleave rather than an ATen
+expression chain, which is a real kernel and not in scope here.
+
+One hazard found while measuring this, worth recording because it presented as
+a crash rather than a wrong number. The two paths build different graphs -- one
+passes `weight_halves` into the custom op, the other passes `None` -- but vLLM
+keys its AOT compile cache on the *config*, so an env-var-only switch shares
+one cached artifact between them. Three passes died with
+`KeyError: 'weight_halves'` inside an AOT-loaded forward after the first pass
+saved an artifact for the other path. The choice is therefore a
+`quantization_config` field, which `ModelConfig.compute_hash` does hash; the
+env override remains for experiments and needs its own `VLLM_CACHE_ROOT`.
+
+### The largest memory item was a RoPE table for a context nobody asks for
+
+The report used to account for memory by subtraction -- weights, imports, and
+an "unexplained ~1.6 GB" left over. A remainder computed from two uncertain
+terms is not a measurement, so `memory_ledger.py` counts instead: unique tensor
+storages keyed by `data_ptr` (so tied weights are counted once), the KV cache
+found by attribute, and every `/proc/self/smaps` mapping bucketed. Allocated is
+separated from resident with `mincore(2)`, because a cache that was allocated
+and never written costs address space, not memory, and counting it would hide
+the remainder rather than explain it.
+
+The first run put 640 MB in an "other" bucket across 114 tensors. Naming the
+largest storages showed what it was:
+
+    model.layers.0.self_attn.rotary_emb.cos_sin_cache   512.0 MB   [1048576, 256]
+    model.layers.3.self_attn.rotary_emb.cos_sin_cache   128.0 MB   [1048576, 64]
+
+Spark advertises a **1 048 576-token context**, and `spark2_5.py` passed
+`max_position=config.max_position_embeddings` straight into `get_rope`, so vLLM
+built `torch.arange(1048576)` cos/sin tables -- two of them, because the
+sliding and full layers are trained with different theta and different partial
+rotary factors, giving one 256-wide table and one 64-wide one. Both fully
+resident by `mincore`. For a deployment whose `max_model_len` is 2048 they need
+**1.25 MB between them**.
+
+vLLM's scheduler cannot produce a position beyond `max_model_len`, so the table
+only has to cover that. Capping it, 3 interleaved passes per arm with
+`VLLM_OMNI_SPARK_ROPE_FULL` toggling the two:
+
+| | settled | peak | init |
+|---|---|---|---|
+| 1M-position tables | 3891 MB | 4019 MB | 13.5 s |
+| **capped to `max_model_len`** | **3273 MB** | **3370 MB** | 13.7 s |
+
+**618 MB settled and 649 MB of peak, for one line and no measurable init
+cost** -- the largest single item found in this study, larger than the single
+weight layout (615 MB) or the blocked repack (489 MB), and unlike those two it
+costs nothing at all. The ledger's storage accounting agrees independently: the
+"other" bucket falls from 640.0 MB to 1.3 MB and unique tensor storage drops
+638.8 MB.
+
+Two things the same ledger exposes, not yet acted on:
+
+* **The KV cache is fully resident** -- 1023.8 MB allocated, 1024.1 MB resident
+  by `mincore`. The "not a memory lever" verdict below was measured while
+  generating 8 tokens; at page level the cache is entirely in core, and it is
+  27% of total memory. That conclusion needs re-opening on this evidence.
+* **`embed_tokens.weight_packed` (128 MB) is kept only because the tinygemm
+  layout was believed un-addressable for a single-row lookup.** That premise is
+  now known to be false -- the deinterleave above reaches any row -- so the
+  row-major copy is probably removable too.
+
+### Peak load memory: the number an edge device actually has to survive
+
+Everything above reports memory *after* the allocator is asked for its arenas
+back. That is the wrong target for a device: `release_after_load()` runs only
+once `load_model` and warm-up have both returned, so a box that cannot hold the
+peak never reaches the release. Sampling `RssAnon` on a timer through the load
+(rather than reading the endpoint) put the W4A16 peak at **5722-5837 MB**
+against ~4480 settled -- a 1.2-1.3 GB spike nothing in the tree had measured.
+
+Most of it was in our own repack path. `unpack_nibbles` expands 4-bit codes to
+int32 -- 4 bytes per weight, eight times the packed form -- because the
+kernel's packer wants int32, and doing that for a whole matrix makes the
+transient the largest object in the process:
+
+| tensor | shape | int32 expansion |
+|---|---|---|
+| `gate_up` | 13312 x 2048 | 104 MiB |
+| `down_proj` | 2048 x 6656 | 52 MiB |
+| `q_k_v_proj` | 3072 x 2048 | 24 MiB |
+| **embedding (tied, so also the head)** | **131072 x 2048** | **1024 MiB** |
+
+against 24 MiB of packed weights for an entire layer.
+
+The packed layout is `[N/64][K][32 bytes]`, so n-blocks are independent and the
+expansion can cover a few thousand rows at a time -- verified byte-identical to
+packing the whole matrix at four shapes plus a ragged last chunk. Measured with
+`VLLM_OMNI_CPU_INT4_REPACK_ROWS` toggling the two in identical code, 3
+interleaved passes, node-local cores:
+
+| | peak | settled | load-time transient | init |
+|---|---|---|---|---|
+| whole-matrix repack | 4502 MB | 3905 MB | 550 MB | 17.7 s |
+| **blocked repack** | **4013 MB** | 3892 MB | **105 MB** | 16.6 s |
+
+**489 MB of peak, for nothing.** Settled memory and init time are unchanged,
+and the repack is in fact *faster* chunked than whole -- 79 ms against 208 ms
+for the embedding at 24 threads, because the working set stays in cache. The
+chunk size is tuned from that sweep: 2048 rows is both the fastest and nearly
+the leanest (16 MiB transient), where 512 costs 116 ms and 32768 costs 129 ms.
+
+Two things this nearly got wrong, both caught by re-measuring rather than
+reasoning. The first blocked pass showed init at 50.2 s against 17.7 -- but it
+is the one pass the harness flagged `[busy]` at load 34.8, and the two clean
+passes read 16.6 and 16.2. And its settled memory looked 239 MB worse, which
+was pass-1 scatter: the medians are 3892 against 3905.
+
+The absolute peak improves less than the 1 GB the arithmetic predicts, because
+the single-layout change above had already removed `to_split_halves`, which was
+doing its own whole-matrix expansion. The two items overlap and their savings
+must not be added.
+
+### Kernel fusion: a negative result
+
+vLLM's CPU build ships `_C::rms_norm` and `_C::fused_add_rms_norm`, and
+`vllm/kernels/vllm_c.py` already wraps both as IR-op implementations -- but
+gates them on `GPGPU_DEVICE = is_cuda_alike() or is_xpu()`, so on CPU every
+norm falls through to the PyTorch path. In isolation the fused kernels are
+**3.6x and 4.5x faster** at Spark's decode shape (5.76 vs 20.99 us; 9.24 vs
+41.14 us). End to end they are **slightly slower**: 72.7 vs 74.0 tok/s.
+
+The reason is that the native norm was never running alone -- inductor fuses it
+into the residual add and the surrounding elementwise work, and an opaque `_C`
+call breaks that fusion for more than the norm itself saves.
+`vllm_omni/model_executor/layers/cpu_ir_ops.py` keeps the registration, off by
+default (`VLLM_OMNI_CPU_FUSED_NORMS=1` to try it on another machine), with the
+measurement recorded. `gelu_and_mul` is deliberately not registered at all: its
+fused CPU kernel is 2.7x *slower* than native at Spark's GEGLU shape (64.11 vs
+23.36 us), and `GeluAndMul.forward_cpu` in vLLM only reaches for it on POWERPC
+anyway.
+
+Two more fusion attempts that lost, recorded so they are not repeated:
+`cpp.min_chunk_size` raised to keep inductor's small elementwise kernels
+single-threaded cost 6% of decode and 19% of prefill; and the head gate written
+as an inline reduction (rather than behind an opaque op) made the whole step
+**1.8x slower** -- 78.8 to 44.5 tok/s -- because inductor's generated loop is
+worse than the matmul it replaced.
+
+## 3. GPU: vLLM-Omni vs SGLang
+
+One L20X (140 GiB HBM), bf16, single sequence, caches disabled on both
+(`enable_prefix_caching=False` / `disable_radix_cache=True`), 2048-token
+prefill so the measurement is not dominated by dispatch overhead.
+
+| engine | init | pp2048 | tg128 |
+|---|---|---|---|
+| vLLM-Omni | 111.3 s | 137 102 tok/s | **451.4 tok/s** |
+| SGLang 0.5.19 | 97.1 s | **142 253 tok/s** | 434.7 tok/s |
+
+A tie within 4% either way: SGLang +3.8% on prefill, vLLM +3.8% on decode.
+
+SGLang has an Intel AMX CPU backend, but the released `sgl-kernel` wheel does
+not ship the CPU ops it needs (`convert_weight_packed` is absent), so SGLang
+could not be measured on CPU without a source build. On the two targets this
+goal is actually about -- CPU and a phone -- SGLang does not run at all today.
+
+## 4. Galaxy S25 (Snapdragon 8 Elite), real device
+
+`vllm_omni/edge/spark_export.py` exports one decode step with the KV cache as
+explicit tensors, sized per layer type: sliding layers get a fixed window at
+any context, full layers grow. Parity against the HF reference is **140.3 dB**
+with identical top-1 and top-5.
+
+### What the device profile said to fix
+
+AI Hub returns per-op cycle counts, and they made the first export's mistake
+obvious. Of a sliding layer at w4a16:
+
+| op | share | what it was |
+|---|---|---|
+| `Output` | 9.9% | handing the recomputed 512-entry window back |
+| `output_..._converted` | 3.1% | quantising that window on the way out |
+| `Slice` x3 | 7.4% | rolling the window to drop the oldest entry |
+| **subtotal** | **20.4%** | **moving a cache-sized tensor across the graph boundary, once per token, in 21 of 28 layers** |
+
+Full-attention layers spent a similar 9.5% on `Concat`, appending the new key
+to the cache.
+
+The fix is a ring buffer: the runtime owns a fixed-size cache and writes the
+new entry into the slot for this position, so the graph returns one K/V entry
+instead of a window and never rolls anything. Ring order needs no fixing up --
+softmax over keys is order-independent and each cached key already carries its
+own rotary phase, so only slots not yet written need masking. The sliding
+cache correspondingly holds `window - 1` entries, because a 512-token window
+counts the current token (transformers' own sliding cache holds 511 for
+exactly this reason).
+
+| layer | roll | ring | |
+|---|---|---|---|
+| sliding, fp16 | 1.689 ms | **1.486 ms** | −12.0% |
+| sliding, w4a16 | 0.917 ms | **0.838 ms** | −8.6% |
+| full, fp16 | 1.559 ms | 1.567 ms | no gain |
+
+Full layers gain nothing, so the default now picks per layer type: ring for
+sliding, roll for full.
+
+### Three things that did not work, and one that changed the answer
+
+**The concat was not the problem.** Joining in score space instead of key
+space removes the concat entirely and is numerically identical (140.3 dB), but
+it needs an explicit max/exp/sum/div in place of a fused `Softmax`, and that
+cannot be quantised -- QNN's context-binary converter exits 14 at w4a16 while
+every fp16 build compiles. Two other shapes failed the same way: a broadcast
+multiply plus `ReduceSum` for the new token's score, and a two-axis broadcast
+multiply for its value contribution. Keeping roll's fused-softmax attention and
+changing only what the layer *returns* captures the win and compiles.
+
+**The cheaper GELU is not cheaper here.** The erf form is 11–13% of every
+layer. The tanh approximation holds 65.2 dB with top-1/top-5 intact, so it was
+adoptable -- but on this NPU it is *slower* (1.535 vs 1.483 ms). The exact form
+stays, which is also the better-fidelity choice.
+
+**Per-graph overhead is negligible, so the per-layer method is sound.**
+Profiling one layer at a time pays the graph's Input/Output boundary once per
+layer, where a real step pays it once for 28. Exporting the architecture's
+repeating 4-layer block (3 sliding + 1 full) measured 6.103 ms against 6.016 ms
+summed from single layers — a 1.4% gap. There is no hidden win in fusing
+layers, and extrapolating from per-layer numbers is legitimate.
+
+**Uncalibrated quantisation produces garbage, which invalidates the earlier
+numbers.** Running the real layer on the device and comparing against the fp32
+reference:
+
+| build | SNR vs reference |
+|---|---|
+| fp16 | 60.6 dB |
+| w4a16, no calibration data | **−1.6 dB** |
+| w4a8, no calibration data | **−1.28 dB** |
+
+Negative SNR means the output is uncorrelated with the truth. Every w4a16
+figure in the first version of this document came from a build in that state.
+Quantisation ranges have to be inferred from activations the model actually
+sees: real ones have `x` at absmax 0.12 and the KV cache at 7.4, where the
+random tensors used for tracing had 0.4 and 2.5.
+
+Re-quantising with activations captured from the reference model on real text
+(4 samples, English, code and Chinese) fixes it:
+
+| scheme | SNR | sliding layer |
+|---|---|---|
+| **w8a16, calibrated** | **36.9 dB** | **0.803 ms** |
+| w16a16, calibrated | 44.7 dB | 1.656 ms |
+| w8a8, calibrated | 5.1 dB — broken | 0.749 ms |
+| w4a16 / w4a8, calibrated | converter fails | — |
+
+int8 *activations* destroy the layer even with calibration, which is the third
+model family in this repo to show it after Qwen3-TTS and MiniCPM-o. int4
+weights do not survive the calibrated path at all. **w8a16 with real
+calibration data is the deployable configuration**, and it is both faster and
+verified where the old w4a16 was neither.
+
+### What is left, and why int4 is not it
+
+Profiling the optimised layer shows the shape of the problem has changed
+completely. `Input` and `Output` -- 26% of the original layer -- are no longer
+in the top ten ops at all. What remains, of 0.803 ms:
+
+| op group | share | can it move? |
+|---|---|---|
+| MLP up + down matmuls | 36.3% | no, this is the model |
+| GELU (fused elementwise) | 17.5% | no, tanh is slower here (measured) |
+| attention matmuls | 10.5% | no |
+| RMSNorm, decomposed | 8.5% | **yes, see below** |
+| cache slice | 4.1% | marginal |
+
+**int4 weights are not the next lever, on two independent counts.** They do not
+compile: AI Hub's calibrated quantise path fails with `QAIRT converter failed
+with exit code 1` for w4a16 and w4a8, with and without `--quantize_io`. And
+they would not help anyway -- on this NPU 4-bit and 8-bit weights time
+identically, for the layer (0.917 vs 0.912 ms) and for the output head (3.868
+vs 3.862 ms). The calibrated *8-bit* build is faster than the uncalibrated
+4-bit one (0.803 vs 0.838). 4-bit buys memory here, not speed.
+
+**The real remaining item is RMSNorm fusion, which calibration costs us.** The
+uncalibrated build contains two fused `rms` ops worth 1.0% of the layer; the
+calibrated build contains eight decomposed `Pow`/`ReduceMean`/`Div`/`Mul` ops
+worth 8.5%. Inserting QDQ nodes around each op breaks QNN's RMSNorm pattern
+match. The calibrated build is still faster overall and is the only correct
+one, so this is a trade worth making today -- but roughly 7.5% of every layer
+is sitting there for whoever can keep the fusion under quantisation.
+(`F.rms_norm` is not a route to it: the TorchScript ONNX exporter has no
+`aten::rms_norm` at any opset.)
+
+### The decode step, end to end
+
+Every component measured on a real S25 at calibrated w8a16, 1k context:
+
+| component | ms each | ms total | SNR |
+|---|---|---|---|
+| sliding layer x21 (ring) | 0.803 | 16.86 | 36.9 dB |
+| full-attention layer x7 (roll) | 0.942 | 6.59 | 37.4 dB |
+| output head x1 | 3.907 | 3.91 | 62.8 dB |
+| **decode step** | | **27.36** | **36.5 tok/s** |
+
+Against the previous best (roll layout, uncalibrated w4a16, plus the same
+head) at 31.01 ms: **11.8% faster, and numerically valid for the first time.**
+
+The output head is 14% of the step and is 98.6% a single 2048x131072 matmul
+tied to the embedding — there is no overhead in it to remove. It was missing
+from the first version of this document entirely.
+
+At 4k context the full-attention layers grow (2.96 ms each at fp16 vs 1.56 at
+1k) while the sliding layers do not move at all, which is what the hybrid
+architecture buys:
+
+| quant | context | Spark hybrid | all-full counterfactual | gain |
+|---|---|---|---|---|
+| fp16 | 1 k | 46.4 ms | 43.7 ms | 0.94x |
+| fp16 | 4 k | 56.2 ms | 83.0 ms | 1.48x |
+
+At 1k in fp16 the hybrid is *slightly slower*: sliding layers rotate all 256
+head dims where full layers rotate 64, and below roughly 2k that RoPE cost
+outweighs the shorter cache. It pays from ~2k onward and keeps paying, because
+the full-layer term grows with context and the sliding term never does.
+
+### Which phone runtime, and which processor
+
+The same exported sliding layer, one Galaxy S25:
+
+| runtime | processor | precision | ms/layer |
+|---|---|---|---|
+| **QNN context binary** | **Hexagon NPU** | **w8a16 calibrated** | **0.803** |
+| TFLite | CPU | int8 | 0.877 |
+| QNN context binary | Hexagon NPU | fp16 | 1.479 |
+| TFLite | Hexagon NPU | fp16 | 1.491 |
+| ONNX Runtime | Hexagon NPU | fp16 | 1.500 |
+| TFLite | Adreno GPU | fp16 | 2.259 |
+| TFLite | CPU | fp16 | 2.958 |
+
+Two readings. At matched fp16 the three NPU runtimes land within 1.4% of each
+other, so the export layout -- static shapes, explicit caches, GQA without
+expanding K/V -- is what sets on-device speed, not the engine; that reproduces
+what the Qwen3-TTS talker step showed on the same device. And the placement is
+the opposite of that TTS result: Spark's decoder wants the NPU (2.0x the CPU,
+1.5x the GPU) because it is matmul-bound, where the TTS vocoder wanted the GPU
+because it was elementwise-bound.
+
+The row that matters for competition is the int8 CPU one. llama.cpp cannot be
+run on an AI Hub device, and on a phone it would use 4-bit weights on the CPU,
+so comparing our NPU path against an fp16 CPU run would flatter us. The
+closest honest proxy AI Hub offers is the same graph, int8, on the S25's own
+CPU: 0.877 ms against our 0.803 ms, so the NPU path is **9.2% ahead of the
+processor and precision class llama.cpp would actually run in**. That is a
+proxy, not llama.cpp itself.
+
+Two independent checks say the proxy is not flattering us either. A 1.11 GiB
+Q4_K_M build on the S25's ~137 GB/s LPDDR5X has a hard bandwidth ceiling of
+7.9 ms/token at 100% of peak and 13.2 ms at a more realistic 60% — against our
+measured 27.4 ms for a step that includes the output head the bandwidth model
+ignores. Per layer, the bandwidth bound gives roughly 0.28-0.47 ms where the
+measured int8 CPU run takes 0.877 ms, i.e. real CPU execution lands at about a
+third of the bandwidth ceiling, which is the usual outcome on a phone.
+
+## 5. Driving an Android phone
+
+Spark-X2.5 is text-only, so any harness must drive from the accessibility tree
+rather than screenshots. Surveyed:
+
+| harness | transport | text-only a11y mode | local OpenAI endpoint | role |
+|---|---|---|---|---|
+| droidrun/mobilerun | ADB + Portal accessibility service | yes | yes, OpenAI-compatible | control |
+| ghost-in-the-droid/android-agent | ADB + uiautomator, no app on device | yes | lists vLLM, base_url undocumented | control |
+| mobileClaw | on-device app, accessibility | vision-oriented | n/a | control |
+| AndroidWorld (DeepMind) | Android emulator | yes | via custom agent | **evaluation**, 116 scored tasks |
+
+Recommendation: droidrun/mobilerun for control (it is the only one whose
+OpenAI-compatible base URL is documented, so it points at a local vLLM server
+unchanged), AndroidWorld for measuring whether the agent actually succeeds.
+
+`analysis/experiments/spark_edge/android_agent/` implements the action space
+those harnesses share -- `tap`, `type_text`, `swipe`, `press_key`, `open_app`,
+`task_complete`, all indexed against accessibility-tree elements, with the
+`adb shell input` command each maps to -- and runs Spark against ten Android
+screens through vLLM's OpenAI server.
+
+Serving works with vLLM's stock parsers: Spark emits
+`<tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call>`,
+which is GLM-4.5's format, so `--tool-call-parser glm45 --reasoning-parser
+glm45 --enable-auto-tool-choice` parses it as-is. No new parser needed.
+
+| config | correct tool | fully correct action | median decision |
+|---|---|---|---|
+| thinking on | 7-8 / 10 | 5-6 / 10 | 8.9 s |
+| thinking off | 5 / 10 | 4 / 10 | **0.61 s** |
+
+(16 CPU cores, bf16, measured under concurrent benchmark load, so latency is
+pessimistic.)
+
+Thinking is worth roughly two of ten decisions and costs 14.6x the latency.
+With thinking off the model collapses toward answering `tap` for everything.
+With it on, the failure mode is the opposite: on the one genuinely ambiguous
+screen (reach Developer options, which is not visible and needs scrolling) it
+reasoned past a 1200-token budget without ever closing `<think>`, so vLLM's
+reasoning parser -- which only emits reasoning once the block closes --
+returned empty content and no tool call at all. A phone agent on this model
+needs a step budget and a fallback, not just a bigger cap.
+
+The clearest wrong action was tapping an already-on Wi-Fi switch instead of
+calling `task_complete`, i.e. undoing the task it had completed.
+
+No Android device or emulator is attached to this host (`/dev/kvm` absent), so
+these numbers measure the model's decisions over real accessibility-tree
+layouts, not a live device.
+
+### 5b. Driving it in a loop, not one decision at a time
+
+The numbers above measure single decisions over fixed screens. A phone agent
+is a loop, and a loop fails differently. `analysis/experiments/spark_edge/
+android_agent/` is now a working toolkit -- uiautomator XML to an indexed
+element list, an ADB device and a scripted simulator that emits the same XML,
+a tool executor, a tolerant tool-call parser and the loop itself (21 tests,
+no phone and no server required).
+
+**Stock `glm45` does not always parse Spark.** Earlier this note said Spark's
+tool format is GLM-4.5's, so vLLM's parsers work as-is. True on 4 of 5 screens.
+On the Wi-Fi toggle screen -- the one where a switch's state decides the
+action -- the model left the format and emitted
+
+    [tap]
+    {"tool": "tap", "parameters": {"index": 1}
+
+unterminated, with `tool_calls` empty. An agent reading only `tool_calls` sees
+a step with no action. `tool_parse.py` recovers the bracketed form, GLM blocks
+and bare JSON, repairing unbalanced braces, and reports every recovery rather
+than silently repairing.
+
+**Navigation survives quantization; recognising completion does not.** Same
+harness, same simulated phone, one run per cell:
+
+| build | decode | "Turn on Wi-Fi." | "Set an alarm for 7:30 AM." |
+|---|---|---|---|
+| bf16 | 41.0 tok/s | done in 4 steps | wrong app (Settings, not Clock), toggles Airplane mode, stuck |
+| **W4A16 g128** | 80.0 tok/s | done in 4 steps | right app, full navigation, then no tool call |
+| W4A8 g128 | 86.6 tok/s | turns Wi-Fi on, then taps it off again (2 runs) | reaches Alarm, then loops |
+
+Every build walks home -> Settings -> Network -> Wi-Fi correctly. Only some
+notice the switch now reads `checked=true` and stop. W4A8 -- the fastest build,
+and the one a speed-first selector picks -- ends with the task undone.
+
+This is what section 2c's fidelity numbers meant. W4A8 reproducing 1/12 greedy
+prompts against its own bf16, and W4A16 2/12, looked like a small difference on
+a proxy metric; it is the difference between finishing a task and reversing it.
+**For agentic work the 4-bit choice is W4A16**, which `weight_path.py` selects
+under `priority="fidelity"` -- 8% of decode for the difference.
+
+Adding an explicit "if the screen already shows what was asked, call
+task_complete" rule to the system prompt did not rescue W4A8. Nor did feeding
+back "the screen did not change", which bf16 was told five times on the alarm
+task before swiping again.
+
+Limits: two tasks, one run each, on a simulated phone -- enough to show a
+failure mode reproduces, not to rank builds by success rate. The simulator
+emits real uiautomator XML, so the parser, the index-to-coordinate mapping and
+the stopping rules run on the path hardware drives; whether a real Settings app
+is laid out the way these fixtures assume is still untested, because this host
+has no `adb`, no emulator and no `/dev/kvm`.
+
+## 6. Where it stands against the goal
+
+- **Runs on CPU**: yes. Faster than llama.cpp at matched precision on both
+  prefill (8.7-15.4x) and decode (1.19-2.65x), and an int8 path adds a further
+  1.6x.
+- **Runs on Galaxy S25**: yes, measured on real hardware -- a full decode
+  step, output head included, is 27.4 ms at 1 k context (36.5 tok/s) in
+  calibrated w8a16, with every component verified at 36.9-62.8 dB against the
+  reference. That is 11.8% faster than the first working version and the first
+  configuration whose output is actually correct: the uncalibrated w4a16 builds
+  quoted earlier score -1.6 dB, i.e. uncorrelated with the truth.
+- **Beats every edge engine**: on prefill, yes, against everything that can
+  run the model, at any precision -- 4.7x over llama.cpp's best quantized
+  build at 32 threads. On decode it depends on load: llama.cpp Q4_K_M wins
+  single-stream by 1.6x, vLLM int8 wins from batch 16 and leads 1.72x at batch
+  64. The single-stream gap is not closable on vLLM 0.28's CPU backend, which
+  ships no dense int4 kernel.
+- **Beats SGLang**: on decode by 3.8% on GPU (prefill 3.8% the other way), and
+  by default on CPU and phone, where SGLang does not run -- its released
+  `sgl-kernel` wheel has no CPU ops and it has no phone backend at all.
+- **Fastest on the phone**: of every runtime and processor combination
+  measurable on the S25, our QNN NPU path is quickest, including a 9.2% margin
+  over the int8-on-CPU class llama.cpp would run in. llama.cpp itself cannot be
+  run on an AI Hub device, so that specific comparison is a proxy.
+- **Phone control**: the action space, an agent loop and measured decision
+  accuracy are in `analysis/experiments/spark_edge/android_agent/`; serving
+  needs no new parser, just `--tool-call-parser glm45 --reasoning-parser
+  glm45`.
+
+Open items: no dense int4 on CPU (upstream kernel gap); int8 long-context
+fidelity would benefit from a smarter scheme than round-to-nearest; on the NPU
+the next ~7.5% per layer is RMSNorm fusion, which calibration currently breaks
+(int4 weights are neither available nor useful there -- 4-bit and 8-bit time
+identically on this hardware); and none of the phone-agent numbers have been run against a live device, since
+this host has no emulator (`/dev/kvm` absent).