The score-space join removed the concat as intended and is numerically
identical (140.3 dB), but it cannot be quantized: replacing torch.softmax
with an explicit max/exp/sum/div made QNN's context-binary converter exit 14
under w4a16, while every fp16 build compiled. Two earlier shapes failed the
same way -- a broadcast multiply plus ReduceSum for the new token's score,
and a two-axis broadcast multiply for its value contribution.
The device profile says the win was never the concat anyway. Of a sliding
layer, Output was 9.9%, its quantise conversion 3.1% and the window Slice
7.4% -- all of it the cost of handing a 512-entry window back across the
graph boundary once per token. So keep roll's attention exactly as it was,
one fused Softmax over concatenated keys, and change only what the layer
returns: the single new K/V entry, which the runtime drops into the ring
slot. Ring order needs no fixing up, because softmax over keys is
order-independent and each cached key already carries its own rotary phase.
Measured on a real Galaxy S25, sliding layer, ms:
w4a16 0.917 -> 0.838 (-8.6%) <- now compiles
fp16 1.689 -> 1.486 (-12.0%)
Full-attention layers keep roll: ring gains them nothing (1.567 vs 1.559 at
fp16, the concat saved being spent on the extra score matmul), so the
default layout is now per layer type.
Parity against the reference is 140.3 dB for the mixed default with
identical top-1/top-5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Reading the device profile of the first export showed a fifth of every
sliding layer went nowhere useful: Output 9.9%, its quantise conversion
3.1%, and the window Slice 7.4%. The layer took a 512-entry window in,
appended the new key, sliced the oldest off and handed the whole recomputed
window back out -- 21 of 28 layers paying to move a cache-sized tensor
across the graph boundary once per token. Full layers paid a similar 9.5%
concatenating the new key onto the cache.
Both are the same mistake, and both go away by joining in score space
instead of key space: score the cache and the new token separately, then
concatenate two vectors of length n and 1 rather than an n x 256 key matrix.
Nothing cache-sized is copied, and each layer returns just the new K/V entry
for the runtime to drop into its ring slot. Softmax over keys does not care
what order the slots are in and each cached key already carries its own
rotary phase, so a ring buffer needs no reordering -- only a mask for slots
not yet written. The sliding cache correspondingly holds window-1 entries,
since a 512-token window counts the current token (transformers' own sliding
cache holds 511 for the same reason).
Measured on a real Galaxy S25, fp16, ms per layer:
sliding 1.689 -> 1.483 (-12.2%)
full 1.559 -> 1.567 (unchanged: the concat saved is spent on the
extra score matmul)
Two shapes had to be written as matmuls rather than the obvious elementwise
forms -- the new token's score (broadcast multiply + ReduceSum) and its
value contribution (a two-axis broadcast multiply). Both compile at fp16 and
both make QNN's context-binary converter exit 14 under w4a16; as matmuls
they compile and are the cheaper op on HTP anyway.
Also adds a selectable GELU approximation, and the measurement that closes
the question: erf GELU is 11-13% of every layer, but tanh is *slower* on
this NPU (1.535 vs 1.483 ms) despite holding 65.2 dB. The exact form stays,
which is also the better-fidelity choice.
Parity against the reference implementation is 136.8 dB with identical
top-1/top-5, and a test pins ring and roll to the same output on a shared
history so the device numbers cannot silently measure a different model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Exports one Spark decode step with the KV cache as explicit tensors, sized
per layer type: the 21 sliding layers hold a fixed 512 entries at any
context, the 7 full-attention layers grow. That asymmetry is the whole
reason the architecture is interesting on a phone, so the two types get
separately sized caches, masks and rotary angles.
Verified against the HF reference at 1024 context: 138.5 dB logits SNR,
identical top-1 and top-5.
Profiled on a real Galaxy S25 (Snapdragon 8 Elite) through AI Hub, QNN
context binary on the NPU, ms per layer:
sliding (512 KV, any ctx) fp16 1.689 w8a16 0.912 w4a16 0.917
full @ 1k fp16 1.559 w8a16 1.129 w4a16 1.127
full @ 4k fp16 2.964 w8a16 1.681 w4a16 1.662
Composed into a step (21 sliding + 7 full), w4a16 gives 27.2 ms at 1k and
30.9 ms at 4k, against 31.6 and 46.5 ms for the same model with every layer
full-attention -- 1.16x and 1.51x. Worth noting the hybrid is 0.94x at 1k in
fp16: sliding layers rotate all 256 head dims where full layers rotate 64,
and below roughly 2k that RoPE cost outweighs the shorter cache.
Also moves the head-wise attention gate off vLLM's CPU linear dispatch. The
gate is [2048, 8], and any linear that narrow is routed to the sgl-kernel
packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim
0; VNNI packing does not at that size, so it reports K and torch.compile
fails tracing the gate multiply. A plain parameter with F.linear gains
nothing less -- a 2048x8 matmul has no use for packed AMX tiles -- and it
keeps the model compilable on CPU and GPU alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
