Commit Graph
2 Commits
Author SHA1 Message Date
tzhouamandClaude Opus 5 658341531a [Edge][Mobile] Spark-X2.5 decode-step export, and a compile-safe head gate
Exports one Spark decode step with the KV cache as explicit tensors, sized
per layer type: the 21 sliding layers hold a fixed 512 entries at any
context, the 7 full-attention layers grow. That asymmetry is the whole
reason the architecture is interesting on a phone, so the two types get
separately sized caches, masks and rotary angles.

Verified against the HF reference at 1024 context: 138.5 dB logits SNR,
identical top-1 and top-5.

Profiled on a real Galaxy S25 (Snapdragon 8 Elite) through AI Hub, QNN
context binary on the NPU, ms per layer:

  sliding (512 KV, any ctx)  fp16 1.689  w8a16 0.912  w4a16 0.917
  full @ 1k                  fp16 1.559  w8a16 1.129  w4a16 1.127
  full @ 4k                  fp16 2.964  w8a16 1.681  w4a16 1.662

Composed into a step (21 sliding + 7 full), w4a16 gives 27.2 ms at 1k and
30.9 ms at 4k, against 31.6 and 46.5 ms for the same model with every layer
full-attention -- 1.16x and 1.51x. Worth noting the hybrid is 0.94x at 1k in
fp16: sliding layers rotate all 256 head dims where full layers rotate 64,
and below roughly 2k that RoPE cost outweighs the shorter cache.

Also moves the head-wise attention gate off vLLM's CPU linear dispatch. The
gate is [2048, 8], and any linear that narrow is routed to the sgl-kernel
packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim
0; VNNI packing does not at that size, so it reports K and torch.compile
fails tracing the gate multiply. A plain parameter with F.linear gains
nothing less -- a 2048x8 matmul has no use for packed AMX tiles -- and it
keeps the model compilable on CPU and GPU alike.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-11 14:49:10 +08:00
tzhouamandClaude Opus 5 e4a565b3df [Edge][Model] Spark-X2.5 (Spark2_5ForCausalLM) support
Spark-X2.5-1.7B is an Apache-2.0 on-device model built for agentic tool
use. vLLM 0.28 has no implementation for it, so it could not be served at
all; llama.cpp already ships one, which made it the only edge engine able
to run the model.

Three traits separate it from a Llama-style decoder:

* Hybrid attention -- layer_types interleaves three 512-token sliding
  layers with one full-attention layer, which is what makes the 1M-token
  context affordable on a phone.
* Per-layer-type RoPE -- sliding layers rotate all 256 head dims with
  theta 1e4, full layers only the first quarter with theta 5e6.
* A head-wise attention output gate -- g_proj reads the same post-norm
  input as QKV and scales each head's attention output by a sigmoid.

The checkpoint also stores QKV pre-fused as a single q_k_v_proj, so
load_weights splits it before vLLM's sharded QKVParallelLinear sees it.

Verified against the official remote-code implementation on transformers
4.57.1 (the version config.json targets; it does not import on the 5.x in
our venvs). In fp32 all three scenarios match exactly -- 32/32 greedy
tokens with a max top-20 logprob difference of 2.2e-05. In bf16 the short
and the 2465-token long-context prompts still match 32/32, which is what
exercises the sliding layers; the tool-calling prompt diverges at token 23
on a near-tie, and fp32 confirms that is rounding rather than a defect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-11 13:47:02 +08:00