[Edge][Model] Spark-X2.5 (Spark2_5ForCausalLM) support
Spark-X2.5-1.7B is an Apache-2.0 on-device model built for agentic tool use. vLLM 0.28 has no implementation for it, so it could not be served at all; llama.cpp already ships one, which made it the only edge engine able to run the model. Three traits separate it from a Llama-style decoder: * Hybrid attention -- layer_types interleaves three 512-token sliding layers with one full-attention layer, which is what makes the 1M-token context affordable on a phone. * Per-layer-type RoPE -- sliding layers rotate all 256 head dims with theta 1e4, full layers only the first quarter with theta 5e6. * A head-wise attention output gate -- g_proj reads the same post-norm input as QKV and scales each head's attention output by a sigmoid. The checkpoint also stores QKV pre-fused as a single q_k_v_proj, so load_weights splits it before vLLM's sharded QKVParallelLinear sees it. Verified against the official remote-code implementation on transformers 4.57.1 (the version config.json targets; it does not import on the 5.x in our venvs). In fp32 all three scenarios match exactly -- 32/32 greedy tokens with a max top-20 logprob difference of 2.2e-05. In bf16 the short and the 2465-token long-context prompts still match 32/32, which is what exercises the sliding layers; the tool-calling prompt diverges at token 23 on a near-tie, and fp32 confirms that is rounding rather than a defect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
This commit is contained in:
1 parent
73cdc7395a
commit
e4a565b3df
7 files changed
+678
No files matched your search
@@ -9,6 +9,11 @@ from vllm.model_executor.models.registry import (
|
||||
)
|
||||
|
||||
_OMNI_MODELS = {
|
||||
"Spark2_5ForCausalLM": (
|
||||
"spark2_5",
|
||||
"spark2_5",
|
||||
"Spark2_5ForCausalLM",
|
||||
),
|
||||
"Qwen2_5OmniForConditionalGeneration": (
|
||||
"qwen2_5_omni",
|
||||
"qwen2_5_omni",
|
||||
|
||||
Reference in new issue
Block a user