[Edge][Model] Spark-X2.5 (Spark2_5ForCausalLM) support
Spark-X2.5-1.7B is an Apache-2.0 on-device model built for agentic tool use. vLLM 0.28 has no implementation for it, so it could not be served at all; llama.cpp already ships one, which made it the only edge engine able to run the model. Three traits separate it from a Llama-style decoder: * Hybrid attention -- layer_types interleaves three 512-token sliding layers with one full-attention layer, which is what makes the 1M-token context affordable on a phone. * Per-layer-type RoPE -- sliding layers rotate all 256 head dims with theta 1e4, full layers only the first quarter with theta 5e6. * A head-wise attention output gate -- g_proj reads the same post-norm input as QKV and scales each head's attention output by a sigmoid. The checkpoint also stores QKV pre-fused as a single q_k_v_proj, so load_weights splits it before vLLM's sharded QKVParallelLinear sees it. Verified against the official remote-code implementation on transformers 4.57.1 (the version config.json targets; it does not import on the 5.x in our venvs). In fp32 all three scenarios match exactly -- 32/32 greedy tokens with a max top-20 logprob difference of 2.2e-05. In bf16 the short and the 2465-token long-context prompts still match 32/32, which is what exercises the sliding layers; the tool-calling prompt diverges at token 23 on a near-tie, and fp32 confirms that is rounding rather than a defect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
This commit is contained in:
1 parent
73cdc7395a
commit
e4a565b3df
7 files changed
+678
No files matched your search
@@ -75,6 +75,9 @@ def _register_omni_hf_configs() -> None:
|
||||
from vllm_omni.model_executor.models.qwen3_tts.configuration_qwen3_tts import (
|
||||
Qwen3TTSConfig,
|
||||
)
|
||||
from vllm_omni.model_executor.models.spark2_5.configuration_spark2_5 import (
|
||||
Spark2_5Config,
|
||||
)
|
||||
from vllm_omni.transformers_utils.configs.cosyvoice3 import CosyVoice3Config
|
||||
from vllm_omni.transformers_utils.configs.glm_tts import GLMTTSConfig
|
||||
from vllm_omni.transformers_utils.configs.omnivoice import OmniVoiceConfig
|
||||
@@ -99,6 +102,7 @@ def _register_omni_hf_configs() -> None:
|
||||
("moss_tts_local", MossTTSLocalConfig),
|
||||
("moss_tts_realtime", MossTTSRealtimeConfig),
|
||||
("qwen3_tts", Qwen3TTSConfig),
|
||||
("spark2_5", Spark2_5Config),
|
||||
("personaplex", PersonaPlexConfig),
|
||||
("nemotron_voicechat", NemotronVoiceChatConfig),
|
||||
("cosyvoice3", CosyVoice3Config),
|
||||
|
||||
Reference in new issue
Block a user