Spark-X2.5-1.7B is an Apache-2.0 on-device model built for agentic tool
use. vLLM 0.28 has no implementation for it, so it could not be served at
all; llama.cpp already ships one, which made it the only edge engine able
to run the model.
Three traits separate it from a Llama-style decoder:
* Hybrid attention -- layer_types interleaves three 512-token sliding
layers with one full-attention layer, which is what makes the 1M-token
context affordable on a phone.
* Per-layer-type RoPE -- sliding layers rotate all 256 head dims with
theta 1e4, full layers only the first quarter with theta 5e6.
* A head-wise attention output gate -- g_proj reads the same post-norm
input as QKV and scales each head's attention output by a sigmoid.
The checkpoint also stores QKV pre-fused as a single q_k_v_proj, so
load_weights splits it before vLLM's sharded QKVParallelLinear sees it.
Verified against the official remote-code implementation on transformers
4.57.1 (the version config.json targets; it does not import on the 5.x in
our venvs). In fp32 all three scenarios match exactly -- 32/32 greedy
tokens with a max top-20 logprob difference of 2.2e-05. In bf16 the short
and the 2465-token long-context prompts still match 32/32, which is what
exercises the sliding layers; the tool-calling prompt diverges at token 23
on a near-tie, and fp32 confirms that is rounding rather than a defect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
- vllm_omni/edge/step_stats.py: process-local counters enabled by
VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core
step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped
per process and merged into an attribution table (share of the 80 ms frame).
Hooks: stage_engine_core_proc (instrument after construction, dump at exit),
chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime),
orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats.
- vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and
the no-preemption KV admission rule (exact bytes/token from the talker KV
geometry); adapt.derive_overrides now sizes max_num_seqs with it and
auto_profile passes the geometry read from the model config.
- Engine re-reads parallel_stage_init after apply_deploy_profile so the edge
profile's orchestrator default takes effect (was read before the profile).
- benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor)
with per-step attribution.
- Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py,
tests/benchmarks/test_cpu_thread_sweep.py, profile default test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* diffusers is no longer imported by vllm_omni.diffusion.data (annotation only);
vllm_omni.diffusion.io_support (diffusion registry + distributed state) is
imported lazily by config_factory and async_omni_engine, so constructing a
Qwen3-TTS pipeline config / AsyncOmni no longer loads diffusers,
x_transformers or whisper (guarded by tests/config/test_import_guard.py).
* requirements/common.txt = core.txt (AR/speech runtime) + media.txt
(diffusion/video/extra audio); VLLM_OMNI_EDGE_BUILD=1 makes setup.py use
requirements/<device>-edge.txt (core only); `edge` extra adds websockets and
sounddevice for on-device clients.
Measured on this host (CUDA venv): `import vllm` 7.0-7.9 s, `import vllm_omni`
adds ~20 s of which ~17 s is transformers' own import (transformers.utils.import_utils)
pulled in by vllm_omni.patch / vllm_omni.version; that floor is not
vLLM-Omni's and is unchanged. Guard test: no diffusion module loaded on the TTS path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* vllm_omni/deploy/edge/qwen3_tts.yaml: single-device Qwen3-TTS profile with a
ramped codec chunk schedule (2->4->8->16->25 frames) that removes the ~80 ms
playback stall of the default 1->25 schedule, decode batch 1, max_num_seqs 4.
* deploy_profile="edge" (Omni/AsyncOmni kwarg, --deploy-profile CLI, headless
serve) resolves vllm_omni/deploy/<profile>/<model_type>.yaml; mutually
exclusive with deploy_config; explicit profiles never fall back silently.
* tests/helpers/stage_config.py: modify_stage_config anchors a relative
base_config before writing the temp YAML so overlay-derived configs resolve.
* async_omni_engine.py also carries the init-timeline hooks used by WP2.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
