Single-stream CPU decode reads every weight to emit one token, so bits per
weight set the token rate. This registers a W4A16 group-wise method
out-of-tree through register_quantization_config -- vLLM itself is untouched
-- using PyTorch's _weight_int4pack_mm_for_cpu tinygemm kernel.
Two paths over one set of weights, because that kernel dequantizes once per
row of activations and so is linear in the batch (12.9x slower than bf16 at
512 rows). Single-row steps take tinygemm; wider batches unpack the weight to
bf16 once and use the ordinary oneDNN/AMX GEMM. Shipping only the first
measured pp512 = 557 tok/s against bf16's 4455; with both, 3589.
The batch threshold is 2 rather than the ~8 rows where the unpack path starts
winning on speed, because running tinygemm on a multi-row batch and then
vLLM's cpu_attention_with_kv_cache in the same forward dies with SIGILL
inside the attention kernel -- an AMX instruction with no valid tile config
loaded. The same prompt is fine in bf16 and fine once every multi-row GEMM
takes the unpack path.
Positioning, measured on a Xeon 8480C with 32 pinned cores and a warm cache
(analysis/spark_cpu_int4.md): vLLM's own GPTQ W4A8 AMX path is faster --
88.7 tok/s decode and 4355 prefill against 85.5 and 3589 here -- but its
config rejects sym=False at 4 bits, so it can only be fed a symmetric grid.
This one is asymmetric and costs +12.5% perplexity against that path's
+27.2%, and needs no AMX. llama.cpp Q4_K_M, for reference, is 98.0 tok/s
decode and 1356 prefill at +7.6% and 5.15 bits per weight.
Expanding a quantized checkpoint back to bf16 offline and scoring it
reproduces the quantized run to within 0.2% perplexity, so both GEMM paths,
the nibble packing and the quantized tied embedding are faithful.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXhbAfkSNRaeqZrus5Z2fx
Spark-X2.5-1.7B is an Apache-2.0 on-device model built for agentic tool
use. vLLM 0.28 has no implementation for it, so it could not be served at
all; llama.cpp already ships one, which made it the only edge engine able
to run the model.
Three traits separate it from a Llama-style decoder:
* Hybrid attention -- layer_types interleaves three 512-token sliding
layers with one full-attention layer, which is what makes the 1M-token
context affordable on a phone.
* Per-layer-type RoPE -- sliding layers rotate all 256 head dims with
theta 1e4, full layers only the first quarter with theta 5e6.
* A head-wise attention output gate -- g_proj reads the same post-norm
input as QKV and scales each head's attention output by a sigmoid.
The checkpoint also stores QKV pre-fused as a single q_k_v_proj, so
load_weights splits it before vLLM's sharded QKVParallelLinear sees it.
Verified against the official remote-code implementation on transformers
4.57.1 (the version config.json targets; it does not import on the 5.x in
our venvs). In fp32 all three scenarios match exactly -- 32/32 greedy
tokens with a max top-20 logprob difference of 2.2e-05. In bf16 the short
and the 2465-token long-context prompts still match 32/32, which is what
exercises the sliding layers; the tool-calling prompt diverges at token 23
on a near-tie, and fp32 confirms that is rounding rather than a defect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Noise is drawn eagerly with the request generators (same RNG consumption as the
eager loop) and passed in, so the compiled post-step is generator-free and
shape-static; used on the fp32/torchao CPU path outside stored (top-p) mode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
torchao's dynamic-activation int8 is numerically wrong under torch.compile on
CPU (cosine 0.02 vs fp32; caused runaway generations), so it is not offered;
weight-only + compile is faithful but slower (78 ms vs 36 ms per frame for the
15 sub-steps), so torch.ao dynamic int8 stays the default. Parity mode now also
records top-5 overlap and KL vs the bf16 model; compiled-step equivalence test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Measured on a Xeon 8480C (12+4 threads, 0.6B): the 15 residual-codebook sub-steps
re-ran the 5-layer predictor over the whole prefix each time (97 ms per frame of
the 134 ms talker step) and vLLM's stock sampler spent 9 ms per step in its
torch.compile'd random-sample kernel on CPU.
- code predictor: forward_cached (KV cache per layer, 2-token prefill then one
token per sub-step) and a shape-static forward_static_step (tensor position +
boolean mask) that torch.compile handles as one graph; incremental decoding
is the default on CPU (VLLM_OMNI_PREDICTOR_INCREMENTAL), optional compile
(VLLM_OMNI_PREDICTOR_COMPILE=step), optional thread cap.
- CPU fast path: dynamic int8 Linear layers (fp32 activations) for the predictor
and its lm heads (VLLM_OMNI_PREDICTOR_INT8, default on; micro-benchmark 70 ->
36 ms per frame for the 15 sub-steps); VLLM_OMNI_PREDICTOR_PARITY_CHECK=1
records top-1 agreement / max logit diff against the bf16 model in situ.
- Qwen3-TTS talker: prefer_model_sampler on CPU with a sample() that mirrors
vLLM's Sampler (logits processors + penalties, temperature, top-k/top-p,
exponential-noise sampling with per-request generators) without the compiled
kernel (VLLM_OMNI_CPU_FAST_SAMPLER, default on).
- step_stats: sampler phase timers and sample-metadata timers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
