[Edge][CPU] KV-cached code predictor, int8-dynamic CPU fast path, model-side CPU sampler

Measured on a Xeon 8480C (12+4 threads, 0.6B): the 15 residual-codebook sub-steps
re-ran the 5-layer predictor over the whole prefix each time (97 ms per frame of
the 134 ms talker step) and vLLM's stock sampler spent 9 ms per step in its
torch.compile'd random-sample kernel on CPU.

- code predictor: forward_cached (KV cache per layer, 2-token prefill then one
  token per sub-step) and a shape-static forward_static_step (tensor position +
  boolean mask) that torch.compile handles as one graph; incremental decoding
  is the default on CPU (VLLM_OMNI_PREDICTOR_INCREMENTAL), optional compile
  (VLLM_OMNI_PREDICTOR_COMPILE=step), optional thread cap.
- CPU fast path: dynamic int8 Linear layers (fp32 activations) for the predictor
  and its lm heads (VLLM_OMNI_PREDICTOR_INT8, default on; micro-benchmark 70 ->
  36 ms per frame for the 15 sub-steps); VLLM_OMNI_PREDICTOR_PARITY_CHECK=1
  records top-1 agreement / max logit diff against the bf16 model in situ.
- Qwen3-TTS talker: prefer_model_sampler on CPU with a sample() that mirrors
  vLLM's Sampler (logits processors + penalties, temperature, top-k/top-p,
  exponential-noise sampling with per-request generators) without the compiled
  kernel (VLLM_OMNI_CPU_FAST_SAMPLER, default on).
- step_stats: sampler phase timers and sample-metadata timers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

tests/model_executor/models/common/test_code_predictor_incremental.py

@@ -0,0 +1,79 @@
+"""KV-cached incremental residual decoding of the Qwen3 code predictor equals the re-prefill forward (CPU)."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from vllm_omni.model_executor.models.common.qwen3_code_predictor import CodePredictorBaseModel
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+def _tiny_config():
+    return SimpleNamespace(
+        hidden_size=64,
+        num_hidden_layers=3,
+        num_attention_heads=4,
+        num_key_value_heads=2,
+        head_dim=16,
+        intermediate_size=96,
+        rms_norm_eps=1e-6,
+        vocab_size=32,
+        num_code_groups=16,
+        rope_theta=10000.0,
+        attention_bias=False,
+    )
+
+
+@torch.inference_mode()
+def test_forward_cached_matches_full_prefix():
+    torch.manual_seed(0)
+    model = CodePredictorBaseModel(_tiny_config()).eval()
+    for p in model.parameters():
+        p.normal_(0, 0.2)
+    bsz, max_seq, hidden = 2, 17, 64
+    x = torch.randn(bsz, max_seq, hidden)
+    pos = torch.arange(max_seq).unsqueeze(0).expand(bsz, -1)
+    full = model(x, pos)  # [B, 17, H], causal over the whole prefix
+
+    caches = model.new_kv_caches(bsz, max_seq, x.device, x.dtype)
+    out_prefill = model.forward_cached(x[:, :2], pos[:, :2], caches, 0)
+    assert torch.allclose(out_prefill, full[:, :2], atol=1e-4, rtol=1e-4)
+    for step in range(2, max_seq):
+        out_step = model.forward_cached(x[:, step : step + 1], pos[:, step : step + 1], caches, step)
+        assert torch.allclose(out_step[:, 0], full[:, step], atol=1e-4, rtol=1e-4), step
+    # caches hold exactly the prefix K/V that a fresh full pass would produce
+    assert caches[0][0].shape == (bsz, 2, max_seq, 16)
+
+
+@torch.inference_mode()
+def test_forward_cached_rejects_multi_token_append():
+    model = CodePredictorBaseModel(_tiny_config()).eval()
+    x = torch.randn(1, 2, 64)
+    pos = torch.arange(2).unsqueeze(0)
+    caches = model.new_kv_caches(1, 17, x.device, x.dtype)
+    model.forward_cached(x, pos, caches, 0)
+    with pytest.raises(ValueError):
+        model.forward_cached(x, pos + 2, caches, 2)
+
+
+@torch.inference_mode()
+def test_forward_static_step_matches_full_prefix():
+    torch.manual_seed(1)
+    model = CodePredictorBaseModel(_tiny_config()).eval()
+    for p in model.parameters():
+        p.normal_(0, 0.2)
+    bsz, max_seq = 2, 17
+    x = torch.randn(bsz, max_seq, 64)
+    pos = torch.arange(max_seq).unsqueeze(0).expand(bsz, -1)
+    full = model(x, pos)
+    caches = model.new_kv_caches(bsz, max_seq, x.device, x.dtype)
+    model.forward_cached(x[:, :2], pos[:, :2], caches, 0)  # eager 2-token prefill
+    for step in range(2, max_seq):
+        pos_t = torch.tensor([step])
+        mask = (torch.arange(max_seq) <= step).view(1, 1, 1, max_seq).expand(bsz, 1, 1, max_seq)
+        out = model.forward_static_step(x[:, step : step + 1], pos[:, step : step + 1], caches, pos_t, mask)
+        assert torch.allclose(out[:, 0], full[:, step], atol=1e-4, rtol=1e-4), step