torchao's dynamic-activation int8 is numerically wrong under torch.compile on
CPU (cosine 0.02 vs fp32; caused runaway generations), so it is not offered;
weight-only + compile is faithful but slower (78 ms vs 36 ms per frame for the
15 sub-steps), so torch.ao dynamic int8 stays the default. Parity mode now also
records top-5 overlap and KL vs the bf16 model; compiled-step equivalence test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Measured on a Xeon 8480C (12+4 threads, 0.6B): the 15 residual-codebook sub-steps
re-ran the 5-layer predictor over the whole prefix each time (97 ms per frame of
the 134 ms talker step) and vLLM's stock sampler spent 9 ms per step in its
torch.compile'd random-sample kernel on CPU.
- code predictor: forward_cached (KV cache per layer, 2-token prefill then one
token per sub-step) and a shape-static forward_static_step (tensor position +
boolean mask) that torch.compile handles as one graph; incremental decoding
is the default on CPU (VLLM_OMNI_PREDICTOR_INCREMENTAL), optional compile
(VLLM_OMNI_PREDICTOR_COMPILE=step), optional thread cap.
- CPU fast path: dynamic int8 Linear layers (fp32 activations) for the predictor
and its lm heads (VLLM_OMNI_PREDICTOR_INT8, default on; micro-benchmark 70 ->
36 ms per frame for the 15 sub-steps); VLLM_OMNI_PREDICTOR_PARITY_CHECK=1
records top-1 agreement / max logit diff against the bf16 model in situ.
- Qwen3-TTS talker: prefer_model_sampler on CPU with a sample() that mirrors
vLLM's Sampler (logits processors + penalties, temperature, top-k/top-p,
exponential-noise sampling with per-request generators) without the compiled
kernel (VLLM_OMNI_CPU_FAST_SAMPLER, default on).
- step_stats: sampler phase timers and sample-metadata timers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
