The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of
rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while
our 0.28-based tree needs seven modules v0.29.0 no longer has
(vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args,
openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant,
models.hunyuan_v1).
Ten files conflicted. Upstream's structure is the base everywhere; the
fork's features are re-attached to it:
- deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config
pops deploy_config (async_omni_engine, cli/serve); the deploy-profile
selector and the resolve_model_config_path chain it needs stay in
entrypoints/utils.py; stage_overrides takes rc1's dict type.
- config_factory keeps the lazy diffusion output-type import; the diffusion
registry import in rc1's new config/resolver.py is made lazy for the same
WP6 reason (TTS path must not load diffusion).
- requirements: our core/media split kept; rc1's transformers>=5.13 floor
and kernels==0.16.1 applied in core.txt / media.txt.
- TTS: rc1's split_granularity and bounded seed plus our output_mode=
codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation;
our private _conditioning_cache_salt is superseded by rc1's public
tts_adapters/base.py::conditioning_cache_salt.
- stage_runtime: rc1's single ExitStack G2 path (lock released before READY)
replaces our two-path block, with the four worker_phase names the init
profiler consumes still emitted; _cpu_placed / host admission kept.
Not yet run against vLLM 0.29.0: the tests listed in the W6 record.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Noise is drawn eagerly with the request generators (same RNG consumption as the
eager loop) and passed in, so the compiled post-step is generator-free and
shape-static; used on the fp32/torchao CPU path outside stored (top-p) mode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
In-situ parity against the bf16 model (VLLM_OMNI_PREDICTOR_PARITY_CHECK=1):
fp32 control keeps 93.8 % top-1 / 95 % top-5 / KL 0.0015; torch.ao dynamic
int8 falls to 22 % top-1 (max logit diff 7), so per-tensor activation
quantization is not acceptable as a default. Adds the ao-nohead variant
(int8 transformer body, fp32 lm heads) for measurement.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
torchao's dynamic-activation int8 is numerically wrong under torch.compile on
CPU (cosine 0.02 vs fp32; caused runaway generations), so it is not offered;
weight-only + compile is faithful but slower (78 ms vs 36 ms per frame for the
15 sub-steps), so torch.ao dynamic int8 stays the default. Parity mode now also
records top-5 overlap and KL vs the bf16 model; compiled-step equivalence test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Measured on a Xeon 8480C (12+4 threads, 0.6B): the 15 residual-codebook sub-steps
re-ran the 5-layer predictor over the whole prefix each time (97 ms per frame of
the 134 ms talker step) and vLLM's stock sampler spent 9 ms per step in its
torch.compile'd random-sample kernel on CPU.
- code predictor: forward_cached (KV cache per layer, 2-token prefill then one
token per sub-step) and a shape-static forward_static_step (tensor position +
boolean mask) that torch.compile handles as one graph; incremental decoding
is the default on CPU (VLLM_OMNI_PREDICTOR_INCREMENTAL), optional compile
(VLLM_OMNI_PREDICTOR_COMPILE=step), optional thread cap.
- CPU fast path: dynamic int8 Linear layers (fp32 activations) for the predictor
and its lm heads (VLLM_OMNI_PREDICTOR_INT8, default on; micro-benchmark 70 ->
36 ms per frame for the 15 sub-steps); VLLM_OMNI_PREDICTOR_PARITY_CHECK=1
records top-1 agreement / max logit diff against the bf16 model in situ.
- Qwen3-TTS talker: prefer_model_sampler on CPU with a sample() that mirrors
vLLM's Sampler (logits processors + penalties, temperature, top-k/top-p,
exponential-noise sampling with per-request generators) without the compiled
kernel (VLLM_OMNI_CPU_FAST_SAMPLER, default on).
- step_stats: sampler phase timers and sample-metadata timers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Counters model.forward_ms / model.sample_ms / model.talker_mtp_ms and
predictor.substep_ms / predictor.loop_ms (VLLM_OMNI_STEP_STATS_DIR); with
VLLM_OMNI_STEP_STATS_SYNC=1 the accelerator is synchronized around the model
timers so async kernels are attributed (attribution runs only).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
