torchao's dynamic-activation int8 is numerically wrong under torch.compile on
CPU (cosine 0.02 vs fp32; caused runaway generations), so it is not offered;
weight-only + compile is faithful but slower (78 ms vs 36 ms per frame for the
15 sub-steps), so torch.ao dynamic int8 stays the default. Parity mode now also
records top-5 overlap and KL vs the bf16 model; compiled-step equivalence test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Measured on a Xeon 8480C (12+4 threads, 0.6B): the 15 residual-codebook sub-steps
re-ran the 5-layer predictor over the whole prefix each time (97 ms per frame of
the 134 ms talker step) and vLLM's stock sampler spent 9 ms per step in its
torch.compile'd random-sample kernel on CPU.
- code predictor: forward_cached (KV cache per layer, 2-token prefill then one
token per sub-step) and a shape-static forward_static_step (tensor position +
boolean mask) that torch.compile handles as one graph; incremental decoding
is the default on CPU (VLLM_OMNI_PREDICTOR_INCREMENTAL), optional compile
(VLLM_OMNI_PREDICTOR_COMPILE=step), optional thread cap.
- CPU fast path: dynamic int8 Linear layers (fp32 activations) for the predictor
and its lm heads (VLLM_OMNI_PREDICTOR_INT8, default on; micro-benchmark 70 ->
36 ms per frame for the 15 sub-steps); VLLM_OMNI_PREDICTOR_PARITY_CHECK=1
records top-1 agreement / max logit diff against the bf16 model in situ.
- Qwen3-TTS talker: prefer_model_sampler on CPU with a sample() that mirrors
vLLM's Sampler (logits processors + penalties, temperature, top-k/top-p,
exponential-noise sampling with per-request generators) without the compiled
kernel (VLLM_OMNI_CPU_FAST_SAMPLER, default on).
- step_stats: sampler phase timers and sample-metadata timers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Counters model.forward_ms / model.sample_ms / model.talker_mtp_ms and
predictor.substep_ms / predictor.loop_ms (VLLM_OMNI_STEP_STATS_DIR); with
VLLM_OMNI_STEP_STATS_SYNC=1 the accelerator is synchronized around the model
timers so async kernels are attributed (attribution runs only).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Measured after the DELTA fix: 53 engine outputs for 52 rows, but the client
still received everything at once because each per-step payload is the newest
row only and the row alignment treated it as a cumulative snapshot (one yield
at the first row, one at the end). accumulate_codec_rows folds delta rows and
the final consolidated matrix into one growing matrix; unit test drives the
generator with per-step deltas and asserts one yield per step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The speech path forced FINAL_ONLY for Qwen3-TTS deployments without async
chunks, so a talker-only server emitted one engine output per request and all
codec frames arrived at once (measured: 1 engine output per request, first
codes at output #1). A talker-only stage (final_output + latent) now keeps
DELTA outputs so codec frames stream per step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- environment_variable_inventory: VLLM_OMNI_CPU_INDUCTOR / HW_PROFILE /
INIT_TIMELINE / STEP_STATS_DIR (public), STAGE_ID / INIT_TIMELINE_DIR
(internal); reference page rows added.
- edge_decoder_client.LocalDecoder: batched_chunked_decode takes a [B, Q, F]
tensor, not a list (the WP5 e2e run failed with a TypeError); the decoder
now also carries the ramp schedule attributes the Code2Wav stage sets.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
vllm_omni/edge/decoder_export.py exports a stateless windowed decoder (context
72 frames + chunk, one program per ramp chunk size) via torch.export (.pt2,
verified here on CPU for chunks 2 and 25 with the real 0.6B decoder: exact
match vs eager, 460 MB fp32 per program) or ExecuTorch XNNPACK (.pte; written
against the executorch 1.x API, not executed: package not installed, install
is permission gated; QNN partitioner export-only/unverified). The WP5 client
gains --decoder-backend executorch --pte-manifest for phone/NPU targets.
benchmarks/tts/edge_results/summarize.py renders all committed JSON as tables.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- vllm_omni/edge/step_stats.py: process-local counters enabled by
VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core
step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped
per process and merged into an attribution table (share of the 80 ms frame).
Hooks: stage_engine_core_proc (instrument after construction, dump at exit),
chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime),
orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats.
- vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and
the no-preemption KV admission rule (exact bytes/token from the talker KV
geometry); adapt.derive_overrides now sizes max_num_seqs with it and
auto_profile passes the geometry read from the model config.
- Engine re-reads parallel_stage_init after apply_deploy_profile so the edge
profile's orchestrator default takes effect (was read before the profile).
- benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor)
with per-step attribution.
- Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py,
tests/benchmarks/test_cpu_thread_sweep.py, profile default test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wraps SharedMemoryConnector: CUDA tensor leaves are exported with
torch.multiprocessing reduce_tensor + an inter-process CUDA event, rebuilt and
cloned by the consumer, released by the producer after an ack file appears.
CPU leaves and metadata keep the shm path. Registered as "CudaIpcConnector";
deploy/qwen3_tts_cuda_ipc.yaml selects it for the talker->code2wav hop.
Adoption in the edge profile is measurement-gated (hop latency bench).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* diffusers is no longer imported by vllm_omni.diffusion.data (annotation only);
vllm_omni.diffusion.io_support (diffusion registry + distributed state) is
imported lazily by config_factory and async_omni_engine, so constructing a
Qwen3-TTS pipeline config / AsyncOmni no longer loads diffusers,
x_transformers or whisper (guarded by tests/config/test_import_guard.py).
* requirements/common.txt = core.txt (AR/speech runtime) + media.txt
(diffusion/video/extra audio); VLLM_OMNI_EDGE_BUILD=1 makes setup.py use
requirements/<device>-edge.txt (core only); `edge` extra adds websockets and
sounddevice for on-device clients.
Measured on this host (CUDA venv): `import vllm` 7.0-7.9 s, `import vllm_omni`
adds ~20 s of which ~17 s is transformers' own import (transformers.utils.import_utils)
pulled in by vllm_omni.patch / vllm_omni.version; that floor is not
vLLM-Omni's and is unchanged. Guard test: no diffusion module loaded on the TTS path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch
(ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores,
big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU
via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu /
npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests).
* vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from
available memory minus weights, max_num_seqs, per-stage disjoint CPU thread
ranges on the big cluster, eager/graph switches for small or unified memory,
adaptive-chunk parameters, timeouts from a load cost model.
* vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk
arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT.
* vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml ->
edge/hardware/<class>.yaml -> derived overrides -> cached calibration,
materialized as a temp deploy YAML (platform overlays are superseded).
* benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY,
taskset, profile overrides) driving stream_latency_bench with auto profiles.
* CPU-only tests for probe snapshots, derivation rules, calibration and auto.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
