tzhouam and Claude Fable 5.1
073d31b9cd
[Edge][WP2] Add initialization timeline and cold-start profiler
...
* vllm_omni/engine/init_timeline.py: InitTimeline phase collector; engine
phases (stages_init, g3_device_lock, g2_spawn/ready) and worker phases
(proc_entry, engine_core_init, init_device, load_model/load_weights,
talker_mtp_init, attn_prewarm, cudagraph_capture, talker_mtp_capture,
code2wav_cudagraph, predictor_setup_compile/warmup/cudagraph) recorded via
worker_phase/timed_phase; workers append JSONL under
VLLM_OMNI_INIT_TIMELINE_DIR (inherited at spawn) and the engine merges them.
Enabled by VLLM_OMNI_INIT_TIMELINE=<json> or log_stats; table logged at INFO,
partial table on orchestrator startup timeout, suggested timeouts logged.
* omni_base.py: named deploy profiles default to stage_init_timeout=900 /
init_timeout=1500 unless given explicitly.
* gpu_model_runner._init_talker_mtp tolerates max_cudagraph_capture_size=None;
gpu_ar_model_runner.shutdown skips CUDA graph teardown on CPU.
* benchmarks/tts/init_profile.py: runs stream_latency_bench --init-only per
configuration (default, cold_cache, eager_stage1, eager_both,
parallel_stage_init, edge, ...) and tabulates phase durations.
* stream_latency_bench: --parallel-stage-init / --deploy-profile flags; the
resource poller no longer shadows Thread._stop and skips GPU sampling on CPU.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 11:01:39 +08:00
tzhouam and Claude Fable 5.1
b70207f57f
[Edge][WP0] Add offline streaming-latency benchmark for chunked TTS
...
Adds benchmarks/tts/stream_latency_bench.py (AsyncOmni driver reusing the
Qwen3-TTS example's prompt builder; --model override, warm-up/repeat,
--init-only, per-chunk arrivals, peak RSS and per-process GPU memory) and
benchmarks/tts/stream_latency_metrics.py (TTFA, underrun-free playback start,
stall at TTFA, RTF streamed/total, chunk gaps, percentiles) with CPU unit
tests. Results go under benchmarks/tts/edge_results/stream_latency/.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 10:24:53 +08:00
