tzhouamandClaude Fable 5.1
073d31b9cd
* vllm_omni/engine/init_timeline.py: InitTimeline phase collector; engine phases (stages_init, g3_device_lock, g2_spawn/ready) and worker phases (proc_entry, engine_core_init, init_device, load_model/load_weights, talker_mtp_init, attn_prewarm, cudagraph_capture, talker_mtp_capture, code2wav_cudagraph, predictor_setup_compile/warmup/cudagraph) recorded via worker_phase/timed_phase; workers append JSONL under VLLM_OMNI_INIT_TIMELINE_DIR (inherited at spawn) and the engine merges them. Enabled by VLLM_OMNI_INIT_TIMELINE=<json> or log_stats; table logged at INFO, partial table on orchestrator startup timeout, suggested timeouts logged. * omni_base.py: named deploy profiles default to stage_init_timeout=900 / init_timeout=1500 unless given explicitly. * gpu_model_runner._init_talker_mtp tolerates max_cudagraph_capture_size=None; gpu_ar_model_runner.shutdown skips CUDA graph teardown on CPU. * benchmarks/tts/init_profile.py: runs stream_latency_bench --init-only per configuration (default, cold_cache, eager_stage1, eager_both, parallel_stage_init, edge, ...) and tabulates phase durations. * stream_latency_bench: --parallel-stage-init / --deploy-profile flags; the resource poller no longer shadows Thread._stop and skips GPU sampling on CPU. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Benchmarks
This directory contains benchmark suites for evaluating different model families and infrastructure components in vLLM-Omni. Each subfolder targets a different benchmark family with its own scripts, configs, and metrics. See the per-subfolder READMEs for detailed usage.
Benchmark families
TTS — Text-to-Speech
Model-agnostic serving benchmarks for TTS models, including Qwen3-TTS and VoxCPM2.
- Layout:
tts/bench_tts.py(serving benchmark driver),tts/model_configs.yaml(model registry),tts/plot_results.py(result plotting) - Dataset: Seed-TTS full or text-only datasets, plus bundled smoke/design prompts under
build_dataset/ - Key metrics: TTFP (time to first audio packet), E2E latency, RTF (real-time factor), throughput (audio seconds / wall-clock second)
Diffusion — Image and Video Generation
Online-serving benchmark for diffusion image/video models, sending requests to the configured vLLM serving endpoint (/v1/chat/completions, /v1/images/generations, or /v1/videos, depending on backend/task).
- Tasks: text-to-image, text-to-video, image-to-image, image-to-video, text+image-to-image, text+image-to-video
- Datasets:
vbench,trace,random - Key metrics: request throughput, latency percentiles, optional SLO attainment
GLM-Image — Text-to-Image and Image-to-Image
Benchmarks for GLM-Image performance across HuggingFace baseline, vLLM-Omni offline inference, and vLLM-Omni online serving.
- Layout:
glm_image/huggingface/(HF baseline),glm_image/vllm-omni/(offline inference),glm_image/benchmark_glm_image.py(online serving) - Tasks: text-to-image and image-to-image
- Key metrics: request/image throughput, latency percentiles, optional per-stage pipeline timings
LingBot-Video — Dense and MoE Parity
Manual cross-runtime validation for the dense LingBot-Video pipeline and the LingBot-Video MoE transformer.
- Dense pipeline: decoded-video MAE, MSE, PSNR, latency, and optional steady-state timings
- MoE transformer: bitwise router, sparse-block, shared-expert, and full-transformer parity
Distributed — RDMA Connector Testing
RDMA environment setup and transfer tests for MooncakeTransferEngineConnector, including pytest-based single-node checks and manual cross-node benchmarks.
- Transfer modes:
copy,zerocopy,gpu(GPUDirect) - Supports: single-node pytest suites and manual multi-node/cross-node transfer testing
Accuracy — Image Generation and Editing Quality
Accuracy benchmarks for image generation/editing models, adapting external suites to vLLM-Omni serving and local judge-evaluation flows.
- Layout:
accuracy/text_to_image/(GEBench),accuracy/image_to_image/(GEdit-Bench) - Method: generation and judge scoring both run through local
vllm-omni serveendpoints
Common serving metrics framework
vllm_omni/benchmarks/ extends vllm bench serve --omni with Omni-specific datasets, backends, and multimodal metrics. Key metrics include:
- Text output: TTFT (time to first token), TPOT (time per output token), ITL (inter-token latency)
- Audio output: TTFP (time to first audio packet), E2E latency, RTF (real-time factor)
- Throughput: request throughput, output token throughput, total token throughput, audio throughput
See vllm_omni/benchmarks/serve.py for the vllm bench serve --omni runner wrapper and vllm_omni/benchmarks/metrics/ for Omni metric definitions.
Adding a new benchmark
- Create a subfolder under
benchmarks/<name>/with scripts, configs if needed, and aREADME.md. - If comparing against another runtime, use clear backend subfolders where applicable, such as
huggingface/andvllm-omni/, or follow the shared TTS serving benchmark pattern intts/. - Add reusable dataset or prompt-building utilities to
build_dataset/if applicable. - Update this README with a link to the new benchmark family.