vllm-omni/benchmarks

Benchmarks

This directory contains benchmark suites for evaluating different model families and infrastructure components in vLLM-Omni. Each subfolder targets a different benchmark family with its own scripts, configs, and metrics. See the per-subfolder READMEs for detailed usage.

Benchmark families

Qwen2.5-Omni — Embedding Wrapper Latency

CPU-observed A/B timings for redundant text embedding removal, with real checkpoint embeddings, warmup, paired blocks, and embedding call-count validation.

TTS — Text-to-Speech

Model-agnostic serving benchmarks for TTS models, including Qwen3-TTS and VoxCPM2.

• Layout: tts/bench_tts.py (serving benchmark driver), tts/model_configs.yaml (model registry), tts/plot_results.py (result plotting)
• Dataset: Seed-TTS full or text-only datasets, plus bundled smoke/design prompts under build_dataset/
• Key metrics: TTFP (time to first audio packet), E2E latency, RTF (real-time factor), throughput (audio seconds / wall-clock second)

Diffusion — Image and Video Generation

Online-serving benchmark for diffusion image/video models, sending requests to the configured vLLM serving endpoint (/v1/chat/completions, /v1/images/generations, or /v1/videos, depending on backend/task).

• Tasks: text-to-image, text-to-video, image-to-image, image-to-video, text+image-to-image, text+image-to-video
• Datasets: vbench, trace, random
• Key metrics: request throughput, latency percentiles, optional SLO attainment

GLM-Image — Text-to-Image and Image-to-Image

Benchmarks for GLM-Image performance across HuggingFace baseline, vLLM-Omni offline inference, and vLLM-Omni online serving.

• Layout: glm_image/huggingface/ (HF baseline), glm_image/vllm-omni/ (offline inference), glm_image/benchmark_glm_image.py (online serving)
• Tasks: text-to-image and image-to-image
• Key metrics: request/image throughput, latency percentiles, optional per-stage pipeline timings

LingBot-Video — Dense and MoE Parity

Manual cross-runtime validation for the dense LingBot-Video pipeline and the LingBot-Video MoE transformer.

• Dense pipeline: decoded-video MAE, MSE, PSNR, latency, and optional steady-state timings
• MoE transformer: bitwise router, sparse-block, shared-expert, and full-transformer parity

LingBot-World — Realtime Streaming Cadence

Client-side benchmark for a WS /v1/realtime/video world-model session, where one request is a whole rollout and one AR block streams out as one video chunk. It measures a cadence rather than a throughput: there is exactly one session, and what matters is whether the next chunk arrives before the viewer finishes the last one.

• Layout: lingbot_world/benchmark_lingbot_world_realtime.py (WS client), lingbot_world/workload.py (rollout and metric math), lingbot_world/configs/ (single-GPU eager and Ulysses-4 compiled deploy configs)
• Workload: built-in image-conditioned rollout with a generated camera script, or a JSON spec with per-chunk actions and mid-rollout prompt updates
• Key metrics: TTFC (time to first chunk), steady-state chunk inter-arrival percentiles, VIDEO_RTF (wall seconds per video second; lower is better), chunk-deadline attainment, and simulated playback underruns

Distributed — RDMA Connector Testing

RDMA environment setup and transfer tests for MooncakeTransferEngineConnector, including pytest-based single-node checks and manual cross-node benchmarks.

• Transfer modes: copy, zerocopy, gpu (GPUDirect)
• Supports: single-node pytest suites and manual multi-node/cross-node transfer testing

Accuracy — Image Generation and Editing Quality

Accuracy benchmarks for image generation/editing models, adapting external suites to vLLM-Omni serving and local judge-evaluation flows.

• Layout: accuracy/text_to_image/ (GEBench), accuracy/image_to_image/ (GEdit-Bench)
• Method: generation and judge scoring both run through local vllm-omni serve endpoints

Common serving metrics framework

vllm_omni/benchmarks/ extends vllm bench serve --omni with Omni-specific datasets, backends, and multimodal metrics. Key metrics include:

• Text output: TTFT (time to first token), TPOT (time per output token), ITL (inter-token latency)
• Audio output: TTFP (time to first audio packet), E2E latency, RTF (real-time factor)
• Throughput: request throughput, output token throughput, total token throughput, audio throughput

See vllm_omni/benchmarks/serve.py for the vllm bench serve --omni runner wrapper and vllm_omni/benchmarks/metrics/ for Omni metric definitions.

Adding a new benchmark

1. Create a subfolder under benchmarks/<name>/ with scripts, configs if needed, and a README.md.
2. If comparing against another runtime, use clear backend subfolders where applicable, such as huggingface/ and vllm-omni/, or follow the shared TTS serving benchmark pattern in tts/.
3. Add reusable dataset or prompt-building utilities to build_dataset/ if applicable.
4. Update this README with a link to the new benchmark family.