vllm-omni/benchmarks/qwen3-omni/README.md

Benchmarks Guide

This README explains how to (1) prepare benchmark datasets and (2) run the provided Qwen3-Omni benchmarks.

1) Prepare the dataset (SeedTTS top100)

cd benchmarks/build_dataset
pip install gdown

# Download SeedTTS test set from Google Drive
gdown --id 1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP

# Extract
tar -xf seedtts_testset.tar

# Copy metadata and extract top-100 prompts
cp seedtts_testset/en/meta.lst meta.lst
python extract_prompts.py -i meta.lst -o top100.txt -n 100

# (Optional) clean up to save space
rm -rf seedtts_testset seedtts_testset.tar meta.lst

Artifacts:

• benchmarks/build_dataset/top100.txt — 100 text prompts (one per line).

2) Run benchmarks

All commands assume repo root (vllm-omni).

A. Transformers benchmark (offline, HF Transformers)

bash benchmarks/qwen3-omni/transformers/eval_qwen3_moe_omni_transformers.sh

What it does:

• Runs qwen3_omni_moe_transformers.py over top100.txt with --num-prompts 100.
• Outputs to benchmarks/qwen3-omni/transformers/benchmark_results/:
  • perf_stats.json — aggregated & per-prompt TPS/latency (thinker/talker/code2wav/overall).
  • results.json — per-prompt outputs and audio paths.
  • audio/ — ~100 generated .wav files.

Key checks:

• overall_tps and *_tps_avg should be non-zero and reasonably stable.
• Investigate any 0/NaN or unusually low TPS / long-tail latency.

B. vLLM Omni end-to-end benchmark (pipeline)

bash benchmarks/qwen3-omni/vllm_omni/eval_qwen3_moe_omni.sh

What it does:

• Runs examples/offline_inference/qwen3_omni/end2end.py with --log-stats.
• Uses benchmarks/build_dataset/top100.txt and writes to:
  • Logs: benchmarks/qwen3-omni/vllm_omni/logs/
    • omni_pipeline_text.orchestrator.stats.jsonl — per-stage latency stats.
    • omni_pipeline_text.overall.stats.jsonl — end-to-end latency/TPS.
    • omni_pipeline_text.stage{0,1,2}.log — per-stage detailed logs/errors.
  • Outputs: benchmarks/qwen3-omni/vllm_omni/outputs/ — ~100 text and .wav files.

Key checks:

• Overall stats: end-to-end latency/TPS should be reasonable.
• Orchestrator stats: per-stage latency should be stable; investigate long tails.
• Stage logs: ensure no errors and no unusually slow stages.

Performance snapshot

The chart below summarizes our measured Qwen3-Omni MoE end-to-end benchmark, comparing vLLM-Omni against HF Transformers. It shows the overall throughput advantage for vLLM-Omni. These are actual experiment results—please refer to this performance when evaluating or reproducing the benchmark.

Directory layout

• benchmarks/build_dataset/ — dataset prep utilities (e.g., SeedTTS top100).
• benchmarks/<model>/vllm_omni/ — vLLM-Omni pipeline benchmarks, logs, outputs.
• Add new tasks under benchmarks/<model>/... with the same pattern: transformers/, vllm_omni/, task-specific README, and (optionally) dataset prep notes.
• benchmarks/<model>/vllm-omni-vs-hf.png — current performance snapshot (overall throughput comparison).
• benchmarks/<model>/transformers/ — HF Transformers benchmarks (offline reference).

Troubleshooting

• Make sure GPU/driver/FlashAttention2 requirements are met for the chosen model.
• If downloads fail, confirm network access to Google Drive (gdown) and Hugging Face.
• If audio files are missing, check for errors in stage logs or model generation.***