vllm-omni/benchmarks/qwen2_5_omni/README.md

Qwen2.5-Omni embedding microbenchmark

Measure the host-observed latency of embed_input_ids before and after the redundant text embedding removal in PR #7477. Both variants use the same real checkpoint embedding table, dependencies, inputs and batch shapes. The baseline wrapper is extracted from the specified Git commit; the fixed wrapper comes from the current checkout. No full model is initialized.

Run

Use Linux, a CUDA GPU, the project dependencies, and a local Qwen/Qwen2.5-Omni-7B checkpoint with its safetensors index and weights. From the repository root:

git fetch origin 01a2f93256975c7ff9565c5414b0fe0225bc4765
PYTHONPATH="$PWD" OMP_NUM_THREADS=1 python benchmarks/qwen2_5_omni/benchmark_embeddings.py \
  --model /path/to/Qwen2.5-Omni-7B \
  --baseline-ref 01a2f93256975c7ff9565c5414b0fe0225bc4765 \
  --cpu-core 2 --lengths 1172 --warmup 50 --blocks 20 --iterations 20 \
  --output-dir /tmp/qwen25-embedding-results

Choose a CPU core in your allowed affinity set. For the original CUDA length sweep, use --lengths 512 1172 4096 16384.

Method

• BF16 checkpoint embeddings, TP=1, CUDA int32 token IDs, CPU multimodal masks.
• One pinned CPU core, one PyTorch thread, garbage collection disabled.
• 50 warmups per variant, then 20 paired blocks of 20 calls with balanced, randomized variant order. Summary: median of block medians.
• Host latency includes synchronization inside the wrapper. Explicit GPU synchronization before and after the timed call is excluded from host latency. Calling-thread CPU time and synchronized completion time are recorded separately.
• CPU frequency is sampled at block boundaries; clocks are not locked.
• Validation checks one text embedding call for the fixed wrapper, two for the multimodal baseline, and torch.equal for identical inputs and batch shapes. This does not establish batch invariance or full-model output equality.

Interleaved audio/video uses a valid synthetic multimodal mask. This benchmark measures wrapper latency, not E2E throughput.

Reference results

Original environment: H100 80GB; Python 3.12.3; vLLM 0.29.0+cu129; PyTorch 2.13.0+cu129; Transformers 5.14.0; Qwen2.5-Omni-7B BF16. At 1172 tokens, the September 13 measurements were:

Input	Before (μs)	After (μs)	Host latency reduction
Text	18.81	18.66	0.80%
Audio	104.32	85.30	18.23%
Image	104.72	86.27	17.62%
Video	104.73	85.76	18.11%
Audio + image + video	567.92	549.54	3.24%
Interleaved audio/video	622.76	597.84	4.00%

Results vary between runs and environments. These local savings did not produce a consistent E2E throughput improvement in the separate repeated E2E experiment.

Output

The output directory contains metadata.json, micro-raw.json, validation.json, and summary.json. Raw records include per-call timings, block timestamps and CPU frequency samples. Timer overhead is measured separately.