vllm-omni/tests/e2e/accuracy/omni_duplex_eval

Omni-DuplexEval CI Guard

Nightly end-to-end scoring regression for [Hothan/Omni-DuplexEval] (RTD 6 subtasks + PR 3 subtasks), driven by vllm bench omni-duplex-eval (generate → evaluate → summarize).

Files

File	Purpose
test_omni_duplex_eval_ci.py	Main pytest: hardcoded thresholds + 6 assertions
test_omni_duplex_eval_ci_mock.py	CPU mock integration tests (no GPU / model needed)
omni_duplex_eval_ci_config.json	Dataset / revision / sample IDs / judge
conftest.py	judge_server fixture (endpoint resolution only)
README.md	This document
results/.gitignore	Track empty results dir for Buildkite artifact upload

Run locally

Judge server must be pre-started; this test never launches it:

# CUDA (H100 / B200), or NPU (A3)
VLLM_DUPLEX_EVAL_JUDGE_URL=http://127.0.0.1:8001 \
  pytest -s -v tests/e2e/accuracy/omni_duplex_eval/ \
    -m "full_model and cuda and (H100 or B200) and cards_1"

The omni_server fixture boots MiniCPM-o-4_5 with the community duplex params (tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py). judge.base_url_env (default VLLM_DUPLEX_EVAL_JUDGE_URL) overrides judge.base_url from the JSON config.

Results output

After Phase 3 (summarize), the full summarize_scores() dict is persisted to results/summary.json (same pattern as PR#6817 perf tests) and printed to stdout:

• results/ is tracked in git via results/.gitignore (empty dir, nothing committed inside).
• On Buildkite nightly, the step uploads results/*.json as artifacts (buildkite-agent artifact upload) even when assertions fail, so a failing run's scores remain downloadable.

Judge server

Judge is a pre-started, OpenAI-compatible endpoint (never launched by this test). It can run on a separate machine / GPU from the omni_server.

Current judge: Qwen2.5-VL-7B-Instruct (judge.model in the JSON config). The 7B model is chosen for Phase A to run on a single GPU (≈16 GB FP16, or ≈5 GB with INT4 quantization); PR 0/1 classification and temporal 0–3 scoring are robust at this size. The 0.00–3.00 content scoring is the weakest link (fine-grained counting / color / spatial checks) — if the guard proves insensitive to regressions, upgrade the judge (e.g. 72B on FP8 H100 or an external API) and re-run the baseline to reset _MIN_* thresholds. Changing the judge never affects the P0 assertions (protocol_pin, sample count, by_task keys) or the CPU mock tests.

Example (any OpenAI-compatible vLLM server):

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-VL-7B-Instruct --port 8001 --max-model-len 65536 \
  --allowed-local-media-path /tmp/omni_duplex_ci

If the judge is started on a different host or the media temporary directory changes, adjust --allowed-local-media-path to match the path used by vllm bench omni-duplex-eval evaluate for materialized videos.

CPU mock tests (no GPU / model / judge needed)

test_omni_duplex_eval_ci_mock.py drives the real test_omni_duplex_eval_ci() guard on CPU by stubbing the heavy I/O:

• _run_cli is replaced by an argv-recording spy (never executes).
• resolve_ref_audio is stubbed to a local path.
• Fake score files are pre-written under tmp_path/scores/<split>/ from sample_ids_per_split (minus exclude_ids), so summarize_scores() and the Phase-4 assertions run for real.

It validates the CLI argument assembly (generate + evaluate), the score-file-count check, protocol_pin, the RTD/PR by_task keys, and all threshold comparisons — including the failure paths. Runs in CI on every PR (core_model + cpu marks):

pytest -v tests/e2e/accuracy/omni_duplex_eval/test_omni_duplex_eval_ci_mock.py

Thresholds (record mode → tighten after first nightly run)

sample_ids_per_split contains real dataset IDs (verified against the Hothan/Omni-DuplexEval splits; PR_correction / PR_event_reminder IDs are inferred from the split row order and must be re-verified once the dataset is reachable). exclude_ids has been removed.

All _MIN_* thresholds are currently 0.0 (record mode): the nightly run produces real scores and persists results/summary.json without failing the gate. Tighten after the first nightly run:

1. Record summarize_scores() output (mean content / temporal / success).
2. Set _MIN_* thresholds = mean - n * std (start n=2) in test_omni_duplex_eval_ci.py. As a starting point, the paper's MiniCPM-o 4.5 scores (100-point scale) map to ≈1.15 content / ≈2.40 temporal / ≈0.20 PR on the code scales (see "Reference paper" below); ~50% of those is a reasonable lower bound.
3. If needed, exclude known bad samples (generate hang / judge failure) via exclude_ids.

Thresholds are bound to the judge model: switching the judge requires re-running steps 1–2. If the 7B judge shows high variance (small models are more prompt-sensitive), widen n (e.g. n=3) or increase the sample size before tightening.

Assertions (in order)

1. protocol_pin == PROTOCOL_PIN (track upstream protocol changes)
2. summary["samples"] == number of selected IDs
3. RTD by_task keys == 6 RTD subtasks
4. PR by_task keys == 3 PR subtasks
5. RTD mean content / temporal scores >= thresholds
6. PR mean_all_success >= threshold

Reference paper & scoring scale

Reference paper: Omni-DuplexEval (arXiv:2605.17360). The paper reports a 100-point scale, while the evaluation code (vllm_omni/benchmarks/duplex/omni_duplex_eval_metrics.py) uses a 3-point content scale (0.00–3.00), a 0–3 integer temporal scale, and a 0/1 PR success flag. The CI thresholds are defined against the code's scales (the source of truth for summarize_scores() output), not the paper's 100-point scale.