[CI][Omni-DuplexEval] Add nightly scoring CI guard (#7773)
Signed-off-by: RyanYun09 <318555231+RyanYun09@users.noreply.github.com>
This commit is contained in:
1 parent
d521b3f917
commit
732d6085be
11 files changed
+1283
-5
No files matched your search
@@ -0,0 +1,135 @@
|
||||
# Omni-DuplexEval CI Guard
|
||||
|
||||
Nightly end-to-end scoring regression for [Hothan/Omni-DuplexEval]
|
||||
(RTD 6 subtasks + PR 3 subtasks), driven by `vllm bench omni-duplex-eval`
|
||||
(generate → evaluate → summarize).
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| ------ | --------- |
|
||||
| `test_omni_duplex_eval_ci.py` | Main pytest: hardcoded thresholds + 6 assertions |
|
||||
| `test_omni_duplex_eval_ci_mock.py` | CPU mock integration tests (no GPU / model needed) |
|
||||
| `omni_duplex_eval_ci_config.json` | Dataset / revision / sample IDs / judge |
|
||||
| `conftest.py` | `judge_server` fixture (endpoint resolution only) |
|
||||
| `README.md` | This document |
|
||||
| `results/.gitignore` | Track empty results dir for Buildkite artifact upload |
|
||||
|
||||
## Run locally
|
||||
|
||||
Judge server must be pre-started; this test never launches it:
|
||||
|
||||
```bash
|
||||
# CUDA (H100 / B200), or NPU (A3)
|
||||
VLLM_DUPLEX_EVAL_JUDGE_URL=http://127.0.0.1:8001 \
|
||||
pytest -s -v tests/e2e/accuracy/omni_duplex_eval/ \
|
||||
-m "full_model and cuda and (H100 or B200) and cards_1"
|
||||
```
|
||||
|
||||
The `omni_server` fixture boots MiniCPM-o-4_5 with the community duplex
|
||||
params (`tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py`).
|
||||
`judge.base_url_env` (default `VLLM_DUPLEX_EVAL_JUDGE_URL`) overrides
|
||||
`judge.base_url` from the JSON config.
|
||||
|
||||
## Results output
|
||||
|
||||
After Phase 3 (summarize), the full `summarize_scores()` dict is persisted
|
||||
to `results/summary.json` (same pattern as PR#6817 perf tests) and printed
|
||||
to stdout:
|
||||
|
||||
- `results/` is tracked in git via `results/.gitignore` (empty dir, nothing
|
||||
committed inside).
|
||||
- On Buildkite nightly, the step uploads `results/*.json` as artifacts
|
||||
(`buildkite-agent artifact upload`) even when assertions fail, so a
|
||||
failing run's scores remain downloadable.
|
||||
|
||||
## Judge server
|
||||
|
||||
Judge is a pre-started, OpenAI-compatible endpoint (never launched by this
|
||||
test). It can run on a separate machine / GPU from the `omni_server`.
|
||||
|
||||
Current judge: **Qwen2.5-VL-7B-Instruct** (`judge.model` in the JSON
|
||||
config). The 7B model is chosen for Phase A to run on a single GPU (≈16 GB
|
||||
FP16, or ≈5 GB with INT4 quantization); PR 0/1 classification and temporal
|
||||
0–3 scoring are robust at this size. The 0.00–3.00 content scoring is the
|
||||
weakest link (fine-grained counting / color / spatial checks) — if the
|
||||
guard proves insensitive to regressions, upgrade the judge (e.g.
|
||||
72B on FP8 H100 or an external API) and **re-run the baseline** to reset
|
||||
`_MIN_*` thresholds. Changing the judge never affects the P0 assertions
|
||||
(`protocol_pin`, sample count, `by_task` keys) or the CPU mock tests.
|
||||
|
||||
Example (any OpenAI-compatible vLLM server):
|
||||
|
||||
```bash
|
||||
python -m vllm.entrypoints.openai.api_server \
|
||||
--model Qwen/Qwen2.5-VL-7B-Instruct --port 8001 --max-model-len 65536 \
|
||||
--allowed-local-media-path /tmp/omni_duplex_ci
|
||||
```
|
||||
|
||||
If the judge is started on a different host or the media temporary directory
|
||||
changes, adjust `--allowed-local-media-path` to match the path used by
|
||||
`vllm bench omni-duplex-eval evaluate` for materialized videos.
|
||||
|
||||
## CPU mock tests (no GPU / model / judge needed)
|
||||
|
||||
`test_omni_duplex_eval_ci_mock.py` drives the *real*
|
||||
`test_omni_duplex_eval_ci()` guard on CPU by stubbing the heavy I/O:
|
||||
|
||||
- `_run_cli` is replaced by an argv-recording spy (never executes).
|
||||
- `resolve_ref_audio` is stubbed to a local path.
|
||||
- Fake score files are pre-written under `tmp_path/scores/<split>/` from
|
||||
`sample_ids_per_split` (minus `exclude_ids`), so `summarize_scores()`
|
||||
and the Phase-4 assertions run for real.
|
||||
|
||||
It validates the CLI argument assembly (generate + evaluate), the
|
||||
score-file-count check, `protocol_pin`, the RTD/PR `by_task` keys, and
|
||||
all threshold comparisons — including the failure paths. Runs in CI on
|
||||
every PR (`core_model` + `cpu` marks):
|
||||
|
||||
```bash
|
||||
pytest -v tests/e2e/accuracy/omni_duplex_eval/test_omni_duplex_eval_ci_mock.py
|
||||
```
|
||||
|
||||
## Thresholds (record mode → tighten after first nightly run)
|
||||
|
||||
`sample_ids_per_split` contains real dataset IDs (verified against the
|
||||
Hothan/Omni-DuplexEval splits; PR_correction / PR_event_reminder IDs are
|
||||
inferred from the split row order and must be re-verified once the dataset
|
||||
is reachable). `exclude_ids` has been removed.
|
||||
|
||||
All `_MIN_*` thresholds are currently **0.0 (record mode)**: the nightly run
|
||||
produces real scores and persists `results/summary.json` without failing the
|
||||
gate. Tighten after the first nightly run:
|
||||
|
||||
1. Record `summarize_scores()` output (mean content / temporal / success).
|
||||
2. Set `_MIN_*` thresholds = `mean - n * std` (start `n=2`) in
|
||||
`test_omni_duplex_eval_ci.py`. As a starting point, the paper's
|
||||
MiniCPM-o 4.5 scores (100-point scale) map to ≈1.15 content / ≈2.40
|
||||
temporal / ≈0.20 PR on the code scales (see "Reference paper" below);
|
||||
~50% of those is a reasonable lower bound.
|
||||
3. If needed, exclude known bad samples (generate hang / judge failure) via
|
||||
`exclude_ids`.
|
||||
|
||||
Thresholds are bound to the judge model: **switching the judge requires
|
||||
re-running steps 1–2**. If the 7B judge shows high variance (small models
|
||||
are more prompt-sensitive), widen `n` (e.g. `n=3`) or increase the sample
|
||||
size before tightening.
|
||||
|
||||
## Assertions (in order)
|
||||
|
||||
1. `protocol_pin == PROTOCOL_PIN` (track upstream protocol changes)
|
||||
2. `summary["samples"]` == number of selected IDs
|
||||
3. RTD `by_task` keys == 6 RTD subtasks
|
||||
4. PR `by_task` keys == 3 PR subtasks
|
||||
5. RTD mean content / temporal scores >= thresholds
|
||||
6. PR `mean_all_success` >= threshold
|
||||
|
||||
## Reference paper & scoring scale
|
||||
|
||||
Reference paper: **Omni-DuplexEval** (arXiv:2605.17360). The paper reports
|
||||
a 100-point scale, while the evaluation code
|
||||
(`vllm_omni/benchmarks/duplex/omni_duplex_eval_metrics.py`) uses a 3-point
|
||||
content scale (0.00–3.00), a 0–3 integer temporal scale, and a 0/1 PR
|
||||
success flag. The CI thresholds are defined against the **code's** scales
|
||||
(the source of truth for `summarize_scores()` output), not the paper's
|
||||
100-point scale.
|
||||
Reference in new issue
Block a user