From 795e4e0160c06ded82d25d0a11f37cf89ee3bb58 Mon Sep 17 00:00:00 2001 From: wangyu <53896905+yenuo26@users.noreply.github.com> Date: Wed, 15 Jul 2026 14:18:58 +0800 Subject: [PATCH] [CI] Add local job runner scripts for L2-L4 pytest execution (#4672) Signed-off-by: wangyu <410167048@qq.com> --- docs/contributing/ci/test_guide.md | 137 +++- .../perf/scripts/run_diffusion_benchmark.py | 2 +- tools/nightly/run_nightly_jobs.sh | 350 +++++++++- tools/run_jobs_common.sh | 653 ++++++++++++++++++ tools/run_merge_jobs.sh | 49 ++ tools/run_ready_jobs.sh | 49 ++ 6 files changed, 1177 insertions(+), 63 deletions(-) create mode 100644 tools/run_jobs_common.sh create mode 100644 tools/run_merge_jobs.sh create mode 100644 tools/run_ready_jobs.sh diff --git a/docs/contributing/ci/test_guide.md b/docs/contributing/ci/test_guide.md index 0db814e08..26996129e 100644 --- a/docs/contributing/ci/test_guide.md +++ b/docs/contributing/ci/test_guide.md @@ -12,12 +12,37 @@ When running test cases, you may need to install the following dependencies: ```bash uv pip install ".[dev]" -apt-get install -y ffmpeg +apt-get install -y espeak-ng jq ``` ## Running Tests Our test scripts use the pytest framework. First, please use `git clone https://github.com/vllm-project/vllm-omni.git` to download the vllm-omni source code. Then, in the root directory of vllm-omni, you can run the following commands in your local test environment to execute the corresponding test cases. +### CI job runners (L2–L4): logs, timing, and timeouts + +[`tools/run_ready_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_ready_jobs.sh), [`tools/run_merge_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_merge_jobs.sh), and [`tools/nightly/run_nightly_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/nightly/run_nightly_jobs.sh) share the same run-time behavior (via [`tools/run_jobs_common.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_jobs_common.sh)): + +**Log layout** (default under `logs/ready_jobs`, `logs/merge_jobs`, or `logs/nightly_jobs`; override with `--log-dir`): + +| Path | Description | +|------|-------------| +| `jobs/*.sh` | Generated per-step bash wrappers | +| `jobs/.job_timeouts` | Job key → `timeout_in_minutes` from Buildkite YAML (when set) | +| `.log` | Tee output for each job | +| `timing_summary.log` | Per-job duration and total wall time after the run completes | + +**Timing:** while each job runs, stderr prints `finished in …` or `failed after …`. When all jobs finish, a summary is printed and written to `timing_summary.log`, for example: + +``` +=== Job timing summary === + TTS_VoxCPM2_Test 25m 40s OK + Omni_Qwen3-Omni_Test 8m 05s TIMED OUT +Total wall time: 33m 45s (2 jobs) +Failed jobs: 1/2 +``` + +**Timeouts:** if a Buildkite step defines `timeout_in_minutes`, the runner wraps that job with `timeout ${N}m` (aligned with CI). Steps without `timeout_in_minutes` rely on any inline `timeout …` already present in the extracted pytest command. On timeout the process exits with code `124` and the summary marks the job as `TIMED OUT`. + === "L1 level" ```bash @@ -28,61 +53,109 @@ Our test scripts use the pytest framework. First, please use `git clone https:// === "L2 level" + **Recommended:** run CI-aligned jobs from the repo root via [`tools/run_ready_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_ready_jobs.sh). The script reads [`.buildkite/test-ready.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-ready.yml), generates per-step bash wrappers, runs pytest, and tees logs under `logs/ready_jobs/` (see [CI job runners](#ci-job-runners-l2l4-logs-timing-and-timeouts) for timing and timeout behavior). Requires `bash`, `python3`, and PyYAML (`pip install pyyaml`). + + ```bash + # All L2 and L1 jobs (default: every pytest step in test-ready.yml) + bash tools/run_ready_jobs.sh + + # Skip L1-style "Simple Test" steps in test-ready.yml (group or "Simple ·" labels) + bash tools/run_ready_jobs.sh --skip-simple + bash tools/run_ready_jobs.sh --skip-simple --model-type diffusion + + # Preview extracted commands without running + bash tools/run_ready_jobs.sh --dry-run + + # Filter by model area (OR semantics): omni | tts | diffusion | all + bash tools/run_ready_jobs.sh --model-type omni + bash tools/run_ready_jobs.sh --model-type tts,diffusion + + # Match a Buildkite label substring (e.g. one model suite) + bash tools/run_ready_jobs.sh --model-type tts --label-substr "VoxCPM2" + + ``` + + **Ad hoc:** run a single test file or marker expression manually: + ```bash cd tests - pytest -s -v -m "core_model and not cpu" --run-level=core_model - ``` - If you only want to run a specific test case, you can use: - ```bash pytest -s -v test_xxxx.py --run-level=core_model + pytest -s -v -m "core_model and distributed_cuda and L4" --run-level=core_model ``` - If you only want to run specific test cases on a particular platform, you can use: - ```bash - pytest -s -v -m "core_model and distributed_cuda and L4" --run-level=core_model - ``` - The latest test commands for various test suites can be found in the [pipeline](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-ready.yml). === "L3 level" + **Recommended:** run CI-aligned jobs from the repo root via [`tools/run_merge_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_merge_jobs.sh). The script reads [`.buildkite/test-merge.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-merge.yml), generates per-step bash wrappers, runs pytest, and tees logs under `logs/merge_jobs/` (see [CI job runners](#ci-job-runners-l2l4-logs-timing-and-timeouts)). Requires `bash`, `python3`, and PyYAML (`pip install pyyaml`). + ```bash - pytest -s -v -m "advanced_model" --run-level=advanced_model + # All L3 and L1 jobs (default: every pytest step in test-merge.yml) + bash tools/run_merge_jobs.sh + + # Skip L1-style "Simple Test" steps in test-merge.yml (group or "Simple ·" labels) + bash tools/run_merge_jobs.sh --skip-simple + bash tools/run_merge_jobs.sh --skip-simple --model-type omni + + # Preview extracted commands without running + bash tools/run_merge_jobs.sh --dry-run + + # Filter by model area (OR semantics): omni | tts | diffusion | all + bash tools/run_merge_jobs.sh --model-type diffusion + bash tools/run_merge_jobs.sh --model-type omni,tts + + # Match a Buildkite label substring + bash tools/run_merge_jobs.sh --model-type diffusion --label-substr "Wan22" + ``` - If you only want to run a specific test case, you can use: + + **Ad hoc:** run a single test file or marker expression manually: + ```bash pytest -s -v test_xxxx.py --run-level=advanced_model + pytest -s -v -m "advanced_model and distributed_cuda and L4" --run-level=advanced_model ``` - If you only want to run specific test cases on a particular platform, you can use: - ```bash - pytest -s -v -m "advanced_model and distributed_cuda and L4" --run-level=advanced_model - ``` - The latest L3 test commands for various test suites can be found in the [pipeline](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-merge.yml). === "L4 level" + **Recommended:** run CI-aligned nightly jobs from the repo root via [`tools/nightly/run_nightly_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/nightly/run_nightly_jobs.sh). The script reads [`.buildkite/test-nightly.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly.yml), generates per-step bash wrappers, runs pytest (perf jobs first, then aggregates perf JSON into Excel), and tees logs under `logs/nightly_jobs/` (see [CI job runners](#ci-job-runners-l2l4-logs-timing-and-timeouts); `generate_nightly_perf_excel` is included in the timing summary). Requires `bash`, `python3`, and PyYAML (`pip install pyyaml`). + ```bash - cd tests - pytest -s -v -m "full_model" --run-level=full_model + # All nightly jobs (default: test-type all, model-type all) + bash tools/nightly/run_nightly_jobs.sh + + # Preview extracted commands without running + bash tools/nightly/run_nightly_jobs.sh --dry-run + + # Test kind (--test-type, OR semantics): perf | acc | function | stability | local | all + bash tools/nightly/run_nightly_jobs.sh --test-type function + bash tools/nightly/run_nightly_jobs.sh --test-type perf,acc + + # Model area (--model-type, OR semantics): omni | tts | diffusion | all + bash tools/nightly/run_nightly_jobs.sh --test-type function --model-type omni + bash tools/nightly/run_nightly_jobs.sh --test-type perf --model-type diffusion + + # Stability scripts (tests/dfx/stability/scripts/) and local marker tests + bash tools/nightly/run_nightly_jobs.sh --test-type stability --model-type omni + bash tools/nightly/run_nightly_jobs.sh --test-type local --model-type tts ``` - If you only want to run a specific test case, you can use: - ```bash - pytest -s -v test_xxxx.py --run-level=full_model - ``` - If you only want to run specific test cases on a particular platform, you can use: - ```bash - pytest -s -v -m "full_model and distributed_cuda and L4" --run-level=full_model - ``` - Note: To run performance tests (defaults to ``test_qwen_omni.json``; use ``--test-config-file tests/dfx/perf/tests/test_tts.json`` for TTS): - ```bash - pytest -s -v tests/dfx/perf/scripts/run_benchmark.py - ``` - The latest L4 (nightly) test commands use the `full_model` marker and `--run-level full_model` (see [test-nightly.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly.yml) and [test-nightly-diffusion.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly-diffusion.yml)). Example: + + Perf steps (label contains ``Perf Test``) run before other jobs; Excel output is written under ``logs/``. Diffusion nightly steps may also appear in [test-nightly-diffusion.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly-diffusion.yml) on CI. + + **Ad hoc:** run a single test file or marker expression manually: ```bash cd tests + pytest -s -v test_xxxx.py --run-level=full_model + pytest -s -v -m "full_model and distributed_cuda and L4" --run-level=full_model pytest -s -v -m "full_model and (omni or tts) and H100" --run-level=full_model ``` + To run a standalone perf benchmark (defaults to ``test_qwen_omni.json``; use ``--test-config-file tests/dfx/perf/tests/test_tts.json`` for TTS): + + ```bash + pytest -s -v tests/dfx/perf/scripts/run_benchmark.py + ``` + === "L5 level" L5 includes stability and reliability testing. Typical commands: diff --git a/tests/dfx/perf/scripts/run_diffusion_benchmark.py b/tests/dfx/perf/scripts/run_diffusion_benchmark.py index 4c185066d..b0860978b 100644 --- a/tests/dfx/perf/scripts/run_diffusion_benchmark.py +++ b/tests/dfx/perf/scripts/run_diffusion_benchmark.py @@ -36,7 +36,7 @@ import pytest from benchmarks.diffusion.backends import endpoint_filename_token, normalize_endpoint -pytestmark = [pytest.mark.diffusion, pytest.mark.full_model] +pytestmark = [pytest.mark.diffusion, pytest.mark.full_model, pytest.mark.local_model] os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn" os.environ.setdefault("DIFFUSION_ATTENTION_BACKEND", "FLASH_ATTN") diff --git a/tools/nightly/run_nightly_jobs.sh b/tools/nightly/run_nightly_jobs.sh index b115a9ee2..7134a36a1 100644 --- a/tools/nightly/run_nightly_jobs.sh +++ b/tools/nightly/run_nightly_jobs.sh @@ -9,7 +9,7 @@ # python tools/nightly/generate_nightly_perf_excel.py runs (even if some perf jobs failed; # the script only aggregates JSON already present under tests/dfx/perf/results). # Excel and generate_nightly_perf_excel.log are written under $REPO_ROOT/logs/; -# per-job tee logs use LOG_DIR (see --log-dir). Then remaining jobs run. +# per-job tee logs use LOG_DIR (see --log-dir; default is timestamped under $REPO_ROOT/logs/). # # Test kind (--test-type / TEST_TYPE), multiple allowed (OR semantics): # Repeat the flag and/or use comma-separated values, e.g. @@ -21,6 +21,8 @@ # function — label has neither "Perf Test" nor "Accuracy Test" (incl. Doc, Multi-Replica, etc.) # stability— fixed dfx stability scripts under tests/dfx/stability/scripts/ (see below) # local — pytest -sv -m " and local_model" from repo root (no YAML step extract) +# When LABEL_SUBSTR is set, also runs matching tests/**/test_*.py and perf JSON configs under +# tests/dfx/perf/tests/*.json via run_benchmark.py / run_diffusion_benchmark.py. # all — no test-kind filter for YAML steps (any leaf step with pytest in test-nightly.yml) # # Model area (--model-type / MODEL_TYPE), multiple allowed (OR semantics): @@ -36,10 +38,12 @@ # local (when included in TEST_TYPE): # From repo root: pytest -sv -m " and local_model" (markers from MODEL_TYPE: omni, tts, # diffusion; all → "(omni or tts or diffusion) and local_model"). Not filtered by nightly YAML. +# LABEL_SUBSTR: if set, restrict to tests/**/test_*.py basenames and tests/dfx/perf/tests/*.json +# whose filename contains the substring (benchmark runner chosen by JSON family). # # stability (when included in TEST_TYPE): -# From repo root: pytest -s -v tests/dfx/stability/scripts/test_stability_*.py -# model_type: omni → qwen3_omni; tts → qwen3_tts; diffusion → qwen_image + wan22; all → all four +# From repo root: pytest -s -v --run-level full_model tests/dfx/stability/scripts/test_stability_*.py +# model_type: omni → qwen3_omni; tts → qwen3_tts + voxcpm2; diffusion → qwen_image + wan22 + hunyuan_image; all → all six # LABEL_SUBSTR: if set, script path / job key / filename must contain it # # Requirements: bash, python3, PyYAML (pip install pyyaml) @@ -58,11 +62,20 @@ # Optional environment: # REPO_ROOT - vllm-omni root (working directory for pytest); see above # YML - path to test-nightly.yml (default: $REPO_ROOT/.buildkite/test-nightly.yml) -# LOG_DIR - logs + generated job scripts (default: $REPO_ROOT/logs/nightly_jobs) +# LOG_DIR - logs + generated job scripts; when unset, a timestamped directory under +# $REPO_ROOT/logs/ is created: +# nightly_jobs_YYYYMMDD-HHMMSS (default / YAML nightly steps) +# nightly_local_jobs_YYYYMMDD-HHMMSS (--test-type local only) +# nightly_stability_jobs_YYYYMMDD-HHMMSS (--test-type stability only) +# per-job *.log plus timing_summary.log after the run; +# perf JSON from tests/dfx/perf/results/ (produced this run) -> $LOG_DIR/perf_results/ # TEST_TYPE - comma-separated and/or repeated flags (default: all); see above # MODEL_TYPE - comma-separated and/or repeated flags (default: all); see above -# LABEL_SUBSTR - YAML mode: substring of Buildkite step label; stability: substring of path/key/filename +# LABEL_SUBSTR - YAML mode: substring of Buildkite step label; stability: substring of path/key/filename; +# local: substring of test_*.py basename or tests/dfx/perf/tests/*.json filename # DRY_RUN=1 - print extracted commands only; do not write scripts or run pytest +# RUN_JOB_TIMEOUT_KILL_AFTER - seconds after SIGTERM before SIGKILL on inline pytest +# timeout (default: 60; baked into generated job scripts) # set -euo pipefail @@ -212,6 +225,28 @@ done TEST_TYPE="$(_finalize_test_type_csv)" MODEL_TYPE="$(_finalize_model_type_csv)" +# Default log directory basename: date + local timestamp; prefix depends on TEST_TYPE. +_nightly_log_dir_basename() { + local ts has_local=0 has_stability=0 has_yaml=0 _t + ts="$(date +%Y%m%d-%H%M%S)" + IFS=',' read -ra _TTOK <<< "${TEST_TYPE}" + for _t in "${_TTOK[@]}"; do + _t="$(printf '%s' "${_t}" | tr '[:upper:]' '[:lower:]' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')" + case "${_t}" in + local) has_local=1 ;; + stability) has_stability=1 ;; + perf | acc | function | all) has_yaml=1 ;; + esac + done + if [[ "${has_local}" -eq 1 && "${has_yaml}" -eq 0 && "${has_stability}" -eq 0 ]]; then + printf 'nightly_local_jobs_%s' "${ts}" + elif [[ "${has_stability}" -eq 1 && "${has_yaml}" -eq 0 && "${has_local}" -eq 0 ]]; then + printf 'nightly_stability_jobs_%s' "${ts}" + else + printf 'nightly_jobs_%s' "${ts}" + fi +} + BUILDKITE_REL=".buildkite/test-nightly.yml" _find_repo_containing_nightly() { @@ -269,7 +304,11 @@ else YML="${REPO_ROOT}/${BUILDKITE_REL}" fi -LOG_DIR="${LOG_DIR:-${REPO_ROOT}/logs/nightly_jobs}" +if [[ -z "${LOG_DIR:-}" ]]; then + LOG_DIR="${REPO_ROOT}/logs/$(_nightly_log_dir_basename)" +fi + +echo "Log directory: ${LOG_DIR}" >&2 # Require nightly YAML only when at least one non-stability test kind needs it (validated in Python too). _needs_nightly_yml=0 @@ -310,6 +349,7 @@ if [[ "${DRY_RUN}" != "1" ]]; then rm -f "${_stale_logs[@]}" fi rm -f "${LOG_DIR}/jobs/.perf_job_keys" + rm -f "${LOG_DIR}/jobs/.job_timeouts" rm -f "${REPO_ROOT}/logs/nightly_perf_manual.xlsx" rm -f "${LOG_DIR}/nightly_perf_manual.xlsx" fi @@ -320,6 +360,7 @@ from __future__ import annotations import os import re +import shlex import stat import sys from pathlib import Path @@ -491,12 +532,60 @@ def job_key_from_step(step: dict, label: str) -> str: return slug +def step_timeout_minutes(step: dict) -> int | None: + raw = step.get("timeout_in_minutes") + if raw is None: + return None + try: + minutes = int(raw) + except (TypeError, ValueError): + return None + return minutes if minutes > 0 else None + + +_INLINE_TIMEOUT_PREFIX = re.compile(r"^timeout\s+(?:-\S+\s+)*\S+\s+", re.IGNORECASE) + + +def timeout_kill_after_seconds() -> int: + raw = os.environ.get("RUN_JOB_TIMEOUT_KILL_AFTER", "60") + try: + seconds = int(raw) + except (TypeError, ValueError): + return 60 + return seconds if seconds > 0 else 60 + + +def prepend_timeout_to_pytest(pytest_line: str, timeout_min: int | None) -> str: + """Prepend GNU timeout directly before pytest (Buildkite inline style).""" + if timeout_min is None: + return pytest_line.strip() + line = _INLINE_TIMEOUT_PREFIX.sub("", pytest_line.strip()) + kill_after = timeout_kill_after_seconds() + return ( + f"timeout --foreground --verbose --kill-after={kill_after} " + f"{timeout_min}m {line}" + ) + + +def _write_job_timeouts_manifest(jobs_dir: Path, job_timeouts: dict[str, int]) -> None: + manifest_path = jobs_dir / ".job_timeouts" + if job_timeouts: + manifest_path.write_text( + "\n".join(f"{k}={v}" for k, v in sorted(job_timeouts.items())) + "\n", + encoding="utf-8", + ) + elif manifest_path.is_file(): + manifest_path.unlink() + + # Fixed stability scripts (when stability is selected); MODEL_TYPES narrows which run. STABILITY_CASES: list[tuple[str, str, tuple[str, ...]]] = [ ("stability_qwen3_omni", "tests/dfx/stability/scripts/test_stability_qwen3_omni.py", ("omni",)), ("stability_qwen3_tts", "tests/dfx/stability/scripts/test_stability_qwen3_tts.py", ("tts",)), + ("stability_voxcpm2", "tests/dfx/stability/scripts/test_stability_voxcpm2.py", ("tts",)), ("stability_qwen_image", "tests/dfx/stability/scripts/test_stability_qwen_image.py", ("diffusion",)), ("stability_wan22", "tests/dfx/stability/scripts/test_stability_wan22.py", ("diffusion",)), + ("stability_hunyuan_image", "tests/dfx/stability/scripts/test_stability_hunyuan_image.py", ("diffusion",)), ] @@ -533,14 +622,148 @@ def local_pytest_marker_expr(model_types: list[str]) -> str: return f"({' or '.join(families)}) and local_model" -def run_local_mode(jobs_dir: Path, model_types: list[str]) -> int: - """Single job: pytest -sv -m ' and local_model' (testpaths from pytest config).""" +def local_test_files_by_filename(substr: str) -> list[str]: + """Return repo-relative posix paths under tests/ whose basename contains substr.""" + tests_root = REPO_ROOT / "tests" + if not tests_root.is_dir(): + return [] + matches: list[str] = [] + for path in sorted(tests_root.rglob("test_*.py")): + if substr in path.name: + matches.append(path.relative_to(REPO_ROOT).as_posix()) + return matches + + +PERF_TESTS_REL = Path("tests/dfx/perf/tests") +RUN_BENCHMARK_REL = Path("tests/dfx/perf/scripts/run_benchmark.py") +RUN_DIFFUSION_BENCHMARK_REL = Path("tests/dfx/perf/scripts/run_diffusion_benchmark.py") +TTS_PERF_JSON_HINTS = ("test_tts", "voxcpm", "higgs_audio") + + +def perf_json_model_family(json_basename: str) -> str: + """Classify a perf JSON config as omni, tts, or diffusion (mirrors nightly YAML runners).""" + name = json_basename.lower() + if name.startswith("test_qwen3_omni"): + return "omni" + if any(hint in name for hint in TTS_PERF_JSON_HINTS): + return "tts" + return "diffusion" + + +def perf_json_runner(json_basename: str) -> Path: + if perf_json_model_family(json_basename) in ("omni", "tts"): + return RUN_BENCHMARK_REL + return RUN_DIFFUSION_BENCHMARK_REL + + +def perf_json_matches_model_type(json_basename: str, model_types: list[str]) -> bool: + if "all" in model_types: + return True + return perf_json_model_family(json_basename) in model_types + + +def local_perf_json_configs(substr: str, model_types: list[str]) -> list[tuple[str, str]]: + """Return (json_rel_posix, runner_rel_posix) for perf configs whose filename contains substr.""" + perf_dir = REPO_ROOT / PERF_TESTS_REL + if not perf_dir.is_dir(): + return [] + matches: list[tuple[str, str]] = [] + for path in sorted(perf_dir.glob("*.json")): + if substr not in path.name: + continue + if not perf_json_matches_model_type(path.name, model_types): + continue + runner = perf_json_runner(path.name) + if not (REPO_ROOT / runner).is_file(): + print(f"# skip (missing runner): {runner}", file=sys.stderr) + continue + matches.append( + ( + path.relative_to(REPO_ROOT).as_posix(), + runner.as_posix(), + ) + ) + return matches + + +def _local_job_slug(label: str) -> str: + slug = re.sub(r"[^\w\-.]+", "_", label, flags=re.UNICODE) + slug = re.sub(r"_+", "_", slug).strip("_") + return slug or "filter" + + +def run_local_mode( + jobs_dir: Path, + model_types: list[str], + perf_job_keys: list[str], +) -> int: + """pytest -sv -m ' and local_model'; optional LABEL_SUBSTR filters.""" marker_expr = local_pytest_marker_expr(model_types) - key = "local_pytest" + matched = 0 + + if LABEL_SUBSTR: + rel_paths = local_test_files_by_filename(LABEL_SUBSTR) + perf_pairs = local_perf_json_configs(LABEL_SUBSTR, model_types) + + if rel_paths: + paths_arg = " ".join(shlex.quote(p) for p in rel_paths) + pytest_line = f'pytest -sv -m "{marker_expr}" {paths_arg}' + slug = _local_job_slug(LABEL_SUBSTR) + key = f"local_pytest_{slug}" + header = ( + f"# Local marker tests — MODEL_TYPE={','.join(model_types)}, " + f"LABEL_SUBSTR={LABEL_SUBSTR!r} (test_*.py), files: {', '.join(rel_paths)}" + ) + script_lines = [ + "#!/usr/bin/env bash", + header, + "set -euo pipefail", + f'cd "{REPO_ROOT}"', + pytest_line, + ] + _write_job_script(key, script_lines, jobs_dir) + matched += 1 + + for json_rel, runner_rel in perf_pairs: + json_name = Path(json_rel).name + pytest_line = ( + f'pytest -s -v -m "{marker_expr}" ' + f"{shlex.quote(runner_rel)} " + f"--test-config-file {shlex.quote(json_rel)}" + ) + slug = _local_job_slug(Path(json_rel).stem) + key = f"local_perf_{slug}" + header = ( + f"# Local perf benchmark — MODEL_TYPE={','.join(model_types)}, " + f"config={json_name!r}, runner={Path(runner_rel).name}" + ) + script_lines = [ + "#!/usr/bin/env bash", + header, + "set -euo pipefail", + f'cd "{REPO_ROOT}"', + pytest_line, + ] + _write_job_script(key, script_lines, jobs_dir) + if key not in perf_job_keys: + perf_job_keys.append(key) + matched += 1 + + if matched == 0: + print( + f"# skip local: no tests/**/test_*.py or " + f"tests/dfx/perf/tests/*.json filename matches " + f"LABEL_SUBSTR={LABEL_SUBSTR!r}", + file=sys.stderr, + ) + return matched + pytest_line = f'pytest -sv -m "{marker_expr}"' + key = "local_pytest" + header = f"# Local marker tests — MODEL_TYPE={','.join(model_types)}" script_lines = [ "#!/usr/bin/env bash", - f"# Local marker tests — MODEL_TYPE={','.join(model_types)}", + header, "set -euo pipefail", f'cd "{REPO_ROOT}"', pytest_line, @@ -566,7 +789,7 @@ def run_stability_mode(jobs_dir: Path, model_types: list[str]) -> int: if not rel_path.is_file(): print(f"# skip (missing file): {rel_path}", file=sys.stderr) continue - pytest_line = f"pytest -s -v {rel_posix}" + pytest_line = f"pytest -s -v --run-level full_model {rel_posix}" matched += 1 script_lines = [ "#!/usr/bin/env bash", @@ -594,13 +817,15 @@ def main() -> None: matched_stability = 0 matched_yaml = 0 + matched_local = 0 perf_job_keys: list[str] = [] + job_timeouts: dict[str, int] = {} if want_stability: matched_stability = run_stability_mode(jobs_dir, model_types) if want_local: - run_local_mode(jobs_dir, model_types) + matched_local = run_local_mode(jobs_dir, model_types, perf_job_keys) if needs_yaml: if not YML.is_file(): @@ -634,8 +859,12 @@ def main() -> None: "set -euo pipefail", f'cd "{REPO_ROOT}"', ] + timeout_min = step_timeout_minutes(step) + if timeout_min is not None: + script_lines.append(f"# Buildkite timeout_in_minutes: {timeout_min}") + job_timeouts[key] = timeout_min script_lines.extend(exports) - script_lines.extend(pys) + script_lines.extend(prepend_timeout_to_pytest(p, timeout_min) for p in pys) _write_job_script(key, script_lines, jobs_dir) if "Perf Test" in label and key not in perf_job_keys: perf_job_keys.append(key) @@ -646,6 +875,7 @@ def main() -> None: manifest_path.write_text("\n".join(perf_job_keys) + "\n", encoding="utf-8") elif manifest_path.is_file(): manifest_path.unlink() + _write_job_timeouts_manifest(jobs_dir, job_timeouts) errs: list[str] = [] if want_stability and matched_stability == 0: @@ -653,6 +883,17 @@ def main() -> None: f"No stability jobs matched TEST_TYPE={test_types!r} MODEL_TYPE={model_types!r} " f"LABEL_SUBSTR={LABEL_SUBSTR!r} under {REPO_ROOT}" ) + if want_local and matched_local == 0: + if LABEL_SUBSTR: + errs.append( + f"No local jobs matched LABEL_SUBSTR={LABEL_SUBSTR!r} " + f"(tests/**/test_*.py or tests/dfx/perf/tests/*.json) " + f"under {REPO_ROOT}" + ) + else: + errs.append( + f"Local mode produced no jobs TEST_TYPE={test_types!r} MODEL_TYPE={model_types!r}" + ) if needs_yaml and matched_yaml == 0: errs.append( f"No YAML steps matched TEST_TYPE={test_types!r} MODEL_TYPE={model_types!r} " @@ -671,29 +912,70 @@ if [[ "${DRY_RUN}" == "1" ]]; then exit 0 fi +# shellcheck source=tools/run_jobs_common.sh +source "${SCRIPT_DIR}/../run_jobs_common.sh" + +PERF_RESULTS_SRC="${REPO_ROOT}/tests/dfx/perf/results" + +# Copy perf benchmark JSON written under tests/dfx/perf/results/ during this run into LOG_DIR. +_collect_perf_result_jsons() { + local since_epoch="${1:?}" + local dest="${LOG_DIR}/perf_results" + + if [[ ! -d "${PERF_RESULTS_SRC}" ]]; then + return 0 + fi + + COLLECT_SINCE_EPOCH="${since_epoch}" \ + python3 - <<'PY' +import os +import shutil +import sys +from pathlib import Path + +repo = Path(os.environ["REPO_ROOT"]).resolve() +src = repo / "tests" / "dfx" / "perf" / "results" +dest = Path(os.environ["LOG_DIR"]).resolve() / "perf_results" +since = float(os.environ["COLLECT_SINCE_EPOCH"]) + +if not src.is_dir(): + sys.exit(0) + +copied = 0 +for path in sorted(src.rglob("*.json")): + if not path.is_file(): + continue + try: + if path.stat().st_mtime < since: + continue + except OSError: + continue + rel = path.relative_to(src) + out = dest / rel + out.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(path, out) + copied += 1 + +if copied: + print(f"Collected {copied} perf result JSON file(s) -> {dest}", file=sys.stderr) +PY +} + # Run jobs: YAML perf steps first (if manifest exists), then generate Excel (always when # manifest non-empty, even if some perf jobs failed; Excel uses whatever JSON exists on disk), # then remaining jobs. Paths align with nightly Buildkite (tests/dfx/perf/results). run_generated_jobs_with_tee() { set -o pipefail local any_fail=0 + local _run_start local perf_list="${LOG_DIR}/jobs/.perf_job_keys" local -a perf_keys=() + _run_jobs_reset_timing + _run_start="$(_run_jobs_epoch_seconds)" if [[ -f "${perf_list}" ]] && [[ -s "${perf_list}" ]]; then mapfile -t perf_keys < "${perf_list}" fi - _run_one_job() { - local _job="$1" - local base out job_status - base="$(basename "${_job}" .sh)" - out="${LOG_DIR}/${base}.log" - echo "==> ${_job} (tee ${out})" >&2 - (cd "${REPO_ROOT}" && bash "${_job}") 2>&1 | tee "${out}" - job_status="${PIPESTATUS[0]}" - return "${job_status}" - } - local k for k in "${perf_keys[@]}"; do [[ -n "${k}" ]] || continue @@ -702,11 +984,11 @@ run_generated_jobs_with_tee() { any_fail=1 continue fi - _run_one_job "${LOG_DIR}/jobs/${k}.sh" || any_fail=1 + _run_one_job_with_timing "${LOG_DIR}/jobs/${k}.sh" || any_fail=1 done if ((${#perf_keys[@]})); then - local excel_out excel_log excel_status excel_repo_logs + local excel_out excel_log excel_status excel_repo_logs excel_start excel_end excel_elapsed excel_repo_logs="${REPO_ROOT}/logs" mkdir -p "${excel_repo_logs}" excel_out="${excel_repo_logs}/nightly_perf_$(date -u +%Y%m%d-%H%M%S).xlsx" @@ -716,6 +998,7 @@ run_generated_jobs_with_tee() { "(report reflects JSON already under tests/dfx/perf/results)." >&2 fi echo "==> python3 tools/nightly/generate_nightly_perf_excel.py -> ${excel_out} (tee ${excel_log})" >&2 + excel_start="$(_run_jobs_epoch_seconds)" ( cd "${REPO_ROOT}" && python3 tools/nightly/generate_nightly_perf_excel.py \ --input-dir "${REPO_ROOT}/tests/dfx/perf/results" \ @@ -723,7 +1006,13 @@ run_generated_jobs_with_tee() { --output-file "${excel_out}" ) 2>&1 | tee "${excel_log}" excel_status="${PIPESTATUS[0]}" - if [[ "${excel_status}" -ne 0 ]]; then + excel_end="$(_run_jobs_epoch_seconds)" + excel_elapsed=$((excel_end - excel_start)) + _run_jobs_record_timing "generate_nightly_perf_excel" "${excel_elapsed}" "${excel_status}" + if [[ "${excel_status}" -eq 0 ]]; then + echo " finished in $(_run_jobs_format_duration "${excel_elapsed}")" >&2 + else + echo " failed after $(_run_jobs_format_duration "${excel_elapsed}") (exit ${excel_status})" >&2 any_fail=1 echo "generate_nightly_perf_excel.py failed (exit ${excel_status}). See ${excel_log}" >&2 fi @@ -739,15 +1028,16 @@ run_generated_jobs_with_tee() { for _job in "${LOG_DIR}/jobs"/*.sh; do base="$(basename "${_job}" .sh)" [[ ${_perf_done["${base}"]+isset} ]] && continue - _run_one_job "${_job}" || any_fail=1 + _run_one_job_with_timing "${_job}" || any_fail=1 done shopt -u nullglob + _collect_perf_result_jsons "${_run_start}" + + _run_jobs_print_timing_summary "${_run_start}" "${any_fail}" if [[ "${any_fail}" -ne 0 ]]; then - echo "One or more selected jobs failed. See logs under ${LOG_DIR}." >&2 return 1 fi - echo "All selected jobs finished OK. Logs: ${LOG_DIR}/*.log" >&2 return 0 } diff --git a/tools/run_jobs_common.sh b/tools/run_jobs_common.sh new file mode 100644 index 000000000..ddc1794dc --- /dev/null +++ b/tools/run_jobs_common.sh @@ -0,0 +1,653 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# +# Shared helpers for tools/run_*_jobs.sh (source this file; do not execute). +# +# Contents: +# 1) Timing / tee runners used by ready, merge, and nightly. +# 2) Ready/merge CLI + YAML extract entrypoint: run_yaml_ci_jobs_main +# (set BUILDKITE_REL and DEFAULT_LOG_SUBDIR in the entry script, then call it). +# +# Job timeouts: generated job scripts embed GNU timeout on the pytest line itself +# (see prepend_timeout_to_pytest in the Python heredoc below). This runner only +# executes bash job.sh and records exit 124 as TIMED OUT via jobs/.job_timeouts +# metadata. + +if [[ -n "${_RUN_JOBS_COMMON_LOADED:-}" ]]; then + return 0 2>/dev/null || exit 0 +fi +_RUN_JOBS_COMMON_LOADED=1 + +if [[ -z "${SCRIPT_DIR:-}" ]]; then + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +fi + +# --------------------------------------------------------------------------- +# Timing / tee helpers (ready, merge, nightly) +# --------------------------------------------------------------------------- + +RUN_JOB_NAMES=() +RUN_JOB_SECONDS=() +RUN_JOB_STATUSES=() + +_run_jobs_reset_timing() { + RUN_JOB_NAMES=() + RUN_JOB_SECONDS=() + RUN_JOB_STATUSES=() +} + +_run_jobs_epoch_seconds() { + date +%s +} + +_run_jobs_format_duration() { + local total="${1}" + local h m s + if [[ "${total}" -lt 60 ]]; then + printf '%ss' "${total}" + return 0 + fi + m=$((total / 60)) + s=$((total % 60)) + if [[ "${m}" -lt 60 ]]; then + if [[ "${s}" -eq 0 ]]; then + printf '%dm' "${m}" + else + printf '%dm %ss' "${m}" "${s}" + fi + return 0 + fi + h=$((m / 60)) + m=$((m % 60)) + if [[ "${m}" -eq 0 && "${s}" -eq 0 ]]; then + printf '%dh' "${h}" + elif [[ "${s}" -eq 0 ]]; then + printf '%dh %dm' "${h}" "${m}" + else + printf '%dh %dm %ss' "${h}" "${m}" "${s}" + fi +} + +_run_jobs_record_timing() { + local name="${1}" + local seconds="${2}" + local status="${3}" + RUN_JOB_NAMES+=("${name}") + RUN_JOB_SECONDS+=("${seconds}") + RUN_JOB_STATUSES+=("${status}") +} + +_run_jobs_lookup_timeout_minutes() { + local base="${1}" + local manifest="${LOG_DIR}/jobs/.job_timeouts" + local line key mins + [[ -f "${manifest}" ]] || return 1 + while IFS='=' read -r key mins; do + [[ -n "${key}" ]] || continue + if [[ "${key}" == "${base}" ]]; then + printf '%s' "${mins}" + return 0 + fi + done < "${manifest}" + return 1 +} + +_run_one_job_with_timing() { + local _job="$1" + local base out job_status start end elapsed timeout_min="" + base="$(basename "${_job}" .sh)" + out="${LOG_DIR}/${base}.log" + if timeout_min="$(_run_jobs_lookup_timeout_minutes "${base}")"; then + echo "==> ${_job} (tee ${out}, pytest inline timeout ${timeout_min}m)" >&2 + else + echo "==> ${_job} (tee ${out})" >&2 + fi + start="$(_run_jobs_epoch_seconds)" + (cd "${REPO_ROOT}" && bash "${_job}") 2>&1 | tee "${out}" + job_status="${PIPESTATUS[0]}" + end="$(_run_jobs_epoch_seconds)" + elapsed=$((end - start)) + _run_jobs_record_timing "${base}" "${elapsed}" "${job_status}" + if [[ "${job_status}" -eq 0 ]]; then + echo " finished in $(_run_jobs_format_duration "${elapsed}")" >&2 + elif [[ "${job_status}" -eq 124 && -n "${timeout_min}" ]]; then + echo " timed out (inline timeout ${timeout_min}m on pytest, exit 124)" >&2 + elif [[ "${job_status}" -eq 124 ]]; then + echo " timed out (exit 124)" >&2 + else + echo " failed after $(_run_jobs_format_duration "${elapsed}") (exit ${job_status})" >&2 + fi + return "${job_status}" +} + +_run_jobs_print_timing_summary() { + local wall_start="${1:-}" + local any_fail="${2:-0}" + local summary_path="${LOG_DIR}/timing_summary.log" + local -a lines=() + local i name secs status status_str failed_count=0 + local wall_elapsed job_count total_elapsed=0 + + lines+=("=== Job timing summary ===") + for i in "${!RUN_JOB_NAMES[@]}"; do + name="${RUN_JOB_NAMES[$i]}" + secs="${RUN_JOB_SECONDS[$i]}" + status="${RUN_JOB_STATUSES[$i]}" + if [[ "${status}" -eq 0 ]]; then + status_str="OK" + elif [[ "${status}" -eq 124 ]]; then + status_str="TIMED OUT" + failed_count=$((failed_count + 1)) + else + status_str="FAILED (exit ${status})" + failed_count=$((failed_count + 1)) + fi + lines+=(" ${name} $(_run_jobs_format_duration "${secs}") ${status_str}") + done + + job_count="${#RUN_JOB_NAMES[@]}" + if [[ -n "${wall_start}" ]]; then + wall_elapsed=$(( $(_run_jobs_epoch_seconds) - wall_start )) + lines+=("Total wall time: $(_run_jobs_format_duration "${wall_elapsed}") (${job_count} jobs)") + elif [[ "${job_count}" -gt 0 ]]; then + for secs in "${RUN_JOB_SECONDS[@]}"; do + total_elapsed=$((total_elapsed + secs)) + done + lines+=("Total job time: $(_run_jobs_format_duration "${total_elapsed}") (${job_count} jobs)") + else + lines+=("Total wall time: 0s (0 jobs)") + fi + + if [[ "${failed_count}" -gt 0 ]]; then + lines+=("Failed jobs: ${failed_count}/${job_count}") + fi + + if [[ "${any_fail}" -ne 0 ]]; then + lines+=("Result: one or more jobs failed. See logs under ${LOG_DIR}.") + else + lines+=("Result: all jobs finished OK. Logs: ${LOG_DIR}/*.log") + fi + + { + for line in "${lines[@]}"; do + printf '%s\n' "${line}" + done + } | tee "${summary_path}" >&2 +} + +# --------------------------------------------------------------------------- +# Ready / merge: CLI + YAML extract + run (tools/run_ready_jobs.sh, +# tools/run_merge_jobs.sh). Nightly does not use this entrypoint. +# +# Entry scripts must set before calling run_yaml_ci_jobs_main: +# BUILDKITE_REL - e.g. .buildkite/test-ready.yml +# DEFAULT_LOG_SUBDIR - e.g. ready_jobs +# --------------------------------------------------------------------------- + +_run_yaml_ci_jobs_usage() { + # Print the entry script header (caller is run_ready_jobs.sh / run_merge_jobs.sh). + sed -n '2,38p' "$0" | sed 's/^# \{0,1\}//' +} + +_run_jobs_split_append_csv_array() { + local -n _arr="${1}" + local _raw="${2}" + local _ifs="${IFS}" + local _part + IFS=',' + for _part in ${_raw}; do + IFS="${_ifs}" + _part="$(printf '%s' "${_part}" | tr '[:upper:]' '[:lower:]' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')" + [[ -n "${_part}" ]] || continue + _arr+=("${_part}") + done +} + +_run_jobs_finalize_model_type_csv() { + if [[ "${MODEL_TYPE_FROM_CLI}" -eq 1 ]]; then + if ((${#MODEL_TYPE_CLI_PARTS[@]} == 0)); then + echo "--model-type requires a non-empty value" >&2 + exit 2 + fi + local -A _seen=() + local -a _out=() + local _x + for _x in "${MODEL_TYPE_CLI_PARTS[@]}"; do + if [[ "${_x}" == all ]]; then + printf '%s' "all" + return 0 + fi + [[ ${_seen["${_x}"]+isset} ]] && continue + _seen["${_x}"]=1 + _out+=("${_x}") + done + (IFS=','; printf '%s' "${_out[*]}") + else + printf '%s' "${MODEL_TYPE_ENV}" + fi +} + +_run_jobs_find_repo_containing_yml() { + local dir="${1:-}" + [[ -n "$dir" ]] || return 1 + dir="$(cd "$dir" && pwd)" || return 1 + while true; do + if [[ -f "${dir}/${BUILDKITE_REL}" ]]; then + printf '%s\n' "$dir" + return 0 + fi + [[ "${dir}" == "/" ]] && return 1 + dir="$(dirname "${dir}")" + done +} + +_run_jobs_derive_repo_root_from_yml() { + local yml="$1" + local d + d="$(cd "$(dirname "${yml}")" && pwd)" + [[ "$(basename "${d}")" == ".buildkite" ]] || return 1 + printf '%s\n' "$(dirname "${d}")" +} + +run_yaml_ci_jobs_main() { + if [[ -z "${BUILDKITE_REL:-}" || -z "${DEFAULT_LOG_SUBDIR:-}" ]]; then + echo "run_yaml_ci_jobs_main: set BUILDKITE_REL and DEFAULT_LOG_SUBDIR before calling." >&2 + exit 1 + fi + + LABEL_SUBSTR="${LABEL_SUBSTR:-}" + MODEL_TYPE_ENV="${MODEL_TYPE:-all}" + MODEL_TYPE_CLI_PARTS=() + MODEL_TYPE_FROM_CLI=0 + DRY_RUN="${DRY_RUN:-0}" + SKIP_SIMPLE=0 + + while [[ $# -gt 0 ]]; do + case "$1" in + -h | --help) + _run_yaml_ci_jobs_usage + exit 0 + ;; + --dry-run) + DRY_RUN=1 + shift + ;; + --log-dir) + LOG_DIR="$2" + shift 2 + ;; + --yaml) + YML="$2" + shift 2 + ;; + --repo-root) + REPO_ROOT="$2" + shift 2 + ;; + --label-substr) + LABEL_SUBSTR="$2" + shift 2 + ;; + --model-type) + MODEL_TYPE_FROM_CLI=1 + _run_jobs_split_append_csv_array MODEL_TYPE_CLI_PARTS "$2" + shift 2 + ;; + --skip-simple) + SKIP_SIMPLE=1 + shift + ;; + *) + echo "Unknown option: $1" >&2 + _run_yaml_ci_jobs_usage >&2 + exit 2 + ;; + esac + done + + MODEL_TYPE="$(_run_jobs_finalize_model_type_csv)" + + if [[ -n "${YML:-}" && -n "${REPO_ROOT:-}" ]]; then + REPO_ROOT="$(cd "${REPO_ROOT}" && pwd)" + YML="$(cd "$(dirname "${YML}")" && pwd)/$(basename "${YML}")" + elif [[ -n "${YML:-}" ]]; then + YML="$(cd "$(dirname "${YML}")" && pwd)/$(basename "${YML}")" + if ! REPO_ROOT="$(_run_jobs_derive_repo_root_from_yml "${YML}")"; then + echo "Could not derive REPO_ROOT from YML=${YML} (expected file at /${BUILDKITE_REL})." >&2 + echo "Set REPO_ROOT explicitly (or pass --repo-root) for pytest working directory." >&2 + exit 2 + fi + elif [[ -n "${REPO_ROOT:-}" ]]; then + REPO_ROOT="$(cd "${REPO_ROOT}" && pwd)" + YML="${REPO_ROOT}/${BUILDKITE_REL}" + else + REPO_ROOT="" + if command -v git >/dev/null 2>&1; then + REPO_ROOT="$(git -C "${PWD}" rev-parse --show-toplevel 2>/dev/null || true)" + fi + if [[ -z "${REPO_ROOT}" ]]; then + REPO_ROOT="$(_run_jobs_find_repo_containing_yml "${PWD}" || true)" + fi + if [[ -z "${REPO_ROOT}" ]]; then + REPO_ROOT="$(_run_jobs_find_repo_containing_yml "${SCRIPT_DIR}" || true)" + fi + if [[ -z "${REPO_ROOT}" ]]; then + echo "Could not locate ${BUILDKITE_REL}. Set REPO_ROOT or YML, run from inside the vllm-omni clone," >&2 + echo "or place this script (or run from a cwd) under the repository tree." >&2 + exit 2 + fi + YML="${REPO_ROOT}/${BUILDKITE_REL}" + fi + + LOG_DIR="${LOG_DIR:-${REPO_ROOT}/logs/${DEFAULT_LOG_SUBDIR}}" + + if [[ ! -f "${YML}" ]]; then + echo "YAML not found: ${YML}" >&2 + exit 2 + fi + + if ! command -v python3 >/dev/null 2>&1; then + echo "python3 is required." >&2 + exit 1 + fi + + mkdir -p "${LOG_DIR}/jobs" + LOG_DIR="$(cd "${LOG_DIR}" && pwd)" + export REPO_ROOT LOG_DIR YML LABEL_SUBSTR MODEL_TYPE DRY_RUN SKIP_SIMPLE + + if [[ "${DRY_RUN}" != "1" ]]; then + shopt -s nullglob + _stale=( "${LOG_DIR}/jobs"/*.sh ) + _stale_logs=( "${LOG_DIR}"/*.log ) + shopt -u nullglob + if ((${#_stale[@]})); then + rm -f "${_stale[@]}" + fi + if ((${#_stale_logs[@]})); then + rm -f "${_stale_logs[@]}" + fi + rm -f "${LOG_DIR}/jobs/.job_timeouts" + fi + + # shellcheck disable=SC2016,SC1078,SC1079 + python3 - <<'PY' +from __future__ import annotations + +import os +import re +import stat +import sys +from pathlib import Path + +try: + import yaml +except ImportError: + print("Missing PyYAML. Install with: pip install pyyaml", file=sys.stderr) + sys.exit(1) + +REPO_ROOT = Path(os.environ["REPO_ROOT"]).resolve() +LOG_DIR = Path(os.environ["LOG_DIR"]).resolve() +YML = Path(os.environ["YML"]).resolve() +LABEL_SUBSTR = (os.environ.get("LABEL_SUBSTR") or "").strip() +DRY_RUN = os.environ.get("DRY_RUN", "0") == "1" +SKIP_SIMPLE = os.environ.get("SKIP_SIMPLE", "0") == "1" + +ALLOWED_MODEL_TYPES = frozenset({"all", "omni", "tts", "diffusion"}) + + +def parse_model_types(raw: str) -> list[str]: + parts = [p.strip().lower() for p in (raw or "").split(",") if p.strip()] + if not parts: + parts = ["all"] + bad = [p for p in parts if p not in ALLOWED_MODEL_TYPES] + if bad: + print( + f"Invalid MODEL_TYPE / --model-type value(s): {bad!r} " + f"(allowed: {', '.join(sorted(ALLOWED_MODEL_TYPES))})", + file=sys.stderr, + ) + sys.exit(2) + if "all" in parts: + return ["all"] + out: list[str] = [] + seen: set[str] = set() + for p in parts: + if p not in seen: + seen.add(p) + out.append(p) + return out + + +PYTEST_CMD_RE = re.compile( + r"(?:timeout\s+\S+\s+)?(?:python3? -m\s+)?pytest\s+[^\n&|;]*" +) + + +def iter_leaf_steps(steps, group: str | None = None): + for raw in steps or []: + if not isinstance(raw, dict): + continue + nested = raw.get("steps") + if isinstance(nested, list) and nested: + g = raw.get("group") + next_group: str | None + if isinstance(g, str): + next_group = g + elif g is not None: + next_group = str(g) + else: + next_group = group + yield from iter_leaf_steps(nested, next_group) + continue + if raw.get("commands"): + yield raw, group + + +def raw_command_text(step: dict) -> str: + raw = step.get("commands") or [] + if isinstance(raw, str): + raw = [raw] + text = "\n".join((c.strip() if isinstance(c, str) else "") for c in raw if c) + return text.replace("$$", "$") + + +def export_lines(text: str) -> list[str]: + out: list[str] = [] + for line in text.splitlines(): + s = line.strip() + if not s or s.startswith("#"): + continue + if s.startswith("export "): + out.append(s) + return out + + +def pytest_lines(text: str) -> list[str]: + out: list[str] = [] + for m in PYTEST_CMD_RE.finditer(text): + line = m.group(0).strip() + line_start = text.rfind("\n", 0, m.start()) + 1 + before = text[line_start : m.start()] + if before.lstrip().startswith("#"): + continue + if line: + out.append(line) + return out + + +def is_simple_test_step(label: str, group: str | None) -> bool: + if group and "Simple Test" in group: + return True + return label.startswith("Simple ·") + + +def label_matches_model_type(label: str, model_types: list[str]) -> bool: + if "all" in model_types: + return True + if "omni" in model_types and "Omni ·" in label: + return True + if "tts" in model_types and "TTS ·" in label: + return True + if "diffusion" in model_types and "Diffusion" in label: + return True + return False + + +def job_key_from_step(step: dict, label: str) -> str: + k = step.get("key") + if isinstance(k, str) and k.strip(): + return k.strip() + slug = re.sub(r"[^\w\-.]+", "_", label or "job", flags=re.UNICODE) + slug = re.sub(r"_+", "_", slug).strip("_") or "job" + return slug + + +def step_timeout_minutes(step: dict) -> int | None: + raw = step.get("timeout_in_minutes") + if raw is None: + return None + try: + minutes = int(raw) + except (TypeError, ValueError): + return None + return minutes if minutes > 0 else None + + +_INLINE_TIMEOUT_PREFIX = re.compile(r"^timeout\s+(?:-\S+\s+)*\S+\s+", re.IGNORECASE) + + +def timeout_kill_after_seconds() -> int: + raw = os.environ.get("RUN_JOB_TIMEOUT_KILL_AFTER", "60") + try: + seconds = int(raw) + except (TypeError, ValueError): + return 60 + return seconds if seconds > 0 else 60 + + +def prepend_timeout_to_pytest(pytest_line: str, timeout_min: int | None) -> str: + """Prepend GNU timeout directly before pytest (Buildkite inline style).""" + if timeout_min is None: + return pytest_line.strip() + line = _INLINE_TIMEOUT_PREFIX.sub("", pytest_line.strip()) + kill_after = timeout_kill_after_seconds() + return ( + f"timeout --foreground --verbose --kill-after={kill_after} " + f"{timeout_min}m {line}" + ) + + +def _write_job_timeouts_manifest(jobs_dir: Path, job_timeouts: dict[str, int]) -> None: + manifest_path = jobs_dir / ".job_timeouts" + if job_timeouts: + manifest_path.write_text( + "\n".join(f"{k}={v}" for k, v in sorted(job_timeouts.items())) + "\n", + encoding="utf-8", + ) + elif manifest_path.is_file(): + manifest_path.unlink() + + +def _write_job_script(key: str, script_lines: list[str], jobs_dir: Path) -> None: + body = "\n".join(script_lines) + "\n" + if DRY_RUN: + print(f"=== {key} ===") + print(body) + return + job_path = jobs_dir / f"{key}.sh" + job_path.write_text(body, encoding="utf-8") + try: + mode = job_path.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH + job_path.chmod(mode) + except OSError: + pass + print(f"generated {job_path}", file=sys.stderr) + + +def main() -> None: + model_types = parse_model_types(os.environ.get("MODEL_TYPE", "all")) + + jobs_dir = LOG_DIR / "jobs" + if not DRY_RUN: + jobs_dir.mkdir(parents=True, exist_ok=True) + + if not YML.is_file(): + print(f"YAML not found: {YML}", file=sys.stderr) + sys.exit(1) + + data = yaml.safe_load(YML.read_text(encoding="utf-8")) + top_steps = (data or {}).get("steps") or [] + + matched_yaml = 0 + job_timeouts: dict[str, int] = {} + for step, grp in iter_leaf_steps(top_steps): + label = step.get("label") or "" + if SKIP_SIMPLE and is_simple_test_step(label, grp): + continue + if LABEL_SUBSTR and LABEL_SUBSTR not in label: + continue + if not label_matches_model_type(label, model_types): + continue + text = raw_command_text(step) + exports = export_lines(text) + pys = pytest_lines(text) + if not pys: + print(f"# skip (no pytest line): {label!r}", file=sys.stderr) + continue + + key = job_key_from_step(step, label) + matched_yaml += 1 + + script_lines = [ + "#!/usr/bin/env bash", + f'# From Buildkite label: {label.replace(chr(10), " ")}', + "set -euo pipefail", + f'cd "{REPO_ROOT}"', + ] + timeout_min = step_timeout_minutes(step) + if timeout_min is not None: + script_lines.append(f"# Buildkite timeout_in_minutes: {timeout_min}") + job_timeouts[key] = timeout_min + script_lines.extend(exports) + script_lines.extend(prepend_timeout_to_pytest(p, timeout_min) for p in pys) + _write_job_script(key, script_lines, jobs_dir) + + if not DRY_RUN: + _write_job_timeouts_manifest(jobs_dir, job_timeouts) + + if matched_yaml == 0: + print( + f"No YAML steps matched MODEL_TYPE={model_types!r} " + f"LABEL_SUBSTR={LABEL_SUBSTR!r} SKIP_SIMPLE={SKIP_SIMPLE!r} in {YML}", + file=sys.stderr, + ) + sys.exit(2) + + +main() +PY + + if [[ "${DRY_RUN}" == "1" ]]; then + exit 0 + fi + + set -o pipefail + local any_fail=0 + local _run_start + local _job + _run_jobs_reset_timing + _run_start="$(_run_jobs_epoch_seconds)" + + shopt -s nullglob + for _job in "${LOG_DIR}/jobs"/*.sh; do + _run_one_job_with_timing "${_job}" || any_fail=1 + done + shopt -u nullglob + + _run_jobs_print_timing_summary "${_run_start}" "${any_fail}" + if [[ "${any_fail}" -ne 0 ]]; then + exit 1 + fi + exit 0 +} diff --git a/tools/run_merge_jobs.sh b/tools/run_merge_jobs.sh new file mode 100644 index 000000000..35b38a8e0 --- /dev/null +++ b/tools/run_merge_jobs.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# +# Extract steps from .buildkite/test-merge.yml that contain pytest, synthesize +# small bash wrappers (exports + pytest), run them, and tee output to logs named +# after each step's Buildkite "key" when present (otherwise a slug of the label). +# +# Model area (--model-type / MODEL_TYPE), multiple allowed (OR semantics): +# e.g. --model-type omni,tts +# omni — label contains "Omni ·" +# tts — label contains "TTS ·" +# diffusion— label contains "Diffusion" +# all — no model filter (default) +# +# --skip-simple — skip steps in the "Simple Test" Buildkite group and labels +# starting with "Simple ·" (L1-style unit tests in the same YAML) +# +# Requirements: bash, python3, PyYAML (pip install pyyaml) +# +# Usage: +# bash tools/run_merge_jobs.sh +# REPO_ROOT=/path/to/vllm-omni bash tools/run_merge_jobs.sh --model-type diffusion --dry-run +# YML=/path/to/vllm-omni/.buildkite/test-merge.yml bash tools/run_merge_jobs.sh +# +# Repository / YAML (no dependency on where this script lives): +# • Set REPO_ROOT (or pass --repo-root) — default YAML is $REPO_ROOT/.buildkite/test-merge.yml +# • Or set YML (or --yaml) — repo root is inferred as parent of the .buildkite directory +# • Or run from inside the clone: git rev-parse --show-toplevel, else walk up from $PWD, +# then from the script's directory, until .buildkite/test-merge.yml exists +# +# Optional environment: +# REPO_ROOT - vllm-omni root (working directory for pytest); see above +# YML - path to test-merge.yml (default: $REPO_ROOT/.buildkite/test-merge.yml) +# LOG_DIR - logs + generated job scripts (default: $REPO_ROOT/logs/merge_jobs); +# per-job *.log plus timing_summary.log after the run +# MODEL_TYPE - comma-separated and/or repeated flags (default: all); see above +# LABEL_SUBSTR - substring of Buildkite step label +# DRY_RUN=1 - print extracted commands only; do not write scripts or run pytest +# +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +BUILDKITE_REL=".buildkite/test-merge.yml" +DEFAULT_LOG_SUBDIR="merge_jobs" + +# shellcheck source=tools/run_jobs_common.sh +source "${SCRIPT_DIR}/run_jobs_common.sh" +run_yaml_ci_jobs_main "$@" diff --git a/tools/run_ready_jobs.sh b/tools/run_ready_jobs.sh new file mode 100644 index 000000000..89f082436 --- /dev/null +++ b/tools/run_ready_jobs.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# +# Extract steps from .buildkite/test-ready.yml that contain pytest, synthesize +# small bash wrappers (exports + pytest), run them, and tee output to logs named +# after each step's Buildkite "key" when present (otherwise a slug of the label). +# +# Model area (--model-type / MODEL_TYPE), multiple allowed (OR semantics): +# e.g. --model-type omni,tts +# omni — label contains "Omni ·" +# tts — label contains "TTS ·" +# diffusion— label contains "Diffusion" +# all — no model filter (default) +# +# --skip-simple — skip steps in the "Simple Test" Buildkite group and labels +# starting with "Simple ·" (L1-style unit tests in the same YAML) +# +# Requirements: bash, python3, PyYAML (pip install pyyaml) +# +# Usage: +# bash tools/run_ready_jobs.sh +# REPO_ROOT=/path/to/vllm-omni bash tools/run_ready_jobs.sh --model-type omni --dry-run +# YML=/path/to/vllm-omni/.buildkite/test-ready.yml bash tools/run_ready_jobs.sh +# +# Repository / YAML (no dependency on where this script lives): +# • Set REPO_ROOT (or pass --repo-root) — default YAML is $REPO_ROOT/.buildkite/test-ready.yml +# • Or set YML (or --yaml) — repo root is inferred as parent of the .buildkite directory +# • Or run from inside the clone: git rev-parse --show-toplevel, else walk up from $PWD, +# then from the script's directory, until .buildkite/test-ready.yml exists +# +# Optional environment: +# REPO_ROOT - vllm-omni root (working directory for pytest); see above +# YML - path to test-ready.yml (default: $REPO_ROOT/.buildkite/test-ready.yml) +# LOG_DIR - logs + generated job scripts (default: $REPO_ROOT/logs/ready_jobs); +# per-job *.log plus timing_summary.log after the run +# MODEL_TYPE - comma-separated and/or repeated flags (default: all); see above +# LABEL_SUBSTR - substring of Buildkite step label +# DRY_RUN=1 - print extracted commands only; do not write scripts or run pytest +# +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +BUILDKITE_REL=".buildkite/test-ready.yml" +DEFAULT_LOG_SUBDIR="ready_jobs" + +# shellcheck source=tools/run_jobs_common.sh +source "${SCRIPT_DIR}/run_jobs_common.sh" +run_yaml_ci_jobs_main "$@"