[CI] Add local job runner scripts for L2-L4 pytest execution (#4672)

Signed-off-by: wangyu <410167048@qq.com>

docs/contributing/ci/test_guide.md

@@ -12,12 +12,37 @@ When running test cases, you may need to install the following dependencies:
 
 ```bash
 uv pip install ".[dev]"
-apt-get install -y ffmpeg
+apt-get install -y espeak-ng jq
 ```
 
 ## Running Tests
 Our test scripts use the pytest framework. First, please use `git clone https://github.com/vllm-project/vllm-omni.git` to download the vllm-omni source code. Then, in the root directory of vllm-omni, you can run the following commands in your local test environment to execute the corresponding test cases.
 
+### CI job runners (L2–L4): logs, timing, and timeouts
+
+[`tools/run_ready_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_ready_jobs.sh), [`tools/run_merge_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_merge_jobs.sh), and [`tools/nightly/run_nightly_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/nightly/run_nightly_jobs.sh) share the same run-time behavior (via [`tools/run_jobs_common.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_jobs_common.sh)):
+
+**Log layout** (default under `logs/ready_jobs`, `logs/merge_jobs`, or `logs/nightly_jobs`; override with `--log-dir`):
+
+| Path | Description |
+|------|-------------|
+| `jobs/*.sh` | Generated per-step bash wrappers |
+| `jobs/.job_timeouts` | Job key → `timeout_in_minutes` from Buildkite YAML (when set) |
+| `<job_key>.log` | Tee output for each job |
+| `timing_summary.log` | Per-job duration and total wall time after the run completes |
+
+**Timing:** while each job runs, stderr prints `finished in …` or `failed after …`. When all jobs finish, a summary is printed and written to `timing_summary.log`, for example:
+
+```
+=== Job timing summary ===
+  TTS_VoxCPM2_Test  25m 40s  OK
+  Omni_Qwen3-Omni_Test  8m 05s  TIMED OUT
+Total wall time: 33m 45s (2 jobs)
+Failed jobs: 1/2
+```
+
+**Timeouts:** if a Buildkite step defines `timeout_in_minutes`, the runner wraps that job with `timeout ${N}m` (aligned with CI). Steps without `timeout_in_minutes` rely on any inline `timeout …` already present in the extracted pytest command. On timeout the process exits with code `124` and the summary marks the job as `TIMED OUT`.
+
 === "L1 level"
 
     ```bash
@@ -28,61 +53,109 @@ Our test scripts use the pytest framework. First, please use `git clone https://
 
 === "L2 level"
 
+    **Recommended:** run CI-aligned jobs from the repo root via [`tools/run_ready_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_ready_jobs.sh). The script reads [`.buildkite/test-ready.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-ready.yml), generates per-step bash wrappers, runs pytest, and tees logs under `logs/ready_jobs/` (see [CI job runners](#ci-job-runners-l2l4-logs-timing-and-timeouts) for timing and timeout behavior). Requires `bash`, `python3`, and PyYAML (`pip install pyyaml`).
+
+    ```bash
+    # All L2 and L1 jobs (default: every pytest step in test-ready.yml)
+    bash tools/run_ready_jobs.sh
+
+    # Skip L1-style "Simple Test" steps in test-ready.yml (group or "Simple ·" labels)
+    bash tools/run_ready_jobs.sh --skip-simple
+    bash tools/run_ready_jobs.sh --skip-simple --model-type diffusion
+
+    # Preview extracted commands without running
+    bash tools/run_ready_jobs.sh --dry-run
+
+    # Filter by model area (OR semantics): omni | tts | diffusion | all
+    bash tools/run_ready_jobs.sh --model-type omni
+    bash tools/run_ready_jobs.sh --model-type tts,diffusion
+
+    # Match a Buildkite label substring (e.g. one model suite)
+    bash tools/run_ready_jobs.sh --model-type tts --label-substr "VoxCPM2"
+
+    ```
+
+    **Ad hoc:** run a single test file or marker expression manually:
+
     ```bash
     cd tests
-    pytest -s -v -m "core_model and not cpu" --run-level=core_model
-    ```
-    If you only want to run a specific test case, you can use:
-    ```bash
     pytest -s -v test_xxxx.py --run-level=core_model
+    pytest -s -v -m "core_model and distributed_cuda and L4" --run-level=core_model
     ```
-    If you only want to run specific test cases on a particular platform, you can use:
-    ```bash
-    pytest -s -v -m "core_model and distributed_cuda and L4"  --run-level=core_model
-    ```
-    The latest test commands for various test suites can be found in the [pipeline](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-ready.yml).
 
 === "L3 level"
 
+    **Recommended:** run CI-aligned jobs from the repo root via [`tools/run_merge_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/run_merge_jobs.sh). The script reads [`.buildkite/test-merge.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-merge.yml), generates per-step bash wrappers, runs pytest, and tees logs under `logs/merge_jobs/` (see [CI job runners](#ci-job-runners-l2l4-logs-timing-and-timeouts)). Requires `bash`, `python3`, and PyYAML (`pip install pyyaml`).
+
     ```bash
-    pytest -s -v -m "advanced_model" --run-level=advanced_model
+    # All L3 and L1 jobs (default: every pytest step in test-merge.yml)
+    bash tools/run_merge_jobs.sh
+
+    # Skip L1-style "Simple Test" steps in test-merge.yml (group or "Simple ·" labels)
+    bash tools/run_merge_jobs.sh --skip-simple
+    bash tools/run_merge_jobs.sh --skip-simple --model-type omni
+
+    # Preview extracted commands without running
+    bash tools/run_merge_jobs.sh --dry-run
+
+    # Filter by model area (OR semantics): omni | tts | diffusion | all
+    bash tools/run_merge_jobs.sh --model-type diffusion
+    bash tools/run_merge_jobs.sh --model-type omni,tts
+
+    # Match a Buildkite label substring
+    bash tools/run_merge_jobs.sh --model-type diffusion --label-substr "Wan22"
+
     ```
-    If you only want to run a specific test case, you can use:
+
+    **Ad hoc:** run a single test file or marker expression manually:
+
     ```bash
     pytest -s -v test_xxxx.py --run-level=advanced_model
+    pytest -s -v -m "advanced_model and distributed_cuda and L4" --run-level=advanced_model
     ```
-    If you only want to run specific test cases on a particular platform, you can use:
-    ```bash
-    pytest -s -v -m "advanced_model and distributed_cuda and L4"  --run-level=advanced_model
-    ```
-    The latest L3 test commands for various test suites can be found in the [pipeline](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-merge.yml).
 
 
 === "L4 level"
 
+    **Recommended:** run CI-aligned nightly jobs from the repo root via [`tools/nightly/run_nightly_jobs.sh`](https://github.com/vllm-project/vllm-omni/blob/main/tools/nightly/run_nightly_jobs.sh). The script reads [`.buildkite/test-nightly.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly.yml), generates per-step bash wrappers, runs pytest (perf jobs first, then aggregates perf JSON into Excel), and tees logs under `logs/nightly_jobs/` (see [CI job runners](#ci-job-runners-l2l4-logs-timing-and-timeouts); `generate_nightly_perf_excel` is included in the timing summary). Requires `bash`, `python3`, and PyYAML (`pip install pyyaml`).
+
     ```bash
-    cd tests
-    pytest -s -v -m "full_model" --run-level=full_model
+    # All nightly jobs (default: test-type all, model-type all)
+    bash tools/nightly/run_nightly_jobs.sh
+
+    # Preview extracted commands without running
+    bash tools/nightly/run_nightly_jobs.sh --dry-run
+
+    # Test kind (--test-type, OR semantics): perf | acc | function | stability | local | all
+    bash tools/nightly/run_nightly_jobs.sh --test-type function
+    bash tools/nightly/run_nightly_jobs.sh --test-type perf,acc
+
+    # Model area (--model-type, OR semantics): omni | tts | diffusion | all
+    bash tools/nightly/run_nightly_jobs.sh --test-type function --model-type omni
+    bash tools/nightly/run_nightly_jobs.sh --test-type perf --model-type diffusion
+
+    # Stability scripts (tests/dfx/stability/scripts/) and local marker tests
+    bash tools/nightly/run_nightly_jobs.sh --test-type stability --model-type omni
+    bash tools/nightly/run_nightly_jobs.sh --test-type local --model-type tts
     ```
-    If you only want to run a specific test case, you can use:
-    ```bash
-    pytest -s -v test_xxxx.py --run-level=full_model
-    ```
-    If you only want to run specific test cases on a particular platform, you can use:
-    ```bash
-    pytest -s -v -m "full_model and distributed_cuda and L4"  --run-level=full_model
-    ```
-    Note: To run performance tests (defaults to ``test_qwen_omni.json``; use ``--test-config-file tests/dfx/perf/tests/test_tts.json`` for TTS):
-    ```bash
-    pytest -s -v tests/dfx/perf/scripts/run_benchmark.py
-    ```
-    The latest L4 (nightly) test commands use the `full_model` marker and `--run-level full_model` (see [test-nightly.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly.yml) and [test-nightly-diffusion.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly-diffusion.yml)). Example:
+
+    Perf steps (label contains ``Perf Test``) run before other jobs; Excel output is written under ``logs/``. Diffusion nightly steps may also appear in [test-nightly-diffusion.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly-diffusion.yml) on CI.
+
+    **Ad hoc:** run a single test file or marker expression manually:
 
     ```bash
     cd tests
+    pytest -s -v test_xxxx.py --run-level=full_model
+    pytest -s -v -m "full_model and distributed_cuda and L4" --run-level=full_model
     pytest -s -v -m "full_model and (omni or tts) and H100" --run-level=full_model
     ```
 
+    To run a standalone perf benchmark (defaults to ``test_qwen_omni.json``; use ``--test-config-file tests/dfx/perf/tests/test_tts.json`` for TTS):
+
+    ```bash
+    pytest -s -v tests/dfx/perf/scripts/run_benchmark.py
+    ```
+
 === "L5 level"
 
     L5 includes stability and reliability testing. Typical commands:

tests/dfx/perf/scripts/run_diffusion_benchmark.py

@@ -36,7 +36,7 @@ import pytest
 
 from benchmarks.diffusion.backends import endpoint_filename_token, normalize_endpoint
 
-pytestmark = [pytest.mark.diffusion, pytest.mark.full_model]
+pytestmark = [pytest.mark.diffusion, pytest.mark.full_model, pytest.mark.local_model]
 
 os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
 os.environ.setdefault("DIFFUSION_ATTENTION_BACKEND", "FLASH_ATTN")

tools/nightly/run_nightly_jobs.sh

@@ -9,7 +9,7 @@
 # python tools/nightly/generate_nightly_perf_excel.py runs (even if some perf jobs failed;
 # the script only aggregates JSON already present under tests/dfx/perf/results).
 # Excel and generate_nightly_perf_excel.log are written under $REPO_ROOT/logs/;
-# per-job tee logs use LOG_DIR (see --log-dir). Then remaining jobs run.
+# per-job tee logs use LOG_DIR (see --log-dir; default is timestamped under $REPO_ROOT/logs/).
 #
 # Test kind (--test-type / TEST_TYPE), multiple allowed (OR semantics):
 #   Repeat the flag and/or use comma-separated values, e.g.
@@ -21,6 +21,8 @@
 #   function — label has neither "Perf Test" nor "Accuracy Test" (incl. Doc, Multi-Replica, etc.)
 #   stability— fixed dfx stability scripts under tests/dfx/stability/scripts/ (see below)
 #   local    — pytest -sv -m "<MODEL_TYPE markers> and local_model" from repo root (no YAML step extract)
+#              When LABEL_SUBSTR is set, also runs matching tests/**/test_*.py and perf JSON configs under
+#              tests/dfx/perf/tests/*.json via run_benchmark.py / run_diffusion_benchmark.py.
 #   all      — no test-kind filter for YAML steps (any leaf step with pytest in test-nightly.yml)
 #
 # Model area (--model-type / MODEL_TYPE), multiple allowed (OR semantics):
@@ -36,10 +38,12 @@
 #   local (when included in TEST_TYPE):
 #     From repo root: pytest -sv -m "<markers> and local_model" (markers from MODEL_TYPE: omni, tts,
 #     diffusion; all → "(omni or tts or diffusion) and local_model"). Not filtered by nightly YAML.
+#     LABEL_SUBSTR: if set, restrict to tests/**/test_*.py basenames and tests/dfx/perf/tests/*.json
+#                   whose filename contains the substring (benchmark runner chosen by JSON family).
 #
 #   stability (when included in TEST_TYPE):
-#     From repo root: pytest -s -v tests/dfx/stability/scripts/test_stability_*.py
-#     model_type: omni → qwen3_omni; tts → qwen3_tts; diffusion → qwen_image + wan22; all → all four
+#     From repo root: pytest -s -v --run-level full_model tests/dfx/stability/scripts/test_stability_*.py
+#     model_type: omni → qwen3_omni; tts → qwen3_tts + voxcpm2; diffusion → qwen_image + wan22 + hunyuan_image; all → all six
 #     LABEL_SUBSTR: if set, script path / job key / filename must contain it
 #
 # Requirements: bash, python3, PyYAML (pip install pyyaml)
@@ -58,11 +62,20 @@
 # Optional environment:
 #   REPO_ROOT     - vllm-omni root (working directory for pytest); see above
 #   YML           - path to test-nightly.yml (default: $REPO_ROOT/.buildkite/test-nightly.yml)
-#   LOG_DIR       - logs + generated job scripts (default: $REPO_ROOT/logs/nightly_jobs)
+#   LOG_DIR       - logs + generated job scripts; when unset, a timestamped directory under
+#                   $REPO_ROOT/logs/ is created:
+#                     nightly_jobs_YYYYMMDD-HHMMSS       (default / YAML nightly steps)
+#                     nightly_local_jobs_YYYYMMDD-HHMMSS (--test-type local only)
+#                     nightly_stability_jobs_YYYYMMDD-HHMMSS (--test-type stability only)
+#                   per-job *.log plus timing_summary.log after the run;
+#                   perf JSON from tests/dfx/perf/results/ (produced this run) -> $LOG_DIR/perf_results/
 #   TEST_TYPE     - comma-separated and/or repeated flags (default: all); see above
 #   MODEL_TYPE    - comma-separated and/or repeated flags (default: all); see above
-#   LABEL_SUBSTR  - YAML mode: substring of Buildkite step label; stability: substring of path/key/filename
+#   LABEL_SUBSTR  - YAML mode: substring of Buildkite step label; stability: substring of path/key/filename;
+#                   local: substring of test_*.py basename or tests/dfx/perf/tests/*.json filename
 #   DRY_RUN=1     - print extracted commands only; do not write scripts or run pytest
+#   RUN_JOB_TIMEOUT_KILL_AFTER - seconds after SIGTERM before SIGKILL on inline pytest
+#                              timeout (default: 60; baked into generated job scripts)
 #
 set -euo pipefail
 
@@ -212,6 +225,28 @@ done
 TEST_TYPE="$(_finalize_test_type_csv)"
 MODEL_TYPE="$(_finalize_model_type_csv)"
 
+# Default log directory basename: date + local timestamp; prefix depends on TEST_TYPE.
+_nightly_log_dir_basename() {
+  local ts has_local=0 has_stability=0 has_yaml=0 _t
+  ts="$(date +%Y%m%d-%H%M%S)"
+  IFS=',' read -ra _TTOK <<< "${TEST_TYPE}"
+  for _t in "${_TTOK[@]}"; do
+    _t="$(printf '%s' "${_t}" | tr '[:upper:]' '[:lower:]' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')"
+    case "${_t}" in
+      local) has_local=1 ;;
+      stability) has_stability=1 ;;
+      perf | acc | function | all) has_yaml=1 ;;
+    esac
+  done
+  if [[ "${has_local}" -eq 1 && "${has_yaml}" -eq 0 && "${has_stability}" -eq 0 ]]; then
+    printf 'nightly_local_jobs_%s' "${ts}"
+  elif [[ "${has_stability}" -eq 1 && "${has_yaml}" -eq 0 && "${has_local}" -eq 0 ]]; then
+    printf 'nightly_stability_jobs_%s' "${ts}"
+  else
+    printf 'nightly_jobs_%s' "${ts}"
+  fi
+}
+
 BUILDKITE_REL=".buildkite/test-nightly.yml"
 
 _find_repo_containing_nightly() {
@@ -269,7 +304,11 @@ else
   YML="${REPO_ROOT}/${BUILDKITE_REL}"
 fi
 
-LOG_DIR="${LOG_DIR:-${REPO_ROOT}/logs/nightly_jobs}"
+if [[ -z "${LOG_DIR:-}" ]]; then
+  LOG_DIR="${REPO_ROOT}/logs/$(_nightly_log_dir_basename)"
+fi
+
+echo "Log directory: ${LOG_DIR}" >&2
 
 # Require nightly YAML only when at least one non-stability test kind needs it (validated in Python too).
 _needs_nightly_yml=0
@@ -310,6 +349,7 @@ if [[ "${DRY_RUN}" != "1" ]]; then
     rm -f "${_stale_logs[@]}"
   fi
   rm -f "${LOG_DIR}/jobs/.perf_job_keys"
+  rm -f "${LOG_DIR}/jobs/.job_timeouts"
   rm -f "${REPO_ROOT}/logs/nightly_perf_manual.xlsx"
   rm -f "${LOG_DIR}/nightly_perf_manual.xlsx"
 fi
@@ -320,6 +360,7 @@ from __future__ import annotations
 
 import os
 import re
+import shlex
 import stat
 import sys
 from pathlib import Path
@@ -491,12 +532,60 @@ def job_key_from_step(step: dict, label: str) -> str:
     return slug
 
 
+def step_timeout_minutes(step: dict) -> int | None:
+    raw = step.get("timeout_in_minutes")
+    if raw is None:
+        return None
+    try:
+        minutes = int(raw)
+    except (TypeError, ValueError):
+        return None
+    return minutes if minutes > 0 else None
+
+
+_INLINE_TIMEOUT_PREFIX = re.compile(r"^timeout\s+(?:-\S+\s+)*\S+\s+", re.IGNORECASE)
+
+
+def timeout_kill_after_seconds() -> int:
+    raw = os.environ.get("RUN_JOB_TIMEOUT_KILL_AFTER", "60")
+    try:
+        seconds = int(raw)
+    except (TypeError, ValueError):
+        return 60
+    return seconds if seconds > 0 else 60
+
+
+def prepend_timeout_to_pytest(pytest_line: str, timeout_min: int | None) -> str:
+    """Prepend GNU timeout directly before pytest (Buildkite inline style)."""
+    if timeout_min is None:
+        return pytest_line.strip()
+    line = _INLINE_TIMEOUT_PREFIX.sub("", pytest_line.strip())
+    kill_after = timeout_kill_after_seconds()
+    return (
+        f"timeout --foreground --verbose --kill-after={kill_after} "
+        f"{timeout_min}m {line}"
+    )
+
+
+def _write_job_timeouts_manifest(jobs_dir: Path, job_timeouts: dict[str, int]) -> None:
+    manifest_path = jobs_dir / ".job_timeouts"
+    if job_timeouts:
+        manifest_path.write_text(
+            "\n".join(f"{k}={v}" for k, v in sorted(job_timeouts.items())) + "\n",
+            encoding="utf-8",
+        )
+    elif manifest_path.is_file():
+        manifest_path.unlink()
+
+
 # Fixed stability scripts (when stability is selected); MODEL_TYPES narrows which run.
 STABILITY_CASES: list[tuple[str, str, tuple[str, ...]]] = [
     ("stability_qwen3_omni", "tests/dfx/stability/scripts/test_stability_qwen3_omni.py", ("omni",)),
     ("stability_qwen3_tts", "tests/dfx/stability/scripts/test_stability_qwen3_tts.py", ("tts",)),
+    ("stability_voxcpm2", "tests/dfx/stability/scripts/test_stability_voxcpm2.py", ("tts",)),
     ("stability_qwen_image", "tests/dfx/stability/scripts/test_stability_qwen_image.py", ("diffusion",)),
     ("stability_wan22", "tests/dfx/stability/scripts/test_stability_wan22.py", ("diffusion",)),
+    ("stability_hunyuan_image", "tests/dfx/stability/scripts/test_stability_hunyuan_image.py", ("diffusion",)),
 ]
 
 
@@ -533,14 +622,148 @@ def local_pytest_marker_expr(model_types: list[str]) -> str:
     return f"({' or '.join(families)}) and local_model"
 
 
-def run_local_mode(jobs_dir: Path, model_types: list[str]) -> int:
-    """Single job: pytest -sv -m '<model markers> and local_model' (testpaths from pytest config)."""
+def local_test_files_by_filename(substr: str) -> list[str]:
+    """Return repo-relative posix paths under tests/ whose basename contains substr."""
+    tests_root = REPO_ROOT / "tests"
+    if not tests_root.is_dir():
+        return []
+    matches: list[str] = []
+    for path in sorted(tests_root.rglob("test_*.py")):
+        if substr in path.name:
+            matches.append(path.relative_to(REPO_ROOT).as_posix())
+    return matches
+
+
+PERF_TESTS_REL = Path("tests/dfx/perf/tests")
+RUN_BENCHMARK_REL = Path("tests/dfx/perf/scripts/run_benchmark.py")
+RUN_DIFFUSION_BENCHMARK_REL = Path("tests/dfx/perf/scripts/run_diffusion_benchmark.py")
+TTS_PERF_JSON_HINTS = ("test_tts", "voxcpm", "higgs_audio")
+
+
+def perf_json_model_family(json_basename: str) -> str:
+    """Classify a perf JSON config as omni, tts, or diffusion (mirrors nightly YAML runners)."""
+    name = json_basename.lower()
+    if name.startswith("test_qwen3_omni"):
+        return "omni"
+    if any(hint in name for hint in TTS_PERF_JSON_HINTS):
+        return "tts"
+    return "diffusion"
+
+
+def perf_json_runner(json_basename: str) -> Path:
+    if perf_json_model_family(json_basename) in ("omni", "tts"):
+        return RUN_BENCHMARK_REL
+    return RUN_DIFFUSION_BENCHMARK_REL
+
+
+def perf_json_matches_model_type(json_basename: str, model_types: list[str]) -> bool:
+    if "all" in model_types:
+        return True
+    return perf_json_model_family(json_basename) in model_types
+
+
+def local_perf_json_configs(substr: str, model_types: list[str]) -> list[tuple[str, str]]:
+    """Return (json_rel_posix, runner_rel_posix) for perf configs whose filename contains substr."""
+    perf_dir = REPO_ROOT / PERF_TESTS_REL
+    if not perf_dir.is_dir():
+        return []
+    matches: list[tuple[str, str]] = []
+    for path in sorted(perf_dir.glob("*.json")):
+        if substr not in path.name:
+            continue
+        if not perf_json_matches_model_type(path.name, model_types):
+            continue
+        runner = perf_json_runner(path.name)
+        if not (REPO_ROOT / runner).is_file():
+            print(f"# skip (missing runner): {runner}", file=sys.stderr)
+            continue
+        matches.append(
+            (
+                path.relative_to(REPO_ROOT).as_posix(),
+                runner.as_posix(),
+            )
+        )
+    return matches
+
+
+def _local_job_slug(label: str) -> str:
+    slug = re.sub(r"[^\w\-.]+", "_", label, flags=re.UNICODE)
+    slug = re.sub(r"_+", "_", slug).strip("_")
+    return slug or "filter"
+
+
+def run_local_mode(
+    jobs_dir: Path,
+    model_types: list[str],
+    perf_job_keys: list[str],
+) -> int:
+    """pytest -sv -m '<model markers> and local_model'; optional LABEL_SUBSTR filters."""
     marker_expr = local_pytest_marker_expr(model_types)
-    key = "local_pytest"
+    matched = 0
+
+    if LABEL_SUBSTR:
+        rel_paths = local_test_files_by_filename(LABEL_SUBSTR)
+        perf_pairs = local_perf_json_configs(LABEL_SUBSTR, model_types)
+
+        if rel_paths:
+            paths_arg = " ".join(shlex.quote(p) for p in rel_paths)
+            pytest_line = f'pytest -sv -m "{marker_expr}" {paths_arg}'
+            slug = _local_job_slug(LABEL_SUBSTR)
+            key = f"local_pytest_{slug}"
+            header = (
+                f"# Local marker tests — MODEL_TYPE={','.join(model_types)}, "
+                f"LABEL_SUBSTR={LABEL_SUBSTR!r} (test_*.py), files: {', '.join(rel_paths)}"
+            )
+            script_lines = [
+                "#!/usr/bin/env bash",
+                header,
+                "set -euo pipefail",
+                f'cd "{REPO_ROOT}"',
+                pytest_line,
+            ]
+            _write_job_script(key, script_lines, jobs_dir)
+            matched += 1
+
+        for json_rel, runner_rel in perf_pairs:
+            json_name = Path(json_rel).name
+            pytest_line = (
+                f'pytest -s -v -m "{marker_expr}" '
+                f"{shlex.quote(runner_rel)} "
+                f"--test-config-file {shlex.quote(json_rel)}"
+            )
+            slug = _local_job_slug(Path(json_rel).stem)
+            key = f"local_perf_{slug}"
+            header = (
+                f"# Local perf benchmark — MODEL_TYPE={','.join(model_types)}, "
+                f"config={json_name!r}, runner={Path(runner_rel).name}"
+            )
+            script_lines = [
+                "#!/usr/bin/env bash",
+                header,
+                "set -euo pipefail",
+                f'cd "{REPO_ROOT}"',
+                pytest_line,
+            ]
+            _write_job_script(key, script_lines, jobs_dir)
+            if key not in perf_job_keys:
+                perf_job_keys.append(key)
+            matched += 1
+
+        if matched == 0:
+            print(
+                f"# skip local: no tests/**/test_*.py or "
+                f"tests/dfx/perf/tests/*.json filename matches "
+                f"LABEL_SUBSTR={LABEL_SUBSTR!r}",
+                file=sys.stderr,
+            )
+        return matched
+
     pytest_line = f'pytest -sv -m "{marker_expr}"'
+    key = "local_pytest"
+    header = f"# Local marker tests — MODEL_TYPE={','.join(model_types)}"
     script_lines = [
         "#!/usr/bin/env bash",
-        f"# Local marker tests — MODEL_TYPE={','.join(model_types)}",
+        header,
         "set -euo pipefail",
         f'cd "{REPO_ROOT}"',
         pytest_line,
@@ -566,7 +789,7 @@ def run_stability_mode(jobs_dir: Path, model_types: list[str]) -> int:
         if not rel_path.is_file():
             print(f"# skip (missing file): {rel_path}", file=sys.stderr)
             continue
-        pytest_line = f"pytest -s -v {rel_posix}"
+        pytest_line = f"pytest -s -v --run-level full_model {rel_posix}"
         matched += 1
         script_lines = [
             "#!/usr/bin/env bash",
@@ -594,13 +817,15 @@ def main() -> None:
 
     matched_stability = 0
     matched_yaml = 0
+    matched_local = 0
     perf_job_keys: list[str] = []
+    job_timeouts: dict[str, int] = {}
 
     if want_stability:
         matched_stability = run_stability_mode(jobs_dir, model_types)
 
     if want_local:
-        run_local_mode(jobs_dir, model_types)
+        matched_local = run_local_mode(jobs_dir, model_types, perf_job_keys)
 
     if needs_yaml:
         if not YML.is_file():
@@ -634,8 +859,12 @@ def main() -> None:
                 "set -euo pipefail",
                 f'cd "{REPO_ROOT}"',
             ]
+            timeout_min = step_timeout_minutes(step)
+            if timeout_min is not None:
+                script_lines.append(f"# Buildkite timeout_in_minutes: {timeout_min}")
+                job_timeouts[key] = timeout_min
             script_lines.extend(exports)
-            script_lines.extend(pys)
+            script_lines.extend(prepend_timeout_to_pytest(p, timeout_min) for p in pys)
             _write_job_script(key, script_lines, jobs_dir)
             if "Perf Test" in label and key not in perf_job_keys:
                 perf_job_keys.append(key)
@@ -646,6 +875,7 @@ def main() -> None:
             manifest_path.write_text("\n".join(perf_job_keys) + "\n", encoding="utf-8")
         elif manifest_path.is_file():
             manifest_path.unlink()
+        _write_job_timeouts_manifest(jobs_dir, job_timeouts)
 
     errs: list[str] = []
     if want_stability and matched_stability == 0:
@@ -653,6 +883,17 @@ def main() -> None:
             f"No stability jobs matched TEST_TYPE={test_types!r} MODEL_TYPE={model_types!r} "
             f"LABEL_SUBSTR={LABEL_SUBSTR!r} under {REPO_ROOT}"
         )
+    if want_local and matched_local == 0:
+        if LABEL_SUBSTR:
+            errs.append(
+                f"No local jobs matched LABEL_SUBSTR={LABEL_SUBSTR!r} "
+                f"(tests/**/test_*.py or tests/dfx/perf/tests/*.json) "
+                f"under {REPO_ROOT}"
+            )
+        else:
+            errs.append(
+                f"Local mode produced no jobs TEST_TYPE={test_types!r} MODEL_TYPE={model_types!r}"
+            )
     if needs_yaml and matched_yaml == 0:
         errs.append(
             f"No YAML steps matched TEST_TYPE={test_types!r} MODEL_TYPE={model_types!r} "
@@ -671,29 +912,70 @@ if [[ "${DRY_RUN}" == "1" ]]; then
   exit 0
 fi
 
+# shellcheck source=tools/run_jobs_common.sh
+source "${SCRIPT_DIR}/../run_jobs_common.sh"
+
+PERF_RESULTS_SRC="${REPO_ROOT}/tests/dfx/perf/results"
+
+# Copy perf benchmark JSON written under tests/dfx/perf/results/ during this run into LOG_DIR.
+_collect_perf_result_jsons() {
+  local since_epoch="${1:?}"
+  local dest="${LOG_DIR}/perf_results"
+
+  if [[ ! -d "${PERF_RESULTS_SRC}" ]]; then
+    return 0
+  fi
+
+  COLLECT_SINCE_EPOCH="${since_epoch}" \
+  python3 - <<'PY'
+import os
+import shutil
+import sys
+from pathlib import Path
+
+repo = Path(os.environ["REPO_ROOT"]).resolve()
+src = repo / "tests" / "dfx" / "perf" / "results"
+dest = Path(os.environ["LOG_DIR"]).resolve() / "perf_results"
+since = float(os.environ["COLLECT_SINCE_EPOCH"])
+
+if not src.is_dir():
+    sys.exit(0)
+
+copied = 0
+for path in sorted(src.rglob("*.json")):
+    if not path.is_file():
+        continue
+    try:
+        if path.stat().st_mtime < since:
+            continue
+    except OSError:
+        continue
+    rel = path.relative_to(src)
+    out = dest / rel
+    out.parent.mkdir(parents=True, exist_ok=True)
+    shutil.copy2(path, out)
+    copied += 1
+
+if copied:
+    print(f"Collected {copied} perf result JSON file(s) -> {dest}", file=sys.stderr)
+PY
+}
+
 # Run jobs: YAML perf steps first (if manifest exists), then generate Excel (always when
 # manifest non-empty, even if some perf jobs failed; Excel uses whatever JSON exists on disk),
 # then remaining jobs. Paths align with nightly Buildkite (tests/dfx/perf/results).
 run_generated_jobs_with_tee() {
   set -o pipefail
   local any_fail=0
+  local _run_start
   local perf_list="${LOG_DIR}/jobs/.perf_job_keys"
   local -a perf_keys=()
+  _run_jobs_reset_timing
+  _run_start="$(_run_jobs_epoch_seconds)"
   if [[ -f "${perf_list}" ]] && [[ -s "${perf_list}" ]]; then
     mapfile -t perf_keys < "${perf_list}"
   fi
 
-  _run_one_job() {
-    local _job="$1"
-    local base out job_status
-    base="$(basename "${_job}" .sh)"
-    out="${LOG_DIR}/${base}.log"
-    echo "==> ${_job}  (tee ${out})" >&2
-    (cd "${REPO_ROOT}" && bash "${_job}") 2>&1 | tee "${out}"
-    job_status="${PIPESTATUS[0]}"
-    return "${job_status}"
-  }
-
   local k
   for k in "${perf_keys[@]}"; do
     [[ -n "${k}" ]] || continue
@@ -702,11 +984,11 @@ run_generated_jobs_with_tee() {
       any_fail=1
       continue
     fi
-    _run_one_job "${LOG_DIR}/jobs/${k}.sh" || any_fail=1
+    _run_one_job_with_timing "${LOG_DIR}/jobs/${k}.sh" || any_fail=1
   done
 
   if ((${#perf_keys[@]})); then
-    local excel_out excel_log excel_status excel_repo_logs
+    local excel_out excel_log excel_status excel_repo_logs excel_start excel_end excel_elapsed
     excel_repo_logs="${REPO_ROOT}/logs"
     mkdir -p "${excel_repo_logs}"
     excel_out="${excel_repo_logs}/nightly_perf_$(date -u +%Y%m%d-%H%M%S).xlsx"
@@ -716,6 +998,7 @@ run_generated_jobs_with_tee() {
         "(report reflects JSON already under tests/dfx/perf/results)." >&2
     fi
     echo "==> python3 tools/nightly/generate_nightly_perf_excel.py -> ${excel_out}  (tee ${excel_log})" >&2
+    excel_start="$(_run_jobs_epoch_seconds)"
     (
       cd "${REPO_ROOT}" && python3 tools/nightly/generate_nightly_perf_excel.py \
         --input-dir "${REPO_ROOT}/tests/dfx/perf/results" \
@@ -723,7 +1006,13 @@ run_generated_jobs_with_tee() {
         --output-file "${excel_out}"
     ) 2>&1 | tee "${excel_log}"
     excel_status="${PIPESTATUS[0]}"
-    if [[ "${excel_status}" -ne 0 ]]; then
+    excel_end="$(_run_jobs_epoch_seconds)"
+    excel_elapsed=$((excel_end - excel_start))
+    _run_jobs_record_timing "generate_nightly_perf_excel" "${excel_elapsed}" "${excel_status}"
+    if [[ "${excel_status}" -eq 0 ]]; then
+      echo "    finished in $(_run_jobs_format_duration "${excel_elapsed}")" >&2
+    else
+      echo "    failed after $(_run_jobs_format_duration "${excel_elapsed}") (exit ${excel_status})" >&2
       any_fail=1
       echo "generate_nightly_perf_excel.py failed (exit ${excel_status}). See ${excel_log}" >&2
     fi
@@ -739,15 +1028,16 @@ run_generated_jobs_with_tee() {
   for _job in "${LOG_DIR}/jobs"/*.sh; do
     base="$(basename "${_job}" .sh)"
     [[ ${_perf_done["${base}"]+isset} ]] && continue
-    _run_one_job "${_job}" || any_fail=1
+    _run_one_job_with_timing "${_job}" || any_fail=1
   done
   shopt -u nullglob
 
+  _collect_perf_result_jsons "${_run_start}"
+
+  _run_jobs_print_timing_summary "${_run_start}" "${any_fail}"
   if [[ "${any_fail}" -ne 0 ]]; then
-    echo "One or more selected jobs failed. See logs under ${LOG_DIR}." >&2
     return 1
   fi
-  echo "All selected jobs finished OK. Logs: ${LOG_DIR}/*.log" >&2
   return 0
 }
 

tools/run_jobs_common.sh

@@ -0,0 +1,653 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+#
+# Shared helpers for tools/run_*_jobs.sh (source this file; do not execute).
+#
+# Contents:
+#   1) Timing / tee runners used by ready, merge, and nightly.
+#   2) Ready/merge CLI + YAML extract entrypoint: run_yaml_ci_jobs_main
+#      (set BUILDKITE_REL and DEFAULT_LOG_SUBDIR in the entry script, then call it).
+#
+# Job timeouts: generated job scripts embed GNU timeout on the pytest line itself
+# (see prepend_timeout_to_pytest in the Python heredoc below). This runner only
+# executes bash job.sh and records exit 124 as TIMED OUT via jobs/.job_timeouts
+# metadata.
+
+if [[ -n "${_RUN_JOBS_COMMON_LOADED:-}" ]]; then
+  return 0 2>/dev/null || exit 0
+fi
+_RUN_JOBS_COMMON_LOADED=1
+
+if [[ -z "${SCRIPT_DIR:-}" ]]; then
+  SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+fi
+
+# ---------------------------------------------------------------------------
+# Timing / tee helpers (ready, merge, nightly)
+# ---------------------------------------------------------------------------
+
+RUN_JOB_NAMES=()
+RUN_JOB_SECONDS=()
+RUN_JOB_STATUSES=()
+
+_run_jobs_reset_timing() {
+  RUN_JOB_NAMES=()
+  RUN_JOB_SECONDS=()
+  RUN_JOB_STATUSES=()
+}
+
+_run_jobs_epoch_seconds() {
+  date +%s
+}
+
+_run_jobs_format_duration() {
+  local total="${1}"
+  local h m s
+  if [[ "${total}" -lt 60 ]]; then
+    printf '%ss' "${total}"
+    return 0
+  fi
+  m=$((total / 60))
+  s=$((total % 60))
+  if [[ "${m}" -lt 60 ]]; then
+    if [[ "${s}" -eq 0 ]]; then
+      printf '%dm' "${m}"
+    else
+      printf '%dm %ss' "${m}" "${s}"
+    fi
+    return 0
+  fi
+  h=$((m / 60))
+  m=$((m % 60))
+  if [[ "${m}" -eq 0 && "${s}" -eq 0 ]]; then
+    printf '%dh' "${h}"
+  elif [[ "${s}" -eq 0 ]]; then
+    printf '%dh %dm' "${h}" "${m}"
+  else
+    printf '%dh %dm %ss' "${h}" "${m}" "${s}"
+  fi
+}
+
+_run_jobs_record_timing() {
+  local name="${1}"
+  local seconds="${2}"
+  local status="${3}"
+  RUN_JOB_NAMES+=("${name}")
+  RUN_JOB_SECONDS+=("${seconds}")
+  RUN_JOB_STATUSES+=("${status}")
+}
+
+_run_jobs_lookup_timeout_minutes() {
+  local base="${1}"
+  local manifest="${LOG_DIR}/jobs/.job_timeouts"
+  local line key mins
+  [[ -f "${manifest}" ]] || return 1
+  while IFS='=' read -r key mins; do
+    [[ -n "${key}" ]] || continue
+    if [[ "${key}" == "${base}" ]]; then
+      printf '%s' "${mins}"
+      return 0
+    fi
+  done < "${manifest}"
+  return 1
+}
+
+_run_one_job_with_timing() {
+  local _job="$1"
+  local base out job_status start end elapsed timeout_min=""
+  base="$(basename "${_job}" .sh)"
+  out="${LOG_DIR}/${base}.log"
+  if timeout_min="$(_run_jobs_lookup_timeout_minutes "${base}")"; then
+    echo "==> ${_job}  (tee ${out}, pytest inline timeout ${timeout_min}m)" >&2
+  else
+    echo "==> ${_job}  (tee ${out})" >&2
+  fi
+  start="$(_run_jobs_epoch_seconds)"
+  (cd "${REPO_ROOT}" && bash "${_job}") 2>&1 | tee "${out}"
+  job_status="${PIPESTATUS[0]}"
+  end="$(_run_jobs_epoch_seconds)"
+  elapsed=$((end - start))
+  _run_jobs_record_timing "${base}" "${elapsed}" "${job_status}"
+  if [[ "${job_status}" -eq 0 ]]; then
+    echo "    finished in $(_run_jobs_format_duration "${elapsed}")" >&2
+  elif [[ "${job_status}" -eq 124 && -n "${timeout_min}" ]]; then
+    echo "    timed out (inline timeout ${timeout_min}m on pytest, exit 124)" >&2
+  elif [[ "${job_status}" -eq 124 ]]; then
+    echo "    timed out (exit 124)" >&2
+  else
+    echo "    failed after $(_run_jobs_format_duration "${elapsed}") (exit ${job_status})" >&2
+  fi
+  return "${job_status}"
+}
+
+_run_jobs_print_timing_summary() {
+  local wall_start="${1:-}"
+  local any_fail="${2:-0}"
+  local summary_path="${LOG_DIR}/timing_summary.log"
+  local -a lines=()
+  local i name secs status status_str failed_count=0
+  local wall_elapsed job_count total_elapsed=0
+
+  lines+=("=== Job timing summary ===")
+  for i in "${!RUN_JOB_NAMES[@]}"; do
+    name="${RUN_JOB_NAMES[$i]}"
+    secs="${RUN_JOB_SECONDS[$i]}"
+    status="${RUN_JOB_STATUSES[$i]}"
+    if [[ "${status}" -eq 0 ]]; then
+      status_str="OK"
+    elif [[ "${status}" -eq 124 ]]; then
+      status_str="TIMED OUT"
+      failed_count=$((failed_count + 1))
+    else
+      status_str="FAILED (exit ${status})"
+      failed_count=$((failed_count + 1))
+    fi
+    lines+=("  ${name}  $(_run_jobs_format_duration "${secs}")  ${status_str}")
+  done
+
+  job_count="${#RUN_JOB_NAMES[@]}"
+  if [[ -n "${wall_start}" ]]; then
+    wall_elapsed=$(( $(_run_jobs_epoch_seconds) - wall_start ))
+    lines+=("Total wall time: $(_run_jobs_format_duration "${wall_elapsed}") (${job_count} jobs)")
+  elif [[ "${job_count}" -gt 0 ]]; then
+    for secs in "${RUN_JOB_SECONDS[@]}"; do
+      total_elapsed=$((total_elapsed + secs))
+    done
+    lines+=("Total job time: $(_run_jobs_format_duration "${total_elapsed}") (${job_count} jobs)")
+  else
+    lines+=("Total wall time: 0s (0 jobs)")
+  fi
+
+  if [[ "${failed_count}" -gt 0 ]]; then
+    lines+=("Failed jobs: ${failed_count}/${job_count}")
+  fi
+
+  if [[ "${any_fail}" -ne 0 ]]; then
+    lines+=("Result: one or more jobs failed. See logs under ${LOG_DIR}.")
+  else
+    lines+=("Result: all jobs finished OK. Logs: ${LOG_DIR}/*.log")
+  fi
+
+  {
+    for line in "${lines[@]}"; do
+      printf '%s\n' "${line}"
+    done
+  } | tee "${summary_path}" >&2
+}
+
+# ---------------------------------------------------------------------------
+# Ready / merge: CLI + YAML extract + run (tools/run_ready_jobs.sh,
+# tools/run_merge_jobs.sh). Nightly does not use this entrypoint.
+#
+# Entry scripts must set before calling run_yaml_ci_jobs_main:
+#   BUILDKITE_REL      - e.g. .buildkite/test-ready.yml
+#   DEFAULT_LOG_SUBDIR - e.g. ready_jobs
+# ---------------------------------------------------------------------------
+
+_run_yaml_ci_jobs_usage() {
+  # Print the entry script header (caller is run_ready_jobs.sh / run_merge_jobs.sh).
+  sed -n '2,38p' "$0" | sed 's/^# \{0,1\}//'
+}
+
+_run_jobs_split_append_csv_array() {
+  local -n _arr="${1}"
+  local _raw="${2}"
+  local _ifs="${IFS}"
+  local _part
+  IFS=','
+  for _part in ${_raw}; do
+    IFS="${_ifs}"
+    _part="$(printf '%s' "${_part}" | tr '[:upper:]' '[:lower:]' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')"
+    [[ -n "${_part}" ]] || continue
+    _arr+=("${_part}")
+  done
+}
+
+_run_jobs_finalize_model_type_csv() {
+  if [[ "${MODEL_TYPE_FROM_CLI}" -eq 1 ]]; then
+    if ((${#MODEL_TYPE_CLI_PARTS[@]} == 0)); then
+      echo "--model-type requires a non-empty value" >&2
+      exit 2
+    fi
+    local -A _seen=()
+    local -a _out=()
+    local _x
+    for _x in "${MODEL_TYPE_CLI_PARTS[@]}"; do
+      if [[ "${_x}" == all ]]; then
+        printf '%s' "all"
+        return 0
+      fi
+      [[ ${_seen["${_x}"]+isset} ]] && continue
+      _seen["${_x}"]=1
+      _out+=("${_x}")
+    done
+    (IFS=','; printf '%s' "${_out[*]}")
+  else
+    printf '%s' "${MODEL_TYPE_ENV}"
+  fi
+}
+
+_run_jobs_find_repo_containing_yml() {
+  local dir="${1:-}"
+  [[ -n "$dir" ]] || return 1
+  dir="$(cd "$dir" && pwd)" || return 1
+  while true; do
+    if [[ -f "${dir}/${BUILDKITE_REL}" ]]; then
+      printf '%s\n' "$dir"
+      return 0
+    fi
+    [[ "${dir}" == "/" ]] && return 1
+    dir="$(dirname "${dir}")"
+  done
+}
+
+_run_jobs_derive_repo_root_from_yml() {
+  local yml="$1"
+  local d
+  d="$(cd "$(dirname "${yml}")" && pwd)"
+  [[ "$(basename "${d}")" == ".buildkite" ]] || return 1
+  printf '%s\n' "$(dirname "${d}")"
+}
+
+run_yaml_ci_jobs_main() {
+  if [[ -z "${BUILDKITE_REL:-}" || -z "${DEFAULT_LOG_SUBDIR:-}" ]]; then
+    echo "run_yaml_ci_jobs_main: set BUILDKITE_REL and DEFAULT_LOG_SUBDIR before calling." >&2
+    exit 1
+  fi
+
+  LABEL_SUBSTR="${LABEL_SUBSTR:-}"
+  MODEL_TYPE_ENV="${MODEL_TYPE:-all}"
+  MODEL_TYPE_CLI_PARTS=()
+  MODEL_TYPE_FROM_CLI=0
+  DRY_RUN="${DRY_RUN:-0}"
+  SKIP_SIMPLE=0
+
+  while [[ $# -gt 0 ]]; do
+    case "$1" in
+      -h | --help)
+        _run_yaml_ci_jobs_usage
+        exit 0
+        ;;
+      --dry-run)
+        DRY_RUN=1
+        shift
+        ;;
+      --log-dir)
+        LOG_DIR="$2"
+        shift 2
+        ;;
+      --yaml)
+        YML="$2"
+        shift 2
+        ;;
+      --repo-root)
+        REPO_ROOT="$2"
+        shift 2
+        ;;
+      --label-substr)
+        LABEL_SUBSTR="$2"
+        shift 2
+        ;;
+      --model-type)
+        MODEL_TYPE_FROM_CLI=1
+        _run_jobs_split_append_csv_array MODEL_TYPE_CLI_PARTS "$2"
+        shift 2
+        ;;
+      --skip-simple)
+        SKIP_SIMPLE=1
+        shift
+        ;;
+      *)
+        echo "Unknown option: $1" >&2
+        _run_yaml_ci_jobs_usage >&2
+        exit 2
+        ;;
+    esac
+  done
+
+  MODEL_TYPE="$(_run_jobs_finalize_model_type_csv)"
+
+  if [[ -n "${YML:-}" && -n "${REPO_ROOT:-}" ]]; then
+    REPO_ROOT="$(cd "${REPO_ROOT}" && pwd)"
+    YML="$(cd "$(dirname "${YML}")" && pwd)/$(basename "${YML}")"
+  elif [[ -n "${YML:-}" ]]; then
+    YML="$(cd "$(dirname "${YML}")" && pwd)/$(basename "${YML}")"
+    if ! REPO_ROOT="$(_run_jobs_derive_repo_root_from_yml "${YML}")"; then
+      echo "Could not derive REPO_ROOT from YML=${YML} (expected file at <repo>/${BUILDKITE_REL})." >&2
+      echo "Set REPO_ROOT explicitly (or pass --repo-root) for pytest working directory." >&2
+      exit 2
+    fi
+  elif [[ -n "${REPO_ROOT:-}" ]]; then
+    REPO_ROOT="$(cd "${REPO_ROOT}" && pwd)"
+    YML="${REPO_ROOT}/${BUILDKITE_REL}"
+  else
+    REPO_ROOT=""
+    if command -v git >/dev/null 2>&1; then
+      REPO_ROOT="$(git -C "${PWD}" rev-parse --show-toplevel 2>/dev/null || true)"
+    fi
+    if [[ -z "${REPO_ROOT}" ]]; then
+      REPO_ROOT="$(_run_jobs_find_repo_containing_yml "${PWD}" || true)"
+    fi
+    if [[ -z "${REPO_ROOT}" ]]; then
+      REPO_ROOT="$(_run_jobs_find_repo_containing_yml "${SCRIPT_DIR}" || true)"
+    fi
+    if [[ -z "${REPO_ROOT}" ]]; then
+      echo "Could not locate ${BUILDKITE_REL}. Set REPO_ROOT or YML, run from inside the vllm-omni clone," >&2
+      echo "or place this script (or run from a cwd) under the repository tree." >&2
+      exit 2
+    fi
+    YML="${REPO_ROOT}/${BUILDKITE_REL}"
+  fi
+
+  LOG_DIR="${LOG_DIR:-${REPO_ROOT}/logs/${DEFAULT_LOG_SUBDIR}}"
+
+  if [[ ! -f "${YML}" ]]; then
+    echo "YAML not found: ${YML}" >&2
+    exit 2
+  fi
+
+  if ! command -v python3 >/dev/null 2>&1; then
+    echo "python3 is required." >&2
+    exit 1
+  fi
+
+  mkdir -p "${LOG_DIR}/jobs"
+  LOG_DIR="$(cd "${LOG_DIR}" && pwd)"
+  export REPO_ROOT LOG_DIR YML LABEL_SUBSTR MODEL_TYPE DRY_RUN SKIP_SIMPLE
+
+  if [[ "${DRY_RUN}" != "1" ]]; then
+    shopt -s nullglob
+    _stale=( "${LOG_DIR}/jobs"/*.sh )
+    _stale_logs=( "${LOG_DIR}"/*.log )
+    shopt -u nullglob
+    if ((${#_stale[@]})); then
+      rm -f "${_stale[@]}"
+    fi
+    if ((${#_stale_logs[@]})); then
+      rm -f "${_stale_logs[@]}"
+    fi
+    rm -f "${LOG_DIR}/jobs/.job_timeouts"
+  fi
+
+  # shellcheck disable=SC2016,SC1078,SC1079
+  python3 - <<'PY'
+from __future__ import annotations
+
+import os
+import re
+import stat
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:
+    print("Missing PyYAML. Install with: pip install pyyaml", file=sys.stderr)
+    sys.exit(1)
+
+REPO_ROOT = Path(os.environ["REPO_ROOT"]).resolve()
+LOG_DIR = Path(os.environ["LOG_DIR"]).resolve()
+YML = Path(os.environ["YML"]).resolve()
+LABEL_SUBSTR = (os.environ.get("LABEL_SUBSTR") or "").strip()
+DRY_RUN = os.environ.get("DRY_RUN", "0") == "1"
+SKIP_SIMPLE = os.environ.get("SKIP_SIMPLE", "0") == "1"
+
+ALLOWED_MODEL_TYPES = frozenset({"all", "omni", "tts", "diffusion"})
+
+
+def parse_model_types(raw: str) -> list[str]:
+    parts = [p.strip().lower() for p in (raw or "").split(",") if p.strip()]
+    if not parts:
+        parts = ["all"]
+    bad = [p for p in parts if p not in ALLOWED_MODEL_TYPES]
+    if bad:
+        print(
+            f"Invalid MODEL_TYPE / --model-type value(s): {bad!r} "
+            f"(allowed: {', '.join(sorted(ALLOWED_MODEL_TYPES))})",
+            file=sys.stderr,
+        )
+        sys.exit(2)
+    if "all" in parts:
+        return ["all"]
+    out: list[str] = []
+    seen: set[str] = set()
+    for p in parts:
+        if p not in seen:
+            seen.add(p)
+            out.append(p)
+    return out
+
+
+PYTEST_CMD_RE = re.compile(
+    r"(?:timeout\s+\S+\s+)?(?:python3? -m\s+)?pytest\s+[^\n&|;]*"
+)
+
+
+def iter_leaf_steps(steps, group: str | None = None):
+    for raw in steps or []:
+        if not isinstance(raw, dict):
+            continue
+        nested = raw.get("steps")
+        if isinstance(nested, list) and nested:
+            g = raw.get("group")
+            next_group: str | None
+            if isinstance(g, str):
+                next_group = g
+            elif g is not None:
+                next_group = str(g)
+            else:
+                next_group = group
+            yield from iter_leaf_steps(nested, next_group)
+            continue
+        if raw.get("commands"):
+            yield raw, group
+
+
+def raw_command_text(step: dict) -> str:
+    raw = step.get("commands") or []
+    if isinstance(raw, str):
+        raw = [raw]
+    text = "\n".join((c.strip() if isinstance(c, str) else "") for c in raw if c)
+    return text.replace("$$", "$")
+
+
+def export_lines(text: str) -> list[str]:
+    out: list[str] = []
+    for line in text.splitlines():
+        s = line.strip()
+        if not s or s.startswith("#"):
+            continue
+        if s.startswith("export "):
+            out.append(s)
+    return out
+
+
+def pytest_lines(text: str) -> list[str]:
+    out: list[str] = []
+    for m in PYTEST_CMD_RE.finditer(text):
+        line = m.group(0).strip()
+        line_start = text.rfind("\n", 0, m.start()) + 1
+        before = text[line_start : m.start()]
+        if before.lstrip().startswith("#"):
+            continue
+        if line:
+            out.append(line)
+    return out
+
+
+def is_simple_test_step(label: str, group: str | None) -> bool:
+    if group and "Simple Test" in group:
+        return True
+    return label.startswith("Simple ·")
+
+
+def label_matches_model_type(label: str, model_types: list[str]) -> bool:
+    if "all" in model_types:
+        return True
+    if "omni" in model_types and "Omni ·" in label:
+        return True
+    if "tts" in model_types and "TTS ·" in label:
+        return True
+    if "diffusion" in model_types and "Diffusion" in label:
+        return True
+    return False
+
+
+def job_key_from_step(step: dict, label: str) -> str:
+    k = step.get("key")
+    if isinstance(k, str) and k.strip():
+        return k.strip()
+    slug = re.sub(r"[^\w\-.]+", "_", label or "job", flags=re.UNICODE)
+    slug = re.sub(r"_+", "_", slug).strip("_") or "job"
+    return slug
+
+
+def step_timeout_minutes(step: dict) -> int | None:
+    raw = step.get("timeout_in_minutes")
+    if raw is None:
+        return None
+    try:
+        minutes = int(raw)
+    except (TypeError, ValueError):
+        return None
+    return minutes if minutes > 0 else None
+
+
+_INLINE_TIMEOUT_PREFIX = re.compile(r"^timeout\s+(?:-\S+\s+)*\S+\s+", re.IGNORECASE)
+
+
+def timeout_kill_after_seconds() -> int:
+    raw = os.environ.get("RUN_JOB_TIMEOUT_KILL_AFTER", "60")
+    try:
+        seconds = int(raw)
+    except (TypeError, ValueError):
+        return 60
+    return seconds if seconds > 0 else 60
+
+
+def prepend_timeout_to_pytest(pytest_line: str, timeout_min: int | None) -> str:
+    """Prepend GNU timeout directly before pytest (Buildkite inline style)."""
+    if timeout_min is None:
+        return pytest_line.strip()
+    line = _INLINE_TIMEOUT_PREFIX.sub("", pytest_line.strip())
+    kill_after = timeout_kill_after_seconds()
+    return (
+        f"timeout --foreground --verbose --kill-after={kill_after} "
+        f"{timeout_min}m {line}"
+    )
+
+
+def _write_job_timeouts_manifest(jobs_dir: Path, job_timeouts: dict[str, int]) -> None:
+    manifest_path = jobs_dir / ".job_timeouts"
+    if job_timeouts:
+        manifest_path.write_text(
+            "\n".join(f"{k}={v}" for k, v in sorted(job_timeouts.items())) + "\n",
+            encoding="utf-8",
+        )
+    elif manifest_path.is_file():
+        manifest_path.unlink()
+
+
+def _write_job_script(key: str, script_lines: list[str], jobs_dir: Path) -> None:
+    body = "\n".join(script_lines) + "\n"
+    if DRY_RUN:
+        print(f"=== {key} ===")
+        print(body)
+        return
+    job_path = jobs_dir / f"{key}.sh"
+    job_path.write_text(body, encoding="utf-8")
+    try:
+        mode = job_path.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH
+        job_path.chmod(mode)
+    except OSError:
+        pass
+    print(f"generated {job_path}", file=sys.stderr)
+
+
+def main() -> None:
+    model_types = parse_model_types(os.environ.get("MODEL_TYPE", "all"))
+
+    jobs_dir = LOG_DIR / "jobs"
+    if not DRY_RUN:
+        jobs_dir.mkdir(parents=True, exist_ok=True)
+
+    if not YML.is_file():
+        print(f"YAML not found: {YML}", file=sys.stderr)
+        sys.exit(1)
+
+    data = yaml.safe_load(YML.read_text(encoding="utf-8"))
+    top_steps = (data or {}).get("steps") or []
+
+    matched_yaml = 0
+    job_timeouts: dict[str, int] = {}
+    for step, grp in iter_leaf_steps(top_steps):
+        label = step.get("label") or ""
+        if SKIP_SIMPLE and is_simple_test_step(label, grp):
+            continue
+        if LABEL_SUBSTR and LABEL_SUBSTR not in label:
+            continue
+        if not label_matches_model_type(label, model_types):
+            continue
+        text = raw_command_text(step)
+        exports = export_lines(text)
+        pys = pytest_lines(text)
+        if not pys:
+            print(f"# skip (no pytest line): {label!r}", file=sys.stderr)
+            continue
+
+        key = job_key_from_step(step, label)
+        matched_yaml += 1
+
+        script_lines = [
+            "#!/usr/bin/env bash",
+            f'# From Buildkite label: {label.replace(chr(10), " ")}',
+            "set -euo pipefail",
+            f'cd "{REPO_ROOT}"',
+        ]
+        timeout_min = step_timeout_minutes(step)
+        if timeout_min is not None:
+            script_lines.append(f"# Buildkite timeout_in_minutes: {timeout_min}")
+            job_timeouts[key] = timeout_min
+        script_lines.extend(exports)
+        script_lines.extend(prepend_timeout_to_pytest(p, timeout_min) for p in pys)
+        _write_job_script(key, script_lines, jobs_dir)
+
+    if not DRY_RUN:
+        _write_job_timeouts_manifest(jobs_dir, job_timeouts)
+
+    if matched_yaml == 0:
+        print(
+            f"No YAML steps matched MODEL_TYPE={model_types!r} "
+            f"LABEL_SUBSTR={LABEL_SUBSTR!r} SKIP_SIMPLE={SKIP_SIMPLE!r} in {YML}",
+            file=sys.stderr,
+        )
+        sys.exit(2)
+
+
+main()
+PY
+
+  if [[ "${DRY_RUN}" == "1" ]]; then
+    exit 0
+  fi
+
+  set -o pipefail
+  local any_fail=0
+  local _run_start
+  local _job
+  _run_jobs_reset_timing
+  _run_start="$(_run_jobs_epoch_seconds)"
+
+  shopt -s nullglob
+  for _job in "${LOG_DIR}/jobs"/*.sh; do
+    _run_one_job_with_timing "${_job}" || any_fail=1
+  done
+  shopt -u nullglob
+
+  _run_jobs_print_timing_summary "${_run_start}" "${any_fail}"
+  if [[ "${any_fail}" -ne 0 ]]; then
+    exit 1
+  fi
+  exit 0
+}

tools/run_merge_jobs.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+#
+# Extract steps from .buildkite/test-merge.yml that contain pytest, synthesize
+# small bash wrappers (exports + pytest), run them, and tee output to logs named
+# after each step's Buildkite "key" when present (otherwise a slug of the label).
+#
+# Model area (--model-type / MODEL_TYPE), multiple allowed (OR semantics):
+#   e.g. --model-type omni,tts
+#   omni     — label contains "Omni ·"
+#   tts      — label contains "TTS ·"
+#   diffusion— label contains "Diffusion"
+#   all      — no model filter (default)
+#
+#   --skip-simple — skip steps in the "Simple Test" Buildkite group and labels
+#                   starting with "Simple ·" (L1-style unit tests in the same YAML)
+#
+# Requirements: bash, python3, PyYAML (pip install pyyaml)
+#
+# Usage:
+#   bash tools/run_merge_jobs.sh
+#   REPO_ROOT=/path/to/vllm-omni bash tools/run_merge_jobs.sh --model-type diffusion --dry-run
+#   YML=/path/to/vllm-omni/.buildkite/test-merge.yml bash tools/run_merge_jobs.sh
+#
+# Repository / YAML (no dependency on where this script lives):
+#   • Set REPO_ROOT (or pass --repo-root) — default YAML is $REPO_ROOT/.buildkite/test-merge.yml
+#   • Or set YML (or --yaml) — repo root is inferred as parent of the .buildkite directory
+#   • Or run from inside the clone: git rev-parse --show-toplevel, else walk up from $PWD,
+#     then from the script's directory, until .buildkite/test-merge.yml exists
+#
+# Optional environment:
+#   REPO_ROOT     - vllm-omni root (working directory for pytest); see above
+#   YML           - path to test-merge.yml (default: $REPO_ROOT/.buildkite/test-merge.yml)
+#   LOG_DIR       - logs + generated job scripts (default: $REPO_ROOT/logs/merge_jobs);
+#                   per-job *.log plus timing_summary.log after the run
+#   MODEL_TYPE    - comma-separated and/or repeated flags (default: all); see above
+#   LABEL_SUBSTR  - substring of Buildkite step label
+#   DRY_RUN=1     - print extracted commands only; do not write scripts or run pytest
+#
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+BUILDKITE_REL=".buildkite/test-merge.yml"
+DEFAULT_LOG_SUBDIR="merge_jobs"
+
+# shellcheck source=tools/run_jobs_common.sh
+source "${SCRIPT_DIR}/run_jobs_common.sh"
+run_yaml_ci_jobs_main "$@"

tools/run_ready_jobs.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+#
+# Extract steps from .buildkite/test-ready.yml that contain pytest, synthesize
+# small bash wrappers (exports + pytest), run them, and tee output to logs named
+# after each step's Buildkite "key" when present (otherwise a slug of the label).
+#
+# Model area (--model-type / MODEL_TYPE), multiple allowed (OR semantics):
+#   e.g. --model-type omni,tts
+#   omni     — label contains "Omni ·"
+#   tts      — label contains "TTS ·"
+#   diffusion— label contains "Diffusion"
+#   all      — no model filter (default)
+#
+#   --skip-simple — skip steps in the "Simple Test" Buildkite group and labels
+#                   starting with "Simple ·" (L1-style unit tests in the same YAML)
+#
+# Requirements: bash, python3, PyYAML (pip install pyyaml)
+#
+# Usage:
+#   bash tools/run_ready_jobs.sh
+#   REPO_ROOT=/path/to/vllm-omni bash tools/run_ready_jobs.sh --model-type omni --dry-run
+#   YML=/path/to/vllm-omni/.buildkite/test-ready.yml bash tools/run_ready_jobs.sh
+#
+# Repository / YAML (no dependency on where this script lives):
+#   • Set REPO_ROOT (or pass --repo-root) — default YAML is $REPO_ROOT/.buildkite/test-ready.yml
+#   • Or set YML (or --yaml) — repo root is inferred as parent of the .buildkite directory
+#   • Or run from inside the clone: git rev-parse --show-toplevel, else walk up from $PWD,
+#     then from the script's directory, until .buildkite/test-ready.yml exists
+#
+# Optional environment:
+#   REPO_ROOT     - vllm-omni root (working directory for pytest); see above
+#   YML           - path to test-ready.yml (default: $REPO_ROOT/.buildkite/test-ready.yml)
+#   LOG_DIR       - logs + generated job scripts (default: $REPO_ROOT/logs/ready_jobs);
+#                   per-job *.log plus timing_summary.log after the run
+#   MODEL_TYPE    - comma-separated and/or repeated flags (default: all); see above
+#   LABEL_SUBSTR  - substring of Buildkite step label
+#   DRY_RUN=1     - print extracted commands only; do not write scripts or run pytest
+#
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+BUILDKITE_REL=".buildkite/test-ready.yml"
+DEFAULT_LOG_SUBDIR="ready_jobs"
+
+# shellcheck source=tools/run_jobs_common.sh
+source "${SCRIPT_DIR}/run_jobs_common.sh"
+run_yaml_ci_jobs_main "$@"