diff --git a/.claude/skills/add-diffusion-model/SKILL.md b/.claude/skills/add-diffusion-model/SKILL.md index c080ca412..c7cdb47ac 100644 --- a/.claude/skills/add-diffusion-model/SKILL.md +++ b/.claude/skills/add-diffusion-model/SKILL.md @@ -288,9 +288,36 @@ Required updates: 5. `examples/offline_inference/xxx/README.md` — offline example docs 6. `examples/online_serve/xxx/README.md` — online serve docs -### Step 8: Add E2E Tests (Recommended) +### Step 8: Add E2E Tests -Create `tests/e2e/online_serving/test_your_model_expansion.py`. +**Follow the [vllm-omni-test skill](../vllm-omni-test/SKILL.md)** for markers, file naming, Buildkite wiring, and run commands. Also read [l4_functionality_tests.inc.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/test_examples/l4_functionality_tests.inc.md) and [CI_5levels.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/CI_5levels.md). + +Classify the model's **CI priority** first: + +| Priority | Required test levels | Files & markers | +|----------|---------------------|-----------------| +| **High** (listed in [#1832](https://github.com/vllm-project/vllm-omni/issues/1832) or on the diffusion hot path) | **L1** · **L2** online · **L3** online + offline · **L4** feature + performance | See table below | +| **Medium** (*normal priority* in L4 docs) | **L3** online + offline · **L4** feature only | Fewer L4 parametrized rows | +| **Low** | **L4** feature only | One or two `*_expansion.py` cases | + +**Per-level deliverables (diffusion / `pytest.mark.diffusion`):** + +| Level | Location | Marker | CI pipeline | Notes | +|-------|----------|--------|-------------|-------| +| **L1** | `tests/diffusion/models/{slug}/`, `tests/diffusion/cache/`, transformer unit tests | `core_model` + `cpu` | `test-ready.yml` | Weight remap, `_sp_plan`, cache enabler registration, shape contracts | +| **L2** | `tests/e2e/online_serving/test_{slug}.py` (and offline if the category is offline-first) | **`core_model` + `advanced_model`** (both on baseline smoke) + `diffusion` + `@hardware_test` / `hardware_marks` | `test-ready.yml` | Default deploy smoke — minimal `num_inference_steps`, single prompt | +| **L3** | `tests/e2e/online_serving/test_{slug}.py` **and** `tests/e2e/offline_inference/test_{slug}.py` when offline matters | Baseline smoke: **`core_model` + `advanced_model`**; heavier cases: `advanced_model` only (+ `diffusion`) | `test-merge.yml` **or** merged into nightly diffusion function job | Real weights, streaming/API paths, LoRA/offload smoke | +| **L4** | `tests/e2e/online_serving/test_{slug}_expansion.py` (+ offline expansion if needed) | `full_model` + `diffusion` | `test-nightly.yml` (X2I / X2V / X2A function groups) | Feature combos per [#1832](https://github.com/vllm-project/vllm-omni/issues/1832); perf → `tests/dfx/perf/tests/test_{model}_vllm_omni.json` | + +**L2 & L3 online — same file, dual marks on the baseline smoke:** The **first / simplest** case in `test_{slug}.py` (default deploy, minimal steps, single prompt) should carry **both** `@pytest.mark.core_model` and `@pytest.mark.advanced_model` on the **same** function so L2 (`test-ready.yml`) and L3 (`test-merge.yml`) share one smoke test. Heavier deploy variants or API paths in the same file use **`advanced_model` only**. When L3 moves to nightly, migrate those heavier cases into `test_{slug}_expansion.py` with `full_model` and remove the dedicated `test-merge.yml` job (see `test_longcat_image_expansion.py`, `test_qwen_image_expansion.py`). + +**L4 design (high priority):** Combine multiple supported features (Cache-DiT, TP, USP, CFG, HSDP, CPU offload, quantization) into **few parametrized** `OmniServerParams` rows so each feature appears in at least one case without exploding GPU jobs. Shard single-GPU vs multi-GPU cases across the nightly X2I/X2V function steps (1-GPU vs `distributed_cuda`). + +**L4 design (medium / low):** One or two parametrized rows covering the best quality/perf trade-off; skip perf JSON unless the model is high priority. + +**Reference implementations:** `tests/e2e/online_serving/test_qwen_image_edit_expansion.py`, `tests/e2e/online_serving/test_longcat_image_expansion.py`, `tests/e2e/online_serving/test_hunyuan_video_15_expansion.py`. + +**Keep model-specific code inside test modules — not `tests/helpers/{slug}.py`:** deploy constants, prompts, sampling dicts, and inline `request_config` / `form_data` belong in each `test_{slug}.py` and `test_{slug}_expansion.py`. Do not add per-model files under `tests/helpers/`; reuse only repo-wide harness (`mark`, `media`, `runtime`, `stage_config`, `assertions`). **L2+ online/offline e2e:** reuse or add `send_*_request` in `tests/helpers/runtime.py` — tests call the handler, not raw `omni.generate` / HTTP. See [vllm-omni-test skill](../vllm-omni-test/SKILL.md) § **Runtime send helpers**. ### Step 9: Add Cache-DiT Acceleration @@ -568,6 +595,7 @@ See the [Profiling Single-Stage Diffusion](../../../docs/contributing/profiling. ## Reference Files +- [vllm-omni-test skill](../vllm-omni-test/SKILL.md) — L1–L4 markers, naming, Buildkite wiring, run commands - [Transformer Adaptation](references/transformer-adaptation.md) — porting transformers from diffusers - [Custom Model Patterns](references/custom-model-patterns.md) — patterns for non-diffusers models - [Parallelism Patterns](references/parallelism-patterns.md) — TP, SP/USP, CFG parallel, HSDP implementation details diff --git a/.claude/skills/add-tts-model/SKILL.md b/.claude/skills/add-tts-model/SKILL.md index 811a359d0..260a7e924 100644 --- a/.claude/skills/add-tts-model/SKILL.md +++ b/.claude/skills/add-tts-model/SKILL.md @@ -305,13 +305,72 @@ def build_voice_clone_prompt(ref_audio_path: str, text: str, codec) -> list: ] ``` +### Test Case Writing (CI Levels) + +**Follow the [vllm-omni-test skill](../vllm-omni-test/SKILL.md)** for markers, file naming (`test_{slug}.py` / `test_{slug}_expansion.py`), Buildkite wiring, and copy-paste run commands. Also read [CI_5levels.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/CI_5levels.md) and [tests_style.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/tests_style.md). + +Classify the model's **CI priority** first (high / medium / low). High-priority TTS models are typically those on the integration hot path or listed in tracking issues such as [#1832](https://github.com/vllm-project/vllm-omni/issues/1832); medium and low tiers cover the long tail. When unsure, ask the reviewer which tier applies. + +| Priority | Required test levels | Files & markers | +|----------|---------------------|-----------------| +| **High** | **L1** unit/logic · **L2** online smoke · **L3** online + offline integration · **L4** feature + performance | See table below | +| **Medium** | **L3** online + offline · **L4** feature only | Skip dedicated L1/L2 unless fixing a logic bug | +| **Low** | **L4** feature only | One or two `*_expansion.py` parametrized cases | + +**Per-level deliverables (TTS / `pytest.mark.tts`):** + +| Level | Location | Marker | CI pipeline | Notes | +|-------|----------|--------|-------------|-------| +| **L1** | `tests/model_executor/…`, `tests/entrypoints/openai_api/…`, stage-processor tests | `core_model` + `cpu` | `test-ready.yml` | Prompt assembly, async_chunk helpers, `serving_speech` validation — no GPU | +| **L2** | `tests/e2e/online_serving/test_{slug}.py` | **`core_model` + `advanced_model`** (both on baseline smoke) + `tts` + `@hardware_test(...)` | `test-ready.yml` (`ready` label) | Default deploy smoke: single `/v1/audio/speech` or offline `OmniRunner` path | +| **L3** | `tests/e2e/online_serving/test_{slug}.py` **and** `tests/e2e/offline_inference/test_{slug}.py` | Baseline smoke: **`core_model` + `advanced_model`**; heavier cases: `advanced_model` only (+ `tts`) | `test-merge.yml` **or** merged into nightly TTS function job | Streaming, voice clone, batch/queue, async_chunk | +| **L4** | `tests/e2e/online_serving/test_{slug}_expansion.py`, optional offline expansion | `full_model` + `tts` | `test-nightly.yml` (`:full_moon: TTS · Function Test with L4`) | Feature matrix; perf → `tests/dfx/perf/tests/test_tts.json` | + +**L2 & L3 online — same file, dual marks on the baseline smoke:** The **first / simplest** case in `test_{slug}.py` (default deploy, single non-streaming `/v1/audio/speech` or equivalent offline path) should carry **both** `@pytest.mark.core_model` **and** `@pytest.mark.advanced_model` on the **same** function so it runs in L2 (`test-ready.yml`, `--run-level core_model`, basic validation) and L3 (`test-merge.yml`, `--run-level advanced_model`, deeper validation) without duplicating the test. In-tree examples: `test_voxcpm2_tts.py::test_text_to_audio_001`, `test_qwen3_tts_customvoice.py::test_text_to_audio_001`. + +Heavier scenarios in the same file use **`advanced_model` only** (streaming, extra languages, concurrency, async_chunk, batch). Example: `test_voice_clone_en_streaming_001` → `advanced_model` only. When migrating L3 to nightly, move those heavier cases into `test_{slug}_expansion.py` with `full_model` and drop the dedicated merge job (see `test_ming_tts_expansion.py`, `test_glm_tts_expansion.py`). + +```python +@pytest.mark.core_model +@pytest.mark.advanced_model +@pytest.mark.tts +@hardware_test(res={"cuda": "L4"}, num_cards=1) +@pytest.mark.parametrize("omni_server", tts_server_params, indirect=True) +def test_voice_clone_en_non_streaming_001(omni_server, openai_client) -> None: + openai_client.send_audio_speech_request({...}) +``` + +**L4 consolidation:** Prefer parametrized `OmniServerParams` rows (`default`, `async_chunk`, feature flags) in one expansion module rather than many merge-only files ([#1832](https://github.com/vllm-project/vllm-omni/issues/1832)). + +**L4 performance (high-priority models):** Add latency / throughput / stress rows in `tests/dfx/perf/tests/test_tts.json`, or a dedicated `tests/dfx/perf/tests/test_{slug}.json` when the model must not join the shared nightly server matrix before integration lands (see VoxCPM2 / Coqui XTTS pattern). Register the model in `benchmarks/tts/model_configs.yaml` for local `bench_tts.py`. Wire a **separate** `test-nightly.yml` Perf Test step when the JSON is not merged into `test_tts.json` yet. + +**Keep model-specific code inside test modules — not `tests/helpers/{slug}.py`:** + +- Put `MODEL`, deploy path, vendored `REF_AUDIO_URL`, `get_prompt()`, and inline `request_config` dicts **in each** `test_{slug}.py`, `test_{slug}_expansion.py`, offline `test_{slug}.py`, and L1 `test_{slug}_*.py` as needed. +- **Do not** add `tests/helpers/{slug}.py` (or `tests/helpers/{model_name}.py`) to deduplicate constants or request builders across those files. A little duplication is intentional; follow in-tree references such as `tests/e2e/online_serving/test_glm_tts.py` and `tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py`. +- `tests/helpers/` is for **repo-wide** harness code only (`mark.py`, `media.py`, `runtime.py`, `stage_config.py`, `assertions.py`, `fixtures/`). Import those; do not extend the tree with per-model modules. + +**Runtime send helpers (`tests/helpers/runtime.py`) — online and offline e2e:** + +| Path | Fixture | Call | +|------|---------|------| +| Online `/v1/*` | `openai_client` | `openai_client.send_*_request(request_config)` | +| Offline inference | `omni_runner_handler` | `omni_runner_handler.send_*_request(request_config)` | + +1. **Grep `runtime.py` first** — reuse `send_omni_request`, `send_diffusion_request`, `send_audio_speech_request` (online + offline Qwen-style TTS), `send_single_stage_tts_request` (Coqui XTTS / MOSS-TTS-Nano offline), etc. +2. **No matching helper** → add `send__request` (or `send__http_request` for negative/dfx) in **`runtime.py`** with general `assert_*` bundled inside, **then** call it from the test. +3. **Test file** holds `request_config` dicts only — not `omni.generate`, not `_collect_audio()`, not raw HTTP/SDK. +4. Wire `tests/helpers/runtime.py` into Buildkite `source_file_dependencies` when you add helpers. + +See [vllm-omni-test skill](../vllm-omni-test/SKILL.md) § **Runtime send helpers** for full tables and exceptions. + ### Deliverables - Updated `serving_speech.py` with all 5 integration points (single commit) - Client scripts and server launcher under `examples/online_serving/text_to_speech//` - Gradio demo with streaming and voice cloning UI in the same dir -- E2E online serving test (`tests/e2e/online_serving/test_.py`) -- Buildkite CI entry in `.buildkite/test-merge.yml` +- E2E tests per **Test Case Writing (CI Levels)** above (priority tier determines L1–L4 scope) +- Buildkite wired per level: `test-ready.yml` (L1/L2), `test-merge.yml` or nightly function job (L3), `test-nightly.yml` (L4) — see [vllm-omni-test skill](../vllm-omni-test/SKILL.md) - New section in `examples/online_serving/text_to_speech/README.md` (table row + per-model section). Do **not** create a top-level `examples/online_serving//` dir or a per-model `README.md` inside `text_to_speech//`. ### E2E test pitfalls to avoid @@ -326,9 +385,9 @@ def build_voice_clone_prompt(ref_audio_path: str, text: str, codec) -> list: - **Use the harness readiness gate.** The fixture waits for HTTP 200 on `/health`; don't add `time.sleep` in tests. If warmup is incomplete, make `/health` return non-200 until you're actually ready. -- **Mark with `@pytest.mark.core_model` + `hardware_test(res={"cuda": "H100"})`** - so the test lands in `test-ready.yml` (triggered by the `ready` label) rather - than only nightly. +- **Mark tests per the CI Levels table** — baseline smoke: `core_model` + `advanced_model`; heavier cases: `advanced_model` only; L4 expansion: `full_model` +- **No per-model helper modules** — do not create `tests/helpers/{slug}.py`; keep constants and `request_config` payloads in the test file +- **Online and offline e2e go through `runtime.py`** — `openai_client.send_audio_speech_request` (online); `omni_runner_handler.send_audio_speech_request` (Qwen-style offline) or `send_single_stage_tts_request` (single-stage offline). Add a new `send_*_request` in `runtime.py` when none fits; do not embed `omni.generate` or HTTP in tests ## Phase 4: Async Chunk (Streaming) @@ -461,8 +520,8 @@ Use this checklist when integrating a new TTS model: - [ ] Voice cloning works (if supported) - [ ] All response formats work (wav, mp3, flac, pcm) - [ ] Client scripts and server launcher created -- [ ] E2E online serving test written (`tests/e2e/online_serving/test_.py`) -- [ ] Buildkite CI entry added to `.buildkite/test-merge.yml` +- [ ] E2E tests added per model priority tier (see **Test Case Writing (CI Levels)**) +- [ ] Buildkite entries match level: `test-ready.yml` / `test-merge.yml` or nightly TTS job / `test-nightly.yml` - [ ] Gradio demo working - [ ] Documentation added (offline + online docs, nav, supported models) @@ -500,5 +559,6 @@ Project docs and adjacent skills: - [TTS audio skill](../vllm-omni-audio-tts/SKILL.md) — supported models and usage - [Fish Speech integration](../vllm-omni-audio-tts/references/fish-speech.md) — complete example of Phases 1–3 - [Qwen3-TTS reference](../vllm-omni-audio-tts/references/qwen-tts.md) — complete example of all 5 phases +- [vllm-omni-test skill](../vllm-omni-test/SKILL.md) — L1–L4 markers, naming, Buildkite wiring, run commands - [Adding a TTS model (developer guide)](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/model/adding_tts_model.md) - `plan/voxcpm2_native_ar_design.md` — VoxCPM2's vLLM-native AR + side-computation pattern (distinct from the generator-based single-stage described above) diff --git a/.claude/skills/readme.md b/.claude/skills/readme.md index f42fcb6ab..b8e9bbb13 100644 --- a/.claude/skills/readme.md +++ b/.claude/skills/readme.md @@ -30,6 +30,12 @@ include: - `precheck-pr`: self-check a branch before creating a PR — validates PR title format, catches dead code, verifies accuracy/perf claims, and confirms merge readiness +- `vllm-omni-test`: guides generation and execution of CI-aligned tests (L1–L4), + pytest marker selection (`core_model` / `advanced_model` / `full_model`, + `omni` / `tts` / `diffusion`), Buildkite wiring (`test-ready.yml`, + `test-merge.yml`, `test-nightly.yml`, `test-weekly.yml`), and copy-paste + local plus CI-like `pytest` commands; see `references/test-routing.md` for + level-to-command mapping - `review-pr`: provides a structured workflow for reviewing pull requests - `vllm-omni-npu-model-runner-upgrade`: upgrades NPU model runners to align with the latest vllm-ascend NPUModelRunner diff --git a/.claude/skills/vllm-omni-test/SKILL.md b/.claude/skills/vllm-omni-test/SKILL.md new file mode 100644 index 000000000..40a81f4cf --- /dev/null +++ b/.claude/skills/vllm-omni-test/SKILL.md @@ -0,0 +1,890 @@ +--- +name: vllm-omni-test +description: Generate and run tests for vllm-project/vllm-omni with CI-aligned levels and markers; wire new tests into Buildkite (test-ready.yml for L1/L2, test-merge.yml for L3, test-nightly.yml for L4). On completion, always provide copy-paste local and CI-like pytest commands plus prerequisites. Use when creating regression tests, adding L1-L4 coverage, selecting pytest markers, or validating fixes from issues/PRs. +--- + +# vLLM-Omni Test Generator & Runner + +## Purpose + +Use this skill to generate minimal, stable test cases and run them with the correct marker/level strategy for [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni). + +**Link convention:** Paths such as `.buildkite/` and `docs/contributing/` live at the **vllm-omni repo root**. Markdown links use repo-relative paths from this skill file (e.g. `../../../.buildkite/test-ready.yml`, `../../../docs/contributing/ci/CI_5levels.md`). + +Default priorities: + +1. Reproducible regression coverage for bug fixes +2. Correct test level and marker selection +3. Low flake, low dependency tests first +4. CI-compatible run commands +5. **Actionable run commands for the human**: whenever you add or change tests, always finish with **copy-paste-ready** `pytest` lines (local: single file and/or single test; CI-like: markers + `--run-level`), plus short **prerequisites** (GPU tier, HF cache, optional `model_prefix`). Do not assume the reader will infer commands from `test-routing.md` alone. + +## Inputs + +- Issue/PR link and summary +- Changed files or suspected code path +- Whether the user wants local quick validation or CI-equivalent validation +- Hardware constraints (CPU only / CUDA / ROCm / NPU) + +## Workflow + +### Step 1: Classify Test Goal + +- **Bugfix regression**: start from a minimal failing scenario and add assertions that prevent recurrence. Before writing tests, output **`required` / `recommended` / `not_needed`**: **`required`** — stable logic/contract bug that should have been caught; **`recommended`** — environment-sensitive but a small regression still helps; **`not_needed`** — one-off external/config failure or existing tests already cover the path. Prefer the narrowest stable L1 (CPU) case; escalate to L2/L3 only when the bug needs real weights or serving. +- **Feature coverage**: verify new behavior and one negative/boundary case. +- **Perf/benchmark claim**: require benchmark-oriented tests and explicit metrics. + +### Step 2: Select Test Level + +- **L1**: unit/logic, deterministic, CPU-friendly, fastest feedback. +- **L2**: basic e2e/integration and platform-dependent checks. +- **L3/L4**: advanced model/integration/perf validation. + +Use [references/test-routing.md](references/test-routing.md) for level-to-marker and command mapping. + +### Step 3: Pick Markers + +Always attach markers deliberately: + +- **Level**: `core_model` (L1/L2) and/or `advanced_model` (L3) and/or `full_model` (L4 nightly) +- **Model type** (required on model-centric e2e — pick exactly one): + - `omni` — end-to-end multimodal LLM pipelines (thinker/talker/stages; Qwen-Omni family) + - `tts` — speech synthesis / TTS-only models (`/v1/audio/speech`, voice clone, etc.) + - `diffusion` — generative diffusion models (image / audio / text / video from noise) +- **Cross-cutting area** (when relevant): `parallel`, `cache`, `example`, `benchmark` +- **Hardware**: `cpu`, `gpu`, `cuda`, `rocm`, `npu`, `L4`, `H100`, `distributed_cuda`, … +- Optional: `slow`, distributed markers when multi-card is required + +**Baseline smoke (L2 + L3):** The simplest e2e case per model — default deploy, minimal request — should usually carry **both** `@pytest.mark.core_model` and `@pytest.mark.advanced_model` on the **same** test function so `test-ready.yml` and `test-merge.yml` share one test. `send_*_request` picks validation depth from `--run-level`. References: `test_voxcpm2_tts.py::test_text_to_audio_001`, `test_qwen3_tts_customvoice.py::test_text_to_audio_001`. Heavier scenarios use **`advanced_model` only**; L4 expansion uses **`full_model`**. + +For hardware-aware tests, prefer `@hardware_test(...)` or `hardware_marks(...)` in `tests/helpers/mark.py`. + +**Diffusion nightly split** (`test-nightly.yml`): all diffusion tests use `pytest.mark.diffusion`, but CI groups them by **output modality**, not by separate pytest markers: + +| Nightly group | Scope | Typical models / paths | +|---------------|--------|-------------------------| +| **Diffusion X2I(&A&T) Model Test** | x2**i** (image), x2**a** (audio), x2**t** (text) and other **non-video** diffusion | Qwen-Image*, BAGEL, FLUX, SD3, Z-Image, LongCat, DreamZero, … — `test_*_expansion.py` under X2I shards; L4 sweep uses `-k "not test_wan and not test_bagel_expansion and not hunyuan"` for L4 | +| **Diffusion X2V Model Test** | x2**v** (video) only | Wan2.2, HunyuanVideo 1.5, Wan VACE, … — e.g. `test_wan22_expansion.py`, `test_hunyuan_video_15_expansion.py` | + +Wire new **video** diffusion expansion tests into the **X2V** group; wire **image/audio/text** diffusion expansion tests into **X2I(&A&T)**. Do not place x2v modules in X2I shards (see comment in `test-nightly.yml` above the X2I group). + +### Naming: generated test module files (L2–L4, model-centric e2e) + +When adding **new** pytest modules whose primary scope is a **specific model** (typical under `tests/e2e/offline_inference/` or `tests/e2e/online_serving/`), use this filename pattern: + +| Level | Filename pattern | Example (model `Qwen/Qwen2.5-Omni-7B`) | +|-------|------------------|----------------------------------------| +| **L2**, **L3** | `test_{lowercase_model_slug}.py` | `test_qwen2_5_omni.py` | +| **L4** | `test_{lowercase_model_slug}_expansion.py` | `test_qwen2_5_omni_expansion.py` | + +**Slug rules for `{lowercase_model_slug}`:** + +1. Start from the HuggingFace-style id (e.g. `Qwen/Qwen2.5-Omni-7B`), but **do not** put the org into the filename: use the **repo segment only** (`Qwen2.5-Omni-7B`), not `Qwen_...` / `qwen_qwen2_5_...`. +2. **Lowercase**; replace `.`, `-`, and whitespace with a single `_` (e.g. `Qwen2.5-Omni-7B` → `qwen2_5_omni`). **Omit** trailing size tokens such as `7b` / `30b` in the basename when a single file covers that model line in the directory (matches `test_qwen2_5_omni.py` in-tree). +3. If two checkpoints in the same folder need separate modules, add a **minimal** disambiguator (e.g. `_7b` vs `_3b`) only then. +4. **L1** unit tests are **not** bound to this pattern; use `tests//test_.py` as today. + +Routing tables and commands: [references/test-routing.md](references/test-routing.md) § *Model-centric e2e filename convention*. + +Existing references: `tests/e2e/offline_inference/test_qwen2_5_omni.py` (L2-style omni), `tests/e2e/offline_inference/test_qwen3_5_9b.py` (L2-style omni, single-stage VL), `tests/e2e/online_serving/test_qwen3_omni_expansion.py` (L4-style omni), `tests/e2e/online_serving/test_qwen_image_edit_expansion.py` / `test_qwen_image_expansion.py` (L4-style diffusion). + +### Step 4: Generate Test Case Skeleton + +**1. Pick the functional scenario** (then choose directory, fixtures, and markers): + +| Scenario | Typical location | Fixtures / runner pattern | Baseline markers & level | +|----------|------------------|---------------------------|---------------------------| +| **Offline inference e2e** | `tests/e2e/offline_inference/` | **Module (default):** `omni_runner` + `omni_runner_handler`. **Function (isolation only):** `omni_runner_function` + `omni_runner_handler_function`. Diffusion/TTS may use `Omni(...).generate` directly | L2: `core_model` + **one of** `omni` / `tts` / `diffusion`; `@hardware_test(...)` when GPU/NPU is required | +| **Online serving e2e** | `tests/e2e/online_serving/` | **Module (default):** `omni_server` + `openai_client`. **Function (isolation only):** `omni_server_function` + `openai_client_function`. Clients: `send_omni_request` (omni), `send_audio_speech_request` (tts), `send_diffusion_request` / `send_video_diffusion_request` / `send_images_generations_request` (diffusion) | Baseline smoke: **`core_model` + `advanced_model`**; heavier paths: `advanced_model` only; L4 expansion: `full_model` | +| **Documentation / runnable examples** | `tests/examples/offline_inference/`, `tests/examples/online_serving/` | **Offline docs (preferred):** extract Python/Bash blocks from the doc README (e.g. `ReadmeSnippet.extract_readme_snippets`), `pytest.mark.parametrize` each snippet, run via `example_runner.run` with a stable `output_subfolder`. **Online docs:** copy client/request scripts into dedicated tests and keep them in sync with the doc page. | Usually **L4**: `advanced_model`, often `example` plus hardware marks matching the nightly docs-example job (see `.buildkite/test-nightly.yml`). Full conventions: [docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md](../../../docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md) (introduced in [PR #1910](https://github.com/vllm-project/vllm-omni/pull/1910): naming, output directory layout, skip rules, avoid trimming `num_inference_steps` without a strong CI reason). | +| **Performance / benchmark** | `tests/dfx/perf/tests/*.json` + `run_*_benchmark.py` | JSON or script-driven server + load config; assert explicit metrics / baselines | L4 Perf: `full_model` + `benchmark`; wire `test-nightly.yml` Perf steps | +| **Invalid parameter / negative HTTP validation** | `tests/dfx/reliability/invalid_param_test/` | Live `omni_server` + low-level `send_*_http_request` with `err_code` / `err_message` | `pytest.mark.slow` + `omni` / `tts` / `diffusion` + `@hardware_marks` (`H100` or `L4`); CI in **`test-weekly.yml`** (not ready/merge/nightly) | + +**If the user’s test plan includes invalid parameter validation / invalid params / negative HTTP / 400 validation:** do **not** add those `test_*` functions to `tests/e2e/online_serving/test_*.py` or `*_expansion.py`. **Move or author them** under `tests/dfx/reliability/invalid_param_test/` in the **endpoint-matching script** (see **Invalid parameter validation** below). Success-path e2e and invalid-param dfx tests must stay in separate modules. + +**1b. Model type — `omni` vs `tts` vs `diffusion`** + +After choosing offline/online/docs/perf, classify the **product under test** and attach **exactly one** model-type marker. All three can live under the same `tests/e2e/` trees; conventions diverge: + +| Dimension | **Omni** (`pytest.mark.omni`) | **TTS** (`pytest.mark.tts`) | **Diffusion** (`pytest.mark.diffusion`) | +|-----------|------------------------------|----------------------------|----------------------------------------| +| **What it is** | Multimodal LLM pipeline (thinker/talker/stages; text + vision + audio I/O) | Speech synthesis / voice models | Generative diffusion (noise → image, audio, text, or video) | +| **Examples** | Qwen2.5-Omni, Qwen3-Omni | Qwen3-TTS, VoxCPM2, Higgs-Audio, Step-Audio2 | Qwen-Image, BAGEL, Wan2.2, HunyuanVideo | +| **Offline runner** | `omni_runner` + `omni_runner_handler`, `generate_multimodal` | `Omni(...).generate` or stage YAML + TTS params | `Omni(...).generate` + `OmniDiffusionSamplingParams` | +| **Online client** | `openai_client.send_omni_request` (chat completions, modalities) | `openai_client.send_audio_speech_request` (`/v1/audio/speech`) | `send_diffusion_request` (chat/T2I), `send_video_diffusion_request` (`/v1/videos`, X2V), or `send_images_generations_http_request` / `send_images_edits_http_request` (DALL-E routes) | +| **Typical assertions** | Stage outputs, text/audio keywords via `OmniRunnerHandler` / response handler | WAV bytes, stream chunks, speech endpoint contract | Image/video dimensions, `final_output_type`, `assert_diffusion_response` | +| **Stage / deploy YAML** | Per-model omni stage configs (`ci/qwen3_omni_moe.yaml`, …) | `qwen3_tts.yaml`, `voxcpm2.yaml`, … | Often default serve; parallel/offload YAML for heavy DiT | +| **Nightly group** (`test-nightly.yml`) | **Omni Model Test** — `-m "full_model and omni"` | **TTS Model Test** — `-m "full_model and tts"` | **Diffusion X2I(&A&T)** *or* **Diffusion X2V** (see Step 3 table; same `diffusion` marker, different YAML group / file shard) | +| **L4 pressure** | Expansion per modality/model as needed | Expansion + accuracy/perf in TTS group | X2I: merge feature combos per [#1832](https://github.com/vllm-project/vllm-omni/issues/1832); X2V: separate nightly group | + +Do **not** mix fixtures across types (e.g. do not use `omni_runner` layout for a pure diffusion or TTS model without mirroring an in-tree test in that family). + +**Diffusion only — X2I(&A&T) vs X2V (nightly routing, not extra markers):** + +- **X2I(&A&T)**: image / audio / text generation — Qwen-Image*, FLUX, SD3, Z-Image, BAGEL (expansion), LongCat, audio diffusion, etc. +- **X2V**: video generation only — Wan2.2, HunyuanVideo 1.5, Wan VACE, LTX video similarity paths. + +When adding a new `test_*_expansion.py`, place it in the matching **nightly group** step (explicit file list in `test-nightly.yml`), not only by marker expression. + +**2. Use the narrowest deterministic skeleton for the scenario** + +*L1 unit / logic (CPU-first):* + +**Mocking rule (L1 only):** use **pytest** integration — **`mocker`** (`pytest-mock`) or **`monkeypatch`** (built-in). **Do not** import or call **`unittest.mock`** (`patch`, `MagicMock`, `@patch`, `with patch(...)`, etc.) in L1 tests; patches must auto-revert with the test lifecycle. + +```python +import pytest + +pytestmark = [pytest.mark.core_model, pytest.mark.cpu] + + +def test_(mocker): + # Prefer mocker.patch / mocker.spy / mocker.Mock — not unittest.mock.patch + fake_fn = mocker.patch("vllm_omni.some.module.expensive_call", return_value=...) + # Act + # Assert + fake_fn.assert_called_once() + + +def test_(monkeypatch): + # For simple env / attribute substitution without a Mock object + monkeypatch.setenv("SOME_FLAG", "1") + monkeypatch.setattr("vllm_omni.some.module.CONST", 42) + ... +``` + +**Avoid in L1:** + +```python +# BAD — do not use in L1 unit tests +from unittest.mock import patch, MagicMock + +@patch("vllm_omni.some.module.fn") +def test_bad(mock_fn): ... + +def test_also_bad(): + with patch("...") as m: ... +``` + +See **L1 unit test constraints (mocking)** below for the full do/don't list. + +*Offline multimodal e2e — **Omni** (representative):* + +```python +@pytest.mark.core_model +@pytest.mark.omni +@hardware_test(...) +@pytest.mark.parametrize("omni_runner", test_params, indirect=True) +def test_(omni_runner, omni_runner_handler) -> None: + request_config = {"prompts": ..., "modalities": [...]} # optional: images, videos, audios + omni_runner_handler.send_omni_request(request_config) +``` + +*Offline generative e2e — **Diffusion** (representative):* + +```python +@pytest.mark.core_model +@pytest.mark.diffusion +@hardware_test(...) +@pytest.mark.parametrize("omni_runner", test_params, indirect=True) +def test_text_to_image_001(omni_runner_handler) -> None: + omni_runner_handler.send_diffusion_request({"prompt": "...", "extra_body": {"num_inference_steps": 4, ...}}) +``` + +*Offline TTS e2e — **Qwen3-TTS** (two-stage; representative):* + +```python +@pytest.mark.advanced_model +@pytest.mark.tts +@hardware_test(...) +@pytest.mark.parametrize("omni_runner", tts_server_params, indirect=True) +def test_text_to_audio_001(omni_runner, omni_runner_handler) -> None: + omni_runner_handler.send_audio_speech_request({ + "input": "...", + "task_type": "Base", + "ref_audio": REF_AUDIO_URL, + "ref_text": REF_TEXT, + }) +``` + +*Offline TTS e2e — **single-stage** (Coqui XTTS, MOSS-TTS-Nano; representative):* + +```python +@pytest.mark.advanced_model +@pytest.mark.tts +@hardware_test(...) +@pytest.mark.parametrize("omni_runner", tts_server_params, indirect=True) +def test_voice_clone_001(omni_runner_handler) -> None: + omni_runner_handler.send_single_stage_tts_request({ + "input": "...", + "language": "en", + "prompt_audio_path": REF_AUDIO_PATH, + "response_format": "wav", + "run_level": "advanced_model", + }) +``` + +*Online serving e2e — **Omni** (representative):* + +```python +@pytest.mark.core_model +@pytest.mark.omni +@hardware_test(...) +@pytest.mark.parametrize("omni_server", test_params, indirect=True) +def test_text_to_text_001(omni_server, openai_client) -> None: + request_config = {"model": omni_server.model, "messages": ..., "modalities": ["text"]} + openai_client.send_omni_request(request_config) +``` + +*Online serving e2e — **TTS** (representative):* + +```python +@pytest.mark.core_model +@pytest.mark.advanced_model +@pytest.mark.tts +@hardware_test(...) +@pytest.mark.parametrize("omni_server", tts_server_params, indirect=True) +def test_text_to_audio_001(omni_server, openai_client) -> None: + request_config = {"model": omni_server.model, "input": "...", "response_format": "wav", ...} + openai_client.send_audio_speech_request(request_config) +``` + +*Online serving e2e — **Diffusion** X2I (representative):* + +```python +@pytest.mark.core_model +@pytest.mark.diffusion +@pytest.mark.parametrize("omni_server", _get_default_case(MODEL), indirect=True) +def test_text_to_image_001(omni_server, openai_client) -> None: + openai_client.send_diffusion_request({...}) # chat completions + extra_body +``` + +*Online serving e2e — **Diffusion** X2V (representative):* + +```python +@pytest.mark.core_model +@pytest.mark.diffusion +def test_text_to_video_001(omni_server, openai_client) -> None: + openai_client.send_video_diffusion_request({"model": ..., "form_data": {...}}) # /v1/videos +``` + +*Documentation example tests:* follow the **Preferred Test Strategy** in [l4_doc_example_tests.inc.md](../../../docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md): dynamic extraction for offline READMEs; explicit copied client code for online pages until extraction is justified; use the documented **naming**, **output directory** (page folder + case id), and **skipping** rules (e.g. Gradio-only scripts). + +*Performance tests:* add or extend entries under `tests/perf/` (and JSON configs where the project uses them), with **explicit baselines** and the same marker/run-level pairing as the CI step that will execute them. + +**3. Cross-cutting rules** + +- Reuse existing fixtures for the chosen scenario; do not mix “online client” assumptions into offline `OmniRunner` tests without a clear reason. +- Avoid external network dependency in assertions unless the scenario is explicitly “online serving” or doc examples that require a model hub (then align with CI secrets/cache). +- Keep **one test function = one intent** (one modality combo, one endpoint contract, or one acceleration combo). +- **E2E test function layout**: **one case → one `test_` function** with a name that states what is validated (endpoint, size/`n`, server flag, or route). **Do not** merge multiple cases into a single test that branches on `request.node.callspec.id`, `param.id`, or `if case_id == ...`. Use `@pytest.mark.parametrize("omni_server", [...], indirect=True)` per function (usually **one** `OmniServerParams` per test). A loop inside one test is OK only when it serves **that function’s single intent** (e.g. three standard sizes in `test_*_sizes_256_512_1024`). +- **Runtime fixture scope** (`tests/helpers/fixtures/runtime.py`): default **`omni_server` / `omni_runner`** (module) + matching client/handler; use **`omni_server_function` / `omni_runner_function`** only when each `test_*` must start a fresh instance (see below). +- **L1 mocks**: never `unittest.mock`; use `mocker` or `monkeypatch` only (see below). +- **API calls (L2+ e2e, online and offline)**: **reuse** `send_*_request` in `tests/helpers/runtime.py` when it exists; **otherwise add** the helper in `runtime.py` first, then call it from the test. General `assert_*` inside `send_*_request`; special `assert_*` only in the test. See **Runtime send helpers** below — do not call `omni.generate`, raw HTTP, or SDK clients from `test_*.py`. +- **Response assertions**: reusable checks on API bodies / decoded media belong in **`tests/helpers/assertions.py`** — not as private `_assert_*` helpers inside `test_*.py` (see below). +- **Model-specific payloads stay in test modules** — per-model `MODEL`, deploy path, `REF_AUDIO_URL`, `get_prompt()`, `_build_request()`, and inline `request_config` dicts live in `test_{slug}.py` / `test_{slug}_expansion.py` (and offline/L1 siblings). **Do not** create `tests/helpers/{slug}.py` to deduplicate them; a little copy across files is preferred (see `test_glm_tts.py`, `test_cosyvoice3_tts_expansion.py`). `tests/helpers/` is repo-wide harness only (`mark`, `media`, `runtime`, `stage_config`, `assertions`, `fixtures/`). + +#### L1 unit test constraints (mocking) + +L1 tests (`core_model and cpu`, under `tests/diffusion/`, `tests/engine/`, `tests/model_executor/`, etc.) must follow the repo’s **pytest-mock** convention (`pytest-mock>=3.10.0` in `pyproject.toml` `[project.optional-dependencies] dev`). + +| Do | Don't | +|----|-------| +| `def test_foo(mocker):` + `mocker.patch(...)`, `mocker.spy(...)`, `mocker.Mock()`, `mocker.MagicMock()`, `mocker.AsyncMock()` | `from unittest.mock import patch, MagicMock, Mock, AsyncMock` | +| `def test_bar(monkeypatch):` + `monkeypatch.setattr` / `setenv` / `delenv` / `setitem` | `@patch(...)` decorator | +| Let `mocker` auto-stop patches after the test | `with patch(...):` / `patch.object(...)` context managers | +| Mirror neighboring L1 tests in the same directory | `unittest.mock.create_autospec` unless an existing file already documents an exception | + +**Rationale:** `mocker` ties patch lifecycle to pytest fixtures (no leaked patches across tests). `unittest.mock` decorators/context managers are easy to compose incorrectly with parametrized or async tests and are inconsistent with in-tree L1 style. + +**Minimal patterns:** + +```python +def test_returns_cached_config(mocker): + loader = mocker.patch( + "vllm_omni.foo.load_yaml", + return_value={"stages": []}, + ) + result = get_deploy_config("ci/foo.yaml") + assert result["stages"] == [] + loader.assert_called_once_with("ci/foo.yaml") + + +def test_skips_when_env_unset(monkeypatch): + monkeypatch.delenv("VLLM_OMNI_FEATURE", raising=False) + assert should_enable_feature() is False +``` + +E2E levels (L2+) generally **avoid mocks**; if a rare L2 stub is unavoidable, still prefer `mocker` over `unittest.mock` for consistency. + +#### Runtime fixtures — scope (`tests/helpers/fixtures/runtime.py`) + +vLLM-Omni e2e tests start a real **OmniServer** (online) or **OmniRunner** (offline). Pick scope by **how often the process must be recreated**, not by test level alone. + +| Scope | Online fixtures | Offline fixtures | When to use | +|-------|-----------------|------------------|-------------| +| **Module** (default) | `omni_server` → `openai_client` | `omni_runner` → `omni_runner_handler` | **Default for L2/L3/L4 expansion** — amortize model/server init across `test_*` in the same module. Same `OmniServerParams` / runner config can be reused by multiple tests. | +| **Function** | `omni_server_function` → `openai_client_function` | `omni_runner_function` → `omni_runner_handler_function` | **Only when required** — each `test_*` must get a **clean** server/runner (no shared engine/GPU state). Typical: `tests/dfx/reliability/`, sleep/wakeup, crash/restart, tests that mutate global server state. | + +**Rules:** + +1. **Default to module scope** (`omni_server` / `omni_runner`) unless the scenario **explicitly** needs a fresh instance per test function. +2. **Indirect parametrize name must match the fixture name**: `@pytest.mark.parametrize("omni_server", ...)` with `omni_server` + `openai_client`; `@pytest.mark.parametrize("omni_server_function", ...)` with `omni_server_function` + `openai_client_function`. Do not mix module fixture with function client (or vice versa). +3. Different `OmniServerParams` per `test_*` (e.g. default vs `--enable-cpu-offload`) is still OK with **module** `omni_server` — pytest parametrizes per test node; only switch to `_function` when isolation between tests matters, not merely because `server_args` differ. +4. One `test_*` with **many** server configs (expansion matrix) → single function + `@pytest.mark.parametrize("omni_server", [...], indirect=True)` + module `omni_server` (see in-tree `test_qwen_image_expansion.py`). + +```python +# Default — module-scoped server (L4 expansion) +@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=H100)], indirect=True) +def test_foo_images_generations_default_1024(omni_server, openai_client) -> None: + openai_client.send_images_generations_request({...}) + +# Function-scoped — reliability / per-test clean state only +@pytest.mark.parametrize( + "omni_server_function", + [pytest.param(OmniServerParams(model=MODEL), marks=H100)], + indirect=True, +) +def test_foo_sleep_wakeup_cycle(omni_server_function, openai_client_function) -> None: + openai_client_function.send_omni_sleep_http_request({...}) + openai_client_function.send_omni_wakeup_http_request({...}) +``` + +#### Runtime send helpers — online **and** offline (`tests/helpers/runtime.py`) + +**L2+ e2e (online serving and offline inference) must call APIs through `tests/helpers/runtime.py`.** Fixtures live in `tests/helpers/fixtures/runtime.py`; **send/assert implementation** lives in `tests/helpers/runtime.py` + `tests/helpers/assertions.py`. + +| Principle | Action | +|-----------|--------| +| **Reuse first** | Grep `runtime.py` for an existing **`send_*_request`** on `OpenAIClientHandler` (online) or `OmniRunnerHandler` (offline) that matches the endpoint / pipeline shape. | +| **Extend when close** | If an existing `send_*_request` almost fits (e.g. missing one optional field), extend it in `runtime.py` — do not fork logic in the test file. | +| **Add when missing** | No suitable helper → add **`send__request`** (high-level: call + general `assert_*`) or **`send___http_request`** (low-level HTTP for negative/dfx) in **`runtime.py` first**, then call it from tests. | +| **Test module owns payload only** | In `test_*.py`: `MODEL`, deploy path, vendored media, `get_prompt()`, and inline **`request_config` dicts** only. **No** `omni.generate(...)`, raw `requests.post`, OpenAI SDK calls, or `_collect_audio()` / `_process_output()` in e2e tests. | + +**Online** (`openai_client` from `omni_server`): `OpenAIClientHandler.send_*_request`. + +**Offline** (`omni_runner_handler` from `omni_runner`): `OmniRunnerHandler.send_*_request`. + +| Do | Don't | +|----|-------| +| Online: `openai_client.send_omni_request`, `send_diffusion_request`, `send_audio_speech_request`, `send_video_diffusion_request`, `send_images_generations_request`, … | `requests.post(f"{base_url}/v1/...", json=…)` or `client.chat.completions.create(...)` inside a test | +| Offline: `omni_runner_handler.send_omni_request`, `send_diffusion_request`, `send_audio_speech_request`, `send_single_stage_tts_request`, `send_single_stage_tts_batch_request`, … | `omni_runner.omni.generate(...)` + hand-rolled tensor/WAV extraction in `test_*.py` | +| Add missing `send_*` to **`runtime.py`** first; bundle general `assert_*` inside the send helper | A one-off `def _post_*` / `def _collect_audio` at the bottom of a test module | +| Mirror naming/style of neighboring `send_*` (docstring, `request_config` dict, optional `run_level`, `err_code` / `err_message` for negative cases) | Different parameter shapes per test file for the same endpoint | + +**Workflow when generating online/offline e2e tests:** + +1. Decide whether the needed check is **general** (every success call of this `send_*`) or **special** (one case / one parameter combo only). See **General vs special assert placement** below. +2. **Search `runtime.py`** for a matching **`send_*_request`**; if missing, add it (with general `assert_*` inside) before writing the test body. +3. In the test module: build `request_config` → call **`send_*_request` only** for the general contract; call **`assert_*` in the test file only** for special, case-specific checks (import from `assertions.py`). +4. Reserve **low-level** `send_*_http_request` for negative/dfx tests (`err_code` / `err_message`) — not for ordinary L2+ success-path e2e. +5. When wiring Buildkite `source_file_dependencies`, include **`tests/helpers/runtime.py`** and/or **`tests/helpers/assertions.py`** when new helpers were added. + +**Exceptions (document in the test docstring why):** models whose offline prompt path cannot go through existing handlers yet (e.g. `test_higgs_audio_v2.py`, `test_voxtral_tts.py` with custom tokenizer compose) may call `omni.generate` directly until a `send_*_request` is added to `runtime.py` — treat as **debt**, not the default for new TTS/diffusion/omni e2e. + +**General vs special assert placement:** + +| Kind | Definition | Where it lives | Test file calls | +|------|------------|----------------|-----------------| +| **General** | Default success contract for this endpoint/client method — e.g. HTTP 200, `data[]` shape, `n` count, `size` dimensions, decodable image/audio, bundled omni/diffusion fields | Implement in `assertions.py`, invoke **inside** `send_*_request` in `runtime.py` (after low-level send / SDK call, when `err_code` is not set) | **`send_*_request` only** — do **not** repeat the same `assert_*` | +| **Special** | Extra check tied to **this test case only** — e.g. seed byte-identical replay, accuracy/CLIP threshold, perf ceiling, model-specific optional field | Add or reuse `assert_*` in `assertions.py` (never inline in test) | `send_*_request` **then** `assert_(...)` **once** for that case | + +Changing `request_config` fields (`size`, `n`, `seed`, server flags) is **not** special validation — the general `assert_*` should read those from `request_config` inside `send_*`. + +**`send_*` ↔ `assert_*` pairing (online `OpenAIClientHandler`):** + +| High-level `send_*` (prefer in L2+ e2e) | Assert already invoked inside `runtime.py` | Low-level HTTP-only sibling | +|----------------------------------------|--------------------------------------------|-----------------------------| +| `send_omni_request` | `assert_omni_response` | `send_chat_completions_http_request` → `assert_http_error` only | +| `send_diffusion_request` | `assert_diffusion_response` | — | +| `send_audio_speech_request` | `assert_audio_speech_response` | `send_audio_speech_http_request` → `assert_http_error` only | +| `send_video_diffusion_request` | `assert_diffusion_response` | `send_videos_*_http_request` → `assert_http_error` only | +| `send_images_generations_request` *(add when needed)* | **`assert_images_generations_response`** *(general — inside send)* | `send_images_generations_http_request` → `assert_http_error` only | +| `send_images_edits_request` *(add when needed)* | **`assert_images_edits_response`** *(general — inside send)* | `send_images_edits_http_request` → `assert_http_error` only | + +**Rule:** Tests call high-level `send_*_request` for the general contract. **Never** call the same bundled `assert_*` again. Call an extra `assert_*` in the test **only** for special, case-specific validation. + +**Common `OpenAIClientHandler` entry points** (non-exhaustive — grep `runtime.py` before adding): + +| Area | High-level (SDK + assert) | Low-level HTTP (`*_http_request`) | +|------|---------------------------|-----------------------------------| +| Omni chat | `send_omni_request` | `send_chat_completions_http_request` | +| Diffusion T2I (chat route) | `send_diffusion_request` | — | +| Diffusion X2V | `send_video_diffusion_request` | `send_videos_create_http_request`, `send_video_content_http_request`, … | +| DALL-E T2I / edit | `send_images_generations_request`, `send_images_edits_request` | `send_images_generations_http_request`, `send_images_edits_http_request` | +| TTS (online) | `send_audio_speech_request` | `send_audio_speech_http_request`, `send_audio_generate_http_request`, … | +| Ops / meta | — | `send_health_http_request`, `send_models_http_request`, `send_omni_sleep_http_request`, … | + +**Common `OmniRunnerHandler` entry points** (offline — grep `runtime.py` before adding): + +| Area | High-level (`send_*_request` + assert) | Notes | +|------|----------------------------------------|-------| +| Omni multimodal | `send_omni_request` | `generate_multimodal` path | +| Diffusion offline | `send_diffusion_request` | chat-route / `OmniTextPrompt` offline | +| Qwen-style TTS | `send_audio_speech_request` | two-stage, `generate_multimodal` + `mm_processor_kwargs` | +| Single-stage TTS (Coqui XTTS, MOSS-TTS-Nano, …) | `send_single_stage_tts_request`, `send_single_stage_tts_batch_request` | `prompt` + `additional_information` + `omni.generate` — **do not** duplicate in tests | +| New model family | **Add** `send__request` here first | Then call from `tests/e2e/offline_inference/test_*.py` | + +L1 tests under `tests/entrypoints/` may use **FastAPI `TestClient`** or direct handler calls with mocks; they do **not** need `OpenAIClientHandler` / `OmniRunnerHandler`. **L2+ online and offline e2e** must use `runtime.py` `send_*` helpers. + +#### Invalid parameter validation (`tests/dfx/reliability/invalid_param_test/`) + +When the user asks for **invalid parameter validation**, **invalid request bodies**, **HTTP 4xx contract tests**, or any case that sends **malformed / out-of-range / mismatched** API payloads against a **live** server: + +1. **Do not** place these in `tests/e2e/online_serving/` or `*_expansion.py`. If already drafted there, **move** the `test_*` into the correct `invalid_param_test` script and delete the duplicate from e2e. +2. **Pick the script by HTTP route** (extend an existing file; add a new `test_invalid_.py` only when no in-tree script covers that route family): + +| API / area | Script | +|------------|--------| +| Omni chat completions, WebSocket video/realtime paths | `test_invalid_omni_chat.py` | +| `POST /v1/audio/speech`, stream, batch, voices | `test_invalid_audio_speech.py` | +| Audio diffusion endpoints | `test_invalid_audio_diffusion.py` | +| `POST /v1/images/generations` | `test_invalid_image_generation.py` | +| `POST /v1/images/edits` | `test_invalid_image_editing.py` | +| `POST/GET/DELETE /v1/videos*` | `test_invalid_video_generation.py` | +| Sleep / wakeup / server control | `test_invalid_server_control.py` | + +3. **Match in-tree style** in the chosen script: + +| Element | Convention | +|---------|------------| +| **Module markers** | `pytestmark = [pytest.mark.slow, pytest.mark.]` | +| **Server fixture** | `_PARAMS` / `_QWEN3_TTS_SPEECH`-style list of `pytest.param(OmniServerParams(...), id="...", marks=hardware_marks(...))`; `@pytest.mark.parametrize("omni_server", _PARAMS, indirect=True)` | +| **Hardware** | `hardware_marks(res={"cuda": "H100"})` for heavy diffusion/omni/video; `hardware_marks(res={"cuda": "L4"})` for smaller TTS models (must match weekly `-m "slow and L4"` step) | +| **HTTP client** | **Low-level** `openai_client.send_*_http_request({..., "err_code": 400, "err_message": (...)} )` — **not** `send_*_request` (success path) | +| **Case shape** | Prefer **one `test_*` per route family** + `@pytest.mark.parametrize("body_spec, err_message", [...])` with stable `id=` per case; or dedicated `test__malformed_json` when not parametrized | +| **Body helpers** | `_minimal__json(omni_server)` / `_minimal_*_form_data()`; merge overrides with `body.update(body_spec)` | +| **Known gaps** | `pytest.mark.skip(reason="…#3649")` as `_SKIP_ISSUE_3649` when server validation is not yet strict (mirror neighboring cases) | +| **Sections** | Route banner comments (`# ─── POST /v1/images/generations ───`) like existing files | +| **Shared fixtures** | Reuse `tests/dfx/reliability/invalid_param_test/conftest.py` (`tiny_png_bytes`, env defaults) | + +4. **Adding a new model** to invalid-param coverage: append a `pytest.param(OmniServerParams(model="...", stage_config_path=..., server_args=...), ...)` entry to the script’s `_PARAMS` list — do **not** create `test_invalid_.py` unless the route is new. + +5. **CI — `.buildkite/test-weekly.yml` only** (not `test-ready.yml` / `test-merge.yml` / `test-nightly.yml`): + +| Weekly step | Command | When your cases run | +|-------------|---------|---------------------| +| **Invalid parameters Test · H100** | `pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and H100"` | Diffusion / omni / video invalid-param tests with `H100` hardware mark | +| **Invalid parameters Test · L4** | `pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and L4"` | TTS / lighter models marked `L4` | + +- **Trigger:** `build.env("WEEKLY") == "1"` or PR label `weekly-test`. +- **Default:** extending an existing `invalid_param_test` script needs **no YAML edit** — the weekly steps already sweep the whole directory. +- **Edit YAML only** when adding a **new hardware queue**, a **new top-level script** that must run in isolation, or a **model-specific weekly shard** (mirror neighboring reliability steps). +- Weekly steps **do not** use `source_file_dependencies`. + +**Example — append to `test_invalid_image_generation.py`:** + +```python +@pytest.mark.parametrize( + "body_spec, err_message", + [ + pytest.param({"seed": -1}, ("seed", "greater_than_equal", "0"), id="seed_negative"), + ], +) +@pytest.mark.parametrize("omni_server", _PARAMS, indirect=True) +def test_images_generations_invalid_requests( + omni_server: OmniServer, + openai_client: OpenAIClientHandler, + body_spec: dict[str, object], + err_message: str | tuple[str, ...], +) -> None: + body = _minimal_images_gen_json(omni_server) + body.update(body_spec) + openai_client.send_images_generations_http_request( + {"json": body, "timeout": 300, "err_code": 400, "err_message": err_message} + ) +``` + +See [references/test-routing.md](references/test-routing.md) **Invalid parameter / weekly CI**. + +#### Assertion helpers (`tests/helpers/assertions.py`) + +**Do not** add module-local helpers such as `_assert_images_generations_payload` or `_send_and_assert_*` in e2e test files. Response/media validation belongs in **`tests/helpers/assertions.py`**, grouped by **category**: + +| Category | Existing anchors | When to extend vs add | +|----------|------------------|------------------------| +| **Image** (chat diffusion + DALL-E JSON) | `assert_image_diffusion_response`, `assert_image_valid` | DALL-E `/v1/images/generations` JSON → add **`assert_images_generations_response`** beside image helpers (reuses `assert_image_valid`); do not fork decode logic into tests | +| **Video** | `assert_video_diffusion_response`, `assert_video_valid` | Extend these for new video contracts | +| **Audio** | `assert_audio_diffusion_response`, `assert_audio_speech_response`, `assert_audio_valid` | Extend for new TTS/audio endpoints | +| **Omni multimodal** | `assert_omni_response` | Extend for new modality combos | +| **HTTP errors** | `assert_http_error`, `assert_err_message_in_text` | Used inside low-level `send_*_http_request` and negative tests | + +| Do | Don't | +|----|-------| +| Put **general** `assert_*` inside `send_*_request` in `runtime.py`; tests call `send_*_request` only | Call `assert_diffusion_response` after `send_diffusion_request` (general assert already bundled) | +| Put **special** `assert_*` in `assertions.py` and call it **in the test** after `send_*_request` when that case needs extra checks | Put general decode/count/size logic in the test because “this case uses `n=4`” (that belongs in general assert reading `request_config`) | +| Extend the matching category function, or add `assert__response` next to its category | `_send_and_assert_*` or PIL/base64 loops in `test_*.py` | +| Use low-level `send_*_http_request` in dfx/negative tests (`err_code`) | Use low-level HTTP send + manual general assert in every L2+ expansion file | + +**Workflow when generating tests:** + +1. Implement or reuse **general** `assert_*` in `assertions.py` (by category). +2. Wire it into **`send_*_request`** in `runtime.py` if not already bundled. +3. In the test: `request_config` → `send_*_request`. Add a separate `assert_*` import/call **only** for special case validation. +4. Update `source_file_dependencies` when `runtime.py` or `assertions.py` changes. + +**Example — DALL-E `/v1/images/generations`:** + +```python +# tests/helpers/assertions.py — general contract (image category) +def assert_images_generations_response(resp_body: dict, request_config: dict, *, run_level: str | None = None) -> None: + ... # data[], n, size → assert_image_valid + +# tests/helpers/runtime.py — general assert INSIDE send +def send_images_generations_request(self, request_config: dict[str, Any], ...) -> list[HttpResponse]: + responses = self.send_images_generations_http_request(request_config, ...) + if request_config.get("err_code") is None: + body = responses[0].json_body + assert isinstance(body, dict) + assert_images_generations_response(body, request_config, run_level=self.run_level) + return responses + +# tests/e2e/.../test_foo_expansion.py — one case per test_* (no case_id branching) +@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=SINGLE_L4)], indirect=True) +def test_foo_images_generations_default_1024(omni_server, openai_client) -> None: + openai_client.send_images_generations_request({"json": body, "timeout": 300}) + +@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=SINGLE_L4)], indirect=True) +def test_foo_images_generations_sizes_256_512_1024(omni_server, openai_client) -> None: + for size in ("256x256", "512x512", "1024x1024"): + openai_client.send_images_generations_request({"json": {**body, "size": size}, "timeout": 300}) + +# chat diffusion — separate test; send_diffusion_request bundles assert_diffusion_response +@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=SINGLE_L4)], indirect=True) +def test_foo_chat_completions_t2i_fallback(omni_server, openai_client) -> None: + openai_client.send_diffusion_request({...}) +``` + +L1 tests under `tests/entrypoints/` may keep **minimal** asserts next to the handler under test when validating a single branch; still prefer `assertions.py` when the same JSON/media check appears in more than one file or level (e.g. L1 protocol test + L4 e2e). + +#### Omni Test Writing Guidance (L1-L4 Layering) + +When the goal is a general Omni/multimodal test case, prioritize mapping the test to the correct purpose, directory, and resource assumptions aligned with the layers (see [CI_5levels.md](../../../docs/contributing/ci/CI_5levels.md)): + +- **L1**: Unit/logic validation on CPU (`core_model and cpu`). Cover input validation, branches, and exception paths (`tests//test_*.py`). **Mock with `mocker` / `monkeypatch` only — not `unittest.mock`.** +- **L2**: Basic e2e (online/offline basic scenarios). Prefer dummy/lightweight models to validate the end-to-end request-to-output-structure/streaming chain (typically `tests/e2e/online_serving/` and `tests/e2e/offline_inference/`). +- **L3/L4**: Important integration, performance, and accuracy validation. L4 emphasizes “full functional scenarios + performance/stress + runnable doc examples” (typically `*_expansion.py` plus related expansion cases). + +Also keep markers consistent with the run level: use `core_model` for L1/L2 and `advanced_model` for L3/L4, and pair with `--run-level` to select the intended CI strategy. + +#### Diffusion Test Writing Guidance (L4 Coverage Combinations) + +When the task involves diffusion models/features, organize L4 test cases following [`#1832`](https://github.com/vllm-project/vllm-omni/issues/1832): combine multiple diffusion features into as few test cases as possible to fit limited CI GPU resources. + +Implementation strategy: + +1. If “full L4 is too heavy”: first provide a **reduced local validation case in L1/L2** (so the key assertion and the contract fix point are covered deterministically). +2. Then provide **CI/nightly-ready L3/L4 high-marked cases** (e.g. `advanced_model`) to broaden coverage under resource constraints. + +#### L4 nightly sub-pillars — Function vs Accuracy vs Perf + +**L4 is not only `test_*_expansion.py`.** In `test-nightly.yml`, each model-type group typically runs **separate jobs**: + +| Sub-pillar | Nightly step label pattern | What you add | Typical paths | +|------------|----------------------------|--------------|---------------| +| **Function** | `· Function Test with …` | E2e expansion / feature matrix | `tests/e2e/online_serving/test__expansion.py` (or offline) | +| **Accuracy** | `· Accuracy Test` | Quality / similarity vs baseline | `tests/e2e/accuracy/test_*.py` | +| **Perf** | `· Perf Test · ` | Throughput / latency / memory benchmarks | `tests/dfx/perf/tests/test__vllm_omni.json` + runner script | +| **Doc** (optional) | `· Doc Test` | Runnable doc examples | `tests/examples/*/test_text_to_image.py`, … | + +**Default when the user asks for “L4 functional cases”:** deliver **Function** pillar only (`*_expansion.py` + Function Test shard in `test-nightly.yml`). **Do not** silently add Perf or Accuracy unless the user also asks for **performance** / **benchmark** / **accuracy** / **full L4** / **full L4 coverage**. + +**When the user explicitly asks for L4 perf (or full L4 including perf):** + +1. **Do not** put throughput/latency assertions inside `test_*_expansion.py` — perf uses the **dfx benchmark harness**, not `omni_server` e2e fixtures. +2. Add or extend a **JSON config** under `tests/dfx/perf/tests/` (mirror in-tree names: `test_qwen_image_vllm_omni.json`, `test_bagel_vllm_omni.json`). +3. Each JSON entry: `test_name`, `server_params` (`model`, `serve_args`), `benchmark_params[]` with explicit **`baseline`** (`throughput_qps`, `latency_mean`, `peak_memory_mb_mean`, …). +4. Run locally via the matching script (model type → runner): + +| Model type | Runner script | Example config | +|------------|---------------|----------------| +| **Diffusion X2I/X2V** | `tests/dfx/perf/scripts/run_diffusion_benchmark.py` | `tests/dfx/perf/tests/test_qwen_image_vllm_omni.json` | +| **TTS** | `tests/dfx/perf/scripts/run_benchmark.py` | `tests/dfx/perf/tests/test_tts.json` (shared) or `test_{slug}.json` (dedicated; use when the model must not join the shared nightly matrix before integration — e.g. `test_voxcpm2.json`, `test_coqui_tts.json`) | +| **Omni** | (see existing Omni Perf steps — `run_benchmark.py` / dedicated omni configs in-tree) | `tests/dfx/perf/tests/test_omni*.json` | + +5. Wire **`test-nightly.yml` · Perf Test** step for that model (separate from Function Test): export `DIFFUSION_BENCHMARK_DIR` / `BENCHMARK_DIR`, run pytest on the script with `--test-config-file`, **upload artifacts** (`buildkite-agent artifact upload`), often **multi-GPU H100** for diffusion perf. +6. **One benchmark scenario → one `test_name` block** in JSON (same spirit as one `test_*` per function case). Combine server `serve_args` + workload in one entry; use multiple `benchmark_params` rows for size/step sweeps under the same server config. + +**Perf local command (diffusion example):** + +```bash +cd tests +export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results +export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN +pytest -s -v dfx/perf/scripts/run_diffusion_benchmark.py \ + --test-config-file dfx/perf/tests/test__vllm_omni.json +``` + +**Clarify in the test plan** which L4 pillars you deliver: `Function only` | `Function + Perf` | `Function + Accuracy + Perf`. + +See [references/test-routing.md](references/test-routing.md) **L4 nightly pillars** for Buildkite step patterns. + +### Step 5: Wire Buildkite (when CI must run the new test) + +If the test is not already collected by an existing pipeline command (for example, L1 tests marked `core_model and cpu` are already covered by the **Simple Unit Test** step in ready/merge), update the appropriate pipeline under `.buildkite/`: + +| Test level | Edit this file | Typical trigger / intent | +|------------|----------------|---------------------------| +| **L1** and **L2** | [`.buildkite/test-ready.yml`](../../../.buildkite/test-ready.yml) | PR **ready** label; L1 CPU + L2 GPU/basic e2e (steps labeled **Omni ·**, **TTS ·**, **Diffusion ·**) | +| **L3** | [`.buildkite/test-merge.yml`](../../../.buildkite/test-merge.yml) | Post-merge; `advanced_model` integration per model type | +| **L4** | [`.buildkite/test-nightly.yml`](../../../.buildkite/test-nightly.yml) | Nightly; grouped by model type (see below) | +| **Invalid param / reliability (weekly)** | [`.buildkite/test-weekly.yml`](../../../.buildkite/test-weekly.yml) | Weekly; `tests/dfx/reliability/invalid_param_test/` — **not** L1–L4 e2e pipelines | + +**Level-specific Buildkite delivery (follow the user’s requested level):** + +| User asks for | Document / edit **only** | Do **not** add by default | +|---------------|---------------------------|---------------------------| +| **L4** (`*_expansion.py`, `full_model`) | `test-nightly.yml` — append file to the matching nightly shard (X2I / X2V / Omni / TTS) **or** note it is already collected by an existing `-m` / `-k` sweep | `test-merge.yml` (L3 / `advanced_model`) or `test-ready.yml` (L2) | +| **L3** | `test-merge.yml` with `advanced_model` + `source_file_dependencies` | `test-nightly.yml` unless user also wants nightly | +| **L2** | `test-ready.yml` with `core_model` + `source_file_dependencies` | merge / nightly pipelines | +| **Invalid param** | `test-weekly.yml` — **Invalid parameters Test** group (`-m "slow and H100"` / `-m "slow and L4"`). Usually **no YAML edit** when appending cases to existing `invalid_param_test` scripts | `test-ready.yml`, `test-merge.yml`, `test-nightly.yml` | + +`test-nightly.yml` shards use **explicit pytest file paths** in `commands` (and PR labels like `diffusion-x2iat-test`); they generally **do not** use `source_file_dependencies`. Only suggest merge/ready YAML when the requested level is L3/L2 (or the user explicitly asks for multi-level CI). + +**`test-nightly.yml` top-level groups** (each uses `full_model` + type marker in pytest `-m`): + +| Group | Type marker | Notes | +|-------|-------------|--------| +| **Omni Model Test** | `omni` | Function / doc / accuracy / **perf** for Qwen-Omni family | +| **TTS Model Test** | `tts` | `-m "full_model and L4 and tts"` (and H100 variants when enabled); **Perf Test** uses `run_benchmark.py` + `test_tts.json` | +| **Diffusion X2I(&A&T) Model Test** | `diffusion` | Image/audio/text diffusion; explicit file shards for **Function**; **Perf Test · <Model>** uses `run_diffusion_benchmark.py` + JSON config | +| **Diffusion X2V Model Test** | `diffusion` | Video-only (Wan, HunyuanVideo, …); PR label `diffusion-x2v-test`; separate **Perf** steps when in-tree | + +Guidelines when editing YAML: + +- **Match markers and `--run-level`** to the test: L1/L2 in `test-ready.yml` use `core_model` + `omni` / `tts` / `diffusion` as appropriate; L3 merge uses `advanced_model`; L4 nightly uses `full_model` with the same type marker. +- **Diffusion L4**: add x2i/x2a/x2t expansion files to an **X2I(&A&T)** step; add x2v expansion files to **X2V** — never rely on a broad `tests/e2e/` sweep alone (shards exist to save GPU and avoid wrong queue). +- **`source_file_dependencies` (required for new E2E jobs in `test-ready.yml` and `test-merge.yml`)**: every step under the **`:card_index_dividers: E2E Test`** group **must** declare `source_file_dependencies` so Buildkite only schedules the job when relevant paths change. Omitting it breaks path-based CI skipping for GPU e2e jobs. +- **Reuse or extend an existing step** when the new test shares the same marker expression, queue, and timeout; otherwise add a new `steps:` entry with the correct `agents.queue`, `timeout_in_minutes`, and docker/kubernetes plugin blocks consistent with neighboring jobs. +- **Job timeout — use `timeout_in_minutes` on the step, not `timeout` before pytest**: when adding or documenting a new E2E job in `test-ready.yml` / `test-merge.yml`, set `timeout_in_minutes` on the step and run `pytest` directly in `commands`. **Do not** wrap pytest in `timeout 40m bash -c "..."` / `timeout 20m bash -c "..."`; Buildkite already enforces the step deadline via `timeout_in_minutes`. +- **Never omit `plugins` in skill examples or PR YAML snippets**: H100 E2E steps on `mithril-h100-pool` use the full `kubernetes` block (`resources`, `volumeMounts`, `env` with `HF_TOKEN` secret, `nodeSelector: gpu-h100-sxm`, `devshm` + `hf-cache` volumes). L4/docker steps on `gpu_1_queue` use the full `docker#v5.2.0` block (`shm-size`, `HF_HOME`, volumes). Do not write `# ... plugins unchanged` — paste the complete block from a sibling step. +- **Platform forks** (e.g. AMD-ready / AMD-merge) live alongside these files; apply the same level → file mapping for those pipelines when the test is platform-specific. + +#### `source_file_dependencies` for E2E Test jobs (`test-ready.yml`, `test-merge.yml`) + +When adding or splitting a step inside the **E2E Test** group, **always** include `source_file_dependencies` on that step. List paths the job actually depends on — at minimum: + +1. **Test module(s)** — every `tests/e2e/...` file the `pytest` command runs (online and offline if both are in one step). +2. **Model implementation** — `vllm_omni/model_executor/models//` and/or `vllm_omni/diffusion/models//` as applicable. +3. **Stage / input plumbing** (omni & multi-stage TTS) — `vllm_omni/model_executor/stage_input_processors/.py` when present. +4. **Deploy / stage YAML** — `vllm_omni/deploy/.yaml` or `vllm_omni/deploy/ci/.yaml` referenced by the test’s `OmniServerParams` / `get_deploy_config_path(...)`. +5. **Test helpers** (when changed in the same PR) — `tests/helpers/runtime.py` for new `send_*` client methods; `tests/helpers/assertions.py` for new shared `assert_*` helpers. + +Use **repo-relative paths** (no leading `./`). Prefer **directory** entries (`.../models/bagel/`) when the whole package matters; use **file** entries for single test modules and YAML configs. Mirror a neighboring step for the same model family (e.g. **Diffusion · Bagel Test** in `test-ready.yml`). + +**Example** (L2 ready — diffusion online smoke, H100 kubernetes): + +```yaml +- label: "Diffusion · Bagel Test" + source_file_dependencies: + - tests/e2e/online_serving/test_bagel.py + - vllm_omni/model_executor/models/bagel/ + - vllm_omni/diffusion/models/bagel/ + - vllm_omni/model_executor/stage_input_processors/bagel.py + - vllm_omni/deploy/bagel.yaml + timeout_in_minutes: 40 + commands: + - | + export VLLM_IMAGE_FETCH_TIMEOUT=60 + pytest -s -v tests/e2e/online_serving/test_bagel.py -m 'core_model' --run-level 'core_model' + agents: + queue: "mithril-h100-pool" + plugins: + - kubernetes: + podSpec: + containers: + - image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT + resources: + limits: + nvidia.com/gpu: 1 + volumeMounts: + - name: devshm + mountPath: /dev/shm + - name: hf-cache + mountPath: /root/.cache/huggingface + env: + - name: HF_HOME + value: /root/.cache/huggingface + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: token + nodeSelector: + node.kubernetes.io/instance-type: gpu-h100-sxm + volumes: + - name: devshm + emptyDir: + medium: Memory + - name: hf-cache + hostPath: + path: /mnt/hf-cache + type: DirectoryOrCreate +``` + +**Example** (L3 merge — TTS online + offline, L4 docker): + +```yaml +- label: "TTS · Qwen3-TTS Base Test" + source_file_dependencies: + - tests/e2e/online_serving/test_qwen3_tts_base.py + - tests/e2e/offline_inference/test_qwen3_tts_base.py + - vllm_omni/model_executor/models/qwen3_tts/ + - vllm_omni/model_executor/stage_input_processors/qwen3_tts.py + - vllm_omni/deploy/qwen3_tts.yaml + timeout_in_minutes: 20 + commands: + - | + export VLLM_LOGGING_LEVEL=DEBUG + export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 + pytest -s -v tests/e2e/online_serving/test_qwen3_tts_base.py tests/e2e/offline_inference/test_qwen3_tts_base.py -m 'advanced_model and cuda' --run-level 'advanced_model' + agents: + queue: "gpu_1_queue" # g6.4xlarge instance on AWS, has 1 L4 GPU + plugins: + - docker#v5.2.0: + image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT + always-pull: true + propagate-environment: true + shm-size: "8gb" + environment: + - "HF_HOME=/fsx/hf_cache" + volumes: + - "/fsx/hf_cache:/fsx/hf_cache" +``` + +**Example** (L3 merge — diffusion online + offline, H100 kubernetes): + +```yaml +- label: "Diffusion · Qwen Image Test" + source_file_dependencies: + - tests/e2e/online_serving/test_qwen_image.py + - tests/e2e/offline_inference/test_qwen_image.py + - vllm_omni/diffusion/models/qwen_image/ + timeout_in_minutes: 40 + commands: + - | + export VLLM_IMAGE_FETCH_TIMEOUT=60 + pytest -s -v \ + tests/e2e/online_serving/test_qwen_image.py \ + tests/e2e/offline_inference/test_qwen_image.py \ + -m 'advanced_model and cuda' --run-level 'advanced_model' + agents: + queue: "mithril-h100-pool" + plugins: + - kubernetes: + podSpec: + containers: + - image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT + resources: + limits: + nvidia.com/gpu: 1 + volumeMounts: + - name: devshm + mountPath: /dev/shm + - name: hf-cache + mountPath: /root/.cache/huggingface + env: + - name: HF_HOME + value: /root/.cache/huggingface + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: token + nodeSelector: + node.kubernetes.io/instance-type: gpu-h100-sxm + volumes: + - name: devshm + emptyDir: + medium: Memory + - name: hf-cache + hostPath: + path: /mnt/hf-cache + type: DirectoryOrCreate +``` + +When extending an **existing** E2E step to run an additional test file, **append** that file (and any new model/deploy paths) to `source_file_dependencies` on the same step. When authoring a PR, mention the dependency list in the **Buildkite change** section of your test plan. + +#### L3 / L4: validating Buildkite from a feature branch (root `pipeline.yml`) + +Production behavior is driven by [`.buildkite/pipeline.yml`](../../../.buildkite/pipeline.yml): it builds the CI image, then uploads **L2** (`test-ready.yml` on non-`main`), **L3** (`test-merge.yml` when `build.branch == "main"`), and **L4** (`test-nightly.yml` when `build.env("NIGHTLY") == "1"`). + +When you add or debug **L3** or **L4** tests and need those child pipelines to run **off `main`** or **without** `NIGHTLY=1`, apply **temporary** edits (revert before merge): + +1. **Point the upload at the right file (optional shortcut)** + In the **Upload Ready Pipeline** step, the command is `buildkite-agent pipeline upload .buildkite/test-ready.yml`. For a one-off run you may change that path to `.buildkite/test-merge.yml` (L3) or `.buildkite/test-nightly.yml` (L4), and adjust the step’s `if:` so it does not conflict with the other upload steps (avoid double-uploading unless intentional). + +2. **L3 — merge pipeline** + The gate `if: build.branch == "main"` lives on the **Upload Merge Pipeline** step in `pipeline.yml` (not inside `test-merge.yml`). **Comment out that `if` line** so `test-merge.yml` is uploaded on your feature branch. Alternatively rely on step (1) instead of the dedicated merge upload step. + +3. **L4 — nightly pipeline** + **Comment out** `if: build.env("NIGHTLY") == "1"` on the **Upload Nightly Pipeline** step in `pipeline.yml` so the nightly definition is uploaded without setting the env var. + Child steps in [`test-nightly.yml`](../../../.buildkite/test-nightly.yml) often repeat `if: build.env("NIGHTLY") == "1"`; **comment out those lines on the steps you need to run**, otherwise they will still be skipped after upload. + +### Step 6: Run Tests + +Pick one command path: + +- **Quick local regression (preferred first)**: single file or `file.py::test_name` +- **CI-like level run**: marker expression + `--run-level` + +Full templates live in [references/test-routing.md](references/test-routing.md). **After authoring tests, always emit concrete commands** (see **Output Format**). + +**Typical copy-paste examples** (run from repo `tests/` directory; requires matching vLLM/vllm-omni install and hardware): + +| Area | Example | +|------|---------| +| **Omni offline L2** | `pytest -s -v e2e/offline_inference/test_qwen2_5_omni.py -m "core_model and omni and not cpu" --run-level=core_model` | +| **Omni online L2** | `pytest -s -v e2e/online_serving/test_qwen3_omni.py -m "core_model and omni" --run-level=core_model` | +| **TTS online L2** | `pytest -s -v e2e/online_serving/test_qwen3_tts_base.py -m "core_model and tts" --run-level=core_model` | +| **Diffusion X2I L2 online** | `pytest -s -v e2e/online_serving/test_qwen_image.py -m "core_model and diffusion" --run-level=core_model` | +| **Diffusion X2V L2 online** | `pytest -s -v e2e/online_serving/test_wan22_t2v.py -m "core_model and diffusion" --run-level=core_model` | +| **Diffusion X2I L4 nightly** | `pytest -s -v e2e/online_serving/test_qwen_image_expansion.py -m "full_model and diffusion and H100" --run-level=full_model` | +| **Diffusion X2I L4 perf (nightly)** | `pytest -s -v dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file dfx/perf/tests/test_qwen_image_vllm_omni.json` | +| **Diffusion X2V L4 nightly** | `pytest -s -v e2e/online_serving/test_wan22_expansion.py -m "full_model and cuda" --run-level=full_model` | +| **Invalid param (weekly H100)** | `pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and H100"` | +| **Invalid param (weekly L4)** | `pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and L4"` | +| **L1 CPU** | `pytest -s -v -m "core_model and cpu"` | + +**Prerequisites to mention when relevant**: GPU model (e.g. L4 vs H100), `HF_HOME` / token for hub weights, module-level `skipif` (NPU/XPU-only gaps), and whether CI already collects the path (e.g. `test_*_expansion.py` glob in `test-nightly.yml`). + +### Step 7: Validate Result Quality + +Before finishing: + +- Is the new assertion directly tied to the bug/feature contract? +- Are API calls made via **`runtime.py` `send_*_request`** (not inline HTTP/SDK in the test file)? +- Is the **general** `assert_*` bundled inside `send_*_request` (not duplicated in the test)? +- Is each scenario a **separate `test_*` function** (no `case_id` / `if param.id` branching across cases)? +- **Fixture scope**: default **`omni_server` / `omni_runner`** (module); **`_function`** variants only when each test must restart the instance? +- Is the test deterministic (no fragile timing/network coupling)? +- Is runtime appropriate for the selected level? +- Are markers and `--run-level` consistent? +- **Invalid-param cases** in `tests/dfx/reliability/invalid_param_test/` (not e2e), using `send_*_http_request` + `err_code`, with `pytest.mark.slow` and correct `H100` / `L4` hardware mark? + +## Output Format + +When completing a request, return: + +1. **Test plan** (level, markers, file target, **L4 pillar(s)** if applicable: Function / Accuracy / Perf / Doc; **or** **Invalid param** pillar with target `invalid_param_test/test_invalid_.py`; module basename for e2e: `test_{slug}.py` / `test_{slug}_expansion.py`) +2. **Generated/updated test file(s)** — model-specific constants and request payloads stay **inside** those test modules (**never** `tests/helpers/{slug}.py`). Extend **`tests/helpers/runtime.py`** / **`tests/helpers/assertions.py`** only when new **repo-wide** `send_*` or shared assert helpers are required (do not leave ad-hoc HTTP or `_assert_*` in test modules). **If L4 Perf was requested**, also list **`tests/dfx/perf/tests/*.json`**. **If invalid-param cases were requested**, list the **`invalid_param_test/` script** and parametrized `test_*` / new `id=` rows — **not** e2e paths. +3. **Buildkite change** — **match the requested level and pillar**: + - **L4 Function**: `test-nightly.yml` **Function Test** shard (file list / note existing sweep). **No** `test-merge.yml` unless user also asked for L3. + - **L4 Perf**: `test-nightly.yml` **Perf Test · <Model>** step (new step or extend commands); include env exports + artifact upload pattern from a sibling perf job. + - **L4 Accuracy**: `test-nightly.yml` **Accuracy Test** step under the same model-type group. + - **L3**: `test-merge.yml` + full `source_file_dependencies` + `agents` + `plugins`. + - **L2**: `test-ready.yml` + same E2E block requirements. + - **Invalid param**: `test-weekly.yml` — note **Invalid parameters Test · H100/L4** group; usually **no YAML change** when only extending existing scripts. + If the user asked only for **L4 functional cases**, state explicitly that **Perf / Accuracy / Invalid param** were not included (offer to add if needed). If the plan mixes **success e2e** and **invalid param**, split deliverables across e2e vs `invalid_param_test/` and call out both CI files. +4. **Run commands (required)** — always include, in fenced `bash` blocks: + - **Local — whole file**: `cd tests` then `pytest -s -v …` + - **Local — single test** (optional but preferred when the change is one function): `pytest -s -v path::test_func …` + - **CI-like** (when not L1 CPU): the same **marker + `--run-level`** pairing the level uses (see [references/test-routing.md](references/test-routing.md) and **Step 6** table above) + - **Prerequisites** (one line): e.g. “Requires CUDA L4 + weights cached locally”; “Requires H100 + `HF_TOKEN`”; “Syntax-only validation when vllm is not installed locally” +5. **Result summary** (pass/fail if you executed tests; if not executed, state that explicitly and what the user should run) + +## Additional Resources + +- Marker and command routing: [references/test-routing.md](references/test-routing.md) +- CI pipelines (vllm-omni): [test-ready.yml](../../../.buildkite/test-ready.yml) (L1/L2), [test-merge.yml](../../../.buildkite/test-merge.yml) (L3), [test-nightly.yml](../../../.buildkite/test-nightly.yml) (L4), [test-weekly.yml](../../../.buildkite/test-weekly.yml) (invalid param / reliability) +- L4 documentation example tests (naming, extraction vs copied scripts, output dirs, skips): [docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md](../../../docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md) — see also [PR #1910](https://github.com/vllm-project/vllm-omni/pull/1910) diff --git a/.claude/skills/vllm-omni-test/references/test-routing.md b/.claude/skills/vllm-omni-test/references/test-routing.md new file mode 100644 index 000000000..445a78399 --- /dev/null +++ b/.claude/skills/vllm-omni-test/references/test-routing.md @@ -0,0 +1,372 @@ +# Test Routing Reference + +Use this reference to map testing goals to levels, markers, and runnable commands. + +**Repo paths** (`.buildkite/`, `docs/contributing/…`): link with repo-relative paths from this file (e.g. `../../../../.buildkite/test-ready.yml`, `../../../../docs/contributing/ci/CI_5levels.md`). + +## Model-centric e2e filename convention (L2–L4) + +When generating a **new** test module tied to a **specific model** under `tests/e2e/offline_inference/` or `tests/e2e/online_serving/`: + +| Level | Pattern | Example (`Qwen/Qwen2.5-Omni-7B`) | +|-------|---------|----------------------------------| +| **L2**, **L3** | `test_{lowercase_model_slug}.py` | `test_qwen2_5_omni.py` | +| **L4** | `test_{lowercase_model_slug}_expansion.py` | `test_qwen2_5_omni_expansion.py` | + +**Slug rules for `{lowercase_model_slug}`** (canonical: [SKILL.md](../SKILL.md) § *Naming: generated test module files*): + +1. Start from the HuggingFace-style id (e.g. `Qwen/Qwen2.5-Omni-7B`), but **do not** put the org into the filename: use the **repo segment only** (`Qwen2.5-Omni-7B`), not `Qwen_...` / `qwen_qwen2_5_...`. +2. **Lowercase**; replace `.`, `-`, and whitespace with a single `_` (e.g. `Qwen2.5-Omni-7B` → `qwen2_5_omni`). **Omit** trailing size tokens such as `7b` / `30b` in the basename when a single file covers that model line in the directory (matches `test_qwen2_5_omni.py` in-tree). +3. If two checkpoints in the same folder need separate modules, add a **minimal** disambiguator (e.g. `_7b` vs `_3b`) **only then**. +4. **L1** unit tests are **not** bound to this pattern; use `tests//test_.py` as today. + +## Model type markers (`omni` / `tts` / `diffusion`) + +Every model-centric e2e test must declare **exactly one** type marker: + +| Marker | Model family | Typical APIs | +|--------|--------------|--------------| +| `pytest.mark.omni` | Multimodal LLM (Qwen-Omni, …) | `send_omni_request`, `omni_runner`, stage YAML | +| `pytest.mark.tts` | TTS / speech synthesis | `send_audio_speech_request`, `/v1/audio/speech` | +| `pytest.mark.diffusion` | Diffusion generative models | `send_diffusion_request`, `send_video_diffusion_request`, `send_images_generations_http_request`, `OmniDiffusionSamplingParams` | + +**Diffusion** uses one marker for all output modalities. Nightly CI splits diffusion by **group**, not by extra markers: + +| Nightly group (`test-nightly.yml`) | Diffusion scope | Examples | +|-----------------------------------|-----------------|----------| +| **Diffusion X2I(&A&T) Model Test** | x2**i** / x2**a** / x2**t** — image, audio, text (non-video) | Qwen-Image*, BAGEL, FLUX, SD3, Z-Image, LongCat, DreamZero | +| **Diffusion X2V Model Test** | x2**v** — video only | Wan2.2, HunyuanVideo 1.5, Wan VACE | + +PR labels for selective nightly runs: `diffusion-x2iat-test`, `diffusion-x2v-test` (plus `nightly-test` / `NIGHTLY=1`). + +## Level and Marker Mapping + +| Goal | Suggested Level | Marker baseline | Typical location | +|------|------------------|-----------------|------------------| +| Unit logic, regression on pure code path | L1 | `core_model and cpu` | `tests//test_*.py` | + +### L1 unit tests — mocking (`mocker`, not `unittest.mock`) + +L1 modules run in **Simple Test** (`test-ready.yml` / `test-merge.yml`: `-m 'core_model and cpu'`). When isolation requires doubles: + +- **Use** the **`mocker`** fixture (`pytest-mock`): `mocker.patch`, `mocker.spy`, `mocker.Mock`, `mocker.MagicMock`, `mocker.AsyncMock`. +- **Use** **`monkeypatch`** for env vars and simple `setattr` / `delattr` without mock objects. +- **Do not use** `unittest.mock` — no `from unittest.mock import ...`, no `@patch`, no `with patch(...)`. + +```python +# Good (L1) +def test_handler(mocker): + mocker.patch("vllm_omni.pkg.fn", return_value=0) + +# Bad (L1) +from unittest.mock import patch + +@patch("vllm_omni.pkg.fn") +def test_handler(mock_fn): ... +``` + +E2E (L2+) should not rely on mocks unless documenting a rare exception; prefer real (or lightweight) execution paths. +| Basic integration/e2e | L2 | `core_model` + **one of** `omni` / `tts` / `diffusion` | `tests/e2e/...` | +| Advanced integration | L3 | `advanced_model` + type marker | `tests/e2e/...` | +| Full function / nightly | L4 | `full_model` (nightly) + type marker | **Function:** `tests/e2e/*_expansion.py`; **Perf:** `tests/dfx/perf/tests/*.json`; **Accuracy:** `tests/e2e/accuracy/` | +| Invalid HTTP / param validation | Weekly (dfx) | `pytest.mark.slow` + type marker + `H100` or `L4` | `tests/dfx/reliability/invalid_param_test/test_invalid_*.py` | + +## Marker Selection Rules + +1. **Level** (pick one): + - `core_model` — L1/L2 (`test-ready.yml`) + - `advanced_model` — L3 (`test-merge.yml`) + - `full_model` — L4 nightly (`test-nightly.yml`); some expansion tests still carry both `advanced_model` and `full_model` during migration +2. **Model type** (pick one): `omni`, `tts`, or `diffusion` +3. **Cross-cutting** when relevant: `parallel`, `cache`, `example`, `benchmark` +4. **Hardware**: `cpu`, `cuda`, `rocm`, `npu`, `L4`, `H100`, `distributed_cuda`, … +5. Multi-card: `@hardware_test(...)` in `tests/helpers/mark.py` + +## Command Templates + +### Quick local checks + +```bash +cd tests +pytest -s -v test_xxxx.py +``` + +### L1 + +```bash +cd tests +pytest -s -v -m "core_model and cpu" +``` + +### L2 + +```bash +cd tests +pytest -s -v -m "core_model and not cpu" --run-level=core_model +``` + +### L3 (merge) + +```bash +cd tests +pytest -s -v -m "advanced_model" --run-level=advanced_model +``` + +### L4 (nightly) + +**Function** (e2e expansion — default when user asks for “L4 functional cases”): + +```bash +cd tests +pytest -s -v e2e/online_serving/test_qwen_image_expansion.py -m "full_model and diffusion and H100" --run-level=full_model +``` + +**Perf** (dfx benchmark harness — separate pillar; not `omni_server` fixtures): + +```bash +cd tests +export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results +export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN +pytest -s -v dfx/perf/scripts/run_diffusion_benchmark.py \ + --test-config-file dfx/perf/tests/test_qwen_image_vllm_omni.json +``` + +Broad marker sweep (when no explicit file shard): + +```bash +cd tests +pytest -s -v -m "full_model" --run-level=full_model +``` + +## L4 nightly pillars (Function / Accuracy / Perf) + +Nightly **L4** for a model is often **multiple jobs** in `test-nightly.yml`, not a single pytest module: + +| Pillar | User says | Deliver | Nightly step | +|--------|-----------|---------|--------------| +| **Function** | “L4 functional cases” / “functional” / `*_expansion.py` | `tests/e2e/.../test__expansion.py` | `· Function Test with H100/L4` | +| **Perf** | “performance” / “perf” / “benchmark” / “full L4” | `tests/dfx/perf/tests/test__vllm_omni.json` + runner | `· Perf Test · ` | +| **Accuracy** | “accuracy” / “similarity” | `tests/e2e/accuracy/test_*.py` | `· Accuracy Test` | + +**Rule:** “L4 functional cases” → **Function only** by default. State that Perf/Accuracy are separate pillars; add them only when requested. + +**Diffusion perf JSON** (`tests/dfx/perf/tests/test__vllm_omni.json`): array of objects with `test_name`, `server_params` (`model`, `serve_args`), `benchmark_params[]`, and per-workload **`baseline`** (`throughput_qps`, `latency_mean`, `peak_memory_mb_mean`). Copy structure from `test_qwen_image_vllm_omni.json` in-tree. + +**Diffusion perf nightly step** (under **Diffusion X2I(&A&T)**; copy full `kubernetes` plugins from `· Perf Test · Qwen-Image`): + +```yaml + - label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · " + key: nightly-diffusion-x2iat-performance- + timeout_in_minutes: 180 + commands: + - export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results + - export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN + - export CACHE_DIT_VERSION=1.3.0 + - | + set +e + pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test__vllm_omni.json + EXIT=$$? + buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json" + buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log" + exit $$EXIT + agents: + queue: "mithril-h100-pool" + plugins: + # … full kubernetes block (often multi-GPU) — copy from Perf Test · Qwen-Image +``` + +| Model type | Perf runner | Config example | +|------------|-------------|----------------| +| Diffusion X2I/X2V | `dfx/perf/scripts/run_diffusion_benchmark.py` | `test_qwen_image_vllm_omni.json` | +| TTS | `dfx/perf/scripts/run_benchmark.py` | `test_tts.json` (`BENCHMARK_DIR`) | +| Omni | `run_benchmark.py` / omni configs in-tree | `test_omni*.json` | + +Do **not** put throughput/latency baselines inside `test_*_expansion.py` — that belongs in the dfx perf JSON + nightly Perf job. + +## Invalid parameter validation (weekly / dfx) + +When the user’s test plan includes **invalid parameter validation**, **invalid request bodies**, or **HTTP 4xx** validation against a live server: + +1. **Do not** author these in `tests/e2e/online_serving/test_*.py` or `*_expansion.py`. **Move** any drafted `test_*` into `tests/dfx/reliability/invalid_param_test/`. +2. **Pick script by route** (extend in-tree file): + +| Route family | Script | +|--------------|--------| +| Omni chat / WS video·realtime | `test_invalid_omni_chat.py` | +| `/v1/audio/speech` (+ stream / batch / voices) | `test_invalid_audio_speech.py` | +| Audio diffusion | `test_invalid_audio_diffusion.py` | +| `/v1/images/generations` | `test_invalid_image_generation.py` | +| `/v1/images/edits` | `test_invalid_image_editing.py` | +| `/v1/videos*` | `test_invalid_video_generation.py` | +| Sleep / wakeup / server control | `test_invalid_server_control.py` | + +3. **Style:** `pytestmark = [pytest.mark.slow, pytest.mark.]`; `_PARAMS` + `hardware_marks`; `send_*_http_request` with `err_code` + `err_message`; parametrized `body_spec` rows with `id=`; `_minimal_*_json()` helpers; `_SKIP_ISSUE_3649` when tracked in [#3649](https://github.com/vllm-project/vllm-omni/issues/3649). +4. **CI:** [`.buildkite/test-weekly.yml`](../../../../.buildkite/test-weekly.yml) group **Reliability Test - Invalid parameters Test** — **not** ready/merge/nightly. + +```bash +# Weekly H100 (diffusion / omni / video invalid-param) +cd tests +pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and H100" + +# Weekly L4 (e.g. Qwen3-TTS speech invalid-param) +pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and L4" + +# Single script / case +pytest -s -v dfx/reliability/invalid_param_test/test_invalid_image_generation.py::test_images_generations_invalid_requests -m "slow and H100" +``` + +**Trigger:** `WEEKLY=1` or PR label `weekly-test`. Appending cases to existing scripts usually needs **no YAML edit** (directory sweep). No `source_file_dependencies` on weekly steps. + +### Platform-targeted examples + +```bash +cd tests +pytest -s -v -m "core_model and distributed_cuda and L4" --run-level=core_model +``` + +### Concrete e2e paths (common in-tree) + +Paths are relative to `tests/` after `cd tests`. + +| Scenario | Example command | +|----------|------------------| +| **Omni** offline L2 | `pytest -s -v e2e/offline_inference/test_qwen2_5_omni.py -m "core_model and omni and not cpu" --run-level=core_model` | +| **Omni** offline L2 — one test | `pytest -s -v e2e/offline_inference/test_qwen2_5_omni.py::test_text_to_text -m "core_model and omni and not cpu" --run-level=core_model` | +| **Omni** online L2 | `pytest -s -v e2e/online_serving/test_qwen3_omni.py -m "core_model and omni" --run-level=core_model` | +| **Omni** offline L2 (Qwen3.5-9B VL) | `pytest -s -v e2e/offline_inference/test_qwen3_5_9b.py -m "core_model and omni and not cpu" --run-level=core_model` | +| **TTS** online L2 | `pytest -s -v e2e/online_serving/test_qwen3_tts_base.py -m "core_model and tts" --run-level=core_model` | +| **TTS** online L2 — one test | `pytest -s -v e2e/online_serving/test_qwen3_tts_base.py::test_text_to_audio_001 -m "core_model and tts" --run-level=core_model` | +| **Diffusion X2I** offline L2 | `pytest -s -v e2e/offline_inference/test_t2i_model.py -m "core_model and diffusion and not cpu" --run-level=core_model` | +| **Diffusion X2I** L2 online smoke | `pytest -s -v e2e/online_serving/test_qwen_image.py e2e/online_serving/test_bagel.py -m "core_model and diffusion" --run-level=core_model` | +| **Diffusion X2V** L2 online smoke | `pytest -s -v e2e/online_serving/test_wan22_t2v.py -m "core_model and diffusion" --run-level=core_model` | +| **Diffusion X2I** L4 online expansion (H100) | `pytest -s -v e2e/online_serving/test_qwen_image_expansion.py -m "full_model and diffusion and H100" --run-level=full_model` | +| **Diffusion X2I** L4 perf (nightly) | `pytest -s -v dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file dfx/perf/tests/test_qwen_image_vllm_omni.json` | +| **Diffusion X2I** L4 online expansion (Edit) | `pytest -s -v e2e/online_serving/test_qwen_image_edit_expansion.py -m "full_model and diffusion and H100" --run-level=full_model` | +| **Diffusion X2V** L4 nightly | `pytest -s -v e2e/online_serving/test_wan22_expansion.py -m "full_model and cuda" --run-level=full_model` | +| **Invalid param** weekly H100 | `pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and H100"` | +| **Invalid param** weekly L4 | `pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and L4"` | + +### Agent / author completion checklist + +When adding or modifying tests, do not stop at “where the file lives” — also deliver: + +1. **Local**: `cd tests` + `pytest -s -v ` (and `path::test_func` when a single case is enough). +2. **CI-like**: marker string + `--run-level` matching the pipeline (`core_model` / `advanced_model` / `full_model`). +3. **L1 mocks**: `mocker` / `monkeypatch` only; no `unittest.mock`. +4. **API client + assert placement**: **General** validation → implement in `assertions.py`, call **inside** `send_*_request` in `runtime.py`; tests call **`send_*_request` only**. **Special** case validation → `assert_*` in `assertions.py`, called **in the test** after `send_*_request`. Low-level `send_*_http_request` is for negative/dfx tests (`err_code`), not ordinary L2+ success e2e. +5. **Shared assertions**: logic in **`tests/helpers/assertions.py`**; general checks inside `send_*_request`; special checks only in the test. No `_assert_*` in `test_*.py`. +6. **One case → one `test_*`**: function name reflects what is validated; no `if case_id == ...` mega-test merging multiple scenarios. +7. **Fixture scope**: default **`omni_server` + `openai_client`** / **`omni_runner` + `omni_runner_handler`** (module). Use **`omni_server_function` + `openai_client_function`** / **`omni_runner_function` + `omni_runner_handler_function`** only when each `test_*` must spawn a fresh instance. Parametrize name must match fixture (`omni_server` vs `omni_server_function`). +8. **Type marker**: `omni`, `tts`, or `diffusion` on every model e2e module. +9. **Diffusion L4 Function**: wire `*_expansion.py` into **X2I(&A&T)** or **X2V** **Function Test** in **`test-nightly.yml` only** — do not add `test-merge.yml` unless the user also requested L3. +10. **Diffusion L4 Perf** (only when requested): add `tests/dfx/perf/tests/test__vllm_omni.json` + **Perf Test · <Model>** step (artifact upload); not part of “L4 functional cases” by default. +11. **E2E Buildkite (L2/L3 only)**: `test-ready.yml` / `test-merge.yml` steps need **`source_file_dependencies`** + full **`agents` + `plugins`**. **L4 nightly** uses explicit file lists or perf scripts in `test-nightly.yml` (no merge job). +12. **Invalid param**: cases in **`tests/dfx/reliability/invalid_param_test/`** (route-matching script), `send_*_http_request` + `err_code`, `pytest.mark.slow` + `H100`/`L4`; CI = **`test-weekly.yml`** only — do not put in e2e or nightly. +13. **Prerequisites**: GPU tier, HF cache/token, and any module `skipif` / platform-only YAML. + +## Buildkite pipeline mapping + +| Level | Repo file | Model-type grouping | +|-------|-----------|---------------------| +| L1, L2 | [`.buildkite/test-ready.yml`](../../../../.buildkite/test-ready.yml) | Steps prefixed **Omni ·**, **TTS ·**, **Diffusion ·** under **E2E Test** — **`source_file_dependencies` required** | +| L3 | [`.buildkite/test-merge.yml`](../../../../.buildkite/test-merge.yml) | Per-model E2E steps; `-m "advanced_model and …"` — **`source_file_dependencies` required** | +| L4 | [`.buildkite/test-nightly.yml`](../../../../.buildkite/test-nightly.yml) | **Omni / TTS / Diffusion X2I(&A&T) / X2V** — each group may have **Function**, **Accuracy**, **Perf**, **Doc** steps | +| Invalid param (weekly) | [`.buildkite/test-weekly.yml`](../../../../.buildkite/test-weekly.yml) | **Invalid parameters Test · H100** / **· L4** — sweeps `tests/dfx/reliability/invalid_param_test/` | + +### `source_file_dependencies` (E2E Test only — ready & merge) + +Required on each step inside the **E2E Test** group in `test-ready.yml` and `test-merge.yml`. Typical entries: + +| Category | Path pattern | +|----------|----------------| +| Test module | `tests/e2e/online_serving/test_.py`, `tests/e2e/offline_inference/test_.py` | +| API client helpers | `tests/helpers/runtime.py` (when the e2e job uses newly added `send_*` methods) | +| Shared assert helpers | `tests/helpers/assertions.py` (when the e2e job uses newly added `assert_*` helpers) | +| AR / omni model | `vllm_omni/model_executor/models//` | +| Diffusion model | `vllm_omni/diffusion/models//` | +| Stage processor | `vllm_omni/model_executor/stage_input_processors/.py` | +| Deploy config | `vllm_omni/deploy/.yaml` or `vllm_omni/deploy/ci/.yaml` | + +Copy the dependency block from the closest in-tree E2E step for the same model family. Update the list whenever the pytest command or server YAML changes. + +### E2E `agents` + `plugins` blocks (required — do not truncate) + +When generating or documenting E2E steps, **always include the complete `plugins` section**. Set **`timeout_in_minutes`** on the step for the job deadline; run **`pytest` directly** in `commands` (no `timeout 40m bash -c` wrapper around pytest). + +Common patterns in `test-ready.yml` / `test-merge.yml`: + +| Queue | Plugin | Required fields | +|-------|--------|-----------------| +| `mithril-h100-pool` | `kubernetes` | `resources.limits.nvidia.com/gpu`, `volumeMounts` (`devshm`, `hf-cache`), `env` (`HF_HOME`, `HF_TOKEN` via `secretKeyRef`), `nodeSelector: gpu-h100-sxm`, `volumes` (`devshm` emptyDir, `hf-cache` hostPath `/mnt/hf-cache`) | +| `gpu_1_queue` / `gpu_4_queue` | `docker#v5.2.0` | `image`, `always-pull`, `propagate-environment`, `shm-size: "8gb"`, `environment: HF_HOME`, `volumes: /fsx/hf_cache` (add `HF_TOKEN` when the step needs hub access) | + +**H100 kubernetes template** (copy from **Diffusion · Bagel Test** / **Diffusion · Qwen Image Test** in-tree): + +```yaml + agents: + queue: "mithril-h100-pool" + plugins: + - kubernetes: + podSpec: + containers: + - image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT + resources: + limits: + nvidia.com/gpu: 1 + volumeMounts: + - name: devshm + mountPath: /dev/shm + - name: hf-cache + mountPath: /root/.cache/huggingface + env: + - name: HF_HOME + value: /root/.cache/huggingface + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: token + nodeSelector: + node.kubernetes.io/instance-type: gpu-h100-sxm + volumes: + - name: devshm + emptyDir: + medium: Memory + - name: hf-cache + hostPath: + path: /mnt/hf-cache + type: DirectoryOrCreate +``` + +**L4 docker template** (copy from **TTS · Qwen3-TTS Base Test** in `test-merge.yml`): + +```yaml + agents: + queue: "gpu_1_queue" + plugins: + - docker#v5.2.0: + image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT + always-pull: true + propagate-environment: true + shm-size: "8gb" + environment: + - "HF_HOME=/fsx/hf_cache" + volumes: + - "/fsx/hf_cache:/fsx/hf_cache" +``` + +When extending an existing step (e.g. add `tests/e2e/offline_inference/test_qwen_image.py` to **Diffusion · Qwen Image Test**), update `source_file_dependencies` and `commands` only; **keep the existing `agents` + `plugins` block unchanged** unless the hardware tier changes. + +The root [`.buildkite/pipeline.yml`](../../../../.buildkite/pipeline.yml) decides **which** child file is uploaded. To run **L3** on a feature branch, comment out `if: build.branch == "main"` on the merge upload step. To run **L4** without `NIGHTLY=1`, comment out the nightly upload step’s `if: build.env("NIGHTLY") == "1"` and the same `if` on relevant steps inside `test-nightly.yml`. Revert such edits before merging. + +Platform-specific pipelines (e.g. AMD) follow the same level → file pairing under `.buildkite/`. + +## Diffusion RFC (#1832) Alignment Tips + +For **X2I(&A&T)** coverage planning: + +- Prioritize high-value feature combinations with minimal case count. +- Split into lightweight L1/L2 validation plus L4 `*_expansion.py` under the **X2I** nightly shards. +- **X2V** models (Wan, HunyuanVideo) stay in the **X2V** group — do not merge into X2I sweeps. + +If hardware is insufficient, provide an executable reduced case plus a deferred full CI/nightly plan. diff --git a/docs/contributing/ci/CI_5levels.md b/docs/contributing/ci/CI_5levels.md index 01310606e..adf29bfa5 100644 --- a/docs/contributing/ci/CI_5levels.md +++ b/docs/contributing/ci/CI_5levels.md @@ -11,6 +11,10 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari Level Scope & Focus + Model Coverage Strategy + Feature Coverage Strategy + Interface Coverage Strategy + Tags Time Cost Test Dir Doc @@ -24,6 +28,10 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari Contribution Guideline & PR checklist / / + / + / + / + / .github/PULL_REQUEST_TEMPLATE.md Test Style (PR Checklist) / / @@ -32,6 +40,10 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari CI Failure Description / / + / + / + / + / CI Failures / / @@ -39,6 +51,10 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari L1
(Unit & Logic) Unit tests for components like entrypoints, models + / + / + / + core_model and cpu <15min /tests/{component_name}/test_xxx @@ -50,10 +66,18 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari L2
(E2E across models & GPU-required UT) - Online & Offline (basic deployment scenarios):
dummy, normal inference function (output format, stream), some instance startup UT + Online (basic deployment scenarios):
dummy, normal inference function (output format, stream), some instance startup UT + High-priority models + online basic scenarios + request success validation + High-priority features (using random lightweight models) + High-priority interfaces (using random lightweight models) + core_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion + Model tests:
/tests/e2e/online_serving/test_{model_name}.py
- /tests/e2e/offline_inference/test_{model_name}.py + Feature tests:
+ /tests/{component_name}/test_xxx
+ Interface tests:
+ /tests/entrypoints/test_xxx Chapter 1
@@ -65,10 +89,19 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari L3
(Important Perf & Integration & Accuracy) Online & Offline (multiple deployment scenarios):
real model, normal inference function, normal accuracy + High/medium-priority models with real weights + online/offline key scenarios + basic accuracy validation + Medium-priority features (using random lightweight models) + Medium-priority interfaces (using random lightweight models) + advanced_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion <30min + Model tests:
/tests/e2e/online_serving/test_{model_name}.py
- /tests/e2e/offline_inference/test_{model_name}.py + /tests/e2e/offline_inference/test_{model_name}.py
+ Feature tests:
+ /tests/{component_name}/test_xxx
+ Interface tests:
+ /tests/entrypoints/test_xxx Chapter 2
@@ -79,18 +112,27 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari L4
(Perf & Integration & Accuracy) - Online & Offline: full functional scenarios + performance test + doc test + Online: full functional scenarios + performance test + doc test + accuracy test + High-priority models: function, performance, accuracy, and doc testing
Medium/low-priority models: function and doc testing + Low-priority features (using random lightweight models) + Low-priority interfaces (using random lightweight models) + full_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion <3 hour - Full Function:
+ Model tests:
/tests/e2e/online_serving/test_{model_name}_expansion.py
- /tests/e2e/offline_inference/test_{model_name}_expansion.py
+ Feature tests:
+ /tests/{component_name}/test_xxx
+ Interface tests:
+ /tests/entrypoints/test_xxx
Performance:
/tests/dfx/perf/tests/test_qwen_omni.json (Omni), test_tts.json (TTS),
and /tests/dfx/perf/tests/test_{diffusion_model}_vllm_omni.json (Diffusion)
Doc Test:
tests/example/online_serving/test_{model_name}.py
- tests/example/offline_inference/test_{model_name}.py + tests/example/offline_inference/test_{model_name}.py
+ Accuracy Test:
+ /tests/e2e/accuracy/test_{model_name}.py Chapter 3
@@ -101,7 +143,11 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari L5
(Stability & Reliability) - Online & Offline: long-term stability test + reliability test + Online: long-term stability test + reliability test + Long-term stability and reliability testing for high-priority models + / + Invalid-parameter validation for high-priority interfaces + slow and hardware_test(H100, L4, etc.) and omni/tts/diffusion Depends on reality Stability: