[SKILLS] Add vllm-omni-test agent skill for CI-aligned test generation (#4434)
Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: [Your Name] <[Your Email]>
This commit is contained in:
wangyu authored and GitHub committed
1 parent
c782746133
commit
dff8cda844
6 files changed
+1419
-17
No files matched your search
@@ -288,9 +288,36 @@ Required updates:
|
||||
5. `examples/offline_inference/xxx/README.md` — offline example docs
|
||||
6. `examples/online_serve/xxx/README.md` — online serve docs
|
||||
|
||||
### Step 8: Add E2E Tests (Recommended)
|
||||
### Step 8: Add E2E Tests
|
||||
|
||||
Create `tests/e2e/online_serving/test_your_model_expansion.py`.
|
||||
**Follow the [vllm-omni-test skill](../vllm-omni-test/SKILL.md)** for markers, file naming, Buildkite wiring, and run commands. Also read [l4_functionality_tests.inc.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/test_examples/l4_functionality_tests.inc.md) and [CI_5levels.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/CI_5levels.md).
|
||||
|
||||
Classify the model's **CI priority** first:
|
||||
|
||||
| Priority | Required test levels | Files & markers |
|
||||
|----------|---------------------|-----------------|
|
||||
| **High** (listed in [#1832](https://github.com/vllm-project/vllm-omni/issues/1832) or on the diffusion hot path) | **L1** · **L2** online · **L3** online + offline · **L4** feature + performance | See table below |
|
||||
| **Medium** (*normal priority* in L4 docs) | **L3** online + offline · **L4** feature only | Fewer L4 parametrized rows |
|
||||
| **Low** | **L4** feature only | One or two `*_expansion.py` cases |
|
||||
|
||||
**Per-level deliverables (diffusion / `pytest.mark.diffusion`):**
|
||||
|
||||
| Level | Location | Marker | CI pipeline | Notes |
|
||||
|-------|----------|--------|-------------|-------|
|
||||
| **L1** | `tests/diffusion/models/{slug}/`, `tests/diffusion/cache/`, transformer unit tests | `core_model` + `cpu` | `test-ready.yml` | Weight remap, `_sp_plan`, cache enabler registration, shape contracts |
|
||||
| **L2** | `tests/e2e/online_serving/test_{slug}.py` (and offline if the category is offline-first) | **`core_model` + `advanced_model`** (both on baseline smoke) + `diffusion` + `@hardware_test` / `hardware_marks` | `test-ready.yml` | Default deploy smoke — minimal `num_inference_steps`, single prompt |
|
||||
| **L3** | `tests/e2e/online_serving/test_{slug}.py` **and** `tests/e2e/offline_inference/test_{slug}.py` when offline matters | Baseline smoke: **`core_model` + `advanced_model`**; heavier cases: `advanced_model` only (+ `diffusion`) | `test-merge.yml` **or** merged into nightly diffusion function job | Real weights, streaming/API paths, LoRA/offload smoke |
|
||||
| **L4** | `tests/e2e/online_serving/test_{slug}_expansion.py` (+ offline expansion if needed) | `full_model` + `diffusion` | `test-nightly.yml` (X2I / X2V / X2A function groups) | Feature combos per [#1832](https://github.com/vllm-project/vllm-omni/issues/1832); perf → `tests/dfx/perf/tests/test_{model}_vllm_omni.json` |
|
||||
|
||||
**L2 & L3 online — same file, dual marks on the baseline smoke:** The **first / simplest** case in `test_{slug}.py` (default deploy, minimal steps, single prompt) should carry **both** `@pytest.mark.core_model` and `@pytest.mark.advanced_model` on the **same** function so L2 (`test-ready.yml`) and L3 (`test-merge.yml`) share one smoke test. Heavier deploy variants or API paths in the same file use **`advanced_model` only**. When L3 moves to nightly, migrate those heavier cases into `test_{slug}_expansion.py` with `full_model` and remove the dedicated `test-merge.yml` job (see `test_longcat_image_expansion.py`, `test_qwen_image_expansion.py`).
|
||||
|
||||
**L4 design (high priority):** Combine multiple supported features (Cache-DiT, TP, USP, CFG, HSDP, CPU offload, quantization) into **few parametrized** `OmniServerParams` rows so each feature appears in at least one case without exploding GPU jobs. Shard single-GPU vs multi-GPU cases across the nightly X2I/X2V function steps (1-GPU vs `distributed_cuda`).
|
||||
|
||||
**L4 design (medium / low):** One or two parametrized rows covering the best quality/perf trade-off; skip perf JSON unless the model is high priority.
|
||||
|
||||
**Reference implementations:** `tests/e2e/online_serving/test_qwen_image_edit_expansion.py`, `tests/e2e/online_serving/test_longcat_image_expansion.py`, `tests/e2e/online_serving/test_hunyuan_video_15_expansion.py`.
|
||||
|
||||
**Keep model-specific code inside test modules — not `tests/helpers/{slug}.py`:** deploy constants, prompts, sampling dicts, and inline `request_config` / `form_data` belong in each `test_{slug}.py` and `test_{slug}_expansion.py`. Do not add per-model files under `tests/helpers/`; reuse only repo-wide harness (`mark`, `media`, `runtime`, `stage_config`, `assertions`). **L2+ online/offline e2e:** reuse or add `send_*_request` in `tests/helpers/runtime.py` — tests call the handler, not raw `omni.generate` / HTTP. See [vllm-omni-test skill](../vllm-omni-test/SKILL.md) § **Runtime send helpers**.
|
||||
|
||||
### Step 9: Add Cache-DiT Acceleration
|
||||
|
||||
@@ -568,6 +595,7 @@ See the [Profiling Single-Stage Diffusion](../../../docs/contributing/profiling.
|
||||
|
||||
## Reference Files
|
||||
|
||||
- [vllm-omni-test skill](../vllm-omni-test/SKILL.md) — L1–L4 markers, naming, Buildkite wiring, run commands
|
||||
- [Transformer Adaptation](references/transformer-adaptation.md) — porting transformers from diffusers
|
||||
- [Custom Model Patterns](references/custom-model-patterns.md) — patterns for non-diffusers models
|
||||
- [Parallelism Patterns](references/parallelism-patterns.md) — TP, SP/USP, CFG parallel, HSDP implementation details
|
||||
|
||||
@@ -305,13 +305,72 @@ def build_voice_clone_prompt(ref_audio_path: str, text: str, codec) -> list:
|
||||
]
|
||||
```
|
||||
|
||||
### Test Case Writing (CI Levels)
|
||||
|
||||
**Follow the [vllm-omni-test skill](../vllm-omni-test/SKILL.md)** for markers, file naming (`test_{slug}.py` / `test_{slug}_expansion.py`), Buildkite wiring, and copy-paste run commands. Also read [CI_5levels.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/CI_5levels.md) and [tests_style.md](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/ci/tests_style.md).
|
||||
|
||||
Classify the model's **CI priority** first (high / medium / low). High-priority TTS models are typically those on the integration hot path or listed in tracking issues such as [#1832](https://github.com/vllm-project/vllm-omni/issues/1832); medium and low tiers cover the long tail. When unsure, ask the reviewer which tier applies.
|
||||
|
||||
| Priority | Required test levels | Files & markers |
|
||||
|----------|---------------------|-----------------|
|
||||
| **High** | **L1** unit/logic · **L2** online smoke · **L3** online + offline integration · **L4** feature + performance | See table below |
|
||||
| **Medium** | **L3** online + offline · **L4** feature only | Skip dedicated L1/L2 unless fixing a logic bug |
|
||||
| **Low** | **L4** feature only | One or two `*_expansion.py` parametrized cases |
|
||||
|
||||
**Per-level deliverables (TTS / `pytest.mark.tts`):**
|
||||
|
||||
| Level | Location | Marker | CI pipeline | Notes |
|
||||
|-------|----------|--------|-------------|-------|
|
||||
| **L1** | `tests/model_executor/…`, `tests/entrypoints/openai_api/…`, stage-processor tests | `core_model` + `cpu` | `test-ready.yml` | Prompt assembly, async_chunk helpers, `serving_speech` validation — no GPU |
|
||||
| **L2** | `tests/e2e/online_serving/test_{slug}.py` | **`core_model` + `advanced_model`** (both on baseline smoke) + `tts` + `@hardware_test(...)` | `test-ready.yml` (`ready` label) | Default deploy smoke: single `/v1/audio/speech` or offline `OmniRunner` path |
|
||||
| **L3** | `tests/e2e/online_serving/test_{slug}.py` **and** `tests/e2e/offline_inference/test_{slug}.py` | Baseline smoke: **`core_model` + `advanced_model`**; heavier cases: `advanced_model` only (+ `tts`) | `test-merge.yml` **or** merged into nightly TTS function job | Streaming, voice clone, batch/queue, async_chunk |
|
||||
| **L4** | `tests/e2e/online_serving/test_{slug}_expansion.py`, optional offline expansion | `full_model` + `tts` | `test-nightly.yml` (`:full_moon: TTS · Function Test with L4`) | Feature matrix; perf → `tests/dfx/perf/tests/test_tts.json` |
|
||||
|
||||
**L2 & L3 online — same file, dual marks on the baseline smoke:** The **first / simplest** case in `test_{slug}.py` (default deploy, single non-streaming `/v1/audio/speech` or equivalent offline path) should carry **both** `@pytest.mark.core_model` **and** `@pytest.mark.advanced_model` on the **same** function so it runs in L2 (`test-ready.yml`, `--run-level core_model`, basic validation) and L3 (`test-merge.yml`, `--run-level advanced_model`, deeper validation) without duplicating the test. In-tree examples: `test_voxcpm2_tts.py::test_text_to_audio_001`, `test_qwen3_tts_customvoice.py::test_text_to_audio_001`.
|
||||
|
||||
Heavier scenarios in the same file use **`advanced_model` only** (streaming, extra languages, concurrency, async_chunk, batch). Example: `test_voice_clone_en_streaming_001` → `advanced_model` only. When migrating L3 to nightly, move those heavier cases into `test_{slug}_expansion.py` with `full_model` and drop the dedicated merge job (see `test_ming_tts_expansion.py`, `test_glm_tts_expansion.py`).
|
||||
|
||||
```python
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.tts
|
||||
@hardware_test(res={"cuda": "L4"}, num_cards=1)
|
||||
@pytest.mark.parametrize("omni_server", tts_server_params, indirect=True)
|
||||
def test_voice_clone_en_non_streaming_001(omni_server, openai_client) -> None:
|
||||
openai_client.send_audio_speech_request({...})
|
||||
```
|
||||
|
||||
**L4 consolidation:** Prefer parametrized `OmniServerParams` rows (`default`, `async_chunk`, feature flags) in one expansion module rather than many merge-only files ([#1832](https://github.com/vllm-project/vllm-omni/issues/1832)).
|
||||
|
||||
**L4 performance (high-priority models):** Add latency / throughput / stress rows in `tests/dfx/perf/tests/test_tts.json`, or a dedicated `tests/dfx/perf/tests/test_{slug}.json` when the model must not join the shared nightly server matrix before integration lands (see VoxCPM2 / Coqui XTTS pattern). Register the model in `benchmarks/tts/model_configs.yaml` for local `bench_tts.py`. Wire a **separate** `test-nightly.yml` Perf Test step when the JSON is not merged into `test_tts.json` yet.
|
||||
|
||||
**Keep model-specific code inside test modules — not `tests/helpers/{slug}.py`:**
|
||||
|
||||
- Put `MODEL`, deploy path, vendored `REF_AUDIO_URL`, `get_prompt()`, and inline `request_config` dicts **in each** `test_{slug}.py`, `test_{slug}_expansion.py`, offline `test_{slug}.py`, and L1 `test_{slug}_*.py` as needed.
|
||||
- **Do not** add `tests/helpers/{slug}.py` (or `tests/helpers/{model_name}.py`) to deduplicate constants or request builders across those files. A little duplication is intentional; follow in-tree references such as `tests/e2e/online_serving/test_glm_tts.py` and `tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py`.
|
||||
- `tests/helpers/` is for **repo-wide** harness code only (`mark.py`, `media.py`, `runtime.py`, `stage_config.py`, `assertions.py`, `fixtures/`). Import those; do not extend the tree with per-model modules.
|
||||
|
||||
**Runtime send helpers (`tests/helpers/runtime.py`) — online and offline e2e:**
|
||||
|
||||
| Path | Fixture | Call |
|
||||
|------|---------|------|
|
||||
| Online `/v1/*` | `openai_client` | `openai_client.send_*_request(request_config)` |
|
||||
| Offline inference | `omni_runner_handler` | `omni_runner_handler.send_*_request(request_config)` |
|
||||
|
||||
1. **Grep `runtime.py` first** — reuse `send_omni_request`, `send_diffusion_request`, `send_audio_speech_request` (online + offline Qwen-style TTS), `send_single_stage_tts_request` (Coqui XTTS / MOSS-TTS-Nano offline), etc.
|
||||
2. **No matching helper** → add `send_<feature>_request` (or `send_<route>_http_request` for negative/dfx) in **`runtime.py`** with general `assert_*` bundled inside, **then** call it from the test.
|
||||
3. **Test file** holds `request_config` dicts only — not `omni.generate`, not `_collect_audio()`, not raw HTTP/SDK.
|
||||
4. Wire `tests/helpers/runtime.py` into Buildkite `source_file_dependencies` when you add helpers.
|
||||
|
||||
See [vllm-omni-test skill](../vllm-omni-test/SKILL.md) § **Runtime send helpers** for full tables and exceptions.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- Updated `serving_speech.py` with all 5 integration points (single commit)
|
||||
- Client scripts and server launcher under `examples/online_serving/text_to_speech/<model>/`
|
||||
- Gradio demo with streaming and voice cloning UI in the same dir
|
||||
- E2E online serving test (`tests/e2e/online_serving/test_<model>.py`)
|
||||
- Buildkite CI entry in `.buildkite/test-merge.yml`
|
||||
- E2E tests per **Test Case Writing (CI Levels)** above (priority tier determines L1–L4 scope)
|
||||
- Buildkite wired per level: `test-ready.yml` (L1/L2), `test-merge.yml` or nightly function job (L3), `test-nightly.yml` (L4) — see [vllm-omni-test skill](../vllm-omni-test/SKILL.md)
|
||||
- New section in `examples/online_serving/text_to_speech/README.md` (table row + per-model section). Do **not** create a top-level `examples/online_serving/<model>/` dir or a per-model `README.md` inside `text_to_speech/<model>/`.
|
||||
|
||||
### E2E test pitfalls to avoid
|
||||
@@ -326,9 +385,9 @@ def build_voice_clone_prompt(ref_audio_path: str, text: str, codec) -> list:
|
||||
- **Use the harness readiness gate.** The fixture waits for HTTP 200 on
|
||||
`/health`; don't add `time.sleep` in tests. If warmup is incomplete, make
|
||||
`/health` return non-200 until you're actually ready.
|
||||
- **Mark with `@pytest.mark.core_model` + `hardware_test(res={"cuda": "H100"})`**
|
||||
so the test lands in `test-ready.yml` (triggered by the `ready` label) rather
|
||||
than only nightly.
|
||||
- **Mark tests per the CI Levels table** — baseline smoke: `core_model` + `advanced_model`; heavier cases: `advanced_model` only; L4 expansion: `full_model`
|
||||
- **No per-model helper modules** — do not create `tests/helpers/{slug}.py`; keep constants and `request_config` payloads in the test file
|
||||
- **Online and offline e2e go through `runtime.py`** — `openai_client.send_audio_speech_request` (online); `omni_runner_handler.send_audio_speech_request` (Qwen-style offline) or `send_single_stage_tts_request` (single-stage offline). Add a new `send_*_request` in `runtime.py` when none fits; do not embed `omni.generate` or HTTP in tests
|
||||
|
||||
## Phase 4: Async Chunk (Streaming)
|
||||
|
||||
@@ -461,8 +520,8 @@ Use this checklist when integrating a new TTS model:
|
||||
- [ ] Voice cloning works (if supported)
|
||||
- [ ] All response formats work (wav, mp3, flac, pcm)
|
||||
- [ ] Client scripts and server launcher created
|
||||
- [ ] E2E online serving test written (`tests/e2e/online_serving/test_<model>.py`)
|
||||
- [ ] Buildkite CI entry added to `.buildkite/test-merge.yml`
|
||||
- [ ] E2E tests added per model priority tier (see **Test Case Writing (CI Levels)**)
|
||||
- [ ] Buildkite entries match level: `test-ready.yml` / `test-merge.yml` or nightly TTS job / `test-nightly.yml`
|
||||
- [ ] Gradio demo working
|
||||
- [ ] Documentation added (offline + online docs, nav, supported models)
|
||||
|
||||
@@ -500,5 +559,6 @@ Project docs and adjacent skills:
|
||||
- [TTS audio skill](../vllm-omni-audio-tts/SKILL.md) — supported models and usage
|
||||
- [Fish Speech integration](../vllm-omni-audio-tts/references/fish-speech.md) — complete example of Phases 1–3
|
||||
- [Qwen3-TTS reference](../vllm-omni-audio-tts/references/qwen-tts.md) — complete example of all 5 phases
|
||||
- [vllm-omni-test skill](../vllm-omni-test/SKILL.md) — L1–L4 markers, naming, Buildkite wiring, run commands
|
||||
- [Adding a TTS model (developer guide)](https://github.com/vllm-project/vllm-omni/blob/main/docs/contributing/model/adding_tts_model.md)
|
||||
- `plan/voxcpm2_native_ar_design.md` — VoxCPM2's vLLM-native AR + side-computation pattern (distinct from the generator-based single-stage described above)
|
||||
@@ -30,6 +30,12 @@ include:
|
||||
- `precheck-pr`: self-check a branch before creating a PR — validates PR title
|
||||
format, catches dead code, verifies accuracy/perf claims, and confirms merge
|
||||
readiness
|
||||
- `vllm-omni-test`: guides generation and execution of CI-aligned tests (L1–L4),
|
||||
pytest marker selection (`core_model` / `advanced_model` / `full_model`,
|
||||
`omni` / `tts` / `diffusion`), Buildkite wiring (`test-ready.yml`,
|
||||
`test-merge.yml`, `test-nightly.yml`, `test-weekly.yml`), and copy-paste
|
||||
local plus CI-like `pytest` commands; see `references/test-routing.md` for
|
||||
level-to-command mapping
|
||||
- `review-pr`: provides a structured workflow for reviewing pull requests
|
||||
- `vllm-omni-npu-model-runner-upgrade`: upgrades NPU model runners to align with the
|
||||
latest vllm-ascend NPUModelRunner
|
||||
|
||||
@@ -0,0 +1,890 @@
|
||||
---
|
||||
name: vllm-omni-test
|
||||
description: Generate and run tests for vllm-project/vllm-omni with CI-aligned levels and markers; wire new tests into Buildkite (test-ready.yml for L1/L2, test-merge.yml for L3, test-nightly.yml for L4). On completion, always provide copy-paste local and CI-like pytest commands plus prerequisites. Use when creating regression tests, adding L1-L4 coverage, selecting pytest markers, or validating fixes from issues/PRs.
|
||||
---
|
||||
|
||||
# vLLM-Omni Test Generator & Runner
|
||||
|
||||
## Purpose
|
||||
|
||||
Use this skill to generate minimal, stable test cases and run them with the correct marker/level strategy for [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni).
|
||||
|
||||
**Link convention:** Paths such as `.buildkite/` and `docs/contributing/` live at the **vllm-omni repo root**. Markdown links use repo-relative paths from this skill file (e.g. `../../../.buildkite/test-ready.yml`, `../../../docs/contributing/ci/CI_5levels.md`).
|
||||
|
||||
Default priorities:
|
||||
|
||||
1. Reproducible regression coverage for bug fixes
|
||||
2. Correct test level and marker selection
|
||||
3. Low flake, low dependency tests first
|
||||
4. CI-compatible run commands
|
||||
5. **Actionable run commands for the human**: whenever you add or change tests, always finish with **copy-paste-ready** `pytest` lines (local: single file and/or single test; CI-like: markers + `--run-level`), plus short **prerequisites** (GPU tier, HF cache, optional `model_prefix`). Do not assume the reader will infer commands from `test-routing.md` alone.
|
||||
|
||||
## Inputs
|
||||
|
||||
- Issue/PR link and summary
|
||||
- Changed files or suspected code path
|
||||
- Whether the user wants local quick validation or CI-equivalent validation
|
||||
- Hardware constraints (CPU only / CUDA / ROCm / NPU)
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Classify Test Goal
|
||||
|
||||
- **Bugfix regression**: start from a minimal failing scenario and add assertions that prevent recurrence. Before writing tests, output **`required` / `recommended` / `not_needed`**: **`required`** — stable logic/contract bug that should have been caught; **`recommended`** — environment-sensitive but a small regression still helps; **`not_needed`** — one-off external/config failure or existing tests already cover the path. Prefer the narrowest stable L1 (CPU) case; escalate to L2/L3 only when the bug needs real weights or serving.
|
||||
- **Feature coverage**: verify new behavior and one negative/boundary case.
|
||||
- **Perf/benchmark claim**: require benchmark-oriented tests and explicit metrics.
|
||||
|
||||
### Step 2: Select Test Level
|
||||
|
||||
- **L1**: unit/logic, deterministic, CPU-friendly, fastest feedback.
|
||||
- **L2**: basic e2e/integration and platform-dependent checks.
|
||||
- **L3/L4**: advanced model/integration/perf validation.
|
||||
|
||||
Use [references/test-routing.md](references/test-routing.md) for level-to-marker and command mapping.
|
||||
|
||||
### Step 3: Pick Markers
|
||||
|
||||
Always attach markers deliberately:
|
||||
|
||||
- **Level**: `core_model` (L1/L2) and/or `advanced_model` (L3) and/or `full_model` (L4 nightly)
|
||||
- **Model type** (required on model-centric e2e — pick exactly one):
|
||||
- `omni` — end-to-end multimodal LLM pipelines (thinker/talker/stages; Qwen-Omni family)
|
||||
- `tts` — speech synthesis / TTS-only models (`/v1/audio/speech`, voice clone, etc.)
|
||||
- `diffusion` — generative diffusion models (image / audio / text / video from noise)
|
||||
- **Cross-cutting area** (when relevant): `parallel`, `cache`, `example`, `benchmark`
|
||||
- **Hardware**: `cpu`, `gpu`, `cuda`, `rocm`, `npu`, `L4`, `H100`, `distributed_cuda`, …
|
||||
- Optional: `slow`, distributed markers when multi-card is required
|
||||
|
||||
**Baseline smoke (L2 + L3):** The simplest e2e case per model — default deploy, minimal request — should usually carry **both** `@pytest.mark.core_model` and `@pytest.mark.advanced_model` on the **same** test function so `test-ready.yml` and `test-merge.yml` share one test. `send_*_request` picks validation depth from `--run-level`. References: `test_voxcpm2_tts.py::test_text_to_audio_001`, `test_qwen3_tts_customvoice.py::test_text_to_audio_001`. Heavier scenarios use **`advanced_model` only**; L4 expansion uses **`full_model`**.
|
||||
|
||||
For hardware-aware tests, prefer `@hardware_test(...)` or `hardware_marks(...)` in `tests/helpers/mark.py`.
|
||||
|
||||
**Diffusion nightly split** (`test-nightly.yml`): all diffusion tests use `pytest.mark.diffusion`, but CI groups them by **output modality**, not by separate pytest markers:
|
||||
|
||||
| Nightly group | Scope | Typical models / paths |
|
||||
|---------------|--------|-------------------------|
|
||||
| **Diffusion X2I(&A&T) Model Test** | x2**i** (image), x2**a** (audio), x2**t** (text) and other **non-video** diffusion | Qwen-Image*, BAGEL, FLUX, SD3, Z-Image, LongCat, DreamZero, … — `test_*_expansion.py` under X2I shards; L4 sweep uses `-k "not test_wan and not test_bagel_expansion and not hunyuan"` for L4 |
|
||||
| **Diffusion X2V Model Test** | x2**v** (video) only | Wan2.2, HunyuanVideo 1.5, Wan VACE, … — e.g. `test_wan22_expansion.py`, `test_hunyuan_video_15_expansion.py` |
|
||||
|
||||
Wire new **video** diffusion expansion tests into the **X2V** group; wire **image/audio/text** diffusion expansion tests into **X2I(&A&T)**. Do not place x2v modules in X2I shards (see comment in `test-nightly.yml` above the X2I group).
|
||||
|
||||
### Naming: generated test module files (L2–L4, model-centric e2e)
|
||||
|
||||
When adding **new** pytest modules whose primary scope is a **specific model** (typical under `tests/e2e/offline_inference/` or `tests/e2e/online_serving/`), use this filename pattern:
|
||||
|
||||
| Level | Filename pattern | Example (model `Qwen/Qwen2.5-Omni-7B`) |
|
||||
|-------|------------------|----------------------------------------|
|
||||
| **L2**, **L3** | `test_{lowercase_model_slug}.py` | `test_qwen2_5_omni.py` |
|
||||
| **L4** | `test_{lowercase_model_slug}_expansion.py` | `test_qwen2_5_omni_expansion.py` |
|
||||
|
||||
**Slug rules for `{lowercase_model_slug}`:**
|
||||
|
||||
1. Start from the HuggingFace-style id (e.g. `Qwen/Qwen2.5-Omni-7B`), but **do not** put the org into the filename: use the **repo segment only** (`Qwen2.5-Omni-7B`), not `Qwen_...` / `qwen_qwen2_5_...`.
|
||||
2. **Lowercase**; replace `.`, `-`, and whitespace with a single `_` (e.g. `Qwen2.5-Omni-7B` → `qwen2_5_omni`). **Omit** trailing size tokens such as `7b` / `30b` in the basename when a single file covers that model line in the directory (matches `test_qwen2_5_omni.py` in-tree).
|
||||
3. If two checkpoints in the same folder need separate modules, add a **minimal** disambiguator (e.g. `_7b` vs `_3b`) only then.
|
||||
4. **L1** unit tests are **not** bound to this pattern; use `tests/<area>/test_<feature>.py` as today.
|
||||
|
||||
Routing tables and commands: [references/test-routing.md](references/test-routing.md) § *Model-centric e2e filename convention*.
|
||||
|
||||
Existing references: `tests/e2e/offline_inference/test_qwen2_5_omni.py` (L2-style omni), `tests/e2e/offline_inference/test_qwen3_5_9b.py` (L2-style omni, single-stage VL), `tests/e2e/online_serving/test_qwen3_omni_expansion.py` (L4-style omni), `tests/e2e/online_serving/test_qwen_image_edit_expansion.py` / `test_qwen_image_expansion.py` (L4-style diffusion).
|
||||
|
||||
### Step 4: Generate Test Case Skeleton
|
||||
|
||||
**1. Pick the functional scenario** (then choose directory, fixtures, and markers):
|
||||
|
||||
| Scenario | Typical location | Fixtures / runner pattern | Baseline markers & level |
|
||||
|----------|------------------|---------------------------|---------------------------|
|
||||
| **Offline inference e2e** | `tests/e2e/offline_inference/` | **Module (default):** `omni_runner` + `omni_runner_handler`. **Function (isolation only):** `omni_runner_function` + `omni_runner_handler_function`. Diffusion/TTS may use `Omni(...).generate` directly | L2: `core_model` + **one of** `omni` / `tts` / `diffusion`; `@hardware_test(...)` when GPU/NPU is required |
|
||||
| **Online serving e2e** | `tests/e2e/online_serving/` | **Module (default):** `omni_server` + `openai_client`. **Function (isolation only):** `omni_server_function` + `openai_client_function`. Clients: `send_omni_request` (omni), `send_audio_speech_request` (tts), `send_diffusion_request` / `send_video_diffusion_request` / `send_images_generations_request` (diffusion) | Baseline smoke: **`core_model` + `advanced_model`**; heavier paths: `advanced_model` only; L4 expansion: `full_model` |
|
||||
| **Documentation / runnable examples** | `tests/examples/offline_inference/`, `tests/examples/online_serving/` | **Offline docs (preferred):** extract Python/Bash blocks from the doc README (e.g. `ReadmeSnippet.extract_readme_snippets`), `pytest.mark.parametrize` each snippet, run via `example_runner.run` with a stable `output_subfolder`. **Online docs:** copy client/request scripts into dedicated tests and keep them in sync with the doc page. | Usually **L4**: `advanced_model`, often `example` plus hardware marks matching the nightly docs-example job (see `.buildkite/test-nightly.yml`). Full conventions: [docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md](../../../docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md) (introduced in [PR #1910](https://github.com/vllm-project/vllm-omni/pull/1910): naming, output directory layout, skip rules, avoid trimming `num_inference_steps` without a strong CI reason). |
|
||||
| **Performance / benchmark** | `tests/dfx/perf/tests/*.json` + `run_*_benchmark.py` | JSON or script-driven server + load config; assert explicit metrics / baselines | L4 Perf: `full_model` + `benchmark`; wire `test-nightly.yml` Perf steps |
|
||||
| **Invalid parameter / negative HTTP validation** | `tests/dfx/reliability/invalid_param_test/` | Live `omni_server` + low-level `send_*_http_request` with `err_code` / `err_message` | `pytest.mark.slow` + `omni` / `tts` / `diffusion` + `@hardware_marks` (`H100` or `L4`); CI in **`test-weekly.yml`** (not ready/merge/nightly) |
|
||||
|
||||
**If the user’s test plan includes invalid parameter validation / invalid params / negative HTTP / 400 validation:** do **not** add those `test_*` functions to `tests/e2e/online_serving/test_*.py` or `*_expansion.py`. **Move or author them** under `tests/dfx/reliability/invalid_param_test/` in the **endpoint-matching script** (see **Invalid parameter validation** below). Success-path e2e and invalid-param dfx tests must stay in separate modules.
|
||||
|
||||
**1b. Model type — `omni` vs `tts` vs `diffusion`**
|
||||
|
||||
After choosing offline/online/docs/perf, classify the **product under test** and attach **exactly one** model-type marker. All three can live under the same `tests/e2e/` trees; conventions diverge:
|
||||
|
||||
| Dimension | **Omni** (`pytest.mark.omni`) | **TTS** (`pytest.mark.tts`) | **Diffusion** (`pytest.mark.diffusion`) |
|
||||
|-----------|------------------------------|----------------------------|----------------------------------------|
|
||||
| **What it is** | Multimodal LLM pipeline (thinker/talker/stages; text + vision + audio I/O) | Speech synthesis / voice models | Generative diffusion (noise → image, audio, text, or video) |
|
||||
| **Examples** | Qwen2.5-Omni, Qwen3-Omni | Qwen3-TTS, VoxCPM2, Higgs-Audio, Step-Audio2 | Qwen-Image, BAGEL, Wan2.2, HunyuanVideo |
|
||||
| **Offline runner** | `omni_runner` + `omni_runner_handler`, `generate_multimodal` | `Omni(...).generate` or stage YAML + TTS params | `Omni(...).generate` + `OmniDiffusionSamplingParams` |
|
||||
| **Online client** | `openai_client.send_omni_request` (chat completions, modalities) | `openai_client.send_audio_speech_request` (`/v1/audio/speech`) | `send_diffusion_request` (chat/T2I), `send_video_diffusion_request` (`/v1/videos`, X2V), or `send_images_generations_http_request` / `send_images_edits_http_request` (DALL-E routes) |
|
||||
| **Typical assertions** | Stage outputs, text/audio keywords via `OmniRunnerHandler` / response handler | WAV bytes, stream chunks, speech endpoint contract | Image/video dimensions, `final_output_type`, `assert_diffusion_response` |
|
||||
| **Stage / deploy YAML** | Per-model omni stage configs (`ci/qwen3_omni_moe.yaml`, …) | `qwen3_tts.yaml`, `voxcpm2.yaml`, … | Often default serve; parallel/offload YAML for heavy DiT |
|
||||
| **Nightly group** (`test-nightly.yml`) | **Omni Model Test** — `-m "full_model and omni"` | **TTS Model Test** — `-m "full_model and tts"` | **Diffusion X2I(&A&T)** *or* **Diffusion X2V** (see Step 3 table; same `diffusion` marker, different YAML group / file shard) |
|
||||
| **L4 pressure** | Expansion per modality/model as needed | Expansion + accuracy/perf in TTS group | X2I: merge feature combos per [#1832](https://github.com/vllm-project/vllm-omni/issues/1832); X2V: separate nightly group |
|
||||
|
||||
Do **not** mix fixtures across types (e.g. do not use `omni_runner` layout for a pure diffusion or TTS model without mirroring an in-tree test in that family).
|
||||
|
||||
**Diffusion only — X2I(&A&T) vs X2V (nightly routing, not extra markers):**
|
||||
|
||||
- **X2I(&A&T)**: image / audio / text generation — Qwen-Image*, FLUX, SD3, Z-Image, BAGEL (expansion), LongCat, audio diffusion, etc.
|
||||
- **X2V**: video generation only — Wan2.2, HunyuanVideo 1.5, Wan VACE, LTX video similarity paths.
|
||||
|
||||
When adding a new `test_*_expansion.py`, place it in the matching **nightly group** step (explicit file list in `test-nightly.yml`), not only by marker expression.
|
||||
|
||||
**2. Use the narrowest deterministic skeleton for the scenario**
|
||||
|
||||
*L1 unit / logic (CPU-first):*
|
||||
|
||||
**Mocking rule (L1 only):** use **pytest** integration — **`mocker`** (`pytest-mock`) or **`monkeypatch`** (built-in). **Do not** import or call **`unittest.mock`** (`patch`, `MagicMock`, `@patch`, `with patch(...)`, etc.) in L1 tests; patches must auto-revert with the test lifecycle.
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
def test_<scenario_name>(mocker):
|
||||
# Prefer mocker.patch / mocker.spy / mocker.Mock — not unittest.mock.patch
|
||||
fake_fn = mocker.patch("vllm_omni.some.module.expensive_call", return_value=...)
|
||||
# Act
|
||||
# Assert
|
||||
fake_fn.assert_called_once()
|
||||
|
||||
|
||||
def test_<env_or_attr>(monkeypatch):
|
||||
# For simple env / attribute substitution without a Mock object
|
||||
monkeypatch.setenv("SOME_FLAG", "1")
|
||||
monkeypatch.setattr("vllm_omni.some.module.CONST", 42)
|
||||
...
|
||||
```
|
||||
|
||||
**Avoid in L1:**
|
||||
|
||||
```python
|
||||
# BAD — do not use in L1 unit tests
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
@patch("vllm_omni.some.module.fn")
|
||||
def test_bad(mock_fn): ...
|
||||
|
||||
def test_also_bad():
|
||||
with patch("...") as m: ...
|
||||
```
|
||||
|
||||
See **L1 unit test constraints (mocking)** below for the full do/don't list.
|
||||
|
||||
*Offline multimodal e2e — **Omni** (representative):*
|
||||
|
||||
```python
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.omni
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
def test_<scenario>(omni_runner, omni_runner_handler) -> None:
|
||||
request_config = {"prompts": ..., "modalities": [...]} # optional: images, videos, audios
|
||||
omni_runner_handler.send_omni_request(request_config)
|
||||
```
|
||||
|
||||
*Offline generative e2e — **Diffusion** (representative):*
|
||||
|
||||
```python
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.diffusion
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
def test_text_to_image_001(omni_runner_handler) -> None:
|
||||
omni_runner_handler.send_diffusion_request({"prompt": "...", "extra_body": {"num_inference_steps": 4, ...}})
|
||||
```
|
||||
|
||||
*Offline TTS e2e — **Qwen3-TTS** (two-stage; representative):*
|
||||
|
||||
```python
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.tts
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_runner", tts_server_params, indirect=True)
|
||||
def test_text_to_audio_001(omni_runner, omni_runner_handler) -> None:
|
||||
omni_runner_handler.send_audio_speech_request({
|
||||
"input": "...",
|
||||
"task_type": "Base",
|
||||
"ref_audio": REF_AUDIO_URL,
|
||||
"ref_text": REF_TEXT,
|
||||
})
|
||||
```
|
||||
|
||||
*Offline TTS e2e — **single-stage** (Coqui XTTS, MOSS-TTS-Nano; representative):*
|
||||
|
||||
```python
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.tts
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_runner", tts_server_params, indirect=True)
|
||||
def test_voice_clone_001(omni_runner_handler) -> None:
|
||||
omni_runner_handler.send_single_stage_tts_request({
|
||||
"input": "...",
|
||||
"language": "en",
|
||||
"prompt_audio_path": REF_AUDIO_PATH,
|
||||
"response_format": "wav",
|
||||
"run_level": "advanced_model",
|
||||
})
|
||||
```
|
||||
|
||||
*Online serving e2e — **Omni** (representative):*
|
||||
|
||||
```python
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.omni
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_server", test_params, indirect=True)
|
||||
def test_text_to_text_001(omni_server, openai_client) -> None:
|
||||
request_config = {"model": omni_server.model, "messages": ..., "modalities": ["text"]}
|
||||
openai_client.send_omni_request(request_config)
|
||||
```
|
||||
|
||||
*Online serving e2e — **TTS** (representative):*
|
||||
|
||||
```python
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.advanced_model
|
||||
@pytest.mark.tts
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_server", tts_server_params, indirect=True)
|
||||
def test_text_to_audio_001(omni_server, openai_client) -> None:
|
||||
request_config = {"model": omni_server.model, "input": "...", "response_format": "wav", ...}
|
||||
openai_client.send_audio_speech_request(request_config)
|
||||
```
|
||||
|
||||
*Online serving e2e — **Diffusion** X2I (representative):*
|
||||
|
||||
```python
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.diffusion
|
||||
@pytest.mark.parametrize("omni_server", _get_default_case(MODEL), indirect=True)
|
||||
def test_text_to_image_001(omni_server, openai_client) -> None:
|
||||
openai_client.send_diffusion_request({...}) # chat completions + extra_body
|
||||
```
|
||||
|
||||
*Online serving e2e — **Diffusion** X2V (representative):*
|
||||
|
||||
```python
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.diffusion
|
||||
def test_text_to_video_001(omni_server, openai_client) -> None:
|
||||
openai_client.send_video_diffusion_request({"model": ..., "form_data": {...}}) # /v1/videos
|
||||
```
|
||||
|
||||
*Documentation example tests:* follow the **Preferred Test Strategy** in [l4_doc_example_tests.inc.md](../../../docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md): dynamic extraction for offline READMEs; explicit copied client code for online pages until extraction is justified; use the documented **naming**, **output directory** (page folder + case id), and **skipping** rules (e.g. Gradio-only scripts).
|
||||
|
||||
*Performance tests:* add or extend entries under `tests/perf/` (and JSON configs where the project uses them), with **explicit baselines** and the same marker/run-level pairing as the CI step that will execute them.
|
||||
|
||||
**3. Cross-cutting rules**
|
||||
|
||||
- Reuse existing fixtures for the chosen scenario; do not mix “online client” assumptions into offline `OmniRunner` tests without a clear reason.
|
||||
- Avoid external network dependency in assertions unless the scenario is explicitly “online serving” or doc examples that require a model hub (then align with CI secrets/cache).
|
||||
- Keep **one test function = one intent** (one modality combo, one endpoint contract, or one acceleration combo).
|
||||
- **E2E test function layout**: **one case → one `test_<scenario>` function** with a name that states what is validated (endpoint, size/`n`, server flag, or route). **Do not** merge multiple cases into a single test that branches on `request.node.callspec.id`, `param.id`, or `if case_id == ...`. Use `@pytest.mark.parametrize("omni_server", [...], indirect=True)` per function (usually **one** `OmniServerParams` per test). A loop inside one test is OK only when it serves **that function’s single intent** (e.g. three standard sizes in `test_*_sizes_256_512_1024`).
|
||||
- **Runtime fixture scope** (`tests/helpers/fixtures/runtime.py`): default **`omni_server` / `omni_runner`** (module) + matching client/handler; use **`omni_server_function` / `omni_runner_function`** only when each `test_*` must start a fresh instance (see below).
|
||||
- **L1 mocks**: never `unittest.mock`; use `mocker` or `monkeypatch` only (see below).
|
||||
- **API calls (L2+ e2e, online and offline)**: **reuse** `send_*_request` in `tests/helpers/runtime.py` when it exists; **otherwise add** the helper in `runtime.py` first, then call it from the test. General `assert_*` inside `send_*_request`; special `assert_*` only in the test. See **Runtime send helpers** below — do not call `omni.generate`, raw HTTP, or SDK clients from `test_*.py`.
|
||||
- **Response assertions**: reusable checks on API bodies / decoded media belong in **`tests/helpers/assertions.py`** — not as private `_assert_*` helpers inside `test_*.py` (see below).
|
||||
- **Model-specific payloads stay in test modules** — per-model `MODEL`, deploy path, `REF_AUDIO_URL`, `get_prompt()`, `_build_request()`, and inline `request_config` dicts live in `test_{slug}.py` / `test_{slug}_expansion.py` (and offline/L1 siblings). **Do not** create `tests/helpers/{slug}.py` to deduplicate them; a little copy across files is preferred (see `test_glm_tts.py`, `test_cosyvoice3_tts_expansion.py`). `tests/helpers/` is repo-wide harness only (`mark`, `media`, `runtime`, `stage_config`, `assertions`, `fixtures/`).
|
||||
|
||||
#### L1 unit test constraints (mocking)
|
||||
|
||||
L1 tests (`core_model and cpu`, under `tests/diffusion/`, `tests/engine/`, `tests/model_executor/`, etc.) must follow the repo’s **pytest-mock** convention (`pytest-mock>=3.10.0` in `pyproject.toml` `[project.optional-dependencies] dev`).
|
||||
|
||||
| Do | Don't |
|
||||
|----|-------|
|
||||
| `def test_foo(mocker):` + `mocker.patch(...)`, `mocker.spy(...)`, `mocker.Mock()`, `mocker.MagicMock()`, `mocker.AsyncMock()` | `from unittest.mock import patch, MagicMock, Mock, AsyncMock` |
|
||||
| `def test_bar(monkeypatch):` + `monkeypatch.setattr` / `setenv` / `delenv` / `setitem` | `@patch(...)` decorator |
|
||||
| Let `mocker` auto-stop patches after the test | `with patch(...):` / `patch.object(...)` context managers |
|
||||
| Mirror neighboring L1 tests in the same directory | `unittest.mock.create_autospec` unless an existing file already documents an exception |
|
||||
|
||||
**Rationale:** `mocker` ties patch lifecycle to pytest fixtures (no leaked patches across tests). `unittest.mock` decorators/context managers are easy to compose incorrectly with parametrized or async tests and are inconsistent with in-tree L1 style.
|
||||
|
||||
**Minimal patterns:**
|
||||
|
||||
```python
|
||||
def test_returns_cached_config(mocker):
|
||||
loader = mocker.patch(
|
||||
"vllm_omni.foo.load_yaml",
|
||||
return_value={"stages": []},
|
||||
)
|
||||
result = get_deploy_config("ci/foo.yaml")
|
||||
assert result["stages"] == []
|
||||
loader.assert_called_once_with("ci/foo.yaml")
|
||||
|
||||
|
||||
def test_skips_when_env_unset(monkeypatch):
|
||||
monkeypatch.delenv("VLLM_OMNI_FEATURE", raising=False)
|
||||
assert should_enable_feature() is False
|
||||
```
|
||||
|
||||
E2E levels (L2+) generally **avoid mocks**; if a rare L2 stub is unavoidable, still prefer `mocker` over `unittest.mock` for consistency.
|
||||
|
||||
#### Runtime fixtures — scope (`tests/helpers/fixtures/runtime.py`)
|
||||
|
||||
vLLM-Omni e2e tests start a real **OmniServer** (online) or **OmniRunner** (offline). Pick scope by **how often the process must be recreated**, not by test level alone.
|
||||
|
||||
| Scope | Online fixtures | Offline fixtures | When to use |
|
||||
|-------|-----------------|------------------|-------------|
|
||||
| **Module** (default) | `omni_server` → `openai_client` | `omni_runner` → `omni_runner_handler` | **Default for L2/L3/L4 expansion** — amortize model/server init across `test_*` in the same module. Same `OmniServerParams` / runner config can be reused by multiple tests. |
|
||||
| **Function** | `omni_server_function` → `openai_client_function` | `omni_runner_function` → `omni_runner_handler_function` | **Only when required** — each `test_*` must get a **clean** server/runner (no shared engine/GPU state). Typical: `tests/dfx/reliability/`, sleep/wakeup, crash/restart, tests that mutate global server state. |
|
||||
|
||||
**Rules:**
|
||||
|
||||
1. **Default to module scope** (`omni_server` / `omni_runner`) unless the scenario **explicitly** needs a fresh instance per test function.
|
||||
2. **Indirect parametrize name must match the fixture name**: `@pytest.mark.parametrize("omni_server", ...)` with `omni_server` + `openai_client`; `@pytest.mark.parametrize("omni_server_function", ...)` with `omni_server_function` + `openai_client_function`. Do not mix module fixture with function client (or vice versa).
|
||||
3. Different `OmniServerParams` per `test_*` (e.g. default vs `--enable-cpu-offload`) is still OK with **module** `omni_server` — pytest parametrizes per test node; only switch to `_function` when isolation between tests matters, not merely because `server_args` differ.
|
||||
4. One `test_*` with **many** server configs (expansion matrix) → single function + `@pytest.mark.parametrize("omni_server", [...], indirect=True)` + module `omni_server` (see in-tree `test_qwen_image_expansion.py`).
|
||||
|
||||
```python
|
||||
# Default — module-scoped server (L4 expansion)
|
||||
@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=H100)], indirect=True)
|
||||
def test_foo_images_generations_default_1024(omni_server, openai_client) -> None:
|
||||
openai_client.send_images_generations_request({...})
|
||||
|
||||
# Function-scoped — reliability / per-test clean state only
|
||||
@pytest.mark.parametrize(
|
||||
"omni_server_function",
|
||||
[pytest.param(OmniServerParams(model=MODEL), marks=H100)],
|
||||
indirect=True,
|
||||
)
|
||||
def test_foo_sleep_wakeup_cycle(omni_server_function, openai_client_function) -> None:
|
||||
openai_client_function.send_omni_sleep_http_request({...})
|
||||
openai_client_function.send_omni_wakeup_http_request({...})
|
||||
```
|
||||
|
||||
#### Runtime send helpers — online **and** offline (`tests/helpers/runtime.py`)
|
||||
|
||||
**L2+ e2e (online serving and offline inference) must call APIs through `tests/helpers/runtime.py`.** Fixtures live in `tests/helpers/fixtures/runtime.py`; **send/assert implementation** lives in `tests/helpers/runtime.py` + `tests/helpers/assertions.py`.
|
||||
|
||||
| Principle | Action |
|
||||
|-----------|--------|
|
||||
| **Reuse first** | Grep `runtime.py` for an existing **`send_*_request`** on `OpenAIClientHandler` (online) or `OmniRunnerHandler` (offline) that matches the endpoint / pipeline shape. |
|
||||
| **Extend when close** | If an existing `send_*_request` almost fits (e.g. missing one optional field), extend it in `runtime.py` — do not fork logic in the test file. |
|
||||
| **Add when missing** | No suitable helper → add **`send_<feature>_request`** (high-level: call + general `assert_*`) or **`send_<route>_<verb>_http_request`** (low-level HTTP for negative/dfx) in **`runtime.py` first**, then call it from tests. |
|
||||
| **Test module owns payload only** | In `test_*.py`: `MODEL`, deploy path, vendored media, `get_prompt()`, and inline **`request_config` dicts** only. **No** `omni.generate(...)`, raw `requests.post`, OpenAI SDK calls, or `_collect_audio()` / `_process_output()` in e2e tests. |
|
||||
|
||||
**Online** (`openai_client` from `omni_server`): `OpenAIClientHandler.send_*_request`.
|
||||
|
||||
**Offline** (`omni_runner_handler` from `omni_runner`): `OmniRunnerHandler.send_*_request`.
|
||||
|
||||
| Do | Don't |
|
||||
|----|-------|
|
||||
| Online: `openai_client.send_omni_request`, `send_diffusion_request`, `send_audio_speech_request`, `send_video_diffusion_request`, `send_images_generations_request`, … | `requests.post(f"{base_url}/v1/...", json=…)` or `client.chat.completions.create(...)` inside a test |
|
||||
| Offline: `omni_runner_handler.send_omni_request`, `send_diffusion_request`, `send_audio_speech_request`, `send_single_stage_tts_request`, `send_single_stage_tts_batch_request`, … | `omni_runner.omni.generate(...)` + hand-rolled tensor/WAV extraction in `test_*.py` |
|
||||
| Add missing `send_*` to **`runtime.py`** first; bundle general `assert_*` inside the send helper | A one-off `def _post_*` / `def _collect_audio` at the bottom of a test module |
|
||||
| Mirror naming/style of neighboring `send_*` (docstring, `request_config` dict, optional `run_level`, `err_code` / `err_message` for negative cases) | Different parameter shapes per test file for the same endpoint |
|
||||
|
||||
**Workflow when generating online/offline e2e tests:**
|
||||
|
||||
1. Decide whether the needed check is **general** (every success call of this `send_*`) or **special** (one case / one parameter combo only). See **General vs special assert placement** below.
|
||||
2. **Search `runtime.py`** for a matching **`send_*_request`**; if missing, add it (with general `assert_*` inside) before writing the test body.
|
||||
3. In the test module: build `request_config` → call **`send_*_request` only** for the general contract; call **`assert_*` in the test file only** for special, case-specific checks (import from `assertions.py`).
|
||||
4. Reserve **low-level** `send_*_http_request` for negative/dfx tests (`err_code` / `err_message`) — not for ordinary L2+ success-path e2e.
|
||||
5. When wiring Buildkite `source_file_dependencies`, include **`tests/helpers/runtime.py`** and/or **`tests/helpers/assertions.py`** when new helpers were added.
|
||||
|
||||
**Exceptions (document in the test docstring why):** models whose offline prompt path cannot go through existing handlers yet (e.g. `test_higgs_audio_v2.py`, `test_voxtral_tts.py` with custom tokenizer compose) may call `omni.generate` directly until a `send_*_request` is added to `runtime.py` — treat as **debt**, not the default for new TTS/diffusion/omni e2e.
|
||||
|
||||
**General vs special assert placement:**
|
||||
|
||||
| Kind | Definition | Where it lives | Test file calls |
|
||||
|------|------------|----------------|-----------------|
|
||||
| **General** | Default success contract for this endpoint/client method — e.g. HTTP 200, `data[]` shape, `n` count, `size` dimensions, decodable image/audio, bundled omni/diffusion fields | Implement in `assertions.py`, invoke **inside** `send_*_request` in `runtime.py` (after low-level send / SDK call, when `err_code` is not set) | **`send_*_request` only** — do **not** repeat the same `assert_*` |
|
||||
| **Special** | Extra check tied to **this test case only** — e.g. seed byte-identical replay, accuracy/CLIP threshold, perf ceiling, model-specific optional field | Add or reuse `assert_*` in `assertions.py` (never inline in test) | `send_*_request` **then** `assert_<special>(...)` **once** for that case |
|
||||
|
||||
Changing `request_config` fields (`size`, `n`, `seed`, server flags) is **not** special validation — the general `assert_*` should read those from `request_config` inside `send_*`.
|
||||
|
||||
**`send_*` ↔ `assert_*` pairing (online `OpenAIClientHandler`):**
|
||||
|
||||
| High-level `send_*` (prefer in L2+ e2e) | Assert already invoked inside `runtime.py` | Low-level HTTP-only sibling |
|
||||
|----------------------------------------|--------------------------------------------|-----------------------------|
|
||||
| `send_omni_request` | `assert_omni_response` | `send_chat_completions_http_request` → `assert_http_error` only |
|
||||
| `send_diffusion_request` | `assert_diffusion_response` | — |
|
||||
| `send_audio_speech_request` | `assert_audio_speech_response` | `send_audio_speech_http_request` → `assert_http_error` only |
|
||||
| `send_video_diffusion_request` | `assert_diffusion_response` | `send_videos_*_http_request` → `assert_http_error` only |
|
||||
| `send_images_generations_request` *(add when needed)* | **`assert_images_generations_response`** *(general — inside send)* | `send_images_generations_http_request` → `assert_http_error` only |
|
||||
| `send_images_edits_request` *(add when needed)* | **`assert_images_edits_response`** *(general — inside send)* | `send_images_edits_http_request` → `assert_http_error` only |
|
||||
|
||||
**Rule:** Tests call high-level `send_*_request` for the general contract. **Never** call the same bundled `assert_*` again. Call an extra `assert_*` in the test **only** for special, case-specific validation.
|
||||
|
||||
**Common `OpenAIClientHandler` entry points** (non-exhaustive — grep `runtime.py` before adding):
|
||||
|
||||
| Area | High-level (SDK + assert) | Low-level HTTP (`*_http_request`) |
|
||||
|------|---------------------------|-----------------------------------|
|
||||
| Omni chat | `send_omni_request` | `send_chat_completions_http_request` |
|
||||
| Diffusion T2I (chat route) | `send_diffusion_request` | — |
|
||||
| Diffusion X2V | `send_video_diffusion_request` | `send_videos_create_http_request`, `send_video_content_http_request`, … |
|
||||
| DALL-E T2I / edit | `send_images_generations_request`, `send_images_edits_request` | `send_images_generations_http_request`, `send_images_edits_http_request` |
|
||||
| TTS (online) | `send_audio_speech_request` | `send_audio_speech_http_request`, `send_audio_generate_http_request`, … |
|
||||
| Ops / meta | — | `send_health_http_request`, `send_models_http_request`, `send_omni_sleep_http_request`, … |
|
||||
|
||||
**Common `OmniRunnerHandler` entry points** (offline — grep `runtime.py` before adding):
|
||||
|
||||
| Area | High-level (`send_*_request` + assert) | Notes |
|
||||
|------|----------------------------------------|-------|
|
||||
| Omni multimodal | `send_omni_request` | `generate_multimodal` path |
|
||||
| Diffusion offline | `send_diffusion_request` | chat-route / `OmniTextPrompt` offline |
|
||||
| Qwen-style TTS | `send_audio_speech_request` | two-stage, `generate_multimodal` + `mm_processor_kwargs` |
|
||||
| Single-stage TTS (Coqui XTTS, MOSS-TTS-Nano, …) | `send_single_stage_tts_request`, `send_single_stage_tts_batch_request` | `prompt` + `additional_information` + `omni.generate` — **do not** duplicate in tests |
|
||||
| New model family | **Add** `send_<family>_request` here first | Then call from `tests/e2e/offline_inference/test_*.py` |
|
||||
|
||||
L1 tests under `tests/entrypoints/` may use **FastAPI `TestClient`** or direct handler calls with mocks; they do **not** need `OpenAIClientHandler` / `OmniRunnerHandler`. **L2+ online and offline e2e** must use `runtime.py` `send_*` helpers.
|
||||
|
||||
#### Invalid parameter validation (`tests/dfx/reliability/invalid_param_test/`)
|
||||
|
||||
When the user asks for **invalid parameter validation**, **invalid request bodies**, **HTTP 4xx contract tests**, or any case that sends **malformed / out-of-range / mismatched** API payloads against a **live** server:
|
||||
|
||||
1. **Do not** place these in `tests/e2e/online_serving/` or `*_expansion.py`. If already drafted there, **move** the `test_*` into the correct `invalid_param_test` script and delete the duplicate from e2e.
|
||||
2. **Pick the script by HTTP route** (extend an existing file; add a new `test_invalid_<area>.py` only when no in-tree script covers that route family):
|
||||
|
||||
| API / area | Script |
|
||||
|------------|--------|
|
||||
| Omni chat completions, WebSocket video/realtime paths | `test_invalid_omni_chat.py` |
|
||||
| `POST /v1/audio/speech`, stream, batch, voices | `test_invalid_audio_speech.py` |
|
||||
| Audio diffusion endpoints | `test_invalid_audio_diffusion.py` |
|
||||
| `POST /v1/images/generations` | `test_invalid_image_generation.py` |
|
||||
| `POST /v1/images/edits` | `test_invalid_image_editing.py` |
|
||||
| `POST/GET/DELETE /v1/videos*` | `test_invalid_video_generation.py` |
|
||||
| Sleep / wakeup / server control | `test_invalid_server_control.py` |
|
||||
|
||||
3. **Match in-tree style** in the chosen script:
|
||||
|
||||
| Element | Convention |
|
||||
|---------|------------|
|
||||
| **Module markers** | `pytestmark = [pytest.mark.slow, pytest.mark.<omni or tts or diffusion>]` |
|
||||
| **Server fixture** | `_PARAMS` / `_QWEN3_TTS_SPEECH`-style list of `pytest.param(OmniServerParams(...), id="...", marks=hardware_marks(...))`; `@pytest.mark.parametrize("omni_server", _PARAMS, indirect=True)` |
|
||||
| **Hardware** | `hardware_marks(res={"cuda": "H100"})` for heavy diffusion/omni/video; `hardware_marks(res={"cuda": "L4"})` for smaller TTS models (must match weekly `-m "slow and L4"` step) |
|
||||
| **HTTP client** | **Low-level** `openai_client.send_*_http_request({..., "err_code": 400, "err_message": (...)} )` — **not** `send_*_request` (success path) |
|
||||
| **Case shape** | Prefer **one `test_*` per route family** + `@pytest.mark.parametrize("body_spec, err_message", [...])` with stable `id=` per case; or dedicated `test_<route>_malformed_json` when not parametrized |
|
||||
| **Body helpers** | `_minimal_<endpoint>_json(omni_server)` / `_minimal_*_form_data()`; merge overrides with `body.update(body_spec)` |
|
||||
| **Known gaps** | `pytest.mark.skip(reason="…#3649")` as `_SKIP_ISSUE_3649` when server validation is not yet strict (mirror neighboring cases) |
|
||||
| **Sections** | Route banner comments (`# ─── POST /v1/images/generations ───`) like existing files |
|
||||
| **Shared fixtures** | Reuse `tests/dfx/reliability/invalid_param_test/conftest.py` (`tiny_png_bytes`, env defaults) |
|
||||
|
||||
4. **Adding a new model** to invalid-param coverage: append a `pytest.param(OmniServerParams(model="...", stage_config_path=..., server_args=...), ...)` entry to the script’s `_PARAMS` list — do **not** create `test_invalid_<model>.py` unless the route is new.
|
||||
|
||||
5. **CI — `.buildkite/test-weekly.yml` only** (not `test-ready.yml` / `test-merge.yml` / `test-nightly.yml`):
|
||||
|
||||
| Weekly step | Command | When your cases run |
|
||||
|-------------|---------|---------------------|
|
||||
| **Invalid parameters Test · H100** | `pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and H100"` | Diffusion / omni / video invalid-param tests with `H100` hardware mark |
|
||||
| **Invalid parameters Test · L4** | `pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and L4"` | TTS / lighter models marked `L4` |
|
||||
|
||||
- **Trigger:** `build.env("WEEKLY") == "1"` or PR label `weekly-test`.
|
||||
- **Default:** extending an existing `invalid_param_test` script needs **no YAML edit** — the weekly steps already sweep the whole directory.
|
||||
- **Edit YAML only** when adding a **new hardware queue**, a **new top-level script** that must run in isolation, or a **model-specific weekly shard** (mirror neighboring reliability steps).
|
||||
- Weekly steps **do not** use `source_file_dependencies`.
|
||||
|
||||
**Example — append to `test_invalid_image_generation.py`:**
|
||||
|
||||
```python
|
||||
@pytest.mark.parametrize(
|
||||
"body_spec, err_message",
|
||||
[
|
||||
pytest.param({"seed": -1}, ("seed", "greater_than_equal", "0"), id="seed_negative"),
|
||||
],
|
||||
)
|
||||
@pytest.mark.parametrize("omni_server", _PARAMS, indirect=True)
|
||||
def test_images_generations_invalid_requests(
|
||||
omni_server: OmniServer,
|
||||
openai_client: OpenAIClientHandler,
|
||||
body_spec: dict[str, object],
|
||||
err_message: str | tuple[str, ...],
|
||||
) -> None:
|
||||
body = _minimal_images_gen_json(omni_server)
|
||||
body.update(body_spec)
|
||||
openai_client.send_images_generations_http_request(
|
||||
{"json": body, "timeout": 300, "err_code": 400, "err_message": err_message}
|
||||
)
|
||||
```
|
||||
|
||||
See [references/test-routing.md](references/test-routing.md) **Invalid parameter / weekly CI**.
|
||||
|
||||
#### Assertion helpers (`tests/helpers/assertions.py`)
|
||||
|
||||
**Do not** add module-local helpers such as `_assert_images_generations_payload` or `_send_and_assert_*` in e2e test files. Response/media validation belongs in **`tests/helpers/assertions.py`**, grouped by **category**:
|
||||
|
||||
| Category | Existing anchors | When to extend vs add |
|
||||
|----------|------------------|------------------------|
|
||||
| **Image** (chat diffusion + DALL-E JSON) | `assert_image_diffusion_response`, `assert_image_valid` | DALL-E `/v1/images/generations` JSON → add **`assert_images_generations_response`** beside image helpers (reuses `assert_image_valid`); do not fork decode logic into tests |
|
||||
| **Video** | `assert_video_diffusion_response`, `assert_video_valid` | Extend these for new video contracts |
|
||||
| **Audio** | `assert_audio_diffusion_response`, `assert_audio_speech_response`, `assert_audio_valid` | Extend for new TTS/audio endpoints |
|
||||
| **Omni multimodal** | `assert_omni_response` | Extend for new modality combos |
|
||||
| **HTTP errors** | `assert_http_error`, `assert_err_message_in_text` | Used inside low-level `send_*_http_request` and negative tests |
|
||||
|
||||
| Do | Don't |
|
||||
|----|-------|
|
||||
| Put **general** `assert_*` inside `send_*_request` in `runtime.py`; tests call `send_*_request` only | Call `assert_diffusion_response` after `send_diffusion_request` (general assert already bundled) |
|
||||
| Put **special** `assert_*` in `assertions.py` and call it **in the test** after `send_*_request` when that case needs extra checks | Put general decode/count/size logic in the test because “this case uses `n=4`” (that belongs in general assert reading `request_config`) |
|
||||
| Extend the matching category function, or add `assert_<endpoint>_response` next to its category | `_send_and_assert_*` or PIL/base64 loops in `test_*.py` |
|
||||
| Use low-level `send_*_http_request` in dfx/negative tests (`err_code`) | Use low-level HTTP send + manual general assert in every L2+ expansion file |
|
||||
|
||||
**Workflow when generating tests:**
|
||||
|
||||
1. Implement or reuse **general** `assert_*` in `assertions.py` (by category).
|
||||
2. Wire it into **`send_*_request`** in `runtime.py` if not already bundled.
|
||||
3. In the test: `request_config` → `send_*_request`. Add a separate `assert_*` import/call **only** for special case validation.
|
||||
4. Update `source_file_dependencies` when `runtime.py` or `assertions.py` changes.
|
||||
|
||||
**Example — DALL-E `/v1/images/generations`:**
|
||||
|
||||
```python
|
||||
# tests/helpers/assertions.py — general contract (image category)
|
||||
def assert_images_generations_response(resp_body: dict, request_config: dict, *, run_level: str | None = None) -> None:
|
||||
... # data[], n, size → assert_image_valid
|
||||
|
||||
# tests/helpers/runtime.py — general assert INSIDE send
|
||||
def send_images_generations_request(self, request_config: dict[str, Any], ...) -> list[HttpResponse]:
|
||||
responses = self.send_images_generations_http_request(request_config, ...)
|
||||
if request_config.get("err_code") is None:
|
||||
body = responses[0].json_body
|
||||
assert isinstance(body, dict)
|
||||
assert_images_generations_response(body, request_config, run_level=self.run_level)
|
||||
return responses
|
||||
|
||||
# tests/e2e/.../test_foo_expansion.py — one case per test_* (no case_id branching)
|
||||
@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=SINGLE_L4)], indirect=True)
|
||||
def test_foo_images_generations_default_1024(omni_server, openai_client) -> None:
|
||||
openai_client.send_images_generations_request({"json": body, "timeout": 300})
|
||||
|
||||
@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=SINGLE_L4)], indirect=True)
|
||||
def test_foo_images_generations_sizes_256_512_1024(omni_server, openai_client) -> None:
|
||||
for size in ("256x256", "512x512", "1024x1024"):
|
||||
openai_client.send_images_generations_request({"json": {**body, "size": size}, "timeout": 300})
|
||||
|
||||
# chat diffusion — separate test; send_diffusion_request bundles assert_diffusion_response
|
||||
@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=SINGLE_L4)], indirect=True)
|
||||
def test_foo_chat_completions_t2i_fallback(omni_server, openai_client) -> None:
|
||||
openai_client.send_diffusion_request({...})
|
||||
```
|
||||
|
||||
L1 tests under `tests/entrypoints/` may keep **minimal** asserts next to the handler under test when validating a single branch; still prefer `assertions.py` when the same JSON/media check appears in more than one file or level (e.g. L1 protocol test + L4 e2e).
|
||||
|
||||
#### Omni Test Writing Guidance (L1-L4 Layering)
|
||||
|
||||
When the goal is a general Omni/multimodal test case, prioritize mapping the test to the correct purpose, directory, and resource assumptions aligned with the layers (see [CI_5levels.md](../../../docs/contributing/ci/CI_5levels.md)):
|
||||
|
||||
- **L1**: Unit/logic validation on CPU (`core_model and cpu`). Cover input validation, branches, and exception paths (`tests/<component>/test_*.py`). **Mock with `mocker` / `monkeypatch` only — not `unittest.mock`.**
|
||||
- **L2**: Basic e2e (online/offline basic scenarios). Prefer dummy/lightweight models to validate the end-to-end request-to-output-structure/streaming chain (typically `tests/e2e/online_serving/` and `tests/e2e/offline_inference/`).
|
||||
- **L3/L4**: Important integration, performance, and accuracy validation. L4 emphasizes “full functional scenarios + performance/stress + runnable doc examples” (typically `*_expansion.py` plus related expansion cases).
|
||||
|
||||
Also keep markers consistent with the run level: use `core_model` for L1/L2 and `advanced_model` for L3/L4, and pair with `--run-level` to select the intended CI strategy.
|
||||
|
||||
#### Diffusion Test Writing Guidance (L4 Coverage Combinations)
|
||||
|
||||
When the task involves diffusion models/features, organize L4 test cases following [`#1832`](https://github.com/vllm-project/vllm-omni/issues/1832): combine multiple diffusion features into as few test cases as possible to fit limited CI GPU resources.
|
||||
|
||||
Implementation strategy:
|
||||
|
||||
1. If “full L4 is too heavy”: first provide a **reduced local validation case in L1/L2** (so the key assertion and the contract fix point are covered deterministically).
|
||||
2. Then provide **CI/nightly-ready L3/L4 high-marked cases** (e.g. `advanced_model`) to broaden coverage under resource constraints.
|
||||
|
||||
#### L4 nightly sub-pillars — Function vs Accuracy vs Perf
|
||||
|
||||
**L4 is not only `test_*_expansion.py`.** In `test-nightly.yml`, each model-type group typically runs **separate jobs**:
|
||||
|
||||
| Sub-pillar | Nightly step label pattern | What you add | Typical paths |
|
||||
|------------|----------------------------|--------------|---------------|
|
||||
| **Function** | `· Function Test with …` | E2e expansion / feature matrix | `tests/e2e/online_serving/test_<model>_expansion.py` (or offline) |
|
||||
| **Accuracy** | `· Accuracy Test` | Quality / similarity vs baseline | `tests/e2e/accuracy/test_<model>*.py` |
|
||||
| **Perf** | `· Perf Test · <Model>` | Throughput / latency / memory benchmarks | `tests/dfx/perf/tests/test_<model>_vllm_omni.json` + runner script |
|
||||
| **Doc** (optional) | `· Doc Test` | Runnable doc examples | `tests/examples/*/test_text_to_image.py`, … |
|
||||
|
||||
**Default when the user asks for “L4 functional cases”:** deliver **Function** pillar only (`*_expansion.py` + Function Test shard in `test-nightly.yml`). **Do not** silently add Perf or Accuracy unless the user also asks for **performance** / **benchmark** / **accuracy** / **full L4** / **full L4 coverage**.
|
||||
|
||||
**When the user explicitly asks for L4 perf (or full L4 including perf):**
|
||||
|
||||
1. **Do not** put throughput/latency assertions inside `test_*_expansion.py` — perf uses the **dfx benchmark harness**, not `omni_server` e2e fixtures.
|
||||
2. Add or extend a **JSON config** under `tests/dfx/perf/tests/` (mirror in-tree names: `test_qwen_image_vllm_omni.json`, `test_bagel_vllm_omni.json`).
|
||||
3. Each JSON entry: `test_name`, `server_params` (`model`, `serve_args`), `benchmark_params[]` with explicit **`baseline`** (`throughput_qps`, `latency_mean`, `peak_memory_mb_mean`, …).
|
||||
4. Run locally via the matching script (model type → runner):
|
||||
|
||||
| Model type | Runner script | Example config |
|
||||
|------------|---------------|----------------|
|
||||
| **Diffusion X2I/X2V** | `tests/dfx/perf/scripts/run_diffusion_benchmark.py` | `tests/dfx/perf/tests/test_qwen_image_vllm_omni.json` |
|
||||
| **TTS** | `tests/dfx/perf/scripts/run_benchmark.py` | `tests/dfx/perf/tests/test_tts.json` (shared) or `test_{slug}.json` (dedicated; use when the model must not join the shared nightly matrix before integration — e.g. `test_voxcpm2.json`, `test_coqui_tts.json`) |
|
||||
| **Omni** | (see existing Omni Perf steps — `run_benchmark.py` / dedicated omni configs in-tree) | `tests/dfx/perf/tests/test_omni*.json` |
|
||||
|
||||
5. Wire **`test-nightly.yml` · Perf Test** step for that model (separate from Function Test): export `DIFFUSION_BENCHMARK_DIR` / `BENCHMARK_DIR`, run pytest on the script with `--test-config-file`, **upload artifacts** (`buildkite-agent artifact upload`), often **multi-GPU H100** for diffusion perf.
|
||||
6. **One benchmark scenario → one `test_name` block** in JSON (same spirit as one `test_*` per function case). Combine server `serve_args` + workload in one entry; use multiple `benchmark_params` rows for size/step sweeps under the same server config.
|
||||
|
||||
**Perf local command (diffusion example):**
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN
|
||||
pytest -s -v dfx/perf/scripts/run_diffusion_benchmark.py \
|
||||
--test-config-file dfx/perf/tests/test_<model>_vllm_omni.json
|
||||
```
|
||||
|
||||
**Clarify in the test plan** which L4 pillars you deliver: `Function only` | `Function + Perf` | `Function + Accuracy + Perf`.
|
||||
|
||||
See [references/test-routing.md](references/test-routing.md) **L4 nightly pillars** for Buildkite step patterns.
|
||||
|
||||
### Step 5: Wire Buildkite (when CI must run the new test)
|
||||
|
||||
If the test is not already collected by an existing pipeline command (for example, L1 tests marked `core_model and cpu` are already covered by the **Simple Unit Test** step in ready/merge), update the appropriate pipeline under `.buildkite/`:
|
||||
|
||||
| Test level | Edit this file | Typical trigger / intent |
|
||||
|------------|----------------|---------------------------|
|
||||
| **L1** and **L2** | [`.buildkite/test-ready.yml`](../../../.buildkite/test-ready.yml) | PR **ready** label; L1 CPU + L2 GPU/basic e2e (steps labeled **Omni ·**, **TTS ·**, **Diffusion ·**) |
|
||||
| **L3** | [`.buildkite/test-merge.yml`](../../../.buildkite/test-merge.yml) | Post-merge; `advanced_model` integration per model type |
|
||||
| **L4** | [`.buildkite/test-nightly.yml`](../../../.buildkite/test-nightly.yml) | Nightly; grouped by model type (see below) |
|
||||
| **Invalid param / reliability (weekly)** | [`.buildkite/test-weekly.yml`](../../../.buildkite/test-weekly.yml) | Weekly; `tests/dfx/reliability/invalid_param_test/` — **not** L1–L4 e2e pipelines |
|
||||
|
||||
**Level-specific Buildkite delivery (follow the user’s requested level):**
|
||||
|
||||
| User asks for | Document / edit **only** | Do **not** add by default |
|
||||
|---------------|---------------------------|---------------------------|
|
||||
| **L4** (`*_expansion.py`, `full_model`) | `test-nightly.yml` — append file to the matching nightly shard (X2I / X2V / Omni / TTS) **or** note it is already collected by an existing `-m` / `-k` sweep | `test-merge.yml` (L3 / `advanced_model`) or `test-ready.yml` (L2) |
|
||||
| **L3** | `test-merge.yml` with `advanced_model` + `source_file_dependencies` | `test-nightly.yml` unless user also wants nightly |
|
||||
| **L2** | `test-ready.yml` with `core_model` + `source_file_dependencies` | merge / nightly pipelines |
|
||||
| **Invalid param** | `test-weekly.yml` — **Invalid parameters Test** group (`-m "slow and H100"` / `-m "slow and L4"`). Usually **no YAML edit** when appending cases to existing `invalid_param_test` scripts | `test-ready.yml`, `test-merge.yml`, `test-nightly.yml` |
|
||||
|
||||
`test-nightly.yml` shards use **explicit pytest file paths** in `commands` (and PR labels like `diffusion-x2iat-test`); they generally **do not** use `source_file_dependencies`. Only suggest merge/ready YAML when the requested level is L3/L2 (or the user explicitly asks for multi-level CI).
|
||||
|
||||
**`test-nightly.yml` top-level groups** (each uses `full_model` + type marker in pytest `-m`):
|
||||
|
||||
| Group | Type marker | Notes |
|
||||
|-------|-------------|--------|
|
||||
| **Omni Model Test** | `omni` | Function / doc / accuracy / **perf** for Qwen-Omni family |
|
||||
| **TTS Model Test** | `tts` | `-m "full_model and L4 and tts"` (and H100 variants when enabled); **Perf Test** uses `run_benchmark.py` + `test_tts.json` |
|
||||
| **Diffusion X2I(&A&T) Model Test** | `diffusion` | Image/audio/text diffusion; explicit file shards for **Function**; **Perf Test · <Model>** uses `run_diffusion_benchmark.py` + JSON config |
|
||||
| **Diffusion X2V Model Test** | `diffusion` | Video-only (Wan, HunyuanVideo, …); PR label `diffusion-x2v-test`; separate **Perf** steps when in-tree |
|
||||
|
||||
Guidelines when editing YAML:
|
||||
|
||||
- **Match markers and `--run-level`** to the test: L1/L2 in `test-ready.yml` use `core_model` + `omni` / `tts` / `diffusion` as appropriate; L3 merge uses `advanced_model`; L4 nightly uses `full_model` with the same type marker.
|
||||
- **Diffusion L4**: add x2i/x2a/x2t expansion files to an **X2I(&A&T)** step; add x2v expansion files to **X2V** — never rely on a broad `tests/e2e/` sweep alone (shards exist to save GPU and avoid wrong queue).
|
||||
- **`source_file_dependencies` (required for new E2E jobs in `test-ready.yml` and `test-merge.yml`)**: every step under the **`:card_index_dividers: E2E Test`** group **must** declare `source_file_dependencies` so Buildkite only schedules the job when relevant paths change. Omitting it breaks path-based CI skipping for GPU e2e jobs.
|
||||
- **Reuse or extend an existing step** when the new test shares the same marker expression, queue, and timeout; otherwise add a new `steps:` entry with the correct `agents.queue`, `timeout_in_minutes`, and docker/kubernetes plugin blocks consistent with neighboring jobs.
|
||||
- **Job timeout — use `timeout_in_minutes` on the step, not `timeout` before pytest**: when adding or documenting a new E2E job in `test-ready.yml` / `test-merge.yml`, set `timeout_in_minutes` on the step and run `pytest` directly in `commands`. **Do not** wrap pytest in `timeout 40m bash -c "..."` / `timeout 20m bash -c "..."`; Buildkite already enforces the step deadline via `timeout_in_minutes`.
|
||||
- **Never omit `plugins` in skill examples or PR YAML snippets**: H100 E2E steps on `mithril-h100-pool` use the full `kubernetes` block (`resources`, `volumeMounts`, `env` with `HF_TOKEN` secret, `nodeSelector: gpu-h100-sxm`, `devshm` + `hf-cache` volumes). L4/docker steps on `gpu_1_queue` use the full `docker#v5.2.0` block (`shm-size`, `HF_HOME`, volumes). Do not write `# ... plugins unchanged` — paste the complete block from a sibling step.
|
||||
- **Platform forks** (e.g. AMD-ready / AMD-merge) live alongside these files; apply the same level → file mapping for those pipelines when the test is platform-specific.
|
||||
|
||||
#### `source_file_dependencies` for E2E Test jobs (`test-ready.yml`, `test-merge.yml`)
|
||||
|
||||
When adding or splitting a step inside the **E2E Test** group, **always** include `source_file_dependencies` on that step. List paths the job actually depends on — at minimum:
|
||||
|
||||
1. **Test module(s)** — every `tests/e2e/...` file the `pytest` command runs (online and offline if both are in one step).
|
||||
2. **Model implementation** — `vllm_omni/model_executor/models/<family>/` and/or `vllm_omni/diffusion/models/<family>/` as applicable.
|
||||
3. **Stage / input plumbing** (omni & multi-stage TTS) — `vllm_omni/model_executor/stage_input_processors/<family>.py` when present.
|
||||
4. **Deploy / stage YAML** — `vllm_omni/deploy/<config>.yaml` or `vllm_omni/deploy/ci/<config>.yaml` referenced by the test’s `OmniServerParams` / `get_deploy_config_path(...)`.
|
||||
5. **Test helpers** (when changed in the same PR) — `tests/helpers/runtime.py` for new `send_*` client methods; `tests/helpers/assertions.py` for new shared `assert_*` helpers.
|
||||
|
||||
Use **repo-relative paths** (no leading `./`). Prefer **directory** entries (`.../models/bagel/`) when the whole package matters; use **file** entries for single test modules and YAML configs. Mirror a neighboring step for the same model family (e.g. **Diffusion · Bagel Test** in `test-ready.yml`).
|
||||
|
||||
**Example** (L2 ready — diffusion online smoke, H100 kubernetes):
|
||||
|
||||
```yaml
|
||||
- label: "Diffusion · Bagel Test"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_bagel.py
|
||||
- vllm_omni/model_executor/models/bagel/
|
||||
- vllm_omni/diffusion/models/bagel/
|
||||
- vllm_omni/model_executor/stage_input_processors/bagel.py
|
||||
- vllm_omni/deploy/bagel.yaml
|
||||
timeout_in_minutes: 40
|
||||
commands:
|
||||
- |
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_bagel.py -m 'core_model' --run-level 'core_model'
|
||||
agents:
|
||||
queue: "mithril-h100-pool"
|
||||
plugins:
|
||||
- kubernetes:
|
||||
podSpec:
|
||||
containers:
|
||||
- image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 1
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: hf-cache
|
||||
mountPath: /root/.cache/huggingface
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /root/.cache/huggingface
|
||||
- name: HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
nodeSelector:
|
||||
node.kubernetes.io/instance-type: gpu-h100-sxm
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: hf-cache
|
||||
hostPath:
|
||||
path: /mnt/hf-cache
|
||||
type: DirectoryOrCreate
|
||||
```
|
||||
|
||||
**Example** (L3 merge — TTS online + offline, L4 docker):
|
||||
|
||||
```yaml
|
||||
- label: "TTS · Qwen3-TTS Base Test"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen3_tts_base.py
|
||||
- tests/e2e/offline_inference/test_qwen3_tts_base.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
|
||||
- vllm_omni/deploy/qwen3_tts.yaml
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- |
|
||||
export VLLM_LOGGING_LEVEL=DEBUG
|
||||
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
|
||||
pytest -s -v tests/e2e/online_serving/test_qwen3_tts_base.py tests/e2e/offline_inference/test_qwen3_tts_base.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
agents:
|
||||
queue: "gpu_1_queue" # g6.4xlarge instance on AWS, has 1 L4 GPU
|
||||
plugins:
|
||||
- docker#v5.2.0:
|
||||
image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
always-pull: true
|
||||
propagate-environment: true
|
||||
shm-size: "8gb"
|
||||
environment:
|
||||
- "HF_HOME=/fsx/hf_cache"
|
||||
volumes:
|
||||
- "/fsx/hf_cache:/fsx/hf_cache"
|
||||
```
|
||||
|
||||
**Example** (L3 merge — diffusion online + offline, H100 kubernetes):
|
||||
|
||||
```yaml
|
||||
- label: "Diffusion · Qwen Image Test"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen_image.py
|
||||
- tests/e2e/offline_inference/test_qwen_image.py
|
||||
- vllm_omni/diffusion/models/qwen_image/
|
||||
timeout_in_minutes: 40
|
||||
commands:
|
||||
- |
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v \
|
||||
tests/e2e/online_serving/test_qwen_image.py \
|
||||
tests/e2e/offline_inference/test_qwen_image.py \
|
||||
-m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
agents:
|
||||
queue: "mithril-h100-pool"
|
||||
plugins:
|
||||
- kubernetes:
|
||||
podSpec:
|
||||
containers:
|
||||
- image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 1
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: hf-cache
|
||||
mountPath: /root/.cache/huggingface
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /root/.cache/huggingface
|
||||
- name: HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
nodeSelector:
|
||||
node.kubernetes.io/instance-type: gpu-h100-sxm
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: hf-cache
|
||||
hostPath:
|
||||
path: /mnt/hf-cache
|
||||
type: DirectoryOrCreate
|
||||
```
|
||||
|
||||
When extending an **existing** E2E step to run an additional test file, **append** that file (and any new model/deploy paths) to `source_file_dependencies` on the same step. When authoring a PR, mention the dependency list in the **Buildkite change** section of your test plan.
|
||||
|
||||
#### L3 / L4: validating Buildkite from a feature branch (root `pipeline.yml`)
|
||||
|
||||
Production behavior is driven by [`.buildkite/pipeline.yml`](../../../.buildkite/pipeline.yml): it builds the CI image, then uploads **L2** (`test-ready.yml` on non-`main`), **L3** (`test-merge.yml` when `build.branch == "main"`), and **L4** (`test-nightly.yml` when `build.env("NIGHTLY") == "1"`).
|
||||
|
||||
When you add or debug **L3** or **L4** tests and need those child pipelines to run **off `main`** or **without** `NIGHTLY=1`, apply **temporary** edits (revert before merge):
|
||||
|
||||
1. **Point the upload at the right file (optional shortcut)**
|
||||
In the **Upload Ready Pipeline** step, the command is `buildkite-agent pipeline upload .buildkite/test-ready.yml`. For a one-off run you may change that path to `.buildkite/test-merge.yml` (L3) or `.buildkite/test-nightly.yml` (L4), and adjust the step’s `if:` so it does not conflict with the other upload steps (avoid double-uploading unless intentional).
|
||||
|
||||
2. **L3 — merge pipeline**
|
||||
The gate `if: build.branch == "main"` lives on the **Upload Merge Pipeline** step in `pipeline.yml` (not inside `test-merge.yml`). **Comment out that `if` line** so `test-merge.yml` is uploaded on your feature branch. Alternatively rely on step (1) instead of the dedicated merge upload step.
|
||||
|
||||
3. **L4 — nightly pipeline**
|
||||
**Comment out** `if: build.env("NIGHTLY") == "1"` on the **Upload Nightly Pipeline** step in `pipeline.yml` so the nightly definition is uploaded without setting the env var.
|
||||
Child steps in [`test-nightly.yml`](../../../.buildkite/test-nightly.yml) often repeat `if: build.env("NIGHTLY") == "1"`; **comment out those lines on the steps you need to run**, otherwise they will still be skipped after upload.
|
||||
|
||||
### Step 6: Run Tests
|
||||
|
||||
Pick one command path:
|
||||
|
||||
- **Quick local regression (preferred first)**: single file or `file.py::test_name`
|
||||
- **CI-like level run**: marker expression + `--run-level`
|
||||
|
||||
Full templates live in [references/test-routing.md](references/test-routing.md). **After authoring tests, always emit concrete commands** (see **Output Format**).
|
||||
|
||||
**Typical copy-paste examples** (run from repo `tests/` directory; requires matching vLLM/vllm-omni install and hardware):
|
||||
|
||||
| Area | Example |
|
||||
|------|---------|
|
||||
| **Omni offline L2** | `pytest -s -v e2e/offline_inference/test_qwen2_5_omni.py -m "core_model and omni and not cpu" --run-level=core_model` |
|
||||
| **Omni online L2** | `pytest -s -v e2e/online_serving/test_qwen3_omni.py -m "core_model and omni" --run-level=core_model` |
|
||||
| **TTS online L2** | `pytest -s -v e2e/online_serving/test_qwen3_tts_base.py -m "core_model and tts" --run-level=core_model` |
|
||||
| **Diffusion X2I L2 online** | `pytest -s -v e2e/online_serving/test_qwen_image.py -m "core_model and diffusion" --run-level=core_model` |
|
||||
| **Diffusion X2V L2 online** | `pytest -s -v e2e/online_serving/test_wan22_t2v.py -m "core_model and diffusion" --run-level=core_model` |
|
||||
| **Diffusion X2I L4 nightly** | `pytest -s -v e2e/online_serving/test_qwen_image_expansion.py -m "full_model and diffusion and H100" --run-level=full_model` |
|
||||
| **Diffusion X2I L4 perf (nightly)** | `pytest -s -v dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file dfx/perf/tests/test_qwen_image_vllm_omni.json` |
|
||||
| **Diffusion X2V L4 nightly** | `pytest -s -v e2e/online_serving/test_wan22_expansion.py -m "full_model and cuda" --run-level=full_model` |
|
||||
| **Invalid param (weekly H100)** | `pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and H100"` |
|
||||
| **Invalid param (weekly L4)** | `pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and L4"` |
|
||||
| **L1 CPU** | `pytest -s -v -m "core_model and cpu"` |
|
||||
|
||||
**Prerequisites to mention when relevant**: GPU model (e.g. L4 vs H100), `HF_HOME` / token for hub weights, module-level `skipif` (NPU/XPU-only gaps), and whether CI already collects the path (e.g. `test_*_expansion.py` glob in `test-nightly.yml`).
|
||||
|
||||
### Step 7: Validate Result Quality
|
||||
|
||||
Before finishing:
|
||||
|
||||
- Is the new assertion directly tied to the bug/feature contract?
|
||||
- Are API calls made via **`runtime.py` `send_*_request`** (not inline HTTP/SDK in the test file)?
|
||||
- Is the **general** `assert_*` bundled inside `send_*_request` (not duplicated in the test)?
|
||||
- Is each scenario a **separate `test_*` function** (no `case_id` / `if param.id` branching across cases)?
|
||||
- **Fixture scope**: default **`omni_server` / `omni_runner`** (module); **`_function`** variants only when each test must restart the instance?
|
||||
- Is the test deterministic (no fragile timing/network coupling)?
|
||||
- Is runtime appropriate for the selected level?
|
||||
- Are markers and `--run-level` consistent?
|
||||
- **Invalid-param cases** in `tests/dfx/reliability/invalid_param_test/` (not e2e), using `send_*_http_request` + `err_code`, with `pytest.mark.slow` and correct `H100` / `L4` hardware mark?
|
||||
|
||||
## Output Format
|
||||
|
||||
When completing a request, return:
|
||||
|
||||
1. **Test plan** (level, markers, file target, **L4 pillar(s)** if applicable: Function / Accuracy / Perf / Doc; **or** **Invalid param** pillar with target `invalid_param_test/test_invalid_<area>.py`; module basename for e2e: `test_{slug}.py` / `test_{slug}_expansion.py`)
|
||||
2. **Generated/updated test file(s)** — model-specific constants and request payloads stay **inside** those test modules (**never** `tests/helpers/{slug}.py`). Extend **`tests/helpers/runtime.py`** / **`tests/helpers/assertions.py`** only when new **repo-wide** `send_*` or shared assert helpers are required (do not leave ad-hoc HTTP or `_assert_*` in test modules). **If L4 Perf was requested**, also list **`tests/dfx/perf/tests/*.json`**. **If invalid-param cases were requested**, list the **`invalid_param_test/` script** and parametrized `test_*` / new `id=` rows — **not** e2e paths.
|
||||
3. **Buildkite change** — **match the requested level and pillar**:
|
||||
- **L4 Function**: `test-nightly.yml` **Function Test** shard (file list / note existing sweep). **No** `test-merge.yml` unless user also asked for L3.
|
||||
- **L4 Perf**: `test-nightly.yml` **Perf Test · <Model>** step (new step or extend commands); include env exports + artifact upload pattern from a sibling perf job.
|
||||
- **L4 Accuracy**: `test-nightly.yml` **Accuracy Test** step under the same model-type group.
|
||||
- **L3**: `test-merge.yml` + full `source_file_dependencies` + `agents` + `plugins`.
|
||||
- **L2**: `test-ready.yml` + same E2E block requirements.
|
||||
- **Invalid param**: `test-weekly.yml` — note **Invalid parameters Test · H100/L4** group; usually **no YAML change** when only extending existing scripts.
|
||||
If the user asked only for **L4 functional cases**, state explicitly that **Perf / Accuracy / Invalid param** were not included (offer to add if needed). If the plan mixes **success e2e** and **invalid param**, split deliverables across e2e vs `invalid_param_test/` and call out both CI files.
|
||||
4. **Run commands (required)** — always include, in fenced `bash` blocks:
|
||||
- **Local — whole file**: `cd tests` then `pytest -s -v <path> …`
|
||||
- **Local — single test** (optional but preferred when the change is one function): `pytest -s -v path::test_func …`
|
||||
- **CI-like** (when not L1 CPU): the same **marker + `--run-level`** pairing the level uses (see [references/test-routing.md](references/test-routing.md) and **Step 6** table above)
|
||||
- **Prerequisites** (one line): e.g. “Requires CUDA L4 + weights cached locally”; “Requires H100 + `HF_TOKEN`”; “Syntax-only validation when vllm is not installed locally”
|
||||
5. **Result summary** (pass/fail if you executed tests; if not executed, state that explicitly and what the user should run)
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- Marker and command routing: [references/test-routing.md](references/test-routing.md)
|
||||
- CI pipelines (vllm-omni): [test-ready.yml](../../../.buildkite/test-ready.yml) (L1/L2), [test-merge.yml](../../../.buildkite/test-merge.yml) (L3), [test-nightly.yml](../../../.buildkite/test-nightly.yml) (L4), [test-weekly.yml](../../../.buildkite/test-weekly.yml) (invalid param / reliability)
|
||||
- L4 documentation example tests (naming, extraction vs copied scripts, output dirs, skips): [docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md](../../../docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md) — see also [PR #1910](https://github.com/vllm-project/vllm-omni/pull/1910)
|
||||
@@ -0,0 +1,372 @@
|
||||
# Test Routing Reference
|
||||
|
||||
Use this reference to map testing goals to levels, markers, and runnable commands.
|
||||
|
||||
**Repo paths** (`.buildkite/`, `docs/contributing/…`): link with repo-relative paths from this file (e.g. `../../../../.buildkite/test-ready.yml`, `../../../../docs/contributing/ci/CI_5levels.md`).
|
||||
|
||||
## Model-centric e2e filename convention (L2–L4)
|
||||
|
||||
When generating a **new** test module tied to a **specific model** under `tests/e2e/offline_inference/` or `tests/e2e/online_serving/`:
|
||||
|
||||
| Level | Pattern | Example (`Qwen/Qwen2.5-Omni-7B`) |
|
||||
|-------|---------|----------------------------------|
|
||||
| **L2**, **L3** | `test_{lowercase_model_slug}.py` | `test_qwen2_5_omni.py` |
|
||||
| **L4** | `test_{lowercase_model_slug}_expansion.py` | `test_qwen2_5_omni_expansion.py` |
|
||||
|
||||
**Slug rules for `{lowercase_model_slug}`** (canonical: [SKILL.md](../SKILL.md) § *Naming: generated test module files*):
|
||||
|
||||
1. Start from the HuggingFace-style id (e.g. `Qwen/Qwen2.5-Omni-7B`), but **do not** put the org into the filename: use the **repo segment only** (`Qwen2.5-Omni-7B`), not `Qwen_...` / `qwen_qwen2_5_...`.
|
||||
2. **Lowercase**; replace `.`, `-`, and whitespace with a single `_` (e.g. `Qwen2.5-Omni-7B` → `qwen2_5_omni`). **Omit** trailing size tokens such as `7b` / `30b` in the basename when a single file covers that model line in the directory (matches `test_qwen2_5_omni.py` in-tree).
|
||||
3. If two checkpoints in the same folder need separate modules, add a **minimal** disambiguator (e.g. `_7b` vs `_3b`) **only then**.
|
||||
4. **L1** unit tests are **not** bound to this pattern; use `tests/<area>/test_<feature>.py` as today.
|
||||
|
||||
## Model type markers (`omni` / `tts` / `diffusion`)
|
||||
|
||||
Every model-centric e2e test must declare **exactly one** type marker:
|
||||
|
||||
| Marker | Model family | Typical APIs |
|
||||
|--------|--------------|--------------|
|
||||
| `pytest.mark.omni` | Multimodal LLM (Qwen-Omni, …) | `send_omni_request`, `omni_runner`, stage YAML |
|
||||
| `pytest.mark.tts` | TTS / speech synthesis | `send_audio_speech_request`, `/v1/audio/speech` |
|
||||
| `pytest.mark.diffusion` | Diffusion generative models | `send_diffusion_request`, `send_video_diffusion_request`, `send_images_generations_http_request`, `OmniDiffusionSamplingParams` |
|
||||
|
||||
**Diffusion** uses one marker for all output modalities. Nightly CI splits diffusion by **group**, not by extra markers:
|
||||
|
||||
| Nightly group (`test-nightly.yml`) | Diffusion scope | Examples |
|
||||
|-----------------------------------|-----------------|----------|
|
||||
| **Diffusion X2I(&A&T) Model Test** | x2**i** / x2**a** / x2**t** — image, audio, text (non-video) | Qwen-Image*, BAGEL, FLUX, SD3, Z-Image, LongCat, DreamZero |
|
||||
| **Diffusion X2V Model Test** | x2**v** — video only | Wan2.2, HunyuanVideo 1.5, Wan VACE |
|
||||
|
||||
PR labels for selective nightly runs: `diffusion-x2iat-test`, `diffusion-x2v-test` (plus `nightly-test` / `NIGHTLY=1`).
|
||||
|
||||
## Level and Marker Mapping
|
||||
|
||||
| Goal | Suggested Level | Marker baseline | Typical location |
|
||||
|------|------------------|-----------------|------------------|
|
||||
| Unit logic, regression on pure code path | L1 | `core_model and cpu` | `tests/<component>/test_*.py` |
|
||||
|
||||
### L1 unit tests — mocking (`mocker`, not `unittest.mock`)
|
||||
|
||||
L1 modules run in **Simple Test** (`test-ready.yml` / `test-merge.yml`: `-m 'core_model and cpu'`). When isolation requires doubles:
|
||||
|
||||
- **Use** the **`mocker`** fixture (`pytest-mock`): `mocker.patch`, `mocker.spy`, `mocker.Mock`, `mocker.MagicMock`, `mocker.AsyncMock`.
|
||||
- **Use** **`monkeypatch`** for env vars and simple `setattr` / `delattr` without mock objects.
|
||||
- **Do not use** `unittest.mock` — no `from unittest.mock import ...`, no `@patch`, no `with patch(...)`.
|
||||
|
||||
```python
|
||||
# Good (L1)
|
||||
def test_handler(mocker):
|
||||
mocker.patch("vllm_omni.pkg.fn", return_value=0)
|
||||
|
||||
# Bad (L1)
|
||||
from unittest.mock import patch
|
||||
|
||||
@patch("vllm_omni.pkg.fn")
|
||||
def test_handler(mock_fn): ...
|
||||
```
|
||||
|
||||
E2E (L2+) should not rely on mocks unless documenting a rare exception; prefer real (or lightweight) execution paths.
|
||||
| Basic integration/e2e | L2 | `core_model` + **one of** `omni` / `tts` / `diffusion` | `tests/e2e/...` |
|
||||
| Advanced integration | L3 | `advanced_model` + type marker | `tests/e2e/...` |
|
||||
| Full function / nightly | L4 | `full_model` (nightly) + type marker | **Function:** `tests/e2e/*_expansion.py`; **Perf:** `tests/dfx/perf/tests/*.json`; **Accuracy:** `tests/e2e/accuracy/` |
|
||||
| Invalid HTTP / param validation | Weekly (dfx) | `pytest.mark.slow` + type marker + `H100` or `L4` | `tests/dfx/reliability/invalid_param_test/test_invalid_*.py` |
|
||||
|
||||
## Marker Selection Rules
|
||||
|
||||
1. **Level** (pick one):
|
||||
- `core_model` — L1/L2 (`test-ready.yml`)
|
||||
- `advanced_model` — L3 (`test-merge.yml`)
|
||||
- `full_model` — L4 nightly (`test-nightly.yml`); some expansion tests still carry both `advanced_model` and `full_model` during migration
|
||||
2. **Model type** (pick one): `omni`, `tts`, or `diffusion`
|
||||
3. **Cross-cutting** when relevant: `parallel`, `cache`, `example`, `benchmark`
|
||||
4. **Hardware**: `cpu`, `cuda`, `rocm`, `npu`, `L4`, `H100`, `distributed_cuda`, …
|
||||
5. Multi-card: `@hardware_test(...)` in `tests/helpers/mark.py`
|
||||
|
||||
## Command Templates
|
||||
|
||||
### Quick local checks
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
pytest -s -v test_xxxx.py
|
||||
```
|
||||
|
||||
### L1
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
pytest -s -v -m "core_model and cpu"
|
||||
```
|
||||
|
||||
### L2
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
pytest -s -v -m "core_model and not cpu" --run-level=core_model
|
||||
```
|
||||
|
||||
### L3 (merge)
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
pytest -s -v -m "advanced_model" --run-level=advanced_model
|
||||
```
|
||||
|
||||
### L4 (nightly)
|
||||
|
||||
**Function** (e2e expansion — default when user asks for “L4 functional cases”):
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
pytest -s -v e2e/online_serving/test_qwen_image_expansion.py -m "full_model and diffusion and H100" --run-level=full_model
|
||||
```
|
||||
|
||||
**Perf** (dfx benchmark harness — separate pillar; not `omni_server` fixtures):
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN
|
||||
pytest -s -v dfx/perf/scripts/run_diffusion_benchmark.py \
|
||||
--test-config-file dfx/perf/tests/test_qwen_image_vllm_omni.json
|
||||
```
|
||||
|
||||
Broad marker sweep (when no explicit file shard):
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
pytest -s -v -m "full_model" --run-level=full_model
|
||||
```
|
||||
|
||||
## L4 nightly pillars (Function / Accuracy / Perf)
|
||||
|
||||
Nightly **L4** for a model is often **multiple jobs** in `test-nightly.yml`, not a single pytest module:
|
||||
|
||||
| Pillar | User says | Deliver | Nightly step |
|
||||
|--------|-----------|---------|--------------|
|
||||
| **Function** | “L4 functional cases” / “functional” / `*_expansion.py` | `tests/e2e/.../test_<model>_expansion.py` | `· Function Test with H100/L4` |
|
||||
| **Perf** | “performance” / “perf” / “benchmark” / “full L4” | `tests/dfx/perf/tests/test_<model>_vllm_omni.json` + runner | `· Perf Test · <Model>` |
|
||||
| **Accuracy** | “accuracy” / “similarity” | `tests/e2e/accuracy/test_<model>*.py` | `· Accuracy Test` |
|
||||
|
||||
**Rule:** “L4 functional cases” → **Function only** by default. State that Perf/Accuracy are separate pillars; add them only when requested.
|
||||
|
||||
**Diffusion perf JSON** (`tests/dfx/perf/tests/test_<slug>_vllm_omni.json`): array of objects with `test_name`, `server_params` (`model`, `serve_args`), `benchmark_params[]`, and per-workload **`baseline`** (`throughput_qps`, `latency_mean`, `peak_memory_mb_mean`). Copy structure from `test_qwen_image_vllm_omni.json` in-tree.
|
||||
|
||||
**Diffusion perf nightly step** (under **Diffusion X2I(&A&T)**; copy full `kubernetes` plugins from `· Perf Test · Qwen-Image`):
|
||||
|
||||
```yaml
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · <Model-Display-Name>"
|
||||
key: nightly-diffusion-x2iat-performance-<slug>
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN
|
||||
- export CACHE_DIT_VERSION=1.3.0
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_<slug>_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
agents:
|
||||
queue: "mithril-h100-pool"
|
||||
plugins:
|
||||
# … full kubernetes block (often multi-GPU) — copy from Perf Test · Qwen-Image
|
||||
```
|
||||
|
||||
| Model type | Perf runner | Config example |
|
||||
|------------|-------------|----------------|
|
||||
| Diffusion X2I/X2V | `dfx/perf/scripts/run_diffusion_benchmark.py` | `test_qwen_image_vllm_omni.json` |
|
||||
| TTS | `dfx/perf/scripts/run_benchmark.py` | `test_tts.json` (`BENCHMARK_DIR`) |
|
||||
| Omni | `run_benchmark.py` / omni configs in-tree | `test_omni*.json` |
|
||||
|
||||
Do **not** put throughput/latency baselines inside `test_*_expansion.py` — that belongs in the dfx perf JSON + nightly Perf job.
|
||||
|
||||
## Invalid parameter validation (weekly / dfx)
|
||||
|
||||
When the user’s test plan includes **invalid parameter validation**, **invalid request bodies**, or **HTTP 4xx** validation against a live server:
|
||||
|
||||
1. **Do not** author these in `tests/e2e/online_serving/test_*.py` or `*_expansion.py`. **Move** any drafted `test_*` into `tests/dfx/reliability/invalid_param_test/`.
|
||||
2. **Pick script by route** (extend in-tree file):
|
||||
|
||||
| Route family | Script |
|
||||
|--------------|--------|
|
||||
| Omni chat / WS video·realtime | `test_invalid_omni_chat.py` |
|
||||
| `/v1/audio/speech` (+ stream / batch / voices) | `test_invalid_audio_speech.py` |
|
||||
| Audio diffusion | `test_invalid_audio_diffusion.py` |
|
||||
| `/v1/images/generations` | `test_invalid_image_generation.py` |
|
||||
| `/v1/images/edits` | `test_invalid_image_editing.py` |
|
||||
| `/v1/videos*` | `test_invalid_video_generation.py` |
|
||||
| Sleep / wakeup / server control | `test_invalid_server_control.py` |
|
||||
|
||||
3. **Style:** `pytestmark = [pytest.mark.slow, pytest.mark.<type>]`; `_PARAMS` + `hardware_marks`; `send_*_http_request` with `err_code` + `err_message`; parametrized `body_spec` rows with `id=`; `_minimal_*_json()` helpers; `_SKIP_ISSUE_3649` when tracked in [#3649](https://github.com/vllm-project/vllm-omni/issues/3649).
|
||||
4. **CI:** [`.buildkite/test-weekly.yml`](../../../../.buildkite/test-weekly.yml) group **Reliability Test - Invalid parameters Test** — **not** ready/merge/nightly.
|
||||
|
||||
```bash
|
||||
# Weekly H100 (diffusion / omni / video invalid-param)
|
||||
cd tests
|
||||
pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and H100"
|
||||
|
||||
# Weekly L4 (e.g. Qwen3-TTS speech invalid-param)
|
||||
pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and L4"
|
||||
|
||||
# Single script / case
|
||||
pytest -s -v dfx/reliability/invalid_param_test/test_invalid_image_generation.py::test_images_generations_invalid_requests -m "slow and H100"
|
||||
```
|
||||
|
||||
**Trigger:** `WEEKLY=1` or PR label `weekly-test`. Appending cases to existing scripts usually needs **no YAML edit** (directory sweep). No `source_file_dependencies` on weekly steps.
|
||||
|
||||
### Platform-targeted examples
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
pytest -s -v -m "core_model and distributed_cuda and L4" --run-level=core_model
|
||||
```
|
||||
|
||||
### Concrete e2e paths (common in-tree)
|
||||
|
||||
Paths are relative to `tests/` after `cd tests`.
|
||||
|
||||
| Scenario | Example command |
|
||||
|----------|------------------|
|
||||
| **Omni** offline L2 | `pytest -s -v e2e/offline_inference/test_qwen2_5_omni.py -m "core_model and omni and not cpu" --run-level=core_model` |
|
||||
| **Omni** offline L2 — one test | `pytest -s -v e2e/offline_inference/test_qwen2_5_omni.py::test_text_to_text -m "core_model and omni and not cpu" --run-level=core_model` |
|
||||
| **Omni** online L2 | `pytest -s -v e2e/online_serving/test_qwen3_omni.py -m "core_model and omni" --run-level=core_model` |
|
||||
| **Omni** offline L2 (Qwen3.5-9B VL) | `pytest -s -v e2e/offline_inference/test_qwen3_5_9b.py -m "core_model and omni and not cpu" --run-level=core_model` |
|
||||
| **TTS** online L2 | `pytest -s -v e2e/online_serving/test_qwen3_tts_base.py -m "core_model and tts" --run-level=core_model` |
|
||||
| **TTS** online L2 — one test | `pytest -s -v e2e/online_serving/test_qwen3_tts_base.py::test_text_to_audio_001 -m "core_model and tts" --run-level=core_model` |
|
||||
| **Diffusion X2I** offline L2 | `pytest -s -v e2e/offline_inference/test_t2i_model.py -m "core_model and diffusion and not cpu" --run-level=core_model` |
|
||||
| **Diffusion X2I** L2 online smoke | `pytest -s -v e2e/online_serving/test_qwen_image.py e2e/online_serving/test_bagel.py -m "core_model and diffusion" --run-level=core_model` |
|
||||
| **Diffusion X2V** L2 online smoke | `pytest -s -v e2e/online_serving/test_wan22_t2v.py -m "core_model and diffusion" --run-level=core_model` |
|
||||
| **Diffusion X2I** L4 online expansion (H100) | `pytest -s -v e2e/online_serving/test_qwen_image_expansion.py -m "full_model and diffusion and H100" --run-level=full_model` |
|
||||
| **Diffusion X2I** L4 perf (nightly) | `pytest -s -v dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file dfx/perf/tests/test_qwen_image_vllm_omni.json` |
|
||||
| **Diffusion X2I** L4 online expansion (Edit) | `pytest -s -v e2e/online_serving/test_qwen_image_edit_expansion.py -m "full_model and diffusion and H100" --run-level=full_model` |
|
||||
| **Diffusion X2V** L4 nightly | `pytest -s -v e2e/online_serving/test_wan22_expansion.py -m "full_model and cuda" --run-level=full_model` |
|
||||
| **Invalid param** weekly H100 | `pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and H100"` |
|
||||
| **Invalid param** weekly L4 | `pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and L4"` |
|
||||
|
||||
### Agent / author completion checklist
|
||||
|
||||
When adding or modifying tests, do not stop at “where the file lives” — also deliver:
|
||||
|
||||
1. **Local**: `cd tests` + `pytest -s -v <path>` (and `path::test_func` when a single case is enough).
|
||||
2. **CI-like**: marker string + `--run-level` matching the pipeline (`core_model` / `advanced_model` / `full_model`).
|
||||
3. **L1 mocks**: `mocker` / `monkeypatch` only; no `unittest.mock`.
|
||||
4. **API client + assert placement**: **General** validation → implement in `assertions.py`, call **inside** `send_*_request` in `runtime.py`; tests call **`send_*_request` only**. **Special** case validation → `assert_*` in `assertions.py`, called **in the test** after `send_*_request`. Low-level `send_*_http_request` is for negative/dfx tests (`err_code`), not ordinary L2+ success e2e.
|
||||
5. **Shared assertions**: logic in **`tests/helpers/assertions.py`**; general checks inside `send_*_request`; special checks only in the test. No `_assert_*` in `test_*.py`.
|
||||
6. **One case → one `test_*`**: function name reflects what is validated; no `if case_id == ...` mega-test merging multiple scenarios.
|
||||
7. **Fixture scope**: default **`omni_server` + `openai_client`** / **`omni_runner` + `omni_runner_handler`** (module). Use **`omni_server_function` + `openai_client_function`** / **`omni_runner_function` + `omni_runner_handler_function`** only when each `test_*` must spawn a fresh instance. Parametrize name must match fixture (`omni_server` vs `omni_server_function`).
|
||||
8. **Type marker**: `omni`, `tts`, or `diffusion` on every model e2e module.
|
||||
9. **Diffusion L4 Function**: wire `*_expansion.py` into **X2I(&A&T)** or **X2V** **Function Test** in **`test-nightly.yml` only** — do not add `test-merge.yml` unless the user also requested L3.
|
||||
10. **Diffusion L4 Perf** (only when requested): add `tests/dfx/perf/tests/test_<slug>_vllm_omni.json` + **Perf Test · <Model>** step (artifact upload); not part of “L4 functional cases” by default.
|
||||
11. **E2E Buildkite (L2/L3 only)**: `test-ready.yml` / `test-merge.yml` steps need **`source_file_dependencies`** + full **`agents` + `plugins`**. **L4 nightly** uses explicit file lists or perf scripts in `test-nightly.yml` (no merge job).
|
||||
12. **Invalid param**: cases in **`tests/dfx/reliability/invalid_param_test/`** (route-matching script), `send_*_http_request` + `err_code`, `pytest.mark.slow` + `H100`/`L4`; CI = **`test-weekly.yml`** only — do not put in e2e or nightly.
|
||||
13. **Prerequisites**: GPU tier, HF cache/token, and any module `skipif` / platform-only YAML.
|
||||
|
||||
## Buildkite pipeline mapping
|
||||
|
||||
| Level | Repo file | Model-type grouping |
|
||||
|-------|-----------|---------------------|
|
||||
| L1, L2 | [`.buildkite/test-ready.yml`](../../../../.buildkite/test-ready.yml) | Steps prefixed **Omni ·**, **TTS ·**, **Diffusion ·** under **E2E Test** — **`source_file_dependencies` required** |
|
||||
| L3 | [`.buildkite/test-merge.yml`](../../../../.buildkite/test-merge.yml) | Per-model E2E steps; `-m "advanced_model and …"` — **`source_file_dependencies` required** |
|
||||
| L4 | [`.buildkite/test-nightly.yml`](../../../../.buildkite/test-nightly.yml) | **Omni / TTS / Diffusion X2I(&A&T) / X2V** — each group may have **Function**, **Accuracy**, **Perf**, **Doc** steps |
|
||||
| Invalid param (weekly) | [`.buildkite/test-weekly.yml`](../../../../.buildkite/test-weekly.yml) | **Invalid parameters Test · H100** / **· L4** — sweeps `tests/dfx/reliability/invalid_param_test/` |
|
||||
|
||||
### `source_file_dependencies` (E2E Test only — ready & merge)
|
||||
|
||||
Required on each step inside the **E2E Test** group in `test-ready.yml` and `test-merge.yml`. Typical entries:
|
||||
|
||||
| Category | Path pattern |
|
||||
|----------|----------------|
|
||||
| Test module | `tests/e2e/online_serving/test_<slug>.py`, `tests/e2e/offline_inference/test_<slug>.py` |
|
||||
| API client helpers | `tests/helpers/runtime.py` (when the e2e job uses newly added `send_*` methods) |
|
||||
| Shared assert helpers | `tests/helpers/assertions.py` (when the e2e job uses newly added `assert_*` helpers) |
|
||||
| AR / omni model | `vllm_omni/model_executor/models/<family>/` |
|
||||
| Diffusion model | `vllm_omni/diffusion/models/<family>/` |
|
||||
| Stage processor | `vllm_omni/model_executor/stage_input_processors/<family>.py` |
|
||||
| Deploy config | `vllm_omni/deploy/<name>.yaml` or `vllm_omni/deploy/ci/<name>.yaml` |
|
||||
|
||||
Copy the dependency block from the closest in-tree E2E step for the same model family. Update the list whenever the pytest command or server YAML changes.
|
||||
|
||||
### E2E `agents` + `plugins` blocks (required — do not truncate)
|
||||
|
||||
When generating or documenting E2E steps, **always include the complete `plugins` section**. Set **`timeout_in_minutes`** on the step for the job deadline; run **`pytest` directly** in `commands` (no `timeout 40m bash -c` wrapper around pytest).
|
||||
|
||||
Common patterns in `test-ready.yml` / `test-merge.yml`:
|
||||
|
||||
| Queue | Plugin | Required fields |
|
||||
|-------|--------|-----------------|
|
||||
| `mithril-h100-pool` | `kubernetes` | `resources.limits.nvidia.com/gpu`, `volumeMounts` (`devshm`, `hf-cache`), `env` (`HF_HOME`, `HF_TOKEN` via `secretKeyRef`), `nodeSelector: gpu-h100-sxm`, `volumes` (`devshm` emptyDir, `hf-cache` hostPath `/mnt/hf-cache`) |
|
||||
| `gpu_1_queue` / `gpu_4_queue` | `docker#v5.2.0` | `image`, `always-pull`, `propagate-environment`, `shm-size: "8gb"`, `environment: HF_HOME`, `volumes: /fsx/hf_cache` (add `HF_TOKEN` when the step needs hub access) |
|
||||
|
||||
**H100 kubernetes template** (copy from **Diffusion · Bagel Test** / **Diffusion · Qwen Image Test** in-tree):
|
||||
|
||||
```yaml
|
||||
agents:
|
||||
queue: "mithril-h100-pool"
|
||||
plugins:
|
||||
- kubernetes:
|
||||
podSpec:
|
||||
containers:
|
||||
- image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 1
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: hf-cache
|
||||
mountPath: /root/.cache/huggingface
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /root/.cache/huggingface
|
||||
- name: HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
nodeSelector:
|
||||
node.kubernetes.io/instance-type: gpu-h100-sxm
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: hf-cache
|
||||
hostPath:
|
||||
path: /mnt/hf-cache
|
||||
type: DirectoryOrCreate
|
||||
```
|
||||
|
||||
**L4 docker template** (copy from **TTS · Qwen3-TTS Base Test** in `test-merge.yml`):
|
||||
|
||||
```yaml
|
||||
agents:
|
||||
queue: "gpu_1_queue"
|
||||
plugins:
|
||||
- docker#v5.2.0:
|
||||
image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
always-pull: true
|
||||
propagate-environment: true
|
||||
shm-size: "8gb"
|
||||
environment:
|
||||
- "HF_HOME=/fsx/hf_cache"
|
||||
volumes:
|
||||
- "/fsx/hf_cache:/fsx/hf_cache"
|
||||
```
|
||||
|
||||
When extending an existing step (e.g. add `tests/e2e/offline_inference/test_qwen_image.py` to **Diffusion · Qwen Image Test**), update `source_file_dependencies` and `commands` only; **keep the existing `agents` + `plugins` block unchanged** unless the hardware tier changes.
|
||||
|
||||
The root [`.buildkite/pipeline.yml`](../../../../.buildkite/pipeline.yml) decides **which** child file is uploaded. To run **L3** on a feature branch, comment out `if: build.branch == "main"` on the merge upload step. To run **L4** without `NIGHTLY=1`, comment out the nightly upload step’s `if: build.env("NIGHTLY") == "1"` and the same `if` on relevant steps inside `test-nightly.yml`. Revert such edits before merging.
|
||||
|
||||
Platform-specific pipelines (e.g. AMD) follow the same level → file pairing under `.buildkite/`.
|
||||
|
||||
## Diffusion RFC (#1832) Alignment Tips
|
||||
|
||||
For **X2I(&A&T)** coverage planning:
|
||||
|
||||
- Prioritize high-value feature combinations with minimal case count.
|
||||
- Split into lightweight L1/L2 validation plus L4 `*_expansion.py` under the **X2I** nightly shards.
|
||||
- **X2V** models (Wan, HunyuanVideo) stay in the **X2V** group — do not merge into X2I sweeps.
|
||||
|
||||
If hardware is insufficient, provide an executable reduced case plus a deferred full CI/nightly plan.
|
||||
@@ -11,6 +11,10 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
<tr>
|
||||
<th>Level</th>
|
||||
<th>Scope & Focus</th>
|
||||
<th>Model Coverage Strategy</th>
|
||||
<th>Feature Coverage Strategy</th>
|
||||
<th>Interface Coverage Strategy</th>
|
||||
<th>Tags</th>
|
||||
<th>Time Cost</th>
|
||||
<th>Test Dir</th>
|
||||
<th>Doc</th>
|
||||
@@ -24,6 +28,10 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
<td>Contribution Guideline & PR checklist</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>.github/PULL_REQUEST_TEMPLATE.md <a href="../tests_style/"> Test Style (PR Checklist)</a></td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
@@ -32,6 +40,10 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
<td>CI Failure Description</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td><a href="../failures/"> CI Failures</a></td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
@@ -39,6 +51,10 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
<tr>
|
||||
<td><strong>L1</strong><br>(Unit & Logic)</td>
|
||||
<td>Unit tests for components like entrypoints, models</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td>/</td>
|
||||
<td><code>core_model and cpu</code></td>
|
||||
<td rowspan="2"><15min</td>
|
||||
<td>/tests/{component_name}/test_xxx</td>
|
||||
<td>
|
||||
@@ -50,10 +66,18 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>L2</strong><br>(E2E across models & GPU-required UT)</td>
|
||||
<td>Online & Offline (basic deployment scenarios):<br>dummy, normal inference function (output format, stream), some instance startup UT</td>
|
||||
<td>Online (basic deployment scenarios):<br>dummy, normal inference function (output format, stream), some instance startup UT</td>
|
||||
<td>High-priority models + online basic scenarios + request success validation</td>
|
||||
<td>High-priority features (using random lightweight models)</td>
|
||||
<td>High-priority interfaces (using random lightweight models)</td>
|
||||
<td><code>core_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
|
||||
<td>
|
||||
<strong>Model tests:</strong><br>
|
||||
/tests/e2e/online_serving/test_{model_name}.py<br>
|
||||
/tests/e2e/offline_inference/test_{model_name}.py
|
||||
<strong>Feature tests:</strong><br>
|
||||
/tests/{component_name}/test_xxx<br>
|
||||
<strong>Interface tests:</strong><br>
|
||||
/tests/entrypoints/test_xxx
|
||||
</td>
|
||||
<td>
|
||||
<a href="#chapter-1-l1-l2-level-testing-unit-testing-and-basic-end-to-end-verification">Chapter 1</a><br>
|
||||
@@ -65,10 +89,19 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
<tr>
|
||||
<td><strong>L3</strong><br>(Important Perf & Integration & Accuracy)</td>
|
||||
<td>Online & Offline (multiple deployment scenarios):<br>real model, normal inference function, normal accuracy</td>
|
||||
<td>High/medium-priority models with real weights + online/offline key scenarios + basic accuracy validation</td>
|
||||
<td>Medium-priority features (using random lightweight models)</td>
|
||||
<td>Medium-priority interfaces (using random lightweight models)</td>
|
||||
<td><code>advanced_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
|
||||
<td><30min</td>
|
||||
<td>
|
||||
<strong>Model tests:</strong><br>
|
||||
/tests/e2e/online_serving/test_{model_name}.py<br>
|
||||
/tests/e2e/offline_inference/test_{model_name}.py
|
||||
/tests/e2e/offline_inference/test_{model_name}.py<br>
|
||||
<strong>Feature tests:</strong><br>
|
||||
/tests/{component_name}/test_xxx<br>
|
||||
<strong>Interface tests:</strong><br>
|
||||
/tests/entrypoints/test_xxx
|
||||
</td>
|
||||
<td>
|
||||
<a href="#chapter-2-l3-level-testing-core-integration-performance-and-accuracy-verification">Chapter 2</a><br>
|
||||
@@ -79,18 +112,27 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>L4</strong><br>(Perf & Integration & Accuracy)</td>
|
||||
<td>Online & Offline: full functional scenarios + performance test + doc test</td>
|
||||
<td>Online: full functional scenarios + performance test + doc test + accuracy test</td>
|
||||
<td>High-priority models: function, performance, accuracy, and doc testing<br>Medium/low-priority models: function and doc testing</td>
|
||||
<td>Low-priority features (using random lightweight models)</td>
|
||||
<td>Low-priority interfaces (using random lightweight models)</td>
|
||||
<td><code>full_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
|
||||
<td><3 hour</td>
|
||||
<td>
|
||||
<strong>Full Function:</strong><br>
|
||||
<strong>Model tests:</strong><br>
|
||||
/tests/e2e/online_serving/test_{model_name}_expansion.py<br>
|
||||
/tests/e2e/offline_inference/test_{model_name}_expansion.py<br>
|
||||
<strong>Feature tests:</strong><br>
|
||||
/tests/{component_name}/test_xxx<br>
|
||||
<strong>Interface tests:</strong><br>
|
||||
/tests/entrypoints/test_xxx<br>
|
||||
<strong>Performance:</strong><br>
|
||||
/tests/dfx/perf/tests/test_qwen_omni.json (Omni), test_tts.json (TTS),<br>
|
||||
and /tests/dfx/perf/tests/test_{diffusion_model}_vllm_omni.json (Diffusion)<br>
|
||||
<strong>Doc Test:</strong><br>
|
||||
tests/example/online_serving/test_{model_name}.py<br>
|
||||
tests/example/offline_inference/test_{model_name}.py
|
||||
tests/example/offline_inference/test_{model_name}.py<br>
|
||||
<strong>Accuracy Test:</strong><br>
|
||||
/tests/e2e/accuracy/test_{model_name}.py
|
||||
</td>
|
||||
<td>
|
||||
<a href="#chapter-3-l4-level-testing-full-functionality-performance-and-documentation-testing">Chapter 3</a><br>
|
||||
@@ -101,7 +143,11 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>L5</strong><br>(Stability & Reliability)</td>
|
||||
<td>Online & Offline: long-term stability test + reliability test</td>
|
||||
<td>Online: long-term stability test + reliability test</td>
|
||||
<td>Long-term stability and reliability testing for high-priority models</td>
|
||||
<td>/</td>
|
||||
<td>Invalid-parameter validation for high-priority interfaces</td>
|
||||
<td><code>slow and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
|
||||
<td> Depends on reality </td>
|
||||
<td>
|
||||
<strong>Stability:</strong><br>
|
||||
|
||||
Reference in new issue
Block a user