anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Refactor][CI] Unify skip-ci targeting and mirror_hardwares pipeline presets (#5254)

Browse Source 
Signed-off-by: wangyu <410167048@qq.com>
Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Signed-off-by: [Your Name] [Your Email]
Co-authored-by: Weiming Liao <fayespica@outlook.com>
This commit is contained in:
wangyuandWeiming Liao authored and GitHub committed 2026-07-24 22:28:43 +08:00
1 parent 536daea107
commit a5d9d57adc
38 files changed
+3018 -4490
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/contributing/README.md
+4 -4
View File
@@ -71,7 +71,7 @@ For additional features and advanced configurations, refer to the:
### Testing
vLLM-Omni uses `pytest` to test the codebase.
Please refer to the [test instructions](./ci/test_guide.md) for detailed testing information.
Please refer to the [test instructions](./ci/test_execution_guide.md) for detailed testing information.
!!! warning
Currently, not all unit tests pass when run on CPU platforms. If you don't have access to a GPU platform to run unit tests locally, rely on the continuous integration system to run the tests for now.
@@ -127,11 +127,11 @@ The skill offers two modes:
The precheck covers five PR types: Bug Fix, Performance, New Model, Diffusion Model, and General. Each type has a tailored checklist that validates evidence quality (repro steps, A/B benchmarks, registry entries, etc.). See the [precheck-pr skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/precheck-pr/SKILL.md) for the full checklist.
### Local Test
Please run the L1 and L2 test cases locally first and attach the results before contacting us to add the "ready" label. Please refer to the [test instructions](./ci/test_guide.md) for running the test cases.
Please run the L1 and L2 test cases locally first and attach the results before contacting us to add the "ready" label. Please refer to the [test instructions](./ci/test_execution_guide.md) for running the test cases.
### Automatic skip-ci (docs and pytest skip marks)
On pull requests and `main` pushes, the bootstrap step in [`.buildkite/cuda/pipeline.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/pipeline.yml) runs [`.buildkite/cuda/scripts/upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/scripts/upload_pipeline.py) against the git diff. When every changed file qualifies, **L2 (`ready`) and L3 (`merge-test`) pipelines are not uploaded**, so the default GPU CI jobs are skipped.
On pull requests and `main` pushes, the bootstrap step in [`.buildkite/cuda/pipeline.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/pipeline.yml) runs [`.buildkite/common/scripts/upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py) against the git diff. When every changed file qualifies, **L2 (`ready`) and L3 (`merge-test`) pipelines are not uploaded**, so the default GPU CI jobs are skipped.
| Change per file | Examples |
| --- | --- |
@@ -148,7 +148,7 @@ These PR shapes all trigger skip-ci:
Skip-ci does **not** apply when the diff also touches product code (for example `vllm_omni/`), or when test files change assertions, imports, fixtures, or other non-skip logic. If the diff cannot be resolved (non-PR branches outside `main`), CI runs as usual.
!!! note
Skipping L2/L3 does **not** disable the Docker image build step. Nightly (L4) upload can still run when the PR has a `nightly-test` label or on scheduled `main` builds with `NIGHTLY=1`. See [CI levels](./ci/CI_5levels.md) for how bootstrap skip-ci relates to diff-gated E2E jobs.
Skipping L2/L3 does **not** disable the Docker image build step. Nightly (L4) upload can still run when the PR has a `nightly-test` label or on scheduled `main` builds with `NIGHTLY=1`. Bootstrap child steps live in `bootstrap-upload-steps.yml`; the hook entry step runs `upload_pipeline.py --upload <platform>/bootstrap-upload-steps.yml`, which injects `if` by step `key` from skip-ci before upload. See [CI Settings — Diff-aware CI](./ci/ci_settings.md#diff-aware-ci) and [Test System Overview](./ci/test_system_overview.md).
### Code Quality
docs/contributing/ci/.nav.yaml
+4 -4
View File
@@ -1,6 +1,6 @@
nav:
- CI_5levels.md
- test_system_overview.md
- ci_settings.md
- test_writing_guide.md
- failures.md
- test_guide.md
- test_markers.md
- test_style.md
- test_execution_guide.md
docs/contributing/ci/ci_settings.md
+344
View File
@@ -0,0 +1,344 @@
# CI Settings
This document describes **where** Buildkite YAML lives in the repo, **how each platform organizes CI**, and **how to add a new job**. It does not document agent queues, GPU types, or container plugin details—those belong in infra / preset files (for example `.buildkite/common/ci_mirror_hardwares.yml`).
For CI levels (L1–L5) and triggers, see [Test System Overview](./test_system_overview.md). For test authoring, see [Test Writing Guide](./test_writing_guide.md). For running tests locally or replaying CI jobs, see [Test Execution Guide](./test_execution_guide.md).
## Directory layout
Canonical layout (prefer these paths for new changes):
```
.buildkite/
├── common/ # Shared across platforms
│ ├── scripts/
│ │ ├── skip_ci.py # skip-ci decision (docs / skip-mark / CI YAML paths)
│ │ ├── upload_pipeline.py # Bootstrap + test-pipeline uploader (CUDA/NPU)
│ │ └── resolve_skip_ci.sh # Shell helpers for AMD/Intel bootstrap
│ └── ci_mirror_hardwares.yml # CUDA uploader presets (referenced by name only)
├── cuda/ # Primary NVIDIA CUDA CI
│ ├── pipeline.yml # Bootstrap entry (hook upload)
│ ├── bootstrap-upload-steps.yml # Bootstrap child steps (upload_pipeline --upload)
│ ├── test-ready.yml # L2
│ ├── test-merge.yml # L3
│ ├── test-nightly.yml # L4
│ ├── test-weekly.yml # L5
│ └── rebase-pipeline.yml
├── npu/
│ ├── pipeline-npu.yml # Bootstrap entry (hook upload)
│ ├── bootstrap-upload-steps.yml # Bootstrap child steps (upload_pipeline --upload)
│ ├── pipeline-npu-a3.yml # A3 variant (when used)
│ ├── test-npu-ready.yml # L2
│ ├── test-npu-nightly.yml # L4
│ └── scripts/
├── amd/
│ ├── test-amd-ready.yml # L2 job definitions (template input)
│ ├── test-amd-merge.yml # L3 job definitions
│ ├── test-template-amd-omni.j2 # Renders final pipeline.yaml
│ └── scripts/
│ ├── bootstrap-amd-omni.sh # Entry: skip-ci → Jinja → upload
│ └── run-amd-test.sh # Wraps pytest inside ROCm docker
├── intel/
│ ├── pipeline-intel.yml # Static Intel XPU pipeline
│ └── scripts/
│ ├── bootstrap-intel-omni.sh
│ └── run-xpu-test.sh
└── release/
├── release-pipeline.yml
└── scripts/
```
**Placement rules**
| Rule | Detail |
| ---- | ------ |
| Platform code under platform dir | New CUDA jobs go in `.buildkite/cuda/`; do not add new top-level `.buildkite/test-*.yml` files. |
| Shared logic in `common/` | Skip-ci and CUDA upload rendering stay in `.buildkite/common/scripts/`. |
| Bootstrap vs test YAML | **Bootstrap** (`pipeline*.yml`) builds images and uploads **child** test pipelines. **Test** YAML (`test-*.yml`) lists pytest steps only. |
| Register CI YAML in `skip_ci.py` | If you add a new whitelisted test pipeline file, update `L2_YAML_FILES`, `L3_YAML_FILES`, or `L45_YAML_FILES` in `.buildkite/common/scripts/skip_ci.py` so skip-ci paths stay correct. |
There are still **legacy copies** at `.buildkite/*.yaml` (without the `cuda/` prefix). Treat `.buildkite/cuda/*` as source of truth.
## Platform comparison
| Platform | Bootstrap entry | Test job files | Upload mechanism | Job hardware in YAML |
| -------- | ---------------- | -------------- | ---------------- | -------------------- |
| **CUDA** | `cuda/pipeline.yml` | `test-ready.yml`, `test-merge.yml`, `test-nightly.yml`, `test-weekly.yml` | `upload_pipeline.py --upload` (expands uploader-only keys) | `mirror_hardwares: <preset>` (string) |
| **NPU** | `npu/pipeline-npu.yml` | `test-npu-ready.yml`, `test-npu-nightly.yml` | `upload_pipeline.py --upload` | `mirror_hardwares: a2b3_npu_1` / `a2b3_npu_4` / `a3_npu_2` |
| **AMD** | `amd/scripts/bootstrap-amd-omni.sh` | `test-amd-ready.yml`, `test-amd-merge.yml` | Jinja (`test-template-amd-omni.j2`) → `pipeline upload` | `agent_pool` + `mirror_hardwares: [amdproduction]` (array, template filter) |
| **Intel** | `intel/scripts/bootstrap-intel-omni.sh` | `intel/pipeline-intel.yml` (steps inline) | Direct `pipeline upload` | Inline `agents.queue` on each step |
## Platform configuration style
=== "CUDA"
**Bootstrap:** [`cuda/pipeline.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/pipeline.yml) (hook upload) + [`cuda/bootstrap-upload-steps.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/bootstrap-upload-steps.yml) (child steps). Document 1 runs [`upload_pipeline.py --upload .buildkite/cuda/bootstrap-upload-steps.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py), which injects `if` by step `key` from skip-ci and uploads image build plus L2–L5 child pipeline upload steps (see [Diff-aware CI — Bootstrap skip](#bootstrap-skip)).
**Test YAML:** `cuda/test-ready.yml` (L2), `test-merge.yml` (L3), `test-nightly.yml` (L4), `test-weekly.yml` (L5). Each file starts with shared `env:` then `steps:`.
| CI level | File | Typical trigger |
| -------- | ---- | ----------------- |
| L2 | `cuda/test-ready.yml` | `ready` label |
| L3 | `cuda/test-merge.yml` | `merge-test` label / main merge |
| L4 | `cuda/test-nightly.yml` | `nightly-test` label or `NIGHTLY=1` |
| L5 | `cuda/test-weekly.yml` | `weekly-test` label or `WEEKLY=1` |
**Upload:** `upload_pipeline.py --upload` expands uploader-only keys before Buildkite upload.
**Hardware in YAML:** `mirror_hardwares: <preset>` (string)—preset names in [`common/ci_mirror_hardwares.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/ci_mirror_hardwares.yml). Do **not** set `agents` / `plugins` on the same step.
**Conventions**
- **`depends_on`:** leaf jobs depend on `upload-ready-pipeline`, `upload-merge-pipeline`, etc.
- **`group` / `label`:** `:card_index_dividers:` groups; labels like `Diffusion · Qwen Image Test`.
- **`commands`:** `timeout … pytest …` with markers and `--run-level` for the pipeline level.
- **`source_file_dependencies`:** required on **E2E Test** leaf jobs in L2/L3; see [Step filtering](#step-filtering).
**Adding a job**
1. Pick the level file (ready / merge / nightly / weekly).
2. Add a step under the right **group** (usually **E2E Test** for model pytest).
3. Set `label`, `commands`, `mirror_hardwares`, `depends_on: upload-<level>-pipeline`.
4. For L2/L3 E2E, add `source_file_dependencies` (pytest + model + deploy YAML prefixes).
5. Dry-run:
```bash
python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml
```
=== "NPU"
**Bootstrap:** [`npu/pipeline-npu.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/npu/pipeline-npu.yml) + [`npu/bootstrap-upload-steps.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/npu/bootstrap-upload-steps.yml)—same split as CUDA; builds A2/B3 and A3 CI images, then uploads child test pipelines.
**Test YAML:** `npu/test-npu-ready.yml` (L2), `test-npu-nightly.yml` (L4).
**Upload:** same as CUDA—`upload_pipeline.py --upload`.
**Hardware in YAML:** `mirror_hardwares` preset (string), expanded to `agents`, top-level `image`, and `plugins`. Presets: `a2b3_npu_1`, `a2b3_npu_4`, `a3_npu_2` in `common/ci_mirror_hardwares.yml`.
**Conventions**
- **`depends_on: upload-ready-pipeline`** (or `upload-nightly-pipeline`) ties jobs to bootstrap upload keys.
- Do not duplicate `agents` / `image` / `plugins` when using `mirror_hardwares`.
**Adding a job**
1. Edit `test-npu-ready.yml` (L2) or `test-npu-nightly.yml` (L4).
2. Add a step with `mirror_hardwares` (add a new preset in `ci_mirror_hardwares.yml` first if needed).
3. Set `commands` to your pytest file and markers.
4. Dry-run:
```bash
python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/npu/test-npu-ready.yml
```
5. Register new pipeline paths in `skip_ci.py` (`L2_YAML_FILES` or `L45_YAML_FILES`) when applicable.
=== "AMD"
**Bootstrap:** [`amd/scripts/bootstrap-amd-omni.sh`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/scripts/bootstrap-amd-omni.sh)—skip-ci, diff filtering, Jinja render, then `buildkite-agent pipeline upload`.
**Test YAML (data):** `amd/test-amd-ready.yml` (L2 / PR), `test-amd-merge.yml` (L3 / main).
**Rendering:** [`test-template-amd-omni.j2`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/test-template-amd-omni.j2) wraps data steps with `amd-build` image build and `amd_<agent_pool>` queues. Do **not** hand-edit generated `pipeline.yaml`.
**Hardware in YAML:** `agent_pool` (for example `mi325_1`) plus `mirror_hardwares: [amdproduction]` (array—Buildkite template filter, not the CUDA/NPU uploader preset mechanism).
**Data file fields**
| Field | Purpose |
| ----- | ------- |
| `agent_pool` | ROCm pool; template maps to `queue: amd_<pool>`. |
| `mirror_hardwares` | Which mirror HW runs the step (for example `[amdproduction]`). |
| `commands` | Passed into `run-amd-test.sh` via `TEST_COMMAND`. |
**Adding a job**
1. Edit `test-amd-ready.yml` or `test-amd-merge.yml`.
2. Copy a neighboring block: `label`, `agent_pool`, `mirror_hardwares`, `commands`, optional `grade`.
3. Regenerate via bootstrap / Jinja; update `skip_ci.py` if you add a new YAML path.
=== "Intel"
**Bootstrap:** [`intel/scripts/bootstrap-intel-omni.sh`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/intel/scripts/bootstrap-intel-omni.sh)—skip-ci, then direct `pipeline upload`.
**Test YAML:** steps live inline in [`intel/pipeline-intel.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/intel/pipeline-intel.yml); runners under `intel/scripts/` (for example `run-xpu-test.sh`).
**Upload:** `buildkite-agent pipeline upload` (no `upload_pipeline.py` mirror expansion).
**Hardware in YAML:** inline `agents.queue` (for example `intel-gpu-omni`) on each step.
**Adding a job**
1. Add a step to `pipeline-intel.yml`, or extend an existing runner script.
2. Match `agents`, `env`, `timeout_in_minutes`, and `command`/`commands` with sibling steps.
3. Keep `pipeline-intel.yml` listed in `L2_YAML_FILES` in `skip_ci.py`.
## Cross-cutting conventions
### Diff-aware CI {#diff-aware-ci}
PR diffs drive **two independent skip layers**. Both read changed files from git, but at different pipeline stages and with different granularity:
| Layer | Script | When | What is skipped | Where you configure |
| ----- | ------ | ---- | --------------- | ------------------- |
| **Bootstrap** | [`skip_ci.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/skip_ci.py) + [`upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py) | Before child test pipelines upload (`cuda/pipeline.yml`, `npu/pipeline-npu.yml`, AMD/Intel bootstraps) | Entire default CI, or whole L2/L3 upload for a platform | Whitelists in `skip_ci.py`; bootstrap `if` injected by step `key` in `upload_pipeline.py` |
| **Step filter** | [`upload_pipeline.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/upload_pipeline.py) | While uploading CUDA L2/L3 YAML | Individual Buildkite steps inside `test-ready.yml` / `test-merge.yml` | `source_file_dependencies` on each step or group |
**Changed files** (both layers):
| Build context | Diff command |
| --- | --- |
| Pull request | `git diff --name-only origin/<base>...<BUILDKITE_COMMIT>` |
| `main` push | `git diff --name-only <commit>^..<commit>` |
| Local dry-run / non-PR | Diff unavailable → bootstrap and step filter both keep all steps; uploader-only keys are still stripped |
Label triggers (`ready`, `merge-test`) are unchanged—diff-aware logic only reduces what runs **after** a pipeline is already scheduled.
#### Bootstrap skip {#bootstrap-skip}
[`skip_ci.py`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/common/scripts/skip_ci.py) classifies the git diff into buckets (docs / qualifying skip-mark / whitelisted L2·L3·L4/L5 YAML / other), then picks one decision path. CUDA/NPU apply that decision by injecting Buildkite `if` on `bootstrap-upload-steps.yml` step keys; AMD/Intel call `gate_bootstrap_ci <platform> <l2|l3>`.
##### Decision overview
| Diff shape | Path | Default L2/L3 |
| --- | --- | --- |
| Product code / non-whitelisted paths | normal CI | all on |
| Docs and/or qualifying skip-mark only | `skip_all` | all off (scheduled `main` NIGHTLY/WEEKLY exceptions below) |
| Whitelisted CI YAML only | yaml-gated (`skip_l2_l3`) | per platform/level matrix |
| Docs/skip-mark **+** whitelisted CI YAML | yaml-gated (`skip_l2_l3`) | same as CI-YAML-only (does **not** widen to normal CI) |
| Non-qualifying skip-mark and no CI YAML | normal CI | all on |
| Diff unavailable | normal CI | all on |
**`skip_all` exceptions (CUDA/NPU bootstrap only):** PR labels (`nightly-test`, `merge-test`, `npu-test`, `weekly-test`, …) do **not** revive jobs. On `main`, scheduled `NIGHTLY=1` still builds the image and uploads L4 plus L2/L3 (with `--e2e`); `WEEKLY=1` still builds the image and uploads L5 (CUDA).
**Yaml-gated nightly/weekly:** L4/L5 upload steps keep their normal label / `NIGHTLY` / `WEEKLY` conditions (for example PR `nightly-test` still uploads nightly). Only L2/L3 upload steps are matrix-gated.
##### Whitelisted CI YAML → platform
Register new files in `L2_YAML_FILES`, `L3_YAML_FILES`, or `L45_YAML_FILES` in `skip_ci.py`.
| Level | File | Platform |
| --- | --- | --- |
| L2 | `.buildkite/cuda/test-ready.yml` | cuda |
| L2 | `.buildkite/npu/test-npu-ready.yml` | npu |
| L2 | `.buildkite/amd/test-amd-ready.yml` | amd |
| L2 | `.buildkite/intel/pipeline-intel.yml` | intel |
| L3 | `.buildkite/cuda/test-merge.yml` | cuda |
| L3 | `.buildkite/amd/test-amd-merge.yml` | amd |
| L4/L5 | `.buildkite/cuda/test-nightly.yml`, `test-weekly.yml` | cuda |
| L4/L5 | `.buildkite/npu/test-npu-nightly.yml` | npu |
##### Yaml-gated category branches
`L2?` / `L3?` / `L45?` = whether any file in that whitelist changed. Untouched platforms/levels end up off.
| ID | L2? | L3? | L45? | Enabled L2/L3 |
| --- | --- | --- | --- | --- |
| G1 | | | ✓ | none (all L2/L3 off) |
| G2 | ✓ | | | L2 only on platforms whose L2 YAML changed |
| G3 | | ✓ | | L3 only on platforms whose L3 YAML changed |
| G4 | ✓ | ✓ | | union of G2 and G3 (same platform ready+merge → both on) |
| G5 | ✓ | | ✓ | same as G2 (L45 rescued by L2 YAML) |
| G6 | | ✓ | ✓ | same as G3 (L45 rescued by L3 YAML) |
| G7 | ✓ | ✓ | ✓ | same as G4 |
##### Yaml-gated examples (platform-level)
Docs/skip-mark mixed with any row below follows the same matrix.
| Diff (whitelist) | Branch | Enabled L2/L3 |
| --- | --- | --- |
| Only `cuda/test-ready.yml` | G2 | `cuda/l2` |
| Only `amd/test-amd-ready.yml` | G2 | `amd/l2` |
| Only `npu/test-npu-ready.yml` | G2 | `npu/l2` |
| Only `intel/pipeline-intel.yml` | G2 | `intel/l2` |
| Only `cuda/test-merge.yml` | G3 | `cuda/l3` |
| Only `amd/test-amd-merge.yml` | G3 | `amd/l3` |
| CUDA ready + CUDA merge | G4 | `cuda/l2` + `cuda/l3` |
| CUDA ready + AMD merge | G4 | `cuda/l2` + `amd/l3` |
| CUDA ready + AMD ready | G2 | `cuda/l2` + `amd/l2` |
| CUDA merge + AMD merge | G3 | `cuda/l3` + `amd/l3` |
| Only nightly / weekly / npu-nightly YAML | G1 | (none) |
| CUDA ready + nightly | G5 | `cuda/l2` |
| CUDA merge + nightly | G6 | `cuda/l3` |
| CUDA ready + merge + nightly | G7 | `cuda/l2` + `cuda/l3` |
##### How platforms consume the decision
| Platform | Mechanism |
| --- | --- |
| **CUDA** | `upload_pipeline.py` injects `if` by step `key` (`image-build`, `upload-ready-pipeline`, `upload-merge-pipeline`, …) from `is_run(cuda, l2/l3)` |
| **NPU** | Same injection; no merge upload step (L3 always off in bootstrap) |
| **AMD / Intel** | `gate_bootstrap_ci <platform> <l2\|l3>` exits 0 when `skip_all` or that target is off |
Unit coverage: `tests/buildkite/test_skip_ci.py`.
#### Step filtering {#step-filtering}
CUDA **L2** (`.buildkite/cuda/test-ready.yml`) and **L3** (`.buildkite/cuda/test-merge.yml`) only. Bootstrap upload entry: `upload_pipeline.py --upload .buildkite/cuda/bootstrap-upload-steps.yml`.
**Uploader-only keys** — removed before Buildkite sees the YAML; never used at runtime on agents:
| Key | Purpose |
| --- | ------- |
| `source_file_dependencies` | List of path **prefixes**. If any changed file equals a prefix or starts with `prefix/`, keep the step (or group); otherwise omit it. |
| `mirror_hardwares` | Expand to `agents` + `plugins` (+ optional `image`) from `ci_mirror_hardwares.yml`. |
**Policy**
- **Always uploaded** (no key): groups outside **E2E Test**—Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
- **Diff-gated**: every **E2E Test** leaf job. List the smallest prefix set per step—pytest file(s), model code under `vllm_omni/model_executor/models/` or `vllm_omni/diffusion/models/`, plus `stage_input_processors/` and `vllm_omni/deploy/*.yaml` when applicable. A **group** may define the key instead; the whole group drops if no prefix matches.
**YAML examples**
```yaml
- label: "Diffusion · Qwen Image Test"
source_file_dependencies:
- tests/e2e/online_serving/test_qwen_image.py
- vllm_omni/diffusion/models/qwen_image/
commands:
- pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model' ...
mirror_hardwares: h100_1
- label: "TTS · Qwen3-TTS CustomVoice Test"
source_file_dependencies:
- tests/e2e/online_serving/test_qwen3_tts_customvoice.py
- vllm_omni/model_executor/models/qwen3_tts/
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
- vllm_omni/deploy/qwen3_tts.yaml
commands:
- pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py ...
mirror_hardwares: l4_4
```
**Local dry-run**
```bash
python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml
python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-merge.yml | grep source_file_dependencies
# (no output expected)
```
On PR builds, `upload_pipeline.py` logs `skip '…' (no changes under …)` for omitted steps.
### Labels and grouping
- Use **groups** for dashboard readability; keep **E2E Test** as the group name CUDA diff filtering expects for `--e2e` nightly runs on main.
- Prefix labels by model domain: **Omni ·**, **TTS ·**, **Diffusion ·**, **Simple ·**, etc., matching existing steps.
### Validation checklist
| Check | Command / location |
| ----- | ------------------ |
| CUDA render | `python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-<level>.yml` |
| No leaked uploader keys | `… \| grep -E 'mirror_hardwares|source_file_dependencies'` → empty |
| Skip-ci / upload unit tests | `pytest tests/buildkite/` |
| Local job replay (CUDA L2+) | `tools/run_ready_jobs.sh`, `tools/run_merge_jobs.sh`, `tools/nightly/run_nightly_jobs.sh` (read YAML from `cuda/`) |
## Related documentation
- [Test System Overview](./test_system_overview.md) — L1–L5 scope, triggers, test pyramid
- [Test Writing Guide](./test_writing_guide.md) — markers, directories, L1–L5 examples
- [Test Execution Guide](./test_execution_guide.md) — running CI-aligned jobs locally
- Implementation: `.buildkite/common/scripts/upload_pipeline.py`, `.buildkite/common/scripts/skip_ci.py`
docs/contributing/ci/failures.md
+1 -1
View File
@@ -79,7 +79,7 @@ For resilience behavior under injected process signals/OOM pressure (validation
### Running Tests Locally
For instructions on running tests locally (including specific test files, functions, and markers), see the [Running Tests](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/test_guide/#running-tests) section in the Test Guide.
For instructions on running tests locally (including specific test files, functions, and markers), see the [Running Tests](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/test_execution_guide/#running-tests) section in the Test Execution guide.
## Step 4: Raise an Issue or Fix It
docs/contributing/ci/test_examples/l4_functionality_tests.inc.md
+1 -1
View File
@@ -37,7 +37,7 @@ Currently all the features are available in online serving mode. Hence, only nee
**Code Style**
- Validation: test that the multimodal output files of your model have the correct shapes. `OpenAIClientHandler.send_diffusion_request` should have taken care of this.
- Test marks: always add `full_model` and `diffusion` for L4 nightly `test_*_expansion.py` cases. Add GPU-related marks if needed. Ref: [Markers for Tests](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/tests_markers/).
- Test marks: always add `full_model` and `diffusion` for L4 nightly `test_*_expansion.py` cases. Add GPU-related marks if needed. Ref: [Markers for Tests](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/test_writing_guide/#markers-for-tests).
- To maximize code reuse, you may refer to
- `tests/conftest.py` for `omni_server` (running server in subprocess) and `openai_client` fixtures (sending requests and validating output), `generate_synthetic_image` and `assert_XXX_valid` helper.
- `tests/helpers/mark.py` for `@hardware_test(...)` and `hardware_marks`.
docs/contributing/ci/test_examples/l4_performance_tests.inc.md
+1 -1
View File
@@ -148,7 +148,7 @@ pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py --assert-baseline \
--test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
```
See also [Markers for Tests](./tests_markers.md) for registered hardware markers (`H100`, `L4`, `cuda`, `distributed_cuda`, …).
See also [Markers for Tests](#markers-for-tests) for registered hardware markers (`H100`, `L4`, `cuda`, `distributed_cuda`, …).
**`server_params` Configuration**
docs/contributing/ci/test_guide.md → docs/contributing/ci/test_execution_guide.md
+3 -3
View File
@@ -1,4 +1,4 @@
# Test Guide
# Test Execution Guide
## Setting Up the Test Environment
### Creating a Container
vLLM-Omni provides an official Docker image for deployment. These images are built upon vLLM Docker images and are available on [Docker Hub](https://hub.docker.com/r/vllm/vllm-omni/tags). The version of vLLM-Omni indicates which vLLM release it is based on.
@@ -191,7 +191,7 @@ Failed jobs: 1/2
The latest L5 CI jobs (reliability + invalid-parameter weekly steps) are in [test-weekly.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/test-weekly.yml).
You can find more information about markers in the documentation: [marker doc](./tests_markers.md)
You can find more information about markers in [Test Writing Guide — Markers for Tests](./test_writing_guide.md#markers-for-tests).
## Adding New Test Cases
Please refer to the [L5 Layering Specification document](./CI_5levels.md).
Please refer to the [Test System Overview](./test_system_overview.md) and [Test Writing Guide](./test_writing_guide.md).
docs/contributing/ci/test_system_overview.md
+210
View File
@@ -0,0 +1,210 @@
# Multi-Level Automated Testing System Overview
## Document Overview
This testing system aims to build a complete, efficient, and well-structured quality assurance framework for the development, integration, and release of model services. It draws on the concept of the test pyramid from modern software engineering, progressively expanding testing activities from basic code logic verification to complex end-to-end (E2E) functionality, performance, accuracy, and even long-term stability validation.
Through five levels (L1-L5) and common (Common) specifications, the system clarifies the testing objectives, scope, execution frequency, and required resources for different development stages (e.g., each commit, PR merge, daily build, pre-release). This ensures that models meet high standards for functionality, performance, and reliability across various deployment scenarios (online serving and offline inference).
<table>
<thead>
<tr>
<th>Level</th>
<th>Scope & Focus</th>
<th>Model Coverage Strategy</th>
<th>Feature Coverage Strategy</th>
<th>Interface Coverage Strategy</th>
<th>Tags</th>
<th>Time Cost</th>
<th>Test Dir</th>
<th>Doc</th>
<th>Frequency</th>
<th>Hardware</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="2"><strong>Common</strong></td>
<td>Contribution Guideline & PR checklist</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td><a href="https://github.com/vllm-project/vllm-omni/blob/main/.github/PULL_REQUEST_TEMPLATE.md">PR Checklist</a></td>
<td>/</td>
<td>/</td>
</tr>
<tr>
<td>CI Failure Description</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td><a href="./failures.md">CI Failures</a></td>
<td>/</td>
<td>/</td>
</tr>
<tr>
<td><strong>L1</strong><br>(Unit & Logic)</td>
<td>Unit tests for components like entrypoints, models</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td><code>core_model and cpu</code></td>
<td rowspan="2">&lt;15min</td>
<td>/tests/{component_name}/test_xxx</td>
<td>
<a href="test_writing_guide.md#l1--l2-level-testing-unit-testing-and-basic-end-to-end-verification">L1 &amp; L2</a><br>
Section 1 L1&amp;L2: Purpose, Test Content, Directory Location, Example
</td>
<td>PR with ready label (also can run locally)</td>
<td>CPU</td>
</tr>
<tr>
<td><strong>L2</strong><br>(E2E across models & GPU-required UT)</td>
<td>Online (basic deployment scenarios):<br>dummy, normal inference function (output format, stream), some instance startup UT</td>
<td>High-priority models + online basic scenarios; request success, non-empty output, format match (no Whisper/accuracy)</td>
<td>High-priority features (using random lightweight models)</td>
<td>High-priority interfaces (using random lightweight models)</td>
<td><code>core_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td>
<strong>Model tests:</strong><br>
/tests/e2e/online_serving/test_{model_name}.py<br>
<strong>Feature tests:</strong><br>
/tests/{component_name}/test_xxx<br>
<strong>Interface tests:</strong><br>
/tests/entrypoints/test_xxx
</td>
<td>
<a href="test_writing_guide.md#l1--l2-level-testing-unit-testing-and-basic-end-to-end-verification">L1 &amp; L2</a><br>
L1&amp;L2: Purpose, Test Content, Directory Location, Example
</td>
<td>PR with ready label</td>
<td>GPU</td>
</tr>
<tr>
<td><strong>L3</strong><br>(Important Perf & Integration & Accuracy)</td>
<td>Online & Offline (multiple deployment scenarios):<br>real model, normal inference function, normal accuracy</td>
<td>High/medium-priority models + key online/offline scenarios; real weights, Whisper/similarity, preset voice gender, basic accuracy</td>
<td>Medium-priority features (using random lightweight models)</td>
<td>Medium-priority interfaces (using random lightweight models)</td>
<td><code>advanced_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td>&lt;30min</td>
<td>
<strong>Model tests:</strong><br>
/tests/e2e/online_serving/test_{model_name}.py<br>
/tests/e2e/offline_inference/test_{model_name}.py<br>
<strong>Feature tests:</strong><br>
/tests/{component_name}/test_xxx<br>
<strong>Interface tests:</strong><br>
/tests/entrypoints/test_xxx
</td>
<td>
<a href="test_writing_guide.md#l3-level-testing-core-integration-performance-and-accuracy-verification">L3</a><br>
L3: Purpose, Test Content, Directory Location, Example
</td>
<td>PR Merged (Also run L1&L2 Tests)</td>
<td>GPU</td>
</tr>
<tr>
<td><strong>L4</strong><br>(Perf & Integration & Accuracy)</td>
<td>Online: full functional scenarios + performance test + doc test + accuracy test</td>
<td>High-priority models: function, performance, accuracy, and doc testing<br>Medium-priority models: function and doc testing</td>
<td>Low-priority features (using real weights)</td>
<td>Low-priority interfaces (using real weights)</td>
<td><code>full_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td>&lt;3 hour</td>
<td>
<strong>Model tests:</strong><br>
/tests/e2e/online_serving/test_{model_name}_expansion.py<br>
<strong>Feature tests:</strong><br>
/tests/{component_name}/test_xxx<br>
<strong>Interface tests:</strong><br>
/tests/entrypoints/test_xxx<br>
<strong>Performance:</strong><br>
/tests/dfx/perf/tests/test_qwen3_omni_*.json (Omni), test_tts.json (TTS),<br>
test_voxcpm2.json, test_higgs_audio_v3.json, and<br>
/tests/dfx/perf/tests/test_{diffusion_model}_vllm_omni.json (Diffusion)<br>
<strong>Doc Test:</strong><br>
tests/examples/online_serving/test_{model_name}.py<br>
tests/examples/offline_inference/test_{model_name}.py<br>
<strong>Accuracy Test:</strong><br>
/tests/e2e/accuracy/test_{model_name}.py
</td>
<td>
<a href="test_writing_guide.md#l4-level-testing-full-functionality-performance-and-documentation-testing">L4</a><br>
L4: Purpose, Test Content, Directory Location, Example
</td>
<td>Nightly</td>
<td>GPU</td>
</tr>
<tr>
<td><strong>L5</strong><br>(Stability & Reliability)</td>
<td>Online: long-term stability test + reliability test</td>
<td>Long-term stability and reliability testing for high-priority models<br>Low-priority models: function and doc testing</td>
<td>/</td>
<td>Invalid-parameter validation for high-priority interfaces</td>
<td><code>slow and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td> Depends on reality </td>
<td>
<strong>Stability:</strong><br>
/tests/dfx/stability/tests/test_qwen3_omni.json<br>
/tests/dfx/stability/tests/test_wan22.json<br>
<strong>Reliability:</strong><br>
tests/dfx/reliability/test_reliability_{model_key}.py<br>
(e.g. <code>test_reliability_qwen3_omni.py</code>, <code>test_reliability_wan22.py</code>, <code>test_reliability_hunyuan_image.py</code>, <code>test_reliability_voxcpm2.py</code>)
</td>
<td>
<a href="test_writing_guide.md#l5-level-testing-stability-and-reliability-testing">L5</a><br>
L5: Purpose, Test Content, Directory Location, Example
</td>
<td>Weekly / Days before Release</td>
<td>GPU</td>
</tr>
</tbody>
</table>
For per-level test authoring (directories, markers, examples), see [Test Writing Guide](./test_writing_guide.md).
## Common Specifications
Before entering specific testing levels, the project establishes two common specifications aimed at standardizing the development process and quickly locating issues.
1. ***PR Checklist ([`.github/PULL_REQUEST_TEMPLATE.md`](https://github.com/vllm-project/vllm-omni/blob/main/.github/PULL_REQUEST_TEMPLATE.md))***: This template defines the self-check items that must be completed before submitting a code review (Pull Request). It ensures that each code change meets basic requirements such as code style, dependency updates, and documentation synchronization before entering the automated testing pipeline, serving as the first manual line of defense for quality assurance.
2. ***CI Failure Explanation ([CI Failures](./failures.md))***: This document archives and explains common failure patterns in the Continuous Integration (CI) pipeline, error log interpretation, and preliminary troubleshooting steps. It helps developers and testers quickly diagnose the causes of automated test failures, improving problem-solving efficiency.
## Notes
### L2 / L3 diff-aware CI (CUDA)
On CUDA **L2** and **L3**, **E2E Test** Buildkite jobs may be omitted at pipeline upload when the PR diff does not touch their path prefixes; other groups still always upload. Full two-layer mechanics and YAML examples: [CI Settings — Diff-aware CI](./ci_settings.md#diff-aware-ci) ([step filtering](./ci_settings.md#step-filtering)).
### Test helper environment variables
Some shared helpers under `tests/helpers/` honor optional environment variables for local debugging. These are **not** set in CI by default.
| Variable | Accepted values | Description |
| -------- | --------------- | ----------- |
| `VLLM_OMNI_KEEP_REQUEST_MEDIA` | `1`, `true`, `yes` (case-insensitive) | When enabled, temporary WAV files created by `tests.helpers.media.convert_audio_bytes_to_text` are **not** deleted when the pytest process exits. By default, each call writes a unique file under the system temp directory via `tempfile.mkstemp` and registers `atexit` cleanup. Use this when debugging audio output validation (Whisper transcription, keyword checks, text–audio similarity). The saved path is logged as `audio data is saved: <path>`. |
Example (Linux / macOS):
```bash
export VLLM_OMNI_KEEP_REQUEST_MEDIA=1
pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -k test_mix_to_text_audio
```
Example (Windows PowerShell):
```powershell
$env:VLLM_OMNI_KEEP_REQUEST_MEDIA = "1"
pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -k test_mix_to_text_audio
```
## Summary
This multi-level testing system achieves continuous, progressive validation of model service quality by tightly integrating testing activities with the development workflow (commit, review, merge, release). From rapid unit testing to comprehensive end-to-end testing, and further to in-depth performance, stability, and reliability verification, each level has clear objectives, collectively building a robust quality protection net. By following this system, teams can deliver high-quality, highly reliable model services more efficiently.
docs/contributing/ci/CI_5levels.md → docs/contributing/ci/test_writing_guide.md
+224 -311
View File
@@ -1,177 +1,9 @@
# Multi-Level Automated Testing System Documentation
# Test Writing Guide
## Document Overview
This document describes how to author tests for each CI level (L1–L5). For the level matrix, CI triggers, and Buildkite upload behavior, see [Test System Overview](./test_system_overview.md).
This testing system aims to build a complete, efficient, and well-structured quality assurance framework for the development, integration, and release of model services. It draws on the concept of the test pyramid from modern software engineering, progressively expanding testing activities from basic code logic verification to complex end-to-end (E2E) functionality, performance, accuracy, and even long-term stability validation.
## Test Directory Structure
Through five levels (L1-L5) and common (Common) specifications, the system clarifies the testing objectives, scope, execution frequency, and required resources for different development stages (e.g., each commit, PR merge, daily build, pre-release). This ensures that models meet high standards for functionality, performance, and reliability across various deployment scenarios (online serving and offline inference).
<table>
<thead>
<tr>
<th>Level</th>
<th>Scope & Focus</th>
<th>Model Coverage Strategy</th>
<th>Feature Coverage Strategy</th>
<th>Interface Coverage Strategy</th>
<th>Tags</th>
<th>Time Cost</th>
<th>Test Dir</th>
<th>Doc</th>
<th>Frequency</th>
<th>Hardware</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="2"><strong>Common</strong></td>
<td>Contribution Guideline & PR checklist</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>.github/PULL_REQUEST_TEMPLATE.md <a href="../tests_style/"> Test Style (PR Checklist)</a></td>
<td>/</td>
<td>/</td>
</tr>
<tr>
<td>CI Failure Description</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td><a href="../failures/"> CI Failures</a></td>
<td>/</td>
<td>/</td>
</tr>
<tr>
<td><strong>L1</strong><br>(Unit & Logic)</td>
<td>Unit tests for components like entrypoints, models</td>
<td>/</td>
<td>/</td>
<td>/</td>
<td><code>core_model and cpu</code></td>
<td rowspan="2">&lt;15min</td>
<td>/tests/{component_name}/test_xxx</td>
<td>
<a href="#chapter-1-l1-l2-level-testing-unit-testing-and-basic-end-to-end-verification">Chapter 1</a><br>
Section 1 L1&amp;L2: Purpose, Test Content, Directory Location, Example
</td>
<td>PR with ready label (also can run locally)</td>
<td>CPU</td>
</tr>
<tr>
<td><strong>L2</strong><br>(E2E across models & GPU-required UT)</td>
<td>Online (basic deployment scenarios):<br>dummy, normal inference function (output format, stream), some instance startup UT</td>
<td>High-priority models + online basic scenarios; request success, non-empty output, format match (no Whisper/accuracy)</td>
<td>High-priority features (using random lightweight models)</td>
<td>High-priority interfaces (using random lightweight models)</td>
<td><code>core_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td>
<strong>Model tests:</strong><br>
/tests/e2e/online_serving/test_{model_name}.py<br>
<strong>Feature tests:</strong><br>
/tests/{component_name}/test_xxx<br>
<strong>Interface tests:</strong><br>
/tests/entrypoints/test_xxx
</td>
<td>
<a href="#chapter-1-l1-l2-level-testing-unit-testing-and-basic-end-to-end-verification">Chapter 1</a><br>
L1&amp;L2: Purpose, Test Content, Directory Location, Example
</td>
<td>PR with ready label</td>
<td>GPU</td>
</tr>
<tr>
<td><strong>L3</strong><br>(Important Perf & Integration & Accuracy)</td>
<td>Online & Offline (multiple deployment scenarios):<br>real model, normal inference function, normal accuracy</td>
<td>High/medium-priority models + key online/offline scenarios; real weights, Whisper/similarity, preset voice gender, basic accuracy</td>
<td>Medium-priority features (using random lightweight models)</td>
<td>Medium-priority interfaces (using random lightweight models)</td>
<td><code>advanced_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td>&lt;30min</td>
<td>
<strong>Model tests:</strong><br>
/tests/e2e/online_serving/test_{model_name}.py<br>
/tests/e2e/offline_inference/test_{model_name}.py<br>
<strong>Feature tests:</strong><br>
/tests/{component_name}/test_xxx<br>
<strong>Interface tests:</strong><br>
/tests/entrypoints/test_xxx
</td>
<td>
<a href="#chapter-2-l3-level-testing-core-integration-performance-and-accuracy-verification">Chapter 2</a><br>
L3: Purpose, Test Content, Directory Location, Example
</td>
<td>PR Merged (Also run L1&L2 Tests)</td>
<td>GPU</td>
</tr>
<tr>
<td><strong>L4</strong><br>(Perf & Integration & Accuracy)</td>
<td>Online: full functional scenarios + performance test + doc test + accuracy test</td>
<td>High-priority models: function, performance, accuracy, and doc testing<br>Medium-priority models: function and doc testing</td>
<td>Low-priority features (using real weights)</td>
<td>Low-priority interfaces (using real weights)</td>
<td><code>full_model and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td>&lt;3 hour</td>
<td>
<strong>Model tests:</strong><br>
/tests/e2e/online_serving/test_{model_name}_expansion.py<br>
<strong>Feature tests:</strong><br>
/tests/{component_name}/test_xxx<br>
<strong>Interface tests:</strong><br>
/tests/entrypoints/test_xxx<br>
<strong>Performance:</strong><br>
/tests/dfx/perf/tests/test_qwen3_omni_*.json (Omni), test_tts.json (TTS),<br>
test_voxcpm2.json, test_higgs_audio_v3.json, and<br>
/tests/dfx/perf/tests/test_{diffusion_model}_vllm_omni.json (Diffusion)<br>
<strong>Doc Test:</strong><br>
tests/examples/online_serving/test_{model_name}.py<br>
tests/examples/offline_inference/test_{model_name}.py<br>
<strong>Accuracy Test:</strong><br>
/tests/e2e/accuracy/test_{model_name}.py
</td>
<td>
<a href="#chapter-3-l4-level-testing-full-functionality-performance-and-documentation-testing">Chapter 3</a><br>
L4: Purpose, Test Content, Directory Location, Example
</td>
<td>Nightly</td>
<td>GPU</td>
</tr>
<tr>
<td><strong>L5</strong><br>(Stability & Reliability)</td>
<td>Online: long-term stability test + reliability test</td>
<td>Long-term stability and reliability testing for high-priority models<br>Low-priority models: function and doc testing</td>
<td>/</td>
<td>Invalid-parameter validation for high-priority interfaces</td>
<td><code>slow and hardware_test(H100, L4, etc.) and omni/tts/diffusion</code></td>
<td> Depends on reality </td>
<td>
<strong>Stability:</strong><br>
/tests/dfx/stability/tests/test_qwen3_omni.json<br>
/tests/dfx/stability/tests/test_wan22.json<br>
<strong>Reliability:</strong><br>
tests/dfx/reliability/test_reliability_{model_key}.py<br>
(e.g. <code>test_reliability_qwen3_omni.py</code>, <code>test_reliability_wan22.py</code>, <code>test_reliability_hunyuan_image.py</code>, <code>test_reliability_voxcpm2.py</code>)
</td>
<td>
<a href="#chapter-4-l5-level-testing-stability-and-reliability-testing">Chapter 4</a><br>
L5: Purpose, Test Content, Directory Location, Example
</td>
<td>Weekly / Days before Release</td>
<td>GPU</td>
</tr>
</tbody>
</table>
---
<details>
<summary> The folder structure for tests file based on the 5 levels design</summary>
Legend: `✅` = test exists, `⬜` = suggested to add.
```
vllm_omni/ tests/
@@ -304,134 +136,219 @@ vllm_omni/ tests/
# vllm_omni/deploy/ instead — see docs/configuration/stage_configs.md.
```
## Markers for Tests
</details>
By adding markers before test functions, tests can later be executed uniformly by simply declaring the corresponding marker type.
### Current Markers
Defined in `pyproject.toml`:
## Common Specifications
| Marker | Description |
| ------------------ | --------------------------------------------------------- |
| `core_model` | L1&L2 tests (run in each PR) |
| `advanced_model` | L3 tests (run on each merge to main) |
| `full_model` | L4 tests (run nightly) |
| `diffusion` | Diffusion model tests |
| `omni` | Omni multimodal model tests |
| `tts` | TTS model tests |
| `cache` | Cache backend tests |
| `parallel` | Parallelism/distributed tests |
| `cpu` | Tests that run on CPU |
| `gpu` | Tests that run on GPU * |
| `cuda` | Tests that run on CUDA * |
| `rocm` | Tests that run on AMD/ROCm * |
| `xpu` | Tests that run on Intel XPU * |
| `npu` | Tests that run on NPU/Ascend * |
| `H100` | Tests that require H100 GPU * |
| `L4` | Tests that require L4 GPU * |
| `MI325` | Tests that require MI325 GPU (AMD/ROCm) * |
| `A2` | Tests that require A2 NPU * |
| `A3` | Tests that require A3 NPU * |
| `distributed_cuda` | Tests that require multi cards on CUDA platform * |
| `distributed_rocm` | Tests that require multi cards on ROCm platform * |
| `distributed_npu` | Tests that require multi cards on NPU platform * |
| `skipif_cuda` | Skip if the num of CUDA cards is less than the required * |
| `skipif_rocm` | Skip if the num of ROCm cards is less than the required * |
| `skipif_npu` | Skip if the num of NPU cards is less than the required * |
| `slow` | Slow tests (may skip in quick CI) |
| `benchmark` | Benchmark tests (decorator on runner test functions; perf JSON uses `full_model` + type marker instead) |
| `local_model` | Tests requiring local / non-HF-hub model weights |
Before entering specific testing levels, the project establishes two common specifications aimed at standardizing the development process and quickly locating issues.
\* Means those markers are auto-added by `@hardware_test` (parametrization decorator) or `hardware_marks` (only returning the list of marks for flexibility).
1. ***PR Checklist ([Tests Style](../ci/tests_style.md))***: This template defines the self-check items that must be completed before submitting a code review (Pull Request). It ensures that each code change meets basic requirements such as code style, dependency updates, and documentation synchronization before entering the automated testing pipeline, serving as the first manual line of defense for quality assurance.
2. ***CI Failure Explanation ([CI Failures](../ci/failures.md))***: This document archives and explains common failure patterns in the Continuous Integration (CI) pipeline, error log interpretation, and preliminary troubleshooting steps. It helps developers and testers quickly diagnose the causes of automated test failures, improving problem-solving efficiency.
#### Example usage for markers
## Notes
```python
from tests.helpers.mark import hardware_test
### Diff-aware Buildkite uploads (`source_file_dependencies`)
L2 (`.buildkite/cuda/test-ready.yml`) and L3 (`.buildkite/cuda/test-merge.yml`) pipelines can **skip unrelated GPU jobs at upload time** based on the PR diff. This is implemented by `.buildkite/cuda/scripts/upload_pipeline.py`, which filters steps before calling `buildkite-agent pipeline upload`.
#### What `source_file_dependencies` is
- A **uploader-only** YAML key on a Buildkite step or group. **Buildkite does not understand it**; `upload_pipeline.py` always **removes** it from the YAML that is uploaded.
- A list of path **prefixes** (directories or individual files). If **any** changed file in the diff equals a prefix or starts with `prefix/`, the step is kept; otherwise the step (or entire group) is **omitted** from the uploaded pipeline.
#### When filtering runs
| Build context | Changed files used for matching |
| --- | --- |
| Pull request | `git diff --name-only origin/<base>...<BUILDKITE_COMMIT>` |
| `main` branch push | `git diff --name-only <commit>^..<commit>` |
| Other (e.g. local dry-run, non-PR branch) | Diff cannot be resolved → **no filtering**; all steps are uploaded and `source_file_dependencies` is still stripped |
Docs-only PRs are handled earlier in bootstrap (`.buildkite/cuda/pipeline.yml`) via skip-ci logic; `source_file_dependencies` applies only to the **uploaded** L2/L3 test pipelines.
#### Where it is configured
| Level | Pipeline file | Upload entry |
| --- | --- | --- |
| L2 | `.buildkite/cuda/test-ready.yml` | `upload_pipeline.py --upload .buildkite/cuda/test-ready.yml` (from `cuda/pipeline.yml`) |
| L3 | `.buildkite/cuda/test-merge.yml` | `upload_pipeline.py --upload .buildkite/cuda/test-merge.yml` (from `cuda/pipeline.yml`) |
Steps **without** `source_file_dependencies` are always uploaded (subject to the usual label conditions: `ready` for L2, `merge-test` for L3).
#### Current skip policy (L2 / L3)
To balance CI cost and coverage:
- **Always run** (no `source_file_dependencies`): baseline groups outside E2E Test—e.g. Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
- **Diff-gated** (`source_file_dependencies` set): **every leaf job under the E2E Test group** in `test-ready.yml` and `test-merge.yml`, regardless of queue (`mithril-h100-pool`, `gpu_1_queue`, or `gpu_4_queue`). Each step lists the smallest set of prefixes that should trigger it—typically:
- pytest file(s) exercised by the job (online and/or offline);
- model code under `vllm_omni/model_executor/models/` or `vllm_omni/diffusion/models/`;
- related `vllm_omni/model_executor/stage_input_processors/` and `vllm_omni/deploy/*.yaml` when applicable.
Adding a new E2E step: add `source_file_dependencies` on the leaf job with those prefixes. Prefer **per-step** deps rather than a broad group-level list unless every child shares the same paths.
#### YAML examples
H100 E2E (kubernetes / `mithril-h100-pool`):
```yaml
- label: "Diffusion · Qwen Image Test"
source_file_dependencies:
- tests/e2e/online_serving/test_qwen_image.py
- vllm_omni/diffusion/models/qwen_image/
commands:
- pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model' ...
agents:
queue: "mithril-h100-pool"
@pytest.mark.core_model
@pytest.mark.omni
@hardware_test(
res={"cuda": "L4", "rocm": "MI325", "npu": "A2"},
num_cards=2,
)
@pytest.mark.parametrize("omni_server", test_params, indirect=True)
def test_video_to_audio()
...
```
Docker E2E (`gpu_1_queue` / `gpu_4_queue`)—same key, same upload-time filtering:
#### Decorator: `@hardware_test`
```yaml
- label: "TTS · Qwen3-TTS CustomVoice Test"
source_file_dependencies:
- tests/e2e/online_serving/test_qwen3_tts_customvoice.py
- vllm_omni/model_executor/models/qwen3_tts/
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
- vllm_omni/deploy/qwen3_tts.yaml
commands:
- pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py ...
agents:
queue: "gpu_4_queue"
This decorator is intended to make hardware-aware, cross-platform test authoring easier and more robust for CI/CD environments. The `hardware_test` decorator in `vllm-omni/tests/helpers/mark.py` performs the following actions:
1. **Applies platform and resource markers**
Adds the appropriate pytest markers for each specified hardware platform (e.g., `cuda`, `rocm`, `xpu`, `npu`) and resource type (e.g., `L4`, `H100`, `MI325`, `B60`, `A2`, `A3`).
```
@pytest.mark.cuda
@pytest.mark.L4
```
2. **Handles multi-card (distributed) scenarios**
For tests requiring multiple cards, it automatically adds distributed markers such as `distributed_cuda`, `distributed_rocm`, or `distributed_npu`.
```
@pytest.mark.distributed_cuda(num_cards=num_cards)
```
3. **Supports flexible card requirements**
Accepts `num_cards` as either a single integer for all platforms or as a dictionary with per-platform values. If not specified, defaults to 1 card per platform.
4. **Integrates resource validation**
On CUDA, adds a skip marker (`skipif_cuda`) if the system does not have the required number of devices.
Support for `skipif_rocm` and `skipif_npu` will be implemented later.
5. **Works with pytest filtering**
Allows tests to be filtered and selected at runtime using standard pytest marker expressions (e.g., `-m "distributed_cuda and L4"`).
##### Example usage for decorator
- Single call for multiple platforms:
```python
@hardware_test(
res={"cuda": "L4", "rocm": "MI325", "xpu": "B60", "npu": "A2"},
num_cards={"cuda": 2, "rocm": 2, "xpu": 2, "npu": 2},
)
```
or
```python
@hardware_test(
res={"cuda": "L4", "rocm": "MI325", "xpu": "B60", "npu": "A2"},
num_cards=2,
)
```
- `res` must be a dict; supported resources: CUDA (L4/H100), ROCm (MI325), XPU (B60), MUSA (S5000), NPU (A2/A3)
- `num_cards` can be int (all platforms) or dict (per platform); defaults to 1 when missing
- Distributed markers (`distributed_cuda`, `distributed_rocm`, `distributed_npu`) are auto-added for multi-card cases
- Filtering examples:
- CUDA only: `pytest -m "distributed_cuda and L4"`
- ROCm only: `pytest -m "distributed_rocm and MI325"`
- NPU only: `pytest -m "distributed_npu"`
#### Function: `hardware_marks`
`hardware_marks` returns a list of pytest mark objects with the same signature as `@hardware_test`. Use it when you need more flexibility, such as attaching hardware marks to individual `pytest.param` entries rather than an entire test function.
```python
from tests.helpers.mark import hardware_marks
MULTI_CARD_MARKS = hardware_marks(
res={"cuda": "H100", "rocm": "MI325", "npu": "A2"}, num_cards=2
)
@pytest.mark.parametrize("omni_server", [
pytest.param(OmniServerParams(...), id="case_001", marks=MULTI_CARD_MARKS),
], indirect=True)
def test_feature(omni_server):
...
```
A **group** may also define `source_file_dependencies`; nested steps inherit filtering as a unit—the whole group is dropped if no prefix matches.
#### JSON `mark` field (L4 perf configs)
#### Local dry-run
Perf JSON under `tests/dfx/perf/tests/` can attach pytest marks per **case** (`test_name` block). Parsed by `tests.dfx.conftest.resolve_pytest_marks` and applied to each parametrized run in `run_benchmark.py` / `run_diffusion_benchmark.py`.
```bash
# Render filtered YAML to stdout (no upload)
python3 .buildkite/cuda/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml
When `mark` is present, it must be an **array** with exactly one ``hardware_marks`` object (same semantics as `hardware_marks()` above), followed by registered pytest marker name strings such as `full_model`, `omni`, `tts`, or `diffusion`.
# Confirm uploader-only keys are stripped
python3 .buildkite/cuda/scripts/upload_pipeline.py .buildkite/cuda/test-merge.yml | grep source_file_dependencies
# (no output expected)
```json
{
"test_name": "test_cosmos3_t2i_official_demo_2gpu",
"mark": [
{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}},
"full_model",
"diffusion"
],
"server_type": "vllm-omni",
"server_params": { "...": "..." },
"benchmark_params": [{ "name": "1024x1024_steps4", "...": "..." }]
}
```
On a PR build, Buildkite logs from `upload_pipeline.py` include lines such as `skip '…' (no changes under …)` for omitted steps.
- Local bulk load: `pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and diffusion and H100"`
- Nightly CI perf steps: `--test-config-file tests/dfx/perf/tests/test_<model>_vllm_omni.json` (file selects cases; no `-m`)
- Result filenames use **runtime** GPU detection (`get_runtime_resource_label`); `H100` is omitted on the default CI pool
#### Related
See [L4 performance test examples](./test_examples/l4_performance_tests.inc.md) for the full schema.
- Implementation: `.buildkite/cuda/scripts/upload_pipeline.py`
- L2/L3 diff skip does **not** replace label-based triggers (`ready`, `merge-test`); it only reduces which steps appear **after** the pipeline is already scheduled.
### Add Support for a New Platform
### Test helper environment variables
If you want to add support for a new platform (e.g., "tpu" for a new accelerator), follow these steps:
Some shared helpers under `tests/helpers/` honor optional environment variables for local debugging. These are **not** set in CI by default.
1. **Extend the marker list in your pytest config** so that platform/resource markers are defined:
```toml
# In pyproject.toml or pytest.ini
[tool.pytest.ini_options]
markers = [
# ... existing markers ...
"tpu: Tests that require TPU device",
"TPU_V3: Tests that require TPU v3 hardware",
"distributed_tpu: Tests that require multiple TPU devices",
]
```
2. **Implement a marker construction function for your platform** in `vllm-omni/tests/helpers/mark.py`:
```python
# In vllm-omni/tests/helpers/mark.py
| Variable | Accepted values | Description |
| -------- | --------------- | ----------- |
| `VLLM_OMNI_KEEP_REQUEST_MEDIA` | `1`, `true`, `yes` (case-insensitive) | When enabled, temporary WAV files created by `tests.helpers.media.convert_audio_bytes_to_text` are **not** deleted when the pytest process exits. By default, each call writes a unique file under the system temp directory via `tempfile.mkstemp` and registers `atexit` cleanup. Use this when debugging audio output validation (Whisper transcription, keyword checks, text–audio similarity). The saved path is logged as `audio data is saved: <path>`. |
def tpu_marks(*, res: str, num_cards: int):
test_platform = pytest.mark.tpu
if res == "TPU_V3":
test_resource = pytest.mark.TPU_V3
else:
raise ValueError(
f"Invalid TPU resource type: {res}. Supported: TPU_V3")
Example (Linux / macOS):
if num_cards == 1:
return [test_platform, test_resource]
else:
test_distributed = pytest.mark.distributed_tpu(num_cards=num_cards)
# Optionally: add skipif_tpu when implemented
return [test_platform, test_resource, test_distributed]
```
3. **Update `hardware_marks` to recognize your new platform**:
In the relevant place (see the `hardware_marks` implementation), add:
```python
if platform == "tpu":
marks = tpu_marks(res=resource, num_cards=cards)
```
(`hardware_test` calls `hardware_marks` internally, so both will pick up the change.)
4. **(Recommended) Add a test using your new markers**:
```python
@hardware_test(
res={"tpu": "TPU_V3"},
num_cards=2,
)
def test_my_tpu_feature():
...
```
```bash
export VLLM_OMNI_KEEP_REQUEST_MEDIA=1
pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -k test_mix_to_text_audio
```
**Summary**:
- Add pytest markers for your new platform/resources
- Implement a marker function (`xxx_marks`)
- Plug into `hardware_marks`
- You're done: tests using `@hardware_test` or `hardware_marks` with your platform now automatically get the correct markers, distribution, and isolation!
Example (Windows PowerShell):
See code in `vllm-omni/tests/helpers/mark.py` for existing examples (`cuda_marks`, `rocm_marks`, `npu_marks`).
```powershell
$env:VLLM_OMNI_KEEP_REQUEST_MEDIA = "1"
pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -k test_mix_to_text_audio
```
## Test case style
## Chapter 1: L1 & L2 Level Testing - Unit Testing and Basic End-to-End Verification
### L1 & L2 Level Testing - Unit Testing and Basic End-to-End Verification
### 1.1 Testing Purpose
#### 1.1 Testing Purpose
L1 and L2 level testing form the foundation of the quality assurance system. L1 level testing focuses on verifying the internal logic correctness of code units (e.g., functions, classes), ensuring each independent component behaves as designed.
@@ -439,7 +356,7 @@ L2 level testing builds upon L1 by introducing GPU resources and verifying that
### 1.2 Testing Content and Scope
#### 1.2 Testing Content and Scope
- ***L1 (Unit & Logic Testing)***:
- - ***Scope***: Tests internal functions and methods of core components such as `entrypoints`, `models`.
@@ -454,7 +371,7 @@ L2 level testing builds upon L1 by introducing GPU resources and verifying that
- **Omni multimodal** (`assert_omni_response`): L2 asserts successful completion; keyword, transcript, and cross-modal similarity checks run only at L3+.
- ***Characteristic***: Requires ***GPU*** resources to perform model computations.
### 1.3 Test Directory and Execution Files
#### 1.3 Test Directory and Execution Files
A clear directory structure is key to managing test cases efficiently.
@@ -464,10 +381,10 @@ A clear directory structure is key to managing test cases efficiently.
- - Online Serving: `/tests/e2e/online_serving/test_{model_name}.py`
- Offline Inference: `/tests/e2e/offline_inference/test_{model_name}.py`
### 1.4 Execution Method and Example
#### 1.4 Execution Method and Example
- ***Trigger Timing***: **`PR with ready label`**. That is, when a developer adds a "ready for review" or similar label to a PR on platforms like GitHub, L1 and L2 tests are automatically triggered.
- ***Diff-aware step skipping***: On L2, **E2E Test** jobs may be omitted at pipeline upload when the PR diff does not touch their [`source_file_dependencies`](#diff-aware-buildkite-uploads-source_file_dependencies) prefixes (H100 and docker queues alike); non-E2E groups still always upload. See [Diff-aware Buildkite uploads](#diff-aware-buildkite-uploads-source_file_dependencies).
- ***Diff-aware step skipping***: On L2, **E2E Test** jobs may be omitted at pipeline upload when the PR diff does not touch their [`source_file_dependencies`](ci_settings.md#step-filtering) prefixes; non-E2E groups still always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
- ***Execution Environment***: L1 uses ***CPU*** environment; L2 requires ***GPU*** environment.
- ***Script Example***:
@@ -535,7 +452,7 @@ def test_cap_and_align_mel_length_no_mismatch(repeats, code_len, max_mel_frames)
<details>
<summary> L2 Test Examples</summary>
You can refer to Test Examples in Chapter 2 to see example test cases that incorporate both L2 and L3 testing logic.
You can refer to Test Examples in the L3 section to see example test cases that incorporate both L2 and L3 testing logic.
</details>
- - ***Run Command***:
@@ -543,9 +460,9 @@ You can refer to Test Examples in Chapter 2 to see example test cases that incor
`pytest -s -v /tests/e2e/online_serving/test_{model_name}.py`
`pytest -s -v -m 'core_model and cpu' --run-level=core_model`
## Chapter 2: L3 Level Testing - Core Integration, Performance, and Accuracy Verification
### L3 Level Testing - Core Integration, Performance, and Accuracy Verification
### 2.1 Testing Purpose
#### 2.1 Testing Purpose
L3 level testing executes after code is merged into the main branch. Its core purpose is to verify the integration effect, key performance indicators, and output accuracy of ***real models*** in ***multiple deployment scenarios***
@@ -553,30 +470,30 @@ L3 level testing executes after code is merged into the main branch. Its core pu
### 2.2 Testing Content and Scope
#### 2.2 Testing Content and Scope
- ***Deployment Scenarios***: Covers richer `online` and `offline` deployment configurations, which may include different hardware configurations, batch sizes, concurrency levels, etc.
- ***Core Verification***:
- 1. ***Inference Functionality***: Ensures real models can perform forward computation normally and return results.
2. ***Accuracy Compliance***: Verifies that the model's evaluation metrics (e.g., accuracy) meet the expected baseline, preventing code changes from introducing accuracy issues.
3. ***Important Performance***: Verifies whether performance (e.g., P99 latency, throughput) in core scenarios meets preset thresholds.
- ***L3 response validation (`advanced_model`)***: At `--run-level advanced_model`, `tests/helpers/assertions.py` adds semantic checks on top of L2 payload gates (see Chapter 1):
- ***L3 response validation (`advanced_model`)***: At `--run-level advanced_model`, `tests/helpers/assertions.py` adds semantic checks on top of L2 payload gates (see L1 & L2 above):
- **Speech / TTS** (`assert_audio_speech_response`): Whisper transcript of returned audio vs `request_config["input"]` (cosine similarity &gt; 0.9 when input text is set); optional `min_audio_bytes` floor; PCM harmonic-to-noise ratio when `response_format` is `pcm`; **preset voice gender** via `_assert_preset_voice_gender_from_audio` when `voice` matches a known preset in `_PRESET_VOICE_GENDER_MAP` (pitch/F0-based classifier; skipped for unknown voices or `pcm` output).
- **Omni multimodal** (`assert_omni_response`): non-empty text/audio outputs; `key_words` in transcript or text; text–audio similarity / containment; preset **`speaker`** gender check (same helper as TTS).
- **Diffusion** (`assert_*_diffusion_response`): image/video dimension and frame-count parity with request parameters where configured.
- **Accuracy suites**: pixel/video similarity and metric baselines under `/tests/e2e/accuracy/` (separate from inline helper assertions).
### 2.3 Test Directory and Execution Files
#### 2.3 Test Directory and Execution Files
- ***Functional Testing***:
- - Online Serving: `/tests/e2e/online_serving/test_{model_name}_expansion.py`
- Offline Inference: `/tests/e2e/offline_inference/test_{model_name}_expansion.py`
- (Note: `_expansion.py` likely means it contains more comprehensive scenario cases compared to L2 tests).
### 2.4 Execution Method and Example
#### 2.4 Execution Method and Example
- ***Trigger Timing***: **`PR Merged`**. Automatically triggered after code review is approved and merged into the main branch (typically via `merge-test` label on the PR before merge).
- ***Diff-aware step skipping***: Same [`source_file_dependencies`](#diff-aware-buildkite-uploads-source_file_dependencies) mechanism as L2—**all E2E Test** leaf steps are diff-gated; other L3 groups always upload. See [Diff-aware Buildkite uploads](#diff-aware-buildkite-uploads-source_file_dependencies).
- ***Diff-aware step skipping***: Same [`source_file_dependencies`](ci_settings.md#step-filtering) mechanism as L2—**all E2E Test** leaf steps are diff-gated; other L3 groups always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
- ***Execution Environment***: ***GPU*** servers.
- ***Script Example***:
@@ -692,27 +609,27 @@ L3 level testing executes after code is merged into the main branch. Its core pu
**Implicit Validation**: `send_omni_request`, `send_audio_speech_request`, and `send_diffusion_request` call `assert_*_response` helpers with the session `--run-level`. At `core_model`, checks are limited to success plus cheap payload sanity (non-empty audio/media, format/content-type, optional `min_audio_bytes`). At `advanced_model` and `full_model`, deep validation adds Whisper transcripts, keyword/similarity, diffusion dimension checks, PCM HNR, etc.
**Audio output debugging**: Deep validation may transcribe returned audio via `convert_audio_bytes_to_text` (Whisper). If an audio keyword or text–audio similarity assertion fails, set `VLLM_OMNI_KEEP_REQUEST_MEDIA=1` before running pytest to keep the intermediate WAV files for inspection (see [Test helper environment variables](#test-helper-environment-variables)).
**Audio output debugging**: Deep validation may transcribe returned audio via `convert_audio_bytes_to_text` (Whisper). If an audio keyword or text–audio similarity assertion fails, set `VLLM_OMNI_KEEP_REQUEST_MEDIA=1` before running pytest to keep the intermediate WAV files for inspection (see [Test helper environment variables](test_system_overview.md#test-helper-environment-variables)).
- ***Run Command (L3 merge)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}.py -m advanced_model --run-level=advanced_model`
- ***Run Command (L4 nightly expansion)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}_expansion.py -m full_model --run-level=full_model`
## Chapter 3: L4 Level Testing - Full Functionality, Performance, and Documentation Testing
### L4 Level Testing - Full Functionality, Performance, and Documentation Testing
### 3.1 Testing Purpose
#### 3.1 Testing Purpose
L4 level testing is a comprehensive quality audit before a version release. It expands upon L3, executing ***full*** functional scenarios, conducting systematic ***performance stress tests***, and simultaneously verifying the correctness of accompanying ***example documentation***. Its purpose is to perform deep validation of the system during off-peak nighttime hours, providing quality trend reports for daytime development and data support for release decisions.
### 3.2 Testing Content and Scope
#### 3.2 Testing Content and Scope
- ***Full Functionality Testing***: Executes all test cases defined in `test_{model_name}_expansion.py`, covering all implemented features, positive flows, boundary conditions, and exception handling.
- ***Performance Testing***: Uses `tests/dfx/perf/tests/test_qwen3_omni_*.json` (Omni), `test_tts.json` / `test_voxcpm2.json` / `test_higgs_audio_v3.json` (TTS), and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (passed to `run_benchmark.py` or `run_diffusion_benchmark.py` via `--test-config-file` in nightly **Perf Test** steps) to drive throughput, latency, and memory benchmarks. Each JSON **case** may declare an optional top-level **`mark`** array: exactly one ``hardware_marks`` object plus pytest marker name strings (`full_model`, `omni` / `tts` / `diffusion`, …). Runners attach those marks to each parametrized `(server, benchmark index)` pair so **local** bulk runs can filter with `-m` (for example `-m "full_model and H100 and diffusion"`). Nightly CI perf jobs select workloads by **`--test-config-file`**, not `-m`. Details are in the Performance Tests example below.
- ***Documentation Testing***: Verifies whether the example code provided to users is runnable and its results match the description.
### 3.3 Test Directory and Execution Files
#### 3.3 Test Directory and Execution Files
- ***Functional Testing***: Same directories as L3.
- ***Performance Test Configuration***: `tests/dfx/perf/tests/test_qwen3_omni_*.json`, `test_tts.json`, `test_voxcpm2.json`, `test_higgs_audio_v3.json`, and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (e.g. `test_qwen_image_vllm_omni.json`, `test_cosmos3_vllm_omni.json`). Optional per-case `mark` for local `-m` filtering is documented in Section 3.4 Performance Tests.
@@ -720,7 +637,7 @@ L4 level testing is a comprehensive quality audit before a version release. It e
- `tests/examples/online_serving/test_{model_name}.py`
- `tests/examples/offline_inference/test_{model_name}.py`
### 3.4 Execution Method and Example
#### 3.4 Execution Method and Example
- ***Trigger Timing***: **`Nightly`**, automatically executed every night.
- ***Execution Environment***: ***GPU*** server clusters to meet the resource demands of performance testing.
@@ -740,20 +657,20 @@ L4 level testing is a comprehensive quality audit before a version release. It e
- ***Run Command***: (Specific commands would depend on the performance testing tool and configuration defined in `nightly.json`).
## Chapter 4: L5 Level Testing - Stability and Reliability Testing
### L5 Level Testing - Stability and Reliability Testing
### 4.1 Testing Purpose
#### 4.1 Testing Purpose
L5 level testing focuses on the performance of model services under ***long-running*** and ***abnormal fault*** scenarios. It aims to uncover deep-seated issues that only manifest under sustained pressure or extreme conditions, such as memory leaks, resource contention, gradual performance degradation, and lack of fault tolerance mechanisms. This is the final, yet crucial, line of defense for ensuring service high availability and production environment robustness.
### 4.2 Testing Content and Scope
#### 4.2 Testing Content and Scope
- ***Long-term Stability (Stability) Testing***: Uses JSON under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` and `test_wan22.json`) to run the service under moderate load for an extended period (e.g., over 12 hours), monitoring whether metrics like memory/VRAM usage, response time, and throughput degrade over time, and whether the service process remains stable.
- ***Reliability Testing***: Uses pytest suites under `tests/dfx/reliability/` to inject controlled faults against a **live** `vllm_omni serve` instance (same **`omni_server` / `omni_server_function`** fixture style as E2E). Current suites emphasize **GPU memory pressure** (CUDA sidecar “memory hog”), **worker / runtime process kill** (`SIGKILL` on `VLLM::Worker` for Qwen3-Omni, `multiprocessing.spawn` for Wan2.2 video workers, or `vLLM-Omni::` for HunyuanImage DiT workers), **large multimodal chat**, **`/v1/videos`**, **`/v1/images/generations`**, or **`/v1/audio/speech`** jobs under OOM, **`/health` → 503** and **fast-fail / non-hanging concurrent** requests after kill, and **OpenAI-style 5xx error contracts** (e.g. text vs text+audio under OOM). **Post-fault recovery** checks exist where enabled (some cases may be `skip` while issues are tracked). See the Reliability `<details>` block in Section 4.4 for file-level responsibilities and CI markers (`slow`, `hardware_test`, POSIX-only kill).
### 4.3 Test Directory and Execution Files
#### 4.3 Test Directory and Execution Files
- ***Stability Test Configuration***: `tests/dfx/stability/tests/test_qwen3_omni.json`, `tests/dfx/stability/tests/test_wan22.json` (one JSON per model / runner family)
- ***Reliability Test Suite*** (`tests/dfx/reliability/`):
@@ -765,7 +682,7 @@ L5 level testing focuses on the performance of model services under ***long-runn
- `conftest.py` — `fault_injector` and `omni_server_after_fault` / `omni_server_after_fault_function` fixtures to run a callable **after** the server is ready.
- `README.md` — Short local run commands for this directory.
### 4.4 Execution Method and Example
#### 4.4 Execution Method and Example
- ***Trigger Timing***: **`Weekly`** (weekly) or **`Days before Release`** (several days before a major release). Due to long execution times, the frequency is lower.
- ***Execution Environment***: ***GPU*** servers, requiring a stable and exclusive testing environment.
@@ -799,7 +716,7 @@ When you want to add L5-level stability test cases, add or extend the appropriat
}
```
#### Parameter Explanation
##### Parameter Explanation
***Overview***
@@ -809,16 +726,16 @@ When you want to add L5-level stability test cases, add or extend the appropriat
| server_params | Yes | Server-side configuration parameters (model, stage configuration, etc.) |
| benchmark_params | Yes | Stability benchmark running parameters (supports multiple configurations) |
#### server_params Configuration
##### server_params Configuration
##### Basic Parameters
###### Basic Parameters
| Parameter | Required | Example | Description |
| ----------------- | -------- | ---------------------------------- | ----------------------------------- |
| model | Yes | "Qwen/Qwen3-Omni-30B-A3B-Instruct" | Model name or path |
| stage_config_name | Yes | "qwen3_omni.yaml" | Stage configuration file name |
##### Dynamic Configuration (update/delete)
###### Dynamic Configuration (update/delete)
Supports incremental modifications based on the basic configuration:
@@ -828,9 +745,9 @@ Supports incremental modifications based on the basic configuration:
| delete | Delete specified configuration items |
***Example***:
You can refer to Test Examples in Chapter 3.4
You can refer to Test Examples in L4 §3.4
#### benchmark_params Configuration
##### benchmark_params Configuration
For stability testing, the key parameters are:
@@ -840,18 +757,18 @@ For stability testing, the key parameters are:
- `max_concurrency`: Maximum number of concurrent requests. When this is used, `request_rate` is set to `inf` internally.
- **num_prompts_per_batch**: Number of prompts sent in each batch. Multiple batches will be executed sequentially within `duration_sec`.
All other optional parameters follow the same rules as the in Chapter 3.4.
All other optional parameters follow the same rules as in L4 §3.4.
</details>
<details>
<summary> Reliability test suite (<code>tests/dfx/reliability</code>)</summary>
#### Purpose and relationship to stability
##### Purpose and relationship to stability
Reliability tests are **short fault-injection** integration runs (L5 **(b)** in `tests/dfx/reliability/README.md`). They complement **stability** JSON-driven long runs: instead of hours of steady traffic, they **perturb** the server (GPU OOM sidecar, fatal signals on selected processes) and check **failure mode** and **latency bounds** (e.g. chat or `/v1/videos` must not hang under concurrent fault-time load).
#### Directory layout
##### Directory layout
| Path | Responsibility |
| ---- | -------------- |
@@ -863,13 +780,13 @@ Reliability tests are **short fault-injection** integration runs (L5 **(b)** in
| `test_reliability_voxcpm2.py` | VoxCPM2: `/v1/audio/speech` under OOM (error contract), **SIGKILL** on `VLLM::` workers and serve/tree targets, health / fast-fail / concurrent speech requests; some OOM cases may be skipped while tracked in issues. |
| `README.md` | Minimal run / collect examples. |
#### Parametrization and markers
##### Parametrization and markers
- Each test module defines a **`RELIABILITY_SCENARIOS`** list (`test_name`, `server_params`: model, `stage_config_name` or diffusion `server_args`, etc.). **`create_reliability_omni_server_params()`** in `tests/dfx/conftest.py` resolves stage paths (including XPU substitutions where applicable) and builds **`OmniServerParams`** lists consumed by **`@pytest.mark.parametrize(..., indirect=True)`** on `omni_server` or `omni_server_function`.
- Cases are tagged **`@pytest.mark.slow`** for weekly / selective CI. GPU-heavy suites use **`@hardware_test(res={"cuda": "H100"}, num_cards=...)`** or **`@hardware_test(res={"cuda": "L4"}, num_cards=1)`** (Qwen3-Omni **2**× H100; Wan2.2 **1**× H100; HunyuanImage DiT **4**× H100; VoxCPM2 **1**× L4).
- **Process-kill** tests use **`@pytest.mark.skipif(os.name == "nt", ...)`** because injection uses POSIX **`pgrep` / `kill`**.
#### CI trigger
##### CI trigger
Weekly Buildkite (`.buildkite/cuda/test-weekly.yml`) runs one step per model suite (trigger: `WEEKLY=1` or PR label `weekly-test`), for example:
@@ -887,7 +804,7 @@ pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py -m "slow"
pytest -s -v tests/dfx/reliability/test_reliability_voxcpm2.py -m "slow"
```
#### Local commands
##### Local commands
```bash
pytest --collect-only tests/dfx/reliability
@@ -897,7 +814,7 @@ pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py -m slow
pytest -s -v tests/dfx/reliability/test_reliability_voxcpm2.py -m slow
```
#### Adding a new model suite
##### Adding a new model suite
1. Add `test_reliability_<model>.py` under `tests/dfx/reliability/`.
2. Define **`RELIABILITY_SCENARIOS`** and pass them through **`create_reliability_omni_server_params()`** with the correct deploy or e2e stage-config directory (same pattern as existing files).
@@ -908,7 +825,3 @@ pytest -s -v tests/dfx/reliability/test_reliability_voxcpm2.py -m slow
- - ***Stability***: `pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/test_stability_wan22.py` (or add `test_stability_<model>.py` alongside a matching JSON config)
- ***Reliability***: `pytest -s -v tests/dfx/reliability/test_reliability_<model>.py -m slow` (current suites: `qwen3_omni`, `wan22`, `hunyuan_image`, `voxcpm2`; add `test_reliability_<suite>.py` and a matching step in `.buildkite/cuda/test-weekly.yml` for new models)
## Summary
This multi-level testing system achieves continuous, progressive validation of model service quality by tightly integrating testing activities with the development workflow (commit, review, merge, release). From rapid unit testing to comprehensive end-to-end testing, and further to in-depth performance, stability, and reliability verification, each level has clear objectives, collectively building a robust quality protection net. By following this system, teams can deliver high-quality, highly reliable model services more efficiently.
docs/contributing/ci/tests_markers.md
-207
View File
@@ -1,207 +0,0 @@
# Markers for Tests
By adding markers before test functions, tests can later be executed uniformly by simply declaring the corresponding marker type.
## Current Markers
Defined in `pyproject.toml`:
| Marker | Description |
| ------------------ | --------------------------------------------------------- |
| `core_model` | L1&L2 tests (run in each PR) |
| `advanced_model` | L3 tests (run on each merge to main) |
| `full_model` | L4 tests (run nightly) |
| `diffusion` | Diffusion model tests |
| `omni` | Omni multimodal model tests |
| `tts` | TTS model tests |
| `cache` | Cache backend tests |
| `parallel` | Parallelism/distributed tests |
| `cpu` | Tests that run on CPU |
| `gpu` | Tests that run on GPU * |
| `cuda` | Tests that run on CUDA * |
| `rocm` | Tests that run on AMD/ROCm * |
| `xpu` | Tests that run on Intel XPU * |
| `npu` | Tests that run on NPU/Ascend * |
| `H100` | Tests that require H100 GPU * |
| `L4` | Tests that require L4 GPU * |
| `MI325` | Tests that require MI325 GPU (AMD/ROCm) * |
| `A2` | Tests that require A2 NPU * |
| `A3` | Tests that require A3 NPU * |
| `distributed_cuda` | Tests that require multi cards on CUDA platform * |
| `distributed_rocm` | Tests that require multi cards on ROCm platform * |
| `distributed_npu` | Tests that require multi cards on NPU platform * |
| `skipif_cuda` | Skip if the num of CUDA cards is less than the required * |
| `skipif_rocm` | Skip if the num of ROCm cards is less than the required * |
| `skipif_npu` | Skip if the num of NPU cards is less than the required * |
| `slow` | Slow tests (may skip in quick CI) |
| `benchmark` | Benchmark tests (decorator on runner test functions; perf JSON uses `full_model` + type marker instead) |
| `local_model` | Tests requiring local / non-HF-hub model weights |
\* Means those markers are auto-added by `@hardware_test` (parametrization decorator) or `hardware_marks` (only returning the list of marks for flexibility).
### Example usage for markers
```python
from tests.helpers.mark import hardware_test
@pytest.mark.core_model
@pytest.mark.omni
@hardware_test(
res={"cuda": "L4", "rocm": "MI325", "npu": "A2"},
num_cards=2,
)
@pytest.mark.parametrize("omni_server", test_params, indirect=True)
def test_video_to_audio()
...
```
### Decorator: `@hardware_test`
This decorator is intended to make hardware-aware, cross-platform test authoring easier and more robust for CI/CD environments. The `hardware_test` decorator in `vllm-omni/tests/helpers/mark.py` performs the following actions:
1. **Applies platform and resource markers**
Adds the appropriate pytest markers for each specified hardware platform (e.g., `cuda`, `rocm`, `xpu`, `npu`) and resource type (e.g., `L4`, `H100`, `MI325`, `B60`, `A2`, `A3`).
```
@pytest.mark.cuda
@pytest.mark.L4
```
2. **Handles multi-card (distributed) scenarios**
For tests requiring multiple cards, it automatically adds distributed markers such as `distributed_cuda`, `distributed_rocm`, or `distributed_npu`.
```
@pytest.mark.distributed_cuda(num_cards=num_cards)
```
3. **Supports flexible card requirements**
Accepts `num_cards` as either a single integer for all platforms or as a dictionary with per-platform values. If not specified, defaults to 1 card per platform.
4. **Integrates resource validation**
On CUDA, adds a skip marker (`skipif_cuda`) if the system does not have the required number of devices.
Support for `skipif_rocm` and `skipif_npu` will be implemented later.
5. **Works with pytest filtering**
Allows tests to be filtered and selected at runtime using standard pytest marker expressions (e.g., `-m "distributed_cuda and L4"`).
#### Example usage for decorator
- Single call for multiple platforms:
```python
@hardware_test(
res={"cuda": "L4", "rocm": "MI325", "xpu": "B60", "npu": "A2"},
num_cards={"cuda": 2, "rocm": 2, "xpu": 2, "npu": 2},
)
```
or
```python
@hardware_test(
res={"cuda": "L4", "rocm": "MI325", "xpu": "B60", "npu": "A2"},
num_cards=2,
)
```
- `res` must be a dict; supported resources: CUDA (L4/H100), ROCm (MI325), XPU (B60), MUSA (S5000), NPU (A2/A3)
- `num_cards` can be int (all platforms) or dict (per platform); defaults to 1 when missing
- Distributed markers (`distributed_cuda`, `distributed_rocm`, `distributed_npu`) are auto-added for multi-card cases
- Filtering examples:
- CUDA only: `pytest -m "distributed_cuda and L4"`
- ROCm only: `pytest -m "distributed_rocm and MI325"`
- NPU only: `pytest -m "distributed_npu"`
### Function: `hardware_marks`
`hardware_marks` returns a list of pytest mark objects with the same signature as `@hardware_test`. Use it when you need more flexibility, such as attaching hardware marks to individual `pytest.param` entries rather than an entire test function.
```python
from tests.helpers.mark import hardware_marks
MULTI_CARD_MARKS = hardware_marks(
res={"cuda": "H100", "rocm": "MI325", "npu": "A2"}, num_cards=2
)
@pytest.mark.parametrize("omni_server", [
pytest.param(OmniServerParams(...), id="case_001", marks=MULTI_CARD_MARKS),
], indirect=True)
def test_feature(omni_server):
...
```
### JSON `mark` field (L4 perf configs)
Perf JSON under `tests/dfx/perf/tests/` can attach pytest marks per **case** (`test_name` block). Parsed by `tests.dfx.conftest.resolve_pytest_marks` and applied to each parametrized run in `run_benchmark.py` / `run_diffusion_benchmark.py`.
When `mark` is present, it must be an **array** with exactly one ``hardware_marks`` object (same semantics as `hardware_marks()` above), followed by registered pytest marker name strings such as `full_model`, `omni`, `tts`, or `diffusion`.
```json
{
"test_name": "test_cosmos3_t2i_official_demo_2gpu",
"mark": [
{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}},
"full_model",
"diffusion"
],
"server_type": "vllm-omni",
"server_params": { "...": "..." },
"benchmark_params": [{ "name": "1024x1024_steps4", "...": "..." }]
}
```
- Local bulk load: `pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and diffusion and H100"`
- Nightly CI perf steps: `--test-config-file tests/dfx/perf/tests/test_<model>_vllm_omni.json` (file selects cases; no `-m`)
- Result filenames use **runtime** GPU detection (`get_runtime_resource_label`); `H100` is omitted on the default CI pool
See [L4 performance test examples](./test_examples/l4_performance_tests.inc.md) for the full schema.
## Add Support for a New Platform
If you want to add support for a new platform (e.g., "tpu" for a new accelerator), follow these steps:
1. **Extend the marker list in your pytest config** so that platform/resource markers are defined:
```toml
# In pyproject.toml or pytest.ini
[tool.pytest.ini_options]
markers = [
# ... existing markers ...
"tpu: Tests that require TPU device",
"TPU_V3: Tests that require TPU v3 hardware",
"distributed_tpu: Tests that require multiple TPU devices",
]
```
2. **Implement a marker construction function for your platform** in `vllm-omni/tests/helpers/mark.py`:
```python
# In vllm-omni/tests/helpers/mark.py
def tpu_marks(*, res: str, num_cards: int):
test_platform = pytest.mark.tpu
if res == "TPU_V3":
test_resource = pytest.mark.TPU_V3
else:
raise ValueError(
f"Invalid TPU resource type: {res}. Supported: TPU_V3")
if num_cards == 1:
return [test_platform, test_resource]
else:
test_distributed = pytest.mark.distributed_tpu(num_cards=num_cards)
# Optionally: add skipif_tpu when implemented
return [test_platform, test_resource, test_distributed]
```
3. **Update `hardware_marks` to recognize your new platform**:
In the relevant place (see the `hardware_marks` implementation), add:
```python
if platform == "tpu":
marks = tpu_marks(res=resource, num_cards=cards)
```
(`hardware_test` calls `hardware_marks` internally, so both will pick up the change.)
4. **(Recommended) Add a test using your new markers**:
```python
@hardware_test(
res={"tpu": "TPU_V3"},
num_cards=2,
)
def test_my_tpu_feature():
...
```
**Summary**:
- Add pytest markers for your new platform/resources
- Implement a marker function (`xxx_marks`)
- Plug into `hardware_marks`
- You're done: tests using `@hardware_test` or `hardware_marks` with your platform now automatically get the correct markers, distribution, and isolation!
See code in `vllm-omni/tests/helpers/mark.py` for existing examples (`cuda_marks`, `rocm_marks`, `npu_marks`).
docs/contributing/ci/tests_style.md
-466
View File
@@ -1,466 +0,0 @@
# Test File Structure and Style Guide
To ensure project maintainability and sustainable development, we encourage contributors to submit test code (unit tests, system tests, or end-to-end tests) alongside their code changes. This document outlines the guidelines for organizing and naming test files.
## Checklist before submitting your test files
1. The file is saved in an appropriate place and the file name is clear.
2. The coding style follows the requirements outlined below.
3. All test functions have appropriate pytest markers.
4. For tests that need run in CI, please ensure it labeled as ``@pytest.mark.core_model` the test is configured under the `./buildkite/` folder.
## Test Types
For more details about our [Five Levels Tests design](../ci/CI_5levels.md).
### Unit Tests and System Tests
For unit tests and system tests, we strongly recommend placing test files in the same directory structure as the source code being tested, using the naming convention `test_*.py`.
### End-to-End (E2E) Tests for Models
End-to-end tests verify the complete functionality of a system or component. For our project, the E2E tests for different omni models are organized into two subdirectories:
- **`tests/e2e/offline_inference/`**: Tests for offline inference modes (e.g., Qwen3Omni offline inference)
- **`tests/e2e/online_serving/`**: Tests for online serving scenarios (e.g., API server tests)
## Test Directory Structure
The ideal directory structure mirrors the source code organization. Legend: `✅` = test exists, `⬜` = suggested to add.
```
vllm_omni/ tests/
├── config/ → ├── config/
│ ├── model.py │ └── test_model.py ⬜
│ └── lora.py │ └── test_lora.py ⬜
│
├── core/ → ├── core/
│ └── sched/ │ └── sched/
│ ├── omni_ar_scheduler.py │ ├── test_omni_ar_scheduler.py ⬜
│ ├── omni_generation_scheduler.py │ ├── test_omni_generation_scheduler.py ⬜
│ └── output.py │ └── test_output.py ✅ currently in entrypoints/test_omni_new_request_data.py (tests output.OmniNewRequestData)
│
├── diffusion/ → ├── diffusion/
│ ├── diffusion_engine.py │ ├── test_diffusion_engine.py ⬜
│ ├── attention/ │ ├── attention/
│ │ ├── layer.py │ │ ├── test_attention_sp.py ✅
│ │ └── backends/ │ │ └── test_flash_attn.py ✅
│ ├── distributed/ │ ├── distributed/
│ │ └── ... │ │ ├── test_comm.py ✅
│ │ │ │ ├── test_cfg_parallel.py ✅
│ │ │ │ └── test_sp_plan_hooks.py ✅
│ ├── lora/ │ ├── lora/
│ │ └── ... │ │ ├── test_base_linear.py ✅
│ │ │ │ └── test_lora_manager.py ✅
│ ├── models/ │ ├── models/
│ │ ├── qwen_image/ │ │ ├── qwen_image/ (e2e coverage)
│ │ ├── ovis_image/ │ │ ├── ovis_image/
│ │ │ └── ... │ │ │ └── test_ovis_image.py ✅
│ │ ├── z_image/ │ │ └── z_image/
│ │ └── ... │ │ └── test_zimage_tp_constraints.py ✅
│ └── worker/ │ └── worker/
│ ├── diffusion_worker.py │ └── test_diffusion_worker.py ✅ file at diffusion/test_diffusion_worker.py
│ └── diffusion_model_runner.py │
│
├── distributed/ → ├── distributed/
│ └── omni_connectors/ │ └── omni_connectors/
│ ├── adapter.py │ ├── test_adapter_and_flow.py ✅
│ ├── kv_transfer_manager.py │ ├── test_basic_connectors.py ✅
│ ├── connectors/ │ ├── test_kv_flow.py ✅
│ └── utils/ │ └── test_omni_connector_configs.py ✅
│
├── engine/ → ├── engine/
│ ├── input_processor.py │ ├── test_input_processor.py ⬜ (no processor.py in source)
│ ├── output_processor.py │ └── test_output_processor.py ⬜
│ └── arg_utils.py │ └── test_arg_utils.py ⬜
│
├── entrypoints/ → ├── entrypoints/
│ ├── stage_utils.py │ ├── test_stage_utils.py ✅
│ ├── cli/ │ ├── cli/ (benchmarks/test_serve_cli.py covers CLI serve)
│ │ └── ... │ │ └── test_*.py ⬜
│ └── openai/ │ └── openai_api/ # maps to entrypoints/openai/
│ ├── api_server.py │ ├── test_api_server.py ⬜ (e2e indirect coverage)
│ ├── serving_chat.py │ ├── test_serving_chat_sampling_params.py ✅
│ ├── serving_speech.py │ ├── test_serving_speech.py ✅
│ └── image_api_utils.py │ └── test_image_server.py ✅
│
├── inputs/ → ├── inputs/
│ ├── data.py │ ├── test_data.py ⬜
│ ├── parse.py │ ├── test_parse.py ⬜
│ └── preprocess.py │ └── test_preprocess.py ✅ currently in entrypoints/test_omni_input_preprocessor.py
│
├── model_executor/ → ├── model_executor/
│ ├── layers/ │ ├── layers/
│ │ └── mrope.py │ │ └── test_mrope.py ⬜
│ ├── model_loader/ │ ├── model_loader/
│ │ └── weight_utils.py │ │ └── test_weight_utils.py ⬜
│ ├── models/ │ ├── models/
│ │ ├── qwen2_5_omni/ │ │ ├── qwen2_5_omni/
│ │ │ ├── qwen2_5_omni_thinker.py │ │ │ ├── test_audio_length.py ✅
│ │ │ ├── qwen2_5_omni_talker.py │ │ │ ├── test_qwen2_5_omni_thinker.py ⬜
│ │ │ └── qwen2_5_omni_token2wav.py │ │ │ ├── test_qwen2_5_omni_talker.py ⬜
│ │ └── qwen3_omni/ │ │ │ └── test_qwen2_5_omni_token2wav.py ⬜
│ │ └── ... │ │ └── qwen3_omni/
│ ├── stage_configs/ │ │ └── test_*.py ⬜
│ │ └── *.yaml │ └── stage_configs/ (used by e2e, test_*.py can be added) ⬜
│ └── stage_input_processors/ │ └── stage_input_processors/
│ └── ... │ └── test_*.py ⬜
│
├── sample/ → ├── sample/
│ └── __init__.py │ └── test_*.py ⬜
│
├── utils/ → ├── utils/
│ └── __init__.py │ └── test_*.py ⬜ (no platform_utils.py currently)
│
├── worker/ → ├── worker/
│ ├── gpu_ar_model_runner.py │ ├── test_gpu_ar_model_runner.py ⬜
│ ├── gpu_ar_worker.py │ ├── test_gpu_ar_worker.py ⬜
│ ├── gpu_generation_model_runner.py │ ├── test_gpu_generation_model_runner.py ✅
│ ├── gpu_generation_worker.py │ ├── test_gpu_generation_worker.py ⬜
│ ├── gpu_model_runner.py │ ├── test_omni_gpu_model_runner.py ✅
│ └── mixins.py │ └── (npu under platforms/npu/worker/) # not worker/npu/
│
├── platforms/ → (no tests/platforms/, e2e and stage_configs provide indirect coverage)
│ ├── cuda/
│ ├── npu/worker/ # NPU worker here, not vllm_omni/worker/npu/
│ ├── rocm/
│ └── xpu/worker/
│
├── outputs.py → test_outputs.py ✅ (at tests root)
├── (logger, patch, request, version) → (no corresponding unit test)
│
└── e2e (tests side only) → ├── e2e/
├── online_serving/ ✅ non-empty
│ ├── test_qwen2_5_omni.py
│ ├── test_async_omni.py
│ ├── test_qwen3_omni.py
│ ├── test_qwen3_omni_expansion.py
│ ├── test_mimo_audio.py
│ └── test_images_generations_lora.py
└── offline_inference/ ✅
├── test_qwen2_5_omni.py
├── test_qwen3_omni.py
├── test_bagel_text2img.py
├── test_z_image.py
├── test_wan22.py
├── test_zimage_tensor_parallel.py
├── test_cache_dit.py
├── test_teacache.py
├── test_stable_audio_expansion.py
├── test_diffusion_cpu_offload.py
├── test_diffusion_layerwise_offload.py
├── test_diffusion_lora.py
├── test_sequence_parallel.py
├── test_qwen_image_edit_expansion.py
└── stage_configs/ (legacy schema, still present
├── bagel_*.yaml for unmigrated models)
└── npu/, rocm/, etc.
# Migrated models (qwen3_omni_moe, qwen2_5_omni, qwen3_tts) live under
# vllm_omni/deploy/ instead — see docs/configuration/stage_configs.md.
examples/ tests
│ └── examples
├── online_serving/ → ├── online_serving/
│ └── {doc_page_title}/README.md │ └── test_{doc_page_title}.py ⬜
└── offline_inference/ → └── offline_inference/
└── {doc_page_title}/README.md └── test_{doc_page_title}.py ⬜
```
### Naming Conventions
- **Unit Tests**: Use `test_<module_name>.py` format. Example: `stage_utils.py` → `test_stage_utils.py`
- **E2E Tests**: Place in `tests/e2e/offline_inference/` or `tests/e2e/online_serving/` with descriptive names. Example: `tests/e2e/offline_inference/test_qwen3_omni.py`, `tests/e2e/offline_inference/test_diffusion_model.py`
- **Expansion Tests**
### Best Practices
1. **Mirror Source Structure**: Test directories should mirror the source code structure
2. **Test Type Indicators**: Use comments to indicate test types (UT for unit tests, E2E for end-to-end tests)
3. **Shared Resources**: Place shared test configurations (e.g., CI configs) in appropriate subdirectories
4. **Consistent Naming**: Follow the `test_*.py` naming convention consistently across all test files
## Test codes requirements
### Coding style
1. **File header**: Add SPDX license header to all test files
2. **Imports**: Pls don't use manual `sys.path` modifications, use standard imports instead.
3. **Test type differentiation**:
- Unit tests: Maintain mock style
- E2E tests for models: Consider using OmniRunner uniformly, avoid decorators
4. **Documentation**: Add docstrings to all test functions
5. **Environment variables**: Set uniformly in `conftest.py` or at the top of files
6. **Type annotations**: Add type annotations to all test function parameters
7. **Pytest Markers**: Add necessary markers like `@pytest.mark.core_model` and use `@hardware_test` to declare hardware requirements (check detailed in [Markers for Tests](../ci/tests_markers.md)).
### Template
#### E2E - Online serving
E2E Online tests for Qwen3-Omni model with mix input and audio+text output. Based on `tests/e2e/online_serving/test_qwen3_omni.py`.
```python
"""
E2E Online tests for Qwen3-Omni model with mix input and audio+text output.
"""
import os
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
import threading
from pathlib import Path
import openai
import pytest
from tests.helpers.media import (
convert_audio_bytes_to_text,
cosine_similarity_text,
generate_synthetic_video,
)
from tests.helpers.runtime import OmniServer, dummy_messages_from_mix_data
from tests.helpers.stage_config import get_deploy_config_path, modify_stage_config
from vllm_omni.platforms import current_omni_platform
# Edit: model name and stage config path
models = ["Qwen/Qwen3-Omni-30B-A3B-Instruct"]
#If you use the default configuration file, you can directly use the following address.
def get_default_config():
return get_deploy_config_path("ci/qwen3_omni_moe.yaml")
#If you need to modify the configuration file, you can use modify_stage_config.
def get_chunk_config():
path = modify_stage_config(
get_default_config(),
updates={
"async_chunk": True,
"stage_args": {
0: {
"engine_args.custom_process_next_stage_input_func": "vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_async_chunk"
},
1: {
"engine_args.custom_process_next_stage_input_func": "vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_async_chunk"
},
},
},
deletes={"stage_args": {2: ["custom_process_input_func"]}},
)
return path
stage_configs = [get_default_config(), CHUNK_CONFIG_PATH]
test_params = [(model, stage_config) for model in models for stage_config in stage_configs]
#Please use this method to launch the online instance.
_omni_server_lock = threading.Lock()
@pytest.fixture(scope="module")
def omni_server(request):
"""Start vLLM-Omni server as a subprocess. Use module scope for multi-stage init (10-20+ min)."""
with _omni_server_lock:
model, stage_config_path = request.param
with OmniServer(
model,
["--stage-configs-path", stage_config_path, "--stage-init-timeout", "120"],
) as server:
yield server
@pytest.fixture
def client(omni_server):
"""OpenAI client for the running vLLM-Omni server."""
return openai.OpenAI(
base_url=f"http://{omni_server.host}:{omni_server.port}/v1",
api_key="EMPTY",
)
#Please use function definitions above the test function to define the prompts and other parameters you need.
def get_system_prompt():
return {
"role": "system",
"content": [
{
"type": "text",
"text": (
"You are Qwen, a virtual human developed by the Qwen Team, "
"Alibaba Group, capable of perceiving auditory and visual inputs, "
"as well as generating text and speech."
),
}
],
}
...
#Please define test case tags according to the instructions in the marker documentation.
@pytest.mark.core_model
@pytest.mark.omni
@pytest.mark.parametrize("omni_server", test_params, indirect=True)
def test_mix_to_text_audio_001(client: openai.OpenAI, omni_server, request) -> None:
# PLEASE FOLLOW THESE TEMPLATE INSTRUCTIONS:
# ============================================================================
# TEMPLATE USAGE GUIDE:
# 1. Copy this entire function as a starting point for multi-modal tests
# 2. Update the test name to reflect your specific test scenario
# 3. Modify input/output modalities as needed (see OPTIONS section below)
# 4. Adjust assertions based on your expected outcomes
# 5. Add custom validation logic for your specific use case
# ============================================================================
#Please list the relevant test points.
"""
Test multi-modal input processing and text/audio output generation via OpenAI API.
Deploy Setting: default yaml
Input Modal: text + audio + video + image
Output Modal: text + audio
Input Setting: stream=True
Datasets: single request
"""
# SECTION 1: TEST SETUP AND INITIALIZATION
# =========================================
# INSTRUCTIONS: Initialize test variables and prepare test environment
# MODIFY: Add any additional test setup required for your scenario
e2e_list = list()
# SECTION 2: TEST DATA GENERATION
# ================================
# INSTRUCTIONS: Generate or load test data for each input modality
# MODIFY: Replace synthetic generators with your actual data sources
# VIDEO DATA - Generate synthetic video for testing
# FORMAT: data:video/mp4;base64,{base64_encoded_video}
# PARAMETERS: width, height, duration_frames
video_data_url = f"data:video/mp4;base64,{generate_synthetic_video(224, 224, 300)['base64']}"
# IMAGE DATA - Generate synthetic image for testing
# FORMAT: data:image/jpeg;base64,{base64_encoded_image}
# PARAMETERS: width, height
image_data_url = f"data:image/jpeg;base64,{generate_synthetic_image(224, 224)['base64']}"
# AUDIO DATA - Generate synthetic audio for testing
# FORMAT: data:audio/wav;base64,{base64_encoded_audio}
# PARAMETERS: duration_seconds, channels
audio_data_url = f"data:audio/wav;base64,{generate_synthetic_audio(5, 1)['base64']}"
# SECTION 3: MESSAGE CONSTRUCTION
# ================================
# INSTRUCTIONS: Assemble the complete message payload for API request
# MODIFY: Add/remove modalities or change prompt structure as needed
# USAGE: Construct a message containing all input modalities
# IMPORTANT: Ensure the message structure matches OpenAI API expectations
# CUSTOMIZATION POINTS:
# - system_prompt: Controls the assistant's behavior
# - content_text: The user's text prompt/question
# - *_data_url: URLs for media content (video/image/audio)
messages = dummy_messages_from_mix_data(
system_prompt=get_system_prompt(),
video_data_url=video_data_url,
image_data_url=image_data_url,
audio_data_url=audio_data_url,
content_text=get_prompt("mix"),
)
# SECTION 4: API REQUEST EXECUTION
# =================================
# INSTRUCTIONS: Make the API call and measure performance
# MODIFY: Add timeout, retry logic, or additional parameters
start_time = time.perf_counter()
chat_completion = client.chat.completions.create(model=omni_server.model, messages=messages, stream=True)
#Call using your preferred method and obtain the final audio and text outputs.
...
# SECTION 5: OUTPUT VALIDATION
# =============================
# INSTRUCTIONS: Verify that outputs meet expected criteria
# MODIFY: Adjust validation logic for your specific requirements
# ASSERTION 1: E2E Validation
# PURPOSE: Verify that the E2E latency is less than the baseline.
current_e2e = time.perf_counter() - start_time
print(f"the request e2e is: {current_e2e}")
e2e_list.append(current_e2e)
print(f"the avg e2e is: {sum(e2e_list) / len(e2e_list)}")
# ASSERTION 2: Text Output Validation
# PURPOSE: Verify that text output was generated with keyword content
assert text_content is not None and len(text_content) >= 2, "No text output is generated"
assert any(
keyword in text_content.lower() for keyword in ["square", "quadrate", "sphere", "globe", "circle", "round"]
), "The output does not contain any of the keywords."
# ASSERTION 3: Cross-Modal Consistency
# PURPOSE: Verify text and audio outputs convey the same information
# CUSTOMIZATION: Adjust similarity threshold (0.9) based on accuracy requirements
assert audio_data is not None, "No audio output is generated"
audio_content = convert_audio_bytes_to_text(audio_data)
print(f"text content is: {text_content}")
print(f"audio content is: {audio_content}")
similarity = cosine_similarity_text(audio_content.lower(), text_content.lower())
print(f"similarity is: {similarity}")
assert similarity > 0.9, "The audio content is not same as the text"
```
#### E2E - Offline inference
```python
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
Offline E2E smoke test for an omni model (video → audio).
"""
import os
from pathlib import Path
import pytest
from vllm.assets.video import VideoAsset
from tests.helpers.mark import hardware_test
from ..multi_stages.conftest import OmniRunner
# Optional: set process start method for workers
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
models = ["{your model name}"] #Edit here to load your model
stage_configs = [str(Path(__file__).parent / "stage_configs" / {your model yaml})] #Edit here to load your model yaml
# Create parameter combinations for model and stage config
test_params = [(model, stage_config) for model in models for stage_config in stage_configs]
# function name: test_{input_modality}_to_{output_modality}
# modality candidate: text, image, audio, video, mixed_modalities
@pytest.mark.core_model
@pytest.mark.omni
@hardware_test(
res={"cuda": "L4", "rocm": "MI325", "npu": "A2"},
num_cards=2,
)
@pytest.mark.parametrize("test_config", test_params)
def test_video_to_audio(omni_runner: type[OmniRunner], model: str) -> None:
"""Offline inference: video input, audio output."""
model, stage_config_path = test_config
with omni_runner(model, seed=42, stage_configs_path=stage_config_path) as runner:
# Prepare inputs
video = VideoAsset(name="sample", num_frames=4).np_ndarrays
outputs = runner.generate_multimodal(
prompts="Describe this video briefly.",
videos=video,
)
# Minimal assertions: got outputs and at least one audio result
assert outputs
has_audio = any(o.final_output_type == "audio" for o in outputs)
assert has_audio
```
docs/contributing/model/adding_diffusion_model.md
+1 -1
View File
@@ -685,7 +685,7 @@ For a fair comparison, keep the same **prompt**, **seed**, **resolution**, **num
To ensure project maintainability and sustainable development, please submit test code (unit tests, system tests, or end-to-end tests) alongside their code changes.
For comprehensive testing guidelines and the definition of test levels (L1-L5), please refer to the [Multi-Level Automated Testing System Documentation](../ci/CI_5levels.md). You are at least required to add an L4 *functionality* test described in that document.
For the definition of test levels (L1-L5), see [Test System Overview](../ci/test_system_overview.md). You are at least required to add an L4 *functionality* test as described in [Test Writing Guide](../ci/test_writing_guide.md#l4-level-testing-full-functionality-performance-and-documentation-testing).
---
docs/contributing/model/adding_omni_model.md
+1 -1
View File
@@ -540,7 +540,7 @@ Do **not** add a no-suffix `thinker2talker` when a `sync_process_input_func` is
## Testing
For comprehensive testing guidelines, please refer to the [Test File Structure and Style Guide](../ci/tests_style.md).
For comprehensive testing guidelines, please refer to the [Test Writing Guide](../ci/test_writing_guide.md).
## Adding a Model Recipe
docs/contributing/model/adding_tts_model.md
+1 -1
View File
@@ -637,7 +637,7 @@ Key points:
## Testing
For general testing conventions, see [tests_style.md](../ci/tests_style.md).
For general testing conventions, see [Test Writing Guide](../ci/test_writing_guide.md).
Recommended test cases for a new TTS model: