[CI/Build] Strengthen pre-commit with markdownlint, SPDX, and policy hooks (#6273)
Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: [Your Name] <your.email@example.com> Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
This commit is contained in:
wangyu authored and GitHub committed
1 parent
10906e3a6d
commit
273ffce130
30 files changed
+1497
-332
No files matched your search
@@ -44,6 +44,149 @@ vLLM-Omni's `pre-commit` hooks will now run automatically every time you commit.
|
||||
pre-commit run --show-diff-on-failure --color=always --all-files # runs on all files (short for --all-files)
|
||||
```
|
||||
|
||||
!!! warning
|
||||
GitHub Actions `pre-commit` **skips** several local gates so a whole-tree
|
||||
`--all-files` run stays green while historical debt is cleaned up. The
|
||||
current `SKIP` list in
|
||||
[`.github/workflows/pre-commit.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.github/workflows/pre-commit.yml)
|
||||
is `check-test-ci-coverage`, `markdownlint-cli2`, `shellcheck`,
|
||||
`check-spdx-header`, and `mypy-3.10`. Those hooks still run on **your
|
||||
commit** for changed files. A passing GitHub pre-commit check does not mean
|
||||
they passed locally. Hooks **not** on that list (forbidden imports,
|
||||
`torch.cuda`, TTS adapter ratchet, Buildkite schema, Ruff, typos, …) run in
|
||||
both places.
|
||||
|
||||
The hooks below are new relative to the previous Omni config (Ruff, typos,
|
||||
actionlint, YAML/whitespace, DCO sign-off, and the pickle-only checker were
|
||||
already there). `check-pickle-imports` is gone: pickle is now one rule inside
|
||||
`check-forbidden-imports`.
|
||||
|
||||
| Hook id | Enforces | GitHub Actions |
|
||||
| ------- | -------- | -------------- |
|
||||
| `markdownlint-cli2` | Markdown in `docs/`, `recipes/`, `README.md`, `CONTRIBUTING.md` (not `.claude/` / `.cursor/` / `CLAUDE.md`) | skipped |
|
||||
| `mypy-3.10` | Type-check changed `vllm_omni/` files (model trees excluded). `tests/` uses `--follow-imports skip` | skipped |
|
||||
| `mypy-3.11` / `3.12` / `3.13` | Same checker, extra Python versions | not installed; `pre-commit run --hook-stage manual mypy-3.12` |
|
||||
| `check-test-ci-coverage` | Every `tests/**/test_*.py` has a CI level mark and a hardware mark/helper | skipped |
|
||||
| `check-tts-adapter-migration` | `self._tts_model_type` branches in `serving_speech.py` must not increase | runs |
|
||||
| `shellcheck` | `*.sh` quoting / undefined vars | skipped |
|
||||
| `check-spdx-header` | Omni SPDX header on `.py` / `.pyi` / `.sh` / `.rs` / `.proto` | skipped |
|
||||
| `check-forbidden-imports` | pickle, stdlib `re`/`base64`, Triton, TileLang, Hugging Face Hub download/API | runs |
|
||||
| `check-torch-cuda-call` | Direct `torch.cuda.*` helpers outside platform adapters | runs |
|
||||
| `check-buildkite` | `.buildkite/*.yml` against the Buildkite schema | runs |
|
||||
|
||||
Related config (not a new hook id): Ruff `TID251` also bans `librosa` and several
|
||||
`torch.cuda.*` names; typos is pinned and ignores git-describe hashes (`+g` +
|
||||
hex) and NumPy `writeable`. Keep the `suggestion` hook last in
|
||||
`.pre-commit-config.yaml` so autofix output does not bury the SKIP tip.
|
||||
|
||||
#### SPDX headers
|
||||
|
||||
Source files must start with:
|
||||
|
||||
```text
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
```
|
||||
|
||||
Rust and proto files use `//` instead of `#`. Empty files are exempt. A stale
|
||||
`Copyright contributors to the vLLM project` line is **rewritten in place** to
|
||||
the Omni copyright; the hook then exits non-zero so you restage the rewrite.
|
||||
|
||||
#### Forbidden imports
|
||||
|
||||
Under `vllm_omni/`, do not use:
|
||||
|
||||
- stdlib `re` → `import regex as re`
|
||||
- stdlib `base64` → `import pybase64` (or `import pybase64 as base64`)
|
||||
- `pickle` / `cloudpickle` (also banned in most tests)
|
||||
- `from huggingface_hub import snapshot_download` / `hf_hub_download` / `HfApi`
|
||||
/ similar → `vllm.transformers_utils.repo_utils`
|
||||
- direct `triton` / `tilelang` → `vllm.triton_utils` / `vllm.tilelang_utils`
|
||||
|
||||
`examples/`, `tests/`, `benchmarks/`, `apps/`, `docs/`, `tools/`, `scripts/`,
|
||||
`.buildkite/`, and `.github/` may keep stdlib `re` and `base64`
|
||||
(`_NON_LIBRARY_DIRS`). Pickle is stricter: tests are not exempt unless the
|
||||
file is on the pickle allowlist.
|
||||
|
||||
#### `torch.cuda` call sites
|
||||
|
||||
Prefer `current_omni_platform` / `OmniPlatform`. Platform adapters under
|
||||
`vllm_omni/platforms/` are allowed; other files must not add new
|
||||
`torch.cuda.*` helpers (`device_count`, `synchronize`, `empty_cache`,
|
||||
`device()`, `manual_seed`, and similar). `import torch` is fine.
|
||||
|
||||
#### shellcheck
|
||||
|
||||
The hook does **not** download a binary. Install `shellcheck` with a signed
|
||||
package manager, then re-run:
|
||||
|
||||
- Debian/Ubuntu/WSL: `sudo apt-get install shellcheck`
|
||||
- Fedora: `sudo dnf install ShellCheck`
|
||||
- macOS: `brew install shellcheck`
|
||||
- Git Bash (MINGW): `scoop install shellcheck` so `shellcheck.exe` is on PATH.
|
||||
WSL is Linux: use `apt-get`, not a Windows `.exe`.
|
||||
|
||||
If no native binary is found, the wrapper prints those commands (and
|
||||
[the upstream install page](https://github.com/koalaman/shellcheck?tab=readme-ov-file#installing))
|
||||
and exits 1. AMD and other vendor scripts are **not** excluded.
|
||||
|
||||
#### mypy
|
||||
|
||||
`mypy-3.10` runs on commit for changed files. `[tool.mypy]` uses Python 3.12,
|
||||
`ignore_missing_imports = true`, and `follow_imports = "silent"`. Wrapper
|
||||
`tools/pre_commit/mypy.py` skips `vllm_omni/model_executor/models/` and
|
||||
`vllm_omni/diffusion/models/`, and type-checks `tests/` with
|
||||
`--follow-imports skip`. Extra versions:
|
||||
|
||||
```bash
|
||||
pre-commit run --hook-stage manual mypy-3.12
|
||||
```
|
||||
|
||||
#### Test CI marks
|
||||
|
||||
`check-test-ci-coverage` requires each collected test module under `tests/` to
|
||||
have at least one CI **level** mark (`core_model`, `advanced_model`,
|
||||
`full_model`, `local_model`, or `slow`) and a **hardware** mark (`cpu`, `cuda`,
|
||||
`H100`, …) or helper (`hardware_test(` / `hardware_marks(`). GitHub Actions
|
||||
skips this hook; local commit does not. See the
|
||||
[test writing guide](./ci/test_writing_guide.md).
|
||||
|
||||
#### TTS adapter ratchet
|
||||
|
||||
`check-tts-adapter-migration` counts `self._tts_model_type` comparisons in
|
||||
`vllm_omni/entrypoints/openai/serving_speech.py`. The count must stay at or
|
||||
below `MAX_MODEL_TYPE_BRANCHES` in
|
||||
[`tools/pre_commit/check_tts_adapter.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_tts_adapter.py).
|
||||
New per-model behavior belongs in
|
||||
`vllm_omni/entrypoints/openai/tts_adapters/`. Removing branches **must** lower
|
||||
the budget in the same change; raising it is a reviewable policy edit.
|
||||
|
||||
#### Markdownlint and Buildkite
|
||||
|
||||
`markdownlint-cli2` auto-fixes `docs/`, `recipes/`, and the root README /
|
||||
CONTRIBUTING (rules in `.markdownlint.yaml`). Skill files under `.claude/` and
|
||||
`.cursor/` are excluded.
|
||||
|
||||
`check-buildkite` validates `.buildkite/*.yml` against the official schema
|
||||
after the same expansion `upload_pipeline.py` uses. Omni-only keys such as
|
||||
`mirror_hardwares` are stripped or rendered first. A few files are skipped in
|
||||
`SKIP_FILES` inside `tools/pre_commit/check_buildkite.py`.
|
||||
|
||||
#### Extending allowlists and budgets
|
||||
|
||||
Prefer fixing the call site. Growing an allowlist or raising a budget is a
|
||||
policy change and needs review — do not edit these just to make the hook pass.
|
||||
|
||||
| Gate | File | What to edit |
|
||||
| ---- | ---- | ------------ |
|
||||
| pickle, `re`, `base64`, Hugging Face Hub, Triton/TileLang | [`tools/pre_commit/check_forbidden_imports.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_forbidden_imports.py) | `CHECK_IMPORTS[<rule>].allowed_files`. Use `allowed_dirs` only when a whole tree should stay exempt. |
|
||||
| `torch.cuda` | [`tools/pre_commit/check_torch_cuda.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_torch_cuda.py) | `ALLOWED_FILES`, or `ALLOWED_PREFIXES` for a platform adapter tree. |
|
||||
| TTS `_tts_model_type` branches | [`tools/pre_commit/check_tts_adapter.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_tts_adapter.py) | `MAX_MODEL_TYPE_BRANCHES` (down only, unless the PR justifies a raise). |
|
||||
| Buildkite skip list | [`tools/pre_commit/check_buildkite.py`](https://github.com/vllm-project/vllm-omni/blob/main/tools/pre_commit/check_buildkite.py) | `SKIP_FILES` for pipelines that cannot be expanded. |
|
||||
|
||||
To skip one hook for a single commit (discouraged): `SKIP=<hook-id> git commit`.
|
||||
To bypass every hook: `git commit --no-verify` (also discouraged).
|
||||
|
||||
### Documentation
|
||||
|
||||
MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file, `mkdocs.yml`.
|
||||
@@ -162,12 +305,14 @@ Only specific types of PRs will be reviewed. The PR title is prefixed appropriat
|
||||
Before submitting a PR, run the [precheck-pr skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/precheck-pr/SKILL.md) with the code agent for a self-review against project conventions:
|
||||
|
||||
The skill offers two modes:
|
||||
|
||||
- **Quick (~3 min):** catches showstoppers — PR title format, missing benchmark claims, rebase status
|
||||
- **Full (~10 min):** thorough maintainer-grade review — dead code scan, copy-paste detection, import hygiene
|
||||
|
||||
The precheck covers five PR types: Bug Fix, Performance, New Model, Diffusion Model, and General. Each type has a tailored checklist that validates evidence quality (repro steps, A/B benchmarks, registry entries, etc.). See the [precheck-pr skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/precheck-pr/SKILL.md) for the full checklist.
|
||||
|
||||
### Local Test
|
||||
|
||||
Please run the L1 and L2 test cases locally first and attach the results before contacting us to add the "ready" label. Please refer to the [test instructions](./ci/test_execution_guide.md) for running the test cases.
|
||||
|
||||
### Automatic skip-ci (docs and pytest skip marks)
|
||||
|
||||
@@ -6,7 +6,7 @@ When you open a PR against vLLM-Omni, several CI checks run automatically:
|
||||
|
||||
| Check | Platform | What it does |
|
||||
| ----- | -------- | ------------ |
|
||||
| **pre-commit** | GitHub Actions | Runs linting (Ruff), formatting, spell-checking (typos), and YAML validation. |
|
||||
| **pre-commit** | GitHub Actions | Runs Ruff, formatting, typos, YAML validation, forbidden imports, `torch.cuda`, TTS adapter ratchet, and Buildkite schema. Local-only gates (SPDX, shellcheck, markdownlint, mypy-3.10, test marks) are in the workflow `SKIP` env — see [Linting](../README.md#linting). |
|
||||
| **Build Wheel** | GitHub Actions | Builds Python wheels for Python 3.11 and 3.12 on Ubuntu. Skipped for docs-only or Markdown-only changes (controlled by `paths-ignore` in the workflow). |
|
||||
| **DCO** | GitHub | Verifies every commit has a `Signed-off-by` line. |
|
||||
| **docs/readthedocs.org:vllm-omni** | Read the Docs | Builds the MkDocs documentation site. |
|
||||
@@ -29,6 +29,13 @@ pre-commit run --all-files
|
||||
|
||||
Then commit the fixes and push.
|
||||
|
||||
A green GitHub pre-commit job can still miss SPDX, shellcheck, mypy-3.10,
|
||||
markdownlint, and test-mark coverage because CI skips those IDs. If your
|
||||
**local** commit hook fails, follow [Linting](../README.md#linting) for the
|
||||
full hook list (forbidden imports, `torch.cuda`, TTS adapter ratchet, and
|
||||
Buildkite schema still run in GHA). Do not grow allowlists or raise
|
||||
`MAX_MODEL_TYPE_BRANCHES` without review.
|
||||
|
||||
### DCO failures
|
||||
|
||||
Every commit must include a `Signed-off-by` line. If you forgot, amend your commits:
|
||||
|
||||
@@ -24,38 +24,39 @@ Deploy YAMLs: `vllm_omni/deploy/` via `tests.helpers.stage_config.get_deploy_con
|
||||
By adding markers before test functions, tests can later be executed uniformly by simply declaring the corresponding marker type.
|
||||
|
||||
### Current Markers
|
||||
|
||||
Defined in `pyproject.toml`:
|
||||
|
||||
| Marker | Description |
|
||||
| ------------------ | --------------------------------------------------------- |
|
||||
| `core_model` | L1&L2 tests (run in each PR) |
|
||||
| `advanced_model` | L3 tests (run on each merge to main) |
|
||||
| `full_model` | L4 tests (run nightly) |
|
||||
| `diffusion` | Diffusion model tests |
|
||||
| `omni` | Omni multimodal model tests |
|
||||
| `tts` | TTS model tests |
|
||||
| `cache` | Cache backend tests |
|
||||
| `parallel` | Parallelism/distributed tests |
|
||||
| `cpu` | Tests that run on CPU |
|
||||
| `gpu` | Tests that run on GPU * |
|
||||
| `cuda` | Tests that run on CUDA * |
|
||||
| `rocm` | Tests that run on AMD/ROCm * |
|
||||
| `xpu` | Tests that run on Intel XPU * |
|
||||
| `npu` | Tests that run on NPU/Ascend * |
|
||||
| `H100` | Tests that require H100 GPU * |
|
||||
| `L4` | Tests that require L4 GPU * |
|
||||
| `MI325` | Tests that require MI325 GPU (AMD/ROCm) * |
|
||||
| `A2` | Tests that require A2 NPU * |
|
||||
| `A3` | Tests that require A3 NPU * |
|
||||
| `distributed_cuda` | Tests that require multi cards on CUDA platform * |
|
||||
| `distributed_rocm` | Tests that require multi cards on ROCm platform * |
|
||||
| `distributed_npu` | Tests that require multi cards on NPU platform * |
|
||||
| `skipif_cuda` | Skip if the num of CUDA cards is less than the required * |
|
||||
| `skipif_rocm` | Skip if the num of ROCm cards is less than the required * |
|
||||
| `skipif_npu` | Skip if the num of NPU cards is less than the required * |
|
||||
| `slow` | Slow tests (may skip in quick CI) |
|
||||
| Marker | Description |
|
||||
| ------------------ | ------------------------------------------------------------------------------------------------------- |
|
||||
| `core_model` | L1&L2 tests (run in each PR) |
|
||||
| `advanced_model` | L3 tests (run on each merge to main) |
|
||||
| `full_model` | L4 tests (run nightly) |
|
||||
| `diffusion` | Diffusion model tests |
|
||||
| `omni` | Omni multimodal model tests |
|
||||
| `tts` | TTS model tests |
|
||||
| `cache` | Cache backend tests |
|
||||
| `parallel` | Parallelism/distributed tests |
|
||||
| `cpu` | Tests that run on CPU |
|
||||
| `gpu` | Tests that run on GPU * |
|
||||
| `cuda` | Tests that run on CUDA * |
|
||||
| `rocm` | Tests that run on AMD/ROCm * |
|
||||
| `xpu` | Tests that run on Intel XPU * |
|
||||
| `npu` | Tests that run on NPU/Ascend * |
|
||||
| `H100` | Tests that require H100 GPU * |
|
||||
| `L4` | Tests that require L4 GPU * |
|
||||
| `MI325` | Tests that require MI325 GPU (AMD/ROCm) * |
|
||||
| `A2` | Tests that require A2 NPU * |
|
||||
| `A3` | Tests that require A3 NPU * |
|
||||
| `distributed_cuda` | Tests that require multi cards on CUDA platform * |
|
||||
| `distributed_rocm` | Tests that require multi cards on ROCm platform * |
|
||||
| `distributed_npu` | Tests that require multi cards on NPU platform * |
|
||||
| `skipif_cuda` | Skip if the num of CUDA cards is less than the required * |
|
||||
| `skipif_rocm` | Skip if the num of ROCm cards is less than the required * |
|
||||
| `skipif_npu` | Skip if the num of NPU cards is less than the required * |
|
||||
| `slow` | Slow tests (may skip in quick CI) |
|
||||
| `benchmark` | Benchmark tests (decorator on runner test functions; perf JSON uses `full_model` + type marker instead) |
|
||||
| `local_model` | Tests requiring local / non-HF-hub model weights |
|
||||
| `local_model` | Tests requiring local / non-HF-hub model weights |
|
||||
|
||||
\* Means those markers are auto-added by `@hardware_test` (parametrization decorator) or `hardware_marks` (only returning the list of marks for flexibility).
|
||||
|
||||
@@ -81,13 +82,13 @@ This decorator is intended to make hardware-aware, cross-platform test authoring
|
||||
|
||||
1. **Applies platform and resource markers**
|
||||
Adds the appropriate pytest markers for each specified hardware platform (e.g., `cuda`, `rocm`, `xpu`, `npu`) and resource type (e.g., `L4`, `H100`, `MI325`, `B60`, `A2`, `A3`).
|
||||
```
|
||||
```python
|
||||
@pytest.mark.cuda
|
||||
@pytest.mark.L4
|
||||
```
|
||||
2. **Handles multi-card (distributed) scenarios**
|
||||
For tests requiring multiple cards, it automatically adds distributed markers such as `distributed_cuda`, `distributed_rocm`, or `distributed_npu`.
|
||||
```
|
||||
```python
|
||||
@pytest.mark.distributed_cuda(num_cards=num_cards)
|
||||
```
|
||||
3. **Supports flexible card requirements**
|
||||
@@ -97,11 +98,11 @@ This decorator is intended to make hardware-aware, cross-platform test authoring
|
||||
On CUDA, adds a skip marker (`skipif_cuda`) if the system does not have the required number of devices.
|
||||
Support for `skipif_rocm` and `skipif_npu` will be implemented later.
|
||||
|
||||
|
||||
5. **Works with pytest filtering**
|
||||
Allows tests to be filtered and selected at runtime using standard pytest marker expressions (e.g., `-m "distributed_cuda and L4"`).
|
||||
|
||||
##### Example usage for decorator
|
||||
|
||||
- Single call for multiple platforms:
|
||||
```python
|
||||
@hardware_test(
|
||||
@@ -220,6 +221,7 @@ If you want to add support for a new platform (e.g., "tpu" for a new accelerator
|
||||
```
|
||||
|
||||
**Summary**:
|
||||
|
||||
- Add pytest markers for your new platform/resources
|
||||
- Implement a marker function (`xxx_marks`)
|
||||
- Plug into `hardware_marks`
|
||||
@@ -237,47 +239,46 @@ L1 and L2 level testing form the foundation of the quality assurance system. L1
|
||||
|
||||
L2 level testing builds upon L1 by introducing GPU resources and verifying that the end-to-end (E2E) process of the model in basic deployment scenarios is smooth. For example, it uses dummy models to confirm that core interfaces like the inference pipeline, output format, and streaming response work properly. The common goal of these two levels is to provide developers with rapid feedback, discovering and fixing issues early in the development cycle.
|
||||
|
||||
|
||||
|
||||
#### 1.2 Testing Content and Scope
|
||||
|
||||
- ***L1 (Unit & Logic Testing)***:
|
||||
- - ***Scope***: Tests internal functions and methods of core components such as `entrypoints`, `models`.
|
||||
- ***Focus***: Branch coverage, exception handling, algorithm logic correctness. Does not involve external dependencies or the complete service stack.
|
||||
- ***Time Cost***: Execution time is controlled within ***15 minutes*** to ensure fast feedback.
|
||||
- ***L2 (Basic End-to-End Testing)***:
|
||||
- - ***Scope***: Covers two basic deployment scenarios: `online` (serving) and `offline` (inference).
|
||||
- ***Focus***: Uses `dummy` weights (via deploy YAML patching at `core_model`) or lightweight real models to verify that the entire chain from request input to result output works normally, including output data structure, streaming (stream) support, and **cheap payload checks** at `--run-level core_model` (see below). Also includes some unit tests that require launching independent service instances.
|
||||
- ***L2 response validation (`core_model`)***: Implemented in `tests/helpers/assertions.py` and invoked by `OpenAIClientHandler` based on `--run-level`. At L2 we require **request success** plus minimal output sanity—not full accuracy:
|
||||
- **Speech / TTS** (`assert_audio_speech_response`): decoded audio must be present and non-empty (or exceed `min_audio_bytes` when set in `request_config`); `response_format` must match the returned content-type (e.g. `wav`, `pcm`). Whisper transcript similarity, PCM HNR, and preset-voice gender checks run only at L3+.
|
||||
- **Diffusion** (`assert_diffusion_response`): at least one non-empty image, video, or audio artifact. Resolution/frame-count parity and other deep checks run only at L3+.
|
||||
- **Omni multimodal** (`assert_omni_response`): L2 asserts successful completion; keyword, transcript, and cross-modal similarity checks run only at L3+.
|
||||
- ***Characteristic***: Requires ***GPU*** resources to perform model computations.
|
||||
- ***L1 (Unit & Logic Testing)***:
|
||||
- ***Scope***: Tests internal functions and methods of core components such as `entrypoints`, `models`.
|
||||
- ***Focus***: Branch coverage, exception handling, algorithm logic correctness. Does not involve external dependencies or the complete service stack.
|
||||
- ***Time Cost***: Execution time is controlled within ***15 minutes*** to ensure fast feedback.
|
||||
- ***L2 (Basic End-to-End Testing)***:
|
||||
- ***Scope***: Covers two basic deployment scenarios: `online` (serving) and `offline` (inference).
|
||||
- ***Focus***: Uses `dummy` weights (via deploy YAML patching at `core_model`) or lightweight real models to verify that the entire chain from request input to result output works normally, including output data structure, streaming (stream) support, and **cheap payload checks** at `--run-level core_model` (see below). Also includes some unit tests that require launching independent service instances.
|
||||
- ***L2 response validation (`core_model`)***: Implemented in `tests/helpers/assertions.py` and invoked by `OpenAIClientHandler` based on `--run-level`. At L2 we require **request success** plus minimal output sanity—not full accuracy:
|
||||
- **Speech / TTS** (`assert_audio_speech_response`): decoded audio must be present and non-empty (or exceed `min_audio_bytes` when set in `request_config`); `response_format` must match the returned content-type (e.g. `wav`, `pcm`). Whisper transcript similarity, PCM HNR, and preset-voice gender checks run only at L3+.
|
||||
- **Diffusion** (`assert_diffusion_response`): at least one non-empty image, video, or audio artifact. Resolution/frame-count parity and other deep checks run only at L3+.
|
||||
- **Omni multimodal** (`assert_omni_response`): L2 asserts successful completion; keyword, transcript, and cross-modal similarity checks run only at L3+.
|
||||
- ***Characteristic***: Requires ***GPU*** resources to perform model computations.
|
||||
|
||||
#### 1.3 Test Directory and Execution Files
|
||||
|
||||
A clear directory structure is key to managing test cases efficiently. See the **Test Dir** column in [Test System Overview](./test_system_overview.md) for the canonical placement rules.
|
||||
|
||||
- ***L1 (component / unit)***: `tests/{component_name}/…` mirroring `vllm_omni/{component_name}/` (e.g. `tests/diffusion/`, `tests/entrypoints/`). Do **not** invent new top-level `tests/` folders unrelated to a component.
|
||||
- ***L2 (basic model E2E)***:
|
||||
- Online serving: `tests/e2e/online_serving/test_{model_name}.py`
|
||||
- Offline inference: `tests/e2e/offline_inference/test_{model_name}.py`
|
||||
- ***Feature-level integration*** (cross-cutting, not a single-model smoke): `tests/e2e/features/<feature>/` (e.g. `fullduplex/`, `custom_pipeline/`, `rlhf_test/`).
|
||||
- ***L1 (component / unit)***: `tests/{component_name}/…` mirroring `vllm_omni/{component_name}/` (e.g. `tests/diffusion/`, `tests/entrypoints/`). Do **not** invent new top-level `tests/` folders unrelated to a component.
|
||||
- ***L2 (basic model E2E)***:
|
||||
- Online serving: `tests/e2e/online_serving/test_{model_name}.py`
|
||||
- Offline inference: `tests/e2e/offline_inference/test_{model_name}.py`
|
||||
- ***Feature-level integration*** (cross-cutting, not a single-model smoke): `tests/e2e/features/<feature>/` (e.g. `fullduplex/`, `custom_pipeline/`, `rlhf_test/`).
|
||||
|
||||
#### 1.4 Execution Method and Example
|
||||
|
||||
- ***Trigger Timing***: **`PR with ready label`**. That is, when a developer adds a "ready for review" or similar label to a PR on platforms like GitHub, L1 and L2 tests are automatically triggered.
|
||||
- ***Diff-aware step skipping***: On L2, **E2E Test** jobs may be omitted at pipeline upload when the PR diff does not touch their [`source_file_dependencies`](ci_settings.md#step-filtering) prefixes; non-E2E groups still always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
|
||||
- ***Execution Environment***: L1 uses ***CPU*** environment; L2 requires ***GPU*** environment.
|
||||
- ***Script Example***:
|
||||
- ***Trigger Timing***: **`PR with ready label`**. That is, when a developer adds a "ready for review" or similar label to a PR on platforms like GitHub, L1 and L2 tests are automatically triggered.
|
||||
- ***Diff-aware step skipping***: On L2, **E2E Test** jobs may be omitted at pipeline upload when the PR diff does not touch their [`source_file_dependencies`](ci_settings.md#step-filtering) prefixes; non-E2E groups still always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
|
||||
- ***Execution Environment***: L1 uses ***CPU*** environment; L2 requires ***GPU*** environment.
|
||||
- ***Script Example***:
|
||||
|
||||
<details>
|
||||
<summary> L1 Test Examples</summary>
|
||||
|
||||
Examples from `tests/model_executor/models/qwen2_5_omni/test_audio_length.py`
|
||||
|
||||
```python
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -331,6 +332,7 @@ def test_cap_and_align_mel_length_no_mismatch(repeats, code_len, max_mel_frames)
|
||||
if max_mel_frames is not None and int(max_mel_frames) > 0 and int(max_mel_frames) >= repeats:
|
||||
assert target_mel_len <= int(max_mel_frames)
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
@@ -338,7 +340,7 @@ def test_cap_and_align_mel_length_no_mismatch(repeats, code_len, max_mel_frames)
|
||||
You can refer to Test Examples in the L3 section to see example test cases that incorporate both L2 and L3 testing logic.
|
||||
</details>
|
||||
|
||||
- - ***Run Command***:
|
||||
- ***Run Command***:
|
||||
|
||||
`pytest -s -v /tests/e2e/online_serving/test_{model_name}.py`
|
||||
`pytest -s -v -m 'core_model and cpu' --run-level=core_model`
|
||||
@@ -351,36 +353,34 @@ L3 level testing executes after code is merged into the main branch. Its core pu
|
||||
|
||||
. It acts as the "quality gatekeeper" for the main branch, ensuring that no merge breaks the core capabilities of the model service. Testing needs to provide clear conclusions within a relatively short time (<30min), balancing test depth with feedback speed.
|
||||
|
||||
|
||||
|
||||
#### 2.2 Testing Content and Scope
|
||||
|
||||
- ***Deployment Scenarios***: Covers richer `online` and `offline` deployment configurations, which may include different hardware configurations, batch sizes, concurrency levels, etc.
|
||||
- ***Core Verification***:
|
||||
- 1. ***Inference Functionality***: Ensures real models can perform forward computation normally and return results.
|
||||
2. ***Accuracy Compliance***: Verifies that the model's evaluation metrics (e.g., accuracy) meet the expected baseline, preventing code changes from introducing accuracy issues.
|
||||
3. ***Important Performance***: Verifies whether performance (e.g., P99 latency, throughput) in core scenarios meets preset thresholds.
|
||||
- ***L3 response validation (`advanced_model`)***: At `--run-level advanced_model`, `tests/helpers/assertions.py` adds semantic checks on top of L2 payload gates (see L1 & L2 above):
|
||||
- **Speech / TTS** (`assert_audio_speech_response`): Whisper transcript of returned audio vs `request_config["input"]` (cosine similarity > 0.9 when input text is set); optional `min_audio_bytes` floor; PCM harmonic-to-noise ratio when `response_format` is `pcm`; **preset voice gender** via `_assert_preset_voice_gender_from_audio` when `voice` matches a known preset in `_PRESET_VOICE_GENDER_MAP` (pitch/F0-based classifier; skipped for unknown voices or `pcm` output).
|
||||
- **Omni multimodal** (`assert_omni_response`): non-empty text/audio outputs; `key_words` in transcript or text; text–audio similarity / containment; preset **`speaker`** gender check (same helper as TTS).
|
||||
- **Diffusion** (`assert_*_diffusion_response`): image/video dimension and frame-count parity with request parameters where configured.
|
||||
- **Accuracy suites**: pixel/video similarity and metric baselines under `/tests/e2e/accuracy/` (separate from inline helper assertions).
|
||||
- ***Deployment Scenarios***: Covers richer `online` and `offline` deployment configurations, which may include different hardware configurations, batch sizes, concurrency levels, etc.
|
||||
- ***Core Verification***:
|
||||
1. ***Inference Functionality***: Ensures real models can perform forward computation normally and return results.
|
||||
2. ***Accuracy Compliance***: Verifies that the model's evaluation metrics (e.g., accuracy) meet the expected baseline, preventing code changes from introducing accuracy issues.
|
||||
3. ***Important Performance***: Verifies whether performance (e.g., P99 latency, throughput) in core scenarios meets preset thresholds.
|
||||
- ***L3 response validation (`advanced_model`)***: At `--run-level advanced_model`, `tests/helpers/assertions.py` adds semantic checks on top of L2 payload gates (see L1 & L2 above):
|
||||
- **Speech / TTS** (`assert_audio_speech_response`): Whisper transcript of returned audio vs `request_config["input"]` (cosine similarity > 0.9 when input text is set); optional `min_audio_bytes` floor; PCM harmonic-to-noise ratio when `response_format` is `pcm`; **preset voice gender** via `_assert_preset_voice_gender_from_audio` when `voice` matches a known preset in `_PRESET_VOICE_GENDER_MAP` (pitch/F0-based classifier; skipped for unknown voices or `pcm` output).
|
||||
- **Omni multimodal** (`assert_omni_response`): non-empty text/audio outputs; `key_words` in transcript or text; text–audio similarity / containment; preset **`speaker`** gender check (same helper as TTS).
|
||||
- **Diffusion** (`assert_*_diffusion_response`): image/video dimension and frame-count parity with request parameters where configured.
|
||||
- **Accuracy suites**: pixel/video similarity and metric baselines under `/tests/e2e/accuracy/` (separate from inline helper assertions).
|
||||
|
||||
#### 2.3 Test Directory and Execution Files
|
||||
|
||||
- ***Functional / model E2E***:
|
||||
- Online serving: `tests/e2e/online_serving/test_{model_name}_expansion.py`
|
||||
- Offline inference: `tests/e2e/offline_inference/test_{model_name}_expansion.py`
|
||||
- Accuracy / similarity: `tests/e2e/accuracy/`
|
||||
- (`_expansion.py` holds broader scenario coverage than the L2 smoke file.)
|
||||
- ***Feature-level integration***: `tests/e2e/features/<feature>/` (not under a new top-level `tests/` directory).
|
||||
- ***Functional / model E2E***:
|
||||
- Online serving: `tests/e2e/online_serving/test_{model_name}_expansion.py`
|
||||
- Offline inference: `tests/e2e/offline_inference/test_{model_name}_expansion.py`
|
||||
- Accuracy / similarity: `tests/e2e/accuracy/`
|
||||
- (`_expansion.py` holds broader scenario coverage than the L2 smoke file.)
|
||||
- ***Feature-level integration***: `tests/e2e/features/<feature>/` (not under a new top-level `tests/` directory).
|
||||
|
||||
#### 2.4 Execution Method and Example
|
||||
|
||||
- ***Trigger Timing***: **`PR Merged`**. Automatically triggered after code review is approved and merged into the main branch (typically via `merge-test` label on the PR before merge).
|
||||
- ***Diff-aware step skipping***: Same [`source_file_dependencies`](ci_settings.md#step-filtering) mechanism as L2—**all E2E Test** leaf steps are diff-gated; other L3 groups always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
|
||||
- ***Execution Environment***: ***GPU*** servers.
|
||||
- ***Script Example***:
|
||||
- ***Trigger Timing***: **`PR Merged`**. Automatically triggered after code review is approved and merged into the main branch (typically via `merge-test` label on the PR before merge).
|
||||
- ***Diff-aware step skipping***: Same [`source_file_dependencies`](ci_settings.md#step-filtering) mechanism as L2—**all E2E Test** leaf steps are diff-gated; other L3 groups always upload. See [CI Settings — Diff-aware CI](ci_settings.md#diff-aware-ci).
|
||||
- ***Execution Environment***: ***GPU*** servers.
|
||||
- ***Script Example***:
|
||||
|
||||
???+ example "Test Examples"
|
||||
|
||||
@@ -496,9 +496,9 @@ L3 level testing executes after code is merged into the main branch. Its core pu
|
||||
|
||||
**Audio output debugging**: Deep validation may transcribe returned audio via `convert_audio_bytes_to_text` (Whisper). If an audio keyword or text–audio similarity assertion fails, set `VLLM_OMNI_KEEP_REQUEST_MEDIA=1` before running pytest to keep the intermediate WAV files for inspection (see [Test helper environment variables](test_system_overview.md#test-helper-environment-variables)).
|
||||
|
||||
- ***Run Command (L3 merge)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}.py -m advanced_model --run-level=advanced_model`
|
||||
- ***Run Command (L3 merge)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}.py -m advanced_model --run-level=advanced_model`
|
||||
|
||||
- ***Run Command (L4 nightly expansion)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}_expansion.py -m full_model --run-level=full_model`
|
||||
- ***Run Command (L4 nightly expansion)***: `pytest -s -v /tests/e2e/online_serving/test_{model_name}_expansion.py -m full_model --run-level=full_model`
|
||||
|
||||
### L4 Level Testing - Full Functionality, Performance, and Documentation Testing
|
||||
|
||||
@@ -506,27 +506,25 @@ L3 level testing executes after code is merged into the main branch. Its core pu
|
||||
|
||||
L4 level testing is a comprehensive quality audit before a version release. It expands upon L3, executing ***full*** functional scenarios, conducting systematic ***performance stress tests***, and simultaneously verifying the correctness of accompanying ***example documentation***. Its purpose is to perform deep validation of the system during off-peak nighttime hours, providing quality trend reports for daytime development and data support for release decisions.
|
||||
|
||||
|
||||
|
||||
#### 3.2 Testing Content and Scope
|
||||
|
||||
- ***Full Functionality Testing***: Executes all test cases defined in `test_{model_name}_expansion.py`, covering all implemented features, positive flows, boundary conditions, and exception handling.
|
||||
- ***Performance Testing***: Uses `tests/dfx/perf/tests/test_qwen3_omni_*.json` (Omni), `test_tts.json` / `test_voxcpm2.json` / `test_higgs_audio_v3.json` (TTS), and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (passed to `run_benchmark.py` or `run_diffusion_benchmark.py` via `--test-config-file` in nightly **Perf Test** steps) to drive throughput, latency, and memory benchmarks. Each JSON **case** may declare an optional top-level **`mark`** array: exactly one ``hardware_marks`` object plus pytest marker name strings (`full_model`, `omni` / `tts` / `diffusion`, …). Runners attach those marks to each parametrized `(server, benchmark index)` pair so **local** bulk runs can filter with `-m` (for example `-m "full_model and H100 and diffusion"`). Nightly CI perf jobs select workloads by **`--test-config-file`**, not `-m`. Details are in the Performance Tests example below.
|
||||
- ***Documentation Testing***: Verifies whether the example code provided to users is runnable and its results match the description.
|
||||
- ***Full Functionality Testing***: Executes all test cases defined in `test_{model_name}_expansion.py`, covering all implemented features, positive flows, boundary conditions, and exception handling.
|
||||
- ***Performance Testing***: Uses `tests/dfx/perf/tests/test_qwen3_omni_*.json` (Omni), `test_tts.json` / `test_voxcpm2.json` / `test_higgs_audio_v3.json` (TTS), and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (passed to `run_benchmark.py` or `run_diffusion_benchmark.py` via `--test-config-file` in nightly **Perf Test** steps) to drive throughput, latency, and memory benchmarks. Each JSON **case** may declare an optional top-level **`mark`** array: exactly one ``hardware_marks`` object plus pytest marker name strings (`full_model`, `omni` / `tts` / `diffusion`, …). Runners attach those marks to each parametrized `(server, benchmark index)` pair so **local** bulk runs can filter with `-m` (for example `-m "full_model and H100 and diffusion"`). Nightly CI perf jobs select workloads by **`--test-config-file`**, not `-m`. Details are in the Performance Tests example below.
|
||||
- ***Documentation Testing***: Verifies whether the example code provided to users is runnable and its results match the description.
|
||||
|
||||
#### 3.3 Test Directory and Execution Files
|
||||
|
||||
- ***Functional Testing***: Same directories as L3.
|
||||
- ***Performance Test Configuration***: `tests/dfx/perf/tests/test_qwen3_omni_*.json`, `test_tts.json`, `test_voxcpm2.json`, `test_higgs_audio_v3.json`, and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (e.g. `test_qwen_image_vllm_omni.json`, `test_cosmos3_vllm_omni.json`). Optional per-case `mark` for local `-m` filtering is documented in Section 3.4 Performance Tests.
|
||||
- ***Documentation Example Tests***:
|
||||
- `tests/examples/online_serving/test_{model_name}.py`
|
||||
- `tests/examples/offline_inference/test_{model_name}.py`
|
||||
- ***Functional Testing***: Same directories as L3.
|
||||
- ***Performance Test Configuration***: `tests/dfx/perf/tests/test_qwen3_omni_*.json`, `test_tts.json`, `test_voxcpm2.json`, `test_higgs_audio_v3.json`, and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (e.g. `test_qwen_image_vllm_omni.json`, `test_cosmos3_vllm_omni.json`). Optional per-case `mark` for local `-m` filtering is documented in Section 3.4 Performance Tests.
|
||||
- ***Documentation Example Tests***:
|
||||
- `tests/examples/online_serving/test_{model_name}.py`
|
||||
- `tests/examples/offline_inference/test_{model_name}.py`
|
||||
|
||||
#### 3.4 Execution Method and Example
|
||||
|
||||
- ***Trigger Timing***: **`Nightly`**, automatically executed every night.
|
||||
- ***Execution Environment***: ***GPU*** server clusters to meet the resource demands of performance testing.
|
||||
- ***Script Example***:
|
||||
- ***Trigger Timing***: **`Nightly`**, automatically executed every night.
|
||||
- ***Execution Environment***: ***GPU*** server clusters to meet the resource demands of performance testing.
|
||||
- ***Script Example***:
|
||||
|
||||
??? example "Test Examples: Documentation Example Tests"
|
||||
|
||||
@@ -540,7 +538,7 @@ L4 level testing is a comprehensive quality audit before a version release. It e
|
||||
|
||||
--8<-- "docs/contributing/ci/test_examples/l4_functionality_tests.inc.md"
|
||||
|
||||
- ***Run Command***: (Specific commands would depend on the performance testing tool and configuration defined in `nightly.json`).
|
||||
- ***Run Command***: (Specific commands would depend on the performance testing tool and configuration defined in `nightly.json`).
|
||||
|
||||
### L5 Level Testing - Stability and Reliability Testing
|
||||
|
||||
@@ -548,33 +546,36 @@ L4 level testing is a comprehensive quality audit before a version release. It e
|
||||
|
||||
L5 level testing focuses on the performance of model services under ***long-running*** and ***abnormal fault*** scenarios. It aims to uncover deep-seated issues that only manifest under sustained pressure or extreme conditions, such as memory leaks, resource contention, gradual performance degradation, and lack of fault tolerance mechanisms. This is the final, yet crucial, line of defense for ensuring service high availability and production environment robustness.
|
||||
|
||||
|
||||
|
||||
#### 4.2 Testing Content and Scope
|
||||
|
||||
- ***Long-term Stability (Stability) Testing***: Uses JSON under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` and `test_wan22.json`) to run the service under moderate load for an extended period (e.g., over 12 hours), monitoring whether metrics like memory/VRAM usage, response time, and throughput degrade over time, and whether the service process remains stable.
|
||||
- ***Reliability Testing***: Uses pytest suites under `tests/dfx/reliability/` to inject controlled faults against a **live** `vllm_omni serve` instance (same **`omni_server` / `omni_server_function`** fixture style as E2E). Current suites emphasize **GPU memory pressure** (CUDA sidecar “memory hog”), **worker / runtime process kill** (`SIGKILL` on `VLLM::Worker` for Qwen3-Omni, `multiprocessing.spawn` for Wan2.2 video workers, or `vLLM-Omni::` for HunyuanImage DiT workers), **large multimodal chat**, **`/v1/videos`**, **`/v1/images/generations`**, or **`/v1/audio/speech`** jobs under OOM, **`/health` → 503** and **fast-fail / non-hanging concurrent** requests after kill, and **OpenAI-style 5xx error contracts** (e.g. text vs text+audio under OOM). **Post-fault recovery** checks exist where enabled (some cases may be `skip` while issues are tracked). See the Reliability `<details>` block in Section 4.4 for file-level responsibilities and CI markers (`slow`, `hardware_test`, POSIX-only kill).
|
||||
- ***Long-term Stability (Stability) Testing***: Uses JSON under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` and `test_wan22.json`) to run the service under moderate load for an extended period (e.g., over 12 hours), monitoring whether metrics like memory/VRAM usage, response time, and throughput degrade over time, and whether the service process remains stable.
|
||||
- ***Reliability Testing***: Uses pytest suites under `tests/dfx/reliability/` to inject controlled faults against a **live** `vllm_omni serve` instance (same **`omni_server` / `omni_server_function`** fixture style as E2E). Current suites emphasize **GPU memory pressure** (CUDA sidecar “memory hog”), **worker / runtime process kill** (`SIGKILL` on `VLLM::Worker` for Qwen3-Omni, `multiprocessing.spawn` for Wan2.2 video workers, or `vLLM-Omni::` for HunyuanImage DiT workers), **large multimodal chat**, **`/v1/videos`**, **`/v1/images/generations`**, or **`/v1/audio/speech`** jobs under OOM, **`/health` → 503** and **fast-fail / non-hanging concurrent** requests after kill, and **OpenAI-style 5xx error contracts** (e.g. text vs text+audio under OOM). **Post-fault recovery** checks exist where enabled (some cases may be `skip` while issues are tracked). See Section 4.3 for file-level responsibilities and Section 4.4 Test Examples (Reliability) for CI markers (`slow`, `hardware_test`, POSIX-only kill).
|
||||
|
||||
#### 4.3 Test Directory and Execution Files
|
||||
|
||||
- ***Stability Test Configuration***: `tests/dfx/stability/tests/test_qwen3_omni.json`, `tests/dfx/stability/tests/test_wan22.json` (one JSON per model / runner family)
|
||||
- ***Reliability Test Suite*** (`tests/dfx/reliability/`):
|
||||
- `test_reliability_qwen3_omni.py` — Qwen3-Omni chat / multimodal reliability (GPU OOM, process kill, recovery, error contract under `--async-chunk` vs default).
|
||||
- `test_reliability_wan22.py` — Wan2.2 I2V video API reliability (`/v1/videos` under OOM and process kill, recovery).
|
||||
- `test_reliability_hunyuan_image.py` — HunyuanImage-3.0-Instruct DiT-only reliability (`/v1/images/generations` under OOM and process kill; deploy `hunyuan_image3_dit.yaml`, H100 × 4).
|
||||
- `helpers.py` — Shared primitives used by current suites: raw HTTP probes for `/v1/chat/completions` and `/health`, OpenAI-style error parsing, GPU OOM sidecar (`inject_gpu_oom` / `stop_gpu_oom_hogs`), and `pgrep`-based process-kill injector construction (`make_process_kill_fault_injector`).
|
||||
- `conftest.py` — `fault_injector` and `omni_server_after_fault` / `omni_server_after_fault_function` fixtures to run a callable **after** the server is ready.
|
||||
- `README.md` — Short local run commands for this directory.
|
||||
- ***Stability Test Configuration***: `tests/dfx/stability/tests/test_qwen3_omni.json`, `tests/dfx/stability/tests/test_wan22.json` (one JSON per model / runner family)
|
||||
- ***Reliability Test Suite*** (`tests/dfx/reliability/`):
|
||||
- `test_reliability_qwen3_omni.py` — Qwen3-Omni chat / multimodal reliability (GPU OOM, process kill, recovery, error contract under `--async-chunk` vs default).
|
||||
- `test_reliability_wan22.py` — Wan2.2 I2V video API reliability (`/v1/videos` under OOM and process kill, recovery).
|
||||
- `test_reliability_hunyuan_image.py` — HunyuanImage-3.0-Instruct DiT-only reliability (`/v1/images/generations` under OOM and process kill; deploy `hunyuan_image3_dit.yaml`, H100 × 4).
|
||||
- `helpers.py` — Shared primitives used by current suites: raw HTTP probes for `/v1/chat/completions` and `/health`, OpenAI-style error parsing, GPU OOM sidecar (`inject_gpu_oom` / `stop_gpu_oom_hogs`), and `pgrep`-based process-kill injector construction (`make_process_kill_fault_injector`).
|
||||
- `conftest.py` — `fault_injector` and `omni_server_after_fault` / `omni_server_after_fault_function` fixtures to run a callable **after** the server is ready.
|
||||
- `README.md` — Short local run commands for this directory.
|
||||
|
||||
#### 4.4 Execution Method and Example
|
||||
|
||||
- ***Trigger Timing***: **`Weekly`** (weekly) or **`Days before Release`** (several days before a major release). Due to long execution times, the frequency is lower.
|
||||
- ***Execution Environment***: ***GPU*** servers, requiring a stable and exclusive testing environment.
|
||||
- ***Script Example***:
|
||||
- ***Trigger Timing***: **`Weekly`** (weekly) or **`Days before Release`** (several days before a major release). Due to long execution times, the frequency is lower.
|
||||
- ***Run Command***:
|
||||
- ***Stability***: `pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/test_stability_wan22.py` (or add `test_stability_<model>.py` alongside a matching JSON config)
|
||||
- ***Reliability***: `pytest -s -v tests/dfx/reliability/test_reliability_<model>.py -m slow` (current suites: `qwen3_omni`, `wan22`, `hunyuan_image`). Weekly CI (`.buildkite/cuda/test-weekly.yml`) runs one step per suite (`WEEKLY=1` or PR label `weekly-test`)
|
||||
- ***Script Example***:
|
||||
|
||||
<details>
|
||||
<summary> Test Examples</summary>
|
||||
|
||||
When you want to add L5-level stability test cases, add or extend the appropriate JSON file under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` for Omni bench traffic, or `test_wan22.json` for diffusion `/v1/videos` workloads). The following illustrates the Qwen3-Omni shape:
|
||||
##### Stability
|
||||
|
||||
When you want to add L5-level stability test cases, add or extend the appropriate JSON file under `tests/dfx/stability/tests/` (for example `test_qwen3_omni.json` for Omni bench traffic, or `test_wan22.json` for diffusion `/v1/videos` workloads). Pair the JSON with `tests/dfx/stability/scripts/test_stability_<model>.py`. The following illustrates the Qwen3-Omni shape:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -600,7 +601,7 @@ When you want to add L5-level stability test cases, add or extend the appropriat
|
||||
}
|
||||
```
|
||||
|
||||
##### Parameter Explanation
|
||||
###### Parameter Explanation
|
||||
|
||||
***Overview***
|
||||
|
||||
@@ -610,91 +611,47 @@ When you want to add L5-level stability test cases, add or extend the appropriat
|
||||
| server_params | Yes | Server-side configuration parameters (model, deploy configuration, etc.) |
|
||||
| benchmark_params | Yes | Stability benchmark running parameters (supports multiple configurations) |
|
||||
|
||||
##### server_params Configuration
|
||||
###### server_params Configuration
|
||||
|
||||
###### Basic Parameters
|
||||
Basic parameters:
|
||||
|
||||
| Parameter | Required | Example | Description |
|
||||
| ----------------- | -------- | ---------------------------------- | ----------------------------------- |
|
||||
| model | Yes | "Qwen/Qwen3-Omni-30B-A3B-Instruct" | Model name or path |
|
||||
| stage_config_name | Yes | "qwen3_omni_moe.yaml" | Deploy configuration file name |
|
||||
|
||||
###### Dynamic Configuration (update/delete)
|
||||
|
||||
Supports incremental modifications based on the basic configuration:
|
||||
Dynamic configuration (`update` / `delete`) supports incremental modifications based on the basic configuration:
|
||||
|
||||
| Operation | Description |
|
||||
| --------- | ------------------------------------ |
|
||||
| update | Update or add configuration items |
|
||||
| delete | Delete specified configuration items |
|
||||
|
||||
***Example***:
|
||||
You can refer to Test Examples in L4 §3.4
|
||||
***Example***: You can refer to Test Examples in L4 §3.4.
|
||||
|
||||
##### benchmark_params Configuration
|
||||
###### benchmark_params Configuration
|
||||
|
||||
For stability testing, the key parameters are:
|
||||
|
||||
- **duration_sec**: Total duration (in seconds) during which benchmark traffic is sent. The stability benchmark will keep sending batches until this duration is reached.
|
||||
- **request_rate** / **max_concurrency**: Exactly one of them must be specified. They control how the traffic is generated for each batch:
|
||||
- `request_rate`: Number of requests per second. The benchmark will send `num_prompts_per_batch` requests at the given rate.
|
||||
- `max_concurrency`: Maximum number of concurrent requests. When this is used, `request_rate` is set to `inf` internally.
|
||||
- **num_prompts_per_batch**: Number of prompts sent in each batch. Multiple batches will be executed sequentially within `duration_sec`.
|
||||
- **duration_sec**: Total duration (in seconds) during which benchmark traffic is sent. The stability benchmark will keep sending batches until this duration is reached.
|
||||
- **request_rate** / **max_concurrency**: Exactly one of them must be specified. They control how the traffic is generated for each batch:
|
||||
- `request_rate`: Number of requests per second. The benchmark will send `num_prompts_per_batch` requests at the given rate.
|
||||
- `max_concurrency`: Maximum number of concurrent requests. When this is used, `request_rate` is set to `inf` internally.
|
||||
- **num_prompts_per_batch**: Number of prompts sent in each batch. Multiple batches will be executed sequentially within `duration_sec`.
|
||||
|
||||
All other optional parameters follow the same rules as in L4 §3.4.
|
||||
|
||||
</details>
|
||||
##### Reliability
|
||||
|
||||
<details>
|
||||
<summary> Reliability test suite (<code>tests/dfx/reliability</code>)</summary>
|
||||
Add pytest modules under `tests/dfx/reliability/` (for example `test_reliability_qwen3_omni.py`, `test_reliability_wan22.py`, `test_reliability_hunyuan_image.py`). Reuse `helpers.py` and `conftest.py` rather than duplicating OOM / process-kill / raw HTTP plumbing.
|
||||
|
||||
##### Purpose and relationship to stability
|
||||
|
||||
Reliability tests are **short fault-injection** integration runs (L5 **(b)** in `tests/dfx/reliability/README.md`). They complement **stability** JSON-driven long runs: instead of hours of steady traffic, they **perturb** the server (GPU OOM sidecar, fatal signals on selected processes) and check **failure mode** and **latency bounds** (e.g. chat or `/v1/videos` must not hang under concurrent fault-time load).
|
||||
|
||||
##### Directory layout
|
||||
|
||||
| Path | Responsibility |
|
||||
| ---- | -------------- |
|
||||
| `helpers.py` | Shared helpers used by current reliability suites: raw `POST`/`GET` probes (`/v1/chat/completions`, `/health`), OpenAI error parsing (`extract_openai_error_contract_from_bytes`), GPU OOM sidecar lifecycle (`inject_gpu_oom`, `stop_gpu_oom_hogs`), and process-kill injector builder (`make_process_kill_fault_injector`). |
|
||||
| `conftest.py` | Pytest fixtures: indirect `fault_injector`, `omni_server_after_fault` / `omni_server_after_fault_function` (run injector after server is ready, then yield server). |
|
||||
| `test_reliability_qwen3_omni.py` | Qwen3-Omni: OOM vs **text vs text+audio** error contract, large multimodal chat under OOM, concurrent pressure, **SIGKILL** on `VLLM::Worker`, `/health` → 503 + fast-fail + concurrent chat; optional OOM recovery scenario (may be skipped while tracked in issues). |
|
||||
| `test_reliability_wan22.py` | Wan2.2 I2V: large `/v1/videos` under OOM, **SIGKILL** on `multiprocessing.spawn` chain, health / fast-fail / concurrent video requests; optional recovery test (may be skipped). |
|
||||
| `test_reliability_hunyuan_image.py` | HunyuanImage DiT-only: large `/v1/images/generations` under OOM, **SIGKILL** on `vLLM-Omni::` workers and serve/tree targets, health / fast-fail / concurrent image requests; some OOM/recovery cases may be skipped while tracked in issues. |
|
||||
| `README.md` | Minimal run / collect examples. |
|
||||
|
||||
##### Parametrization and markers
|
||||
###### Parametrization and markers
|
||||
|
||||
- Each test module defines a **`RELIABILITY_SCENARIOS`** list (`test_name`, `server_params`: model, `stage_config_name` or diffusion `server_args`, etc.). **`create_reliability_omni_server_params()`** in `tests/dfx/conftest.py` resolves stage paths (including XPU substitutions where applicable) and builds **`OmniServerParams`** lists consumed by **`@pytest.mark.parametrize(..., indirect=True)`** on `omni_server` or `omni_server_function`.
|
||||
- Cases are tagged **`@pytest.mark.slow`** for weekly / selective CI. GPU-heavy suites use **`@hardware_test(res={"cuda": "H100"}, num_cards=...)`** (Qwen3-Omni **2**× H100; Wan2.2 **1**× H100; HunyuanImage DiT **4**× H100).
|
||||
- **Process-kill** tests use **`@pytest.mark.skipif(os.name == "nt", ...)`** because injection uses POSIX **`pgrep` / `kill`**.
|
||||
|
||||
##### CI trigger
|
||||
|
||||
Weekly Buildkite (`.buildkite/cuda/test-weekly.yml`) runs one step per model suite (trigger: `WEEKLY=1` or PR label `weekly-test`), for example:
|
||||
|
||||
| Buildkite step | Test file | CI hardware |
|
||||
| -------------- | --------- | ----------- |
|
||||
| Reliability Test - qwen3-omni | `test_reliability_qwen3_omni.py` | H100 × 2 (`mithril-h100-pool`) |
|
||||
| Reliability Test - wan22 | `test_reliability_wan22.py` | H100 × 2 (`mithril-h100-pool`) |
|
||||
| Reliability Test - hunyuan-image | `test_reliability_hunyuan_image.py` | H100 × 4 (`mithril-h100-pool`) |
|
||||
|
||||
```bash
|
||||
pytest -s -v tests/dfx/reliability/test_reliability_qwen3_omni.py -m "slow"
|
||||
pytest -s -v tests/dfx/reliability/test_reliability_wan22.py -m "slow"
|
||||
pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py -m "slow"
|
||||
```
|
||||
|
||||
##### Local commands
|
||||
|
||||
```bash
|
||||
pytest --collect-only tests/dfx/reliability
|
||||
pytest -s -v tests/dfx/reliability/test_reliability_qwen3_omni.py -m slow
|
||||
pytest -s -v tests/dfx/reliability/test_reliability_wan22.py -m slow
|
||||
pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py -m slow
|
||||
```
|
||||
|
||||
##### Adding a new model suite
|
||||
###### Adding a new model suite
|
||||
|
||||
1. Add `test_reliability_<model>.py` under `tests/dfx/reliability/`.
|
||||
2. Define **`RELIABILITY_SCENARIOS`** and pass them through **`create_reliability_omni_server_params()`** with the correct deploy or e2e deploy-config directory (same pattern as existing files).
|
||||
@@ -702,6 +659,3 @@ pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py -m slow
|
||||
4. Register **`slow`** (and **`hardware_test`** if needed); extend **`.buildkite/cuda/test-weekly.yml`** when the suite should run in weekly L5.
|
||||
|
||||
</details>
|
||||
|
||||
- - ***Stability***: `pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/test_stability_wan22.py` (or add `test_stability_<model>.py` alongside a matching JSON config)
|
||||
- ***Reliability***: `pytest -s -v tests/dfx/reliability/test_reliability_<model>.py -m slow` (current suites: `qwen3_omni`, `wan22`, `hunyuan_image`; add `test_reliability_<suite>.py` and a matching step in `.buildkite/cuda/test-weekly.yml` for new models)
|
||||
@@ -78,7 +78,7 @@ Profile before optimizing.
|
||||
new TTS model must pass all three levels:
|
||||
|
||||
| Layer | Catches | Tool |
|
||||
|-------|---------|------|
|
||||
| ------- | --------- | ------ |
|
||||
| Offline RTF / duration | Throughput regressions, missing audio, wrong sample rate | `end2end.py`, pytest e2e |
|
||||
| Browser streaming playback | Delta-vs-cumulative bugs, chunk boundary glitches, TTFP regressions | Gradio demo over `/v1/audio/speech?stream=true` |
|
||||
| Concurrent requests | Per-request state leaks, codec window round-robin gaps | `max_num_seqs>1` smoke with 4+ parallel prompts |
|
||||
@@ -94,10 +94,10 @@ the symptom is crosstalk or truncation under load, nothing in single-request tes
|
||||
vLLM-Omni supports TTS models as multi-stage pipelines where each stage runs independently
|
||||
and can be placed on different devices. Qwen3-TTS has two stages:
|
||||
|
||||
| Stage | Name | Input | Output |
|
||||
|-------|------|-------|--------|
|
||||
| 0 | Code Predictor (AR) | Text tokens | Discrete RVQ codec codes |
|
||||
| 1 | Code2Wav (Decoder) | RVQ codec codes | Audio waveform |
|
||||
| Stage | Name | Input | Output |
|
||||
| ----- | ------------------- | --------------- | ------------------------ |
|
||||
| 0 | Code Predictor (AR) | Text tokens | Discrete RVQ codec codes |
|
||||
| 1 | Code2Wav (Decoder) | RVQ codec codes | Audio waveform |
|
||||
|
||||
Each stage is a separate model class configured independently via YAML. The two stages
|
||||
are connected by the `async_chunk` framework, which enables inter-stage streaming for
|
||||
@@ -170,7 +170,7 @@ and Qwen3-Omni).
|
||||
|
||||
When adding a new TTS model, create the following structure:
|
||||
|
||||
```
|
||||
```text
|
||||
vllm_omni/model_executor/models/
|
||||
your_model_name/
|
||||
__init__.py
|
||||
@@ -201,7 +201,7 @@ it would not be picked up — the hub README is the documented surface.
|
||||
**Qwen3-TTS reference files:**
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| ------ | --------- |
|
||||
| `models/qwen3_tts/qwen3_tts.py` | Unified model class |
|
||||
| `models/qwen3_tts/qwen3_tts_code_predictor_vllm.py` | Stage 0 - optimized AR |
|
||||
| `models/qwen3_tts/qwen3_tts_code2wav.py` | Stage 1 - decoder |
|
||||
@@ -528,9 +528,9 @@ Understanding what's available in stage outputs:
|
||||
- `stage_list[source_id].engine_outputs` - list of `EngineCoreOutput` objects
|
||||
- Each `EngineCoreOutput` has `outputs` - list of `RequestOutput` objects
|
||||
- Each `RequestOutput` has:
|
||||
- `token_ids` - generated token IDs
|
||||
- `multimodal_output` - dict with keys matching your model's `OmniOutput.multimodal_outputs`
|
||||
- `prompt_token_ids` - original prompt token IDs
|
||||
- `token_ids` - generated token IDs
|
||||
- `multimodal_output` - dict with keys matching your model's `OmniOutput.multimodal_outputs`
|
||||
- `prompt_token_ids` - original prompt token IDs
|
||||
|
||||
### Batch mode (non-streaming)
|
||||
|
||||
@@ -610,6 +610,7 @@ def ar2decoder_async_chunk(
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- `transfer_manager` is the `OmniChunkTransferAdapter` that owns the chunk lifecycle
|
||||
- Each call appends one AR decode step's output; a chunk is emitted every `chunk_size` steps
|
||||
- The final (possibly partial) chunk is flushed when `is_finished` is true
|
||||
@@ -664,6 +665,7 @@ Create `vllm_omni/entrypoints/openai/tts_adapters/your_model.py`:
|
||||
|
||||
```python
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""YourModel serving adapter."""
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
@@ -802,7 +804,7 @@ directly from the model's own generator.
|
||||
|
||||
### Directory structure
|
||||
|
||||
```
|
||||
```text
|
||||
vllm_omni/model_executor/models/your_model_name/
|
||||
__init__.py
|
||||
modeling_your_model_name.py # unified class: load_weights + forward + streaming
|
||||
@@ -950,10 +952,17 @@ is correct. Stage the modified files and commit again — do not revert the chan
|
||||
Common failures and fixes:
|
||||
|
||||
| Check | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| ------- | ------- | ----- |
|
||||
| `ruff F841` | Local variable assigned but never used | Remove the extraction or forward it to the model call |
|
||||
| `ruff E402` | Module-level import not at top of file | Move import to the top-level import block |
|
||||
| `ruff format` | Line length, spacing, or quote style | Accept the auto-fix, stage, and re-commit |
|
||||
| `check-spdx-header` | Missing header, or copyright still says `vLLM project` | Use `Copyright contributors to the vLLM-Omni project`; restage the rewrite |
|
||||
| `check-forbidden-imports` | Stdlib `re`/`base64`, pickle, Hugging Face Hub API, or direct Triton/TileLang | `import regex as re` and `pybase64`; do not grow the allowlist without review |
|
||||
| `check-torch-cuda-call` | New `torch.cuda.*` call site | Use `current_omni_platform`; do not grow `ALLOWED_FILES` without review |
|
||||
| `check-tts-adapter-migration` | New `_tts_model_type` branch in `serving_speech.py` | Put logic in `tts_adapters/`; lower `MAX_MODEL_TYPE_BRANCHES` when removing branches |
|
||||
| `check-test-ci-coverage` | Test file missing level or hardware mark | Add `core_model`/… plus `cpu`/`cuda`/`hardware_test(` |
|
||||
| `shellcheck` | Native `shellcheck` missing, or script warning | Install via apt/dnf/brew (or `shellcheck.exe` on Windows). See [Linting](../README.md#linting) |
|
||||
| `mypy-3.10` / `markdownlint-cli2` / `check-buildkite` | Types, docs markdown, or Buildkite YAML | See [Linting](../README.md#linting) |
|
||||
|
||||
### DCO sign-off
|
||||
|
||||
@@ -1008,7 +1017,7 @@ Adding a TTS model to vLLM-Omni involves:
|
||||
### Qwen3-TTS Reference Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| ------ | --------- |
|
||||
| `models/qwen3_tts/qwen3_tts.py` | Unified model class |
|
||||
| `models/qwen3_tts/qwen3_tts_code_predictor_vllm.py` | AR stage with vLLM fused ops |
|
||||
| `models/qwen3_tts/qwen3_tts_code2wav.py` | Decoder stage with `chunked_decode_streaming()` |
|
||||
|
||||
Reference in new issue
Block a user