[CI][Perf] Add JSON mark support and paired parametrize for L4 perf, so as to further support performance‑case selection for NPU‑CI. (#5084)
Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
This commit is contained in:
wangyu authored and GitHub committed
1 parent
5c3900960b
commit
2a49b2b217
31 files changed
+3475
-2191
No files matched your search
@@ -126,11 +126,12 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
<strong>Interface tests:</strong><br>
|
||||
/tests/entrypoints/test_xxx<br>
|
||||
<strong>Performance:</strong><br>
|
||||
/tests/dfx/perf/tests/test_qwen_omni.json (Omni), test_tts.json (TTS),<br>
|
||||
and /tests/dfx/perf/tests/test_{diffusion_model}_vllm_omni.json (Diffusion)<br>
|
||||
/tests/dfx/perf/tests/test_qwen3_omni_*.json (Omni), test_tts.json (TTS),<br>
|
||||
test_voxcpm2.json, test_higgs_audio_v3.json, and<br>
|
||||
/tests/dfx/perf/tests/test_{diffusion_model}_vllm_omni.json (Diffusion)<br>
|
||||
<strong>Doc Test:</strong><br>
|
||||
tests/example/online_serving/test_{model_name}.py<br>
|
||||
tests/example/offline_inference/test_{model_name}.py<br>
|
||||
tests/examples/online_serving/test_{model_name}.py<br>
|
||||
tests/examples/offline_inference/test_{model_name}.py<br>
|
||||
<strong>Accuracy Test:</strong><br>
|
||||
/tests/e2e/accuracy/test_{model_name}.py
|
||||
</td>
|
||||
@@ -708,16 +709,16 @@ L4 level testing is a comprehensive quality audit before a version release. It e
|
||||
### 3.2 Testing Content and Scope
|
||||
|
||||
- ***Full Functionality Testing***: Executes all test cases defined in `test_{model_name}_expansion.py`, covering all implemented features, positive flows, boundary conditions, and exception handling.
|
||||
- ***Performance Testing***: Uses `tests/dfx/perf/tests/test_qwen_omni.json`, `tests/dfx/perf/tests/test_tts.json`, and diffusion configs in the form `tests/dfx/perf/tests/test_*_vllm_omni.json` (passed to `run_benchmark.py` via `--test-config-file`) to drive performance testing tools for stress, load, and endurance tests, collecting metrics like throughput, response time, and resource utilization.
|
||||
- ***Performance Testing***: Uses `tests/dfx/perf/tests/test_qwen3_omni_*.json` (Omni), `test_tts.json` / `test_voxcpm2.json` / `test_higgs_audio_v3.json` (TTS), and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (passed to `run_benchmark.py` or `run_diffusion_benchmark.py` via `--test-config-file` in nightly **Perf Test** steps) to drive throughput, latency, and memory benchmarks. Each JSON **case** may declare an optional top-level **`mark`** array: exactly one ``hardware_marks`` object plus pytest marker name strings (`full_model`, `omni` / `tts` / `diffusion`, …). Runners attach those marks to each parametrized `(server, benchmark index)` pair so **local** bulk runs can filter with `-m` (for example `-m "full_model and H100 and diffusion"`). Nightly CI perf jobs select workloads by **`--test-config-file`**, not `-m`. Details are in the Performance Tests example below.
|
||||
- ***Documentation Testing***: Verifies whether the example code provided to users is runnable and its results match the description.
|
||||
|
||||
### 3.3 Test Directory and Execution Files
|
||||
|
||||
- ***Functional Testing***: Same directories as L3.
|
||||
- ***Performance Test Configuration***: `tests/dfx/perf/tests/test_qwen_omni.json`, `tests/dfx/perf/tests/test_tts.json`, and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (e.g. `test_qwen_image_vllm_omni.json`)
|
||||
- ***Performance Test Configuration***: `tests/dfx/perf/tests/test_qwen3_omni_*.json`, `test_tts.json`, `test_voxcpm2.json`, `test_higgs_audio_v3.json`, and diffusion configs `tests/dfx/perf/tests/test_*_vllm_omni.json` (e.g. `test_qwen_image_vllm_omni.json`, `test_cosmos3_vllm_omni.json`). Optional per-case `mark` for local `-m` filtering is documented in Section 3.4 Performance Tests.
|
||||
- ***Documentation Example Tests***:
|
||||
- - `tests/example/online_serving/test_{model_name}.py`
|
||||
- `tests/example/offline_inference/test_{model_name}.py`
|
||||
- `tests/examples/online_serving/test_{model_name}.py`
|
||||
- `tests/examples/offline_inference/test_{model_name}.py`
|
||||
|
||||
### 3.4 Execution Method and Example
|
||||
|
||||
|
||||
@@ -1,14 +1,41 @@
|
||||
When you want to add L4-level ***performance test*** cases, you can refer to the following format for case addition in `tests/dfx/perf/tests/test_qwen_omni.json`, `tests/dfx/perf/tests/test_tts.json`, or diffusion configs such as `tests/dfx/perf/tests/test_*_vllm_omni.json` (selected via `pytest ... run_benchmark.py --test-config-file <path>`):
|
||||
When you want to add L4-level ***performance test*** cases, add entries to JSON files under `tests/dfx/perf/tests/` and run them via `tests/dfx/perf/scripts/run_benchmark.py` (omni / TTS) or `run_diffusion_benchmark.py` (diffusion).
|
||||
|
||||
**Config file layout (in-tree examples)**
|
||||
|
||||
| Model type | Runner | Example JSON files |
|
||||
| ---------- | ------ | ------------------ |
|
||||
| Omni | `run_benchmark.py` | `test_qwen3_omni_no_async_chunk.json`, `test_qwen3_omni_async_chunk.json`, `test_qwen3_omni_vllm_text.json`, `test_qwen3_omni_multi_replicas.json` |
|
||||
| TTS | `run_benchmark.py` | `test_tts.json`, `test_voxcpm2.json`, `test_higgs_audio_v3.json` |
|
||||
| Diffusion | `run_diffusion_benchmark.py` | `test_qwen_image_vllm_omni.json`, `test_bagel_vllm_omni.json`, `test_wan22_i2v_vllm_omni.json`, `test_cosmos3_vllm_omni.json`, … |
|
||||
|
||||
**How runners pick cases**
|
||||
|
||||
Without **`--test-config-file`**, each runner scans all `*.json` under `tests/dfx/perf/tests/` but only keeps its own model type:
|
||||
|
||||
- **`run_benchmark.py`**: omni and TTS cases only (skips diffusion JSON).
|
||||
- **`run_diffusion_benchmark.py`**: diffusion cases only (skips omni / TTS JSON).
|
||||
|
||||
Diffusion cases are detected when the JSON has `server_type` (typically `"vllm-omni"`) or `"diffusion"` in the `mark` array. Omni / TTS JSON has neither.
|
||||
|
||||
**Running perf cases**
|
||||
|
||||
Pass **`--test-config-file`** to run one JSON file as-is, or omit it for the bulk scan above and filter with pytest **`-m`** on each case's JSON `mark` (for example `pytest … run_benchmark.py -m "full_model and tts and H100"`).
|
||||
|
||||
```JSON
|
||||
{
|
||||
"test_name": "test_qwen3_omni",
|
||||
"test_name": "test_qwen3_omni_async_chunk",
|
||||
"mark": [
|
||||
{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}},
|
||||
"full_model",
|
||||
"omni"
|
||||
],
|
||||
"server_params": {
|
||||
"model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
|
||||
"stage_config_name": "qwen3_omni.yaml"
|
||||
},
|
||||
"benchmark_params": [
|
||||
{
|
||||
"name": "random_10p",
|
||||
"dataset_name": "random",
|
||||
"num_prompts": [10, 20],
|
||||
"max_concurrency": [1, 4],
|
||||
@@ -33,8 +60,95 @@ When you want to add L4-level ***performance test*** cases, you can refer to the
|
||||
| Field | Required | Description |
|
||||
| ---------------- | -------- | --------------------------------------------------------------- |
|
||||
| test_name | Yes | Unique identifier for the test case |
|
||||
| mark | No | Pytest marks for this case (see **`mark` field** below). Omit only for configs not meant to be filtered by `-m`. |
|
||||
| server_params | Yes | Server-side configuration parameters |
|
||||
| benchmark_params | Yes | Benchmark running parameters (supports multiple configurations) |
|
||||
| server_type | Diffusion only | When set (e.g. `"vllm-omni"`), routes the case to `run_diffusion_benchmark.py`. |
|
||||
| benchmark_endpoint | Diffusion only | API path for the benchmark client (e.g. `/v1/videos`, `/v1/images/generations`). |
|
||||
|
||||
**`mark` field**
|
||||
|
||||
Optional top-level field on each perf JSON **case object** (one per `test_name`). `run_benchmark.py` and `run_diffusion_benchmark.py` read it via `tests.dfx.conftest.resolve_pytest_marks` and attach the marks to the corresponding `pytest.param`, so you can filter locally with `-m`.
|
||||
|
||||
When `mark` is present, it must be an **array** with exactly one ``hardware_marks`` object (same shape as `@hardware_test` / `hardware_marks()` in `tests/helpers/mark.py`), followed by registered pytest marker name strings such as `full_model`, `omni`, `tts`, `diffusion`, or `local_model`.
|
||||
|
||||
Supported form:
|
||||
|
||||
| Form | Example | Effect |
|
||||
| ---- | ------- | ------ |
|
||||
| Array (single platform) | `"mark": [{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}}, "full_model", "diffusion"]` | Hardware marks from `hardware_marks(...)`; `num_cards` > 1 adds `distributed_*` (+ CUDA `skipif_cuda` when applicable). String entries become extra pytest marks. |
|
||||
| Array (multi-platform) | `"mark": [{"hardware_marks": {"res": {"cuda": "H100", "rocm": "MI325", "npu": "A2"}, "num_cards": {"cuda": 2, "rocm": 2, "npu": 1}}}, "full_model", "omni"]` | Same as above, but declares **multiple platforms** in one case. Filter examples: `-m "full_model and H100 and cuda"`, `-m "full_model and MI325 and rocm"`. |
|
||||
|
||||
Recommended for L4 perf cases:
|
||||
|
||||
```JSON
|
||||
{
|
||||
"test_name": "test_bagel_single_device_single_stage_t2i",
|
||||
"mark": [
|
||||
{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 1}},
|
||||
"full_model",
|
||||
"diffusion"
|
||||
],
|
||||
"server_type": "vllm-omni",
|
||||
"server_params": { "...": "..." },
|
||||
"benchmark_params": [ { "name": "1024x1024_steps20", "...": "..." } ]
|
||||
}
|
||||
```
|
||||
|
||||
Multi-GPU diffusion (example: Cosmos3 with `cfg-parallel-size=2`):
|
||||
|
||||
```JSON
|
||||
{
|
||||
"test_name": "test_cosmos3_t2i_official_demo_2gpu",
|
||||
"mark": [
|
||||
{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}},
|
||||
"full_model",
|
||||
"diffusion"
|
||||
],
|
||||
"server_type": "vllm-omni",
|
||||
"benchmark_endpoint": "/v1/images/generations",
|
||||
"server_params": { "...": "..." },
|
||||
"benchmark_params": [ { "name": "1024x1024_steps4", "...": "..." } ]
|
||||
}
|
||||
```
|
||||
|
||||
- Omni perf: include `"omni"` in the `mark` array
|
||||
- TTS perf: include `"tts"` in the `mark` array
|
||||
- Diffusion perf: include `"diffusion"` in the `mark` array
|
||||
- HunyuanImage local-weight cases: add `"local_model"` to the `mark` array
|
||||
|
||||
**Parametrization IDs**
|
||||
|
||||
Each `(server, benchmark index)` pair becomes one `pytest.param` with id `{test_name}-{suffix}`. The suffix comes from `benchmark_params[].name` when set (for example `test_tts-p0`, `test_omni-p1`); otherwise it is derived from fields like `task` / `eval_phase`.
|
||||
|
||||
**Benchmark result filenames**
|
||||
|
||||
Result files use the **runtime** hardware label from `get_runtime_resource_label()` (detected GPU/NPU name on the machine that ran the job), **not** `mark.hardware_marks.res`. On the default H100 CI pool, `H100` is **omitted** from filenames (`resource_label_for_filename`).
|
||||
|
||||
Examples:
|
||||
|
||||
- Omni/TTS: `result_{test_name}_{optional_hw}_{dataset}_....json` under `BENCHMARK_DIR`
|
||||
- Diffusion: one aggregate `diffusion_result_{config_stem}_{optional_hw}_{timestamp}.json` per source JSON file (array of all runs from that file)
|
||||
|
||||
**Local commands**
|
||||
|
||||
```bash
|
||||
# Bulk load + filter by JSON mark
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and H100 and diffusion"
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py -m "full_model and omni and H100"
|
||||
|
||||
# Single file (same as nightly CI Perf steps)
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py \
|
||||
--test-config-file tests/dfx/perf/tests/test_bagel_vllm_omni.json
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
|
||||
--test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json
|
||||
|
||||
# Optional baseline assertion (default off)
|
||||
pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py --assert-baseline \
|
||||
--test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
|
||||
```
|
||||
|
||||
See also [Markers for Tests](./tests_markers.md) for registered hardware markers (`H100`, `L4`, `cuda`, `distributed_cuda`, …).
|
||||
|
||||
**`server_params` Configuration**
|
||||
|
||||
@@ -79,10 +193,12 @@ You can add any benchmark running parameters you need here. For all optional par
|
||||
1. Change the --xxx-xx-xx running parameters to xxx_xx_xx format and fill them as keys in the JSON file.
|
||||
2. For boolean variables in the running parameters, modify them to forms such as ignore_eos: true/false and fill them into the JSON file.
|
||||
3. Optionally add a `baseline` object (see **Baseline thresholds** below). If you omit `baseline` or leave it empty, the performance test still runs but does not assert metric thresholds from this field.
|
||||
4. The qps and concurrency modes are recommended to be mutually exclusive. For detailed explanations, see the table below:
|
||||
4. Set `"name"` on each `benchmark_params` entry for stable pytest ids and readable result keys.
|
||||
5. The qps and concurrency modes are recommended to be mutually exclusive. For detailed explanations, see the table below:
|
||||
|
||||
| Parameter | Type | Required | Example/Values | Description |
|
||||
| --------------- | ----------- | -------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| name | string | No | `"1024x1024_steps4"` | Stable suffix for pytest param id and logs. |
|
||||
| num_prompts | int / array | Yes | 10,[10, 20, 30] | Number of requests. Supports single values or arrays. If a single value is used, it will be automatically expanded to match the number of qps or max_concurrency, e.g., [10,10,10]. If an array is used, its length must match the number of qps or max_concurrency. |
|
||||
| request_rate | float / array | No | 0.5, [0.5, 1, inf] | Queries per second. Supports single values or arrays. If a single value is used, it will be automatically expanded to match the number of num_prompts, e.g., [1,1,1]. If an array is used, its length must match the number of num_prompts. |
|
||||
| max_concurrency | int / array | No | 1, [1, 2, 3] | Maximum concurrent in-flight requests. Same array / expansion rules as `request_rate` (mutually exclusive with QPS mode). |
|
||||
|
||||
@@ -149,12 +149,18 @@ Failed jobs: 1/2
|
||||
pytest -s -v -m "full_model and distributed_cuda and L4" --run-level=full_model
|
||||
pytest -s -v -m "full_model and (omni or tts) and H100" --run-level=full_model
|
||||
```
|
||||
|
||||
To run a standalone perf benchmark (defaults to ``test_qwen_omni.json``; use ``--test-config-file tests/dfx/perf/tests/test_tts.json`` for TTS):
|
||||
|
||||
If you only want to run specific test cases on a particular platform, you can use:
|
||||
```bash
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py
|
||||
pytest -s -v -m "full_model and distributed_cuda and L4" --run-level=full_model
|
||||
```
|
||||
Note: ``run_benchmark.py`` and ``run_diffusion_benchmark.py`` accept an optional ``--test-config-file``. If omitted, each loads every ``*.json`` under ``tests/dfx/perf/tests/`` (omni/tts vs diffusion split by ``is_diffusion_perf_config``) and pytest ``-m`` filters by each case's JSON ``mark``:
|
||||
```bash
|
||||
pytest -sv tests/dfx/perf/scripts/run_benchmark.py -m "full_model and tts and H100"
|
||||
pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and diffusion and H100"
|
||||
pytest -sv tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json
|
||||
pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_cosmos3_vllm_omni.json
|
||||
```
|
||||
Nightly **Perf Test** jobs in ``test-nightly.yml`` use ``--test-config-file`` only (no ``-m``). E2e L4 function tests still use ``full_model`` + ``--run-level full_model`` (see [test-nightly.yml](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/test-nightly.yml)). Example:
|
||||
|
||||
=== "L5 level"
|
||||
|
||||
|
||||
@@ -33,7 +33,8 @@ Defined in `pyproject.toml`:
|
||||
| `skipif_rocm` | Skip if the num of ROCm cards is less than the required * |
|
||||
| `skipif_npu` | Skip if the num of NPU cards is less than the required * |
|
||||
| `slow` | Slow tests (may skip in quick CI) |
|
||||
| `benchmark` | Benchmark tests |
|
||||
| `benchmark` | Benchmark tests (decorator on runner test functions; perf JSON uses `full_model` + type marker instead) |
|
||||
| `local_model` | Tests requiring local / non-HF-hub model weights |
|
||||
|
||||
\* Means those markers are auto-added by `@hardware_test` (parametrization decorator) or `hardware_marks` (only returning the list of marks for flexibility).
|
||||
|
||||
@@ -94,7 +95,7 @@ This decorator is intended to make hardware-aware, cross-platform test authoring
|
||||
num_cards=2,
|
||||
)
|
||||
```
|
||||
- `res` must be a dict; supported resources: CUDA (L4/H100), ROCm (MI325), NPU (A2/A3)
|
||||
- `res` must be a dict; supported resources: CUDA (L4/H100), ROCm (MI325), XPU (B60), MUSA (S5000), NPU (A2/A3)
|
||||
- `num_cards` can be int (all platforms) or dict (per platform); defaults to 1 when missing
|
||||
- Distributed markers (`distributed_cuda`, `distributed_rocm`, `distributed_npu`) are auto-added for multi-card cases
|
||||
- Filtering examples:
|
||||
@@ -120,6 +121,32 @@ def test_feature(omni_server):
|
||||
...
|
||||
```
|
||||
|
||||
### JSON `mark` field (L4 perf configs)
|
||||
|
||||
Perf JSON under `tests/dfx/perf/tests/` can attach pytest marks per **case** (`test_name` block). Parsed by `tests.dfx.conftest.resolve_pytest_marks` and applied to each parametrized run in `run_benchmark.py` / `run_diffusion_benchmark.py`.
|
||||
|
||||
When `mark` is present, it must be an **array** with exactly one ``hardware_marks`` object (same semantics as `hardware_marks()` above), followed by registered pytest marker name strings such as `full_model`, `omni`, `tts`, or `diffusion`.
|
||||
|
||||
```json
|
||||
{
|
||||
"test_name": "test_cosmos3_t2i_official_demo_2gpu",
|
||||
"mark": [
|
||||
{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}},
|
||||
"full_model",
|
||||
"diffusion"
|
||||
],
|
||||
"server_type": "vllm-omni",
|
||||
"server_params": { "...": "..." },
|
||||
"benchmark_params": [{ "name": "1024x1024_steps4", "...": "..." }]
|
||||
}
|
||||
```
|
||||
|
||||
- Local bulk load: `pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and diffusion and H100"`
|
||||
- Nightly CI perf steps: `--test-config-file tests/dfx/perf/tests/test_<model>_vllm_omni.json` (file selects cases; no `-m`)
|
||||
- Result filenames use **runtime** GPU detection (`get_runtime_resource_label`); `H100` is omitted on the default CI pool
|
||||
|
||||
See [L4 performance test examples](./test_examples/l4_performance_tests.inc.md) for the full schema.
|
||||
|
||||
## Add Support for a New Platform
|
||||
|
||||
If you want to add support for a new platform (e.g., "tpu" for a new accelerator), follow these steps:
|
||||
|
||||
Reference in new issue
Block a user