anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[CI][Perf] migrate diffusion DFX benches to vllm bench serve --omni (#7737)

Browse Source 
Signed-off-by: wangyu <410167048@qq.com>
This commit is contained in:
wangyu authored and GitHub committed 2026-09-21 19:16:39 -07:00
1 parent a4ad73e08b
commit 0570b0da7e
30 files changed
+2040 -729
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/contributing/ci/test_examples/l4_performance_tests.inc.md
+32 -20
View File
@@ -7,16 +7,17 @@ When you want to add L4-level ***performance test*** cases, add entries to JSON
| Omni (nightly) | `run_benchmark.py` | `test_qwen3_omni_no_async_chunk.json`, `test_qwen3_omni_async_chunk.json` (`full_model` without `slow` in `mark`) |
| Omni (weekly) | `run_benchmark.py` | `test_qwen3_omni_async_chunk.json` (CUDA only), `test_qwen3_omni_vllm_text.json`, `test_qwen3_omni_multi_replicas.json` (`slow` in `mark`; **Perf Test** in `test-weekly.yml`) |
| TTS | `run_benchmark.py` | `test_tts.json`, `test_voxcpm2.json`, `test_higgs_audio_v3.json` |
| Diffusion | `run_diffusion_benchmark.py` | `test_qwen_image_vllm_omni.json`, `test_bagel_vllm_omni.json`, `test_wan22_i2v_vllm_omni.json`, `test_cosmos3_vllm_omni.json`, … |
| Diffusion (`/v1/chat/completions`) | `run_diffusion_benchmark.py` | `test_qwen_image_vllm_omni.json`, `test_bagel_vllm_omni.json`, … |
| Diffusion (`/v1/images/generations`, `/v1/images/edits`, `/v1/videos`) | `run_benchmark.py` | `test_wan22_i2v_vllm_omni.json`, `test_cosmos3_vllm_omni.json`, `test_lingbot_video_vllm_omni.json`, … |
#### How runners pick cases
Without **`--test-config-file`**, each runner scans all `*.json` under `tests/dfx/perf/tests/` but only keeps its own model type:
- **`run_benchmark.py`**: omni and TTS cases only (skips diffusion JSON).
- **`run_diffusion_benchmark.py`**: diffusion cases only (skips omni / TTS JSON).
- **`run_benchmark.py`**: cases whose `benchmark_params` use ``dataset_name`` (omni, TTS, and image/video OpenAI generation via `vllm bench serve --omni`).
- **`run_diffusion_benchmark.py`**: cases whose `benchmark_params` use ``dataset`` (diffusion client schema, including custom jsonl).
Diffusion cases are detected when the JSON has `server_type` (typically `"vllm-omni"`) or `"diffusion"` in the `mark` array. Omni / TTS JSON has neither.
Diffusion-script cases are detected by `is_diffusion_perf_config()`: presence of ``dataset`` (without ``dataset_name``) in `benchmark_params`. Marks / `server_type` / endpoint values are not used for this split.
#### Running perf cases
@@ -60,16 +61,16 @@ Pass **`--test-config-file`** to load one JSON file, or omit it for the bulk sca
##### Overview
| Field | Required | Description |
| ------------------ | -------------- | -------------------------------------------- |
| test_name | Yes | Unique identifier for the test case |
| mark | No | Pytest marks; see **`mark` field** below |
| server_params | Yes | Server-side configuration parameters |
| benchmark_params | Yes | Benchmark running parameters |
| server_type | Diffusion only | Routes case to run_diffusion_benchmark.py |
| benchmark_endpoint | Diffusion only | Benchmark API path |
| Field | Required | Description |
| ------------------ | --------- | ---------------------------------------------------------------------------------------- |
| test_name | Yes | Unique identifier for the test case |
| mark | No | Pytest marks; see **`mark` field** below |
| server_params | Yes | Server-side configuration parameters |
| benchmark_params | Yes | Benchmark running parameters |
| server_type | Diffusion | Only for diffusion-script JSON; omit on omni-bench generation cases |
| benchmark_endpoint | Optional | Legacy diffusion custom-jsonl alias; prefer `benchmark_params[].endpoint` |
Omit `mark` only for configs not meant to be filtered by `-m`. `server_type` is typically `"vllm-omni"`. `benchmark_endpoint` examples: `/v1/videos`, `/v1/images/generations`.
Omit `mark` only for configs not meant to be filtered by `-m`. Cases that call `/v1/images/edits`, `/v1/images/generations`, or `/v1/videos` use the same `benchmark_params` schema as Omni/TTS (`dataset_name`, `endpoint`, `extra_body`) and are executed by `run_benchmark.py` — do not set `server_type` or `task` on those cases. Remaining diffusion cases (usually `/v1/chat/completions`, or custom jsonl) stay on `run_diffusion_benchmark.py` and may keep `server_type`.
#### `mark` field
@@ -100,7 +101,7 @@ Recommended for L4 perf cases:
}
```
Multi-GPU diffusion (example: Cosmos3 with `cfg-parallel-size=2`):
Multi-GPU generation via omni bench (example: Cosmos3 with `cfg-parallel-size=2`):
```JSON
{
@@ -110,10 +111,17 @@ Multi-GPU diffusion (example: Cosmos3 with `cfg-parallel-size=2`):
"full_model",
"diffusion"
],
"server_type": "vllm-omni",
"benchmark_endpoint": "/v1/images/generations",
"server_params": { "...": "..." },
"benchmark_params": [ { "name": "1024x1024_steps4", "...": "..." } ]
"benchmark_params": [
{
"name": "1024x1024_steps4",
"dataset_name": "random",
"endpoint": "/v1/images/generations",
"num_prompts": 3,
"max_concurrency": 1,
"extra_body": { "width": 1024, "height": 1024, "num_inference_steps": 4 }
}
]
}
```
@@ -132,8 +140,8 @@ Result files use the **runtime** hardware label from `get_runtime_resource_label
Examples:
- Omni/TTS: `result_{test_name}_{optional_hw}_{dataset}_....json` under `BENCHMARK_DIR`
- Diffusion: one aggregate `diffusion_result_{config_stem}_{optional_hw}_{timestamp}.json` per source JSON file (array of all runs from that file)
- Omni/TTS and OpenAI generation endpoints (`/v1/images/*`, `/v1/videos`): `result_{test_name}_{optional_hw}_{dataset}_....json` under `BENCHMARK_DIR`
- Remaining diffusion (`run_diffusion_benchmark.py`): one aggregate `diffusion_result_{config_stem}_{optional_hw}_{timestamp}.json` per source JSON file
#### Local commands
@@ -145,6 +153,9 @@ pytest -s -v tests/dfx/perf/scripts/run_benchmark.py -m "full_model and omni and
# Single file (same selectors as the CI Perf steps)
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py \
--test-config-file tests/dfx/perf/tests/test_bagel_vllm_omni.json
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
--test-config-file tests/dfx/perf/tests/test_cosmos3_vllm_omni.json \
-m "H100 and B200 and cards_2"
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
--test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json \
-m "H100 and full_model and not slow"
@@ -203,7 +214,8 @@ You can add any benchmark running parameters you need here. For all optional par
2. For boolean variables in the running parameters, modify them to forms such as ignore_eos: true/false and fill them into the JSON file.
3. Optionally add a `baseline` object (see **Baseline thresholds** below). If you omit `baseline` or leave it empty, the performance test still runs but does not assert metric thresholds from this field.
4. Set `"name"` on each `benchmark_params` entry for stable pytest ids and readable result keys.
5. The qps and concurrency modes are recommended to be mutually exclusive. For detailed explanations, see the table below:
5. Image/video generation cases (`/v1/images/generations`, `/v1/images/edits`, `/v1/videos`) use this same schema: set `endpoint` to the API path, put width/height/steps/frames in `extra_body`, use `dataset_name: random` for text-only inputs or `random-mm` when the request needs a synthetic image/video. Do not set `server_type` or `task`, and do not use `random-request-config` or kebab-case diffusion client fields.
6. The qps and concurrency modes are recommended to be mutually exclusive. For detailed explanations, see the table below:
| Parameter | Type | Required | Example/Values | Description |
| --------------- | ------------- | -------- | -------------------- | ------------------------------------ |
docs/contributing/ci/test_execution_guide.md
+1 -1
View File
@@ -165,7 +165,7 @@ Failed jobs: 1/2
pytest -sv tests/dfx/perf/scripts/run_benchmark.py -m "full_model and tts and H100"
pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and diffusion and H100"
pytest -sv tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json
pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_cosmos3_vllm_omni.json
pytest -sv tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_cosmos3_vllm_omni.json
```
Nightly **Perf Test** jobs in [``test-nightly.yml``](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/test-nightly.yml) use ``--test-config-file`` only (no ``-m``). Weekly **Perf Test** in [``test-weekly.yml``](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/cuda/test-weekly.yml) runs ``test_qwen3_omni_vllm_text.json`` and ``test_qwen3_omni_multi_replicas.json`` (JSON ``mark`` includes ``slow``). E2e L4 function tests use ``full_model`` + ``--run-level full_model``. Example:
docs/contributing/ci/test_writing_guide.md
+2 -1
View File
@@ -159,7 +159,8 @@ When `mark` is present, it must be an **array** with exactly one ``hardware_mark
}
```
- Local bulk load: `pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and diffusion and H100"`
- Local bulk load: `pytest -sv tests/dfx/perf/scripts/run_benchmark.py -m "full_model and H100"` (omni/TTS and `/v1/images/*` + `/v1/videos` diffusion)
- Diffusion chat-completions remaining cases: `pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and diffusion and H100"`
- Nightly CI perf steps: `--test-config-file tests/dfx/perf/tests/test_<model>_vllm_omni.json` (file selects cases; no `-m`)
- Result filenames use **runtime** GPU detection (`get_runtime_resource_label`); `H100` is omitted on the default CI pool