vllm-omni/docs/contributing/ci/test_examples/l4_performance_tests.inc.md

When you want to add L4-level performance test cases, add entries to JSON files under tests/dfx/perf/tests/ and run them via tests/dfx/perf/scripts/run_benchmark.py (omni / TTS) or run_diffusion_benchmark.py (diffusion).

Config file layout (in-tree examples)

Model type	Runner	Example JSON files
Omni (nightly)	run_benchmark.py	test_qwen3_omni_no_async_chunk.json, test_qwen3_omni_async_chunk.json (full_model without slow in mark)
Omni (weekly)	run_benchmark.py	test_qwen3_omni_async_chunk.json (CUDA only), test_qwen3_omni_vllm_text.json, test_qwen3_omni_multi_replicas.json (slow in mark; Perf Test in test-weekly.yml)
TTS	run_benchmark.py	test_tts.json, test_voxcpm2.json, test_higgs_audio_v3.json
Diffusion	run_diffusion_benchmark.py	test_qwen_image_vllm_omni.json, test_bagel_vllm_omni.json, test_wan22_i2v_vllm_omni.json, test_cosmos3_vllm_omni.json, …

How runners pick cases

Without --test-config-file, each runner scans all *.json under tests/dfx/perf/tests/ but only keeps its own model type:

• run_benchmark.py: omni and TTS cases only (skips diffusion JSON).
• run_diffusion_benchmark.py: diffusion cases only (skips omni / TTS JSON).

Diffusion cases are detected when the JSON has server_type (typically "vllm-omni") or "diffusion" in the mark array. Omni / TTS JSON has neither.

Running perf cases

Pass --test-config-file to load one JSON file, or omit it for the bulk scan above. In either mode, pytest -m filters each case using its JSON mark. A single file can contain nightly, weekly, and platform-specific case objects. For example, the nightly object in test_qwen3_omni_async_chunk.json declares both H100 and A3, while its slow weekly object is CUDA-only; there is no NPU weekly counterpart. Use -m "H100 and full_model and not slow", -m "H100 and full_model and slow", or -m "npu" to select the intended schedule and platform.

{
    "test_name": "test_qwen3_omni_async_chunk",
    "mark": [
        {"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}},
        "full_model",
        "omni"
    ],
    "server_params": {
        "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
        "stage_config_name": "qwen3_omni_moe.yaml"
    },
    "benchmark_params": [
        {
            "name": "random_10p",
            "dataset_name": "random",
            "num_prompts": [10, 20],
            "max_concurrency": [1, 4],
            "random_input_len": 2500,
            "random_output_len": 900,
            "ignore_eos": true,
            "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
            "baseline": {
                "H100": {
                    "mean_ttft_ms": [500, 800],
                    "mean_audio_ttfp_ms": [2000, 3500],
                    "mean_audio_rtf": [0.25, 0.35]
                }
            }
        }
    ]
}

Parameter Explanation

Overview

Field	Required	Description
test_name	Yes	Unique identifier for the test case
mark	No	Pytest marks; see mark field below
server_params	Yes	Server-side configuration parameters
benchmark_params	Yes	Benchmark running parameters
server_type	Diffusion only	Routes case to run_diffusion_benchmark.py
benchmark_endpoint	Diffusion only	Benchmark API path

Omit mark only for configs not meant to be filtered by -m. server_type is typically "vllm-omni". benchmark_endpoint examples: /v1/videos, /v1/images/generations.

mark field

Optional top-level field on each perf JSON case object (one per test_name). run_benchmark.py and run_diffusion_benchmark.py read it via tests.dfx.conftest.resolve_pytest_marks and attach the marks to the corresponding pytest.param, so you can filter locally with -m.

When mark is present, it must be an array with exactly one hardware_marks object (same shape as @hardware_test / hardware_marks() in tests/helpers/mark.py), followed by registered pytest marker name strings such as full_model, slow, omni, tts, diffusion, or local_model.

Supported form:

Form	Example	Effect
Array (single platform)	"mark": [{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}}, "full_model", "diffusion"]	Hardware marks from hardware_marks(...); num_cards adds cards_{n} (including cards_1 when omitted). String entries become extra pytest marks.
Array (multi-platform)	"mark": [{"hardware_marks": {"res": {"cuda": "H100", "rocm": "MI325"}, "num_cards": 2}}, "full_model", "omni"]	Same as above, but declares multiple platforms on one item. All platforms must share the same num_cards (int or identical dict values). Different counts need @hardware_test or one hardware_marks() per pytest.param. Filter examples: -m "full_model and H100 and cards_2 and cuda", -m "full_model and MI325 and cards_2 and rocm".

Recommended for L4 perf cases:

{
    "test_name": "test_bagel_single_device_single_stage_t2i",
    "mark": [
        {"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 1}},
        "full_model",
        "diffusion"
    ],
    "server_type": "vllm-omni",
    "server_params": { "...": "..." },
    "benchmark_params": [ { "name": "1024x1024_steps20", "...": "..." } ]
}

Multi-GPU diffusion (example: Cosmos3 with cfg-parallel-size=2):

{
    "test_name": "test_cosmos3_t2i_official_demo_2gpu",
    "mark": [
        {"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 2}},
        "full_model",
        "diffusion"
    ],
    "server_type": "vllm-omni",
    "benchmark_endpoint": "/v1/images/generations",
    "server_params": { "...": "..." },
    "benchmark_params": [ { "name": "1024x1024_steps4", "...": "..." } ]
}

• Omni perf: include "omni" in the mark array
• TTS perf: include "tts" in the mark array
• Diffusion perf: include "diffusion" in the mark array
• HunyuanImage local-weight cases: add "local_model" to the mark array

Parametrization IDs

Each (server, benchmark index) pair becomes one pytest.param with id {test_name}-{suffix}. The suffix comes from benchmark_params[].name when set (for example test_tts-p0, test_omni-p1); otherwise it is derived from fields like task / eval_phase.

Benchmark result filenames

Result files use the runtime hardware label from get_runtime_resource_label() (detected GPU/NPU name on the machine that ran the job), not mark.hardware_marks.res. On the default H100 CI pool, H100 is omitted from filenames (resource_label_for_filename).

Examples:

• Omni/TTS: result_{test_name}_{optional_hw}_{dataset}_....json under BENCHMARK_DIR
• Diffusion: one aggregate diffusion_result_{config_stem}_{optional_hw}_{timestamp}.json per source JSON file (array of all runs from that file)

Local commands

# Bulk load + filter by JSON mark
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "full_model and H100 and diffusion"
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py -m "full_model and omni and H100"

# Single file (same selectors as the CI Perf steps)
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py \
  --test-config-file tests/dfx/perf/tests/test_bagel_vllm_omni.json
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
  --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json \
  -m "H100 and full_model and not slow"
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
  --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json \
  -m "H100 and full_model and slow"
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
  --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json \
  -m "npu"

See also Markers for Tests for registered hardware markers (H100, L4, cuda, cards_1, …).

server_params Configuration

Basic Parameters

Parameter	Required	Example	Description
model	Yes	Qwen/Qwen3-Omni-30B-A3B…	Model name or path
stage_config_name	Yes	qwen3_omni_moe.yaml	Deploy configuration file name

Dynamic Configuration (update/delete)

Supports incremental modifications based on the basic configuration:

Operation	Description
update	Update or add configuration items
delete	Delete specified configuration items

Example

"update": {
    "async_chunk": true,  // Enable asynchronous chunk processing
    "stages": {
        "0": {
            "max_num_seqs": 64
        }
    }
},
"delete": {
    "stages": {
        "2": ["max_num_batched_tokens"]  // Delete this deploy override for stage 2
    }
}

benchmark_params Configuration

You can add any benchmark running parameters you need here. For all optional parameters, refer to the benchmark documentation. General modifications are as follows:

1. Change the --xxx-xx-xx running parameters to xxx_xx_xx format and fill them as keys in the JSON file.
2. For boolean variables in the running parameters, modify them to forms such as ignore_eos: true/false and fill them into the JSON file.
3. Optionally add a baseline object (see Baseline thresholds below). If you omit baseline or leave it empty, the performance test still runs but does not assert metric thresholds from this field.
4. Set "name" on each benchmark_params entry for stable pytest ids and readable result keys.
5. The qps and concurrency modes are recommended to be mutually exclusive. For detailed explanations, see the table below:

Parameter	Type	Required	Example/Values	Description
name	string	No	"1024x1024_steps4"	Stable suffix for pytest param id
num_prompts	int / array	Yes	10, [10, 20, 30]	Number of requests (scalar or sweep)
request_rate	float / array	No	0.5, [0.5, 1, inf]	Queries per second (QPS mode)
max_concurrency	int / array	No	1, [1, 2, 3]	Max in-flight requests
baseline	object	No	see below	Optional hardware-nested thresholds

num_prompts, request_rate, and max_concurrency accept scalars or arrays; single values expand to match sweep length. QPS and concurrency modes are mutually exclusive. baseline top-level keys must be hardware pytest markers from pyproject.toml; metric names and values are free-form (scalars or sweep-aligned lists). Omit baseline to run without threshold assertions.