[Test] L5 Long-Term Stability Test and GPU Memory Monitoring Main L5 last (#1799)
This commit is contained in:
1 parent
65d76a4127
commit
c14ab819e3
15 files changed
+1613
-101
No files matched your search
@@ -86,7 +86,7 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
/tests/e2e/online_serving/test_{model_name}_expansion.py<br>
|
||||
/tests/e2e/offline_inference/test_{model_name}_expansion.py<br>
|
||||
<strong>Performance:</strong><br>
|
||||
/tests/e2e/perf/nightly.json<br>
|
||||
/tests/dfx/perf/tests/test.json<br>
|
||||
<strong>Doc Test:</strong><br>
|
||||
tests/example/online_serving/test_{model_name}.py<br>
|
||||
tests/example/offline_inference/test_{model_name}.py
|
||||
@@ -104,7 +104,7 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
|
||||
<td> Depends on reality </td>
|
||||
<td>
|
||||
<strong>Stability:</strong><br>
|
||||
tests/e2e/stability/weekly.json<br>
|
||||
/tests/dfx/stability/tests/test.json<br>
|
||||
<strong>Reliability:</strong><br>
|
||||
tests/e2e/reliability/test_{model_name}.py
|
||||
</td>
|
||||
@@ -529,13 +529,13 @@ L4 level testing is a comprehensive quality audit before a version release. It e
|
||||
### 3.2 Testing Content and Scope
|
||||
|
||||
- ***Full Functionality Testing***: Executes all test cases defined in `test_{model_name}_expansion.py`, covering all implemented features, positive flows, boundary conditions, and exception handling.
|
||||
- ***Performance Testing***: Uses the `/tests/e2e/perf/nightly.json` configuration file to drive performance testing tools for stress, load, and endurance tests, collecting metrics like throughput, response time, and resource utilization.
|
||||
- ***Performance Testing***: Uses the `tests/dfx/perf/tests/test.json` configuration file to drive performance testing tools for stress, load, and endurance tests, collecting metrics like throughput, response time, and resource utilization.
|
||||
- ***Documentation Testing***: Verifies whether the example code provided to users is runnable and its results match the description.
|
||||
|
||||
### 3.3 Test Directory and Execution Files
|
||||
|
||||
- ***Functional Testing***: Same directories as L3.
|
||||
- ***Performance Test Configuration***: `/tests/e2e/perf/nightly.json`
|
||||
- ***Performance Test Configuration***: `tests/dfx/perf/tests/test.json`
|
||||
- ***Documentation Example Tests***:
|
||||
- - `tests/example/online_serving/test_{model_name}.py`
|
||||
- `tests/example/offline_inference/test_{model_name}.py`
|
||||
@@ -552,7 +552,7 @@ L4 level testing is a comprehensive quality audit before a version release. It e
|
||||
|
||||
--8<-- "docs/contributing/ci/test_examples/doc_example_tests.inc.md"
|
||||
|
||||
When you want to add L4-level ***performance test*** cases, you can refer to the following format for case addition in tests/perf/tests/test.json:
|
||||
When you want to add L4-level ***performance test*** cases, you can refer to the following format for case addition in tests/dfx/perf/tests/test.json:
|
||||
|
||||
```JSON
|
||||
{
|
||||
@@ -654,12 +654,12 @@ L5 level testing focuses on the performance of model services under ***long-runn
|
||||
|
||||
### 4.2 Testing Content and Scope
|
||||
|
||||
- ***Long-term Stability (Stability) Testing***: Uses the `tests/e2e/stability/weekly.json` configuration to run the service under moderate load for an extended period (e.g., over 12 hours), monitoring whether metrics like memory/VRAM usage, response time, and throughput degrade over time, and whether the service process remains stable.
|
||||
- ***Long-term Stability (Stability) Testing***: Uses the `tests/dfx/stability/tests/test.json` configuration to run the service under moderate load for an extended period (e.g., over 12 hours), monitoring whether metrics like memory/VRAM usage, response time, and throughput degrade over time, and whether the service process remains stable.
|
||||
- ***Reliability Testing***: Uses `tests/e2e/reliability/test_{model_name}.py` to actively simulate various fault and abnormal scenarios, such as: dependent service interruption, abnormal input data, network flicker, hardware resource preemption, etc., to verify the system's fault tolerance, self-healing, and graceful degradation capabilities.
|
||||
|
||||
### 4.3 Test Directory and Execution Files
|
||||
|
||||
- ***Stability Test Configuration***: `tests/e2e/stability/weekly.json`
|
||||
- ***Stability Test Configuration***: `tests/dfx/stability/tests/test.json`
|
||||
- ***Reliability Test Suite***: `tests/e2e/reliability/test_{model_name}.py`
|
||||
|
||||
### 4.4 Execution Method and Example
|
||||
@@ -669,12 +669,79 @@ L5 level testing focuses on the performance of model services under ***long-runn
|
||||
- ***Script Example***:
|
||||
<details>
|
||||
<summary> Test Examples</summary>
|
||||
```python
|
||||
# WIP
|
||||
|
||||
When you want to add L5-level stability test cases, you can refer to the following format for case addition in `tests/dfx/stability/tests/test.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"test_name": "test_qwen3_omni_stability",
|
||||
"server_params": {
|
||||
"model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
|
||||
"stage_config_name": "qwen3_omni.yaml"
|
||||
},
|
||||
"benchmark_params": [
|
||||
{
|
||||
"dataset_name": "random",
|
||||
"backend": "openai-chat-omni",
|
||||
"endpoint": "/v1/chat/completions",
|
||||
"duration_sec": 43200,
|
||||
"request_rate": 0.5,
|
||||
"num_prompts_per_batch": 20,
|
||||
"random_input_len": 2500,
|
||||
"random_output_len": 900,
|
||||
"ignore_eos": true,
|
||||
"percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Parameter Explanation
|
||||
|
||||
***Overview***
|
||||
|
||||
| Field | Required | Description |
|
||||
| ---------------- | -------- | --------------------------------------------------------------------------- |
|
||||
| test_name | Yes | Unique identifier for the stability test case |
|
||||
| server_params | Yes | Server-side configuration parameters (model, stage configuration, etc.) |
|
||||
| benchmark_params | Yes | Stability benchmark running parameters (supports multiple configurations) |
|
||||
|
||||
#### server_params Configuration
|
||||
|
||||
##### Basic Parameters
|
||||
|
||||
| Parameter | Required | Example | Description |
|
||||
| ----------------- | -------- | ---------------------------------- | ----------------------------------- |
|
||||
| model | Yes | "Qwen/Qwen3-Omni-30B-A3B-Instruct" | Model name or path |
|
||||
| stage_config_name | Yes | "qwen3_omni.yaml" | Stage configuration file name |
|
||||
|
||||
##### Dynamic Configuration (update/delete)
|
||||
|
||||
Supports incremental modifications based on the basic configuration:
|
||||
|
||||
| Operation | Description |
|
||||
| --------- | ------------------------------------ |
|
||||
| update | Update or add configuration items |
|
||||
| delete | Delete specified configuration items |
|
||||
|
||||
***Example***:
|
||||
You can refer to Test Examples in Chapter 3.4
|
||||
|
||||
#### benchmark_params Configuration
|
||||
|
||||
For stability testing, the key parameters are:
|
||||
|
||||
- **duration_sec**: Total duration (in seconds) during which benchmark traffic is sent. The stability benchmark will keep sending batches until this duration is reached.
|
||||
- **request_rate** / **max_concurrency**: Exactly one of them must be specified. They control how the traffic is generated for each batch:
|
||||
- `request_rate`: Number of requests per second. The benchmark will send `num_prompts_per_batch` requests at the given rate.
|
||||
- `max_concurrency`: Maximum number of concurrent requests. When this is used, `request_rate` is set to `inf` internally.
|
||||
- **num_prompts_per_batch**: Number of prompts sent in each batch. Multiple batches will be executed sequentially within `duration_sec`.
|
||||
|
||||
All other optional parameters follow the same rules as the in Chapter 3.4.
|
||||
|
||||
</details>
|
||||
|
||||
- - ***Stability***: (Execution would be driven by the configuration in `weekly.json`).
|
||||
- - ***Stability***: `pytest -s -v tests/dfx/stability/scripts/test_{model_name}.py`
|
||||
- ***Reliability***: `pytest -s -v tests/e2e/reliability/test_{model_name}.py`
|
||||
|
||||
## Summary
|
||||
|
||||
Reference in new issue
Block a user