[Bugfix] Fix video benchmark timeout and failure accounting (#7259)
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
This commit is contained in:
1 parent
b58ff5cb8b
commit
031612f250
3 files changed
+120
-73
No files matched your search
@@ -17,18 +17,18 @@ The main entrypoint is:
|
||||
vllm serve Qwen/Qwen-Image --omni --port 8099
|
||||
```
|
||||
|
||||
2. Run a minimal benchmark:
|
||||
1. Run a minimal benchmark:
|
||||
|
||||
```bash
|
||||
python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
|
||||
--base-url http://localhost:8099 \
|
||||
--model Qwen/Qwen-Image \
|
||||
--task t2i \
|
||||
--dataset vbench \
|
||||
--num-prompts 5
|
||||
--base-url http://localhost:8099 \
|
||||
--model Qwen/Qwen-Image \
|
||||
--task t2i \
|
||||
--dataset vbench \
|
||||
--num-prompts 5
|
||||
```
|
||||
|
||||
**Notes**
|
||||
### Notes
|
||||
|
||||
- By default, image tasks talk to `http://<host>:<port>/v1/chat/completions`; video tasks talk to `/v1/videos`.
|
||||
- If you run the server on another host or port, pass `--base-url` accordingly.
|
||||
@@ -52,14 +52,14 @@ Example (`t2v`):
|
||||
|
||||
```bash
|
||||
python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
|
||||
--base-url http://localhost:8099 \
|
||||
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--task t2v \
|
||||
--dataset vbench \
|
||||
--num-prompts 50 \
|
||||
--width 640 --height 480 \
|
||||
--num-frames 81 --fps 16 \
|
||||
--num-inference-steps 40
|
||||
--base-url http://localhost:8099 \
|
||||
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--task t2v \
|
||||
--dataset vbench \
|
||||
--num-prompts 50 \
|
||||
--width 640 --height 480 \
|
||||
--num-frames 81 --fps 16 \
|
||||
--num-inference-steps 40
|
||||
```
|
||||
|
||||
Note: `vbench` can also be used for other tasks such as `t2i` / `i2v` (and `i2i`). For `t2i`, the loader reuses VBench t2v text prompts; for `i2v` / `i2i`, it loads the VBench i2v dataset (with image paths).
|
||||
@@ -140,6 +140,21 @@ Traffic / concurrency flags:
|
||||
- `--request-rate`: Target request rate (requests/second). If set to `inf`, the script sends all requests immediately.
|
||||
- `--max-concurrency`: Max number of in-flight requests (default: `1`). This can hard-cap the achieved QPS: if it is too small, requests will queue behind the semaphore, and both achieved throughput and observed SLO attainment can be skewed.
|
||||
|
||||
### Video job timeout and failures
|
||||
|
||||
`--video-job-timeout` controls how long each `/v1/videos` job may be polled,
|
||||
including time queued on the server (default: `900` seconds). For long video
|
||||
jobs at high concurrency, use a larger budget, for example
|
||||
`--video-job-timeout 1800`. This also applies to warmup jobs. The polling budget
|
||||
starts after job creation; it excludes time waiting for the client concurrency
|
||||
semaphore. Individual HTTP calls still use the aiohttp session timeout.
|
||||
|
||||
An expired job is counted as failed and deleted, which cancels queued or running
|
||||
server generation. The benchmark continues processing the remaining requests.
|
||||
The progress bar counts both successful and failed requests. The final report
|
||||
shows the failure count and the first 10 errors; `--output-file` saves all errors
|
||||
with their request IDs in `request_errors`.
|
||||
|
||||
### Batched warmup note
|
||||
|
||||
For batched serving runs, warm the same in-flight shape you plan to measure.
|
||||
|
||||
Reference in new issue
Block a user