[Bench] Add audio-streaming continuity metric for TTS (#3618)

Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Yueqian LinandHongsheng Liu authored and GitHub committed 2026-05-20 05:08:56 +00:00
1 parent 06ab723afc
commit f99e8b2d3c
7 files changed
+350 -10

No files matched your search

+21 -3
View File
@@ -40,7 +40,7 @@ vllm bench serve --omni \
--num-prompts 20 --num-warmups 2 \
--extra-body '{"task_type":"Base"}' \
--max-concurrency 1 --request-rate inf \
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration \
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
--save-result --result-dir ./results
```
@@ -58,7 +58,7 @@ vllm bench serve --omni \
--num-prompts 20 --num-warmups 2 \
--extra-body '{"voice":"Vivian","language":"English","task_type":"CustomVoice"}' \
--max-concurrency 1 --request-rate inf \
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration \
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
--save-result --result-dir ./results
```
@@ -76,10 +76,28 @@ vllm bench serve --omni \
--num-prompts 20 --num-warmups 2 \
--extra-body '{"task_type":"VoiceDesign","language":"English"}' \
--max-concurrency 1 --request-rate inf \
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration \
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
--save-result --result-dir ./results
```
#### Streaming continuity (`audio_underrun`)
`audio_underrun` (seconds) is the per-request worst-case buffer deficit
under a realtime-rate player simulation. The audio-speech backend records
chunk arrival times during the SSE stream and surfaces:
- `Mean / Median / P{50,99} AUDIO_UNDERRUN (s)` per the standard percentile
output - any positive value means at least one inter-chunk gap was longer
than the chunk's audio duration.
- `Streaming continuity OK rate` - fraction of requests whose worst-case
underrun stayed under the threshold (default 100 ms, the commonly cited
"audible gap" budget). Override the threshold with
`VLLM_OMNI_BENCH_AUDIO_CONTINUITY_THRESHOLD_S=<float>`.
This captures the failure mode where `RTF_p50 < 1` (server keeps up in
aggregate) but per-stream chunk arrival is bursty enough that listeners
still hear gaps - common at high concurrency on streaming TTS pipelines.
#### Add WER / SIM / UTMOS to any of the above
Append `--seed-tts-wer-eval` (and optionally `SEED_TTS_EVAL_DEVICE=cuda:0`
+1 -1
View File
@@ -118,7 +118,7 @@ def build_bench_args(
"--num-warmups",
"2",
"--percentile-metrics",
"ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
"ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun",
]
if resolved_dataset_path: