[Bench] Add audio-streaming continuity metric for TTS (#3618)
Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
06ab723afc
commit
f99e8b2d3c
7 files changed
+350
-10
No files matched your search
@@ -40,7 +40,7 @@ vllm bench serve --omni \
|
||||
--num-prompts 20 --num-warmups 2 \
|
||||
--extra-body '{"task_type":"Base"}' \
|
||||
--max-concurrency 1 --request-rate inf \
|
||||
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration \
|
||||
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
|
||||
--save-result --result-dir ./results
|
||||
```
|
||||
|
||||
@@ -58,7 +58,7 @@ vllm bench serve --omni \
|
||||
--num-prompts 20 --num-warmups 2 \
|
||||
--extra-body '{"voice":"Vivian","language":"English","task_type":"CustomVoice"}' \
|
||||
--max-concurrency 1 --request-rate inf \
|
||||
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration \
|
||||
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
|
||||
--save-result --result-dir ./results
|
||||
```
|
||||
|
||||
@@ -76,10 +76,28 @@ vllm bench serve --omni \
|
||||
--num-prompts 20 --num-warmups 2 \
|
||||
--extra-body '{"task_type":"VoiceDesign","language":"English"}' \
|
||||
--max-concurrency 1 --request-rate inf \
|
||||
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration \
|
||||
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
|
||||
--save-result --result-dir ./results
|
||||
```
|
||||
|
||||
#### Streaming continuity (`audio_underrun`)
|
||||
|
||||
`audio_underrun` (seconds) is the per-request worst-case buffer deficit
|
||||
under a realtime-rate player simulation. The audio-speech backend records
|
||||
chunk arrival times during the SSE stream and surfaces:
|
||||
|
||||
- `Mean / Median / P{50,99} AUDIO_UNDERRUN (s)` per the standard percentile
|
||||
output - any positive value means at least one inter-chunk gap was longer
|
||||
than the chunk's audio duration.
|
||||
- `Streaming continuity OK rate` - fraction of requests whose worst-case
|
||||
underrun stayed under the threshold (default 100 ms, the commonly cited
|
||||
"audible gap" budget). Override the threshold with
|
||||
`VLLM_OMNI_BENCH_AUDIO_CONTINUITY_THRESHOLD_S=<float>`.
|
||||
|
||||
This captures the failure mode where `RTF_p50 < 1` (server keeps up in
|
||||
aggregate) but per-stream chunk arrival is bursty enough that listeners
|
||||
still hear gaps - common at high concurrency on streaming TTS pipelines.
|
||||
|
||||
#### Add WER / SIM / UTMOS to any of the above
|
||||
|
||||
Append `--seed-tts-wer-eval` (and optionally `SEED_TTS_EVAL_DEVICE=cuda:0`
|
||||
|
||||
@@ -118,7 +118,7 @@ def build_bench_args(
|
||||
"--num-warmups",
|
||||
"2",
|
||||
"--percentile-metrics",
|
||||
"ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
|
||||
"ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun",
|
||||
]
|
||||
|
||||
if resolved_dataset_path:
|
||||
|
||||
Reference in new issue
Block a user