273 lines
11 KiB
Markdown
273 lines
11 KiB
Markdown
# TTS Universal Benchmark
|
||
|
||
A model-agnostic serving benchmark for TTS models in vllm-omni. One CLI
|
||
(`bench_tts.py`) + one YAML registry (`model_configs.yaml`) drive perf and
|
||
quality runs for every registered checkpoint: **Qwen3-TTS** (Base / CustomVoice)
|
||
and **VoxCPM2** today, more to come.
|
||
|
||
The same three task types — `voice_clone`, `default_voice`, `voice_design` —
|
||
are wired into both the manual CLI and the DFX nightly CI matrix
|
||
(`tests/dfx/perf/tests/test_tts.json`).
|
||
|
||
## Quick start
|
||
|
||
### 1. Start the server
|
||
|
||
```bash
|
||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8000
|
||
```
|
||
|
||
The server auto-loads its Deploy YAML from `vllm_omni/deploy/qwen3_tts.yaml`
|
||
(Pipeline + Deploy schema introduced in #2383). No explicit `--deploy-config`
|
||
flag is needed for a registered model.
|
||
|
||
### 2. Run the benchmark (`vllm bench serve --omni`)
|
||
|
||
The primary, directly-controllable path. Copy-paste one of these and tweak
|
||
any bench flag (sampling params, endpoint, extra body, warmups, etc.):
|
||
|
||
#### voice_clone (Qwen3-TTS-Base, seed-tts dataset)
|
||
|
||
```bash
|
||
vllm bench serve --omni \
|
||
--host 127.0.0.1 --port 8000 \
|
||
--model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
|
||
--backend openai-audio-speech \
|
||
--endpoint /v1/audio/speech \
|
||
--dataset-name seed-tts \
|
||
--dataset-path /path/to/seed-tts-eval \
|
||
--seed-tts-locale en \
|
||
--num-prompts 20 --num-warmups 2 \
|
||
--extra-body '{"task_type":"Base"}' \
|
||
--max-concurrency 1 --request-rate inf \
|
||
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
|
||
--save-result --result-dir ./results
|
||
```
|
||
|
||
#### default_voice (Qwen3-TTS-CustomVoice, bundled seed_tts_smoke)
|
||
|
||
```bash
|
||
vllm bench serve --omni \
|
||
--host 127.0.0.1 --port 8000 \
|
||
--model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||
--backend openai-audio-speech \
|
||
--endpoint /v1/audio/speech \
|
||
--dataset-name seed-tts-text \
|
||
--dataset-path benchmarks/build_dataset/seed_tts_smoke \
|
||
--seed-tts-locale en \
|
||
--num-prompts 20 --num-warmups 2 \
|
||
--extra-body '{"voice":"Vivian","language":"English","task_type":"CustomVoice"}' \
|
||
--max-concurrency 1 --request-rate inf \
|
||
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
|
||
--save-result --result-dir ./results
|
||
```
|
||
|
||
#### voice_design (Qwen3-TTS-CustomVoice, bundled seed_tts_design)
|
||
|
||
```bash
|
||
vllm bench serve --omni \
|
||
--host 127.0.0.1 --port 8000 \
|
||
--model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||
--backend openai-audio-speech \
|
||
--endpoint /v1/audio/speech \
|
||
--dataset-name seed-tts-design \
|
||
--dataset-path benchmarks/build_dataset/seed_tts_design \
|
||
--seed-tts-locale en \
|
||
--num-prompts 20 --num-warmups 2 \
|
||
--extra-body '{"task_type":"VoiceDesign","language":"English"}' \
|
||
--max-concurrency 1 --request-rate inf \
|
||
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
|
||
--save-result --result-dir ./results
|
||
```
|
||
|
||
#### Streaming continuity (`audio_underrun`)
|
||
|
||
`audio_underrun` (seconds) is the per-request worst-case buffer deficit
|
||
under a realtime-rate player simulation. The audio-speech backend records
|
||
chunk arrival times during the SSE stream and surfaces:
|
||
|
||
- `Mean / Median / P{50,99} AUDIO_UNDERRUN (s)` per the standard percentile
|
||
output - any positive value means at least one inter-chunk gap was longer
|
||
than the chunk's audio duration.
|
||
- `Streaming continuity OK rate` - fraction of requests whose worst-case
|
||
underrun stayed under the threshold (default 100 ms, the commonly cited
|
||
"audible gap" budget). Override the threshold with
|
||
`VLLM_OMNI_BENCH_AUDIO_CONTINUITY_THRESHOLD_S=<float>`.
|
||
|
||
This captures the failure mode where `RTF_p50 < 1` (server keeps up in
|
||
aggregate) but per-stream chunk arrival is bursty enough that listeners
|
||
still hear gaps - common at high concurrency on streaming TTS pipelines.
|
||
|
||
#### Add WER / SIM / UTMOS to any of the above
|
||
|
||
Append `--seed-tts-wer-eval` (and optionally `SEED_TTS_EVAL_DEVICE=cuda:0`
|
||
in the env, per PR #2558). This triggers the seed-tts-eval protocol:
|
||
Whisper-large-v3 ASR → WER, WavLM embeddings → SIM, balacoon/utmos → UTMOS.
|
||
|
||
### 3. Convenience wrapper (`bench_tts.py`)
|
||
|
||
If you're running the **canonical** configuration for a registered model,
|
||
`bench_tts.py` loads the right defaults from `model_configs.yaml` and
|
||
emits the exact `vllm bench serve --omni` command above — useful for
|
||
concurrency sweeps and multi-task runs:
|
||
|
||
```bash
|
||
# Smallest smoke — 5 prompts, concurrency=1
|
||
python benchmarks/tts/bench_tts.py \
|
||
--model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
|
||
--task voice_clone \
|
||
--dataset-path /path/to/seed-tts-eval \
|
||
--concurrency 1 --num-prompts 5 \
|
||
--output-dir ./results
|
||
|
||
# Full concurrency sweep
|
||
python benchmarks/tts/bench_tts.py \
|
||
--model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
|
||
--task voice_clone \
|
||
--dataset-path /path/to/seed-tts-eval \
|
||
--concurrency 1 2 4 8 16 32 \
|
||
--num-prompts 20 \
|
||
--output-dir ./results
|
||
|
||
# With WER / SIM / UTMOS quality eval (adds ASR + embedding compute)
|
||
python benchmarks/tts/bench_tts.py \
|
||
--model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
|
||
--task voice_clone \
|
||
--dataset-path /path/to/seed-tts-eval \
|
||
--wer-eval \
|
||
--concurrency 4 --num-prompts 200 \
|
||
--output-dir ./results
|
||
```
|
||
|
||
#### Fixed IndexTTS 2.5 performance sweep
|
||
|
||
IndexTTS 2.5 is registered in `model_configs.yaml` like the other TTS models.
|
||
Use `--served-model-name` when the server was launched from a local native
|
||
bundle instead of the registry key. The command below reproduces the acceptance
|
||
workload: Seed-TTS Eval EN, `n=500`, concurrency `4/8/16/32`, five warmups, and
|
||
dataset seed 0.
|
||
|
||
```bash
|
||
python benchmarks/tts/bench_tts.py \
|
||
--model IndexTeam/IndexTTS-2.5 \
|
||
--served-model-name /path/to/indextts-2.5 \
|
||
--task voice_clone --locale en \
|
||
--dataset-path /path/to/seedtts_testset \
|
||
--host 127.0.0.1 --port 8092 \
|
||
--concurrency 4 8 16 32 \
|
||
--num-prompts 500 --num-warmups 5 \
|
||
--output-dir ./results/indextts2_5 \
|
||
-- \
|
||
--tokenizer /path/to/indextts-2.5/qwen0.6bemo4-merge \
|
||
--seed 0 --metric-percentiles 50,95,99 --disable-tqdm --save-detailed
|
||
```
|
||
|
||
The wrapper benchmarks an already-running server. For reproducible quality
|
||
comparisons, add `--request-seed 42`; production-performance runs should omit
|
||
it. The trailing `--seed 0` controls Seed-TTS row selection rather than model
|
||
sampling.
|
||
|
||
### 4. Plot a sweep
|
||
|
||
```bash
|
||
python benchmarks/tts/plot_results.py \
|
||
--results ./results/*.json \
|
||
--output ./results/curve.png
|
||
```
|
||
|
||
Outputs TTFP / RTF / throughput curves (and a markdown table) for every
|
||
`(task, concurrency)` combination in the result set.
|
||
|
||
## Task types
|
||
|
||
| Task | Dataset | Request body | Checkpoints that support it |
|
||
|-----------------|-------------------|-----------------------------------------------------|------------------------------------------|
|
||
| `voice_clone` | `seed-tts` | `ref_audio` + `ref_text` + `task_type=Base` | `Qwen3-TTS-*-Base`, `VoxCPM2` |
|
||
| `default_voice` | `seed-tts-text` | `voice=Vivian` + `task_type=CustomVoice` | `Qwen3-TTS-*-CustomVoice` |
|
||
| `voice_design` | `seed-tts-design` | `instructions=<natural-language description>` + `task_type=VoiceDesign` | `Qwen3-TTS-*-CustomVoice` |
|
||
|
||
**`-CustomVoice` checkpoints do NOT ship `speaker_encoder` weights**, so
|
||
voice_clone requests raise `ValueError` at model runtime. Use `-Base` for
|
||
voice_clone.
|
||
|
||
## Adding a new TTS model
|
||
|
||
Drop an entry into `model_configs.yaml` — no Python changes required:
|
||
|
||
```yaml
|
||
models:
|
||
<org>/<model-id>:
|
||
supported_tasks: [voice_clone] # or default_voice / voice_design
|
||
backend: openai-audio-speech # vllm bench serve backend
|
||
endpoint: /v1/audio/speech # OpenAI-compatible endpoint
|
||
task_extra_body: # merged into every request's body
|
||
voice_clone:
|
||
task_type: Base
|
||
```
|
||
|
||
Then add the model's Deploy YAML under `vllm_omni/deploy/<model>.yaml`
|
||
(Pipeline + Deploy schema) and it's immediately benchable.
|
||
|
||
## Datasets
|
||
|
||
| Dataset | Bundled? | Format | Source |
|
||
|--------------------|----------|-------------------|----------------------------------------------------------------|
|
||
| `seed-tts-design` | ✅ | 5-field meta.lst | `benchmarks/build_dataset/seed_tts_design/en/meta.lst` (20 prompts) |
|
||
| `seed_tts_smoke` | ✅ | 4-field meta.lst | `benchmarks/build_dataset/seed_tts_smoke/en/meta.lst` (20 text-only) |
|
||
| `seed-tts` | ❌ | 4-field meta.lst + WAVs | Google-Drive: [BytedanceSpeech/seed-tts-eval][seedtts] (~1.2 GB) |
|
||
| `seed-tts-text` | ❌ | 4-field meta.lst | Same archive as `seed-tts` (wav column unused) |
|
||
|
||
[seedtts]: https://github.com/BytedanceSpeech/seed-tts-eval
|
||
|
||
For manual voice_clone / default_voice runs against the full corpus, follow
|
||
`benchmarks/build_dataset/download_process_data_seedtts.md` and point
|
||
`--dataset-path` at the extracted `seedtts_testset` directory.
|
||
|
||
## DFX nightly CI
|
||
|
||
`tests/dfx/perf/tests/test_tts.json` wires three perf regimes plus quality:
|
||
|
||
| eval_phase | concurrency | purpose | Baseline metrics |
|
||
|---------------|-------------|---------------------------------------------------------|-----------------------------------------|
|
||
| `latency` | 1 | Single-request TTFP / RTF SLO | `median_audio_ttfp_ms`, `median_audio_rtf` |
|
||
| `throughput` | 8 | Codec-batching cliff sentinel (PDF #272 concurrency≥8) | `median_audio_ttfp_ms`, `median_audio_rtf` |
|
||
| `quality` | 4 | WER / SIM / UTMOS regression (disabled in CI by default)| `mean_audio_rtf` |
|
||
|
||
Why `median_*` for latency/throughput and `mean_*` for quality: latency
|
||
distributions have cold-start tails that drag the mean; quality aggregates
|
||
over 200 prompts so single-request outliers don't matter.
|
||
|
||
Quality entries are `enabled: false` in CI because seed-tts-eval is not
|
||
staged in the Buildkite container (matches the precedent in
|
||
PR #2558 — quality runs are manual / release-validation, not nightly).
|
||
|
||
## Concurrency cliff regression sentinel
|
||
|
||
Observed on H20-3e, Qwen3-TTS-1.7B (measured pre-merge on this branch):
|
||
|
||
| Task | Model | c=1 | c=4 | **c=8** | c=16 | c=32 |
|
||
|---------------|---------------|--------|--------|------------|--------|--------|
|
||
| voice_clone | 1.7B-Base | RTF 0.15 / TTFP 165ms | 0.28 / 412ms | **0.49 / 1701ms** | 0.72 / 3355ms | 0.77 / 3772ms |
|
||
| voice_design | 1.7B-CustomVoice | RTF 0.08 / TTFP 53ms | 0.11 / 154ms | **0.21 / 872ms** | 0.33 / 1801ms | 0.38 / 1989ms |
|
||
|
||
Both models show a **4–6× TTFP jump from c=4 to c=8** while audio throughput
|
||
saturates around c=4–8 — the codec-bs=1 bottleneck documented in
|
||
vllm-project/vllm-omni#272. The `throughput` CI regime at c=8 is the
|
||
sentinel for regressions in this area.
|
||
|
||
## File layout
|
||
|
||
```
|
||
benchmarks/tts/
|
||
├── README.md (this file)
|
||
├── bench_tts.py CLI — serve-mode benchmark driver
|
||
├── plot_results.py Generate per-task / per-concurrency curves
|
||
└── model_configs.yaml Model registry (supported tasks + extra body)
|
||
```
|
||
|
||
## Related
|
||
|
||
- Upstream seed-tts-eval integration: vllm-project/vllm-omni#2558
|
||
- Pipeline + Deploy schema: vllm-project/vllm-omni#2383
|
||
- Concurrency cliff RFC: vllm-project/vllm-omni#272
|