[Perf] Fix Qwen3-TTS latency regression (#3485)
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
12 files changed
+315
-85
No files matched your search
@@ -531,16 +531,18 @@ for result in response.json()["results"]:
|
||||
|
||||
All items are fanned out to `generate()` concurrently. The engine's stage worker automatically batches them up to the configured `max_batch_size` and queues the rest — no client-side throttling needed.
|
||||
|
||||
For best throughput, set both stages' `max_num_seqs` to ≥4 via `--stage-overrides`:
|
||||
For best throughput, set both stages' `max_num_seqs` above 1 via `--stage-overrides`. On the current Qwen3-TTS CustomVoice benchmark, stage 1 performed best at `max_num_seqs: 10`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||||
--omni --port 8091 --trust-remote-code --enforce-eager \
|
||||
--stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},
|
||||
"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}'
|
||||
--stage-overrides '{"0":{"max_num_seqs":10,"gpu_memory_utilization":0.2},
|
||||
"1":{"max_num_seqs":10,"gpu_memory_utilization":0.2}}'
|
||||
```
|
||||
|
||||
The bundled `qwen3_tts.yaml` uses `max_num_seqs: 1` (single request) on both stages. Bumping to 4 yields roughly 4× throughput on the talker and lets stage 1 batch chunks across in-flight requests.
|
||||
The bundled `qwen3_tts.yaml` uses a multi-request default and lets stage 1 batch chunks across in-flight requests. For latency-sensitive deployments, avoid forcing stage 1 back to `max_num_seqs: 1`; benchmark before reducing it below `10`.
|
||||
|
||||
The bundled config also sets `initial_codec_chunk_frames: 1`. This emits only the first audio chunk early for lower TTFA, then returns to the normal `codec_chunk_frames` window so Code2Wav does not repeatedly decode tiny overlapping chunks.
|
||||
|
||||
## Supported Models
|
||||
|
||||
|
||||
Reference in new issue
Block a user