anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Model] Optimize Qwen3-TTS streaming throughput and first audio on CUDA (#8163)

Browse Source 
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
Sy03 authored and GitHub committed 2026-09-26 22:51:04 +08:00
1 parent 8a49c65bf9
commit e1303e5067
55 files changed
+4222 -230
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/configuration/stage_configs.md
+64 -9
View File
@@ -337,6 +337,41 @@ supply `mtp_sampling_params` and `get_mtp_seed(sampling_params)` for model-local
(with `mtp_sample_steps` and `mtp_sample_vocab_size`) as needed. Qwen3-TTS retains
its existing `talker_mtp` entry point for V1.
### Qwen3-TTS first-frame delivery and rollback
The standard `qwen3_tts.yaml`, `qwen3_tts_high_concurrency.yaml`,
`qwen3_tts_mrv2.yaml` and `qwen3_tts_high_concurrency_mrv2.yaml` profiles
enable the `talker_first_audio` connector option, as does the experimental
single-GPU profile. This changes the default CUDA MRV2 streaming path:
residual prediction runs eagerly after the Talker sample, the Talker loads
an additional first-frame decoder with its weights and CUDA graphs, and the
orchestrator orders its audio before subsequent Code2Wav chunks.
Code2Wav retains full-audio prefix graphs alongside the optional state-only
graphs. Each request's delivery marker selects the graph and audio trimming;
enabling the option alone never suppresses audio or disables prefix batching
for requests that retain regular codec delivery.
The path requires asynchronous chunks, TP/PP 1, an in-process executor and
disabled prefix caching. Requests with reference codes and unsupported
runners or platforms retain regular Code2Wav delivery.
To restore regular codec delivery and avoid loading the Talker's additional
decoder, use a deploy overlay with the relevant base profile:
```yaml
base_config: qwen3_tts.yaml
connectors:
connector_of_shared_memory:
extra:
talker_first_audio: false
```
First-packet latency measures when PCM starts arriving. Time to first audible
audio (TTFA) also includes any leading silence in the generated audio. An
earlier first packet therefore does not necessarily improve TTFA; measure
both for the intended voice and workload.
### Included performance work
- Request snapshots have a fast path for immutable scalar leaves, including
@@ -372,14 +407,34 @@ speaker similarity before adopting either batching preset for a production
workload. Floating-point decoder outputs can differ across batch sizes; this PR
does not claim bitwise or quality equivalence.
### Optional MPS deployment
### Experimental MPS deployment
NVIDIA MPS is an optional operator setting for colocated CUDA processes, not a
YAML option or a library default. This PR does not establish a throughput or
first-packet latency benefit from MPS. Measure the exact deployment with and
without MPS before enabling it.
NVIDIA MPS lets colocated CUDA stage processes share GPU execution resources.
It is disabled by default. The experimental
`qwen3_tts_high_concurrency_mrv2_single_gpu.yaml` profile sets `cuda_mps: true`,
alongside cached residual prediction, fused sampling, time-major codec
convolutions, first-frame delivery, and larger graph batches.
Use only assigned GPUs and an independent MPS pipe directory. A private MPS
server does not provide exclusive GPU ownership or MIG isolation. For a
single-GPU deployment, explicitly place both stages on that GPU; the supplied
high-concurrency profile places its two stages on different GPUs by default.
The runtime requires `nvidia-cuda-mps-control` on `PATH` and one explicit CUDA
GPU per local EngineCore stage, with `parallel_stage_init: false`.
Use numeric GPU ordinals for stage placement and visibility so initialization
locks identify the physical GPU before MPS remaps it. It starts a private MPS daemon for each selected
GPU and stops its own daemon after the stages exit. If
`CUDA_MPS_PIPE_DIRECTORY` already names an operator-managed daemon, the runtime
reuses it without stopping it. Diffusion and remote stages are unsupported.
Set `cuda_mps: false` in a deploy overlay to disable automatic MPS management.
Stage `env.CUDA_MPS_PIPE_DIRECTORY` overrides the parent setting when selecting
the daemon; an explicit empty string selects a private daemon. Stages sharing
a GPU reuse the first stage's daemon. Later stages may omit the setting or
explicitly match that policy; a conflicting explicit setting fails before
that stage starts. The parent environment is unchanged.
MPS can improve throughput under concurrent load while increasing first-packet
or first-audible-audio latency, particularly at low request rates. For a
latency-sensitive workload, compare the same profile with `cuda_mps: false`.
An inherited operator-managed MPS daemon must also be disabled by its owner
for that comparison; this switch only controls automatic MPS management.
MPS does not reserve a GPU. Use only assigned GPUs, explicitly place stages on
the intended GPU, and warm the complete pipeline before measuring performance.
docs/design/qwen3_omni_tts_performance_optimization.md
+68 -7
View File
@@ -93,13 +93,13 @@ Each optimization stacks on the previous one. The summary plots below show the c
**Benchmark environment:**
| | Qwen3-Omni | Qwen3-TTS |
| --- |-----------------------------| --- |
| **GPU** | A100 | H200 |
| | Qwen3-Omni | Qwen3-TTS |
| --- | ----------------------------- | --- |
| **GPU** | A100 | H200 |
| **Model** | Qwen3-Omni-30B-A3B-Instruct | Qwen3-TTS-12Hz-1.7B-CustomVoice |
| **vLLM** | v0.17.0 | v0.18.0 |
| **vllm-omni** | commit 199f7832 | v0.18.0rc2 |
| **CUDA** | 12.9 | 12.8 |
| **vLLM** | v0.17.0 | v0.18.0 |
| **vllm-omni** | commit 199f7832 | v0.18.0rc2 |
| **CUDA** | 12.9 | 12.8 |
This post walks through each optimization in the same order they are typically enabled in practice, then ends with deployment playbooks for both models.
@@ -310,7 +310,68 @@ self._compiled_model_fwd = torch.compile(
`mode="default"` is used instead of `mode="reduce-overhead"` to avoid conflicts with vLLM's own CUDA graph capture on the main Talker model. `dynamic=True` handles the growing sequence length without recompilation.
These optimizations are always-on in the current codebase - all Qwen3-TTS benchmark results in this post include them.
The historical benchmark results above use re-prefill. It remains the default
for Qwen3-TTS and the shared Qwen3-Omni predictor.
### Experimental Qwen3-TTS single-GPU throughput profile
The opt-in `qwen3_tts_high_concurrency_mrv2_single_gpu.yaml` profile uses a
Qwen3-TTS-specific execution path on CUDA with BF16 weights:
- `code_predictor_kv_cache` reuses each layer's keys and values across the 15
residual-codebook predictions within one audio frame. The first step writes
two positions; subsequent steps write one. Scratch storage is bounded by the
configured batch capacity, overwritten for every frame, and never shared
between requests across frames. The worker's serialized CUDA stream owns it;
it is separate from the Talker's paged KV cache.
- A short causal/GQA Triton kernel masks unwritten K/V loads. Dynamic-batch
`torch.compile` warms the predictor before the runner captures whole-MTP
CUDA graphs. This path creates no inner CUDA graphs. Unsupported devices and
dtypes retain re-prefill; FP16 retains its existing precision safeguards.
- `code_predictor_fused_sampling` fuses temperature scaling, top-k filtering,
and Gumbel argmax when the runner supplies FP32 uniforms. It preserves top-k
ties and uses the supplied randomness. Explicit generator paths retain the
existing sampler. Qwen3-Omni does not read either TTS option.
- `decode_cudnn_benchmark` selects convolution algorithms during Code2Wav
warmup and graph capture. The process's cuDNN flags are restored afterwards,
including on capture failure. Captured graphs retain the selected algorithms.
- `decode_time_major_conv` keeps codec convolution activations in time-major
layout to reduce transposes and copies between layers.
- `talker_first_audio` completes residual prediction immediately after the
first codebook sample and decodes the first frame in the Talker process.
The codec advances its state without repeating the delivered frame; the
orchestrator orders that frame before subsequent codec chunks. This path
requires CUDA, MRV2, asynchronous chunks, a single-process executor, TP/PP 1,
and disabled prefix caching. Reference-code requests retain codec delivery.
Supported Qwen3-TTS profiles enable it explicitly; other models do not opt in.
The predictor, sampling, convolution-layout and autotuning options default to
false outside this throughput profile. It uses fixed 1/25-frame chunks,
72 frames of left context, Talker/codec capacities of 128/64, codec batches
through 8, and synchronous codec scheduling. Client concurrency is independent
of those capacities. This profile also sets `cuda_mps: true`; the runtime owns
a private MPS daemon unless an operator supplies an existing MPS pipe directory.
Ordinary profiles leave MPS disabled. See
[experimental MPS deployment](../configuration/stage_configs.md#experimental-mps-deployment)
for requirements and ownership. Startup includes compilation and graph capture.
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
--deploy-config vllm_omni/deploy/qwen3_tts_high_concurrency_mrv2_single_gpu.yaml \
--api-server-count 4 --stage-init-timeout 1800 --init-timeout 2400
```
KV reuse changes GEMM shapes and attention reduction order; cuDNN autotuning
can also change waveform numerics. This path is not bitwise equivalent to
re-prefill, and aggregate WER alone cannot establish perceptual or speaker
similarity equivalence. Validate quality on the intended workload before
adopting it. Qwen3-Omni keeps its existing predictor and audio-delivery behavior.
For a controlled ablation, copy the profile and disable individual options while keeping capacities, chunking, hardware, client concurrency,
and warmup identical. Report repeated runs, first-audio latency and failures
alongside throughput. Warm requests measure a ready service; they do not measure
cold-start latency.
---