[Model] Add single-stage Qwen3-TTS with in-Talker streaming codec decode (#8259)
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
43 files changed
+3565
-90
No files matched your search
@@ -319,7 +319,9 @@ ROCm and MUSA. Select one of these deployment profiles:
|
||||
| `qwen3_tts_mrv2.yaml` | V2 | B1 | Explicit MRV2 profile (same runner selection as the default) |
|
||||
| `qwen3_tts_high_concurrency_mrv2.yaml` | V2 | B1, B2 | Opt-in throughput tuning |
|
||||
| `qwen3_tts_high_concurrency_mrv2_b4.yaml` | V2 | B1, B2, B3, B4 | Experimental throughput / buffered playback |
|
||||
| `qwen3_tts_high_concurrency_mrv2_single_gpu.yaml` | V2 | B1–B8 | Experimental two-stage deployment on one GPU, with MPS |
|
||||
| `qwen3_tts_high_concurrency.yaml` | V1 | Existing defaults | V1 high-concurrency control |
|
||||
| `qwen3_tts_fused_single_gpu.yaml` | V2 | N/A (in-Talker decoder) | Opt-in single-stage CUDA pipeline on one GPU |
|
||||
|
||||
```bash
|
||||
# MRV2 is the default; pass a copy with `model_runner: v1` to force V1.
|
||||
@@ -350,12 +352,73 @@ supply `mtp_sampling_params` and `get_mtp_seed(sampling_params)` for model-local
|
||||
(with `mtp_sample_steps` and `mtp_sample_vocab_size`) as needed. Qwen3-TTS retains
|
||||
its existing `talker_mtp` entry point for V1.
|
||||
|
||||
### Selecting one or two Qwen3-TTS stages
|
||||
|
||||
Choose the deployment profile explicitly with `--deploy-config`. The default
|
||||
`qwen3_tts` pipeline retains two stages: the Talker publishes codec chunks to
|
||||
Code2Wav, which produces audio. Its stage devices can be placed on the same
|
||||
GPU or on separate GPUs. `qwen3_tts_fused` is a separate, opt-in pipeline for
|
||||
one CUDA GPU: the Talker owns the stateful codec decoder and emits PCM directly,
|
||||
without a Code2Wav engine or stage connector.
|
||||
|
||||
```bash
|
||||
# Two stages on one GPU; the two-stage pipeline remains independently selectable.
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
|
||||
--deploy-config vllm_omni/deploy/qwen3_tts_high_concurrency_mrv2_single_gpu.yaml
|
||||
|
||||
# One stage on one GPU; no inter-stage transport or Code2Wav scheduling.
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
|
||||
--deploy-config vllm_omni/deploy/qwen3_tts_fused_single_gpu.yaml
|
||||
```
|
||||
|
||||
The single-stage profile requires CUDA, V2, asynchronous chunks, TP=PP=1,
|
||||
an in-process worker and disabled prefix caching. Select the full profile;
|
||||
adding `talker_stream_decode: true` to a two-stage Talker is rejected because
|
||||
its latent output belongs to the Code2Wav input contract. Single-stage options
|
||||
live in `stages[].additional_config`; two-stage transport and first-frame
|
||||
options live in `connectors.*.extra`.
|
||||
|
||||
The single-stage profile defaults `talker_stream_first_audio` to `false` for
|
||||
throughput. To opt into a separate first-frame delivery queue, save this
|
||||
overlay beside the bundled deployment files and pass its path to
|
||||
`--deploy-config`:
|
||||
|
||||
```yaml
|
||||
base_config: qwen3_tts_fused_single_gpu.yaml
|
||||
stages:
|
||||
- stage_id: 0
|
||||
additional_config:
|
||||
talker_stream_decode: true
|
||||
talker_stream_first_audio: true
|
||||
ref_code_context_frames: 72
|
||||
code_predictor_kv_cache: true
|
||||
code_predictor_fused_sampling: true
|
||||
code_predictor_fused: true
|
||||
```
|
||||
|
||||
This queue uses PCM from the single-stage decoder. The two-stage
|
||||
`talker_first_audio` option below uses its own first-frame decoder and does
|
||||
not require single-stage execution. The two options cannot be enabled
|
||||
together. Single-stage prefix-cache reset with
|
||||
`reset_running_requests=True` returns `False` while requests are active;
|
||||
wait for completion or abort them before retrying.
|
||||
|
||||
Reference-cloning context is selected independently by each profile. The
|
||||
single-stage profile explicitly uses 72 reference frames, matching the
|
||||
`qwen3_tts.yaml` context boundary. A shorter
|
||||
`additional_config.ref_code_context_frames` (for example 25) can change
|
||||
timbre continuity even when WER is similar;
|
||||
the throughput measurements for CustomVoice do not qualify Base voice
|
||||
cloning quality. Reference layout, grouping and priming are model hooks;
|
||||
the shared worker does not select Qwen's context length or codebook layout.
|
||||
|
||||
### Qwen3-TTS first-frame delivery and rollback
|
||||
|
||||
The standard `qwen3_tts.yaml`, `qwen3_tts_high_concurrency.yaml`,
|
||||
`qwen3_tts_mrv2.yaml` and `qwen3_tts_high_concurrency_mrv2.yaml` profiles
|
||||
enable the `talker_first_audio` connector option, as does the experimental
|
||||
single-GPU profile. This changes the default CUDA MRV2 streaming path:
|
||||
two-stage `qwen3_tts_high_concurrency_mrv2_single_gpu.yaml` profile.
|
||||
This changes the default CUDA MRV2 streaming path:
|
||||
residual prediction runs eagerly after the Talker sample, the Talker loads
|
||||
an additional first-frame decoder with its weights and CUDA graphs, and the
|
||||
orchestrator orders its audio before subsequent Code2Wav chunks.
|
||||
|
||||
Reference in new issue
Block a user