[Model] Add Breeze-TTS-2 two-stage AR TTS support (talker + codec, streaming) (#7084)
Signed-off-by: noname <liuzhiwei08@kuaishou.com> Signed-off-by: noname <133508552+lzwnoname@users.noreply.github.com> Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: noname <liuzhiwei08@kuaishou.com> Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
46 files changed
+6067
-4
No files matched your search
@@ -13,7 +13,8 @@ For the full list of supported architectures across all modalities, see
|
||||
## Supported Models
|
||||
|
||||
| Model | HuggingFace repo | Voice cloning | Streaming | Voice presets / upload | Gradio demo |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
|---|---|---|---|---|---|
|
||||
| Breeze-TTS-2 | `BreezeBlue/Breeze-TTS-2` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | speaker tags (`S0`..`S9`, default `S0`) | — |
|
||||
| Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | uploaded audio voice only; no presets | ✓ |
|
||||
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
|
||||
| GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
|
||||
@@ -1025,3 +1026,63 @@ The demo handles voice-preset selection and reference-audio upload. `voxtral_tts
|
||||
- Voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`).
|
||||
- Voice cloning is gated upstream and may require a recent `mistral_common`.
|
||||
- A standalone CLI client is not yet shipped; the gradio demo is the canonical reference for now.
|
||||
|
||||
## Breeze-TTS-2
|
||||
|
||||
Two-stage AR TTS (T5Gemma2 + Qwen3 talker with a depth decoder → bundled Qwen3-TTS codec) at 24 kHz mono, from [BreezeBlue](https://huggingface.co/BreezeBlue/Breeze-TTS-2). Four modes are selected automatically from the request fields: plain (`input`), voice design (`instructions`), clone (`ref_audio`+`ref_text`), and voice direction (reference trio + `instructions`).
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
vllm serve BreezeBlue/Breeze-TTS-2 --omni --port 8091
|
||||
# or:
|
||||
./breeze_tts_2/run_server.sh
|
||||
```
|
||||
|
||||
The deploy config at `vllm_omni/deploy/breeze_tts_2.yaml` auto-loads (async-chunk streaming, 8-frame inter-stage chunks).
|
||||
|
||||
### Sending requests
|
||||
|
||||
```bash
|
||||
# Plain synthesis (speaker tags S0..S9, default S0)
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is Breeze TTS 2 on vLLM-Omni.",
|
||||
"voice": "S0",
|
||||
"response_format": "wav",
|
||||
"sample_rate": 24000
|
||||
}' --output breeze_plain.wav
|
||||
|
||||
# Voice direction (clone a reference, then steer the delivery)
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "We need to discuss what happened last night.",
|
||||
"ref_audio": "file:///path/to/reference.wav",
|
||||
"ref_text": "The exact transcript of the reference audio.",
|
||||
"instructions": "Speak slowly with a restrained, serious tone.",
|
||||
"response_format": "wav"
|
||||
}' --output breeze_direction.wav
|
||||
```
|
||||
|
||||
### Streaming PCM
|
||||
|
||||
```bash
|
||||
curl -N -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Streaming output from Breeze TTS 2.",
|
||||
"stream": true,
|
||||
"stream_format": "audio",
|
||||
"response_format": "pcm",
|
||||
"sample_rate": 24000
|
||||
}' --output breeze_stream.pcm
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Only `sample_rate=24000` is accepted; `speed` must be `1.0`.
|
||||
- `guidance_scale`/`cfg_scale` must be `1.0`; `negative_prompt` is rejected (CFG companion support is a follow-up).
|
||||
- `instructions` without `ref_audio` is voice design; with `ref_audio`+`ref_text` it is voice direction.
|
||||
- See [`recipes/BreezeBlue/Breeze-TTS-2.md`](../../../recipes/BreezeBlue/Breeze-TTS-2.md) for the full recipe and the offline example under [`examples/offline_inference/text_to_speech/breeze_tts_2/`](../../offline_inference/text_to_speech/breeze_tts_2/).
|
||||
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
# Launch Breeze-TTS-2 online serving with the async-chunk deploy config.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
MODEL="${1:-BreezeBlue/Breeze-TTS-2}"
|
||||
PORT="${PORT:-8091}"
|
||||
|
||||
exec vllm-omni serve "${MODEL}" \
|
||||
--deploy-config vllm_omni/deploy/breeze_tts_2.yaml \
|
||||
--omni --port "${PORT}"
|
||||
Reference in new issue
Block a user