vllm-omni/recipes/BreezeBlue/Breeze-TTS-2.md

# Breeze-TTS-2

> Text-to-speech serving (plain / voice design / voice clone / voice direction)

## Summary

- Vendor: BreezeBlue
- Model: `BreezeBlue/Breeze-TTS-2` (`BreezeForConditionalGeneration`)
- Task: Text-to-speech with speaker tags, natural-language style instructions, single-reference voice cloning, and reference+instruction "voice direction"
- Mode: Online serving with the OpenAI-compatible `/v1/audio/speech` API (streaming and non-streaming); offline `Omni` example
- Hardware: 1x NVIDIA L20 48GB (CUDA)
- Maintainer: Community

## When to use this recipe

Use this recipe when you want a known-good starting point for serving
Breeze-TTS-2 with vLLM-Omni. Breeze-TTS-2 is a two-stage AR TTS pipeline:

| Stage | Components | Output |
| ----- | ---------- | ------ |
| 0 (talker, `LLM_AR`) | T5Gemma2 text encoder + Qwen3 backbone + 2052-way codebook-0 head + 12-layer depth decoder | 16-codebook codec frames |
| 1 (codec, `LLM_GENERATION`) | Bundled Qwen3-TTS audio tokenizer decoder (Mimi fallback) | 24 kHz mono waveform |

Four prompt modes are selected automatically from the request fields:

| Mode | Trigger | Fields |
| ---- | ------- | ------ |
| `tts_plain` | only `input` (default `voice=S0`) | `input`, `voice` |
| `tts_instruction` (voice design) | `input` + `instructions` | style/emotion/pace directives, not spoken |
| `ref_clone_tata` (clone) | `ref_audio` + `ref_text` + `input` | reference clip and its exact transcript |
| `ref_edit_tata` (voice direction) | reference trio + `instructions` | keep the reference timbre, direct the delivery |

Current scope: greedy sampling with `cfg_scale=1.0` and a default
`repetition_penalty=1.1` on generated codebook-0 tokens; CFG ≠ 1.0 and
`negative_prompt` are rejected with explicit client errors until companion
request support lands.

The [official inference example](https://github.com/breezeblue-ai/breeze-tts/blob/main/infer.py)
also uses repetition penalty `1.1`, but its
[runtime defaults](https://github.com/breezeblue-ai/breeze-tts/blob/main/breeze_infer/runtime.py)
sample both the backbone and depth decoder with temperature `0.9` and top-k
`50`. The [official voice-design examples](https://huggingface.co/BreezeBlue/Breeze-TTS-2#-voice-design)
use CFG `4`. Those sampling and guidance settings are outside the current
port's scope, so the examples below do not establish parity with the official
voice-design outputs.

## Supported model contract

| Item | Value |
| ---- | ----- |
| Tasks | Text-to-speech through `/v1/audio/speech` (online) and the offline `Omni` example |
| Modes | plain, voice design (`instructions`), voice clone (`ref_audio` + `ref_text`), voice direction (reference + `instructions`) |
| Languages | English and Chinese (model card) |
| Reference audio | exactly one clip per request with its exact transcript; encoded by the bundled Qwen3-TTS tokenizer to 16 codebooks at 12.5 Hz |
| Output | 24 kHz mono; `wav` or `pcm`; raw audio streaming (`stream_format="audio"`) or SSE `speech.audio.delta` (`stream_format="sse"`) |
| Length | prompt bounded by stage-0 `max_model_len=4096`; synthesis bounded by `max_new_tokens` codec frames (default 2048, about 164 s) |
| Sampling | greedy (`temperature=0`) with `cfg_scale=1.0` and `repetition_penalty=1.1`; other guidance values and `negative_prompt` are rejected |
| Deployment profile | `vllm_omni/deploy/breeze_tts_2.yaml`: two stages on one GPU, async-chunk streaming with 8-frame codec chunks, stage 0 `gpu_memory_utilization=0.80`, stage 1 `0.15` |

## References

- Issue: [#6656 [New Model]: Breeze TTS 2](https://github.com/vllm-project/vllm-omni/issues/6656)
- Related examples under `examples/`:
  [`examples/online_serving/text_to_speech/breeze_tts_2/`](../../examples/online_serving/text_to_speech/breeze_tts_2/),
  [`examples/offline_inference/text_to_speech/breeze_tts_2/`](../../examples/offline_inference/text_to_speech/breeze_tts_2/)

## Hardware

- Accelerator model and per-device memory: 1x NVIDIA L20 48GB
- Number of devices: 1 (both stages are placed on device 0)
- Device interconnect: not applicable (single device)
- Host memory: no special requirement; per-request codec state is small
- Qualification scope: functional serving of all four prompt modes, streaming PCM, and 4 concurrent mixed-mode requests (greedy, `cfg_scale=1.0`). No other accelerator is qualified by this recipe.

## Software environment

- OS: Linux
- Python: 3.12.12
- Driver / runtime: NVIDIA driver 580.82.07, CUDA 13.0 (PyTorch 2.13.0+cu130)
- vLLM version: 0.28.0
- vLLM-Omni version or commit: the commit that introduced this recipe (branch based on `main` @ `67ef0b64`)
- Transformers: 5.14.1

## Command

Start the server from the repository root. The deploy config
(`vllm_omni/deploy/breeze_tts_2.yaml`, async chunk streaming by default)
auto-loads from the checkpoint's `model_type`; pass it explicitly if you
customized it:

```bash
vllm serve BreezeBlue/Breeze-TTS-2 \
    --deploy-config vllm_omni/deploy/breeze_tts_2.yaml \
    --omni --port 8091
```

## Verification

Plain synthesis (speaker tag `S0`..`S9`, default `S0`):

```bash
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "model": "BreezeBlue/Breeze-TTS-2",
        "input": "Hello, this is Breeze TTS 2 running on vLLM-Omni.",
        "voice": "S0",
        "response_format": "wav",
        "sample_rate": 24000
    }' --output breeze_plain.wav
```

Voice design (instruction only, no reference audio):

```bash
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "(laugh) Welcome to our story time.",
        "instructions": "A warm young woman, clear voice, lively delivery.",
        "response_format": "wav"
    }' --output breeze_design.wav
```

Voice cloning (`ref_audio` and `ref_text` must be provided together):

```bash
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "This is new text spoken in the cloned voice.",
        "ref_audio": "file:///path/to/reference.wav",
        "ref_text": "The exact transcript of the reference audio.",
        "response_format": "wav"
    }' --output breeze_clone.wav
```

Voice direction (reference + instruction):

```bash
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "We need to discuss what happened last night.",
        "ref_audio": "file:///path/to/reference.wav",
        "ref_text": "The exact transcript of the reference audio.",
        "instructions": "Speak slowly with a restrained, serious tone.",
        "response_format": "wav"
    }' --output breeze_direction.wav
```

Streaming raw PCM (`stream_format="audio"`):

```bash
curl -N -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Streaming output from Breeze TTS 2.",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm",
        "sample_rate": 24000
    }' --output breeze_stream.pcm
```

Expected results: finite, non-silent 24 kHz mono WAV files; the streaming example
returns raw PCM bytes. Use `stream_format="sse"` for `speech.audio.delta`
events containing base64-encoded audio. On 1x L20 (greedy, single request after
warm-up) a few seconds of speech completes in roughly 3.4–3.8 s wall time.

## Notes

- **Sample rate**: only `24000` is accepted; other values fail validation
  before inference.
- **Speed / seeds / multi-reference**: `speed` must be `1.0`;
  `task_type=VoiceDesign`, `ref_audio_2`, and speaker embeddings are not
  supported yet.
- **CFG**: `guidance_scale`/`cfg_scale` must be `1.0`. Non-1.0 values and
  `negative_prompt` return a client error.
- **Repetition penalty**: the default `1.1` matches upstream's penalty on
  generated codebook-0 tokens and discourages repeated silence. Override it
  with `extra_params: {"repetition_penalty": 1.0}` to disable it, or another
  finite positive value. Text and reference audio are excluded from its history.
  The legacy `tools/breeze_tts_2_upstream_golden.py` uses penalty `1.0`; set
  the same serving override when comparing against those golden frames.
- **Length budget**: prompt length is bounded by stage 0 `max_model_len=4096`;
  synthesis length by `max_new_tokens` (default 2048 frames ≈ 164 s at the
  12.5 Hz codec frame rate) — set `max_new_tokens` for shorter caps.
- **Checkpoint layout**: the checkpoint must contain the `audio_tokenizer/`
  subdirectory (bundled Qwen3-TTS codec) for the default streaming path; the
  Mimi fallback is used only when that directory is absent and does not
  support async-chunk streaming.
- **License**: the reference inference code is Apache 2.0; the checkpoints are
  distributed under the separate BreezeBlue Research and Non-Commercial
  License. Commercial use requires written authorization from BreezeBlue.

## Supported features

| Feature | Status | Notes |
| ------- | ------ | ----- |
| Streaming output | ✓ | `async_chunk: true`, 8-frame inter-stage chunks; raw audio or SSE `speech.audio.delta` |
| Voice cloning | ✓ | one reference clip plus transcript per request |
| Voice design / voice direction | ✓ | `instructions` field, with or without a reference |
| Classifier-free guidance | ✗ | `cfg_scale` must be `1.0`; `negative_prompt` rejected (follow-up) |
| Non-greedy sampling | ✗ | greedy decoding only |
| Prefix caching | ✗ | disabled in the deploy config |
| Tensor / pipeline parallelism | untested | single-GPU profile only |
| Quantization | ✗ | not supported in this release |
| CUDA graphs | not tuned | depth decoder and stage-1 codec run eagerly |
| Platforms | CUDA | qualified on 1x NVIDIA L20 |