vllm-omni/recipes/BreezeBlue/Breeze-TTS-2.md

Breeze-TTS-2

Breeze-TTS-2 generates 24 kHz English and Chinese speech. The integration supports Voice Design from text instructions, Voice Clone from a reference recording and transcript, and Voice Direction combining reference conditioning with instructions.

Installation and serving

Follow the project's source installation guide for vLLM and vLLM-Omni. The integration loads the released checkpoint directly. The original Breeze repository and the qwen-tts package are not required. The common requirements include soxr to reproduce reference-audio resampling.

vllm-omni serve BreezeBlue/Breeze-TTS-2 --omni --host 127.0.0.1 --port 8091

The default configuration uses one CUDA GPU, BF16 generation and codec computation, and FP32 reference encoding. Decoded waveform tensors are converted to FP32 before response formatting. It targets GPUs with at least 16 GiB available for the deployment; other workloads consume additional memory.

On WSL2, set vLLM's supported pinned-memory option before starting the server:

export VLLM_WSL2_ENABLE_PIN_MEMORY=1

Native Linux enables pinned memory by default. Startup warmup exercises plain generation, CFG, reference conditioning and two-request batches. New text lengths or batch sizes can require additional graph capture or compilation; warm the workload before measuring steady performance.

Voice Design

curl --fail http://127.0.0.1:8091/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "BreezeBlue/Breeze-TTS-2",
    "input": "你好，欢迎使用语音合成系统。",
    "instructions": "年轻女性，声音温暖自然，使用标准普通话。",
    "extra_params": {"guidance_scale": 4.0},
    "response_format": "wav",
    "seed": 42,
    "max_new_tokens": 250
  }' --output breeze.wav

The voice field selects a speaker tag, such as S0 or [S0]; omitting it uses [S0]. These are checkpoint prompt tags, not a catalog of named voices. Omitting instructions uses Speak clearly and naturally.. An explicitly empty instruction remains empty. Vocal-event tokens, including (laughs) and [笑], are passed to the checkpoint unchanged.

Voice Clone and Voice Direction

Add ref_audio and its exact ref_text transcript. Reference audio accepts the shared speech API's audio URLs and base64 data URLs, either as a string or a one-item list. The shared resolver's format, duration and local-file access restrictions apply.

{
  "model": "BreezeBlue/Breeze-TTS-2",
  "input": "Please read this new sentence clearly and naturally.",
  "ref_audio": "data:audio/wav;base64,<base64-encoded-reference>",
  "ref_text": "The words actually spoken in the reference recording.",
  "instructions": "Keep the reference voice. Speak gently and calmly.",
  "extra_params": {"guidance_scale": 4.0},
  "response_format": "wav",
  "seed": 42
}

Use guidance_scale: 1.0 with the default instruction for ordinary cloning. Reference audio plus a delivery instruction and CFG provides Voice Direction. When explicitly specified, task_type is Base for reference conditioning and VoiceDesign without reference audio. The legacy CustomVoice value also remains accepted.

Stereo references are averaged to mono and resampled to 24 kHz with soxr HQ. Reference text is encoded independently from the target text. The reference codec preserves full causal attention during offline encoding, as used by the released Breeze runtime; newer Transformers sliding-window defaults would change long references. Reference convolution uses full FP32 precision.

Sampling and output

Supported extra_params:

Parameter	Default	Meaning
guidance_scale	1.0	Any finite positive value; values other than 1 use two CFG branches
temperature	0.9	Zero selects greedy generation
top_k	50	Zero or -1 disables top-k filtering
top_p	1.0	Nucleus sampling probability in (0, 1]
repetition_penalty	1.1	Positive penalty applied to the first codebook's generated history

cfg_scale remains an alias for guidance_scale; guidance_scale takes precedence when both are supplied.

CFG combines conditional and unconditional logits in the Qwen3 output head and all 15 remaining depth codebooks. Both branches retain reference audio and reference text; the unconditional branch removes only target instructions.

max_new_tokens limits generated audio frames, including EOS. Each frame is 80 ms; the default is 1500 tokens, subject to the remaining 2048-token context. Both CFG branches stop at the same generation limit despite unequal prompts. Each request owns its seed, generator, audio-code history and codec state. Different GPU kernels and batch shapes can still change sampled outputs.

For streamed PCM, use "stream": true, "stream_format": "audio", and "response_format": "pcm". Output is mono signed 16-bit little-endian PCM at 24 kHz. Complete WAV responses accumulate the same internal streaming output. With the default deployment, the first chunks contain 1, 2, 4 and 5 codec frames, then remain at 5 frames.

Non-streaming throughput profile

For concurrent complete responses, select the throughput deployment from the repository root.

vllm-omni serve BreezeBlue/Breeze-TTS-2 --omni \
  --deploy-config vllm_omni/deploy/breeze_tts_2_throughput.yaml \
  --host 127.0.0.1 --port 8091

Request a complete WAV with "stream": false and omit stream_format. Setting "stream_format": "audio" enables streaming even when stream is false.

curl --fail http://127.0.0.1:8091/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "BreezeBlue/Breeze-TTS-2",
    "input": "Hello, welcome to the speech synthesis demonstration.",
    "stream": false,
    "response_format": "wav",
    "seed": 42,
    "max_new_tokens": 250
  }' --output breeze-complete.wav

The profile was tested on one H100 80 GB. It sets max_num_seqs: 32 on both stages and reserves 8 GiB of stage-0 KV cache for thirty-two 2048-token contexts. These are physical sequence slots: stage 0 can run up to thirty-two plain requests (guidance_scale: 1.0) or sixteen CFG requests, since each CFG request occupies two slots. Mixed workloads share the same limit; additional requests queue. The default deployment's memory target does not apply to this larger profile.

The internal codec uses fixed 16-frame chunks from the first chunk onward, with a shorter final chunk when needed. cfg_prefill_delay_tokens: 0 removes the default eight-frame admission delay, so waiting requests can enter as soon as scheduling and KV capacity permit. The stateful codec still requires async_chunk: true for these complete HTTP responses. Warm the expected text lengths and concurrency before measuring the profile.

The shared benchmark defaults to streaming. Override both stream and stream_format to measure complete responses, and set the generation limit explicitly in the request body:

python benchmarks/tts/bench_tts.py \
  --model BreezeBlue/Breeze-TTS-2 --task default_voice --locale en \
  --dataset-path benchmarks/build_dataset/seed_tts_smoke \
  --host 127.0.0.1 --port 8091 --concurrency 1 2 4 8 16 32 64 \
  --num-prompts 128 --num-warmups 64 \
  --output-dir results/breeze-nonstream -- \
  --extra-body '{"stream":false,"stream_format":null,"response_format":"pcm","max_new_tokens":384,"seed":42,"extra_params":{"temperature":0.9,"top_k":50,"top_p":1,"repetition_penalty":1.1,"guidance_scale":1}}'

Use the same dataset and sampling settings when comparing deployment profiles. Verify completed and failed request counts in each result. For complete responses, compare throughput and RTF; first-response timing represents completion latency. Run ASR separately: the benchmark's WER mode forces streaming.

Pipeline and optimization

Stage 0 batches T5Gemma2 text encoding, optional reference-audio encoding, vLLM's native Qwen3 backbone with paged KV caching, and Breeze's Llama depth decoder. Text segments up to 128 tokens use padded CUDA Graphs with separate attention masks. Longer segments use dynamically compiled encoder layers without retaining a separate graph workspace for each input length. The depth decoder uses fused QKV/MLP projections, compiled layer kernels, fixed frame-local KV buffers and a CUDA Graph covering all 15 depth steps. Graphs pad requests to power-of-two buckets, keeping CFG branches separate. Compiled layers reuse kernels across batch sizes, avoiding a new compilation for every shrinking batch. Only real requests advance their random state, and returned tensors exclude padded rows. Positive temperature, top-k and top-p are mutable GPU inputs to the captured sampler. Changing those settings, or changing a positive CFG scale within the paired path, reuses the graph. Greedy decoding uses a separate graph that skips sorting and random sampling. Each request owns its random-number generator.

Reference encoding uses dynamic-shape compilation with CUDA Graphs disabled for that path, so large reference-convolution workspaces remain reusable by the allocator. Every convolution retains each recording's actual boundary; the final downsampler repeats its last valid state for partial frames, matching independent encoding. Static convolution metadata removes GPU-to-CPU padding calculations. Startup compiles reference batches of one and two. Waveform storage is aligned to 1920-sample frames, with that frame width marked static for compilation. Actual sample lengths still govern every convolution boundary. This avoids nested dynamic ceil/mod expressions that otherwise make compiler tiling spend minutes simplifying reference shapes.

Stage 1 reuses Qwen3-TTS's stateful codec and its CUDA Graph decoder with the checkpoint's audio_tokenizer/ weights. Shared-memory chunks connect stages. Only the generated-code decode path is captured: reference conditioning was already consumed in stage 0, so the Qwen ICL reference-prefix graphs and their large convolution workspaces are unnecessary for Breeze.

The Qwen3 backbone uses compiled piecewise prefill and full decode graphs. Stage 0 uses synchronous scheduling to keep CFG branches at the same generation step; stage 1 uses asynchronous scheduling. CFG admission reserves enough KV pages for both branches' complete generation and preserves their actual position IDs. Cancellation finishes both branches.

The default admission policy lets active requests produce eight audio frames before adding another prefill. This gives playback a buffer before text and reference encoding share the device with an existing stream. It increases a queued request's first-audio latency; benchmark it separately from an idle server.

The default max_num_seqs: 2 counts physical branches: it permits two plain requests or one CFG request at a time. Additional CFG requests queue. Prefix caching and chunked prefill are disabled because prompt token IDs reserve positions for externally encoded embeddings. The deployment requires async_chunk: true, including for complete HTTP responses.

The accelerated implementation retains the breeze_tts_2 model package, breeze_tts_2 and breeze_tts_2_codec stage names, and default breeze_tts_2.yaml deployment. The shorter breeze_tts.yaml and breeze_tts_throughput.yaml deployment filenames remain compatibility aliases. For a copied custom configuration, use the current breeze_tts_2.yaml as the starting point: codec chunk controls use codec_chunk_frames, initial_codec_chunk_frames and codec_chunk_ramp. Defaults now use a 2048-token context and temperature 0.9; set temperature to zero explicitly when comparing greedy generation with the earlier runtime.

The supported execution scope is one CUDA GPU, TP=1, PP=1 and unquantized released weights. Named voices, speaker embeddings, dual CFG and multiple reference recordings are not supported. The original public fast runtime also does not expose dual CFG.

Offline inference and validation

vllm_omni.model_executor.models.breeze_tts_2.prompt.build_breeze_prompt accepts a checkpoint tokenizer, text, instructions, sampling parameters, and optional ref_audio=(waveform, sample_rate) plus ref_text. Pass its result to Omni.generate() using the default Breeze deployment. Waveform arrays use soundfile's (samples, channels) convention.

CPU tests compare cached depth attention and CFG against Hugging Face Llama and cover request-owned sampling, payload transport, chunk delivery and CFG admission. GPU tests compare padded batched text graphs with independent eager encoders and exercise depth graphs across CFG scales and batch shapes.

pytest -v tests/model_executor/models/test_breeze_tts_2.py \
  tests/model_executor/models/test_breeze_tts_2_registration.py \
  tests/core/sched/test_omni_cfg_ar_scheduler.py -m 'core_model and cpu'
pytest -v tests/model_executor/models/test_breeze_tts_2_graphs.py -m 'core_model and cuda'
pytest -v tests/e2e/online_serving/test_breeze_tts_2.py \
  -m 'core_model and tts' --run-level=core_model
pytest -v tests/e2e/online_serving/test_breeze_tts_2.py \
  -m 'advanced_model and tts' --run-level=advanced_model

Serving tests use the shared client for Chinese CFG design, English PCM, concurrent CFG requests, reference cloning and direction. Core tests use dummy weights and eight-frame requests; advanced tests use real weights and the shared content checks. These checks do not establish perceptual quality parity. The Chinese content check uses Whisper large-v3 as its primary transcriber: small misrecognizes homophones on the fixed test clip. The expected text and shared similarity threshold are unchanged; there is no failed-check retry. Ensure enough host memory and disk space for the large-v3 test model when Whisper runs on CPU alongside the serving process.

The English PCM test uses a 0 dB harmonic-to-noise floor because the original eager runtime scores 0.56 dB on that same prompt and seed, below the shared 1 dB floor. Raw PCM is not transcribed by the shared helper. Separate matched evaluations must retain audio and compare intelligibility, speaker similarity, instruction effects, first audio, RTF and playback underruns.

The shared serving benchmark supports default speech, Voice Design and Voice Clone. For Voice Design:

python benchmarks/tts/bench_tts.py \
  --model BreezeBlue/Breeze-TTS-2 --task voice_design --locale en \
  --host 127.0.0.1 --port 8091 --concurrency 1 2 \
  --num-prompts 4 --num-warmups 1 --request-seed 42 --output-len 250 \
  --output-dir results/breeze

For a local model path, add --served-model-name /path/to/Breeze-TTS-2. For cloning, select --task voice_clone --dataset-path /path/to/seed-tts-eval with the desired locale. Reference conditioning with instructions and other CFG scales also require identical requests and sampling settings in the reference runtime and this deployment.

The reference source and released weights have separate licenses. Consult the reference repository and checkpoint model card.