vllm-omni/docs/user_guide/examples/offline_inference/text_to_speech.md

Text-To-Speech

Source https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech.

vLLM-Omni supports several autoregressive TTS models. They share a common CLI shape (--text, --ref-audio, --ref-text, --output-dir) and live together in this hub. Each model has its own subdirectory containing a single end2end.py script; this README is the single doc entry point.

For online serving, see examples/online_serving/<model>/. For the full list of supported architectures across all modalities, see Supported Models.

Supported Models

Model	HuggingFace repo	Stages	Voice cloning	Streaming	Special modes	Sample rate
VoxCPM2	openbmb/VoxCPM2	single (native AR)	✓	—	continuation (--ref-audio + --ref-text)	48 kHz
VoxCPM	local model dir	split	✓	✓ (voxcpm_async_chunk.yaml)	—	24 kHz
CosyVoice3	FunAudioLLM/Fun-CosyVoice3-0.5B-2512	2 (talker + code2wav)	✓	✓ (async_chunk: true default)	—	22.05 kHz
Fish Speech S2 Pro	fishaudio/s2-pro	dual-AR	✓	✓ (--streaming)	—	44.1 kHz
OmniVoice	k2-fsa/OmniVoice	2 (gen + dec)	✓	—	voice design (--instruct), language (--lang)	24 kHz
Qwen3-TTS	Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}	2 (talker + code2wav)	✓ (Base)	✓	3 task variants (--query-type)	24 kHz
Voxtral TTS	mistralai/Voxtral-4B-TTS-2603	varies	✓	✓	voice presets (--voice)	24 kHz

Common Quick Start

Most models share this invocation shape:

python examples/offline_inference/text_to_speech/<model>/end2end.py \
    --text "Hello, this is a test." \
    --ref-audio /path/to/reference.wav \
    --ref-text  "Transcript of the reference audio."

--ref-audio and --ref-text are optional (text-only synthesis works without them) and must be provided together for voice cloning. The exotic scripts — Qwen3-TTS, Voxtral TTS, CosyVoice3 — accept additional model-specific flags documented in their per-model section below.

---

VoxCPM2

Single-stage native AR TTS at 48 kHz. Pipeline: feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE.

Prerequisites

pip install voxcpm
# or, for a local source checkout:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm

Quick start

python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
    --model openbmb/VoxCPM2 \
    --text "Hello, this is a VoxCPM2 demo."

Voice cloning

Pass a reference audio for isolated cloning, or both --ref-audio + --ref-text for prompt continuation:

python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
    --text "Hello, this is a voice clone demo." \
    --ref-audio /path/to/reference.wav \
    --ref-text  "Transcript of the reference audio."

Notes

• Output: 48 kHz mono WAV.
• Stage config: vllm_omni/model_executor/stage_configs/voxcpm2.yaml (default).

---

VoxCPM

Split-stage TTS. The hub example covers single TTS, single voice cloning, and streaming. Use benchmarks/voxcpm/ for warmup, batch JSONL prompts, profiler injection, and offline TTFP / RTF measurement.

Prerequisites

pip install voxcpm soundfile
# or use a local source tree:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src

If the native VoxCPM config.json does not contain HF metadata such as model_type, prepare a persistent HF-compatible config directory and point the stage configs to it via VLLM_OMNI_VOXCPM_HF_CONFIG_PATH:

export VOXCPM_MODEL=/path/to/voxcpm-model
export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config
mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"
cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json"
cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true
python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)'

Quick start

python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
    --model "$VOXCPM_MODEL" \
    --text "This is a split-stage VoxCPM synthesis example running on vLLM Omni."

Voice cloning

python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
    --model "$VOXCPM_MODEL" \
    --text "This sentence is synthesized with a cloned voice." \
    --ref-audio /path/to/reference.wav \
    --ref-text  "The exact transcript spoken in reference.wav."

Streaming

Pass the async-chunk stage config:

python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
    --model "$VOXCPM_MODEL" \
    --stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
    --text "This is a split-stage VoxCPM streaming example running on vLLM Omni."

Notes

• voxcpm.yaml is the default non-streaming stage config; voxcpm_async_chunk.yaml enables streaming.
• Streaming is currently single-request oriented.
• --ref-text must be the real transcript of --ref-audio; mismatched text degrades quality.
• For online serving, see examples/online_serving/voxcpm. For benchmark reporting, see benchmarks/voxcpm.

---

CosyVoice3

2-stage TTS pipeline (talker + code2wav) at 22.05 kHz.

Prerequisites

uv pip install -e .
# Includes soundfile, onnxruntime, x-transformers, einops via requirements.

Download the model snapshot:

from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512',
                  local_dir='pretrained_models/Fun-CosyVoice3-0.5B')

If your downloaded checkpoint lacks config.json, add it:

{
    "model_type": "cosyvoice3",
    "architectures": ["CosyVoice3Model"]
}

This is required because AutoConfig.register("cosyvoice3", CosyVoice3Config) only registers the class mapping; the loader still reads model_type from config.json to select the class.

Quick start

python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
    --model pretrained_models/Fun-CosyVoice3-0.5B \
    --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN

Voice cloning

Pass a reference audio. Note that CosyVoice3's --prompt-text is a system-style prompt for the GPT stage, not a reference transcript:

python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
    --model pretrained_models/Fun-CosyVoice3-0.5B \
    --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN \
    --ref-audio prompt.wav \
    --prompt-text "You are a helpful assistant.<|endofprompt|>Testing my voices. Why should I not?"

Notes

• Stage 0 (talker) emits speech tokens; stage 1 (code2wav) runs flow matching + HiFiGAN to synthesize waveform.
• Deploy config auto-loads from vllm_omni/deploy/cosyvoice3.yaml based on HF model_type. Pass --deploy-config <path> to override.
• async_chunk: true is the default; pass --no-async-chunk to switch to the legacy synchronous path.

---

Fish Speech S2 Pro

4B dual-AR text-to-speech model from FishAudio with the DAC codec at 44.1 kHz.

Prerequisites

pip install fish-speech

Quick start

python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
    --text "Hello, this is a test of the Fish Speech text to speech system."

Voice cloning

python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
    --text "Hello, this is a cloned voice." \
    --ref-audio /path/to/reference.wav \
    --ref-text  "Transcript of the reference audio."

Streaming

python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
    --text "Hello, this is a streaming test." \
    --streaming

Streaming requires async_chunk: true in the stage config.

Notes

• Output: 44.1 kHz mono WAV.
• DAC codec weights (codec.pth) are loaded lazily from the model directory.

---

OmniVoice

Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design).

Prerequisites

huggingface-cli download k2-fsa/OmniVoice

Voice cloning requires transformers>=5.3.0. Auto and design modes work with transformers>=4.57.0.

Quick start (auto voice)

python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --model k2-fsa/OmniVoice \
    --text "Hello, this is a test."

Voice cloning

python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --model k2-fsa/OmniVoice \
    --text "Hello, this is a test." \
    --ref-audio ref.wav \
    --ref-text  "This is the reference transcription."

Voice design

python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --model k2-fsa/OmniVoice \
    --text "Hello, this is a test." \
    --instruct "female, low pitch, british accent"

Language hint

python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --model k2-fsa/OmniVoice \
    --text "你好，这是一个测试。" \
    --lang zh

Notes

• Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking.
• Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz.

---

Qwen3-TTS

3-task-variant TTS with 24 kHz output. Has its own argparse surface (this script does not follow the common --text / --ref-audio shape).

Prerequisites

For ROCm builds, replace onnxruntime with onnxruntime-rocm:

pip uninstall onnxruntime
pip install onnxruntime-rocm

Task variants

• CustomVoice: predefined speaker (speaker ID) with optional style instruction.
• VoiceDesign: text + descriptive instruction designs a new voice.
• Base: voice cloning from reference audio + transcript.

# Single sample
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type VoiceDesign
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base

# Base variant has an additional mode flag:
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag icl       # default
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag xvec_only # x_vector_only_mode

# Batch (multiple prompts in one run)
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice --use-batch-sample

Streaming

python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
    --query-type CustomVoice \
    --streaming \
    --output-dir /tmp/out_stream

Streaming requires async_chunk: true in the stage config.

Batched decoding

The Code2Wav stage supports batched decoding through the SpeechTokenizer. Configure both stages with max_num_seqs > 1 via --stage-overrides and pass multiple prompts via --txt-prompts:

python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
    --query-type CustomVoice \
    --txt-prompts examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt \
    --batch-size 4 \
    --stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}'

--batch-size must match a CUDA-graph capture size (1, 2, 4, 8, 16…).

Notes

• Run --help for the full argument surface.
• See qwen3_tts/end2end.py for the prompt-length-estimation logic the Talker uses.

---

Voxtral TTS

Voxtral-4B-TTS (Mistral). Has its own argparse surface; uses voice presets and the mistral_common SpeechRequest protocol.

Prerequisites

Latest mistral_common with SpeechRequest support:

pip install -e /path/to/mistral-common  # or upgrade from PyPI when available

Quick start (voice preset)

python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
    --write-audio --voice cheerful_female \
    --model mistralai/Voxtral-4B-TTS-2603 \
    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"

Voice cloning (capability gated upstream)

python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
    --write-audio \
    --model mistralai/Voxtral-4B-TTS-2603 \
    --text "This is a test message." \
    --ref-audio path/to/reference_audio.wav

Streaming + concurrency

python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
    --num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
    --model mistralai/Voxtral-4B-TTS-2603 \
    --text "..."

Available voice presets are listed on the HF model card (mistralai/Voxtral-4B-TTS-2603).

Notes

• --num-prompts N replicates the prompt for performance measurement.
• --concurrency M requires --streaming and must evenly divide --num-prompts.
• Run --help for the full argument surface.

Example materials

??? abstract "cosyvoice3/end2end.py" py --8<-- "examples/offline_inference/text_to_speech/cosyvoice3/end2end.py" ??? abstract "fish_speech/end2end.py" py --8<-- "examples/offline_inference/text_to_speech/fish_speech/end2end.py" ??? abstract "omnivoice/end2end.py" py --8<-- "examples/offline_inference/text_to_speech/omnivoice/end2end.py" ??? abstract "qwen3_tts/benchmark_prompts.txt" txt --8<-- "examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt" ??? abstract "qwen3_tts/end2end.py" py --8<-- "examples/offline_inference/text_to_speech/qwen3_tts/end2end.py" ??? abstract "voxcpm/end2end.py" py --8<-- "examples/offline_inference/text_to_speech/voxcpm/end2end.py" ??? abstract "voxcpm2/end2end.py" py --8<-- "examples/offline_inference/text_to_speech/voxcpm2/end2end.py" ??? abstract "voxtral_tts/end2end.py" py --8<-- "examples/offline_inference/text_to_speech/voxtral_tts/end2end.py"