vllm-omni/docs/user_guide/examples/online_serving/text_to_speech.md

Text-To-Speech (Online Serving)

Source https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech.

vLLM-Omni exposes TTS models through the OpenAI-compatible POST /v1/audio/speech endpoint, launched with vllm serve <model> --omni. Each TTS model has its own subdirectory containing client snippets, gradio demos, and helper scripts; this README is the single doc entry point for all of them.

For offline inference, see examples/offline_inference/text_to_speech. For the full list of supported architectures across all modalities, see Supported Models.

Supported Models

Model	HuggingFace repo	Voice cloning	Streaming	Voice presets / upload	Gradio demo
CosyVoice3	FunAudioLLM/Fun-CosyVoice3-0.5B-2512	✓ (ref_audio+ref_text)	✓ (PCM stream)	—	—
Fish Speech S2 Pro	fishaudio/s2-pro	✓ (ref_audio+ref_text)	✓ (PCM stream)	—	✓
higgs-audio v2	bosonai/higgs-audio-v2-generation-3B-base	✓ (ref_audio+ref_text)	✓ (codec_streaming)	—	—
GLM-TTS	zai-org/GLM-TTS	✓ (ref_audio+ref_text, required)	✓ (PCM stream)	—	✓
OmniVoice	k2-fsa/OmniVoice	(offline only)	—	—	—
Qwen3-TTS	Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}	✓ (Base)	✓ (PCM + WebSocket)	✓ (presets + /v1/audio/voices upload)	✓ (standard + FastRTC)
VoxCPM2	openbmb/VoxCPM2	✓	✓ (AudioWorklet via gradio)	—	✓
Voxtral TTS	mistralai/Voxtral-4B-TTS-2603	✓ (gated upstream)	✓	✓ (presets)	✓

For offline inference of any of these models, see the offline TTS hub.

Common Quick Start

Launch the server (defaults shown — adjust --port, --gpu-memory-utilization, etc. as needed):

vllm serve <hf-repo-or-local-path> --omni --port 8091

Send a TTS request via curl:

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "default",
        "response_format": "wav"
    }' --output output.wav

Or via Python httpx:

import httpx

response = httpx.post(
    "http://localhost:8091/v1/audio/speech",
    json={
        "input": "Hello, how are you?",
        "voice": "default",
        "response_format": "wav",
    },
    timeout=300.0,
)
open("output.wav", "wb").write(response.content)

Or via the OpenAI SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
    model="<hf-repo>",
    voice="default",
    input="Hello, how are you?",
)
response.stream_to_file("output.wav")

Streaming PCM output (where supported) — set stream=true with response_format="pcm":

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "default",
        "stream": true,
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others).

For full request-shape documentation (all parameters, response formats, error codes), see the Speech API reference.

---

CosyVoice3

2-stage TTS (talker + flow-matching code2wav) at 24 kHz. Voice cloning only — every request needs ref_audio + ref_text; there are no built-in voice presets.

Prerequisites

huggingface-cli download FunAudioLLM/Fun-CosyVoice3-0.5B-2512

If your downloaded checkpoint lacks config.json, add one with {"model_type": "cosyvoice3", "architectures": ["CosyVoice3Model"]} (the loader reads model_type to select the class).

Launch

vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remote-code
# or:
./cosyvoice3/run_server.sh

Streaming is on by default via async_chunk: true in vllm_omni/deploy/cosyvoice3.yaml. Pass --no-async-chunk (or NO_ASYNC_CHUNK=1 ./cosyvoice3/run_server.sh) for the legacy synchronous path.

CLI client

The client defaults to the official upstream zero-shot prompt, so it runs without extra flags:

cd examples/online_serving/text_to_speech/cosyvoice3
python speech_client.py --text "收到好友从远方寄来的生日礼物。"

Pass your own reference clip and transcript for a different voice:

python speech_client.py --text "Hello, this is a cloned voice." \
    --ref-audio /path/to/reference.wav \
    --ref-text "Transcript of the reference audio."

Stream PCM instead of WAV:

python speech_client.py --text "Hello world" --stream --output output.pcm

The client supports --api-base, --model, --text, --ref-audio, --ref-text, --response-format, --stream, --output.

Notes

• Stage 0 (talker) emits speech tokens; stage 1 (code2wav) runs flow matching + HiFiGAN to synthesize waveform.
• Deploy config auto-loads from vllm_omni/deploy/cosyvoice3.yaml based on HF model_type. Pass --deploy-config <path> to override.
• For offline inference and the end-to-end script, see the offline CosyVoice3 section.

---

Fish Speech S2 Pro

4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec.

Prerequisites

pip install fish-speech

Launch

vllm serve fishaudio/s2-pro --omni --port 8091
# or:
./fish_speech/run_server.sh

The deploy config auto-loads from vllm_omni/deploy/fish_qwen3_omni.yaml (the HF model_type on the fishaudio checkpoint is fish_qwen3_omni).

Voice cloning

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is a cloned voice.",
        "voice": "default",
        "ref_audio": "https://example.com/reference.wav",
        "ref_text": "Transcript of the reference audio."
    }' --output cloned.wav

CLI client

cd examples/online_serving/text_to_speech/fish_speech
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Hello world" --stream --output output.pcm

Gradio demo

./fish_speech/run_gradio_demo.sh             # launches server + Gradio
python fish_speech/gradio_demo.py --api-base http://localhost:8091  # if server already running

Notes

• Output: 44.1 kHz mono.
• Streaming PCM player command must use -r 44100.

---

higgs-audio v2

2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) that streams chunks back to the client.

Prerequisites

Voice clone uses HF's HiggsAudioV2TokenizerModel loaded from k2-fsa/OmniVoice/audio_tokenizer/ (~806 MB subdir; the boson-ai standalone tokenizer Hub repo's model.safetensors is the 3B talker LM, not the codec):

pip install -U "transformers>=5.3.0"

Launch

GPUS=6,7 PORT=8094 bash examples/online_serving/text_to_speech/higgs_audio_v2/run_server.sh

Deploy config auto-loads from vllm_omni/deploy/higgs_audio_v2.yaml.

Sending requests

# Plain TTS
python higgs_audio_v2/batch_speech_client.py \
    --base-url http://localhost:8094 \
    --output-dir /tmp/higgs_out \
    --prompts "Hello world." "The quick brown fox jumps over the lazy dog."

# Voice cloning — pass a reference clip and its transcript together
python higgs_audio_v2/batch_speech_client.py \
    --base-url http://localhost:8094 \
    --output-dir /tmp/higgs_clone \
    --ref-audio /path/to/reference.wav \
    --ref-text  "Exact transcript spoken in reference.wav." \
    --prompts "Hello, this is a cloned voice."

Notes

• Output: 24 kHz mono.
• --ref-text must be the real transcript of --ref-audio; mismatched text degrades cloned-voice quality.
• Out of scope (rejected with explicit 4xx): multi-speaker [SPEAKERn] tags inside input, profile: text-only speaker descriptions, the ref_audio_in_system_message system-block variant, chunked long-form generation, and per-request voice / instructions / task_type / language / speed != 1.0 / x_vector_only_mode / speaker_embedding.

GLM-TTS

2-stage TTS (AR + DiT flow-matching) at 24 kHz. Every request requires ref_audio + ref_text.

Launch

vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091
# or:
bash examples/online_serving/text_to_speech/glm_tts/run_server.sh /path/to/GLM-TTS

Sending requests

# Voice cloning (required)
python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
    --text "你好，这是语音克隆测试。" \
    --ref-audio file:///path/to/ref.wav \
    --ref-text "这是参考音频的文本内容。"

# Custom format
python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
    --text "Hello, this is a voice cloning test." \
    --ref-audio file:///path/to/ref.wav \
    --ref-text "Transcript of the reference audio." \
    --response-format mp3 -o output.mp3

Gradio demo

bash examples/online_serving/text_to_speech/glm_tts/run_gradio_demo.sh

Notes

• Output: 24 kHz mono WAV via HiFT vocoder.
• ref_audio + ref_text are required together on every request. Reference audio should be 3-10 seconds.
• Voice cloning feature extraction (WhisperVQ, CampPlus, mel) runs on the model side — no external dependency on the serving layer.

---

OmniVoice

Zero-shot multilingual TTS (600+ languages). Online serving currently exposes auto voice only; voice cloning and voice design are available offline.

Prerequisites

huggingface-cli download k2-fsa/OmniVoice

Voice cloning (offline) needs transformers>=5.3.0; auto voice works with transformers>=4.57.0.

Launch

vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code
# or:
./omnivoice/run_server.sh

CLI client

cd examples/online_serving/text_to_speech/omnivoice
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Bonjour, comment allez-vous?" --language French

The client supports --api-base, --model, --text, --response-format, --language, --output.

Notes

• Voice cloning and voice design require offline inference; see the offline OmniVoice section.

---

Qwen3-TTS

Three model variants exposed via separate checkpoints:

Variant	HF repo	Use
CustomVoice	Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice	Predefined speakers (vivian, ryan, …) with optional style instructions
VoiceDesign	Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign	Natural-language voice style description
Base	Qwen/Qwen3-TTS-12Hz-1.7B-Base	Voice cloning from a reference audio

Each variant ships smaller 0.6B companions where available.

Launch

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
# or:
./qwen3_tts/run_server.sh                # default: CustomVoice
./qwen3_tts/run_server.sh VoiceDesign
./qwen3_tts/run_server.sh Base

Executor backend

Single-GPU serves now default to the uniproc executor (lower IPC overhead, the Base cloning use case from #2603 / #2604). vllm_omni/deploy/qwen3_tts.yaml is the only Qwen3-TTS deploy config; pass --deploy-config <path> to override.

To opt out of chunked streaming, pass --no-async-chunk — the pipeline auto-dispatches to the end-to-end codec processor.

Sending requests

# CustomVoice with a predefined speaker
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --text "今天天气真好" \
    --voice ryan \
    --instructions "用开心的语气说"

# VoiceDesign with a style description
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
    --task-type VoiceDesign \
    --text "哥哥，你回来啦" \
    --instructions "体现撒娇稚嫩的萝莉女声，音调偏高"

# Base voice cloning
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --task-type Base \
    --text "Hello, this is a cloned voice" \
    --ref-audio /path/to/reference.wav \
    --ref-text "Original transcript of the reference audio"

Voices endpoint

List available voices, or upload a custom one for Base cloning:

# List
curl http://localhost:8091/v1/audio/voices

# Upload
curl -X POST http://localhost:8091/v1/audio/voices \
    -F "audio_sample=@/path/to/voice_sample.wav" \
    -F "consent=user_consent_id" \
    -F "name=custom_voice_1" \
    -F "ref_text=The exact transcript of the audio sample." \
    -F "speaker_description=warm narrator"

Uploaded voices are then usable as voice="custom_voice_1" on subsequent requests.

Streaming PCM

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English",
        "stream": true,
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Streaming requires response_format="pcm" and async_chunk: true on the stage config (default in qwen3_tts.yaml). speed is not supported when streaming.

Streaming WebSocket

The /v1/audio/speech/stream endpoint accepts text incrementally, splits it at sentence boundaries, and emits one PCM stream per sentence:

python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you? I am fine."
python qwen3_tts/streaming_speech_client.py --text "..." --simulate-stt --stt-delay 0.1

Gradio demos

./qwen3_tts/run_gradio_demo.sh                              # CustomVoice (default)
./qwen3_tts/run_gradio_demo.sh --task-type VoiceDesign
./qwen3_tts/run_gradio_demo.sh --task-type Base

# FastRTC variant (gapless WebRTC streaming):
pip install fastrtc
python qwen3_tts/gradio_fastrtc_demo.py --api-base http://localhost:8000

Speaker embedding interpolation

qwen3_tts/speaker_embedding_interpolation.py blends two predefined speakers' embeddings to produce intermediate voices. See the script for usage.

Batch client

qwen3_tts/batch_speech_client.py issues many concurrent requests for throughput measurement.

Notes

• Base voice cloning has per-request reference-audio cost; the uniproc default keeps IPC overhead off the critical path. See the executor-backend section above for background.
• vllm_omni/deploy/qwen3_tts.yaml is the default deploy config (loaded by HF model_type); per-stage runtime overrides are available via --stage-N-<field> <value>.

---

---

VoxCPM2

Single-stage native AR TTS at 48 kHz.

Launch

vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000

Deploy config auto-loads from vllm_omni/deploy/voxcpm2.yaml. Pass --deploy-config <path> to override or --stage-N-<field> <value> for per-stage runtime tweaks.

Sending requests

# Zero-shot synthesis
python voxcpm2/openai_speech_client.py --text "Hello, this is VoxCPM2."

# Voice cloning
python voxcpm2/openai_speech_client.py \
    --text "This should sound like the reference speaker." \
    --ref-audio /path/to/reference.wav

The ref_audio field accepts local file paths (auto-base64), HTTP URLs, or data:audio/wav;base64,... data URIs.

Gradio demo (gapless streaming via AudioWorklet)

python voxcpm2/gradio_demo.py

Uses an AudioWorklet-based player adapted from the Qwen3-TTS demo for gap-free playback. Audio is streamed from the OpenAI Speech endpoint with stream=true.

---

Voxtral TTS

Voxtral-4B-TTS (Mistral). Uses the mistral_common SpeechRequest protocol; voice presets are model-specific.

Prerequisites

Latest mistral_common with SpeechRequest support:

pip install -e /path/to/mistral-common  # or upgrade from PyPI when available

Launch

vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091

Deploy config auto-loads from vllm_omni/deploy/voxtral_tts.yaml.

Gradio demo

python voxtral_tts/gradio_demo.py

The demo handles voice-preset selection and reference-audio upload. voxtral_tts/text_preprocess.py provides the text-normalization helpers used by the demo (also available for other clients).

Notes

• Voice presets are listed on the HF model card (mistralai/Voxtral-4B-TTS-2603).
• Voice cloning is gated upstream and may require a recent mistral_common.
• A standalone CLI client is not yet shipped; the gradio demo is the canonical reference for now.

Example materials

??? abstract "cosyvoice3/run_server.sh"

--8<-- "examples/online_serving/text_to_speech/cosyvoice3/run_server.sh"

??? abstract "cosyvoice3/speech_client.py"

--8<-- "examples/online_serving/text_to_speech/cosyvoice3/speech_client.py"

??? abstract "fish_speech/gradio_demo.py" py --8<-- "examples/online_serving/text_to_speech/fish_speech/gradio_demo.py" ??? abstract "fish_speech/run_gradio_demo.sh" sh --8<-- "examples/online_serving/text_to_speech/fish_speech/run_gradio_demo.sh" ??? abstract "fish_speech/run_server.sh" sh --8<-- "examples/online_serving/text_to_speech/fish_speech/run_server.sh" ??? abstract "fish_speech/speech_client.py" py --8<-- "examples/online_serving/text_to_speech/fish_speech/speech_client.py" ??? abstract "higgs_audio_v2/batch_speech_client.py" py --8<-- "examples/online_serving/text_to_speech/higgs_audio_v2/batch_speech_client.py" ??? abstract "higgs_audio_v2/run_server.sh" sh --8<-- "examples/online_serving/text_to_speech/higgs_audio_v2/run_server.sh" ??? abstract "glm_tts/gradio_demo.py" py --8<-- "examples/online_serving/text_to_speech/glm_tts/gradio_demo.py" ??? abstract "glm_tts/openai_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py" ??? abstract "glm_tts/run_gradio_demo.sh"sh --8<-- "examples/online_serving/text_to_speech/glm_tts/run_gradio_demo.sh" ??? abstract "glm_tts/run_server.sh"sh --8<-- "examples/online_serving/text_to_speech/glm_tts/run_server.sh" ??? abstract "omnivoice/run_server.sh"sh --8<-- "examples/online_serving/text_to_speech/omnivoice/run_server.sh" ??? abstract "omnivoice/speech_client.py"py --8<-- "examples/online_serving/text_to_speech/omnivoice/speech_client.py" ??? abstract "qwen3_tts/batch_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/batch_speech_client.py" ??? abstract "qwen3_tts/gradio_demo.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/gradio_demo.py" ??? abstract "qwen3_tts/openai_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/openai_speech_client.py" ??? abstract "qwen3_tts/run_gradio_demo.sh"sh --8<-- "examples/online_serving/text_to_speech/qwen3_tts/run_gradio_demo.sh" ??? abstract "qwen3_tts/run_server.sh"sh --8<-- "examples/online_serving/text_to_speech/qwen3_tts/run_server.sh" ??? abstract "qwen3_tts/speaker_embedding_interpolation.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/speaker_embedding_interpolation.py" ??? abstract "qwen3_tts/streaming_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/streaming_speech_client.py" ??? abstract "qwen3_tts/tts_common.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/tts_common.py" ??? abstract "voxcpm2/gradio_demo.py"py --8<-- "examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py" ??? abstract "voxcpm2/openai_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py" ??? abstract "voxtral_tts/gradio_demo.py"py --8<-- "examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py" ??? abstract "voxtral_tts/text_preprocess.py"py --8<-- "examples/online_serving/text_to_speech/voxtral_tts/text_preprocess.py" ``````