vllm-omni/examples/online_serving/text_to_speech/README.md

Text-To-Speech (Online Serving)

vLLM-Omni exposes TTS models through the OpenAI-compatible POST /v1/audio/speech endpoint, launched with vllm serve <model> --omni. Each TTS model has its own subdirectory containing client snippets, gradio demos, and helper scripts; this README is the single doc entry point for all of them.

For offline inference, see examples/offline_inference/text_to_speech. For the full list of supported architectures across all modalities, see Supported Models.

Supported Models

Model	HuggingFace repo	Voice cloning	Streaming	Voice presets / upload	Gradio demo
Breeze-TTS-2	BreezeBlue/Breeze-TTS-2	✓ (ref_audio+ref_text)	✓ (PCM stream)	speaker tags (S0..S9, default S0)	—
Audio8 TTS Preview	Audio8/Audio8-TTS-Preview-0.6b	✓ (ref_audio+ref_text)	✓ (PCM stream)	uploaded audio voice only; no presets	✓
Fish Speech S2 Pro	fishaudio/s2-pro	✓ (ref_audio+ref_text)	✓ (PCM stream)	—	✓
GLM-TTS	zai-org/GLM-TTS	✓ (ref_audio+ref_text, required)	✓ (PCM stream)	—	✓
IndexTTS-2	IndexTeam/IndexTTS-2	✓ (ref_audio or uploaded voice)	stream=true response, non-chunk	uploaded audio voice only; no presets	—
IndexTTS-2.5	native checkpoints/ bundle	✓ (ref_audio or uploaded voice)	stream=true response, non-chunk	uploaded audio voice only; no presets	—
Ming-omni-tts	inclusionAI/Ming-omni-tts-0.5B	✓ (ref_audio / speaker_embedding)	✓ (PCM stream)	IP labels + structured instructions	—
Ming-flash-omni-TTS	Jonathan1909/Ming-flash-omni-2.0	— (caption-controlled)	—	caption fields (instructions)	—
MOSS-TTS-Nano	OpenMOSS-Team/MOSS-TTS-Nano	✓ (ref_audio required)	✓ (PCM stream)	—	✓
OmniVoice	k2-fsa/OmniVoice	✓	—	—	—
Qwen3-TTS	Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}	✓ (Base)	✓ (PCM + WebSocket)	✓ (presets + /v1/audio/voices upload)	✓ (standard + FastRTC)
VoxCPM2	openbmb/VoxCPM2	✓	✓ (AudioWorklet via gradio)	—	✓
dots.tts	dots-studio/dots.tts-soar	— (text-only)	✓ (PCM stream)	— (default placeholder only)	—
Voxtral TTS	mistralai/Voxtral-4B-TTS-2603	✓ (gated upstream)	✓	✓ (presets)	✓

CosyVoice3 is intentionally absent: no online example exists for it yet. See its offline section instead.

Common Quick Start

Launch the server (defaults shown — adjust --port, --gpu-memory-utilization, etc. as needed):

vllm serve <hf-repo-or-local-path> --omni --port 8091

Send a TTS request via curl. These generic snippets assume a model with a preset/default voice; voice-cloning-only models such as IndexTTS-2 require ref_audio or an uploaded audio voice (see model-specific sections below).

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "default",
        "response_format": "wav"
    }' --output output.wav

Or via Python httpx:

import httpx

response = httpx.post(
    "http://localhost:8091/v1/audio/speech",
    json={
        "input": "Hello, how are you?",
        "voice": "default",
        "response_format": "wav",
    },
    timeout=300.0,
)
open("output.wav", "wb").write(response.content)

Or via the OpenAI SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
    model="<hf-repo>",
    voice="default",
    input="Hello, how are you?",
)
response.stream_to_file("output.wav")

Streaming PCM output (where supported) — set stream=true, stream_format="audio", and response_format="pcm":

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "default",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for IndexTTS-2, and 24 kHz for many others).

For full request-shape documentation (all parameters, response formats, error codes), see the Speech API reference.

---

dots.tts

Single-stage, text-only TTS at 48 kHz. The current integration supports no-reference synthesis; voice cloning, reference audio, named voices, and precomputed speaker embeddings are not supported yet.

Launch

vllm serve dots-studio/dots.tts-soar --omni --trust-remote-code --port 8091

Sending requests

# Non-streaming WAV
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is a dots TTS online-serving test.",
        "voice": "default",
        "response_format": "wav"
    }' --output output.wav

# Streaming 48 kHz mono PCM
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is a streaming dots TTS test.",
        "voice": "default",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 48000 -e signed -b 16 -c 1 -

---

GLM-TTS

2-stage TTS (AR + DiT flow-matching) at 24 kHz. Every request requires ref_audio + ref_text.

Launch

vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091
# or:
bash examples/online_serving/text_to_speech/glm_tts/run_server.sh /path/to/GLM-TTS

Sending requests

# Voice cloning (required)
python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
    --text "你好，这是语音克隆测试。" \
    --ref-audio file:///path/to/ref.wav \
    --ref-text "这是参考音频的文本内容。"

# Custom format
python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
    --text "Hello, this is a voice cloning test." \
    --ref-audio file:///path/to/ref.wav \
    --ref-text "Transcript of the reference audio." \
    --response-format mp3 -o output.mp3

Gradio demo

bash examples/online_serving/text_to_speech/glm_tts/run_gradio_demo.sh

Notes

• Output: 24 kHz mono WAV via HiFT vocoder.
• ref_audio + ref_text are required together on every request. Reference audio should be 3-10 seconds.
• Voice cloning feature extraction (WhisperVQ, CampPlus, mel) runs on the model side — no external dependency on the serving layer.

---

IndexTTS-2 and IndexTTS-2.5

2-stage TTS at 22.05 kHz. Requests use ref_audio for voice cloning, or an uploaded audio voice from /v1/audio/voices. IndexTTS-2.5 uses a multilingual tokenizer, CAMPPlus speaker projection, EnhancedCodec, and code-only Stage 0→1 transfer by default. Both versions support emotion conditioning via emo_audio, emo_text, or emo_vector passed in extra_params.

Prerequisites

IndexTTS-2 uses its existing frontend and does not require the IndexTTS-2.5 text dependencies. Before launching IndexTTS-2.5, install its optional frontend dependencies:

pip install 'vllm-omni[indextts2]'

Launch

vllm serve IndexTeam/IndexTTS-2 --omni --trust-remote-code --port 8092
# or, to pass the bundled deploy config explicitly:
bash examples/online_serving/text_to_speech/indextts2/run_server.sh

# IndexTTS-2.5 default: use_gpt_latent=false
MODEL_VERSION=2.5 \
MODEL=/path/to/indextts-2.5 \
bash examples/online_serving/text_to_speech/indextts2/run_server.sh

Sending requests

# Voice cloning (ref_audio required)
python examples/online_serving/text_to_speech/indextts2/speech_client.py \
    --text "你好，世界！" \
    --ref-audio /path/to/reference.wav

# With emotion audio
python examples/online_serving/text_to_speech/indextts2/speech_client.py \
    --text "今天心情很好！" \
    --ref-audio /path/to/ref.wav \
    --emo-audio /path/to/happy.wav

# IndexTTS-2.5 Japanese request
python examples/online_serving/text_to_speech/indextts2/speech_client.py \
    --model-version 2.5 \
    --model /path/to/indextts-2.5 \
    --lang ja \
    --text "こんにちは、音声合成のテストです。" \
    --ref-audio /path/to/ref.wav

# IndexTTS-2.5 native speed control
python examples/online_serving/text_to_speech/indextts2/speech_client.py \
    --model-version 2.5 \
    --model /path/to/indextts-2.5 \
    --speed 2.0 \
    --text "这是两倍速度的语音合成测试。" \
    --ref-audio /path/to/ref.wav

Notes

• Output: 22.05 kHz mono WAV.
• Provide ref_audio on the documented raw request path, or pass voice only when it names an uploaded audio voice; neither version provides a built-in text-only preset voice.
• IndexTTS-2.5 request controls lang and text_normalization are carried in extra_params; the client does this for --model-version 2.5.
• IndexTTS-2.5 accepts the public speed request field in [0.5, 2.0]: 2.0 generates shorter, faster speech and 0.5 generates longer, slower speech. This is native Stage 1 duration control, so serving does not apply the generic playback-speed adjustment a second time. IndexTTS-2 does not use this control.
• IndexTTS-2.5 accepts language codes such as zh, en, zhen (mixed Chinese/English), ja, and yue. Mandarin is a vLLM-Omni convenience alias for zh. Japanese (ja) uses fugashi tokenization and produces audio, but it does not automatically expand numbers, dates, or percentages; callers should first write those inputs as readable Japanese text.
• A request seed controls Stage 0 AR sampling and per-request CFM noise. Different concurrent batch compositions do not guarantee a bit-identical waveform.
• IndexTTS-2.5 Stage 0 uses plain vLLM sampling and does not reproduce the official default num_beams=3 beam search. For parity comparisons, run upstream with num_beams=1; output quality can differ from the official beam-search result.
• Emotion params (emo_audio, emo_text, emo_vector, emo_alpha, use_emo_text, use_random) are passed via the extra_params field. Official precedence is use_emo_text > emo_vector > emo_audio > same emotion as the speaker reference.
• IndexTTS-2.5 uses vllm_omni/deploy/indextts2_5.yaml with the official code-only Stage 0 to Stage 1 contract.
• For IndexTTS-2.5, set MODEL to the local native bundle; asset discovery accepts its nested checkpoints/ layout.

---

Audio8 TTS Preview

0.6B DualAR TTS at 44.1 kHz, 11 languages, zero-shot voice cloning.

Prerequisites

None beyond the base install: the neural audio codec is implemented in tree and its weights (codec.pth) ship with the checkpoint.

Launch

vllm serve Audio8/Audio8-TTS-Preview-0.6b --omni --port 8092
# or:
./audio8_tts/run_server.sh

The deploy config auto-loads from vllm_omni/deploy/audio8_tts.yaml (HF model_type is arktts). Do not pass --trust-remote-code: vllm-omni registers its own arktts config, and transformers would otherwise prefer the checkpoint's remote code and bypass it.

Text-only synthesis

curl -X POST http://localhost:8092/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Welcome to Audio8 TTS.",
        "response_format": "wav"
    }' --output output.wav

Voice cloning

curl -X POST http://localhost:8092/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Welcome to Audio8 TTS.",
        "ref_audio": "https://example.com/reference.wav",
        "ref_text": "The exact transcript of the reference recording."
    }' --output cloned.wav

ref_audio also accepts a data:audio/wav;base64,... URL. ref_text is mandatory whenever ref_audio is present and must match the recording. Uploading a voice via POST /v1/audio/voices lets the server reuse the encoded reference codes across requests (voice: "<name>").

Streaming

python audio8_tts/speech_client.py --text "Welcome to Audio8 TTS." --stream --output out.pcm
ffplay -f s16le -ar 44100 -ac 1 out.pcm

Gradio demo

./audio8_tts/run_gradio_demo.sh          # server + demo
python audio8_tts/gradio_demo.py --api-base http://localhost:8092   # demo only

Benchmark

export VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=44100   # required: harness defaults to 24 kHz
vllm bench serve --omni --host 127.0.0.1 --port 8092 \
    --model Audio8/Audio8-TTS-Preview-0.6b \
    --backend openai-audio-speech --endpoint /v1/audio/speech \
    --dataset-name seed-tts-text --dataset-path benchmarks/build_dataset/seed_tts_smoke \
    --seed-tts-locale en --num-prompts 20 --num-warmups 2 \
    --max-concurrency 1 --request-rate inf \
    --percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun

Measured on 1x H20 shared with an unrelated job holding ~80% SM, so treat these as a lower bound. 20 prompts from seed_tts_smoke/en, mean audio 4.5 s, deploy defaults:

	HF reference (batch=1)	vllm-omni c=1	c=4	c=8
RTF (mean)	1.008	0.19	0.30	0.54
Throughput (req/s)	0.23	1.18	2.84	2.93
E2E latency (mean, ms)	4375	850	1349	2361
Time to first packet (mean, ms)	n/a (no streaming)	63	105	1124
Underrun p99 / continuity OK	n/a	0 s / 100%	0 s / 100%	0 s / 100%

~5.3x lower per-request RTF than the upstream reference implementation at concurrency 1, ~12.8x aggregate throughput at concurrency 8.

Stage-0 max_num_seqs: first-packet vs throughput

This subsection is operator tuning guidance, not a claim this PR optimizes anything: the shipped deploy default is unchanged (max_num_seqs: 4). The numbers below are a single run on the same contended H20 as above (one unrelated job holding ~80% SM, no repeats), so read them as directional, not as measured speedups. Same setup, varying stage 0's max_num_seqs (stage 1 stays at 1):

stage-0 max_num_seqs	client concurrency	req/s	RTF	TTFP mean / p99 (ms)	underrun p99 (s)
4 (default)	1	1.18	0.19	63 / 67	0.00
4	4	2.84	0.30	105 / 148	0.00
4	8	2.93	0.54	1124 / 1642	0.00
8	1	1.16	0.19	64 / 70	0.00
8	4	2.88	0.30	107 / 148	0.00
8	8	3.72	0.40	247 / 733	0.22
8	16	3.91	0.65	1366 / 2530	0.29

Reading of this:

• Below the limit (concurrency <= max_num_seqs) the setting is irrelevant, as expected — c=1 and c=4 are identical for both.
• At c=8, raising the limit to 8 admits every request instead of queueing half of them: in this single run, roughly +27% throughput and ~4.5x lower first-packet latency.
• Throughput saturates around 3.7-3.9 req/s (~17 s of audio per second). Past c=8 you only buy queueing delay.
• The cost of admitting more streams is a 0.22-0.29 s worst-case buffer deficit in 1 of 20 requests (Streaming continuity OK rate 95%), because stage 1 runs max_num_seqs: 1 and the codec window round-robins across streams. Client-side prebuffering (the bundled Gradio demo holds 2 chunks) absorbs that; raising stage 1's max_num_seqs is the untested knob if you need both.

The shipped default stays at 4 because every measured point there had zero underrun. Raise it to 8 if first-packet latency under load matters more than a rare sub-300 ms gap.

Notes

• Output: 44.1 kHz mono; ~21.5 codec frames per second.
• No built-in speaker presets. Omit voice for a random timbre, or clone one.
• max_new_tokens caps generated codec frames (1 frame ~= 46 ms).
• Context is 2048 packed text+audio positions (max_model_len: 2048).
• Do not send temperature: 0: greedy decoding is degenerate for this checkpoint and truncates the utterance. The deploy defaults (0.7 / top_k 50 / top_p 0.9) mirror generation_config.json.

---

Fish Speech S2 Pro

4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec, which is vendored in vLLM-Omni (no extra packages required).

Kvcache attention fast path

Fish Speech S2 Pro uses a Triton decode-only kvcache attention fast path by default on CUDA builds. Set VLLM_OMNI_FISH_KVCACHE_ATTN=0 to disable it, or VLLM_OMNI_FISH_KVCACHE_ATTN=required to fail fast if the fast path cannot be installed.

# Verify fast path availability.
python - <<'PY'
from vllm_omni.attention import fish_kvcache_attn

print(fish_kvcache_attn.is_available())
print(fish_kvcache_attn.load_error())
PY

# Optional: disable the runtime fast path.
export VLLM_OMNI_FISH_KVCACHE_ATTN=0

Launch

vllm serve fishaudio/s2-pro --omni --port 8091
# or:
./fish_speech/run_server.sh

The deploy config auto-loads from vllm_omni/deploy/fish_qwen3_omni.yaml (the HF model_type on the fishaudio checkpoint is fish_qwen3_omni).

Voice cloning

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is a cloned voice.",
        "voice": "default",
        "ref_audio": "https://example.com/reference.wav",
        "ref_text": "Transcript of the reference audio."
    }' --output cloned.wav

CLI client

cd examples/online_serving/text_to_speech/fish_speech
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Hello world" --stream --output output.pcm

Gradio demo

./fish_speech/run_gradio_demo.sh             # launches server + Gradio
python fish_speech/gradio_demo.py --api-base http://localhost:8091  # if server already running

Notes

• Output: 44.1 kHz mono.
• Streaming PCM player command must use -r 44100.

---

Ming-omni-tts

Dense 0.5B two-stage TTS served through /v1/audio/speech. Ming uses the standard speech endpoint plus structured controls in instructions, voice, language, ref_audio, ref_text, and speaker_embedding.

Launch

bash examples/online_serving/text_to_speech/ming_tts/run_server.sh

Equivalent manual command:

vllm-omni serve inclusionAI/Ming-omni-tts-0.5B \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --host 0.0.0.0 --port 8091 \
    --enforce-eager --omni

Sending requests

python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \
    --text "你好，这是 Ming 在线语音合成测试。"

Structured dialect control:

python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \
    --text "我觉得社会企业同个人都有责任" \
    --instruction-json '{"方言":"广粤话"}' \
    --ref-audio /path/to/yue_prompt.wav

Zero-shot cloning:

python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \
    --text "我们的愿景是构建未来服务业的数字化基础设施，为世界带来更多微小而美好的改变。" \
    --ref-audio /path/to/10002287-00000094.wav \
    --ref-text "在此奉劝大家别乱打美白针。"

Notes

• run_curl.sh keeps a small sanity subset; use the Ming README for the broader request cookbook.
• Online serving is speech-shaped today; music-only bgm and text-to-audio tta remain offline examples.
• Full request details live in ming_tts/README.md.

---

Ming-flash-omni-TTS

Standalone talker-only deployment of Ming-flash-omni-2.0. Voice is controlled through caption text passed via instructions.

Launch

# from repo root
bash examples/online_serving/text_to_speech/ming_flash_omni_tts/run_server.sh

Equivalent manual command:

vllm serve Jonathan1909/Ming-flash-omni-2.0 \
    --deploy-config vllm_omni/deploy/ming_flash_omni_tts.yaml \
    --host 0.0.0.0 --port 8091 \
    --trust-remote-code --omni

Sending requests

python examples/online_serving/text_to_speech/ming_flash_omni_tts/speech_client.py \
    --text "我们当迎着阳光辛勤耕作，去摘取，去制作，去品尝，去馈赠。" \
    --output ming_online.wav

ASMR-style caption via instructions:

python examples/online_serving/text_to_speech/ming_flash_omni_tts/speech_client.py \
    --text "我会一直在这里陪着你，直到你慢慢、慢慢地沉入那个最温柔的梦里……好吗？" \
    --instructions "这是一种ASMR耳语，属于一种旨在引发特殊感官体验的创意风格。这个女性使用轻柔的普通话进行耳语，声音气音成分重。" \
    --output ming_online_asmr.wav

Notes

• Server uses use_zero_spk_emb=True and the cookbook decode defaults (max_decode_steps=200, cfg=2.0, sigma=0.25, temperature=0.0). For other caption fields (语速, 基频, IP, BGM, etc.) or overriding decode args, use the offline example where additional_information is set explicitly.
• This is the online counterpart of examples/offline_inference/text_to_speech/ming_flash_omni_tts/.
• For multimodal Ming-flash-omni online serving, see examples/online_serving/ming_flash_omni/.

---

MOSS-TTS Local Transformer v1.5

For a single H200, the optional moss_tts_local_h200.yaml deployment places both the talker and codec on logical GPU 0:

CUDA_VISIBLE_DEVICES=0 VLLM_OMNI_EVENT_DRIVEN_ORCH=1 \
    vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
    --omni --trust-remote-code \
    --deploy-config vllm_omni/deploy/moss_tts_local_h200.yaml \
    --disable-log-stats

Run from the repository root and select an available physical GPU with CUDA_VISIBLE_DEVICES. The command enables event-driven orchestration and disables detailed per-request statistics logging to reduce CPU overhead at high concurrency. Remove --disable-log-stats when those statistics are needed; keep these settings identical when comparing performance. This preset configures 256 request slots per stage, a 32 GiB talker KV cache, talker CUDA Graph buckets through 512 scheduled tokens, and codec batch buckets through 256. It requires more than 80 GiB of GPU memory; request capacity also depends on input and generated lengths. The default moss_tts_local.yaml remains available for smaller deployments.

The preset enables the optional codec backend with hf_overrides.codec_attention_backend: triton on stage 1. It preserves the streaming ring-cache mask and uses BF16 attention with 64-dimensional heads; other attention shapes use PyTorch SDPA. Set the backend to sdpa to use the default attention implementation. Codec terminal tails share execution only when they already map to the same padded CUDA Graph; returned audio retains each request's actual length.

Voice cloning requests use ref_audio and ref_text. For streaming output, set stream: true, stream_format: "audio", and response_format: "pcm"; the native PCM format is 48 kHz, stereo, signed 16-bit little-endian. Compute audio throughput as PCM bytes / (48000 * 2 * 2) / elapsed seconds, and compare the same requests, concurrency, and reference-cache state.

MOSS-TTS-Nano

Single-stage 0.1B AR LM + MOSS-Audio-Tokenizer-Nano codec at 48 kHz mono. Every request must include ref_audio; there are no built-in speaker presets.

The OpenAI-schema voice and ref_text fields are accepted but ignored — voice_clone does not consume a transcript, and upstream's continuation mode (the only path that accepts prompt_text) emits near-silent output, so it is not exposed here. Sample reference clips ship in the upstream repo under assets/audio/.

Launch

vllm serve OpenMOSS-Team/MOSS-TTS-Nano --omni --port 8091
# or:
./moss_tts_nano/run_server.sh

The deploy config at vllm_omni/deploy/moss_tts_nano.yaml auto-loads; no --deploy-config, --trust-remote-code, or --enforce-eager flags are needed.

Sending requests

# One-off fetch of a sample reference clip; cache under XDG_CACHE_HOME.
REF_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/moss-tts-nano"
mkdir -p "$REF_DIR"
REF_WAV="$REF_DIR/zh_1.wav"
[ -s "$REF_WAV" ] || curl -L -o "$REF_WAV" https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS-Nano/main/assets/audio/zh_1.wav
REF_AUDIO=$(base64 -w 0 "$REF_WAV")

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d "{
        \"input\": \"你好，这是语音合成测试。\",
        \"ref_audio\": \"data:audio/wav;base64,${REF_AUDIO}\",
        \"response_format\": \"wav\"
    }" --output output.wav

Streaming PCM

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d "{
        \"input\": \"Hello, streaming output from MOSS-TTS-Nano.\",
        \"ref_audio\": \"data:audio/wav;base64,${REF_AUDIO}\",
        \"stream\": true,
        \"stream_format\": \"audio\",
        \"response_format\": \"pcm\"
    }" --no-buffer | play -t raw -r 48000 -e signed -b 16 -c 1 -

Gradio demo

# Option 1: launch server + Gradio together
./moss_tts_nano/run_gradio_demo.sh

# Option 2: server already running
python moss_tts_nano/gradio_demo.py --api-base http://localhost:8091

Then open http://localhost:7860 in your browser.

Notes

• Output is 48 kHz mono PCM (the upstream tokenizer is internally stereo at 48 kHz; the wrapper averages to mono before reaching the engine).
• Standard /v1/audio/speech request shape: input, ref_audio (base64 data URL), response_format, stream, max_new_tokens. The voice and ref_text fields from the OpenAI schema are accepted but ignored.

---

OmniVoice

Zero-shot multilingual TTS (600+ languages). Online serving currently exposes auto voice only; voice cloning and voice design are available offline.

Prerequisites

huggingface-cli download k2-fsa/OmniVoice

Voice cloning (offline) needs transformers>=5.3.0; auto voice works with transformers>=4.57.0.

Launch

vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code
# or:
./omnivoice/run_server.sh

CLI client

cd examples/online_serving/text_to_speech/omnivoice
# Text-only (auto voice)
python speech_client.py --text "Hello, how are you?"

# Language hint
python speech_client.py --text "Bonjour, comment allez-vous?" --language French
# Voice cloning (reference audio + optional ref_text)
python speech_client.py \
--text "Bonjour, comment allez-vous?" \
--ref-audio /path/to/ref_audio.wav \
--ref-text "Bonjour, comment allez-vous?"

# Style instruction (voice design-style control)
python speech_client.py \
--text "Bonjour, comment allez-vous?" \
--language French \
--instructions "loud voice"

# Deterministic output with seed parameter
python speech_client.py --text "Hello, how are you?" --seed 42

The client supports --api-base, --model, --text, --response-format, --language, --voice, --ref-audio, --ref-text, --instructions, --seed, and --output.

Qwen3-TTS

Three model variants exposed via separate checkpoints:

Variant	HF repo	Use
CustomVoice	Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice	Predefined speakers (vivian, ryan, …) with optional style instructions
VoiceDesign	Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign	Natural-language voice style description
Base	Qwen/Qwen3-TTS-12Hz-1.7B-Base	Voice cloning from a reference audio

Each variant ships smaller 0.6B companions where available.

Launch

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
# or:
./qwen3_tts/run_server.sh                # default: CustomVoice
./qwen3_tts/run_server.sh VoiceDesign
./qwen3_tts/run_server.sh Base

Executor backend

Single-GPU serves now default to the uniproc executor (lower IPC overhead, the Base cloning use case from #2603 / #2604). vllm_omni/deploy/qwen3_tts.yaml is the only Qwen3-TTS deploy config; pass --deploy-config <path> to override.

To opt out of chunked streaming, pass --no-async-chunk — the pipeline auto-dispatches to the end-to-end codec processor.

Sending requests

# CustomVoice with a predefined speaker
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --text "今天天气真好" \
    --speaker ryan \
    --instructions "用开心的语气说"

# VoiceDesign with a style description
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
    --task-type VoiceDesign \
    --text "哥哥，你回来啦" \
    --instructions "体现撒娇稚嫩的萝莉女声，音调偏高"

# Base voice cloning
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --task-type Base \
    --text "Hello, this is a cloned voice" \
    --ref-audio /path/to/reference.wav \
    --ref-text "Original transcript of the reference audio"

Voices endpoint

List available voices, or upload a custom one for Base cloning:

# List
curl http://localhost:8091/v1/audio/voices

# Upload
curl -X POST http://localhost:8091/v1/audio/voices \
    -F "audio_sample=@/path/to/voice_sample.wav" \
    -F "consent=user_consent_id" \
    -F "name=custom_voice_1" \
    -F "ref_text=The exact transcript of the audio sample." \
    -F "speaker_description=warm narrator"

For Qwen3-TTS, uploaded voices are Base voice-cloning inputs and require a Base checkpoint. When a request names an uploaded voice, the server infers task_type="Base". Built-in presets such as vivian and ryan remain CustomVoice speakers and require a CustomVoice checkpoint.

Precomputed custom voices

For reused Base voice-cloning speakers, precompute the reference artifacts once and load them at server startup. Precomputed voices use the same Base task and checkpoint-matching rules as uploaded voices:

python qwen3_tts/precompute_custom_voice.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --voice-name alice \
    --ref-audio /path/to/reference.wav \
    --ref-text "Original transcript of the reference audio" \
    --mode icl \
    --output-dir /path/to/custom_voices

--mode icl stores both speaker_embedding and ref_code; --mode xvec stores only the speaker embedding. Add the output directory to a deploy config:

custom_voice_dir: /path/to/custom_voices

Then start the server with that config and call the Speech API with only the voice name:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --deploy-config /path/to/qwen3_tts_custom_voice.yaml

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{"input":"Hello from a precomputed voice.","voice":"alice","task_type":"Base"}' \
    --output alice.wav

Streaming PCM

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Raw PCM streaming requires stream_format="audio", response_format="pcm", and async_chunk: true in the deploy config (default in qwen3_tts.yaml). speed is not supported when streaming.

Streaming WebSocket

The /v1/audio/speech/stream endpoint accepts text incrementally and, by default, synthesizes the buffered text as one continuous request on input.done:

python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you? I am fine."
python qwen3_tts/streaming_speech_client.py --text "..." --simulate-stt --stt-delay 0.1

For per-sentence audio (including Indic danda ।) before input.done:

python qwen3_tts/streaming_speech_client.py \
    --text "नमस्ते। कैसे हो?" \
    --split-granularity sentence

input.done flushes without closing, so repeating --text synthesizes several utterances over one connection:

python qwen3_tts/streaming_speech_client.py \
    --text "First utterance." \
    --text "Second utterance, same connection."

To receive word-level timestamps, launch the server with a forced aligner:

vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --omni \
    --deploy-config vllm_omni/deploy/qwen3_tts.yaml \
    --trust-remote-code \
    --forced-aligner Qwen/Qwen3-ForcedAligner-0.6B

Then request PCM JSON sidecar chunks:

python qwen3_tts/streaming_speech_client.py \
    --text "Hello world. How are you?" \
    --stream-audio \
    --response-format pcm \
    --word-timestamps

The client writes one PCM file per sentence and a matching sentence_XXX_timestamps.json sidecar.

Non-streaming requests can also ask for timestamps: pass "word_timestamps": true to POST /v1/audio/speech and read the X-Word-Timestamps response header (JSON list of {word, start_ms, end_ms}, ASCII-escaped). Past 4 KB the header is replaced by X-Word-Timestamps-Omitted: oversize; bytes=<n>; limit=4096 and the audio still returns — use the WebSocket path for long transcripts.

To see the alignment instead of reading a JSON sidecar, run the word-timestamp Gradio demo (server must be launched with --forced-aligner):

python qwen3_tts/word_timestamps_demo.py --api-base http://localhost:8091

Each sentence's audio plays in an <audio> element while its text is rendered as inline word spans; the current word highlights as audio.currentTime crosses each start_ms. The Stop (barge-in) button cuts playback and reports the last-spoken word, useful for the voice-agent barge-in case.

Gradio demos

./qwen3_tts/run_gradio_demo.sh                              # CustomVoice (default)
./qwen3_tts/run_gradio_demo.sh --task-type VoiceDesign
./qwen3_tts/run_gradio_demo.sh --task-type Base

Speaker embedding interpolation

qwen3_tts/speaker_embedding_interpolation.py blends two predefined speakers' embeddings to produce intermediate voices. See the script for usage.

Batch client

qwen3_tts/batch_speech_client.py issues many concurrent requests for throughput measurement.

Notes

• Base voice cloning has uniproc-vs-mp tradeoffs depending on per-request reference audio cost; see the executor-backend section above.
• With async chunking, Qwen3-TTS Base voice cloning sends the full reference context in the first Code2Wav packet, then caches that prefix on the Code2Wav stage for follow-up chunks in the same request.
• vllm_omni/deploy/qwen3_tts.yaml is the default deploy config (loaded by HF model_type); per-stage runtime overrides are available via --stage-N-<field> <value>.

---

VoxCPM2

Single-stage native AR TTS at 48 kHz.

Launch

vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000

Deploy config auto-loads from vllm_omni/deploy/voxcpm2.yaml. Pass --deploy-config <path> to override or --stage-N-<field> <value> for per-stage runtime tweaks.

Sending requests

# Zero-shot synthesis
python voxcpm2/openai_speech_client.py --text "Hello, this is VoxCPM2."

# Voice cloning
python voxcpm2/openai_speech_client.py \
    --text "This should sound like the reference speaker." \
    --ref-audio /path/to/reference.wav

The ref_audio field accepts local file paths (auto-base64), HTTP URLs, or data:audio/wav;base64,... data URIs.

Precomputed custom voices

For repeated VoxCPM2 speakers, precompute the prompt cache and load it through custom_voice_dir:

python voxcpm2/precompute_custom_voice.py \
    --model openbmb/VoxCPM2 \
    --voice-name alice \
    --ref-audio /path/to/reference.wav \
    --mode ref_continuation \
    --prompt-text "Original transcript of the reference audio" \
    --output-dir /path/to/custom_voices

Add the output directory to the deploy config:

custom_voice_dir: /path/to/custom_voices

After startup, /v1/audio/voices lists alice, and /v1/audio/speech can use voice="alice" without sending ref_audio.

Gradio demo (gapless streaming via AudioWorklet)

python voxcpm2/gradio_demo.py

Uses an AudioWorklet-based player adapted from the Qwen3-TTS demo for gap-free playback. Raw PCM audio is streamed from the OpenAI Speech endpoint with stream=true and stream_format="audio".

---

Voxtral TTS

Voxtral-4B-TTS (Mistral). Uses the mistral_common SpeechRequest protocol; voice presets are model-specific.

Prerequisites

Latest mistral_common with SpeechRequest support:

pip install -e /path/to/mistral-common  # or upgrade from PyPI when available

Launch

vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091

Deploy config auto-loads from vllm_omni/deploy/voxtral_tts.yaml.

Gradio demo

python voxtral_tts/gradio_demo.py

The demo handles voice-preset selection and reference-audio upload. voxtral_tts/text_preprocess.py provides the text-normalization helpers used by the demo (also available for other clients).

Notes

• Voice presets are listed on the HF model card (mistralai/Voxtral-4B-TTS-2603).
• Voice cloning is gated upstream and may require a recent mistral_common.
• A standalone CLI client is not yet shipped; the gradio demo is the canonical reference for now.

Breeze-TTS-2

Two-stage AR TTS (T5Gemma2 + Qwen3 talker with a depth decoder → bundled Qwen3-TTS codec) at 24 kHz mono, from BreezeBlue. Four modes are selected automatically from the request fields: plain (input), voice design (instructions), clone (ref_audio+ref_text), and voice direction (reference trio + instructions).

Launch

vllm serve BreezeBlue/Breeze-TTS-2 --omni --port 8091
# or:
./breeze_tts_2/run_server.sh

The deploy config at vllm_omni/deploy/breeze_tts_2.yaml auto-loads (async-chunk streaming, 8-frame inter-stage chunks).

Sending requests

# Plain synthesis (speaker tags S0..S9, default S0)
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is Breeze TTS 2 on vLLM-Omni.",
        "voice": "S0",
        "response_format": "wav",
        "sample_rate": 24000
    }' --output breeze_plain.wav

# Voice direction (clone a reference, then steer the delivery)
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "We need to discuss what happened last night.",
        "ref_audio": "file:///path/to/reference.wav",
        "ref_text": "The exact transcript of the reference audio.",
        "instructions": "Speak slowly with a restrained, serious tone.",
        "response_format": "wav"
    }' --output breeze_direction.wav

Streaming PCM

curl -N -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Streaming output from Breeze TTS 2.",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm",
        "sample_rate": 24000
    }' --output breeze_stream.pcm

Notes

• Only sample_rate=24000 is accepted; speed must be 1.0.
• guidance_scale/cfg_scale must be 1.0; negative_prompt is rejected (CFG companion support is a follow-up).
• instructions without ref_audio is voice design; with ref_audio+ref_text it is voice direction.
• See recipes/BreezeBlue/Breeze-TTS-2.md for the full recipe and the offline example under examples/offline_inference/text_to_speech/breeze_tts_2/.