0 Fork 0

Files

T

NancyFyongandNancyFyong e74fe3c6e8 [Model] Add Audio8 TTS Preview 0.6B (DualAR, 44.1 kHz codec) (#6157 )

Signed-off-by: NancyFyong <NancyFyong@users.noreply.github.com>
Co-authored-by: NancyFyong <NancyFyong@users.noreply.github.com>

2026-09-08 19:00:25 -07:00

29 KiB

Raw Blame History

Text-To-Speech

vLLM-Omni supports several autoregressive TTS models. They share a mostly common CLI shape (--text, --ref-audio, --ref-text, plus an output-path flag — --output-dir for most, --output for OmniVoice) and live together in this hub. Each model has its own subdirectory containing a single end2end.py script; this README is the single doc entry point.

For online serving, see examples/online_serving/text_to_speech/. For the full list of supported architectures across all modalities, see Supported Models.

Supported Models

Model	HuggingFace repo	Stages	Voice cloning	Streaming	Special modes	Sample rate
Audio8 TTS Preview	`Audio8/Audio8-TTS-Preview-0.6b`	dual-AR	✓	✓	11 languages	44.1 kHz
CosyVoice3	`FunAudioLLM/Fun-CosyVoice3-0.5B-2512`	2 (talker + code2wav)	✓	✓	—	24 kHz
Fish Speech S2 Pro	`fishaudio/s2-pro`	dual-AR	✓	✓	—	44.1 kHz
Gepard-1.0	`nineninesix/gepard-1.0`	single (native AR + NanoCodec)	— (zero-shot; cloning WIP)	— (serving WIP)	zero-shot	22.05 kHz
GLM-TTS	`zai-org/GLM-TTS`	2 (AR + DiT)	✓ (required)	✓	—	24 kHz
Ming-omni-tts	`inclusionAI/Ming-omni-tts-0.5B`	2 (AR + audio VAE)	✓	✓	style / IP / dialect / TTA / podcast	44.1 kHz
Ming-flash-omni-TTS	`Jonathan1909/Ming-flash-omni-2.0`	single (talker only)	— (caption-controlled)	—	style / IP / basic captions	44.1 kHz
MOSS-TTS-Nano	`OpenMOSS-Team/MOSS-TTS-Nano`	single (AR + codec)	✓ (required)	✓	voice_clone, continuation	48 kHz
OmniVoice	`k2-fsa/OmniVoice`	2 (gen + dec)	✓	—	voice design, language hint	24 kHz
Qwen3-TTS	`Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}`	2 (talker + code2wav)	✓ (Base)	✓	3 task variants	24 kHz
VoxCPM2	`openbmb/VoxCPM2`	single (native AR)	✓	✓ (online)	continuation	48 kHz
dots.tts	`dots-studio/dots.tts-soar`	single (native AR)	— (text-only)	✓ (online)	no-reference synthesis	48 kHz
IndexTTS-2	`IndexTeam/IndexTTS-2`	2 (AR talker + S2Mel DiT + BigVGAN)	✓ (required)	—	emotion control (`--emo-audio`, `--emo-text`, `--emo-vector`)	22.05 kHz
IndexTTS-2.5	native `checkpoints/` bundle	2 (AR talker + EnhancedCodec + S2Mel DiT + BigVGAN)	✓ (required)	—	multilingual (`--lang`) + emotion control	22.05 kHz
Voxtral TTS	`mistralai/Voxtral-4B-TTS-2603`	varies	✓	✓	voice presets	24 kHz

Common Quick Start

Most models share this invocation shape:

python examples/offline_inference/text_to_speech/<model>/end2end.py \
    --text "Hello, this is a test." \
    --ref-audio /path/to/reference.wav \
    --ref-text  "Transcript of the reference audio."

--ref-audio and --ref-text are optional (text-only synthesis works without them) and must be provided together for voice cloning. The exotic scripts — Qwen3-TTS, Voxtral TTS, CosyVoice3 — accept additional model-specific flags documented in their per-model section below. Qwen3-TTS in particular uses its own argparse surface (--query-type, --audio-path, etc.) and does not follow the common shape; see its section.

Audio8 TTS Preview

0.6B DualAR text-to-speech model (Audio8) with its own 44.1 kHz neural audio codec: a slow AR transformer predicts one semantic token per codec frame, and a 4-layer fast AR predicts that frame's remaining 9 codebooks.

Prerequisites

No extra packages: the codec architecture ships in tree and its weights (codec.pth) come with the checkpoint.

Quick start

python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
    --text "Welcome to Audio8 TTS, a compact multilingual text to speech model."

Voice cloning

python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
    --text "Welcome to Audio8 TTS." \
    --ref-audio /path/to/reference.wav \
    --ref-text  "The exact transcript of the reference recording."

The reference transcript must match what is actually spoken in the clip; a mismatch degrades both stability and speaker similarity.

Streaming

python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
    --text "Welcome to Audio8 TTS." --streaming

Notes

Output: 44.1 kHz mono WAV; the codec emits ~21.5 frames/s (2048 samples per frame).
Context is 2048 packed text+audio positions, so max_model_len is 2048.
Greedy decoding (temperature: 0) is degenerate for this checkpoint — it ends the utterance almost immediately. Keep the deploy defaults (temperature 0.7, top_k 50, top_p 0.9), which also match generation_config.json.
Repetition-Aware Sampling is applied to the semantic token: a repeat inside the last 10 frames is re-drawn from a flatter distribution instead of being masked out.
--num-prompts 4 smoke-tests concurrency (stage 0 runs max_num_seqs: 4).

Measured behaviour (1x H20, shared GPU)

--streaming on a 58-character English sentence: TTFA ~0.74 s (4-frame first chunk), 7 chunks, 1.55 s wall for 3.16 s of audio (RTF ~0.49). Treat these as smoke-test magnitudes, not a benchmark.

CosyVoice3

2-stage TTS pipeline (talker + code2wav) at 24 kHz.

Prerequisites

uv pip install -e .
# Includes soundfile, onnxruntime, x-transformers, einops via requirements.

Download the model snapshot:

from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512',
                  local_dir='pretrained_models/Fun-CosyVoice3-0.5B')

If your downloaded checkpoint lacks config.json, add it:

{
    "model_type": "cosyvoice3",
    "architectures": ["CosyVoice3Model"]
}

This is required because AutoConfig.register("cosyvoice3", CosyVoice3Config) only registers the class mapping; the loader still reads model_type from config.json to select the class.

Quick start

python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
    --model pretrained_models/Fun-CosyVoice3-0.5B \
    --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN

Voice cloning

If --ref-audio is omitted, the script downloads the upstream zero_shot_prompt.wav from the CosyVoice repo into the current directory. To use your own clip, pass --ref-audio /path/to/reference.wav, and modify --prompt-text correspondingly.

python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
    --model pretrained_models/Fun-CosyVoice3-0.5B \
    --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN \
    --ref-audio /path/to/reference.wav \
    --prompt-text "You are a helpful assistant.<|endofprompt|>Trascript in your ref audio clip"

Streaming

Streaming is enabled by default via async_chunk: true in vllm_omni/deploy/cosyvoice3.yaml. Pass --no-async-chunk on vllm serve to switch to the legacy synchronous path.

Notes

Stage 0 (talker) emits speech tokens; stage 1 (code2wav) runs flow matching + HiFiGAN to synthesize waveform.
Deploy config auto-loads from vllm_omni/deploy/cosyvoice3.yaml based on HF model_type. Pass --deploy-config <path> to override.

GLM-TTS

2-stage TTS pipeline (AR + DiT flow-matching) at 24 kHz. Every request requires reference audio and its transcript for zero-shot voice cloning.

Quick start

python examples/offline_inference/text_to_speech/glm_tts/end2end.py \
    --model zai-org/GLM-TTS \
    --text "你好，这是语音合成测试。" \
    --ref-audio /path/to/reference.wav \
    --ref-text "这是参考音频的文本内容。" \
    --output-dir ./output

Architecture

Text → [Stage 0: AR] → Speech Tokens → [Stage 1: DiT + HiFT] → Audio (24 kHz)
        (Llama-based)    (32k vocab)      (Flow Matching)

Notes

--ref-audio and --ref-text are required together; GLM-TTS does not support text-only synthesis.
Reference audio should be 3-10 seconds.
First run may be slow due to lazy loading of WhisperVQ tokenizer and CampPlus ONNX speaker embedder.
Default sampling: temperature=1.0, top_k=25, top_p=0.8 (RAS method).
The --model path should point to the repository root (not llm/ subdirectory).

Fish Speech S2 Pro

4B dual-AR text-to-speech model from FishAudio with the DAC codec at 44.1 kHz. No extra packages are required; the DAC codec is vendored in vLLM-Omni.

Quick start

python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
    --text "Hello, this is a test of the Fish Speech text to speech system."

Voice cloning

python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
    --text "Hello, this is a cloned voice." \
    --ref-audio /path/to/reference.wav \
    --ref-text  "Transcript of the reference audio."

Streaming

python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
    --text "Hello, this is a streaming test." \
    --streaming

Streaming requires async_chunk: true in the deploy config.

Notes

Output: 44.1 kHz mono WAV.
DAC codec weights (codec.pth) are loaded lazily from the model directory.

Ming-omni-tts

Dense 0.5B two-stage TTS pipeline (AR + flow + audio VAE) at 44.1 kHz. The example covers style, IP voice, music-only generation, text-to-audio events, emotion, dialect, zero-shot cloning, podcast, speech+BGM, and speech+environment-sound cases.

Quick start

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case style \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager

Voice cloning

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case zero_shot \
    --ref-audio /path/to/reference.wav \
    --ref-text "在此奉劝大家别乱打美白针。" \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager

Streaming

python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
    --case basic \
    --ref-audio /path/to/reference.wav \
    --streaming \
    --deploy-config vllm_omni/deploy/ming_tts.yaml \
    --enforce-eager

Notes

style, ip, bgm, and tta do not require reference audio.
Reference-audio cases use --ref-audio; zero_shot also requires --ref-text.
podcast uses multiple references via --ref-audio-paths.
Full case details live in ming_tts/README.md.

Ming-flash-omni-TTS

Standalone talker-only deployment of Ming-flash-omni-2.0 at 44.1 kHz. Voice is controlled through caption fields (风格 / IP / 语速/基频/音量) rather than reference audio.

Prerequisites

The example calls into vllm_omni.model_executor.models.ming_flash_omni.prompt_utils for the default prompt and instruction builder; no extra pip install on top of the base vLLM-Omni install.

Quick start

python examples/offline_inference/text_to_speech/ming_flash_omni_tts/end2end.py --case style

Cases

# ASMR-style whisper (caption-driven)
python examples/offline_inference/text_to_speech/ming_flash_omni_tts/end2end.py --case style

# IP voice (preset character voice via caption)
python examples/offline_inference/text_to_speech/ming_flash_omni_tts/end2end.py --case ip

# Basic speed/pitch/volume control
python examples/offline_inference/text_to_speech/ming_flash_omni_tts/end2end.py --case basic

Override the default text per case with --text, write to a custom path with --output.

Notes

Talker-only deployment — for the multimodal Ming-flash-omni example, see examples/offline_inference/ming_flash_omni/.
Deploy config: vllm_omni/deploy/ming_flash_omni_tts.yaml (single GPU, enforce_eager, max_num_seqs: 1).
Decode defaults from the Ming cookbook: max_decode_steps=200, cfg=2.0, sigma=0.25, temperature=0.0, use_zero_spk_emb=True.

MOSS-TTS-Nano

Single-stage 0.1B AR LM + MOSS-Audio-Tokenizer-Nano codec at 48 kHz mono (mixed down from upstream stereo). ZH / EN / JA. Every request requires a reference clip via --ref-audio.

No built-in speaker presets. --ref-audio is required on every call. Default --mode voice_clone matches upstream's recommended workflow; --mode continuation is exposed for completeness but upstream's continuation-with-prompt path emits very short / near-silent output, so it is rarely useful in practice. Sample reference clips ship in the upstream repo under assets/audio/ (e.g. zh_1.wav, en_2.wav, jp_2.wav).

Quick start

# Fetch a sample reference clip (one-off, user-scoped cache).
REF_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/moss-tts-nano"
mkdir -p "$REF_DIR"
[ -s "$REF_DIR/zh_1.wav" ] || \
    curl -L -o "$REF_DIR/zh_1.wav" https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS-Nano/main/assets/audio/zh_1.wav

python examples/offline_inference/text_to_speech/moss_tts_nano/end2end.py \
    --text "你好，这是MOSS-TTS-Nano的语音合成演示。" \
    --ref-audio "$REF_DIR/zh_1.wav"

The first run downloads OpenMOSS-Team/MOSS-TTS-Nano and OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano from Hugging Face.

Reproducible runs

python examples/offline_inference/text_to_speech/moss_tts_nano/end2end.py \
    --text "Deterministic test." \
    --ref-audio "$REF_DIR/en_2.wav" \
    --seed 42

Notes

Output: 48 kHz mono WAV (the tokenizer is internally stereo at 48 kHz; the wrapper averages to mono before reaching the engine).
Deploy config: vllm_omni/deploy/moss_tts_nano.yaml (auto-loaded; override with --deploy-config).
Default --max-new-frames 375 ≈ 14 s of audio; raise for longer outputs.
--ref-text is rejected in voice_clone mode and required only with --mode continuation.
Run --help for the full sampling-knob surface (--audio-temperature, --audio-top-k, --audio-top-p, --text-temperature).

OmniVoice

Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design).

Prerequisites

huggingface-cli download k2-fsa/OmniVoice

Voice cloning requires transformers>=5.3.0. Auto and design modes work with transformers>=4.57.0.

Quick start (auto voice)

python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --model k2-fsa/OmniVoice \
    --text "Hello, this is a test."

Voice cloning

python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --model k2-fsa/OmniVoice \
    --text "Hello, this is a test." \
    --ref-audio ref.wav \
    --ref-text  "This is the reference transcription."

Voice design

python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --model k2-fsa/OmniVoice \
    --text "Hello, this is a test." \
    --instruct "female, low pitch, british accent"

Language hint

python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --model k2-fsa/OmniVoice \
    --text "你好，这是一个测试。" \
    --lang zh

Seed for Reproducibility

python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --model k2-fsa/OmniVoice \
    --text "Hello, this is a test." \
    --seed 42

Notes

Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking.
Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz.

Qwen3-TTS

3-task-variant TTS with 24 kHz output. Has its own argparse surface (this script does not follow the common --text / --ref-audio shape).

Prerequisites

For ROCm builds, replace onnxruntime with onnxruntime-rocm:

pip uninstall onnxruntime
pip install onnxruntime-rocm

Task variants

CustomVoice: predefined speaker (speaker ID) with optional style instruction.
VoiceDesign: text + descriptive instruction designs a new voice.
Base: voice cloning from reference audio + transcript.

# Single sample
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type VoiceDesign
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base

# Base with a custom reference audio (Qwen3-TTS uses --audio-path, not --ref-audio):
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
    --query-type Base --audio-path /path/to/reference.wav

# Base variant has an additional mode flag:
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag icl       # default
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag xvec_only # x_vector_only_mode

# Batch (multiple prompts in one run)
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice --use-batch-sample

Streaming

python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
    --query-type CustomVoice \
    --streaming \
    --output-dir /tmp/out_stream

Streaming requires async_chunk: true in the deploy config.

Word Timestamps

Word-level timestamps are currently a serving-path feature: launch vllm-omni serve with --forced-aligner and request word_timestamps (see examples/online_serving/text_to_speech/README.md). An offline example is not provided in this release.

Batched decoding

The Code2Wav stage supports batched decoding through the SpeechTokenizer. Pass multiple prompts via --txt-prompts and set --batch-size accordingly. To raise max_num_seqs on either stage, create an overlay based on vllm_omni/deploy/qwen3_tts.yaml and pass it with --deploy-config:

python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
    --query-type CustomVoice \
    --txt-prompts examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt \
    --batch-size 4 \
    --deploy-config /path/to/qwen3_tts_batched.yaml

--batch-size must match a CUDA-graph capture size (1, 2, 4, 8, 16…).

Notes

Run --help for the full argument surface.
See qwen3_tts/end2end.py for the prompt-length-estimation logic the Talker uses.

Gepard-1.0

Single-stage native AR TTS at 22.05 kHz. Pipeline: Qwen3.5 backbone → 32 FSQ codebook heads (one frame/step) → NeMo NanoCodec. Backbone runs under vLLM paged attention; the 32-head sampling + learned embedding feedback ride the native-AR runner hooks. enforce_eager (CUDA graph is a perf follow-up).

Prerequisites

The NanoCodec decoder needs NeMo (NVIDIA Open Model License), installed separately:

uv pip install "nemo_toolkit[tts]==2.7.3" \
    --overrides <(printf '%s\n' "transformers>=5.5.3" "huggingface-hub>=1.0")

The overrides are what make that command work. nemo_toolkit[tts] declares transformers~=4.57.0, so a plain install downgrades the version requirements/common.txt pins and takes huggingface-hub back to 0.x with it — after which import vllm fails. Both packages are named here rather than left to resolve on their own: transformers 5.x happens to require huggingface-hub 1.x, but relying on that would make the second repair an accident of the first. --constraint cannot be used instead, since it narrows a version range rather than overriding a dependency a package declares, and the two sets never intersect. NeMo 2.7.3 decodes correctly against transformers 5.x, so nothing is lost by holding it there.

(The same clash is why this is documented here rather than declared in pyproject.toml: an extra resolves together with the base dependencies, so vllm-omni[gepard] would be unsatisfiable rather than opt-in.)

On a host whose CUDA toolkit cannot build kernels for the GPU — no nvcc/ninja, or a consumer Blackwell (sm_120) card — also set VLLM_USE_FLASHINFER_SAMPLER=0. vLLM's sampler JIT-compiles through FlashInfer on first use, and where that build fails the engine dies inside profile_run long after every import has succeeded, which reads as a model failure rather than an environment one. The PyTorch sampler it falls back to costs nothing here.

Quick start (zero-shot, default voice)

python examples/offline_inference/text_to_speech/gepard/end2end.py \
    --text "Hello, this is a Gepard demo."

Voice cloning

Not yet — PR1 is zero-shot only (the learned null_prefix default voice). Reference-audio cloning (ref_compressor speaker prefix) is a follow-up PR.

Notes

Output: 22.05 kHz mono WAV.
First run downloads two checkpoints, not one: the model itself, and the NanoCodec decoder it names in codec_id (nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps by default). Both are fetched automatically; expect the first startup to be correspondingly slower.
Offline only for now; the /v1/audio/speech adapter is a follow-up PR.
Deploy config: vllm_omni/deploy/gepard.yaml (the example's default; copy it and pass --deploy-config to change it).
Generation length and reproducibility are stage settings, not script flags. One output token is one audio frame, so max_tokens in the YAML is the frame budget; seed makes the in-model 32-head sampling reproducible. The script deliberately passes no SamplingParams: one supplied by a caller replaces the stage defaults wholesale rather than merging, which would drop the pipeline's stop token and run every request to max_tokens.
Text length: short texts are repeated internally to match the training layout (the checkpoint's text_repetition block); the upper bound is the stage's max_model_len, enforced by the engine. Empty text is rejected rather than voiced.
VLLM_GEPARD_GREEDY=1 swaps the 32-head Gumbel-max sampling for argmax, for reproducible comparisons that do not depend on a seed.

VoxCPM2

Single-stage native AR TTS at 48 kHz. Pipeline: feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE.

Prerequisites

pip install voxcpm
# or, for a local source checkout:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm

Quick start

python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
    --model openbmb/VoxCPM2 \
    --text "Hello, this is a VoxCPM2 demo."

Voice cloning

Pass a reference audio for isolated cloning, or both --ref-audio + --ref-text for prompt continuation:

python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
    --text "Hello, this is a voice clone demo." \
    --ref-audio /path/to/reference.wav \
    --ref-text  "Transcript of the reference audio."

Streaming

Streaming is exposed through the online OpenAI Speech API (stream=true). See examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py for an AudioWorklet-based gapless streaming player; the offline end2end.py script does not expose a streaming path.

Notes

Output: 48 kHz mono WAV.
Deploy config: vllm_omni/deploy/voxcpm2.yaml (auto-loaded by HF model_type).

dots.tts

Single-stage native AR TTS at 48 kHz (rednote-hilab). Pipeline: Qwen2.5-1.5B base LM → DiT (10-step Euler flow matching) → patch_encoder AR loopback → AudioVAE (streaming decode). Same "vLLM-native base LM + side-path computation" pattern as VoxCPM2, with a plain Qwen2 backbone instead of MiniCPM4 and no FSQ / residual LM stage.

Quick start

python examples/offline_inference/text_to_speech/dots_tts/end2end.py \
    --model dots-studio/dots.tts-soar \
    --text "Hello, this is a test of dots TTS running on vLLM Omni."

Voice cloning

Not wired in this release. Generation is text-only and does not consume reference audio, a named voice, or a precomputed speaker embedding. The CAM++ x-vector speaker encoder weights load, but end2end.py has no --ref-audio/--ref-text flags yet.

Streaming

The AudioVAE decoder emits incremental audio through its internal streaming path (init_stream_state / stream_step / stream_flush). Online streaming is exposed through the OpenAI-compatible /v1/audio/speech endpoint with stream=true; see the online serving guide. The offline end2end.py script returns the consolidated waveform.

Notes

Output: 48 kHz mono WAV.
Deploy config: vllm_omni/deploy/dots_tts.yaml (auto-loaded by HF model_type).
Checkpoints: dots-studio/dots.tts-soar is the validated default. dots.tts-base shares the same architecture but is unvalidated in this repo. dots.tts-mf (MeanFlow, 2-4 step) is not supported yet.
Known limitation: no CUDA graph capture and no batched side-path computation yet, so concurrent requests do not currently scale (each request's DiT Euler steps run serially). See recipes/rednote-hilab/dots.tts.md for details and the roadmap.

IndexTTS-2 and IndexTTS-2.5

Both versions use a 2-stage TTS pipeline and produce 22.05 kHz mono audio. Stage 0 is a GPT AR talker. IndexTTS-2 Stage 1 uses RepCodec semantic embeddings, GPT latent, S2Mel CFM/DiT, and BigVGAN. IndexTTS-2.5 replaces RepCodec with EnhancedCodec and uses the official code-only path (use_gpt_latent=false). Every request requires reference audio for zero-shot voice cloning.

Quick start

IndexTTS-2:

python examples/offline_inference/text_to_speech/indextts2/end2end.py \
    --model IndexTeam/IndexTTS-2 \
    --text "你好，这是一个语音合成测试。" \
    --ref-audio /path/to/reference.wav

IndexTTS-2 uses its existing frontend and does not require the IndexTTS-2.5 text dependencies. Before running IndexTTS-2.5, install its optional frontend dependencies:

pip install 'vllm-omni[indextts2]'

IndexTTS-2.5 with a native repository/lab bundle whose model root contains checkpoints/:

python examples/offline_inference/text_to_speech/indextts2/end2end.py \
    --model /path/to/indextts-2.5 \
    --model-version 2.5 \
    --lang ja \
    --text "こんにちは、音声合成のテストです。" \
    --ref-audio /path/to/reference.wav

Emotion control

# Emotion from reference audio
python examples/offline_inference/text_to_speech/indextts2/end2end.py \
    --model IndexTeam/IndexTTS-2 \
    --text "今天天气真好！" \
    --ref-audio /path/to/ref.wav \
    --emo-audio /path/to/happy.wav

# Emotion from 8-dim vector (happy angry sad afraid disgusted melancholy surprised calm)
python examples/offline_inference/text_to_speech/indextts2/end2end.py \
    --model IndexTeam/IndexTTS-2 \
    --text "今天天气真好！" \
    --ref-audio /path/to/ref.wav \
    --emo-vector 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

# Emotion from text description
python examples/offline_inference/text_to_speech/indextts2/end2end.py \
    --model IndexTeam/IndexTTS-2 \
    --text "今天天气真好！" \
    --ref-audio /path/to/ref.wav \
    --emo-text "happy and excited"

Notes

--ref-audio is required — neither version provides text-only synthesis.
IndexTTS-2.5 Stage 0 (AR Talker) generates mel codes from text + reference audio using plain vLLM sampling. It does not reproduce the official default num_beams=3 beam search. For parity comparisons, run upstream with num_beams=1; output quality can differ from the official beam-search result.
IndexTTS-2.5 accepts language codes such as zh, en, zhen (mixed Chinese/English), ja, and yue. Mandarin is a vLLM-Omni convenience alias for zh. Japanese (ja) uses fugashi tokenization and produces audio, but it does not automatically expand numbers, dates, or percentages; write those inputs as readable Japanese text before calling the model. Use --no-text-normalization only when the input is already normalized.
A seed controls Stage 0 AR sampling and per-request CFM noise. Different concurrent batch compositions do not guarantee a bit-identical waveform.
Stage 1 (semantic codec + S2Mel + BigVGAN): CFM/DiT converts the complete semantic-code sequence to waveform.
Deploy configs: vllm_omni/deploy/indextts2.yaml and indextts2_5.yaml.
async_chunk=false is intentional for IndexTTS-2.5 correctness: S2Mel consumes the completed semantic sequence.

Voxtral TTS

Voxtral-4B-TTS (Mistral). Has its own argparse surface; uses voice presets and the mistral_common SpeechRequest protocol.

Prerequisites

Latest mistral_common with SpeechRequest support:

pip install -e /path/to/mistral-common  # or upgrade from PyPI when available

Quick start (voice preset)

python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
    --write-audio --voice cheerful_female \
    --model mistralai/Voxtral-4B-TTS-2603 \
    --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"

Voice cloning (capability gated upstream)

python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
    --write-audio \
    --model mistralai/Voxtral-4B-TTS-2603 \
    --text "This is a test message." \
    --ref-audio path/to/reference_audio.wav

Streaming + concurrency

python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
    --num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
    --model mistralai/Voxtral-4B-TTS-2603 \
    --text "..."

Available voice presets are listed on the HF model card (mistralai/Voxtral-4B-TTS-2603).

Notes

--num-prompts N replicates the prompt for performance measurement.
--concurrency M requires --streaming and must evenly divide --num-prompts.
Run --help for the full argument surface.

29 KiB Raw Blame History

Text-To-Speech

Supported Models

Common Quick Start

Audio8 TTS Preview

Prerequisites

Quick start

Voice cloning

Streaming

Notes

Measured behaviour (1x H20, shared GPU)

CosyVoice3

Prerequisites

Quick start

Voice cloning

Streaming

Notes

GLM-TTS

Quick start

Architecture

Notes

Fish Speech S2 Pro

Quick start

Voice cloning

Streaming

Notes

Ming-omni-tts

Quick start

Voice cloning

Streaming

Notes

Ming-flash-omni-TTS

Prerequisites

Quick start

Cases

Notes

MOSS-TTS-Nano

Quick start

Reproducible runs

Notes

OmniVoice

Prerequisites

Quick start (auto voice)

Voice cloning

Voice design

Language hint

Seed for Reproducibility

Notes

Qwen3-TTS

Prerequisites

Task variants

Streaming

Word Timestamps

Batched decoding

Notes

Gepard-1.0

Prerequisites

Quick start (zero-shot, default voice)

Voice cloning

Notes

VoxCPM2

Prerequisites

Quick start

Voice cloning

Streaming

Notes

dots.tts

Quick start

Voice cloning

Streaming

Notes

IndexTTS-2 and IndexTTS-2.5

Quick start

Emotion control

Notes

Voxtral TTS

Prerequisites

Quick start (voice preset)

Voice cloning (capability gated upstream)

Streaming + concurrency

29 KiB

Raw Blame History