0 Fork 0

Files

T

Sy03 dd765fbf64 [Model] Select CosyVoice3 streaming defaults by device capability (#8433 )

Signed-off-by: Sy03 <1370724210@qq.com>

2026-10-03 21:59:36 -07:00

34 KiB

Raw Blame History

Text-To-Speech (Online Serving)

Source https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech.

vLLM-Omni exposes TTS models through the OpenAI-compatible POST /v1/audio/speech endpoint, launched with vllm serve <model> --omni. Each TTS model has its own subdirectory containing client snippets, gradio demos, and helper scripts; this README is the single doc entry point for all of them.

For offline inference, see examples/offline_inference/text_to_speech. For the full list of supported architectures across all modalities, see Supported Models.

Supported Models

Model	HuggingFace repo	Voice cloning	Streaming	Voice presets / upload	Gradio demo
AuK / AuK-Flash	assembled local bundle	✓ (`ref_audio`)	HTTP audio after full generation	—	—
CosyVoice3	`FunAudioLLM/Fun-CosyVoice3-0.5B-2512`	✓ (`ref_audio`+`ref_text`)	✓ (PCM stream)	—	—
Fish Speech S2 Pro	`fishaudio/s2-pro`	✓ (`ref_audio`+`ref_text`)	✓ (PCM stream)	—	✓
Gepard-1.0	`nineninesix/gepard-1.0`	— (zero-shot default voice)	✓ (PCM / WAV stream)	`default` only	—
higgs-audio v2	`bosonai/higgs-audio-v2-generation-3B-base`	✓ (`ref_audio`+`ref_text`)	✓ (codec_streaming)	—	—
GLM-TTS	`zai-org/GLM-TTS`	✓ (`ref_audio`+`ref_text`, required)	✓ (PCM stream)	—	✓
OmniVoice	`k2-fsa/OmniVoice`	(offline only)	—	—	—
Qwen3-TTS	`Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}`	✓ (Base)	✓ (PCM + WebSocket)	✓ (presets + `/v1/audio/voices` upload)	✓ (standard + FastRTC)
VoxCPM2	`openbmb/VoxCPM2`	✓	✓ (AudioWorklet via gradio)	—	✓
Voxtral TTS	`mistralai/Voxtral-4B-TTS-2603`	✓ (gated upstream)	✓	✓ (presets)	✓

For offline inference of any of these models, see the offline TTS hub.

Common Quick Start

Launch the server (defaults shown — adjust --port, --gpu-memory-utilization, etc. as needed):

vllm serve <hf-repo-or-local-path> --omni --port 8091

Send a TTS request via curl:

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "default",
        "response_format": "wav"
    }' --output output.wav

Or via Python httpx:

import httpx

response = httpx.post(
    "http://localhost:8091/v1/audio/speech",
    json={
        "input": "Hello, how are you?",
        "voice": "default",
        "response_format": "wav",
    },
    timeout=300.0,
)
open("output.wav", "wb").write(response.content)

Or via the OpenAI SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
    model="<hf-repo>",
    voice="default",
    input="Hello, how are you?",
)
response.stream_to_file("output.wav")

Streaming PCM output (where supported) — set stream=true, stream_format="audio", and response_format="pcm":

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "default",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for Gepard, 24 kHz for the others).

For full request-shape documentation (all parameters, response formats, error codes), see the Speech API reference.

AuK / AuK-Flash

AuK serves speech generation and reference-voice synthesis at 24 kHz. Assemble a local bundle before serving:

python tools/prepare_auk_checkpoint.py \
    --auk-dir ckpts/AuK --qwen-dir ckpts/Qwen2.5-Omni-3B --out ckpts/auk-omni
MODEL=ckpts/auk-omni bash examples/online_serving/text_to_speech/auk/run_server.sh

AuK accepts the complete instruction after you fill in the local recipe template or the official AuK cookbook. Pass it in instructions and leave input empty. For instruct TTS, fill in Generate speech based on the following description: "{voice description}". The content to speak is: "{text}".; for zero-shot TTS, editing, enhancement and separation, use the corresponding complete instruction from the cookbook. Set duration_seconds for text-only generation; with ref_audio, omitting it keeps the source length.

For compatibility with clients that can only send input, AuK treats an input-only request as the complete instruction and logs a recommendation to move it to instructions with input="".

extra_params accepts num_inference_steps, guidance_scale, sway, t_grid, and vae_sample; seed is a top-level field.

python examples/online_serving/text_to_speech/auk/speech_client.py \
    --model ckpts/auk-omni \
    --instructions 'Generate speech based on the following description: "A calm young woman speaking warmly and slowly.". The content to speak is: "Welcome back, how was your day?".' \
    --duration-seconds 3 --seed 7

Complete-instruction editing also does not need a separate text argument:

python examples/online_serving/text_to_speech/auk/speech_client.py \
    --model ckpts/auk-omni --instructions "Keep pure speech voice, remove noise and reverberation." \
    --ref-audio noisy.wav --seed 7 --output clean.wav

The client accepts local reference files, URLs, and data URLs. --stream writes signed 16-bit 24 kHz PCM after the complete waveform is ready. See speech_client.py --help for editing and sampling options.

CosyVoice3

2-stage TTS (talker + flow-matching code2wav) at 24 kHz. Voice cloning only — every request needs ref_audio + ref_text; there are no built-in voice presets.

Prerequisites

huggingface-cli download FunAudioLLM/Fun-CosyVoice3-0.5B-2512

If your downloaded checkpoint lacks config.json, add one with {"model_type": "cosyvoice3", "architectures": ["CosyVoice3Model"]} (the loader reads model_type to select the class).

Launch

vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remote-code
# or:
./cosyvoice3/run_server.sh

Streaming is on by default. On CUDA Hopper GPUs with at least 140 GiB of device memory (H200), serving automatically selects cosyvoice3_packed_streaming_optimized_standard.yaml when nvidia-cuda-mps-control is available on PATH: AR32 / codec32, standard sampling, packed Flow, cached ISTFT, HiFT decode graphs, and local CUDA MPS. No --hf-overrides is needed. Other devices and environments without the MPS control tool use cosyvoice3.yaml with RAS sampling and AR8 / codec8.

The optimized profile is not a universal hardware configuration. Its packed kernels are Hopper-only; the talker's Model Runner V2 is unsupported on NPU/XPU, and CUDA MPS requires a local NVIDIA GPU and its control tool. Larger batches, 25-frame chunks, and graph allocations also require per-device validation on smaller or slower GPUs. Automatic selection retains the generic profile outside the validated device class; an explicit deploy configuration takes precedence.

The H200 default targets streaming throughput at C32/C64. It changes sampling from RAS to standard (temperature 0.7, top-p 0.8, top-k 20, repetition penalty 1.21); output lengths and quality can differ. HiFT captures a recurring exact shape on its third use, which adds latency to that request, and its graph pool needs memory headroom beyond the engine reservations. This profile is not a claim of optimal first-audio latency at low concurrency. Pass --deploy-config cosyvoice3.yaml to retain the previous configuration, or --deploy-config <path> to select another profile.

Pass --no-async-chunk (or NO_ASYNC_CHUNK=1 ./cosyvoice3/run_server.sh) for the legacy synchronous path; use --deploy-config cosyvoice3.yaml alongside it when selecting that path on H200.

Cross-request Stage-1 flow batching is enabled in the H200 profile. For the generic configuration, enable it in the server environment when concurrent requests should share a flow-estimator call:

export COSYVOICE3_BATCH_FLOW=1
vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remote-code

Batching preserves output lengths and streaming cache alignment, but the different GEMM shapes can produce small waveform differences compared with processing each request separately. Use --deploy-config cosyvoice3.yaml and leave COSYVOICE3_BATCH_FLOW unset (or set it to 0) when request-independent numerical behavior is required. Set COSYVOICE3_BATCH_FLOW_DEBUG=1 to log the observed group-size distribution and enable detailed Stage-1 profiler scopes; diagnostics are disabled by default to avoid per-step profiling overhead.

CLI client

The client defaults to the official upstream zero-shot prompt, so it runs without extra flags:

cd examples/online_serving/text_to_speech/cosyvoice3
python speech_client.py --text "收到好友从远方寄来的生日礼物。"

Pass your own reference clip and transcript for a different voice:

python speech_client.py --text "Hello, this is a cloned voice." \
    --ref-audio /path/to/reference.wav \
    --ref-text "Transcript of the reference audio."

Stream PCM instead of WAV:

python speech_client.py --text "Hello world" --stream --output output.pcm

The client supports --api-base, --model, --text, --ref-audio, --ref-text, --response-format, --stream, --output.

Notes

Stage 0 (talker) emits speech tokens; stage 1 (code2wav) runs flow matching + HiFiGAN to synthesize waveform.
Deploy config auto-loads the device-specific CosyVoice3 default described above. Pass --deploy-config <path> to override.
For offline inference and the end-to-end script, see the offline CosyVoice3 section.

Fish Speech S2 Pro

4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec, which is vendored in vLLM-Omni (no extra packages required).

Launch

vllm serve fishaudio/s2-pro --omni --port 8091
# or:
./fish_speech/run_server.sh

The deploy config auto-loads from vllm_omni/deploy/fish_qwen3_omni.yaml (the HF model_type on the fishaudio checkpoint is fish_qwen3_omni).

Voice cloning

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is a cloned voice.",
        "voice": "default",
        "ref_audio": "https://example.com/reference.wav",
        "ref_text": "Transcript of the reference audio."
    }' --output cloned.wav

CLI client

cd examples/online_serving/text_to_speech/fish_speech
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Hello world" --stream --output output.pcm

Gradio demo

./fish_speech/run_gradio_demo.sh             # launches server + Gradio
python fish_speech/gradio_demo.py --api-base http://localhost:8091  # if server already running

Notes

Output: 44.1 kHz mono.
Streaming PCM player command must use -r 44100.

Gepard-1.0

Single-stage native AR TTS at 22.05 kHz mono. Zero-shot only: omit voice or pass "default". Voice cloning from reference audio is not available yet.

Prerequisites

Same NeMo NanoCodec install as the offline Gepard section. On a host whose CUDA toolkit cannot build kernels — no nvcc/ninja, or a consumer Blackwell (sm_120) card — also export VLLM_USE_FLASHINFER_SAMPLER=0 before launch. That env is not a deploy-YAML field.

Launch

vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
    --stage-init-timeout 900 \
    --deploy-config vllm_omni/deploy/gepard.yaml
# or:
./gepard/run_server.sh

--stage-init-timeout 900 matches the online e2e fixture and the offline example. Serve defaults to 300s, which is often too short for a cold download of the talker plus NanoCodec.

The packaged vllm_omni/deploy/gepard.yaml must be passed with --deploy-config. The checkpoint self-identifies as qwen3_5_text, so omitting the YAML launches a diffusion fallback instead of the Gepard pipeline. The YAML sets async_chunk: false, max_num_seqs: 4, and currently pins seed: 42, so serving is deterministic by default until that YAML seed is removed. Pass an explicit per-request seed in tests and clients rather than depending on either default.

Sending requests

python examples/online_serving/text_to_speech/gepard/speech_client.py \
    --text "Hello, this is Gepard speaking."

python examples/online_serving/text_to_speech/gepard/speech_client.py \
    --text "Hello, this is Gepard speaking." --seed 7 --stream --output output.pcm

Gepard accepts input (required), voice ("default"), response_format, stream / stream_format, max_new_tokens, and seed. Unsupported fields include speed, language, instructions, task_type, ref_audio, ref_text, extra_params, and word_timestamps.

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, this is Gepard speaking.",
        "voice": "default",
        "seed": 7
    }' --output output.wav

Notes

Output: 22.05 kHz mono. max_new_tokens is a frame budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms at 21.5 fps; adapter bounds 1..4096).
Supported request fields: input (required), voice (default only), response_format (wav default; wav/pcm/flac/mp3 non-streaming; opus 400 because 22.05 kHz is not an Opus sample rate; streaming pcm/wav only), stream / stream_format, max_new_tokens, seed.
Unsupported: speed, extra_params (including temperature/top_p/top_k), ref_audio, ref_text, speaker_embedding, task_type, instructions, language, and word_timestamps.
Concurrent requests at max_num_seqs: 4 are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below max_num_seqs and treat preemption as out of scope until the platform fix lands.
Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.
For the request-field table, see the Speech API Gepard section.

higgs-audio v2

2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) that streams chunks back to the client.

Prerequisites

Voice clone uses HF's HiggsAudioV2TokenizerModel loaded from k2-fsa/OmniVoice/audio_tokenizer/ (~806 MB subdir; the boson-ai standalone tokenizer Hub repo's model.safetensors is the 3B talker LM, not the codec):

pip install -U "transformers>=5.3.0"

Launch

GPUS=6,7 PORT=8094 bash examples/online_serving/text_to_speech/higgs_audio_v2/run_server.sh

Deploy config auto-loads from vllm_omni/deploy/higgs_audio_v2.yaml.

Sending requests

# Plain TTS
python higgs_audio_v2/batch_speech_client.py \
    --base-url http://localhost:8094 \
    --output-dir /tmp/higgs_out \
    --prompts "Hello world." "The quick brown fox jumps over the lazy dog."

# Voice cloning — pass a reference clip and its transcript together
python higgs_audio_v2/batch_speech_client.py \
    --base-url http://localhost:8094 \
    --output-dir /tmp/higgs_clone \
    --ref-audio /path/to/reference.wav \
    --ref-text  "Exact transcript spoken in reference.wav." \
    --prompts "Hello, this is a cloned voice."

Notes

Output: 24 kHz mono.
--ref-text must be the real transcript of --ref-audio; mismatched text degrades cloned-voice quality.
Out of scope (rejected with explicit 4xx): multi-speaker [SPEAKERn] tags inside input, profile: text-only speaker descriptions, the ref_audio_in_system_message system-block variant, chunked long-form generation, and per-request voice / instructions / task_type / language / speed != 1.0 / x_vector_only_mode / speaker_embedding.

GLM-TTS

2-stage TTS (AR + DiT flow-matching) at 24 kHz. Every request requires ref_audio + ref_text.

Launch

vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091
# or:
bash examples/online_serving/text_to_speech/glm_tts/run_server.sh /path/to/GLM-TTS

Sending requests

# Voice cloning (required)
python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
    --text "你好，这是语音克隆测试。" \
    --ref-audio file:///path/to/ref.wav \
    --ref-text "这是参考音频的文本内容。"

# Custom format
python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
    --text "Hello, this is a voice cloning test." \
    --ref-audio file:///path/to/ref.wav \
    --ref-text "Transcript of the reference audio." \
    --response-format mp3 -o output.mp3

Gradio demo

bash examples/online_serving/text_to_speech/glm_tts/run_gradio_demo.sh

Notes

Output: 24 kHz mono WAV via HiFT vocoder.
ref_audio + ref_text are required together on every request. Reference audio should be 3-10 seconds.
Voice cloning feature extraction (WhisperVQ, CampPlus, mel) runs on the model side — no external dependency on the serving layer.

OmniVoice

Zero-shot multilingual TTS (600+ languages). Online serving currently exposes auto voice only; voice cloning and voice design are available offline.

Prerequisites

huggingface-cli download k2-fsa/OmniVoice

Voice cloning (offline) needs transformers>=5.3.0; auto voice works with transformers>=4.57.0.

Launch

vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code
# or:
./omnivoice/run_server.sh

CLI client

cd examples/online_serving/text_to_speech/omnivoice
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Bonjour, comment allez-vous?" --language French

The client supports --api-base, --model, --text, --response-format, --language, --output.

Notes

Voice cloning and voice design require offline inference; see the offline OmniVoice section.

Qwen3-TTS

Three model variants exposed via separate checkpoints:

Variant	HF repo	Use
CustomVoice	`Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`	Predefined speakers (`vivian`, `ryan`, …) with optional style instructions
VoiceDesign	`Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign`	Natural-language voice style description
Base	`Qwen/Qwen3-TTS-12Hz-1.7B-Base`	Voice cloning from a reference audio

Each variant ships smaller 0.6B companions where available.

Launch

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
# or:
./qwen3_tts/run_server.sh                # default: CustomVoice
./qwen3_tts/run_server.sh VoiceDesign
./qwen3_tts/run_server.sh Base

For a local deployment with multiple API frontend processes sharing one set of TTS stage engines, add --api-server-count:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --omni --api-server-count 2 --port 8091

This mode supports local EngineCore stages with one local process group per replica, including stages configured with multiple num_replicas. Headless, remote, intra-replica data-parallel, and Ray stage deployments are not supported. Runtime voice upload and deletion are disabled with multiple API frontends because their registries are process-local. Built-in voices, inline ref_audio, and voices restored from custom_voice_dir at startup are supported.

Executor backend

Single-GPU serves now default to the uniproc executor (lower IPC overhead, the Base cloning use case from #2603 / #2604). vllm_omni/deploy/qwen3_tts.yaml is the only Qwen3-TTS deploy config; pass --deploy-config <path> to override.

To opt out of chunked streaming, pass --no-async-chunk — the pipeline auto-dispatches to the end-to-end codec processor.

Tuning stage 1 `max_num_seqs` per task type

The bundled qwen3_tts.yaml ships stage 1 (Code2Wav) at max_num_seqs: 10, tuned for Base voice cloning: stage-1 lifetimes are long (~3 s/req), so admitting up to 10 concurrent codec sequences lets requests progress in parallel in the scheduler — ~2× TTFA p95 at c=4 / c=8 (1× H100, 1.7B-Base, seed-tts) at an 8–12 % audio-throughput cost.

CustomVoice / VoiceDesign have much shorter stage-1 lifetimes (~50–200 ms) and are TTFA-optimal at max_num_seqs: 1. Override the default when serving those task types:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni \
    --stage-overrides '{"1": {"max_num_seqs": 1}}'

Sending requests

# CustomVoice with a predefined speaker
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --text "今天天气真好" \
    --speaker ryan \
    --instructions "用开心的语气说"

# VoiceDesign with a style description
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
    --task-type VoiceDesign \
    --text "哥哥，你回来啦" \
    --instructions "体现撒娇稚嫩的萝莉女声，音调偏高"

# Base voice cloning
python qwen3_tts/openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --task-type Base \
    --text "Hello, this is a cloned voice" \
    --ref-audio /path/to/reference.wav \
    --ref-text "Original transcript of the reference audio"

Voices endpoint

List available voices, or upload a custom one for Base cloning:

# List
curl http://localhost:8091/v1/audio/voices

# Upload
curl -X POST http://localhost:8091/v1/audio/voices \
    -F "audio_sample=@/path/to/voice_sample.wav" \
    -F "consent=user_consent_id" \
    -F "name=custom_voice_1" \
    -F "ref_text=The exact transcript of the audio sample." \
    -F "speaker_description=warm narrator"

For Qwen3-TTS, uploaded voices are Base voice-cloning inputs and require a Base checkpoint. When a request names an uploaded voice, the server infers task_type="Base". Built-in presets such as vivian and ryan remain CustomVoice speakers and require a CustomVoice checkpoint.

The runtime upload and delete routes require a single API frontend; with --api-server-count > 1, use inline ref_audio or restore precomputed voices from custom_voice_dir at startup.

Precomputed custom voices

For reused Base voice-cloning speakers, precompute the reference artifacts once and load them at server startup. Precomputed voices use the same Base task and checkpoint-matching rules as uploaded voices:

python qwen3_tts/precompute_custom_voice.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --voice-name alice \
    --ref-audio /path/to/reference.wav \
    --ref-text "Original transcript of the reference audio" \
    --mode icl \
    --output-dir /path/to/custom_voices

--mode icl stores both speaker_embedding and ref_code; --mode xvec stores only the speaker embedding. Add the output directory to a deploy config:

custom_voice_dir: /path/to/custom_voices

Then start the server with that config and call the Speech API with only the voice name:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --deploy-config /path/to/qwen3_tts_custom_voice.yaml

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{"input":"Hello from a precomputed voice.","voice":"alice","task_type":"Base"}' \
    --output alice.wav

Streaming PCM

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English",
        "stream": true,
        "stream_format": "audio",
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Raw PCM streaming requires stream_format="audio", response_format="pcm", and async_chunk: true in the deploy config (default in qwen3_tts.yaml). speed is not supported when streaming.

Streaming WebSocket

The /v1/audio/speech/stream endpoint accepts text incrementally and synthesizes the buffered text as one continuous request on input.done:

python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you? I am fine."
python qwen3_tts/streaming_speech_client.py --text "..." --simulate-stt --stt-delay 0.1

input.done flushes without closing, so repeating --text synthesizes several utterances over one connection:

python qwen3_tts/streaming_speech_client.py \
    --text "First utterance." \
    --text "Second utterance, same connection."

Gradio demos

./qwen3_tts/run_gradio_demo.sh                              # CustomVoice (default)
./qwen3_tts/run_gradio_demo.sh --task-type VoiceDesign
./qwen3_tts/run_gradio_demo.sh --task-type Base

# FastRTC variant (gapless WebRTC streaming):
pip install fastrtc
python qwen3_tts/gradio_fastrtc_demo.py --api-base http://localhost:8000

Speaker embedding interpolation

qwen3_tts/speaker_embedding_interpolation.py blends two predefined speakers' embeddings to produce intermediate voices. See the script for usage.

Batch client

qwen3_tts/batch_speech_client.py issues many concurrent requests for throughput measurement.

Notes

Base voice cloning has uniproc-vs-mp tradeoffs depending on per-request reference audio cost; see the executor-backend section above.
With async chunking, Qwen3-TTS Base voice cloning sends the full reference context in the first Code2Wav packet, then caches that prefix on the Code2Wav stage for follow-up chunks in the same request.
vllm_omni/deploy/qwen3_tts.yaml is the default deploy config (loaded by HF model_type); per-stage runtime overrides are available via --stage-N-<field> <value>.
Under vocoder-bound overload (single-stream rtf_p99 ≥ 1 at the target concurrency), set active_stream_window: 2 at the top of the deploy yaml to cap simultaneously active Stage 1 streams. Off by default; trades TTFP for streaming continuity. See #3592 for the mechanism and tradeoff numbers.

VoxCPM2

Single-stage native AR TTS at 48 kHz.

Launch

vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000

Deploy config auto-loads from vllm_omni/deploy/voxcpm2.yaml. Pass --deploy-config <path> to override or --stage-N-<field> <value> for per-stage runtime tweaks.

Sending requests

# Zero-shot synthesis
python voxcpm2/openai_speech_client.py --text "Hello, this is VoxCPM2."

# Voice cloning
python voxcpm2/openai_speech_client.py \
    --text "This should sound like the reference speaker." \
    --ref-audio /path/to/reference.wav

The ref_audio field accepts local file paths (auto-base64), HTTP URLs, or data:audio/wav;base64,... data URIs.

Precomputed custom voices

For repeated VoxCPM2 speakers, precompute the prompt cache and load it through custom_voice_dir:

python voxcpm2/precompute_custom_voice.py \
    --model openbmb/VoxCPM2 \
    --voice-name alice \
    --ref-audio /path/to/reference.wav \
    --mode ref_continuation \
    --prompt-text "Original transcript of the reference audio" \
    --output-dir /path/to/custom_voices

Add the output directory to the deploy config:

custom_voice_dir: /path/to/custom_voices

After startup, /v1/audio/voices lists alice, and /v1/audio/speech can use voice="alice" without sending ref_audio.

Gradio demo (gapless streaming via AudioWorklet)

python voxcpm2/gradio_demo.py

Uses an AudioWorklet-based player adapted from the Qwen3-TTS demo for gap-free playback. Raw PCM audio is streamed from the OpenAI Speech endpoint with stream=true and stream_format="audio".

Voxtral TTS

Voxtral-4B-TTS (Mistral). Uses the mistral_common SpeechRequest protocol; voice presets are model-specific.

Prerequisites

Latest mistral_common with SpeechRequest support:

pip install -e /path/to/mistral-common  # or upgrade from PyPI when available

Launch

vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091

Deploy config auto-loads from vllm_omni/deploy/voxtral_tts.yaml.

Gradio demo

python voxtral_tts/gradio_demo.py

The demo handles voice-preset selection and reference-audio upload. voxtral_tts/text_preprocess.py provides the text-normalization helpers used by the demo (also available for other clients).

Notes

Voice presets are listed on the HF model card (mistralai/Voxtral-4B-TTS-2603).
Voice cloning is gated upstream and may require a recent mistral_common.
A standalone CLI client is not yet shipped; the gradio demo is the canonical reference for now.

Example materials

??? abstract "cosyvoice3/run_server.sh"

--8<-- "examples/online_serving/text_to_speech/cosyvoice3/run_server.sh"

??? abstract "cosyvoice3/speech_client.py"

--8<-- "examples/online_serving/text_to_speech/cosyvoice3/speech_client.py"

??? abstract "fish_speech/gradio_demo.py" py --8<-- "examples/online_serving/text_to_speech/fish_speech/gradio_demo.py" ??? abstract "fish_speech/run_gradio_demo.sh" sh --8<-- "examples/online_serving/text_to_speech/fish_speech/run_gradio_demo.sh" ??? abstract "fish_speech/run_server.sh" sh --8<-- "examples/online_serving/text_to_speech/fish_speech/run_server.sh" ??? abstract "fish_speech/speech_client.py" py --8<-- "examples/online_serving/text_to_speech/fish_speech/speech_client.py" ??? abstract "higgs_audio_v2/batch_speech_client.py" py --8<-- "examples/online_serving/text_to_speech/higgs_audio_v2/batch_speech_client.py" ??? abstract "higgs_audio_v2/run_server.sh" sh --8<-- "examples/online_serving/text_to_speech/higgs_audio_v2/run_server.sh" ??? abstract "glm_tts/gradio_demo.py" py --8<-- "examples/online_serving/text_to_speech/glm_tts/gradio_demo.py" ??? abstract "glm_tts/openai_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py" ??? abstract "glm_tts/run_gradio_demo.sh"sh --8<-- "examples/online_serving/text_to_speech/glm_tts/run_gradio_demo.sh" ??? abstract "glm_tts/run_server.sh"sh --8<-- "examples/online_serving/text_to_speech/glm_tts/run_server.sh" ??? abstract "omnivoice/run_server.sh"sh --8<-- "examples/online_serving/text_to_speech/omnivoice/run_server.sh" ??? abstract "omnivoice/speech_client.py"py --8<-- "examples/online_serving/text_to_speech/omnivoice/speech_client.py" ??? abstract "qwen3_tts/batch_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/batch_speech_client.py" ??? abstract "qwen3_tts/gradio_demo.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/gradio_demo.py" ??? abstract "qwen3_tts/openai_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/openai_speech_client.py" ??? abstract "qwen3_tts/run_gradio_demo.sh"sh --8<-- "examples/online_serving/text_to_speech/qwen3_tts/run_gradio_demo.sh" ??? abstract "qwen3_tts/run_server.sh"sh --8<-- "examples/online_serving/text_to_speech/qwen3_tts/run_server.sh" ??? abstract "qwen3_tts/speaker_embedding_interpolation.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/speaker_embedding_interpolation.py" ??? abstract "qwen3_tts/streaming_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/streaming_speech_client.py" ??? abstract "qwen3_tts/tts_common.py"py --8<-- "examples/online_serving/text_to_speech/qwen3_tts/tts_common.py" ??? abstract "voxcpm2/gradio_demo.py"py --8<-- "examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py" ??? abstract "voxcpm2/openai_speech_client.py"py --8<-- "examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py" ??? abstract "voxtral_tts/gradio_demo.py"py --8<-- "examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py" ??? abstract "voxtral_tts/text_preprocess.py"py --8<-- "examples/online_serving/text_to_speech/voxtral_tts/text_preprocess.py" ``````

34 KiB Raw Blame History Unescape Escape

Text-To-Speech (Online Serving)

Supported Models

Common Quick Start

AuK / AuK-Flash

CosyVoice3

Prerequisites

Launch

CLI client

Notes

Fish Speech S2 Pro

Launch

Voice cloning

CLI client

Gradio demo

Notes

Gepard-1.0

Prerequisites

Launch

Sending requests

Notes

higgs-audio v2

Prerequisites

Launch

Sending requests

Notes

GLM-TTS

Launch

Sending requests

Gradio demo

Notes

OmniVoice

Prerequisites

Launch

CLI client

Notes

Qwen3-TTS

Launch

Executor backend

Tuning stage 1 max_num_seqs per task type

Sending requests

Voices endpoint

Precomputed custom voices

Streaming PCM

Streaming WebSocket

Gradio demos

Speaker embedding interpolation

Batch client

Notes

VoxCPM2

Launch

Sending requests

Precomputed custom voices

Gradio demo (gapless streaming via AudioWorklet)

Voxtral TTS

Prerequisites

Launch

Gradio demo

Notes

Example materials

34 KiB

Raw Blame History

Tuning stage 1 `max_num_seqs` per task type