vllm-omni/examples/online_serving/qwen3_tts

Qwen3-TTS

🛠️ Installation

Please refer to README.md

Supported Models

Model	Task Type	Description
Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice	CustomVoice	Predefined speaker voices with optional style control
Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign	VoiceDesign	Natural language voice style description
Qwen/Qwen3-TTS-12Hz-1.7B-Base	Base	Voice cloning from reference audio
Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice	CustomVoice	Smaller/faster variant
Qwen/Qwen3-TTS-12Hz-0.6B-Base	Base	Smaller/faster variant for voice cloning

Gradio Demo

Two interactive Gradio demos are available, both supporting all 3 task types:

Demo	File	Transport	Streaming Quality
Standard	gradio_demo.py	HTTP chunked	May have small gaps between chunks
FastRTC	gradio_fastrtc_demo.py	WebRTC	Gapless streaming (requires pip install fastrtc)

# Option 1: Launch server + Standard Gradio together
./run_gradio_demo.sh                                # CustomVoice (default)
./run_gradio_demo.sh --task-type VoiceDesign        # VoiceDesign
./run_gradio_demo.sh --task-type Base               # Voice cloning

# Option 2: If server is already running
python gradio_demo.py --api-base http://localhost:8000

# Option 3: FastRTC demo (gapless streaming)
pip install fastrtc
python gradio_fastrtc_demo.py --api-base http://localhost:8000

Then open http://localhost:7860 in your browser.

Run examples (Qwen3-TTS)

Launch the Server

The default deploy config is located at vllm_omni/deploy/qwen3_tts.yaml and is loaded automatically by the model registry — no --deploy-config flag needed for default use. Platform-specific deltas (NPU, ROCm, XPU) are merged in automatically from the platforms: block of the same YAML based on the detected runtime.

# CustomVoice model (predefined speakers)
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --omni \
    --port 8091

# VoiceDesign model
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
    --omni \
    --port 8091

# Base model (voice cloning)
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --omni \
    --port 8091

If you have custom stage configs file, launch the server with command below

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --stage-configs-path /path/to/stage_configs_file \
    --omni \
    --port 8091

Sync vs async-chunk mode

Qwen3-TTS supports both chunked streaming (default, lower latency) and synchronous end-to-end modes from the same deploy YAML. The bundled qwen3_tts.yaml ships with async_chunk: true; flip with --no-async-chunk and the pipeline automatically dispatches to the end-to-end codec processor:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091 \
    --no-async-chunk

No variant YAML or extra flag is needed — the StagePipelineConfig on each stage declares both processor functions and the runtime picks based on the async_chunk: bool.

Alternatively, use the convenience script:

./run_server.sh                  # Default: CustomVoice model
./run_server.sh CustomVoice      # CustomVoice model
./run_server.sh VoiceDesign      # VoiceDesign model
./run_server.sh Base             # Base (voice clone) model

Send TTS Request

Get into the example folder

cd examples/online_serving/qwen3_tts

Send request via python

# CustomVoice: Use predefined speaker
python openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --text "你好，我是通义千问" \
    --voice vivian \
    --language Chinese

# CustomVoice with style instruction
python openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --text "今天天气真好" \
    --voice ryan \
    --instructions "用开心的语气说"

# VoiceDesign: Describe the voice style
python openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
    --task-type VoiceDesign \
    --text "哥哥，你回来啦" \
    --instructions "体现撒娇稚嫩的萝莉女声，音调偏高"

# Base: Voice cloning
python openai_speech_client.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --task-type Base \
    --text "Hello, this is a cloned voice" \
    --ref-audio /path/to/reference.wav \
    --ref-text "Original transcript of the reference audio"

The Python client supports the following command-line arguments:

• --api-base: API base URL (default: http://localhost:8091)
• --model (or -m): Model name/path (default: Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice)
• --task-type (or -t): TTS task type. Options: CustomVoice, VoiceDesign, Base
• --text: Text to synthesize (required)
• --voice: Speaker/voice name (default: vivian). Options: vivian, ryan, aiden, etc.
• --language: Language. Options: Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
• --instructions: Voice style/emotion instructions
• --ref-audio: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server.
• --ref-text: Reference audio transcript for voice cloning (Base task).
• --response-format: Audio output format (default: wav). Options: wav, mp3, flac, pcm, aac, opus
• --output (or -o): Output audio file path (default: tts_output.wav)

Send request via curl

# Simple TTS request
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English"
    }' --output output.wav

# With style instruction
curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "I am so excited!",
        "voice": "vivian",
        "instructions": "Speak with great enthusiasm"
    }' --output excited.wav

# List available voices in CustomVoice models
curl http://localhost:8091/v1/audio/voices

Using OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")

response = client.audio.speech.create(
    model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    voice="vivian",
    input="Hello, how are you?",
)

response.stream_to_file("output.wav")

Using Python httpx

import httpx

response = httpx.post(
    "http://localhost:8091/v1/audio/speech",
    json={
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English",
    },
    timeout=300.0,
)

with open("output.wav", "wb") as f:
    f.write(response.content)

API Reference

Voices Endpoint

GET /v1/audio/voices

List all available voices/speakers from the loaded model, including both built-in model voices and uploaded custom voices.

Response Example:

{
  "voices": ["vivian", "ryan", "custom_voice_1"],
  "uploaded_voices": [
    {
      "name": "custom_voice_1",
      "consent": "user_consent_id",
      "created_at": 1738660000,
      "file_size": 1024000,
      "mime_type": "audio/wav"
    }
  ]
}

POST /v1/audio/voices

Upload a new voice sample for voice cloning in Base task TTS requests.

Form Parameters:

• audio_sample (required): Audio file (max 10MB, supported formats: wav, mp3, flac, ogg, aac, webm, mp4)
• consent (required): Consent recording ID
• name (required): Name for the new voice
• ref_text (optional): Transcript of the audio. Enables in-context voice cloning (higher quality).
• speaker_description (optional): Free-form description of the voice (e.g. "warm narrator", "energetic presenter").

Response Example:

{
  "success": true,
  "voice": {
    "name": "custom_voice_1",
    "consent": "user_consent_id",
    "created_at": 1738660000,
    "mime_type": "audio/wav",
    "file_size": 1024000,
    "ref_text": "The exact transcript of the audio sample.",
    "speaker_description": "warm narrator"
  }
}

Fields ref_text and speaker_description are omitted when not provided at upload time.

Usage Example:

curl -X POST http://localhost:8000/v1/audio/voices \
  -F "audio_sample=@/path/to/voice_sample.wav" \
  -F "consent=user_consent_id" \
  -F "name=custom_voice_1" \
  -F "ref_text=The exact transcript of the audio sample." \
  -F "speaker_description=warm narrator"

Endpoint

POST /v1/audio/speech
Content-Type: application/json

This endpoint follows the OpenAI Audio Speech API format with additional Qwen3-TTS parameters.

Request Body

{
    "input": "Text to synthesize",
    "voice": "vivian",
    "response_format": "wav",
    "task_type": "CustomVoice",
    "language": "Auto",
    "instructions": "Optional style instructions",
    "ref_audio":  "HTTP URL, base64 data URL, or file:// URI for voice cloning",
    "ref_text": "Reference audio transcript",
    "x_vector_only_mode": false,
    "max_new_tokens": 2048
}

Note: The model field is optional when serving a single model, as the server already knows which model is loaded.

Response

Returns binary audio data with appropriate Content-Type header (e.g., audio/wav).

Parameters

OpenAI Standard Parameters

Parameter	Type	Default	Description
input	string	required	Text to synthesize
model	string	server's model	Model to use (optional, should match server if specified)
voice	string	"vivian"	Speaker name (e.g., vivian, ryan, aiden)
response_format	string	"wav"	Audio format: wav, mp3, flac, pcm, aac, opus
speed	float	1.0	Playback speed (0.25-4.0, not supported with stream=true)

vLLM-Omni Extension Parameters

Parameter	Type	Default	Description
task_type	string	"CustomVoice"	Task: CustomVoice, VoiceDesign, or Base
language	string	"Auto"	Language (see supported languages below)
instructions	string	""	Voice style/emotion instructions
max_new_tokens	int	2048	Maximum tokens to generate
initial_codec_chunk_frames	int	null	Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load.
stream	bool	false	Stream raw PCM chunks as they are decoded (requires response_format="pcm")

Supported languages: Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian

Voice Clone Parameters (Base task)

Parameter	Type	Required	Description
ref_audio	string	Yes	Reference audio (HTTP URL, base64 data URL, or file:// URI with --allowed-local-media-path)
ref_text	string	No	Transcript of reference audio (for ICL mode)
x_vector_only_mode	bool	No	Use speaker embedding only (no ICL)

Streaming

Set stream=true with response_format="pcm" to receive raw PCM audio chunks as they are decoded (one chunk per Code2Wav window, default 25 frames; configurable in the stage config):

curl -X POST http://localhost:8091/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English",
        "stream": true,
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -

Constraints:

• stream=true requires response_format="pcm" (raw 16-bit signed PCM, 24 kHz mono).
• speed adjustment is not supported when streaming.
• Requires the server stage config to have async_chunk: true (default in qwen3_tts.yaml).

Streaming Text Input (WebSocket)

The /v1/audio/speech/stream WebSocket endpoint accepts text incrementally, buffers and splits it at sentence boundaries, and generates audio per sentence.

When stream_audio=true, each sentence is emitted as audio.start, one or more binary PCM frames, and audio.done.

Quick Start

python streaming_speech_client.py \
    --text "Hello world. How are you? I am fine."

python streaming_speech_client.py \
    --text "Hello world. How are you? I am fine." \
    --simulate-stt --stt-delay 0.1

WebSocket Protocol

Client -> Server:

{"type": "session.config", "voice": "Vivian", "task_type": "CustomVoice", "language": "Auto", "split_granularity": "sentence", "stream_audio": true, "response_format": "pcm"}
{"type": "input.text", "text": "Hello, how are you? "}
{"type": "input.done"}

Server -> Client:

{"type": "audio.start", "sentence_index": 0, "sentence_text": "Hello, how are you?", "format": "pcm", "sample_rate": 24000}
// binary PCM frame(s)
{"type": "audio.done", "sentence_index": 0, "total_bytes": 96000, "error": false}
{"type": "session.done", "total_sentences": 1}

Choosing an Execution Backend: Uniproc vs Multiprocessing

Qwen3-TTS stage configs support two execution backends controlled by the distributed_executor_backend engine arg. The performance tradeoff between them is both hardware- and task-dependent, so there is no single best default (see #2603, #2604 for the full investigation).

Backend	Stage config setting	Behaviour
Uniproc (default, world_size=1)	distributed_executor_backend omitted	Both stages run inside the orchestrator process. Avoids IPC serialisation, D2H copies, and msgpack overhead between stages.
Multiprocessing	distributed_executor_backend: "mp"	Each stage runs in its own subprocess. The Talker can continue decoding while Code2Wav runs the vocoder in parallel, improving pipeline utilisation under concurrency.

Note: When distributed_executor_backend is omitted and world_size=1, vLLM automatically uses the uniproc executor. When world_size > 1, it defaults to mp.

When uniproc wins

The uniproc path eliminates inter-process data transfer (D2H copies, msgpack serialisation/deserialisation, tensor detaching). This matters most when per-request processing is heavy relative to autoregressive decode.

The Base cloning task involves reference-audio encoding on every request, making IPC overhead a larger fraction of total cost. Qwen3-Omni shows a similar pattern.

When multiprocessing (mp) wins

For lighter per-request workloads, process-level parallelism between the Talker and Code2Wav stages dominates.

CustomVoice is lighter per-request (no reference audio encoding), so the process-level parallelism of mp outweighs its serialisation cost at concurrency ≥ 4.

How to switch

To use the uniproc executor on a single-GPU setup, pass the qwen3_tts_uniproc.yaml stage config:

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --omni \
    --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_uniproc.yaml \
    --port 8091

Limitations

• Single request: Batch processing is not yet optimized for online serving.

Troubleshooting

1. TTS model did not produce audio output: Ensure you're using the correct model variant for your task type (CustomVoice task → CustomVoice model, etc.)
2. Connection refused: Make sure the server is running on the correct port
3. Out of memory: Use smaller model variant (Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice) or reduce --gpu-memory-utilization
4. Unsupported speaker: Use /v1/audio/voices to list available voices for the loaded model
5. Voice clone fails: Ensure you're using the Base model variant for voice cloning