Support Qwen3 tts online serving (#968)
Signed-off-by: linyueqian <linyueqian@outlook.com>
This commit is contained in:
Yueqian Lin authored and GitHub committed
1 parent
4a3a1b6de8
commit
77ff875170
8 files changed
+772
-21
No files matched your search
@@ -0,0 +1,170 @@
|
||||
# Qwen3-TTS Online Serving
|
||||
|
||||
This directory contains examples for running Qwen3-TTS models with vLLM-Omni's online serving API.
|
||||
|
||||
## Supported Models
|
||||
|
||||
| Model | Task Type | Description |
|
||||
|-------|-----------|-------------|
|
||||
| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | CustomVoice | Predefined speaker voices with optional style control |
|
||||
| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description |
|
||||
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Start the Server
|
||||
|
||||
```bash
|
||||
# CustomVoice model (default)
|
||||
./run_server.sh
|
||||
|
||||
# Or specify task type
|
||||
./run_server.sh CustomVoice
|
||||
./run_server.sh VoiceDesign
|
||||
./run_server.sh Base
|
||||
```
|
||||
|
||||
### 2. Run the Client
|
||||
|
||||
```bash
|
||||
# CustomVoice: Use predefined speaker
|
||||
python openai_speech_client.py \
|
||||
--text "你好，我是通义千问" \
|
||||
--voice Vivian \
|
||||
--language Chinese
|
||||
|
||||
# CustomVoice with style instruction
|
||||
python openai_speech_client.py \
|
||||
--text "今天天气真好" \
|
||||
--voice Ryan \
|
||||
--instructions "用开心的语气说"
|
||||
|
||||
# VoiceDesign: Describe the voice style
|
||||
python openai_speech_client.py \
|
||||
--task-type VoiceDesign \
|
||||
--text "哥哥，你回来啦" \
|
||||
--instructions "体现撒娇稚嫩的萝莉女声，音调偏高"
|
||||
|
||||
# Base: Voice cloning
|
||||
python openai_speech_client.py \
|
||||
--task-type Base \
|
||||
--text "Hello, this is a cloned voice" \
|
||||
--ref-audio /path/to/reference.wav \
|
||||
--ref-text "Original transcript of the reference audio"
|
||||
```
|
||||
|
||||
### 3. Using curl
|
||||
|
||||
```bash
|
||||
# Simple TTS request
|
||||
curl -X POST http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, how are you?",
|
||||
"voice": "Vivian",
|
||||
"language": "English"
|
||||
}' --output output.wav
|
||||
|
||||
# With style instruction
|
||||
curl -X POST http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "I am so excited!",
|
||||
"voice": "Vivian",
|
||||
"instructions": "Speak with great enthusiasm"
|
||||
}' --output excited.wav
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### Endpoint
|
||||
|
||||
```
|
||||
POST /v1/audio/speech
|
||||
```
|
||||
|
||||
This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters.
|
||||
|
||||
### Request Body
|
||||
|
||||
```json
|
||||
{
|
||||
"input": "Text to synthesize",
|
||||
"voice": "Vivian",
|
||||
"response_format": "wav",
|
||||
"task_type": "CustomVoice",
|
||||
"language": "Auto",
|
||||
"instructions": "Optional style instructions",
|
||||
"ref_audio": "URL or base64 for voice cloning",
|
||||
"ref_text": "Reference audio transcript",
|
||||
"x_vector_only_mode": false,
|
||||
"max_new_tokens": 2048
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** The `model` field is optional when serving a single model, as the server already knows which model is loaded.
|
||||
|
||||
### Response
|
||||
|
||||
Returns audio data in the requested format (default: WAV).
|
||||
|
||||
## Parameters
|
||||
|
||||
### Standard OpenAI Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `input` | string | required | Text to synthesize |
|
||||
| `voice` | string | "Vivian" | Speaker/voice name |
|
||||
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus |
|
||||
| `speed` | float | 1.0 | Playback speed (0.25-4.0) |
|
||||
| `model` | string | optional | Model name (optional when serving single model) |
|
||||
|
||||
### Qwen3-TTS Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base |
|
||||
| `language` | string | "Auto" | Language: Auto, Chinese, English, Japanese, Korean |
|
||||
| `instructions` | string | "" | Voice style/emotion instructions |
|
||||
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
|
||||
|
||||
### Voice Clone Parameters (Base task)
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `ref_audio` | string | Yes* | Reference audio (file path, URL, or base64) |
|
||||
| `ref_text` | string | No | Transcript of reference audio (for ICL mode) |
|
||||
| `x_vector_only_mode` | bool | false | Use speaker embedding only (no ICL) |
|
||||
|
||||
## Python Usage
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
# Simple request
|
||||
response = httpx.post(
|
||||
"http://localhost:8000/v1/audio/speech",
|
||||
json={
|
||||
"model": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
|
||||
"input": "Hello world",
|
||||
"voice": "Vivian",
|
||||
},
|
||||
timeout=300.0,
|
||||
)
|
||||
|
||||
with open("output.wav", "wb") as f:
|
||||
f.write(response.content)
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- **No streaming**: Audio is generated completely before being returned. Streaming will be supported after the pipeline is disaggregated (see RFC #938).
|
||||
- **Single request**: Batch processing is not yet optimized for online serving.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
1. **Connection refused**: Make sure the server is running on the correct port
|
||||
2. **Out of memory**: Reduce `--gpu-memory-utilization` in run_server.sh
|
||||
3. **Unsupported speaker**: Check supported speakers via model documentation
|
||||
4. **Voice clone fails**: Ensure you're using the Base model variant for voice cloning
|
||||
@@ -0,0 +1,235 @@
|
||||
"""OpenAI-compatible client for Qwen3-TTS via /v1/audio/speech endpoint.
|
||||
|
||||
This script demonstrates how to use the OpenAI-compatible speech API
|
||||
to generate audio from text using Qwen3-TTS models.
|
||||
|
||||
Examples:
|
||||
# CustomVoice task (predefined speaker)
|
||||
python openai_speech_client.py --text "Hello, how are you?" --voice Vivian
|
||||
|
||||
# CustomVoice with emotion instruction
|
||||
python openai_speech_client.py --text "I'm so happy!" --voice Vivian \
|
||||
--instructions "Speak with excitement"
|
||||
|
||||
# VoiceDesign task (voice from description)
|
||||
python openai_speech_client.py --text "Hello world" \
|
||||
--task-type VoiceDesign \
|
||||
--instructions "A warm, friendly female voice"
|
||||
|
||||
# Base task (voice cloning)
|
||||
python openai_speech_client.py --text "Hello world" \
|
||||
--task-type Base \
|
||||
--ref-audio "https://example.com/reference.wav" \
|
||||
--ref-text "This is the reference transcript"
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import os
|
||||
|
||||
import httpx
|
||||
|
||||
# Default server configuration
|
||||
DEFAULT_API_BASE = "http://localhost:8000"
|
||||
DEFAULT_API_KEY = "EMPTY"
|
||||
|
||||
|
||||
def encode_audio_to_base64(audio_path: str) -> str:
|
||||
"""Encode a local audio file to base64 data URL."""
|
||||
if not os.path.exists(audio_path):
|
||||
raise FileNotFoundError(f"Audio file not found: {audio_path}")
|
||||
|
||||
# Detect MIME type from extension
|
||||
audio_path_lower = audio_path.lower()
|
||||
if audio_path_lower.endswith(".wav"):
|
||||
mime_type = "audio/wav"
|
||||
elif audio_path_lower.endswith((".mp3", ".mpeg")):
|
||||
mime_type = "audio/mpeg"
|
||||
elif audio_path_lower.endswith(".flac"):
|
||||
mime_type = "audio/flac"
|
||||
elif audio_path_lower.endswith(".ogg"):
|
||||
mime_type = "audio/ogg"
|
||||
else:
|
||||
mime_type = "audio/wav" # Default
|
||||
|
||||
with open(audio_path, "rb") as f:
|
||||
audio_bytes = f.read()
|
||||
audio_b64 = base64.b64encode(audio_bytes).decode("utf-8")
|
||||
return f"data:{mime_type};base64,{audio_b64}"
|
||||
|
||||
|
||||
def run_tts_generation(args) -> None:
|
||||
"""Run TTS generation via OpenAI-compatible /v1/audio/speech API."""
|
||||
|
||||
# Build request payload
|
||||
payload = {
|
||||
"model": args.model,
|
||||
"input": args.text,
|
||||
"voice": args.voice,
|
||||
"response_format": args.response_format,
|
||||
}
|
||||
|
||||
# Add optional parameters
|
||||
if args.instructions:
|
||||
payload["instructions"] = args.instructions
|
||||
if args.task_type:
|
||||
payload["task_type"] = args.task_type
|
||||
if args.language:
|
||||
payload["language"] = args.language
|
||||
if args.max_new_tokens:
|
||||
payload["max_new_tokens"] = args.max_new_tokens
|
||||
|
||||
# Voice clone parameters (Base task)
|
||||
if args.ref_audio:
|
||||
if args.ref_audio.startswith(("http://", "https://")):
|
||||
payload["ref_audio"] = args.ref_audio
|
||||
else:
|
||||
payload["ref_audio"] = encode_audio_to_base64(args.ref_audio)
|
||||
if args.ref_text:
|
||||
payload["ref_text"] = args.ref_text
|
||||
if args.x_vector_only:
|
||||
payload["x_vector_only_mode"] = True
|
||||
|
||||
print(f"Task type: {args.task_type or 'CustomVoice'}")
|
||||
print(f"Text: {args.text}")
|
||||
print(f"Voice: {args.voice}")
|
||||
print("Generating audio...")
|
||||
|
||||
# Make the API call
|
||||
api_url = f"{args.api_base}/v1/audio/speech"
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {args.api_key}",
|
||||
}
|
||||
|
||||
with httpx.Client(timeout=300.0) as client:
|
||||
response = client.post(api_url, json=payload, headers=headers)
|
||||
|
||||
if response.status_code != 200:
|
||||
print(f"Error: {response.status_code}")
|
||||
print(response.text)
|
||||
return
|
||||
|
||||
# Save audio response
|
||||
output_path = args.output or "tts_output.wav"
|
||||
with open(output_path, "wb") as f:
|
||||
f.write(response.content)
|
||||
print(f"Audio saved to: {output_path}")
|
||||
|
||||
|
||||
def parse_args():
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="OpenAI-compatible client for Qwen3-TTS via /v1/audio/speech",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog=__doc__,
|
||||
)
|
||||
|
||||
# Server configuration
|
||||
parser.add_argument(
|
||||
"--api-base",
|
||||
type=str,
|
||||
default=DEFAULT_API_BASE,
|
||||
help=f"API base URL (default: {DEFAULT_API_BASE})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--api-key",
|
||||
type=str,
|
||||
default=DEFAULT_API_KEY,
|
||||
help="API key (default: EMPTY)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
"-m",
|
||||
type=str,
|
||||
default="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
|
||||
help="Model name/path",
|
||||
)
|
||||
|
||||
# Task configuration
|
||||
parser.add_argument(
|
||||
"--task-type",
|
||||
"-t",
|
||||
type=str,
|
||||
default=None,
|
||||
choices=["CustomVoice", "VoiceDesign", "Base"],
|
||||
help="TTS task type (default: CustomVoice)",
|
||||
)
|
||||
|
||||
# Input text
|
||||
parser.add_argument(
|
||||
"--text",
|
||||
type=str,
|
||||
required=True,
|
||||
help="Text to synthesize",
|
||||
)
|
||||
|
||||
# Voice/speaker
|
||||
parser.add_argument(
|
||||
"--voice",
|
||||
type=str,
|
||||
default="Vivian",
|
||||
help="Speaker/voice name (default: Vivian). Options: Vivian, Ryan, etc.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--language",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Language: Auto, Chinese, English, etc.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--instructions",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Voice style/emotion instructions",
|
||||
)
|
||||
|
||||
# Base (voice clone) parameters
|
||||
parser.add_argument(
|
||||
"--ref-audio",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Reference audio file path or URL for voice cloning (Base task)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ref-text",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Reference audio transcript for voice cloning (Base task)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--x-vector-only",
|
||||
action="store_true",
|
||||
help="Use x-vector only mode for voice cloning (no ICL)",
|
||||
)
|
||||
|
||||
# Generation parameters
|
||||
parser.add_argument(
|
||||
"--max-new-tokens",
|
||||
type=int,
|
||||
default=None,
|
||||
help="Maximum new tokens to generate",
|
||||
)
|
||||
|
||||
# Output
|
||||
parser.add_argument(
|
||||
"--response-format",
|
||||
type=str,
|
||||
default="wav",
|
||||
choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
|
||||
help="Audio output format (default: wav)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
"-o",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Output audio file path (default: tts_output.wav)",
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
args = parse_args()
|
||||
run_tts_generation(args)
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
# Launch vLLM-Omni server for Qwen3-TTS models
|
||||
#
|
||||
# Usage:
|
||||
# ./run_server.sh # Default: CustomVoice model
|
||||
# ./run_server.sh CustomVoice # CustomVoice model
|
||||
# ./run_server.sh VoiceDesign # VoiceDesign model
|
||||
# ./run_server.sh Base # Base (voice clone) model
|
||||
|
||||
set -e
|
||||
|
||||
TASK_TYPE="${1:-CustomVoice}"
|
||||
|
||||
case "$TASK_TYPE" in
|
||||
CustomVoice)
|
||||
MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice"
|
||||
;;
|
||||
VoiceDesign)
|
||||
MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign"
|
||||
;;
|
||||
Base)
|
||||
MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-Base"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown task type: $TASK_TYPE"
|
||||
echo "Supported: CustomVoice, VoiceDesign, Base"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "Starting Qwen3-TTS server with model: $MODEL"
|
||||
|
||||
vllm-omni serve "$MODEL" \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--trust-remote-code \
|
||||
--enforce-eager \
|
||||
--omni
|
||||
Reference in new issue
Block a user