anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Doc] [skip ci]Sync. (#1363)

Browse Source 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
This commit is contained in:
AliciaandYueqian Lin authored and GitHub committed 2026-02-27 17:28:35 +08:00
1 parent bc13f7cf38
commit 64f5ef615c
15 files changed
+1122 -141
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/online_serving/qwen3_tts/README.md
+135 -106
View File
@@ -1,40 +1,75 @@
# Qwen3-TTS
This directory contains examples for running Qwen3-TTS models with vLLM-Omni's online serving API.
## 🛠️ Installation
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
## Supported Models
| Model | Task Type | Description |
|-------|-----------|-------------|
| Model | Task Type | Description |
| -------------------------------------- | ----------- | ----------------------------------------------------- |
| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | CustomVoice | Predefined speaker voices with optional style control |
| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description |
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio |
| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant |
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning |
| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description |
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio |
| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant |
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning |
## Quick Start
## Run examples (Qwen3-TTS)
### 1. Start the Server
```bash
# CustomVoice model (default)
./run_server.sh
# Or specify task type
./run_server.sh CustomVoice
./run_server.sh VoiceDesign
./run_server.sh Base
```
Or launch directly with vllm serve:
### Launch the Server
```bash
# CustomVoice model (predefined speakers)
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
--omni --port 8091 --trust-remote-code --enforce-eager
--omni \
--port 8091 \
--trust-remote-code \
--enforce-eager
# VoiceDesign model
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
--omni \
--port 8091 \
--trust-remote-code \
--enforce-eager
# Base model (voice cloning)
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
--omni \
--port 8091 \
--trust-remote-code \
--enforce-eager
```
### 2. Run the Client
If you have custom stage configs file, launch the server with command below
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--stage-configs-path /path/to/stage_configs_file \
--omni \
--port 8091 \
--trust-remote-code \
--enforce-eager
```
Alternatively, use the convenience script:
```bash
./run_server.sh # Default: CustomVoice model
./run_server.sh CustomVoice # CustomVoice model
./run_server.sh VoiceDesign # VoiceDesign model
./run_server.sh Base # Base (voice clone) model
```
### Send TTS Request
Get into the example folder
```bash
cd examples/online_serving/qwen3_tts
```
#### Send request via python
```bash
# CustomVoice: Use predefined speaker
@@ -65,7 +100,21 @@ python openai_speech_client.py \
--ref-text "Original transcript of the reference audio"
```
### 3. Using curl
The Python client supports the following command-line arguments:
- `--api-base`: API base URL (default: `http://localhost:8091`)
- `--model` (or `-m`): Model name/path (default: `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`)
- `--task-type` (or `-t`): TTS task type. Options: `CustomVoice`, `VoiceDesign`, `Base`
- `--text`: Text to synthesize (required)
- `--voice`: Speaker/voice name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc.
- `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`
- `--instructions`: Voice style/emotion instructions
- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task)
- `--ref-text`: Reference audio transcript for voice cloning (Base task)
- `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus`
- `--output` (or `-o`): Output audio file path (default: `tts_output.wav`)
#### Send request via curl
```bash
# Simple TTS request
@@ -90,24 +139,48 @@ curl -X POST http://localhost:8091/v1/audio/speech \
curl http://localhost:8091/v1/audio/voices
```
## Server Configuration
### Using OpenAI SDK
### TTS-Specific CLI Parameters
```python
from openai import OpenAI
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--tts-max-instructions-length` | int | 500 | Maximum character length for voice style instructions |
client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
**Example with custom instructions length:**
response = client.audio.speech.create(
model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
voice="vivian",
input="Hello, how are you?",
)
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--omni \
--port 8000 \
--tts-max-instructions-length 1000
response.stream_to_file("output.wav")
```
This is useful when you need longer voice style descriptions, especially for the VoiceDesign task type.
### Using Python httpx
```python
import httpx
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English",
},
timeout=300.0,
)
with open("output.wav", "wb") as f:
f.write(response.content)
```
### FAQ
If you encounter error about backend of librosa, try to install ffmpeg with command below.
```
sudo apt update
sudo apt install ffmpeg
```
## API Reference
@@ -115,6 +188,7 @@ This is useful when you need longer voice style descriptions, especially for the
```
POST /v1/audio/speech
Content-Type: application/json
```
This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters.
@@ -146,8 +220,7 @@ Lists available voices for the loaded model:
"ref_audio": "URL or base64 for voice cloning",
"ref_text": "Reference audio transcript",
"x_vector_only_mode": false,
"max_new_tokens": 2048,
"stream": false
"max_new_tokens": 2048
}
```
@@ -155,84 +228,46 @@ Lists available voices for the loaded model:
### Response
Returns audio data in the requested format (default: WAV).
Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/wav`).
## Parameters
### OpenAI Standard Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `input` | string | **required** | Text to synthesize |
| `model` | string | server's model | Model to use (optional, should match server if specified) |
| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) |
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus |
| `speed` | float | 1.0 | Playback speed (0.25-4.0, not supported with `stream=true`) |
| Parameter | Type | Default | Description |
| ----------------- | ------ | -------------- | ----------------------------------------------------------- |
| `input` | string | **required** | Text to synthesize |
| `model` | string | server's model | Model to use (optional, should match server if specified) |
| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) |
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus |
| `speed` | float | 1.0 | Playback speed (0.25-4.0, not supported with `stream=true`) |
### vLLM-Omni Extension Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base |
| `language` | string | "Auto" | Language (see supported languages below) |
| `instructions` | string | "" | Voice style/emotion instructions |
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
| Parameter | Type | Default | Description |
| ---------------- | ------ | ------------- | ---------------------------------------------------------------------------- |
| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base |
| `language` | string | "Auto" | Language (see supported languages below) |
| `instructions` | string | "" | Voice style/emotion instructions |
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
**Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
### Voice Clone Parameters (Base task)
| Parameter | Type | Default | Description |
|-----------|------|----------|-------------|
| `ref_audio` | string | null | Reference audio (URL or base64 data URL) |
| `ref_text` | string | null | Transcript of reference audio |
| `x_vector_only_mode` | bool | null | Use speaker embedding only (no ICL) |
## Python Usage
### Using OpenAI SDK
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
voice="vivian",
input="Hello, how are you?",
)
response.stream_to_file("output.wav")
```
### Using httpx
```python
import httpx
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English",
},
timeout=300.0,
)
with open("output.wav", "wb") as f:
f.write(response.content)
```
| Parameter | Type | Required | Description |
| -------------------- | ------ | -------- | -------------------------------------------- |
| `ref_audio` | string | **Yes** | Reference audio (URL or base64 data URL) |
| `ref_text` | string | No | Transcript of reference audio (for ICL mode) |
| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) |
## Streaming
Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded
(one chunk per 25-frame Code2Wav window, ~2 s latency to first byte):
(one chunk per 25-frame Code2Wav window):
```bash
# Stream raw PCM and pipe directly to a player
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
@@ -242,18 +277,12 @@ curl -X POST http://localhost:8091/v1/audio/speech \
"stream": true,
"response_format": "pcm"
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
# Or save the raw PCM for offline processing
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input": "Hello!", "stream": true, "response_format": "pcm"}' \
--output output.pcm
```
**Constraints:**
- `stream=true` requires `response_format="pcm"` (raw 16-bit signed PCM, 24 kHz mono).
- `speed` adjustment is not supported when streaming (`speed` must be 1.0 or omitted).
- Requires the server to be launched with a stage config that sets `async_chunk: true`.
- `speed` adjustment is not supported when streaming.
- Requires the server stage config to have `async_chunk: true` (default in `qwen3_tts.yaml`).
## Limitations
@@ -261,7 +290,7 @@ curl -X POST http://localhost:8091/v1/audio/speech \
## Troubleshooting
1. **"TTS model did not produce audio output"**: Ensure you're using the correct model variant for your task type (CustomVoice task → CustomVoice model, etc.)
1. **TTS model did not produce audio output**: Ensure you're using the correct model variant for your task type (CustomVoice task → CustomVoice model, etc.)
2. **Connection refused**: Make sure the server is running on the correct port
3. **Out of memory**: Use smaller model variant (`Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice`) or reduce `--gpu-memory-utilization`
4. **Unsupported speaker**: Use `/v1/audio/voices` to list available voices for the loaded model