[Doc]Fix - Align with repo. (#1176)
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
This commit is contained in:
Alicia authored and GitHub committed
1 parent
d6f93b0f37
commit
ce1ae26084
13 files changed
+335
-29
No files matched your search
@@ -2,9 +2,10 @@
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/bagel>.
|
||||
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
Please refer to [README.md](../../../README.md)
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
|
||||
|
||||
## Run examples (BAGEL-7B-MoT)
|
||||
|
||||
@@ -74,7 +75,7 @@ python openai_chat_client.py \
|
||||
|
||||
BAGEL-7B-MoT supports **multiple modality modes** for different use cases.
|
||||
|
||||
The default yaml configuration deploys Thinker and DiT on the same GPU. You can use the default configuration file: [`bagel.yaml`](../../../vllm_omni/model_executor/stage_configs/bagel.yaml)
|
||||
The default yaml configuration deploys Thinker and DiT on the same GPU. You can use the default configuration file: [`bagel.yaml`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/model_executor/stage_configs/bagel.yaml)
|
||||
|
||||
| Modality | Input | Output | Description |
|
||||
| ----------- | ------------ | ------ | -------------------------------------- |
|
||||
@@ -230,3 +231,14 @@ sudo apt install ffmpeg
|
||||
| Stage-0 (Thinker) | **15.04 GiB** **+ KV Cache** |
|
||||
| Stage-1 (DiT) | **26.50 GiB** |
|
||||
| Total | **~42 GiB + KV Cache** |
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "openai_chat_client.py"
|
||||
``````py
|
||||
--8<-- "examples/online_serving/bagel/openai_chat_client.py"
|
||||
``````
|
||||
??? abstract "run_server.sh"
|
||||
``````sh
|
||||
--8<-- "examples/online_serving/bagel/run_server.sh"
|
||||
``````
|
||||
@@ -1,7 +1,8 @@
|
||||
# LoRA-Inference
|
||||
# Online LoRA Inference (Diffusion)
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/lora_inference>.
|
||||
|
||||
|
||||
This example shows how to use **per-request LoRA** with vLLM-Omni diffusion models via the OpenAI-compatible Chat Completions API.
|
||||
|
||||
> Note: The LoRA adapter path must be readable on the **server** machine (usually a local path or a mounted directory).
|
||||
@@ -55,15 +56,17 @@ lora_adapter/
|
||||
└── adapter_model.safetensors
|
||||
```
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "openai_chat_client.py"
|
||||
``````py
|
||||
--8<-- "examples/online_serving/lora_inference/openai_chat_client.py"
|
||||
``````
|
||||
??? abstract "run_curl_lora_inference.sh"
|
||||
``````py
|
||||
``````sh
|
||||
--8<-- "examples/online_serving/lora_inference/run_curl_lora_inference.sh"
|
||||
``````
|
||||
??? abstract "run_server.sh"
|
||||
``````py
|
||||
``````sh
|
||||
--8<-- "examples/online_serving/lora_inference/run_server.sh"
|
||||
``````
|
||||
@@ -0,0 +1,189 @@
|
||||
# Qwen3-TTS Online Serving
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_tts>.
|
||||
|
||||
|
||||
This directory contains examples for running Qwen3-TTS models with vLLM-Omni's online serving API.
|
||||
|
||||
## Supported Models
|
||||
|
||||
| Model | Task Type | Description |
|
||||
|-------|-----------|-------------|
|
||||
| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | CustomVoice | Predefined speaker voices with optional style control |
|
||||
| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description |
|
||||
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Start the Server
|
||||
|
||||
```bash
|
||||
# CustomVoice model (default)
|
||||
./run_server.sh
|
||||
|
||||
# Or specify task type
|
||||
./run_server.sh CustomVoice
|
||||
./run_server.sh VoiceDesign
|
||||
./run_server.sh Base
|
||||
```
|
||||
|
||||
### 2. Run the Client
|
||||
|
||||
```bash
|
||||
# CustomVoice: Use predefined speaker
|
||||
python openai_speech_client.py \
|
||||
--text "你好，我是通义千问" \
|
||||
--voice Vivian \
|
||||
--language Chinese
|
||||
|
||||
# CustomVoice with style instruction
|
||||
python openai_speech_client.py \
|
||||
--text "今天天气真好" \
|
||||
--voice Ryan \
|
||||
--instructions "用开心的语气说"
|
||||
|
||||
# VoiceDesign: Describe the voice style
|
||||
python openai_speech_client.py \
|
||||
--model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
|
||||
--task-type VoiceDesign \
|
||||
--text "哥哥，你回来啦" \
|
||||
--instructions "体现撒娇稚嫩的萝莉女声，音调偏高"
|
||||
|
||||
# Base: Voice cloning
|
||||
python openai_speech_client.py \
|
||||
--model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
|
||||
--task-type Base \
|
||||
--text "Hello, this is a cloned voice" \
|
||||
--ref-audio /path/to/reference.wav \
|
||||
--ref-text "Original transcript of the reference audio"
|
||||
```
|
||||
|
||||
### 3. Using curl
|
||||
|
||||
```bash
|
||||
# Simple TTS request
|
||||
curl -X POST http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, how are you?",
|
||||
"voice": "Vivian",
|
||||
"language": "English"
|
||||
}' --output output.wav
|
||||
|
||||
# With style instruction
|
||||
curl -X POST http://localhost:8000/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "I am so excited!",
|
||||
"voice": "Vivian",
|
||||
"instructions": "Speak with great enthusiasm"
|
||||
}' --output excited.wav
|
||||
|
||||
# List available voices in CustomVoice models
|
||||
curl http://localhost:8000/v1/audio/voices
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### Endpoint
|
||||
|
||||
```
|
||||
POST /v1/audio/speech
|
||||
```
|
||||
|
||||
This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters.
|
||||
|
||||
### Request Body
|
||||
|
||||
```json
|
||||
{
|
||||
"input": "Text to synthesize",
|
||||
"voice": "Vivian",
|
||||
"response_format": "wav",
|
||||
"task_type": "CustomVoice",
|
||||
"language": "Auto",
|
||||
"instructions": "Optional style instructions",
|
||||
"ref_audio": "URL or base64 for voice cloning",
|
||||
"ref_text": "Reference audio transcript",
|
||||
"x_vector_only_mode": false,
|
||||
"max_new_tokens": 2048
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** The `model` field is optional when serving a single model, as the server already knows which model is loaded.
|
||||
|
||||
### Response
|
||||
|
||||
Returns audio data in the requested format (default: WAV).
|
||||
|
||||
## Parameters
|
||||
|
||||
### Standard OpenAI Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `input` | string | required | Text to synthesize |
|
||||
| `voice` | string | "Vivian" | Speaker/voice name |
|
||||
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus |
|
||||
| `speed` | float | 1.0 | Playback speed (0.25-4.0) |
|
||||
| `model` | string | optional | Model name (optional when serving single model) |
|
||||
|
||||
### Qwen3-TTS Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base |
|
||||
| `language` | string | "Auto" | Language: Auto, Chinese, English, Japanese, Korean |
|
||||
| `instructions` | string | "" | Voice style/emotion instructions |
|
||||
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
|
||||
|
||||
### Voice Clone Parameters (Base task)
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `ref_audio` | string | Yes* | Reference audio (file path, URL, or base64) |
|
||||
| `ref_text` | string | No | Transcript of reference audio (for ICL mode) |
|
||||
| `x_vector_only_mode` | bool | false | Use speaker embedding only (no ICL) |
|
||||
|
||||
## Python Usage
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
# Simple request
|
||||
response = httpx.post(
|
||||
"http://localhost:8000/v1/audio/speech",
|
||||
json={
|
||||
"model": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
|
||||
"input": "Hello world",
|
||||
"voice": "Vivian",
|
||||
},
|
||||
timeout=300.0,
|
||||
)
|
||||
|
||||
with open("output.wav", "wb") as f:
|
||||
f.write(response.content)
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- **No streaming**: Audio is generated completely before being returned. Streaming will be supported after the pipeline is disaggregated (see RFC #938).
|
||||
- **Single request**: Batch processing is not yet optimized for online serving.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
1. **Connection refused**: Make sure the server is running on the correct port
|
||||
2. **Out of memory**: Reduce `--gpu-memory-utilization` in run_server.sh
|
||||
3. **Unsupported speaker**: Check supported speakers via model documentation
|
||||
4. **Voice clone fails**: Ensure you're using the Base model variant for voice cloning
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "openai_speech_client.py"
|
||||
``````py
|
||||
--8<-- "examples/online_serving/qwen3_tts/openai_speech_client.py"
|
||||
``````
|
||||
??? abstract "run_server.sh"
|
||||
``````sh
|
||||
--8<-- "examples/online_serving/qwen3_tts/run_server.sh"
|
||||
``````
|
||||
Reference in new issue
Block a user