diff --git a/docs/user_guide/examples/online_serving/qwen3_tts.md b/docs/user_guide/examples/online_serving/qwen3_tts.md index e3de00515..993505b87 100644 --- a/docs/user_guide/examples/online_serving/qwen3_tts.md +++ b/docs/user_guide/examples/online_serving/qwen3_tts.md @@ -17,6 +17,31 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/ | `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant | | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning | +## Gradio Demo + +Two interactive Gradio demos are available, both supporting all 3 task types: + +| Demo | File | Transport | Streaming Quality | +| ---- | ---- | --------- | ----------------- | +| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks | +| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) | + +```bash +# Option 1: Launch server + Standard Gradio together +./run_gradio_demo.sh # CustomVoice (default) +./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign +./run_gradio_demo.sh --task-type Base # Voice cloning + +# Option 2: If server is already running +python gradio_demo.py --api-base http://localhost:8000 + +# Option 3: FastRTC demo (gapless streaming) +pip install fastrtc +python gradio_fastrtc_demo.py --api-base http://localhost:8000 +``` + +Then open http://localhost:7860 in your browser. + ## Run examples (Qwen3-TTS) ### Launch the Server @@ -112,7 +137,7 @@ The Python client supports the following command-line arguments: - `--voice`: Speaker/voice name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc. - `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian` - `--instructions`: Voice style/emotion instructions -- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server. +- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task) - `--ref-text`: Reference audio transcript for voice cloning (Base task) - `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus` - `--output` (or `-o`): Output audio file path (default: `tts_output.wav`) @@ -187,6 +212,59 @@ sudo apt install ffmpeg ## API Reference +### Voices Endpoint + +#### GET /v1/audio/voices + +List all available voices/speakers from the loaded model, including both built-in model voices and uploaded custom voices. + +**Response Example:** +```json +{ + "voices": ["vivian", "ryan", "custom_voice_1"], + "uploaded_voices": [ + { + "name": "custom_voice_1", + "consent": "user_consent_id", + "created_at": 1738660000, + "file_size": 1024000, + "mime_type": "audio/wav" + } + ] +} +``` + +#### POST /v1/audio/voices + +Upload a new voice sample for voice cloning in Base task TTS requests. + +**Form Parameters:** +- `audio_sample` (required): Audio file (max 10MB, supported formats: wav, mp3, flac, ogg, aac, webm, mp4) +- `consent` (required): Consent recording ID +- `name` (required): Name for the new voice + +**Response Example:** +```json +{ + "success": true, + "voice": { + "name": "custom_voice_1", + "consent": "user_consent_id", + "created_at": 1738660000, + "mime_type": "audio/wav", + "file_size": 1024000 + } +} +``` + +**Usage Example:** +```bash +curl -X POST http://localhost:8000/v1/audio/voices \ + -F "audio_sample=@/path/to/voice_sample.wav" \ + -F "consent=user_consent_id" \ + -F "name=custom_voice_1" +``` + ### Endpoint ``` @@ -196,20 +274,6 @@ Content-Type: application/json This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters. -### Voices Endpoint - -``` -GET /v1/audio/voices -``` - -Lists available voices for the loaded model: - -```json -{ - "voices": ["aiden", "dylan", "eric", "one_anna", "ryan", "serena", "sohee", "uncle_fu", "vivian"] -} -``` - ### Request Body ```json @@ -220,7 +284,7 @@ Lists available voices for the loaded model: "task_type": "CustomVoice", "language": "Auto", "instructions": "Optional style instructions", - "ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning", + "ref_audio": "URL or base64 for voice cloning", "ref_text": "Reference audio transcript", "x_vector_only_mode": false, "max_new_tokens": 2048 @@ -253,6 +317,7 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w | `language` | string | "Auto" | Language (see supported languages below) | | `instructions` | string | "" | Voice style/emotion instructions | | `max_new_tokens` | int | 2048 | Maximum tokens to generate | +| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. | | `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | **Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian @@ -261,7 +326,7 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w | Parameter | Type | Required | Description | | -------------------- | ------ | -------- | -------------------------------------------- | -| `ref_audio` | string | **Yes** | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) | +| `ref_audio` | string | **Yes** | Reference audio (URL or base64 data URL) | | `ref_text` | string | No | Transcript of reference audio (for ICL mode) | | `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | @@ -301,11 +366,27 @@ curl -X POST http://localhost:8091/v1/audio/speech \ ## Example materials +??? abstract "gradio_demo.py" + ``````py + --8<-- "examples/online_serving/qwen3_tts/gradio_demo.py" + `````` +??? abstract "gradio_fastrtc_demo.py" + ``````py + --8<-- "examples/online_serving/qwen3_tts/gradio_fastrtc_demo.py" + `````` ??? abstract "openai_speech_client.py" ``````py --8<-- "examples/online_serving/qwen3_tts/openai_speech_client.py" `````` +??? abstract "run_gradio_demo.sh" + ``````sh + --8<-- "examples/online_serving/qwen3_tts/run_gradio_demo.sh" + `````` ??? abstract "run_server.sh" ``````sh --8<-- "examples/online_serving/qwen3_tts/run_server.sh" `````` +??? abstract "tts_common.py" + ``````py + --8<-- "examples/online_serving/qwen3_tts/tts_common.py" + `````` diff --git a/examples/online_serving/qwen3_tts/README.md b/examples/online_serving/qwen3_tts/README.md index 6984ebd42..99442c29e 100644 --- a/examples/online_serving/qwen3_tts/README.md +++ b/examples/online_serving/qwen3_tts/README.md @@ -14,6 +14,31 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/ | `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant | | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning | +## Gradio Demo + +Two interactive Gradio demos are available, both supporting all 3 task types: + +| Demo | File | Transport | Streaming Quality | +| ---- | ---- | --------- | ----------------- | +| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks | +| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) | + +```bash +# Option 1: Launch server + Standard Gradio together +./run_gradio_demo.sh # CustomVoice (default) +./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign +./run_gradio_demo.sh --task-type Base # Voice cloning + +# Option 2: If server is already running +python gradio_demo.py --api-base http://localhost:8000 + +# Option 3: FastRTC demo (gapless streaming) +pip install fastrtc +python gradio_fastrtc_demo.py --api-base http://localhost:8000 +``` + +Then open http://localhost:7860 in your browser. + ## Run examples (Qwen3-TTS) ### Launch the Server diff --git a/examples/online_serving/qwen3_tts/gradio_demo.py b/examples/online_serving/qwen3_tts/gradio_demo.py new file mode 100644 index 000000000..b8a54c6a6 --- /dev/null +++ b/examples/online_serving/qwen3_tts/gradio_demo.py @@ -0,0 +1,404 @@ +"""Gradio demo for Qwen3-TTS online serving via /v1/audio/speech API. + +Uses Gradio's built-in audio component for playback. Supports streaming +(progressive PCM chunks) and non-streaming (full audio download) modes. + +For gapless streaming playback, see gradio_fastrtc_demo.py which uses WebRTC. + +Supports all 3 task types: + - CustomVoice: Predefined speaker with optional style instructions + - VoiceDesign: Natural language voice description + - Base: Voice cloning from reference audio (upload or URL) + +Usage: + # Start the server first (see run_server.sh), then: + python gradio_demo.py --api-base http://localhost:8000 + + # Or use run_gradio_demo.sh to start both server and demo together. +""" + +import argparse +import io + +import gradio as gr +import httpx +import numpy as np +import soundfile as sf +from tts_common import ( + PCM_SAMPLE_RATE, + SUPPORTED_LANGUAGES, + TASK_TYPES, + add_common_args, + build_payload, + fetch_voices, + on_task_type_change, + stream_pcm_chunks, +) + + +def generate_speech( + api_base: str, + text: str, + task_type: str, + voice: str, + language: str, + instructions: str, + ref_audio: tuple | None, + ref_audio_url: str, + ref_text: str, + x_vector_only: bool, + response_format: str, + speed: float, +): + """Non-streaming: call /v1/audio/speech and return audio.""" + payload = build_payload( + text, + task_type, + voice, + language, + instructions, + ref_audio, + ref_audio_url, + ref_text, + x_vector_only, + response_format, + speed, + stream=False, + ) + + try: + with httpx.Client(timeout=300.0) as client: + resp = client.post( + f"{api_base}/v1/audio/speech", + json=payload, + headers={ + "Content-Type": "application/json", + "Authorization": "Bearer EMPTY", + }, + ) + except httpx.TimeoutException: + raise gr.Error("Request timed out. The server may be busy.") + except httpx.ConnectError: + raise gr.Error(f"Cannot connect to server at {api_base}. Make sure the vLLM server is running.") + + if resp.status_code != 200: + raise gr.Error(f"Server error ({resp.status_code}): {resp.text}") + + content_type = resp.headers.get("content-type", "") + if "application/json" in content_type: + try: + error_data = resp.json() + raise gr.Error(f"Server error: {error_data}") + except ValueError: + pass + + try: + if response_format == "pcm": + audio_np = np.frombuffer(resp.content, dtype=np.int16).astype(np.float32) / 32767.0 + return (PCM_SAMPLE_RATE, audio_np) + audio_np, sample_rate = sf.read(io.BytesIO(resp.content)) + if audio_np.ndim > 1: + audio_np = audio_np[:, 0] + return (sample_rate, audio_np.astype(np.float32)) + except Exception as e: + raise gr.Error(f"Failed to decode audio response: {e}") + + +def generate_speech_stream( + api_base: str, + text: str, + task_type: str, + voice: str, + language: str, + instructions: str, + ref_audio: tuple | None, + ref_audio_url: str, + ref_text: str, + x_vector_only: bool, + response_format: str, + speed: float, + chunk_seconds: float = 0.25, +): + """Streaming: yield individual PCM chunks for progressive playback.""" + payload = build_payload( + text, + task_type, + voice, + language, + instructions, + ref_audio, + ref_audio_url, + ref_text, + x_vector_only, + response_format, + speed, + stream=True, + ) + + min_chunk_samples = int(PCM_SAMPLE_RATE * chunk_seconds) + # Pre-buffer 2 chunks before first playback to avoid the gap + # between chunk 1 and chunk 2 during progressive streaming. + prebuffer_chunks = 2 + pending: list[np.ndarray] = [] + pending_len = 0 + chunks_yielded = 0 + prebuffer: list[np.ndarray] = [] + + try: + for samples in stream_pcm_chunks(api_base, payload): + float_samples = samples.astype(np.float32) / 32767.0 + pending.append(float_samples) + pending_len += len(float_samples) + if pending_len >= min_chunk_samples: + audio_chunk = np.concatenate(pending) + pending.clear() + pending_len = 0 + + if chunks_yielded < prebuffer_chunks: + prebuffer.append(audio_chunk) + chunks_yielded += 1 + if chunks_yielded == prebuffer_chunks: + yield (PCM_SAMPLE_RATE, np.concatenate(prebuffer)) + prebuffer.clear() + else: + yield (PCM_SAMPLE_RATE, audio_chunk) + # Flush: prebuffer (if stream was shorter than 2 chunks) + remaining pending + remaining = prebuffer + pending + if remaining: + yield (PCM_SAMPLE_RATE, np.concatenate(remaining)) + except httpx.TimeoutException: + raise gr.Error("Request timed out. The server may be busy.") + except httpx.ConnectError: + raise gr.Error(f"Cannot connect to server at {api_base}. Make sure the vLLM server is running.") + + +def on_stream_change(stream: bool): + """When streaming is enabled, lock format to PCM and disable speed.""" + if stream: + return ( + gr.update(value="pcm", interactive=False), + gr.update(interactive=False), + ) + return ( + gr.update(value="wav", interactive=True), + gr.update(interactive=True), + ) + + +def build_interface(api_base: str, stream_chunk_seconds: float = 0.25): + """Build the Gradio interface.""" + voices = fetch_voices(api_base) + + css = """ + #generate-btn button { width: 100%; } + .task-info { padding: 8px 12px; border-radius: 6px; + background: #f0f4ff; margin-bottom: 8px; } + """ + + with gr.Blocks(css=css, title="Qwen3-TTS Demo") as demo: + gr.Markdown("# Qwen3-TTS Online Serving Demo") + gr.Markdown(f"**Server:** `{api_base}`") + + with gr.Row(): + # Left column: inputs + with gr.Column(scale=3): + text_input = gr.Textbox( + label="Text to Synthesize", + placeholder="Enter text here, e.g., Hello, how are you?", + lines=4, + ) + + with gr.Row(): + task_type = gr.Radio( + choices=TASK_TYPES, + value="CustomVoice", + label="Task Type", + scale=2, + ) + language = gr.Dropdown( + choices=SUPPORTED_LANGUAGES, + value="Auto", + label="Language", + scale=1, + ) + + voice = gr.Dropdown( + choices=voices, + value=voices[0] if voices else None, + label="Speaker", + visible=True, + ) + + instructions = gr.Textbox( + label="Instructions", + placeholder=("e.g., Speak with excitement / A warm, friendly female voice"), + lines=2, + visible=True, + info="Optional style/emotion instructions", + ) + + ref_audio = gr.Audio( + label="Reference Audio (upload for voice cloning)", + type="numpy", + sources=["upload", "microphone"], + visible=False, + ) + ref_audio_url = gr.Textbox( + label="Reference Audio URL", + placeholder=("https://example.com/reference.wav (alternative to uploading)"), + lines=1, + visible=False, + ) + ref_text = gr.Textbox( + label="Reference Audio Transcript", + placeholder=("Transcript of the reference audio (optional, improves quality)"), + lines=2, + visible=False, + ) + x_vector_only = gr.Checkbox( + label="Use x-vector only", + value=False, + visible=False, + info=("Skip reference transcript, use speaker embedding only (lower quality)"), + ) + + with gr.Row(): + response_format = gr.Dropdown( + choices=["wav", "mp3", "flac", "pcm", "aac", "opus"], + value="wav", + label="Audio Format", + scale=1, + ) + speed = gr.Slider( + minimum=0.25, + maximum=4.0, + value=1.0, + step=0.05, + label="Speed", + scale=1, + ) + stream_checkbox = gr.Checkbox( + label="Stream output", + value=False, + info=( + "Enable streaming (PCM format, speed disabled). " + "For gapless streaming, use gradio_fastrtc_demo.py" + ), + scale=1, + ) + + generate_btn = gr.Button( + "Generate Speech", + variant="primary", + size="lg", + elem_id="generate-btn", + ) + + # Right column: output + with gr.Column(scale=2): + audio_output = gr.Audio( + label="Generated Audio", + interactive=False, + streaming=True, + autoplay=True, + ) + gr.Markdown( + "### Task Types\n" + "- **CustomVoice**: Use a predefined speaker " + "(Vivian, Ryan, etc.) with optional style instructions\n" + "- **VoiceDesign**: Describe the desired voice in natural " + "language (instructions required)\n" + "- **Base**: Clone a voice from reference audio " + "(upload a file or provide a URL)" + ) + + # Dynamic UI updates + task_type.change( + fn=on_task_type_change, + inputs=[task_type], + outputs=[ + voice, + instructions, + ref_audio, + ref_audio_url, + ref_text, + x_vector_only, + ], + ) + + stream_checkbox.change( + fn=on_stream_change, + inputs=[stream_checkbox], + outputs=[response_format, speed], + ) + + all_inputs = [ + text_input, + task_type, + voice, + language, + instructions, + ref_audio, + ref_audio_url, + ref_text, + x_vector_only, + response_format, + speed, + ] + + def dispatch(stream_enabled, *args): + if stream_enabled: + yield from generate_speech_stream( + api_base, + *args, + chunk_seconds=stream_chunk_seconds, + ) + else: + yield generate_speech(api_base, *args) + + generate_btn.click( + fn=dispatch, + inputs=[stream_checkbox] + all_inputs, + outputs=[audio_output], + ) + + demo.queue() + return demo + + +def parse_args(): + parser = argparse.ArgumentParser(description="Gradio demo for Qwen3-TTS online serving.") + add_common_args(parser) + parser.add_argument( + "--stream-chunk-seconds", + type=float, + default=0.25, + help=( + "Seconds of audio to buffer per streaming chunk (default: 0.25). " + "Lower = smoother but more overhead, higher = chunkier." + ), + ) + return parser.parse_args() + + +def main(): + args = parse_args() + + print(f"Connecting to vLLM server at: {args.api_base}") + demo = build_interface( + args.api_base, + stream_chunk_seconds=args.stream_chunk_seconds, + ) + + try: + demo.launch( + server_name=args.host, + server_port=args.port, + share=args.share, + ) + except KeyboardInterrupt: + print("\nShutting down...") + + +if __name__ == "__main__": + main() diff --git a/examples/online_serving/qwen3_tts/gradio_fastrtc_demo.py b/examples/online_serving/qwen3_tts/gradio_fastrtc_demo.py new file mode 100644 index 000000000..a40ebfd89 --- /dev/null +++ b/examples/online_serving/qwen3_tts/gradio_fastrtc_demo.py @@ -0,0 +1,274 @@ +"""Gradio demo for Qwen3-TTS using FastRTC for gapless streaming audio. + +Uses WebRTC via FastRTC to deliver audio chunks as a continuous real-time +stream, avoiding the inter-chunk gaps present in Gradio's built-in streaming. + +Requires: pip install fastrtc + +Supports all 3 task types: + - CustomVoice: Predefined speaker with optional style instructions + - VoiceDesign: Natural language voice description + - Base: Voice cloning from reference audio (URL only in this demo) + +Usage: + # Start the server first (see run_server.sh), then: + python gradio_fastrtc_demo.py --api-base http://localhost:8000 + + # With public sharing: + python gradio_fastrtc_demo.py --api-base http://localhost:8000 --share +""" + +import argparse +import logging +from collections.abc import Generator + +import gradio as gr +import numpy as np +from fastrtc import WebRTC +from tts_common import ( + PCM_SAMPLE_RATE, + SUPPORTED_LANGUAGES, + TASK_TYPES, + add_common_args, + build_payload, + fetch_voices, + stream_pcm_chunks, +) + +logger = logging.getLogger(__name__) + + +def make_tts_generator(api_base: str): + """Create a generator function for FastRTC's mode='receive'. + + FastRTC calls this with the additional_inputs values whenever the user + clicks "Start Stream". It must be a synchronous generator yielding + (sample_rate, int16_ndarray) tuples. + """ + + def tts_generate( + text: str, + task_type: str, + voice: str, + language: str, + instructions: str, + ref_audio_url: str, + ref_text: str, + x_vector_only: bool, + ) -> Generator[tuple[int, np.ndarray], None, None]: + if not text or not text.strip(): + return + + payload = build_payload( + text, + task_type, + voice, + language, + instructions, + ref_audio=None, # WebRTC UI doesn't support file upload + ref_audio_url=ref_audio_url, + ref_text=ref_text, + x_vector_only=x_vector_only, + stream=True, + ) + + try: + for samples in stream_pcm_chunks(api_base, payload): + yield (PCM_SAMPLE_RATE, samples) + except Exception: + logger.exception("TTS streaming error") + + return tts_generate + + +def build_interface(api_base: str): + """Build a custom Gradio Blocks UI matching gradio_demo.py layout.""" + voices = fetch_voices(api_base) + handler = make_tts_generator(api_base) + + css = """ + #generate-btn button { width: 100%; } + .task-info { padding: 8px 12px; border-radius: 6px; + background: #f0f4ff; margin-bottom: 8px; } + """ + + with gr.Blocks(css=css, title="Qwen3-TTS (FastRTC)") as demo: + gr.Markdown("# Qwen3-TTS Online Serving Demo (FastRTC)") + gr.Markdown(f"**Server:** `{api_base}` | **Transport:** WebRTC (gapless streaming)") + + with gr.Row(): + # Left column: inputs + with gr.Column(scale=3): + text_input = gr.Textbox( + label="Text to Synthesize", + placeholder="Enter text here, e.g., Hello, how are you?", + lines=4, + ) + + with gr.Row(): + task_type = gr.Radio( + choices=TASK_TYPES, + value="CustomVoice", + label="Task Type", + scale=2, + ) + language = gr.Dropdown( + choices=SUPPORTED_LANGUAGES, + value="Auto", + label="Language", + scale=1, + ) + + voice = gr.Dropdown( + choices=voices, + value=voices[0] if voices else None, + label="Speaker", + visible=True, + ) + + instructions = gr.Textbox( + label="Instructions", + placeholder=("e.g., Speak with excitement / A warm, friendly female voice"), + lines=2, + visible=True, + info="Optional style/emotion instructions", + ) + + # Base (voice clone) controls + # Note: file upload is not supported with WebRTC, + # use a URL instead. + ref_audio_url = gr.Textbox( + label="Reference Audio URL", + placeholder=("https://example.com/reference.wav (alternative to uploading)"), + lines=1, + visible=False, + ) + ref_text = gr.Textbox( + label="Reference Audio Transcript", + placeholder=("Transcript of the reference audio (optional, improves quality)"), + lines=2, + visible=False, + ) + x_vector_only = gr.Checkbox( + label="Use x-vector only", + value=False, + visible=False, + info=("Skip reference transcript, use speaker embedding only (lower quality)"), + ) + + generate_btn = gr.Button( + "Generate Speech", + variant="primary", + size="lg", + elem_id="generate-btn", + ) + + # Right column: output + with gr.Column(scale=2): + webrtc_output = WebRTC( + label="Generated Audio (WebRTC)", + mode="receive", + modality="audio", + ) + gr.Markdown( + "### Task Types\n" + "- **CustomVoice**: Use a predefined speaker " + "(Vivian, Ryan, etc.) with optional style instructions\n" + "- **VoiceDesign**: Describe the desired voice in natural " + "language (instructions required)\n" + "- **Base**: Clone a voice from reference audio " + "(provide a URL)" + ) + + # Dynamic UI updates for task type + # FastRTC doesn't support file upload, so we only toggle + # 5 components (no ref_audio) + def on_task_type_change_fastrtc(task_type: str): + if task_type == "CustomVoice": + return ( + gr.update(visible=True), # voice + gr.update(visible=True, info="Optional style/emotion instructions"), + gr.update(visible=False), # ref_audio_url + gr.update(visible=False), # ref_text + gr.update(visible=False), # x_vector_only + ) + elif task_type == "VoiceDesign": + return ( + gr.update(visible=False), + gr.update(visible=True, info="Required: describe the voice style"), + gr.update(visible=False), + gr.update(visible=False), + gr.update(visible=False), + ) + elif task_type == "Base": + return ( + gr.update(visible=False), + gr.update(visible=False), + gr.update(visible=True), + gr.update(visible=True), + gr.update(visible=True), + ) + return ( + gr.update(visible=True), + gr.update(visible=True), + gr.update(visible=False), + gr.update(visible=False), + gr.update(visible=False), + ) + + task_type.change( + fn=on_task_type_change_fastrtc, + inputs=[task_type], + outputs=[ + voice, + instructions, + ref_audio_url, + ref_text, + x_vector_only, + ], + ) + + # Wire up WebRTC streaming + all_inputs = [ + text_input, + task_type, + voice, + language, + instructions, + ref_audio_url, + ref_text, + x_vector_only, + ] + + webrtc_output.stream( + fn=handler, + inputs=all_inputs, + outputs=[webrtc_output], + trigger=generate_btn.click, + ) + + demo.queue() + return demo + + +def parse_args(): + parser = argparse.ArgumentParser(description="Gradio FastRTC demo for Qwen3-TTS (gapless streaming).") + add_common_args(parser) + return parser.parse_args() + + +def main(): + args = parse_args() + logging.basicConfig(level=logging.INFO) + print(f"Connecting to vLLM server at: {args.api_base}") + + demo = build_interface(args.api_base) + demo.launch( + server_name=args.host, + server_port=args.port, + share=args.share, + ) + + +if __name__ == "__main__": + main() diff --git a/examples/online_serving/qwen3_tts/run_gradio_demo.sh b/examples/online_serving/qwen3_tts/run_gradio_demo.sh new file mode 100644 index 000000000..1c7fe9e37 --- /dev/null +++ b/examples/online_serving/qwen3_tts/run_gradio_demo.sh @@ -0,0 +1,203 @@ +#!/bin/bash +# Launch both vLLM server and Gradio demo for Qwen3-TTS +# +# Usage: +# ./run_gradio_demo.sh # Default: CustomVoice +# ./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign model +# ./run_gradio_demo.sh --task-type Base --gradio-port 7861 +# +# Options: +# --task-type TYPE Task type: CustomVoice, VoiceDesign, Base (default: CustomVoice) +# --server-port PORT Port for vLLM server (default: 8000) +# --gradio-port PORT Port for Gradio demo (default: 7860) +# --server-host HOST Host for vLLM server (default: 0.0.0.0) +# --gradio-ip IP IP for Gradio demo (default: 127.0.0.1) +# --share Share Gradio demo publicly + +set -e + +# Default values +TASK_TYPE="CustomVoice" +SERVER_PORT=8000 +GRADIO_PORT=7860 +SERVER_HOST="0.0.0.0" +GRADIO_IP="127.0.0.1" +GRADIO_SHARE=false + +# Parse command line arguments +while [[ $# -gt 0 ]]; do + case $1 in + --task-type) + TASK_TYPE="$2" + shift 2 + ;; + --server-port) + SERVER_PORT="$2" + shift 2 + ;; + --gradio-port) + GRADIO_PORT="$2" + shift 2 + ;; + --server-host) + SERVER_HOST="$2" + shift 2 + ;; + --gradio-ip) + GRADIO_IP="$2" + shift 2 + ;; + --share) + GRADIO_SHARE=true + shift + ;; + --help) + echo "Usage: $0 [OPTIONS]" + echo "" + echo "Options:" + echo " --task-type TYPE Task type: CustomVoice, VoiceDesign, Base (default: CustomVoice)" + echo " --server-port PORT Port for vLLM server (default: 8000)" + echo " --gradio-port PORT Port for Gradio demo (default: 7860)" + echo " --server-host HOST Host for vLLM server (default: 0.0.0.0)" + echo " --gradio-ip IP IP for Gradio demo (default: 127.0.0.1)" + echo " --share Share Gradio demo publicly" + echo "" + exit 0 + ;; + *) + echo "Unknown option: $1" + echo "Use --help for usage information" + exit 1 + ;; + esac +done + +# Map task type to model +case "$TASK_TYPE" in + CustomVoice) + MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice" + ;; + VoiceDesign) + MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign" + ;; + Base) + MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-Base" + ;; + *) + echo "Unknown task type: $TASK_TYPE" + echo "Supported: CustomVoice, VoiceDesign, Base" + exit 1 + ;; +esac + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +API_BASE="http://localhost:${SERVER_PORT}" + +echo "==========================================" +echo "Qwen3-TTS Gradio Demo" +echo "==========================================" +echo "Task Type : $TASK_TYPE" +echo "Model : $MODEL" +echo "Server : http://${SERVER_HOST}:${SERVER_PORT}" +echo "Gradio : http://${GRADIO_IP}:${GRADIO_PORT}" +echo "==========================================" + +# Cleanup on exit +cleanup() { + echo "" + echo "Shutting down..." + if [ -n "$SERVER_PID" ]; then + echo "Stopping vLLM server (PID: $SERVER_PID)..." + kill "$SERVER_PID" 2>/dev/null || true + wait "$SERVER_PID" 2>/dev/null || true + fi + if [ -n "$GRADIO_PID" ]; then + echo "Stopping Gradio demo (PID: $GRADIO_PID)..." + kill "$GRADIO_PID" 2>/dev/null || true + wait "$GRADIO_PID" 2>/dev/null || true + fi + echo "Cleanup complete" + exit 0 +} +trap cleanup SIGINT SIGTERM + +# Start vLLM server +echo "" +echo "Starting vLLM server..." +LOG_FILE="/tmp/vllm_tts_server_${SERVER_PORT}.log" + +vllm-omni serve "$MODEL" \ + --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \ + --host "$SERVER_HOST" \ + --port "$SERVER_PORT" \ + --gpu-memory-utilization 0.9 \ + --trust-remote-code \ + --enforce-eager \ + --omni 2>&1 | tee "$LOG_FILE" & +SERVER_PID=$! + +# Wait for server startup +echo "" +echo "Waiting for vLLM server to be ready..." +STARTUP_FLAG="/tmp/vllm_tts_startup_flag_${SERVER_PORT}.tmp" +rm -f "$STARTUP_FLAG" + +( + tail -f "$LOG_FILE" 2>/dev/null | grep -m 1 "Application startup complete" > /dev/null && touch "$STARTUP_FLAG" +) & +TAIL_PID=$! + +MAX_WAIT=300 +ELAPSED=0 +while [ $ELAPSED -lt $MAX_WAIT ]; do + if [ -f "$STARTUP_FLAG" ]; then + kill "$TAIL_PID" 2>/dev/null || true + wait "$TAIL_PID" 2>/dev/null || true + echo "" + echo "vLLM server is ready!" + break + fi + if ! kill -0 "$SERVER_PID" 2>/dev/null; then + kill "$TAIL_PID" 2>/dev/null || true + echo "" + echo "Error: vLLM server failed to start" + exit 1 + fi + sleep 1 + ELAPSED=$((ELAPSED + 1)) +done + +rm -f "$STARTUP_FLAG" + +if [ $ELAPSED -ge $MAX_WAIT ]; then + kill "$TAIL_PID" 2>/dev/null || true + echo "Error: Server startup timed out after ${MAX_WAIT}s" + kill "$SERVER_PID" 2>/dev/null || true + exit 1 +fi + +# Start Gradio demo +echo "" +echo "Starting Gradio demo..." +cd "$SCRIPT_DIR" +GRADIO_CMD=("python" "gradio_demo.py" "--api-base" "$API_BASE" "--ip" "$GRADIO_IP" "--port" "$GRADIO_PORT") +if [ "$GRADIO_SHARE" = true ]; then + GRADIO_CMD+=("--share") +fi + +"${GRADIO_CMD[@]}" & +GRADIO_PID=$! + +echo "" +echo "==========================================" +echo "Both services are running!" +echo "==========================================" +echo "vLLM Server : http://${SERVER_HOST}:${SERVER_PORT}" +echo "Gradio Demo : http://${GRADIO_IP}:${GRADIO_PORT}" +echo "" +echo "Press Ctrl+C to stop both services" +echo "==========================================" +echo "" + +wait $SERVER_PID $GRADIO_PID || true +cleanup diff --git a/examples/online_serving/qwen3_tts/tts_common.py b/examples/online_serving/qwen3_tts/tts_common.py new file mode 100644 index 000000000..8e1c77c39 --- /dev/null +++ b/examples/online_serving/qwen3_tts/tts_common.py @@ -0,0 +1,218 @@ +"""Shared constants, helpers, and payload building for Qwen3-TTS Gradio demos.""" + +import base64 +import io + +import gradio as gr +import httpx +import numpy as np +import soundfile as sf + +SUPPORTED_LANGUAGES = [ + "Auto", + "Chinese", + "English", + "Japanese", + "Korean", + "German", + "French", + "Russian", + "Portuguese", + "Spanish", + "Italian", +] + +TASK_TYPES = ["CustomVoice", "VoiceDesign", "Base"] + +PCM_SAMPLE_RATE = 24000 + +DEFAULT_API_BASE = "http://localhost:8000" + + +def fetch_voices(api_base: str) -> list[str]: + """Fetch available voices from the server.""" + try: + with httpx.Client(timeout=10.0) as client: + resp = client.get( + f"{api_base}/v1/audio/voices", + headers={"Authorization": "Bearer EMPTY"}, + ) + if resp.status_code == 200: + data = resp.json() + return data.get("voices", ["Vivian", "Ryan"]) + except Exception: + pass + return ["Vivian", "Ryan"] + + +def encode_audio_to_base64(audio_data: tuple) -> str: + """Encode Gradio audio input (sample_rate, numpy_array) to base64 data URL.""" + sample_rate, audio_np = audio_data + + if audio_np.dtype != np.int16: + if audio_np.dtype in (np.float32, np.float64): + audio_np = np.clip(audio_np, -1.0, 1.0) + audio_np = (audio_np * 32767).astype(np.int16) + else: + audio_np = audio_np.astype(np.int16) + + buf = io.BytesIO() + sf.write(buf, audio_np, sample_rate, format="WAV") + wav_b64 = base64.b64encode(buf.getvalue()).decode("utf-8") + return f"data:audio/wav;base64,{wav_b64}" + + +def build_payload( + text: str, + task_type: str, + voice: str, + language: str, + instructions: str, + ref_audio: tuple | None, + ref_audio_url: str, + ref_text: str, + x_vector_only: bool, + response_format: str = "pcm", + speed: float = 1.0, + stream: bool = True, +) -> dict: + """Build the /v1/audio/speech request payload. + + Raises gr.Error for invalid input so callers don't need to validate. + """ + if not text or not text.strip(): + raise gr.Error("Please enter text to synthesize.") + + payload: dict = { + "input": text.strip(), + "response_format": "pcm" if stream else response_format, + "stream": stream, + } + if not stream: + payload["speed"] = speed + + if task_type: + payload["task_type"] = task_type + if language: + payload["language"] = language + + if task_type == "CustomVoice": + if voice: + payload["voice"] = voice + if instructions and instructions.strip(): + payload["instructions"] = instructions.strip() + + elif task_type == "VoiceDesign": + if not instructions or not instructions.strip(): + raise gr.Error("VoiceDesign task requires voice style instructions.") + payload["instructions"] = instructions.strip() + + elif task_type == "Base": + ref_audio_url_stripped = ref_audio_url.strip() if ref_audio_url else "" + if ref_audio_url_stripped: + payload["ref_audio"] = ref_audio_url_stripped + elif ref_audio is not None: + payload["ref_audio"] = encode_audio_to_base64(ref_audio) + else: + raise gr.Error("Base (voice clone) task requires reference audio. Upload a file or provide a URL.") + if ref_text and ref_text.strip(): + payload["ref_text"] = ref_text.strip() + if x_vector_only: + payload["x_vector_only_mode"] = True + + return payload + + +def on_task_type_change(task_type: str): + """Update UI visibility based on selected task type.""" + if task_type == "CustomVoice": + return ( + gr.update(visible=True), # voice dropdown + gr.update(visible=True, info="Optional style/emotion instructions"), + gr.update(visible=False), # ref_audio + gr.update(visible=False), # ref_audio_url + gr.update(visible=False), # ref_text + gr.update(visible=False), # x_vector_only + ) + elif task_type == "VoiceDesign": + return ( + gr.update(visible=False), + gr.update(visible=True, info="Required: describe the voice style"), + gr.update(visible=False), + gr.update(visible=False), + gr.update(visible=False), + gr.update(visible=False), + ) + elif task_type == "Base": + return ( + gr.update(visible=False), + gr.update(visible=False), + gr.update(visible=True), + gr.update(visible=True), + gr.update(visible=True), + gr.update(visible=True), + ) + return ( + gr.update(visible=True), + gr.update(visible=True), + gr.update(visible=False), + gr.update(visible=False), + gr.update(visible=False), + gr.update(visible=False), + ) + + +def stream_pcm_chunks(api_base: str, payload: dict): + """Stream raw PCM bytes from the server, yielding int16 numpy arrays. + + Handles odd-byte boundaries between network chunks. + """ + leftover = b"" + with httpx.Client(timeout=300.0) as client: + with client.stream( + "POST", + f"{api_base}/v1/audio/speech", + json=payload, + headers={ + "Content-Type": "application/json", + "Authorization": "Bearer EMPTY", + }, + ) as resp: + if resp.status_code != 200: + resp.read() + raise gr.Error(f"Server error ({resp.status_code}): {resp.text}") + for chunk in resp.iter_bytes(): + if not chunk: + continue + raw = leftover + chunk + usable = len(raw) - (len(raw) % 2) + leftover = raw[usable:] + if usable == 0: + continue + yield np.frombuffer(raw[:usable], dtype=np.int16).copy() + + +def add_common_args(parser): + """Add CLI arguments shared by both demos.""" + parser.add_argument( + "--api-base", + default=DEFAULT_API_BASE, + help=f"Base URL for the vLLM API server (default: {DEFAULT_API_BASE}).", + ) + parser.add_argument( + "--host", + default="0.0.0.0", + help="Host/IP for Gradio server (default: 0.0.0.0).", + ) + parser.add_argument( + "--port", + type=int, + default=7860, + help="Port for Gradio server (default: 7860).", + ) + parser.add_argument( + "--share", + action="store_true", + help="Share the Gradio demo publicly.", + ) + return parser