[Feat][Qwen3-tts]: Add Gradio demo for online serving (#1231)

Signed-off-by: lishunyang <lishunyang12@163.com>
Signed-off-by: linyueqian <linyueqian@outlook.com>
Co-authored-by: linyueqian <linyueqian@outlook.com>

docs/user_guide/examples/online_serving/qwen3_tts.md

@@ -17,6 +17,31 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
 | `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant                                |
 | `Qwen/Qwen3-TTS-12Hz-0.6B-Base`        | Base        | Smaller/faster variant for voice cloning              |
 
+## Gradio Demo
+
+Two interactive Gradio demos are available, both supporting all 3 task types:
+
+| Demo | File | Transport | Streaming Quality |
+| ---- | ---- | --------- | ----------------- |
+| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks |
+| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) |
+
+```bash
+# Option 1: Launch server + Standard Gradio together
+./run_gradio_demo.sh                                # CustomVoice (default)
+./run_gradio_demo.sh --task-type VoiceDesign        # VoiceDesign
+./run_gradio_demo.sh --task-type Base               # Voice cloning
+
+# Option 2: If server is already running
+python gradio_demo.py --api-base http://localhost:8000
+
+# Option 3: FastRTC demo (gapless streaming)
+pip install fastrtc
+python gradio_fastrtc_demo.py --api-base http://localhost:8000
+```
+
+Then open http://localhost:7860 in your browser.
+
 ## Run examples (Qwen3-TTS)
 
 ### Launch the Server
@@ -112,7 +137,7 @@ The Python client supports the following command-line arguments:
 - `--voice`: Speaker/voice name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc.
 - `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`
 - `--instructions`: Voice style/emotion instructions
-- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server.
+- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task)
 - `--ref-text`: Reference audio transcript for voice cloning (Base task)
 - `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus`
 - `--output` (or `-o`): Output audio file path (default: `tts_output.wav`)
@@ -187,6 +212,59 @@ sudo apt install ffmpeg
 
 ## API Reference
 
+### Voices Endpoint
+
+#### GET /v1/audio/voices
+
+List all available voices/speakers from the loaded model, including both built-in model voices and uploaded custom voices.
+
+**Response Example:**
+```json
+{
+  "voices": ["vivian", "ryan", "custom_voice_1"],
+  "uploaded_voices": [
+    {
+      "name": "custom_voice_1",
+      "consent": "user_consent_id",
+      "created_at": 1738660000,
+      "file_size": 1024000,
+      "mime_type": "audio/wav"
+    }
+  ]
+}
+```
+
+#### POST /v1/audio/voices
+
+Upload a new voice sample for voice cloning in Base task TTS requests.
+
+**Form Parameters:**
+- `audio_sample` (required): Audio file (max 10MB, supported formats: wav, mp3, flac, ogg, aac, webm, mp4)
+- `consent` (required): Consent recording ID
+- `name` (required): Name for the new voice
+
+**Response Example:**
+```json
+{
+  "success": true,
+  "voice": {
+    "name": "custom_voice_1",
+    "consent": "user_consent_id",
+    "created_at": 1738660000,
+    "mime_type": "audio/wav",
+    "file_size": 1024000
+  }
+}
+```
+
+**Usage Example:**
+```bash
+curl -X POST http://localhost:8000/v1/audio/voices \
+  -F "audio_sample=@/path/to/voice_sample.wav" \
+  -F "consent=user_consent_id" \
+  -F "name=custom_voice_1"
+```
+
 ### Endpoint
 
 ```
@@ -196,20 +274,6 @@ Content-Type: application/json
 
 This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters.
 
-### Voices Endpoint
-
-```
-GET /v1/audio/voices
-```
-
-Lists available voices for the loaded model:
-
-```json
-{
-    "voices": ["aiden", "dylan", "eric", "one_anna", "ryan", "serena", "sohee", "uncle_fu", "vivian"]
-}
-```
-
 ### Request Body
 
 ```json
@@ -220,7 +284,7 @@ Lists available voices for the loaded model:
     "task_type": "CustomVoice",
     "language": "Auto",
     "instructions": "Optional style instructions",
-    "ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning",
+    "ref_audio": "URL or base64 for voice cloning",
     "ref_text": "Reference audio transcript",
     "x_vector_only_mode": false,
     "max_new_tokens": 2048
@@ -253,6 +317,7 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w
 | `language`       | string | "Auto"        | Language (see supported languages below)                                     |
 | `instructions`   | string | ""            | Voice style/emotion instructions                                             |
 | `max_new_tokens` | int    | 2048          | Maximum tokens to generate                                                   |
+| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. |
 | `stream`         | bool   | false         | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
 
 **Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
@@ -261,7 +326,7 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w
 
 | Parameter            | Type   | Required | Description                                  |
 | -------------------- | ------ | -------- | -------------------------------------------- |
-| `ref_audio`          | string | **Yes**  | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) |
+| `ref_audio`          | string | **Yes**  | Reference audio (URL or base64 data URL)     |
 | `ref_text`           | string | No       | Transcript of reference audio (for ICL mode) |
 | `x_vector_only_mode` | bool   | No       | Use speaker embedding only (no ICL)          |
 
@@ -301,11 +366,27 @@ curl -X POST http://localhost:8091/v1/audio/speech \
 
 ## Example materials
 
+??? abstract "gradio_demo.py"
+    ``````py
+    --8<-- "examples/online_serving/qwen3_tts/gradio_demo.py"
+    ``````
+??? abstract "gradio_fastrtc_demo.py"
+    ``````py
+    --8<-- "examples/online_serving/qwen3_tts/gradio_fastrtc_demo.py"
+    ``````
 ??? abstract "openai_speech_client.py"
     ``````py
     --8<-- "examples/online_serving/qwen3_tts/openai_speech_client.py"
     ``````
+??? abstract "run_gradio_demo.sh"
+    ``````sh
+    --8<-- "examples/online_serving/qwen3_tts/run_gradio_demo.sh"
+    ``````
 ??? abstract "run_server.sh"
     ``````sh
     --8<-- "examples/online_serving/qwen3_tts/run_server.sh"
     ``````
+??? abstract "tts_common.py"
+    ``````py
+    --8<-- "examples/online_serving/qwen3_tts/tts_common.py"
+    ``````

examples/online_serving/qwen3_tts/README.md

@@ -14,6 +14,31 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
 | `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant                                |
 | `Qwen/Qwen3-TTS-12Hz-0.6B-Base`        | Base        | Smaller/faster variant for voice cloning              |
 
+## Gradio Demo
+
+Two interactive Gradio demos are available, both supporting all 3 task types:
+
+| Demo | File | Transport | Streaming Quality |
+| ---- | ---- | --------- | ----------------- |
+| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks |
+| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) |
+
+```bash
+# Option 1: Launch server + Standard Gradio together
+./run_gradio_demo.sh                                # CustomVoice (default)
+./run_gradio_demo.sh --task-type VoiceDesign        # VoiceDesign
+./run_gradio_demo.sh --task-type Base               # Voice cloning
+
+# Option 2: If server is already running
+python gradio_demo.py --api-base http://localhost:8000
+
+# Option 3: FastRTC demo (gapless streaming)
+pip install fastrtc
+python gradio_fastrtc_demo.py --api-base http://localhost:8000
+```
+
+Then open http://localhost:7860 in your browser.
+
 ## Run examples (Qwen3-TTS)
 
 ### Launch the Server

examples/online_serving/qwen3_tts/gradio_demo.py

@@ -0,0 +1,404 @@
+"""Gradio demo for Qwen3-TTS online serving via /v1/audio/speech API.
+
+Uses Gradio's built-in audio component for playback. Supports streaming
+(progressive PCM chunks) and non-streaming (full audio download) modes.
+
+For gapless streaming playback, see gradio_fastrtc_demo.py which uses WebRTC.
+
+Supports all 3 task types:
+  - CustomVoice: Predefined speaker with optional style instructions
+  - VoiceDesign: Natural language voice description
+  - Base: Voice cloning from reference audio (upload or URL)
+
+Usage:
+    # Start the server first (see run_server.sh), then:
+    python gradio_demo.py --api-base http://localhost:8000
+
+    # Or use run_gradio_demo.sh to start both server and demo together.
+"""
+
+import argparse
+import io
+
+import gradio as gr
+import httpx
+import numpy as np
+import soundfile as sf
+from tts_common import (
+    PCM_SAMPLE_RATE,
+    SUPPORTED_LANGUAGES,
+    TASK_TYPES,
+    add_common_args,
+    build_payload,
+    fetch_voices,
+    on_task_type_change,
+    stream_pcm_chunks,
+)
+
+
+def generate_speech(
+    api_base: str,
+    text: str,
+    task_type: str,
+    voice: str,
+    language: str,
+    instructions: str,
+    ref_audio: tuple | None,
+    ref_audio_url: str,
+    ref_text: str,
+    x_vector_only: bool,
+    response_format: str,
+    speed: float,
+):
+    """Non-streaming: call /v1/audio/speech and return audio."""
+    payload = build_payload(
+        text,
+        task_type,
+        voice,
+        language,
+        instructions,
+        ref_audio,
+        ref_audio_url,
+        ref_text,
+        x_vector_only,
+        response_format,
+        speed,
+        stream=False,
+    )
+
+    try:
+        with httpx.Client(timeout=300.0) as client:
+            resp = client.post(
+                f"{api_base}/v1/audio/speech",
+                json=payload,
+                headers={
+                    "Content-Type": "application/json",
+                    "Authorization": "Bearer EMPTY",
+                },
+            )
+    except httpx.TimeoutException:
+        raise gr.Error("Request timed out. The server may be busy.")
+    except httpx.ConnectError:
+        raise gr.Error(f"Cannot connect to server at {api_base}. Make sure the vLLM server is running.")
+
+    if resp.status_code != 200:
+        raise gr.Error(f"Server error ({resp.status_code}): {resp.text}")
+
+    content_type = resp.headers.get("content-type", "")
+    if "application/json" in content_type:
+        try:
+            error_data = resp.json()
+            raise gr.Error(f"Server error: {error_data}")
+        except ValueError:
+            pass
+
+    try:
+        if response_format == "pcm":
+            audio_np = np.frombuffer(resp.content, dtype=np.int16).astype(np.float32) / 32767.0
+            return (PCM_SAMPLE_RATE, audio_np)
+        audio_np, sample_rate = sf.read(io.BytesIO(resp.content))
+        if audio_np.ndim > 1:
+            audio_np = audio_np[:, 0]
+        return (sample_rate, audio_np.astype(np.float32))
+    except Exception as e:
+        raise gr.Error(f"Failed to decode audio response: {e}")
+
+
+def generate_speech_stream(
+    api_base: str,
+    text: str,
+    task_type: str,
+    voice: str,
+    language: str,
+    instructions: str,
+    ref_audio: tuple | None,
+    ref_audio_url: str,
+    ref_text: str,
+    x_vector_only: bool,
+    response_format: str,
+    speed: float,
+    chunk_seconds: float = 0.25,
+):
+    """Streaming: yield individual PCM chunks for progressive playback."""
+    payload = build_payload(
+        text,
+        task_type,
+        voice,
+        language,
+        instructions,
+        ref_audio,
+        ref_audio_url,
+        ref_text,
+        x_vector_only,
+        response_format,
+        speed,
+        stream=True,
+    )
+
+    min_chunk_samples = int(PCM_SAMPLE_RATE * chunk_seconds)
+    # Pre-buffer 2 chunks before first playback to avoid the gap
+    # between chunk 1 and chunk 2 during progressive streaming.
+    prebuffer_chunks = 2
+    pending: list[np.ndarray] = []
+    pending_len = 0
+    chunks_yielded = 0
+    prebuffer: list[np.ndarray] = []
+
+    try:
+        for samples in stream_pcm_chunks(api_base, payload):
+            float_samples = samples.astype(np.float32) / 32767.0
+            pending.append(float_samples)
+            pending_len += len(float_samples)
+            if pending_len >= min_chunk_samples:
+                audio_chunk = np.concatenate(pending)
+                pending.clear()
+                pending_len = 0
+
+                if chunks_yielded < prebuffer_chunks:
+                    prebuffer.append(audio_chunk)
+                    chunks_yielded += 1
+                    if chunks_yielded == prebuffer_chunks:
+                        yield (PCM_SAMPLE_RATE, np.concatenate(prebuffer))
+                        prebuffer.clear()
+                else:
+                    yield (PCM_SAMPLE_RATE, audio_chunk)
+        # Flush: prebuffer (if stream was shorter than 2 chunks) + remaining pending
+        remaining = prebuffer + pending
+        if remaining:
+            yield (PCM_SAMPLE_RATE, np.concatenate(remaining))
+    except httpx.TimeoutException:
+        raise gr.Error("Request timed out. The server may be busy.")
+    except httpx.ConnectError:
+        raise gr.Error(f"Cannot connect to server at {api_base}. Make sure the vLLM server is running.")
+
+
+def on_stream_change(stream: bool):
+    """When streaming is enabled, lock format to PCM and disable speed."""
+    if stream:
+        return (
+            gr.update(value="pcm", interactive=False),
+            gr.update(interactive=False),
+        )
+    return (
+        gr.update(value="wav", interactive=True),
+        gr.update(interactive=True),
+    )
+
+
+def build_interface(api_base: str, stream_chunk_seconds: float = 0.25):
+    """Build the Gradio interface."""
+    voices = fetch_voices(api_base)
+
+    css = """
+    #generate-btn button { width: 100%; }
+    .task-info { padding: 8px 12px; border-radius: 6px;
+                 background: #f0f4ff; margin-bottom: 8px; }
+    """
+
+    with gr.Blocks(css=css, title="Qwen3-TTS Demo") as demo:
+        gr.Markdown("# Qwen3-TTS Online Serving Demo")
+        gr.Markdown(f"**Server:** `{api_base}`")
+
+        with gr.Row():
+            # Left column: inputs
+            with gr.Column(scale=3):
+                text_input = gr.Textbox(
+                    label="Text to Synthesize",
+                    placeholder="Enter text here, e.g., Hello, how are you?",
+                    lines=4,
+                )
+
+                with gr.Row():
+                    task_type = gr.Radio(
+                        choices=TASK_TYPES,
+                        value="CustomVoice",
+                        label="Task Type",
+                        scale=2,
+                    )
+                    language = gr.Dropdown(
+                        choices=SUPPORTED_LANGUAGES,
+                        value="Auto",
+                        label="Language",
+                        scale=1,
+                    )
+
+                voice = gr.Dropdown(
+                    choices=voices,
+                    value=voices[0] if voices else None,
+                    label="Speaker",
+                    visible=True,
+                )
+
+                instructions = gr.Textbox(
+                    label="Instructions",
+                    placeholder=("e.g., Speak with excitement / A warm, friendly female voice"),
+                    lines=2,
+                    visible=True,
+                    info="Optional style/emotion instructions",
+                )
+
+                ref_audio = gr.Audio(
+                    label="Reference Audio (upload for voice cloning)",
+                    type="numpy",
+                    sources=["upload", "microphone"],
+                    visible=False,
+                )
+                ref_audio_url = gr.Textbox(
+                    label="Reference Audio URL",
+                    placeholder=("https://example.com/reference.wav (alternative to uploading)"),
+                    lines=1,
+                    visible=False,
+                )
+                ref_text = gr.Textbox(
+                    label="Reference Audio Transcript",
+                    placeholder=("Transcript of the reference audio (optional, improves quality)"),
+                    lines=2,
+                    visible=False,
+                )
+                x_vector_only = gr.Checkbox(
+                    label="Use x-vector only",
+                    value=False,
+                    visible=False,
+                    info=("Skip reference transcript, use speaker embedding only (lower quality)"),
+                )
+
+                with gr.Row():
+                    response_format = gr.Dropdown(
+                        choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
+                        value="wav",
+                        label="Audio Format",
+                        scale=1,
+                    )
+                    speed = gr.Slider(
+                        minimum=0.25,
+                        maximum=4.0,
+                        value=1.0,
+                        step=0.05,
+                        label="Speed",
+                        scale=1,
+                    )
+                    stream_checkbox = gr.Checkbox(
+                        label="Stream output",
+                        value=False,
+                        info=(
+                            "Enable streaming (PCM format, speed disabled). "
+                            "For gapless streaming, use gradio_fastrtc_demo.py"
+                        ),
+                        scale=1,
+                    )
+
+                generate_btn = gr.Button(
+                    "Generate Speech",
+                    variant="primary",
+                    size="lg",
+                    elem_id="generate-btn",
+                )
+
+            # Right column: output
+            with gr.Column(scale=2):
+                audio_output = gr.Audio(
+                    label="Generated Audio",
+                    interactive=False,
+                    streaming=True,
+                    autoplay=True,
+                )
+                gr.Markdown(
+                    "### Task Types\n"
+                    "- **CustomVoice**: Use a predefined speaker "
+                    "(Vivian, Ryan, etc.) with optional style instructions\n"
+                    "- **VoiceDesign**: Describe the desired voice in natural "
+                    "language (instructions required)\n"
+                    "- **Base**: Clone a voice from reference audio "
+                    "(upload a file or provide a URL)"
+                )
+
+        # Dynamic UI updates
+        task_type.change(
+            fn=on_task_type_change,
+            inputs=[task_type],
+            outputs=[
+                voice,
+                instructions,
+                ref_audio,
+                ref_audio_url,
+                ref_text,
+                x_vector_only,
+            ],
+        )
+
+        stream_checkbox.change(
+            fn=on_stream_change,
+            inputs=[stream_checkbox],
+            outputs=[response_format, speed],
+        )
+
+        all_inputs = [
+            text_input,
+            task_type,
+            voice,
+            language,
+            instructions,
+            ref_audio,
+            ref_audio_url,
+            ref_text,
+            x_vector_only,
+            response_format,
+            speed,
+        ]
+
+        def dispatch(stream_enabled, *args):
+            if stream_enabled:
+                yield from generate_speech_stream(
+                    api_base,
+                    *args,
+                    chunk_seconds=stream_chunk_seconds,
+                )
+            else:
+                yield generate_speech(api_base, *args)
+
+        generate_btn.click(
+            fn=dispatch,
+            inputs=[stream_checkbox] + all_inputs,
+            outputs=[audio_output],
+        )
+
+        demo.queue()
+    return demo
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description="Gradio demo for Qwen3-TTS online serving.")
+    add_common_args(parser)
+    parser.add_argument(
+        "--stream-chunk-seconds",
+        type=float,
+        default=0.25,
+        help=(
+            "Seconds of audio to buffer per streaming chunk (default: 0.25). "
+            "Lower = smoother but more overhead, higher = chunkier."
+        ),
+    )
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+
+    print(f"Connecting to vLLM server at: {args.api_base}")
+    demo = build_interface(
+        args.api_base,
+        stream_chunk_seconds=args.stream_chunk_seconds,
+    )
+
+    try:
+        demo.launch(
+            server_name=args.host,
+            server_port=args.port,
+            share=args.share,
+        )
+    except KeyboardInterrupt:
+        print("\nShutting down...")
+
+
+if __name__ == "__main__":
+    main()

examples/online_serving/qwen3_tts/gradio_fastrtc_demo.py

@@ -0,0 +1,274 @@
+"""Gradio demo for Qwen3-TTS using FastRTC for gapless streaming audio.
+
+Uses WebRTC via FastRTC to deliver audio chunks as a continuous real-time
+stream, avoiding the inter-chunk gaps present in Gradio's built-in streaming.
+
+Requires: pip install fastrtc
+
+Supports all 3 task types:
+  - CustomVoice: Predefined speaker with optional style instructions
+  - VoiceDesign: Natural language voice description
+  - Base: Voice cloning from reference audio (URL only in this demo)
+
+Usage:
+    # Start the server first (see run_server.sh), then:
+    python gradio_fastrtc_demo.py --api-base http://localhost:8000
+
+    # With public sharing:
+    python gradio_fastrtc_demo.py --api-base http://localhost:8000 --share
+"""
+
+import argparse
+import logging
+from collections.abc import Generator
+
+import gradio as gr
+import numpy as np
+from fastrtc import WebRTC
+from tts_common import (
+    PCM_SAMPLE_RATE,
+    SUPPORTED_LANGUAGES,
+    TASK_TYPES,
+    add_common_args,
+    build_payload,
+    fetch_voices,
+    stream_pcm_chunks,
+)
+
+logger = logging.getLogger(__name__)
+
+
+def make_tts_generator(api_base: str):
+    """Create a generator function for FastRTC's mode='receive'.
+
+    FastRTC calls this with the additional_inputs values whenever the user
+    clicks "Start Stream". It must be a synchronous generator yielding
+    (sample_rate, int16_ndarray) tuples.
+    """
+
+    def tts_generate(
+        text: str,
+        task_type: str,
+        voice: str,
+        language: str,
+        instructions: str,
+        ref_audio_url: str,
+        ref_text: str,
+        x_vector_only: bool,
+    ) -> Generator[tuple[int, np.ndarray], None, None]:
+        if not text or not text.strip():
+            return
+
+        payload = build_payload(
+            text,
+            task_type,
+            voice,
+            language,
+            instructions,
+            ref_audio=None,  # WebRTC UI doesn't support file upload
+            ref_audio_url=ref_audio_url,
+            ref_text=ref_text,
+            x_vector_only=x_vector_only,
+            stream=True,
+        )
+
+        try:
+            for samples in stream_pcm_chunks(api_base, payload):
+                yield (PCM_SAMPLE_RATE, samples)
+        except Exception:
+            logger.exception("TTS streaming error")
+
+    return tts_generate
+
+
+def build_interface(api_base: str):
+    """Build a custom Gradio Blocks UI matching gradio_demo.py layout."""
+    voices = fetch_voices(api_base)
+    handler = make_tts_generator(api_base)
+
+    css = """
+    #generate-btn button { width: 100%; }
+    .task-info { padding: 8px 12px; border-radius: 6px;
+                 background: #f0f4ff; margin-bottom: 8px; }
+    """
+
+    with gr.Blocks(css=css, title="Qwen3-TTS (FastRTC)") as demo:
+        gr.Markdown("# Qwen3-TTS Online Serving Demo (FastRTC)")
+        gr.Markdown(f"**Server:** `{api_base}` | **Transport:** WebRTC (gapless streaming)")
+
+        with gr.Row():
+            # Left column: inputs
+            with gr.Column(scale=3):
+                text_input = gr.Textbox(
+                    label="Text to Synthesize",
+                    placeholder="Enter text here, e.g., Hello, how are you?",
+                    lines=4,
+                )
+
+                with gr.Row():
+                    task_type = gr.Radio(
+                        choices=TASK_TYPES,
+                        value="CustomVoice",
+                        label="Task Type",
+                        scale=2,
+                    )
+                    language = gr.Dropdown(
+                        choices=SUPPORTED_LANGUAGES,
+                        value="Auto",
+                        label="Language",
+                        scale=1,
+                    )
+
+                voice = gr.Dropdown(
+                    choices=voices,
+                    value=voices[0] if voices else None,
+                    label="Speaker",
+                    visible=True,
+                )
+
+                instructions = gr.Textbox(
+                    label="Instructions",
+                    placeholder=("e.g., Speak with excitement / A warm, friendly female voice"),
+                    lines=2,
+                    visible=True,
+                    info="Optional style/emotion instructions",
+                )
+
+                # Base (voice clone) controls
+                # Note: file upload is not supported with WebRTC,
+                # use a URL instead.
+                ref_audio_url = gr.Textbox(
+                    label="Reference Audio URL",
+                    placeholder=("https://example.com/reference.wav (alternative to uploading)"),
+                    lines=1,
+                    visible=False,
+                )
+                ref_text = gr.Textbox(
+                    label="Reference Audio Transcript",
+                    placeholder=("Transcript of the reference audio (optional, improves quality)"),
+                    lines=2,
+                    visible=False,
+                )
+                x_vector_only = gr.Checkbox(
+                    label="Use x-vector only",
+                    value=False,
+                    visible=False,
+                    info=("Skip reference transcript, use speaker embedding only (lower quality)"),
+                )
+
+                generate_btn = gr.Button(
+                    "Generate Speech",
+                    variant="primary",
+                    size="lg",
+                    elem_id="generate-btn",
+                )
+
+            # Right column: output
+            with gr.Column(scale=2):
+                webrtc_output = WebRTC(
+                    label="Generated Audio (WebRTC)",
+                    mode="receive",
+                    modality="audio",
+                )
+                gr.Markdown(
+                    "### Task Types\n"
+                    "- **CustomVoice**: Use a predefined speaker "
+                    "(Vivian, Ryan, etc.) with optional style instructions\n"
+                    "- **VoiceDesign**: Describe the desired voice in natural "
+                    "language (instructions required)\n"
+                    "- **Base**: Clone a voice from reference audio "
+                    "(provide a URL)"
+                )
+
+        # Dynamic UI updates for task type
+        # FastRTC doesn't support file upload, so we only toggle
+        # 5 components (no ref_audio)
+        def on_task_type_change_fastrtc(task_type: str):
+            if task_type == "CustomVoice":
+                return (
+                    gr.update(visible=True),  # voice
+                    gr.update(visible=True, info="Optional style/emotion instructions"),
+                    gr.update(visible=False),  # ref_audio_url
+                    gr.update(visible=False),  # ref_text
+                    gr.update(visible=False),  # x_vector_only
+                )
+            elif task_type == "VoiceDesign":
+                return (
+                    gr.update(visible=False),
+                    gr.update(visible=True, info="Required: describe the voice style"),
+                    gr.update(visible=False),
+                    gr.update(visible=False),
+                    gr.update(visible=False),
+                )
+            elif task_type == "Base":
+                return (
+                    gr.update(visible=False),
+                    gr.update(visible=False),
+                    gr.update(visible=True),
+                    gr.update(visible=True),
+                    gr.update(visible=True),
+                )
+            return (
+                gr.update(visible=True),
+                gr.update(visible=True),
+                gr.update(visible=False),
+                gr.update(visible=False),
+                gr.update(visible=False),
+            )
+
+        task_type.change(
+            fn=on_task_type_change_fastrtc,
+            inputs=[task_type],
+            outputs=[
+                voice,
+                instructions,
+                ref_audio_url,
+                ref_text,
+                x_vector_only,
+            ],
+        )
+
+        # Wire up WebRTC streaming
+        all_inputs = [
+            text_input,
+            task_type,
+            voice,
+            language,
+            instructions,
+            ref_audio_url,
+            ref_text,
+            x_vector_only,
+        ]
+
+        webrtc_output.stream(
+            fn=handler,
+            inputs=all_inputs,
+            outputs=[webrtc_output],
+            trigger=generate_btn.click,
+        )
+
+        demo.queue()
+    return demo
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description="Gradio FastRTC demo for Qwen3-TTS (gapless streaming).")
+    add_common_args(parser)
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+    logging.basicConfig(level=logging.INFO)
+    print(f"Connecting to vLLM server at: {args.api_base}")
+
+    demo = build_interface(args.api_base)
+    demo.launch(
+        server_name=args.host,
+        server_port=args.port,
+        share=args.share,
+    )
+
+
+if __name__ == "__main__":
+    main()

examples/online_serving/qwen3_tts/run_gradio_demo.sh

@@ -0,0 +1,203 @@
+#!/bin/bash
+# Launch both vLLM server and Gradio demo for Qwen3-TTS
+#
+# Usage:
+#   ./run_gradio_demo.sh                                    # Default: CustomVoice
+#   ./run_gradio_demo.sh --task-type VoiceDesign            # VoiceDesign model
+#   ./run_gradio_demo.sh --task-type Base --gradio-port 7861
+#
+# Options:
+#   --task-type TYPE        Task type: CustomVoice, VoiceDesign, Base (default: CustomVoice)
+#   --server-port PORT      Port for vLLM server (default: 8000)
+#   --gradio-port PORT      Port for Gradio demo (default: 7860)
+#   --server-host HOST      Host for vLLM server (default: 0.0.0.0)
+#   --gradio-ip IP          IP for Gradio demo (default: 127.0.0.1)
+#   --share                 Share Gradio demo publicly
+
+set -e
+
+# Default values
+TASK_TYPE="CustomVoice"
+SERVER_PORT=8000
+GRADIO_PORT=7860
+SERVER_HOST="0.0.0.0"
+GRADIO_IP="127.0.0.1"
+GRADIO_SHARE=false
+
+# Parse command line arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --task-type)
+            TASK_TYPE="$2"
+            shift 2
+            ;;
+        --server-port)
+            SERVER_PORT="$2"
+            shift 2
+            ;;
+        --gradio-port)
+            GRADIO_PORT="$2"
+            shift 2
+            ;;
+        --server-host)
+            SERVER_HOST="$2"
+            shift 2
+            ;;
+        --gradio-ip)
+            GRADIO_IP="$2"
+            shift 2
+            ;;
+        --share)
+            GRADIO_SHARE=true
+            shift
+            ;;
+        --help)
+            echo "Usage: $0 [OPTIONS]"
+            echo ""
+            echo "Options:"
+            echo "  --task-type TYPE        Task type: CustomVoice, VoiceDesign, Base (default: CustomVoice)"
+            echo "  --server-port PORT      Port for vLLM server (default: 8000)"
+            echo "  --gradio-port PORT      Port for Gradio demo (default: 7860)"
+            echo "  --server-host HOST      Host for vLLM server (default: 0.0.0.0)"
+            echo "  --gradio-ip IP          IP for Gradio demo (default: 127.0.0.1)"
+            echo "  --share                 Share Gradio demo publicly"
+            echo ""
+            exit 0
+            ;;
+        *)
+            echo "Unknown option: $1"
+            echo "Use --help for usage information"
+            exit 1
+            ;;
+    esac
+done
+
+# Map task type to model
+case "$TASK_TYPE" in
+    CustomVoice)
+        MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice"
+        ;;
+    VoiceDesign)
+        MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign"
+        ;;
+    Base)
+        MODEL="Qwen/Qwen3-TTS-12Hz-1.7B-Base"
+        ;;
+    *)
+        echo "Unknown task type: $TASK_TYPE"
+        echo "Supported: CustomVoice, VoiceDesign, Base"
+        exit 1
+        ;;
+esac
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+API_BASE="http://localhost:${SERVER_PORT}"
+
+echo "=========================================="
+echo "Qwen3-TTS Gradio Demo"
+echo "=========================================="
+echo "Task Type : $TASK_TYPE"
+echo "Model     : $MODEL"
+echo "Server    : http://${SERVER_HOST}:${SERVER_PORT}"
+echo "Gradio    : http://${GRADIO_IP}:${GRADIO_PORT}"
+echo "=========================================="
+
+# Cleanup on exit
+cleanup() {
+    echo ""
+    echo "Shutting down..."
+    if [ -n "$SERVER_PID" ]; then
+        echo "Stopping vLLM server (PID: $SERVER_PID)..."
+        kill "$SERVER_PID" 2>/dev/null || true
+        wait "$SERVER_PID" 2>/dev/null || true
+    fi
+    if [ -n "$GRADIO_PID" ]; then
+        echo "Stopping Gradio demo (PID: $GRADIO_PID)..."
+        kill "$GRADIO_PID" 2>/dev/null || true
+        wait "$GRADIO_PID" 2>/dev/null || true
+    fi
+    echo "Cleanup complete"
+    exit 0
+}
+trap cleanup SIGINT SIGTERM
+
+# Start vLLM server
+echo ""
+echo "Starting vLLM server..."
+LOG_FILE="/tmp/vllm_tts_server_${SERVER_PORT}.log"
+
+vllm-omni serve "$MODEL" \
+    --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
+    --host "$SERVER_HOST" \
+    --port "$SERVER_PORT" \
+    --gpu-memory-utilization 0.9 \
+    --trust-remote-code \
+    --enforce-eager \
+    --omni 2>&1 | tee "$LOG_FILE" &
+SERVER_PID=$!
+
+# Wait for server startup
+echo ""
+echo "Waiting for vLLM server to be ready..."
+STARTUP_FLAG="/tmp/vllm_tts_startup_flag_${SERVER_PORT}.tmp"
+rm -f "$STARTUP_FLAG"
+
+(
+    tail -f "$LOG_FILE" 2>/dev/null | grep -m 1 "Application startup complete" > /dev/null && touch "$STARTUP_FLAG"
+) &
+TAIL_PID=$!
+
+MAX_WAIT=300
+ELAPSED=0
+while [ $ELAPSED -lt $MAX_WAIT ]; do
+    if [ -f "$STARTUP_FLAG" ]; then
+        kill "$TAIL_PID" 2>/dev/null || true
+        wait "$TAIL_PID" 2>/dev/null || true
+        echo ""
+        echo "vLLM server is ready!"
+        break
+    fi
+    if ! kill -0 "$SERVER_PID" 2>/dev/null; then
+        kill "$TAIL_PID" 2>/dev/null || true
+        echo ""
+        echo "Error: vLLM server failed to start"
+        exit 1
+    fi
+    sleep 1
+    ELAPSED=$((ELAPSED + 1))
+done
+
+rm -f "$STARTUP_FLAG"
+
+if [ $ELAPSED -ge $MAX_WAIT ]; then
+    kill "$TAIL_PID" 2>/dev/null || true
+    echo "Error: Server startup timed out after ${MAX_WAIT}s"
+    kill "$SERVER_PID" 2>/dev/null || true
+    exit 1
+fi
+
+# Start Gradio demo
+echo ""
+echo "Starting Gradio demo..."
+cd "$SCRIPT_DIR"
+GRADIO_CMD=("python" "gradio_demo.py" "--api-base" "$API_BASE" "--ip" "$GRADIO_IP" "--port" "$GRADIO_PORT")
+if [ "$GRADIO_SHARE" = true ]; then
+    GRADIO_CMD+=("--share")
+fi
+
+"${GRADIO_CMD[@]}" &
+GRADIO_PID=$!
+
+echo ""
+echo "=========================================="
+echo "Both services are running!"
+echo "=========================================="
+echo "vLLM Server : http://${SERVER_HOST}:${SERVER_PORT}"
+echo "Gradio Demo : http://${GRADIO_IP}:${GRADIO_PORT}"
+echo ""
+echo "Press Ctrl+C to stop both services"
+echo "=========================================="
+echo ""
+
+wait $SERVER_PID $GRADIO_PID || true
+cleanup

examples/online_serving/qwen3_tts/tts_common.py

@@ -0,0 +1,218 @@
+"""Shared constants, helpers, and payload building for Qwen3-TTS Gradio demos."""
+
+import base64
+import io
+
+import gradio as gr
+import httpx
+import numpy as np
+import soundfile as sf
+
+SUPPORTED_LANGUAGES = [
+    "Auto",
+    "Chinese",
+    "English",
+    "Japanese",
+    "Korean",
+    "German",
+    "French",
+    "Russian",
+    "Portuguese",
+    "Spanish",
+    "Italian",
+]
+
+TASK_TYPES = ["CustomVoice", "VoiceDesign", "Base"]
+
+PCM_SAMPLE_RATE = 24000
+
+DEFAULT_API_BASE = "http://localhost:8000"
+
+
+def fetch_voices(api_base: str) -> list[str]:
+    """Fetch available voices from the server."""
+    try:
+        with httpx.Client(timeout=10.0) as client:
+            resp = client.get(
+                f"{api_base}/v1/audio/voices",
+                headers={"Authorization": "Bearer EMPTY"},
+            )
+        if resp.status_code == 200:
+            data = resp.json()
+            return data.get("voices", ["Vivian", "Ryan"])
+    except Exception:
+        pass
+    return ["Vivian", "Ryan"]
+
+
+def encode_audio_to_base64(audio_data: tuple) -> str:
+    """Encode Gradio audio input (sample_rate, numpy_array) to base64 data URL."""
+    sample_rate, audio_np = audio_data
+
+    if audio_np.dtype != np.int16:
+        if audio_np.dtype in (np.float32, np.float64):
+            audio_np = np.clip(audio_np, -1.0, 1.0)
+            audio_np = (audio_np * 32767).astype(np.int16)
+        else:
+            audio_np = audio_np.astype(np.int16)
+
+    buf = io.BytesIO()
+    sf.write(buf, audio_np, sample_rate, format="WAV")
+    wav_b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
+    return f"data:audio/wav;base64,{wav_b64}"
+
+
+def build_payload(
+    text: str,
+    task_type: str,
+    voice: str,
+    language: str,
+    instructions: str,
+    ref_audio: tuple | None,
+    ref_audio_url: str,
+    ref_text: str,
+    x_vector_only: bool,
+    response_format: str = "pcm",
+    speed: float = 1.0,
+    stream: bool = True,
+) -> dict:
+    """Build the /v1/audio/speech request payload.
+
+    Raises gr.Error for invalid input so callers don't need to validate.
+    """
+    if not text or not text.strip():
+        raise gr.Error("Please enter text to synthesize.")
+
+    payload: dict = {
+        "input": text.strip(),
+        "response_format": "pcm" if stream else response_format,
+        "stream": stream,
+    }
+    if not stream:
+        payload["speed"] = speed
+
+    if task_type:
+        payload["task_type"] = task_type
+    if language:
+        payload["language"] = language
+
+    if task_type == "CustomVoice":
+        if voice:
+            payload["voice"] = voice
+        if instructions and instructions.strip():
+            payload["instructions"] = instructions.strip()
+
+    elif task_type == "VoiceDesign":
+        if not instructions or not instructions.strip():
+            raise gr.Error("VoiceDesign task requires voice style instructions.")
+        payload["instructions"] = instructions.strip()
+
+    elif task_type == "Base":
+        ref_audio_url_stripped = ref_audio_url.strip() if ref_audio_url else ""
+        if ref_audio_url_stripped:
+            payload["ref_audio"] = ref_audio_url_stripped
+        elif ref_audio is not None:
+            payload["ref_audio"] = encode_audio_to_base64(ref_audio)
+        else:
+            raise gr.Error("Base (voice clone) task requires reference audio. Upload a file or provide a URL.")
+        if ref_text and ref_text.strip():
+            payload["ref_text"] = ref_text.strip()
+        if x_vector_only:
+            payload["x_vector_only_mode"] = True
+
+    return payload
+
+
+def on_task_type_change(task_type: str):
+    """Update UI visibility based on selected task type."""
+    if task_type == "CustomVoice":
+        return (
+            gr.update(visible=True),  # voice dropdown
+            gr.update(visible=True, info="Optional style/emotion instructions"),
+            gr.update(visible=False),  # ref_audio
+            gr.update(visible=False),  # ref_audio_url
+            gr.update(visible=False),  # ref_text
+            gr.update(visible=False),  # x_vector_only
+        )
+    elif task_type == "VoiceDesign":
+        return (
+            gr.update(visible=False),
+            gr.update(visible=True, info="Required: describe the voice style"),
+            gr.update(visible=False),
+            gr.update(visible=False),
+            gr.update(visible=False),
+            gr.update(visible=False),
+        )
+    elif task_type == "Base":
+        return (
+            gr.update(visible=False),
+            gr.update(visible=False),
+            gr.update(visible=True),
+            gr.update(visible=True),
+            gr.update(visible=True),
+            gr.update(visible=True),
+        )
+    return (
+        gr.update(visible=True),
+        gr.update(visible=True),
+        gr.update(visible=False),
+        gr.update(visible=False),
+        gr.update(visible=False),
+        gr.update(visible=False),
+    )
+
+
+def stream_pcm_chunks(api_base: str, payload: dict):
+    """Stream raw PCM bytes from the server, yielding int16 numpy arrays.
+
+    Handles odd-byte boundaries between network chunks.
+    """
+    leftover = b""
+    with httpx.Client(timeout=300.0) as client:
+        with client.stream(
+            "POST",
+            f"{api_base}/v1/audio/speech",
+            json=payload,
+            headers={
+                "Content-Type": "application/json",
+                "Authorization": "Bearer EMPTY",
+            },
+        ) as resp:
+            if resp.status_code != 200:
+                resp.read()
+                raise gr.Error(f"Server error ({resp.status_code}): {resp.text}")
+            for chunk in resp.iter_bytes():
+                if not chunk:
+                    continue
+                raw = leftover + chunk
+                usable = len(raw) - (len(raw) % 2)
+                leftover = raw[usable:]
+                if usable == 0:
+                    continue
+                yield np.frombuffer(raw[:usable], dtype=np.int16).copy()
+
+
+def add_common_args(parser):
+    """Add CLI arguments shared by both demos."""
+    parser.add_argument(
+        "--api-base",
+        default=DEFAULT_API_BASE,
+        help=f"Base URL for the vLLM API server (default: {DEFAULT_API_BASE}).",
+    )
+    parser.add_argument(
+        "--host",
+        default="0.0.0.0",
+        help="Host/IP for Gradio server (default: 0.0.0.0).",
+    )
+    parser.add_argument(
+        "--port",
+        type=int,
+        default=7860,
+        help="Port for Gradio server (default: 7860).",
+    )
+    parser.add_argument(
+        "--share",
+        action="store_true",
+        help="Share the Gradio demo publicly.",
+    )
+    return parser