[Model] Add Ming-omni-tts dense 0.5B pipeline (#2906)

Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>

examples/online_serving/text_to_speech/README.md

@@ -16,6 +16,7 @@ For the full list of supported architectures across all modalities, see
 |---|---|---|---|---|---|
 | Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
 | GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
+| Ming-omni-tts | `inclusionAI/Ming-omni-tts-0.5B` | ✓ (`ref_audio` / `speaker_embedding`) | ✓ (PCM stream) | IP labels + structured `instructions` | — |
 | Ming-flash-omni-TTS | `Jonathan1909/Ming-flash-omni-2.0` | — (caption-controlled) | — | caption fields (`instructions`) | — |
 | MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | ✓ (`ref_audio` required) | ✓ (PCM stream) | — | ✓ |
 | OmniVoice | `k2-fsa/OmniVoice` | ✓ | — | — | — |
@@ -202,6 +203,51 @@ python fish_speech/gradio_demo.py --api-base http://localhost:8091  # if server
 
 ---
 
+## Ming-omni-tts
+
+Dense 0.5B two-stage TTS served through `/v1/audio/speech`. Ming uses the standard speech endpoint plus structured controls in `instructions`, `voice`, `language`, `ref_audio`, `ref_text`, and `speaker_embedding`.
+
+### Launch
+```bash
+bash examples/online_serving/text_to_speech/ming_tts/run_server.sh
+```
+Equivalent manual command:
+```bash
+vllm-omni serve inclusionAI/Ming-omni-tts-0.5B \
+    --deploy-config vllm_omni/deploy/ming_tts.yaml \
+    --host 0.0.0.0 --port 8091 \
+    --enforce-eager --omni
+```
+
+### Sending requests
+```bash
+python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \
+    --text "你好，这是 Ming 在线语音合成测试。"
+```
+
+Structured dialect control:
+```bash
+python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \
+    --text "我觉得社会企业同个人都有责任" \
+    --instruction-json '{"方言":"广粤话"}' \
+    --ref-audio /path/to/yue_prompt.wav
+```
+
+Zero-shot cloning:
+```bash
+python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \
+    --text "我们的愿景是构建未来服务业的数字化基础设施，为世界带来更多微小而美好的改变。" \
+    --ref-audio /path/to/10002287-00000094.wav \
+    --ref-text "在此奉劝大家别乱打美白针。"
+```
+
+### Notes
+- `run_curl.sh` keeps a small sanity subset; use the Ming README for the broader request cookbook.
+- Online serving is speech-shaped today; music-only `bgm` and text-to-audio `tta` remain offline examples.
+- Full request details live in [`ming_tts/README.md`](ming_tts/README.md).
+
+---
+
 ## Ming-flash-omni-TTS
 
 Standalone talker-only deployment of Ming-flash-omni-2.0. Voice is controlled through caption text passed via `instructions`.

examples/online_serving/text_to_speech/ming_tts/README.md

@@ -0,0 +1,113 @@
+# Ming-omni-tts Online Serving
+
+Serve the dense `inclusionAI/Ming-omni-tts-0.5B` two-stage TTS model through
+the OpenAI-compatible `/v1/audio/speech` endpoint.
+
+## Start Server
+
+```bash
+vllm-omni serve inclusionAI/Ming-omni-tts-0.5B \
+    --deploy-config vllm_omni/deploy/ming_tts.yaml \
+    --omni \
+    --port 8091 \
+    --enforce-eager
+```
+
+Or:
+
+```bash
+cd examples/online_serving/text_to_speech/ming_tts
+./run_server.sh
+```
+
+The tested ROCm environment is summarized in the
+[Ming recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/inclusionAI/Ming-omni-tts-0.5B.md).
+
+## Send Requests
+
+The Python client targets `http://localhost:8091/v1` with `api_key=EMPTY`; it
+does not call OpenAI's hosted API.
+
+```bash
+python openai_speech_client.py \
+    --text "你好，这是 Ming 在线语音合成测试。" \
+    --max-new-tokens 200
+```
+
+Style or dialect controls can be plain text or Ming JSON. The upstream
+dialect example also uses `yue_prompt.wav` for speaker conditioning:
+
+```bash
+python openai_speech_client.py \
+    --text "我觉得社会企业同个人都有责任" \
+    --instruction-json '{"方言":"广粤话"}' \
+    --ref-audio /path/to/yue_prompt.wav \
+    --max-new-tokens 200
+```
+
+When `--ref-audio` is supplied without `--ref-text`, the server extracts the
+Ming speaker embedding, matching upstream `use_spk_emb=True`, without using the
+audio as a zero-shot prompt.
+
+Reference-audio cloning:
+
+```bash
+python openai_speech_client.py \
+    --text "我们的愿景是构建未来服务业的数字化基础设施，为世界带来更多微小而美好的改变。" \
+    --ref-audio /path/to/10002287-00000094.wav \
+    --ref-text "在此奉劝大家别乱打美白针。" \
+    --max-new-tokens 200
+```
+
+Podcast-style multi-speaker prompt:
+
+```bash
+python openai_speech_client.py \
+    --text " speaker_1:你可以说一下，就大概说一下，可能虽然我也不知道，我看过那部电影没有。
+ speaker_2:就是那个叫什么，变相一节课的嘛。
+ speaker_1:嗯。
+ speaker_2:一部搞笑的电影。
+ speaker_1:一部搞笑的。" \
+    --ref-audio /path/to/CTS-CN-F2F-2019-11-11-423-012-A.wav \
+    --ref-audio /path/to/CTS-CN-F2F-2019-11-11-423-012-B.wav \
+    --ref-text " speaker_1:并且我们还要进行每个月还要考核 笔试的话还要进行笔试，做个，当服务员还要去笔试了
+ speaker_2:对啊，这真的很奇怪，就是 单纯的因，单纯自己工资不高，只是因为可能人家那个店比较出名一点，就对你苛刻要求"
+```
+
+Streaming PCM:
+
+```bash
+python openai_speech_client.py \
+    --text "你好，这是流式输出测试。" \
+    --stream \
+    --output ming_output.pcm
+```
+
+`run_curl.sh` keeps small smoke checks:
+
+```bash
+./run_curl.sh basic
+REF_AUDIO=/path/to/reference.wav REF_TEXT="在此奉劝大家别乱打美白针。" ./run_curl.sh zero_shot
+./run_curl.sh stream
+```
+
+## Request Fields
+
+| Field | Ming meaning |
+|-------|--------------|
+| `input` | target text |
+| `instructions` | plain style text, or JSON object for structured Ming controls |
+| `voice` | Ming IP voice label unless it resolves to an uploaded speaker |
+| `language` | Ming `方言` control |
+| `ref_audio` | speaker reference; with `ref_text`, also supplies the prompt waveform |
+| `ref_text` | transcript enabling zero-shot or podcast prompt-latent conditioning |
+| `speaker_embedding` | 192-d Ming speaker embedding |
+| `max_new_tokens` | Ming `max_decode_steps` |
+
+## Notes
+
+- `ref_audio` accepts local paths through the client, remote URLs, `file://`,
+  or `data:` URLs.
+- Non-streaming responses return WAV bytes; streaming responses return PCM.
+- Music-only `bgm` generation is offline-only until the API exposes Ming
+  prompt-mode selection.

examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py

@@ -0,0 +1,225 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""OpenAI-compatible client for Ming-omni-tts via /v1/audio/speech.
+
+Examples:
+    python openai_speech_client.py --text "你好，世界"
+    python openai_speech_client.py --text "我觉得社会企业同个人都有责任" \
+        --instruction-json '{"方言":"广粤话"}' --ref-audio yue_prompt.wav --max-new-tokens 200
+    python openai_speech_client.py \
+        --text "我们的愿景是构建未来服务业的数字化基础设施，为世界带来更多微小而美好的改变。" \
+        --ref-audio 10002287-00000094.wav --ref-text "在此奉劝大家别乱打美白针。" \
+        --max-new-tokens 200
+    python openai_speech_client.py --text "speaker_1:你好。 speaker_2:你好。" \
+        --ref-audio speaker_1.wav --ref-audio speaker_2.wav --ref-text "speaker_1:你好。 speaker_2:你好。"
+    python openai_speech_client.py --text "你好，这是流式输出测试。" \
+        --stream --output ming_output.pcm
+"""
+
+import argparse
+import base64
+import json
+import os
+
+import httpx
+
+DEFAULT_API_BASE = "http://localhost:8091"
+DEFAULT_API_KEY = "EMPTY"
+DEFAULT_MODEL = "inclusionAI/Ming-omni-tts-0.5B"
+EXPECTED_SPEAKER_EMBEDDING_DIM = 192
+
+
+def encode_audio_to_base64(audio_path: str) -> str:
+    """Encode a local audio file to a base64 data URL."""
+    if not os.path.exists(audio_path):
+        raise FileNotFoundError(f"Audio file not found: {audio_path}")
+    ext = audio_path.lower().rsplit(".", 1)[-1]
+    mime_map = {
+        "wav": "audio/wav",
+        "mp3": "audio/mpeg",
+        "flac": "audio/flac",
+        "ogg": "audio/ogg",
+        "aac": "audio/aac",
+    }
+    mime_type = mime_map.get(ext, "audio/wav")
+    with open(audio_path, "rb") as f:
+        audio_b64 = base64.b64encode(f.read()).decode("utf-8")
+    return f"data:{mime_type};base64,{audio_b64}"
+
+
+def load_speaker_embedding(path: str) -> list[float]:
+    """Load and validate a 192-d Ming speaker embedding JSON file."""
+    with open(path, encoding="utf-8") as f:
+        data = json.load(f)
+
+    if not isinstance(data, list):
+        raise ValueError("speaker_embedding file must contain a JSON list")
+    if len(data) != EXPECTED_SPEAKER_EMBEDDING_DIM:
+        raise ValueError(
+            f"Ming dense speaker_embedding must have {EXPECTED_SPEAKER_EMBEDDING_DIM} values, got {len(data)}"
+        )
+
+    values = []
+    for index, value in enumerate(data):
+        try:
+            values.append(float(value))
+        except (TypeError, ValueError) as exc:
+            raise ValueError(f"speaker_embedding[{index}] must be a number, got {value!r}") from exc
+    return values
+
+
+def build_instruction_payload(args) -> str | None:
+    """Return a string payload for the API `instructions` field."""
+    if args.instructions and args.instruction_json:
+        raise ValueError("Use either --instructions or --instruction-json, not both")
+    if args.instruction_json:
+        parsed = json.loads(args.instruction_json)
+        return json.dumps(parsed, ensure_ascii=False)
+    return args.instructions
+
+
+def validate_args(args) -> None:
+    """Fail fast on invalid combinations before hitting the server."""
+    if args.ref_text and not args.ref_audio:
+        raise ValueError("--ref-audio is required when --ref-text is provided")
+    if args.speaker_embedding and args.ref_audio and len(args.ref_audio) > 1:
+        raise ValueError("--speaker-embedding cannot be combined with multiple --ref-audio values")
+
+
+def run_tts(args) -> None:
+    """Generate speech via the OpenAI-compatible /v1/audio/speech API."""
+    validate_args(args)
+
+    payload = {
+        "model": args.model,
+        "input": args.text,
+        "response_format": args.response_format,
+    }
+
+    if args.voice:
+        payload["voice"] = args.voice
+    if args.task_type:
+        payload["task_type"] = args.task_type
+    if args.dialect:
+        payload["language"] = args.dialect
+
+    instructions = build_instruction_payload(args)
+    if instructions:
+        payload["instructions"] = instructions
+
+    if args.ref_audio:
+        ref_audio = []
+        for audio in args.ref_audio:
+            if audio.startswith(("http://", "https://", "data:")):
+                ref_audio.append(audio)
+            else:
+                ref_audio.append(encode_audio_to_base64(audio))
+        payload["ref_audio"] = ref_audio[0] if len(ref_audio) == 1 else ref_audio
+    if args.ref_text:
+        payload["ref_text"] = args.ref_text
+    if args.speaker_embedding:
+        payload["speaker_embedding"] = load_speaker_embedding(args.speaker_embedding)
+    if args.max_new_tokens:
+        payload["max_new_tokens"] = args.max_new_tokens
+    if args.stream:
+        payload["stream"] = True
+        payload["response_format"] = "pcm"
+
+    api_url = f"{args.api_base}/v1/audio/speech"
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": f"Bearer {args.api_key}",
+    }
+
+    print(f"Model: {args.model}")
+    print(f"Text: {args.text}")
+    print(f"Payload keys: {sorted(payload)}")
+
+    if args.stream:
+        output_path = args.output or "ming_output.pcm"
+        with httpx.Client(timeout=300.0) as client:
+            with client.stream("POST", api_url, json=payload, headers=headers) as response:
+                if response.status_code != 200:
+                    print(f"Error: {response.status_code}")
+                    print(response.read().decode())
+                    return
+                with open(output_path, "wb") as f:
+                    for chunk in response.iter_bytes():
+                        f.write(chunk)
+        print(f"Streamed PCM audio to: {output_path}")
+        return
+
+    with httpx.Client(timeout=300.0) as client:
+        response = client.post(api_url, json=payload, headers=headers)
+
+    if response.status_code != 200:
+        print(f"Error: {response.status_code}")
+        print(response.text)
+        return
+
+    try:
+        text = response.content.decode("utf-8")
+        if text.startswith('{"error"'):
+            print(f"Error: {text}")
+            return
+    except UnicodeDecodeError:
+        pass
+
+    output_path = args.output or "ming_output.wav"
+    with open(output_path, "wb") as f:
+        f.write(response.content)
+    print(f"Audio saved to: {output_path}")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="OpenAI-compatible client for Ming-omni-tts via /v1/audio/speech")
+    parser.add_argument("--api-base", default=DEFAULT_API_BASE, help="API base URL")
+    parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="API key")
+    parser.add_argument("--model", "-m", default=DEFAULT_MODEL, help="Model name or path")
+    parser.add_argument("--text", required=True, help="Text to synthesize")
+    parser.add_argument(
+        "--task-type",
+        default=None,
+        choices=["CustomVoice", "VoiceDesign", "Base"],
+        help="Optional compatibility task type. Ming accepts the same field but primarily uses prompt metadata.",
+    )
+    parser.add_argument(
+        "--voice",
+        default=None,
+        help="Maps to Ming `IP` when using built-in character voices, or to an uploaded voice sample name",
+    )
+    parser.add_argument("--dialect", default=None, help="Maps to Ming `方言`")
+    parser.add_argument("--instructions", default=None, help="Free-form Ming instruction string")
+    parser.add_argument(
+        "--instruction-json",
+        default=None,
+        help='Structured Ming instruction JSON, for example \'{"情感":"高兴"}\'',
+    )
+    parser.add_argument(
+        "--ref-audio",
+        action="append",
+        default=None,
+        help="Reference audio path, URL, or data URL. Repeat for podcast-style multi-speaker prompts.",
+    )
+    parser.add_argument("--ref-text", default=None, help="Reference transcript for cloning")
+    parser.add_argument(
+        "--speaker-embedding", default=None, help="Path to a JSON file containing a 192-d speaker embedding"
+    )
+    parser.add_argument("--max-new-tokens", type=int, default=None, help="Override ming_max_decode_steps")
+    parser.add_argument("--stream", action="store_true", help="Enable streaming PCM output")
+    parser.add_argument(
+        "--response-format",
+        default="wav",
+        choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
+        help="Audio format when not streaming",
+    )
+    parser.add_argument("--output", "-o", default=None, help="Output file path")
+    args = parser.parse_args()
+    try:
+        run_tts(args)
+    except (FileNotFoundError, ValueError, json.JSONDecodeError) as exc:
+        raise SystemExit(f"Error: {exc}") from exc
+
+
+if __name__ == "__main__":
+    main()

examples/online_serving/text_to_speech/ming_tts/run_curl.sh

@@ -0,0 +1,76 @@
+#!/bin/bash
+set -euo pipefail
+
+MODE="${1:-basic}"
+HOST="${HOST:-localhost}"
+PORT="${PORT:-8091}"
+MODEL="${MODEL:-inclusionAI/Ming-omni-tts-0.5B}"
+API_URL="http://${HOST}:${PORT}/v1/audio/speech"
+TEXT="${TEXT:-你好，这是 Ming 在线语音合成测试。}"
+OUTPUT="${OUTPUT:-ming_output.wav}"
+STREAM_OUTPUT="${STREAM_OUTPUT:-ming_output.pcm}"
+REF_AUDIO="${REF_AUDIO:-}"
+REF_TEXT="${REF_TEXT:-}"
+
+post_json() {
+    local payload="$1"
+    local output_path="$2"
+    curl -X POST "$API_URL" \
+        -H "Content-Type: application/json" \
+        -H "Authorization: Bearer EMPTY" \
+        -d "$payload" \
+        --output "$output_path"
+}
+
+case "$MODE" in
+    basic)
+        post_json "{
+            \"model\": \"${MODEL}\",
+            \"input\": \"${TEXT}\",
+            \"response_format\": \"wav\"
+        }" "$OUTPUT"
+        ;;
+    zero_shot)
+        if [ -z "$REF_AUDIO" ] || [ -z "$REF_TEXT" ]; then
+            echo "zero_shot requires REF_AUDIO and REF_TEXT" >&2
+            exit 1
+        fi
+        python - <<'PY' > /tmp/ming_zero_shot_payload.json
+import base64
+import json
+import mimetypes
+import os
+from pathlib import Path
+
+path = Path(os.environ["REF_AUDIO"])
+mime_type = mimetypes.guess_type(path.name)[0] or "audio/wav"
+payload = {
+    "model": os.environ["MODEL"],
+    "input": os.environ["TEXT"],
+    "ref_audio": f"data:{mime_type};base64,{base64.b64encode(path.read_bytes()).decode('utf-8')}",
+    "ref_text": os.environ["REF_TEXT"],
+    "response_format": "wav",
+}
+print(json.dumps(payload, ensure_ascii=False))
+PY
+        curl -X POST "$API_URL" \
+            -H "Content-Type: application/json" \
+            -H "Authorization: Bearer EMPTY" \
+            --data-binary @/tmp/ming_zero_shot_payload.json \
+            --output "$OUTPUT"
+        rm -f /tmp/ming_zero_shot_payload.json
+        ;;
+    stream)
+        post_json "{
+            \"model\": \"${MODEL}\",
+            \"input\": \"${TEXT}\",
+            \"stream\": true,
+            \"response_format\": \"pcm\"
+        }" "$STREAM_OUTPUT"
+        ;;
+    *)
+        echo "Unknown mode: $MODE" >&2
+        echo "Supported sanity checks: basic, zero_shot, stream" >&2
+        exit 1
+        ;;
+esac

examples/online_serving/text_to_speech/ming_tts/run_server.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# Launch vLLM-Omni server for Ming-omni-tts.
+#
+# Usage:
+#   ./run_server.sh
+#   PORT=8000 ./run_server.sh
+
+set -e
+
+DIR="$(cd "$(dirname "$0")" && pwd)"
+ROOT="$(cd "$DIR/../../../.." && pwd)"
+
+MODEL="${MODEL:-inclusionAI/Ming-omni-tts-0.5B}"
+PORT="${PORT:-8091}"
+DEPLOY_CONFIG="${DEPLOY_CONFIG:-$ROOT/vllm_omni/deploy/ming_tts.yaml}"
+
+echo "Starting Ming-omni-tts server with model: $MODEL"
+echo "Deploy config: $DEPLOY_CONFIG"
+
+vllm-omni serve "$MODEL" \
+    --deploy-config "$DEPLOY_CONFIG" \
+    --host 0.0.0.0 \
+    --port "$PORT" \
+    --enforce-eager \
+    --omni