[Model] Ming-flash-omni-2.0 Omni-Speech and TTS (#2890)
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com> Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com> Signed-off-by: LHXuuu <xulianhao.xlh@antgroup.com> Co-authored-by: LHXuuu <xulianhao.xlh@antgroup.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
32 files changed
+4620
-261
No files matched your search
@@ -4,38 +4,49 @@
|
||||
|
||||
Please refer to [README.md](../../../README.md)
|
||||
|
||||
## Deployment modes
|
||||
|
||||
| Mode | Launch command | Output |
|
||||
|------|---------------|--------|
|
||||
| Thinker only (multimodal understanding) | `vllm serve ... --omni` | Text |
|
||||
| Thinker + Talker (omni-speech) | `vllm serve ... --omni --stage-configs-path ming_flash_omni.yaml` | Text + Audio |
|
||||
|
||||
For standalone TTS (talker only), see [`examples/online_serving/ming_flash_omni_tts/`](../ming_flash_omni_tts/).
|
||||
|
||||
## Run examples (Ming-flash-omni 2.0)
|
||||
|
||||
### Launch the Server
|
||||
|
||||
**Thinker only (text output):**
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091
|
||||
```
|
||||
|
||||
If you have custom stage configs file, launch the server with command below
|
||||
**Thinker + Talker (omni-speech, text + audio output):**
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni --port 8091 \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni.yaml
|
||||
```
|
||||
|
||||
Pass `--stage-configs-path /path/to/your_config.yaml` to use a custom stage
|
||||
config.
|
||||
|
||||
### Send Multi-modal Request
|
||||
|
||||
#### Send request via python
|
||||
Shared Python client (supports `text | use_image | use_audio | use_video |
|
||||
use_mixed_modalities`; pass `--image-path` / `--audio-path` / `--video-path`
|
||||
for local files or URLs, `--modalities text` for output, `--help` for the
|
||||
full flag list):
|
||||
|
||||
```bash
|
||||
python examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py --model Jonathan1909/Ming-flash-omni-2.0 --query-type use_mixed_modalities --port 8091 --host "localhost" --modalities text
|
||||
python examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--model Jonathan1909/Ming-flash-omni-2.0 \
|
||||
--query-type use_mixed_modalities \
|
||||
--port 8091 --host localhost \
|
||||
--modalities text
|
||||
```
|
||||
|
||||
The Python client supports the following command-line arguments:
|
||||
|
||||
- `--query-type` (or `-q`): Query type. Options: `text`, `use_audio`, `use_image`, `use_video`, `use_mixed_modalities`
|
||||
- `--video-path` (or `-v`): Path to local video file or URL. If not provided and query-type uses video, uses default video URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs. Example: `--video-path /path/to/video.mp4` or `--video-path https://example.com/video.mp4`
|
||||
- `--image-path` (or `-i`): Path to local image file or URL. If not provided and query-type uses image, uses default image URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common image formats: JPEG, PNG, GIF, WebP. Example: `--image-path /path/to/image.jpg` or `--image-path https://example.com/image.png`
|
||||
- `--audio-path` (or `-a`): Path to local audio file or URL. If not provided and query-type uses audio, uses default audio URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common audio formats: MP3, WAV, OGG, FLAC, M4A. Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
|
||||
- `--prompt` (or `-p`): Custom text prompt/question. If not provided, uses default prompt for the selected query type. Example: `--prompt "What are the main activities shown in this video?"`
|
||||
- `--modalities`: Output modalities. For now, only `text` is supported. Example: `--modalities text`
|
||||
|
||||
|
||||
#### Send request via curl
|
||||
Parameterized curl wrapper in this directory:
|
||||
|
||||
```bash
|
||||
bash run_curl_multimodal_generation.sh text
|
||||
@@ -47,83 +58,17 @@ bash run_curl_multimodal_generation.sh use_mixed_modalities
|
||||
|
||||
## Modality control
|
||||
|
||||
Ming-flash-omni 2.0 currently supports text output only (thinker stage).
|
||||
| `modalities` | Server config | Output |
|
||||
|-------------|--------------|--------|
|
||||
| `["text"]` or omitted | Thinker only | Text |
|
||||
| `["audio"]` | Thinker + Talker | Audio (speech) |
|
||||
| `["text", "audio"]` | Thinker + Talker | Text + Audio |
|
||||
|
||||
| Modalities | Output |
|
||||
|------------|--------|
|
||||
| `["text"]` | Text only |
|
||||
| Not specified | Text only (default) |
|
||||
For ready-to-copy curl examples (text / audio / multimodal input, SSE
|
||||
streaming, reasoning mode), see the recipe at
|
||||
[`recipes/inclusionAI/Ming-flash-omni-2.0.md`](../../../recipes/inclusionAI/Ming-flash-omni-2.0.md).
|
||||
|
||||
### Using curl
|
||||
|
||||
```bash
|
||||
curl http://localhost:8091/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Jonathan1909/Ming-flash-omni-2.0",
|
||||
"messages": [
|
||||
{"role": "system", "content": [{"type": "text", "text": "你是一个友好的AI助手。\n\ndetailed thinking off"}]},
|
||||
{"role": "user", "content": "请详细介绍鹦鹉的生活习性。"}
|
||||
],
|
||||
"modalities": ["text"]
|
||||
}'
|
||||
```
|
||||
|
||||
### Using OpenAI Python SDK
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8091/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Jonathan1909/Ming-flash-omni-2.0",
|
||||
messages=[
|
||||
{"role": "system", "content": [{"type": "text", "text": "你是一个友好的AI助手。\n\ndetailed thinking off"}]},
|
||||
{"role": "user", "content": "请详细介绍鹦鹉的生活习性。"},
|
||||
],
|
||||
modalities=["text"],
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Multi-modal input with OpenAI Python SDK
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8091/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Jonathan1909/Ming-flash-omni-2.0",
|
||||
messages=[
|
||||
{"role": "system", "content": [{"type": "text", "text": "你是一个友好的AI助手。\n\ndetailed thinking off"}]},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg"}},
|
||||
{"type": "text", "text": "Describe this image in detail."},
|
||||
],
|
||||
},
|
||||
],
|
||||
modalities=["text"],
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Streaming Output
|
||||
|
||||
To enable streaming output:
|
||||
|
||||
```bash
|
||||
python examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type use_image \
|
||||
--model Jonathan1909/Ming-flash-omni-2.0 \
|
||||
--modalities text \
|
||||
--stream
|
||||
```
|
||||
|
||||
Or with the OpenAI Python SDK:
|
||||
## OpenAI Python SDK — streaming
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
@@ -146,59 +91,5 @@ for chunk in response:
|
||||
print()
|
||||
```
|
||||
|
||||
Or using curl:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8091/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Jonathan1909/Ming-flash-omni-2.0",
|
||||
"messages": [
|
||||
{"role": "system", "content": [{"type": "text", "text": "你是一个友好的AI助手。\n\ndetailed thinking off"}]},
|
||||
{"role": "user", "content": "请详细介绍鹦鹉的生活习性。"}
|
||||
],
|
||||
"modalities": ["text"],
|
||||
"stream": true,
|
||||
}'
|
||||
```
|
||||
|
||||
|
||||
## Reasoning (Thinking Mode)
|
||||
|
||||
To enable reasoning/thinking mode, change `detailed thinking off` to `detailed thinking on` in the system prompt:
|
||||
|
||||
### Using curl
|
||||
|
||||
```bash
|
||||
curl http://localhost:8091/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Jonathan1909/Ming-flash-omni-2.0",
|
||||
"messages": [
|
||||
{"role": "system", "content": [{"type": "text", "text": "你是一个友好的AI助手。\n\ndetailed thinking on"}]},
|
||||
{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://example.com/math_problem.png"}},
|
||||
{"type": "text", "text": "Solve this math problem step by step."}
|
||||
]}
|
||||
],
|
||||
"modalities": ["text"]
|
||||
}'
|
||||
```
|
||||
|
||||
### Using OpenAI Python SDK
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8091/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Jonathan1909/Ming-flash-omni-2.0",
|
||||
messages=[
|
||||
{"role": "system", "content": [{"type": "text", "text": "你是一个友好的AI助手。\n\ndetailed thinking on"}]},
|
||||
{"role": "user", "content": "If a train travels 120 km in 2 hours, what is its average speed?"},
|
||||
],
|
||||
modalities=["text"],
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
The `--stream` flag on the Python client script above shows the same pattern
|
||||
driven by the shared multimodal client.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Ming-flash-omni Standalone TTS (Online Serving)
|
||||
|
||||
This directory contains online e2e examples for **Ming-flash-omni-2.0 standalone talker deployment**.
|
||||
|
||||
Server uses:
|
||||
|
||||
- `model`: `Jonathan1909/Ming-flash-omni-2.0`
|
||||
- `stage config`: `vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml`
|
||||
|
||||
## Launch the Server
|
||||
|
||||
```bash
|
||||
# from repo root
|
||||
bash examples/online_serving/ming_flash_omni_tts/run_server.sh
|
||||
```
|
||||
|
||||
Equivalent manual command:
|
||||
|
||||
```bash
|
||||
vllm serve Jonathan1909/Ming-flash-omni-2.0 \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml \
|
||||
--host 0.0.0.0 \
|
||||
--port 8091 \
|
||||
--trust-remote-code \
|
||||
--omni
|
||||
```
|
||||
|
||||
## Send TTS Request
|
||||
|
||||
Python client:
|
||||
|
||||
```bash
|
||||
python examples/online_serving/ming_flash_omni_tts/speech_client.py \
|
||||
--text "我们当迎着阳光辛勤耕作,去摘取,去制作,去品尝,去馈赠。" \
|
||||
--output ming_online.wav
|
||||
```
|
||||
|
||||
Long-form `instructions` (e.g. ASMR whisper style) via the client:
|
||||
|
||||
```bash
|
||||
python examples/online_serving/ming_flash_omni_tts/speech_client.py \
|
||||
--text "我会一直在这里陪着你,直到你慢慢、慢慢地沉入那个最温柔的梦里……好吗?" \
|
||||
--instructions "这是一种ASMR耳语,属于一种旨在引发特殊感官体验的创意风格。这个女性使用轻柔的普通话进行耳语,声音气音成分重。音量极低,紧贴麦克风,语速极慢,旨在制造触发听者颅内快感的声学刺激。" \
|
||||
--output ming_online_asmr.wav
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- This is the **online serving** counterpart of `examples/offline_inference/ming_flash_omni_tts/`.
|
||||
- The server uses `use_zero_spk_emb=True` and the default decode args
|
||||
(`max_decode_steps=200`, `cfg=2.0`, `sigma=0.25`, `temperature=0.0`).
|
||||
For other caption fields (`语速`, `基频`, `IP`, BGM, etc.) or overriding
|
||||
decode args, use the offline e2e example where `additional_information`
|
||||
is set explicitly.
|
||||
@@ -0,0 +1,25 @@
|
||||
#!/bin/bash
|
||||
# Launch vLLM-Omni server for Ming-flash-omni-2.0 standalone talker (TTS).
|
||||
#
|
||||
# Usage:
|
||||
# ./run_server.sh
|
||||
# MODEL=/path/to/local/model ./run_server.sh
|
||||
# PORT=8091 ./run_server.sh
|
||||
# HOST=127.0.0.1 ./run_server.sh # bind only to loopback
|
||||
|
||||
set -e
|
||||
|
||||
MODEL="${MODEL:-Jonathan1909/Ming-flash-omni-2.0}"
|
||||
HOST="${HOST:-0.0.0.0}"
|
||||
PORT="${PORT:-8091}"
|
||||
STAGE_CONFIG="${STAGE_CONFIG:-vllm_omni/model_executor/stage_configs/ming_flash_omni_tts.yaml}"
|
||||
|
||||
echo "Starting Ming standalone TTS server with model: $MODEL"
|
||||
echo "Stage config: $STAGE_CONFIG"
|
||||
|
||||
vllm serve "$MODEL" \
|
||||
--stage-configs-path "$STAGE_CONFIG" \
|
||||
--host "$HOST" \
|
||||
--port "$PORT" \
|
||||
--trust-remote-code \
|
||||
--omni
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Client for Ming standalone TTS via /v1/audio/speech endpoint."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
|
||||
import httpx
|
||||
|
||||
DEFAULT_API_BASE = "http://localhost:8091"
|
||||
DEFAULT_API_KEY = "EMPTY"
|
||||
DEFAULT_MODEL = "Jonathan1909/Ming-flash-omni-2.0"
|
||||
|
||||
|
||||
def run_tts(args) -> None:
|
||||
payload = {
|
||||
"model": args.model,
|
||||
"input": args.text,
|
||||
"response_format": args.response_format,
|
||||
}
|
||||
|
||||
instructions = args.instructions
|
||||
if args.instruction_json:
|
||||
if instructions:
|
||||
sys.exit("--instructions and --instruction-json are mutually exclusive")
|
||||
|
||||
try:
|
||||
parsed = json.loads(args.instruction_json)
|
||||
except json.JSONDecodeError as exc:
|
||||
sys.exit(f"--instruction-json must be valid JSON: {exc}")
|
||||
if not isinstance(parsed, dict):
|
||||
sys.exit("--instruction-json must decode to a JSON object")
|
||||
# Re-encode with ensure_ascii=False so UTF-8 Chinese keys/values
|
||||
# arrive at the server intact rather than as \\uXXXX escapes.
|
||||
instructions = json.dumps(parsed, ensure_ascii=False)
|
||||
if instructions:
|
||||
payload["instructions"] = instructions
|
||||
|
||||
print(f"Model: {args.model}")
|
||||
print(f"Text: {args.text}")
|
||||
print("Generating audio...")
|
||||
|
||||
api_url = f"{args.api_base}/v1/audio/speech"
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {args.api_key}",
|
||||
}
|
||||
|
||||
with httpx.Client(timeout=300.0) as client:
|
||||
response = client.post(api_url, json=payload, headers=headers)
|
||||
|
||||
if response.status_code != 200:
|
||||
print(f"Error: {response.status_code}")
|
||||
print(response.text)
|
||||
return
|
||||
|
||||
output_path = args.output or "ming_tts_output.wav"
|
||||
with open(output_path, "wb") as f:
|
||||
f.write(response.content)
|
||||
print(f"Audio saved to: {output_path}")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Ming standalone TTS speech client")
|
||||
parser.add_argument("--api-base", default=DEFAULT_API_BASE, help="API base URL")
|
||||
parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="API key")
|
||||
parser.add_argument("--model", "-m", default=DEFAULT_MODEL, help="Model name or local path")
|
||||
parser.add_argument("--text", required=True, help="Text to synthesize")
|
||||
parser.add_argument(
|
||||
"--response-format",
|
||||
default="wav",
|
||||
choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
|
||||
help="Audio format (default: wav)",
|
||||
)
|
||||
parser.add_argument("--output", "-o", default=None, help="Output file path")
|
||||
parser.add_argument(
|
||||
"--instructions",
|
||||
default=None,
|
||||
help="Free-form style description (mapped to caption 风格 on the server).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--instruction-json",
|
||||
default=None,
|
||||
help=(
|
||||
"Structured caption JSON forwarded as `instructions`. Accepts Ming "
|
||||
"caption keys: 方言, 风格, 语速, 基频, 音量, 情感, IP, 说话人, BGM. "
|
||||
),
|
||||
)
|
||||
args = parser.parse_args()
|
||||
run_tts(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user