anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Docs] Consolidate moss_tts_nano + ming_flash_omni_tts into TTS hub (#3358)

Browse Source 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
Yueqian Lin authored and GitHub committed 2026-05-05 17:36:16 +00:00
1 parent 44cde33eb5
commit 5cf3f7947b
20 files changed
+210 -494
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/online_serving/ming_flash_omni/README.md
+1 -1
View File
@@ -11,7 +11,7 @@ Please refer to [README.md](../../../README.md)
| Thinker + Talker (omni-speech, default) | `vllm serve ... --omni` | Text + Audio |
| Thinker only (multimodal understanding) | `vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
For standalone TTS (talker only), see [`examples/online_serving/ming_flash_omni_tts/`](../ming_flash_omni_tts/).
For standalone TTS (talker only), see the [Ming-flash-omni-TTS section in the Text-To-Speech hub](../text_to_speech/README.md#ming-flash-omni-tts).
## Run examples (Ming-flash-omni 2.0)
examples/online_serving/ming_flash_omni_tts/README.md
-54
View File
@@ -1,54 +0,0 @@
# Ming-flash-omni Standalone TTS (Online Serving)
This directory contains online e2e examples for **Ming-flash-omni-2.0 standalone talker deployment**.
Server uses:
- `model`: `Jonathan1909/Ming-flash-omni-2.0`
- `deploy config`: `vllm_omni/deploy/ming_flash_omni_tts.yaml`
## Launch the Server
```bash
# from repo root
bash examples/online_serving/ming_flash_omni_tts/run_server.sh
```
Equivalent manual command:
```bash
vllm serve Jonathan1909/Ming-flash-omni-2.0 \
--deploy-config vllm_omni/deploy/ming_flash_omni_tts.yaml \
--host 0.0.0.0 \
--port 8091 \
--trust-remote-code \
--omni
```
## Send TTS Request
Python client:
```bash
python examples/online_serving/ming_flash_omni_tts/speech_client.py \
--text "我们当迎着阳光辛勤耕作，去摘取，去制作，去品尝，去馈赠。" \
--output ming_online.wav
```
Long-form `instructions` (e.g. ASMR whisper style) via the client:
```bash
python examples/online_serving/ming_flash_omni_tts/speech_client.py \
--text "我会一直在这里陪着你，直到你慢慢、慢慢地沉入那个最温柔的梦里……好吗？" \
--instructions "这是一种ASMR耳语，属于一种旨在引发特殊感官体验的创意风格。这个女性使用轻柔的普通话进行耳语，声音气音成分重。音量极低，紧贴麦克风，语速极慢，旨在制造触发听者颅内快感的声学刺激。" \
--output ming_online_asmr.wav
```
## Notes
- This is the **online serving** counterpart of `examples/offline_inference/ming_flash_omni_tts/`.
- The server uses `use_zero_spk_emb=True` and the default decode args
(`max_decode_steps=200`, `cfg=2.0`, `sigma=0.25`, `temperature=0.0`).
For other caption fields (`语速`, `基频`, `IP`, BGM, etc.) or overriding
decode args, use the offline e2e example where `additional_information`
is set explicitly.
examples/online_serving/moss_tts_nano/README.md
-147
View File
@@ -1,147 +0,0 @@
# MOSS-TTS-Nano
## Model checkpoint
| Model | Description |
|-------|-------------|
| `OpenMOSS-Team/MOSS-TTS-Nano` | 0.1B AR LM + MOSS-Audio-Tokenizer-Nano codec, 48 kHz mono (mixed down from upstream stereo), ZH/EN/JA |
> **No built-in speaker presets.** Every request must include `ref_audio`.
> The server uses upstream's recommended `voice_clone` mode (per
> upstream's README and `infer.py` example). The OpenAI-schema `voice`
> and `ref_text` fields are accepted but ignored — `voice_clone` does
> not consume a transcript, and upstream's `continuation` mode (the only
> path that accepts `prompt_text`) emits near-silent output with a
> reference clip + transcript pair, so it is not exposed here.
>
> Sample reference clips are available in the upstream repo under
> [`assets/audio/`](https://github.com/OpenMOSS/MOSS-TTS-Nano/tree/main/assets/audio)
> (e.g. `zh_1.wav`, `en_2.wav`, `jp_2.wav`).
## Gradio Demo
An interactive Gradio demo is available with custom voice cloning and
streaming support. Upload your own reference audio in the UI.
```bash
# Option 1: Launch server + Gradio together
./run_gradio_demo.sh
# Option 2: If server is already running
python gradio_demo.py --api-base http://localhost:8091
```
Then open http://localhost:7860 in your browser.
## Launch the Server
```bash
vllm serve OpenMOSS-Team/MOSS-TTS-Nano --omni --port 8091
```
The deploy config at `vllm_omni/deploy/moss_tts_nano.yaml` auto-loads; no
`--stage-configs-path`, `--trust-remote-code`, or `--enforce-eager` flags
are needed.
Or use the convenience script:
```bash
./run_server.sh
```
## Send TTS Request
Every request needs `ref_audio` (base64 data URL). Reuse a saved sample:
```bash
# Fetch a sample reference clip from the upstream repo (one-off).
# Cache under XDG_CACHE_HOME so it survives across runs and stays user-scoped.
REF_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/moss-tts-nano"
mkdir -p "$REF_DIR"
REF_WAV="$REF_DIR/zh_1.wav"
[ -s "$REF_WAV" ] || curl -L -o "$REF_WAV" https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS-Nano/main/assets/audio/zh_1.wav
REF_AUDIO=$(base64 -w 0 "$REF_WAV")
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d "{
\"input\": \"你好，这是语音合成测试。\",
\"ref_audio\": \"data:audio/wav;base64,${REF_AUDIO}\",
\"response_format\": \"wav\"
}" --output output.wav
```
### Using Python
```python
import base64
import os
import urllib.request
from pathlib import Path
import httpx
ref_dir = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache")) / "moss-tts-nano"
ref_dir.mkdir(parents=True, exist_ok=True)
ref_wav = ref_dir / "zh_1.wav"
if not ref_wav.exists() or ref_wav.stat().st_size == 0:
urllib.request.urlretrieve(
"https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS-Nano/main/assets/audio/zh_1.wav",
ref_wav,
)
with ref_wav.open("rb") as f:
ref_audio_b64 = base64.b64encode(f.read()).decode("ascii")
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "你好，这是语音合成测试。",
"ref_audio": f"data:audio/wav;base64,{ref_audio_b64}",
"response_format": "wav",
},
timeout=300.0,
)
with open("output.wav", "wb") as f:
f.write(response.content)
```
### Streaming
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d "{
\"input\": \"Hello, streaming output from MOSS-TTS-Nano.\",
\"ref_audio\": \"data:audio/wav;base64,${REF_AUDIO}\",
\"stream\": true,
\"response_format\": \"pcm\"
}" --no-buffer | play -t raw -r 48000 -e signed -b 16 -c 1 -
```
**Note:** Output is 48 kHz mono PCM. Upstream's audio tokenizer is internally stereo at 48 kHz; the model wrapper averages the two channels into mono before reaching the engine, so playback duration / pitch are correct against the WAV header's 48 kHz rate.
## API Parameters
MOSS-TTS-Nano uses the standard `/v1/audio/speech` endpoint.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `input` | string | **required** | Text to synthesize (ZH / EN / JA) |
| `ref_audio` | string | **required** | Base64 data URL of the reference audio clip |
| `ref_text` | string | accepted, ignored | Schema-compatible field; voice_clone mode does not consume a transcript |
| `response_format` | string | `"wav"` | Audio format: wav, mp3, flac, pcm |
| `stream` | bool | false | Stream raw PCM chunks |
| `max_new_tokens` | int | 4096 | Maximum tokens to generate |
The `voice` and `ref_text` fields from the OpenAI schema are accepted but
ignored — there are no built-in speaker presets in MOSS-TTS-Nano, and
upstream's voice_clone mode does not consume a transcript.
## Troubleshooting
1. **`libnvrtc.so.13: cannot open shared object file`**: torchaudio 2.10+ defaults to torchcodec which requires NVRTC. vLLM-Omni patches this automatically at model load time to use soundfile instead.
2. **Connection refused**: Ensure the server is running on the correct port.
3. **Flashinfer version mismatch**: Set `FLASHINFER_DISABLE_VERSION_CHECK=1` if you see version warnings.
4. **Out of memory**: The default `gpu_memory_utilization=0.3` is conservative. Increase it in the stage config if you have more VRAM available.
examples/online_serving/text_to_speech/README.md
+102 -1
View File
@@ -15,9 +15,11 @@ For the full list of supported architectures across all modalities, see
| Model | HuggingFace repo | Voice cloning | Streaming | Voice presets / upload | Gradio demo |
|---|---|---|---|---|---|
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
| Ming-flash-omni-TTS | `Jonathan1909/Ming-flash-omni-2.0` | — (caption-controlled) | — | caption fields (`instructions`) | — |
| MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | ✓ (`ref_audio` required) | ✓ (PCM stream) | — | ✓ |
| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | ✓ (Base) | ✓ (PCM + WebSocket) | ✓ (presets + `/v1/audio/voices` upload) | ✓ (standard + FastRTC) |
| VoxCPM | local model dir | ✓ | ✓ (PCM stream) | — | — |
| VoxCPM | `openbmb/VoxCPM-0.5B` | ✓ | ✓ (PCM stream) | — | — |
| VoxCPM2 | `openbmb/VoxCPM2` | ✓ | ✓ (AudioWorklet via gradio) | — | ✓ |
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✓ (gated upstream) | ✓ | ✓ (presets) | ✓ |
@@ -141,6 +143,105 @@ python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server
---
## Ming-flash-omni-TTS
Standalone talker-only deployment of Ming-flash-omni-2.0. Voice is controlled through caption text passed via `instructions`.
### Launch
```bash
# from repo root
bash examples/online_serving/text_to_speech/ming_flash_omni_tts/run_server.sh
```
Equivalent manual command:
```bash
vllm serve Jonathan1909/Ming-flash-omni-2.0 \
--deploy-config vllm_omni/deploy/ming_flash_omni_tts.yaml \
--host 0.0.0.0 --port 8091 \
--trust-remote-code --omni
```
### Sending requests
```bash
python examples/online_serving/text_to_speech/ming_flash_omni_tts/speech_client.py \
--text "我们当迎着阳光辛勤耕作，去摘取，去制作，去品尝，去馈赠。" \
--output ming_online.wav
```
ASMR-style caption via `instructions`:
```bash
python examples/online_serving/text_to_speech/ming_flash_omni_tts/speech_client.py \
--text "我会一直在这里陪着你，直到你慢慢、慢慢地沉入那个最温柔的梦里……好吗？" \
--instructions "这是一种ASMR耳语，属于一种旨在引发特殊感官体验的创意风格。这个女性使用轻柔的普通话进行耳语，声音气音成分重。" \
--output ming_online_asmr.wav
```
### Notes
- Server uses `use_zero_spk_emb=True` and the cookbook decode defaults (`max_decode_steps=200`, `cfg=2.0`, `sigma=0.25`, `temperature=0.0`). For other caption fields (`语速`, `基频`, `IP`, BGM, etc.) or overriding decode args, use the offline example where `additional_information` is set explicitly.
- This is the online counterpart of [`examples/offline_inference/text_to_speech/ming_flash_omni_tts/`](../../offline_inference/text_to_speech/ming_flash_omni_tts/).
- For multimodal Ming-flash-omni online serving, see [`examples/online_serving/ming_flash_omni/`](../../ming_flash_omni/).
---
## MOSS-TTS-Nano
Single-stage 0.1B AR LM + MOSS-Audio-Tokenizer-Nano codec at 48 kHz mono. Every request must include `ref_audio`; there are no built-in speaker presets.
> The OpenAI-schema `voice` and `ref_text` fields are accepted but ignored — `voice_clone` does not consume a transcript, and upstream's `continuation` mode (the only path that accepts `prompt_text`) emits near-silent output, so it is not exposed here. Sample reference clips ship in the upstream repo under [`assets/audio/`](https://github.com/OpenMOSS/MOSS-TTS-Nano/tree/main/assets/audio).
### Launch
```bash
vllm serve OpenMOSS-Team/MOSS-TTS-Nano --omni --port 8091
# or:
./moss_tts_nano/run_server.sh
```
The deploy config at `vllm_omni/deploy/moss_tts_nano.yaml` auto-loads; no `--stage-configs-path`, `--trust-remote-code`, or `--enforce-eager` flags are needed.
### Sending requests
```bash
# One-off fetch of a sample reference clip; cache under XDG_CACHE_HOME.
REF_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/moss-tts-nano"
mkdir -p "$REF_DIR"
REF_WAV="$REF_DIR/zh_1.wav"
[ -s "$REF_WAV" ] || curl -L -o "$REF_WAV" https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS-Nano/main/assets/audio/zh_1.wav
REF_AUDIO=$(base64 -w 0 "$REF_WAV")
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d "{
\"input\": \"你好，这是语音合成测试。\",
\"ref_audio\": \"data:audio/wav;base64,${REF_AUDIO}\",
\"response_format\": \"wav\"
}" --output output.wav
```
### Streaming PCM
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d "{
\"input\": \"Hello, streaming output from MOSS-TTS-Nano.\",
\"ref_audio\": \"data:audio/wav;base64,${REF_AUDIO}\",
\"stream\": true,
\"response_format\": \"pcm\"
}" --no-buffer | play -t raw -r 48000 -e signed -b 16 -c 1 -
```
### Gradio demo
```bash
# Option 1: launch server + Gradio together
./moss_tts_nano/run_gradio_demo.sh
# Option 2: server already running
python moss_tts_nano/gradio_demo.py --api-base http://localhost:8091
```
Then open http://localhost:7860 in your browser.
### Notes
- Output is 48 kHz mono PCM (the upstream tokenizer is internally stereo at 48 kHz; the wrapper averages to mono before reaching the engine).
- Standard `/v1/audio/speech` request shape: `input`, `ref_audio` (base64 data URL), `response_format`, `stream`, `max_new_tokens`. The `voice` and `ref_text` fields from the OpenAI schema are accepted but ignored.
---
## OmniVoice
Zero-shot multilingual TTS (600+ languages). Online serving currently exposes **auto voice** only; voice cloning and voice design are available offline.
examples/online_serving/ming_flash_omni_tts/run_server.sh → examples/online_serving/text_to_speech/ming_flash_omni_tts/run_server.sh
View File
File renamed without changes.
examples/online_serving/ming_flash_omni_tts/speech_client.py → examples/online_serving/text_to_speech/ming_flash_omni_tts/speech_client.py
View File
File renamed without changes.
examples/online_serving/moss_tts_nano/gradio_demo.py → examples/online_serving/text_to_speech/moss_tts_nano/gradio_demo.py
View File
File renamed without changes.
examples/online_serving/moss_tts_nano/run_gradio_demo.sh → examples/online_serving/text_to_speech/moss_tts_nano/run_gradio_demo.sh
View File
File renamed without changes.
examples/online_serving/moss_tts_nano/run_server.sh → examples/online_serving/text_to_speech/moss_tts_nano/run_server.sh
View File
File renamed without changes.