[Misc] Remove Dynin-Omni and dots.tts support (#7655)
Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
wangyuandYueqian Lin authored and GitHub committed
1 parent
3f94c417be
commit
6a03e45c34
56 files changed
+133
-12749
No files matched your search
@@ -2,7 +2,6 @@
|
||||
# The docs generator combines these names with the serving mode for each page H1.
|
||||
aura_omni: AURA Omni
|
||||
covo_audio: Covo-Audio-Chat
|
||||
dynin_omni: Dynin-Omni
|
||||
lance: Lance
|
||||
mimo_audio: MiMo-Audio
|
||||
ming_flash_omni: Ming-flash-omni 2.0
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
# Dynin-Omni: Offline inference
|
||||
|
||||
This folder contains a unified offline inference entrypoint:
|
||||
|
||||
- `end2end.py`
|
||||
|
||||
## 1. Environment Setup
|
||||
|
||||
Run from repository root:
|
||||
|
||||
```bash
|
||||
cd <REPO_ROOT>
|
||||
```
|
||||
|
||||
If needed, install this repo in editable mode:
|
||||
|
||||
```bash
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
## 2. Extra Dependencies (EMOVA)
|
||||
|
||||
Install the following packages for EMOVA-related components:
|
||||
|
||||
```bash
|
||||
pip install \
|
||||
"phonemizer==3.3.0" \
|
||||
"Unidecode==1.4.0" \
|
||||
"hydra-core==1.3.2" \
|
||||
"pytorch-lightning==1.1.0" \
|
||||
"wget==3.2" \
|
||||
"wrapt==2.1.1" \
|
||||
"onnx==1.20.1" \
|
||||
"frozendict==2.4.7" \
|
||||
"inflect==7.5.0" \
|
||||
"braceexpand==0.1.7" \
|
||||
"webdataset==1.0.2" \
|
||||
"torch-stft==0.1.4" \
|
||||
"editdistance==0.8.1"
|
||||
```
|
||||
|
||||
## 3. Hardware and VRAM Requirements
|
||||
|
||||
This example uses a 3-stage pipeline on one GPU by default
|
||||
([`dynin_omni.yaml`](../../../vllm_omni/deploy/dynin_omni.yaml)):
|
||||
|
||||
- Stage-0 (`token2text`): `gpu_memory_utilization: 0.5`
|
||||
- Stage-1 (`token2image`): `gpu_memory_utilization: 0.1`
|
||||
- Stage-2 (`token2audio`): `gpu_memory_utilization: 0.1`
|
||||
|
||||
### Requested GPU Memory Budget from `gpu_memory_utilization`
|
||||
|
||||
| Stage | Utilization | A100 80GB | H200 141GB |
|
||||
| :-- | :-- | :-- | :-- |
|
||||
| Stage-0 (token2text) | 0.5 | ~40.0 GB | ~70.5 GB |
|
||||
| Stage-1 (token2image) | 0.1 | ~8.0 GB | ~14.1 GB |
|
||||
| Stage-2 (token2audio) | 0.1 | ~8.0 GB | ~14.1 GB |
|
||||
| Total requested budget | 0.7 | ~56.0 GB | ~98.7 GB |
|
||||
|
||||
### Observed Runtime Signal (from your log)
|
||||
|
||||
- Stage-0 reported: `Model loading took 15.12 GiB memory` (weights footprint signal).
|
||||
- Stages 1/2 can still add runtime memory depending on task path and backend allocations.
|
||||
- Keep extra headroom for CUDA/PyTorch overhead and temporary allocations.
|
||||
|
||||
### GPU Compatibility
|
||||
|
||||
- Confirmed target GPUs for this setup: **NVIDIA H200**, **NVIDIA A100**.
|
||||
- CI/e2e coverage in this repo also includes CUDA **L4** markers for Dynin tests.
|
||||
|
||||
## 4. End2End Run Examples
|
||||
|
||||
```bash
|
||||
# t2t
|
||||
python <REPO_ROOT>/examples/offline_inference/dynin_omni/end2end.py \
|
||||
--task t2t --model snu-aidas/Dynin-Omni --text <INSTRUCTION_TEXT>
|
||||
|
||||
# i2t
|
||||
python <REPO_ROOT>/examples/offline_inference/dynin_omni/end2end.py \
|
||||
--task i2t --model snu-aidas/Dynin-Omni --image <IMAGE_PATH> --text "Please describe this image in detail."
|
||||
|
||||
# s2t
|
||||
python <REPO_ROOT>/examples/offline_inference/dynin_omni/end2end.py \
|
||||
--task s2t --model snu-aidas/Dynin-Omni --audio <AUDIO_PATH> --text "Transcribe the given audio."
|
||||
|
||||
# t2i
|
||||
python <REPO_ROOT>/examples/offline_inference/dynin_omni/end2end.py \
|
||||
--task t2i --model snu-aidas/Dynin-Omni --text <INSTRUCTION_TEXT>
|
||||
|
||||
# v2t
|
||||
python <REPO_ROOT>/examples/offline_inference/dynin_omni/end2end.py \
|
||||
--task v2t --model snu-aidas/Dynin-Omni --video <VIDEO_PATH> --text "Describe this video in detail."
|
||||
|
||||
# i2i
|
||||
python <REPO_ROOT>/examples/offline_inference/dynin_omni/end2end.py \
|
||||
--task i2i --model snu-aidas/Dynin-Omni --image <IMAGE_PATH> --text <INSTRUCTION_TEXT>
|
||||
|
||||
# t2s
|
||||
python <REPO_ROOT>/examples/offline_inference/dynin_omni/end2end.py \
|
||||
--task t2s --model snu-aidas/Dynin-Omni --text <INSTRUCTION_TEXT>
|
||||
```
|
||||
|
||||
## 5. Notes
|
||||
|
||||
- Outputs are saved under task-specific directories in `/tmp` by default.
|
||||
- You can override output path with `--output-dir`.
|
||||
- If you want to force local config resolution, pass `--dynin-config-path <PATH_TO_DYNIN_OMNI_YAML>`.
|
||||
- If you see the warning
|
||||
`max_num_batched_tokens (32768) exceeds max_num_seqs * max_model_len (4096)`,
|
||||
reduce `max_num_batched_tokens` in the deploy config (for example, `4096` in the CI config).
|
||||
File diff suppressed because it is too large.
Load diff
@@ -13,7 +13,7 @@ list of supported architectures across all modalities, see
|
||||
## Supported Models
|
||||
|
||||
| Model | HuggingFace repo | Stages | Voice cloning | Streaming | Special modes | Sample rate |
|
||||
|---|---|---|---|---|---|---|
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| AuK | `tencent/AuK`, `tencent/AuK-Flash` (assembled with `tools/prepare_auk_checkpoint.py`) | 2 (thinker encoder + DiT) | ✓ | — | instruction-driven editing, enhancement, separation; no example script (use `vllm_omni.model_extras.auk`) | 24 kHz |
|
||||
| Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | dual-AR | ✓ | ✓ | 11 languages | 44.1 kHz |
|
||||
| Breeze-TTS-2 | `BreezeBlue/Breeze-TTS-2` | 2 (talker + codec) | ✓ | ✓ (async chunk) | voice design / voice direction (`--instruction`) | 24 kHz |
|
||||
@@ -27,7 +27,6 @@ list of supported architectures across all modalities, see
|
||||
| OmniVoice | `k2-fsa/OmniVoice` | 2 (gen + dec) | ✓ | — | voice design, language hint | 24 kHz |
|
||||
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | 2 (talker + code2wav) | ✓ (Base) | ✓ | 3 task variants | 24 kHz |
|
||||
| VoxCPM2 | `openbmb/VoxCPM2` | single (native AR) | ✓ | ✓ (online) | continuation | 48 kHz |
|
||||
| dots.tts | `dots-studio/dots.tts-soar` | single (native AR) | — (text-only) | ✓ (online) | no-reference synthesis | 48 kHz |
|
||||
| IndexTTS-2 | `IndexTeam/IndexTTS-2` | 2 (AR talker + S2Mel DiT + BigVGAN) | ✓ (required) | — | emotion control (`--emo-audio`, `--emo-text`, `--emo-vector`) | 22.05 kHz |
|
||||
| IndexTTS-2.5 | native `checkpoints/` bundle | 2 (AR talker + EnhancedCodec + S2Mel DiT + BigVGAN) | ✓ (required) | — | multilingual (`--lang`) + emotion control | 22.05 kHz |
|
||||
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | varies | ✓ | ✓ | voice presets | 24 kHz |
|
||||
@@ -54,32 +53,38 @@ codec: a slow AR transformer predicts one semantic token per codec frame, and a
|
||||
4-layer fast AR predicts that frame's remaining 9 codebooks.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
No extra packages: the codec architecture ships in tree and its weights
|
||||
(`codec.pth`) come with the checkpoint.
|
||||
|
||||
### Quick start
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
|
||||
--text "Welcome to Audio8 TTS, a compact multilingual text to speech model."
|
||||
```
|
||||
|
||||
### Voice cloning
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
|
||||
--text "Welcome to Audio8 TTS." \
|
||||
--ref-audio /path/to/reference.wav \
|
||||
--ref-text "The exact transcript of the reference recording."
|
||||
```
|
||||
|
||||
The reference transcript must match what is actually spoken in the clip;
|
||||
a mismatch degrades both stability and speaker similarity.
|
||||
|
||||
### Streaming
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
|
||||
--text "Welcome to Audio8 TTS." --streaming
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 44.1 kHz mono WAV; the codec emits ~21.5 frames/s (2048 samples per frame).
|
||||
- Context is 2048 packed text+audio positions, so `max_model_len` is 2048.
|
||||
- Greedy decoding (`temperature: 0`) is degenerate for this checkpoint — it
|
||||
@@ -92,6 +97,7 @@ python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
|
||||
- `--num-prompts 4` smoke-tests concurrency (stage 0 runs `max_num_seqs: 4`).
|
||||
|
||||
### Measured behaviour (1x H20, shared GPU)
|
||||
|
||||
`--streaming` on a 58-character English sentence: TTFA ~0.74 s (4-frame first
|
||||
chunk), 7 chunks, 1.55 s wall for 3.16 s of audio (RTF ~0.49). Treat these as
|
||||
smoke-test magnitudes, not a benchmark.
|
||||
@@ -103,12 +109,14 @@ smoke-test magnitudes, not a benchmark.
|
||||
2-stage TTS pipeline (`talker` + `code2wav`) at 24 kHz.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
```bash
|
||||
uv pip install -e .
|
||||
# Includes soundfile, onnxruntime, x-transformers, einops via requirements.
|
||||
```
|
||||
|
||||
Download the model snapshot:
|
||||
|
||||
```python
|
||||
from huggingface_hub import snapshot_download
|
||||
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512',
|
||||
@@ -116,15 +124,18 @@ snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512',
|
||||
```
|
||||
|
||||
If your downloaded checkpoint lacks `config.json`, add it:
|
||||
|
||||
```json
|
||||
{
|
||||
"model_type": "cosyvoice3",
|
||||
"architectures": ["CosyVoice3Model"]
|
||||
}
|
||||
```
|
||||
|
||||
This is required because `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers the class mapping; the loader still reads `model_type` from `config.json` to select the class.
|
||||
|
||||
### Quick start
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
|
||||
--model pretrained_models/Fun-CosyVoice3-0.5B \
|
||||
@@ -132,9 +143,11 @@ python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
|
||||
```
|
||||
|
||||
### Voice cloning
|
||||
|
||||
If `--ref-audio` is omitted, the script downloads the upstream
|
||||
[`zero_shot_prompt.wav`](https://github.com/FunAudioLLM/CosyVoice/blob/main/asset/zero_shot_prompt.wav) from the CosyVoice repo into the current directory.
|
||||
To use your own clip, pass `--ref-audio /path/to/reference.wav`, and modify `--prompt-text` correspondingly.
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
|
||||
--model pretrained_models/Fun-CosyVoice3-0.5B \
|
||||
@@ -144,9 +157,11 @@ python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
Streaming is enabled by default via `async_chunk: true` in `vllm_omni/deploy/cosyvoice3.yaml`. Pass `--no-async-chunk` on `vllm serve` to switch to the legacy synchronous path.
|
||||
|
||||
### Notes
|
||||
|
||||
- Stage 0 (`talker`) emits speech tokens; stage 1 (`code2wav`) runs flow matching + HiFiGAN to synthesize waveform.
|
||||
- Deploy config auto-loads from `vllm_omni/deploy/cosyvoice3.yaml` based on HF `model_type`. Pass `--deploy-config <path>` to override.
|
||||
|
||||
@@ -157,6 +172,7 @@ Streaming is enabled by default via `async_chunk: true` in `vllm_omni/deploy/cos
|
||||
2-stage TTS pipeline (AR + DiT flow-matching) at 24 kHz. Every request requires reference audio and its transcript for zero-shot voice cloning.
|
||||
|
||||
### Quick start
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/glm_tts/end2end.py \
|
||||
--model zai-org/GLM-TTS \
|
||||
@@ -167,12 +183,14 @@ python examples/offline_inference/text_to_speech/glm_tts/end2end.py \
|
||||
```
|
||||
|
||||
### Architecture
|
||||
|
||||
```text
|
||||
Text → [Stage 0: AR] → Speech Tokens → [Stage 1: DiT + HiFT] → Audio (24 kHz)
|
||||
(Llama-based) (32k vocab) (Flow Matching)
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- `--ref-audio` and `--ref-text` are **required** together; GLM-TTS does not support text-only synthesis.
|
||||
- Reference audio should be 3-10 seconds.
|
||||
- First run may be slow due to lazy loading of WhisperVQ tokenizer and CampPlus ONNX speaker embedder.
|
||||
@@ -187,12 +205,14 @@ Text → [Stage 0: AR] → Speech Tokens → [Stage 1: DiT + HiFT] → Audio (24
|
||||
No extra packages are required; the DAC codec is vendored in vLLM-Omni.
|
||||
|
||||
### Quick start
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
|
||||
--text "Hello, this is a test of the Fish Speech text to speech system."
|
||||
```
|
||||
|
||||
### Voice cloning
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
|
||||
--text "Hello, this is a cloned voice." \
|
||||
@@ -201,14 +221,17 @@ python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
|
||||
--text "Hello, this is a streaming test." \
|
||||
--streaming
|
||||
```
|
||||
|
||||
Streaming requires `async_chunk: true` in the deploy config.
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 44.1 kHz mono WAV.
|
||||
- DAC codec weights (`codec.pth`) are loaded lazily from the model directory.
|
||||
|
||||
@@ -219,6 +242,7 @@ Streaming requires `async_chunk: true` in the deploy config.
|
||||
Dense 0.5B two-stage TTS pipeline (`AR + flow` + audio VAE) at 44.1 kHz. The example covers style, IP voice, music-only generation, text-to-audio events, emotion, dialect, zero-shot cloning, podcast, speech+BGM, and speech+environment-sound cases.
|
||||
|
||||
### Quick start
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
|
||||
--case style \
|
||||
@@ -227,6 +251,7 @@ python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
|
||||
```
|
||||
|
||||
### Voice cloning
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
|
||||
--case zero_shot \
|
||||
@@ -237,6 +262,7 @@ python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
|
||||
--case basic \
|
||||
@@ -247,6 +273,7 @@ python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- `style`, `ip`, `bgm`, and `tta` do not require reference audio.
|
||||
- Reference-audio cases use `--ref-audio`; `zero_shot` also requires `--ref-text`.
|
||||
- `podcast` uses multiple references via `--ref-audio-paths`.
|
||||
@@ -259,14 +286,17 @@ python examples/offline_inference/text_to_speech/ming_tts/end2end.py \
|
||||
Standalone talker-only deployment of Ming-flash-omni-2.0 at 44.1 kHz. Voice is controlled through caption fields (`风格` / `IP` / `语速`/`基频`/`音量`) rather than reference audio.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
The example calls into `vllm_omni.model_executor.models.ming_flash_omni.prompt_utils` for the default prompt and instruction builder; no extra pip install on top of the base vLLM-Omni install.
|
||||
|
||||
### Quick start
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/ming_flash_omni_tts/end2end.py --case style
|
||||
```
|
||||
|
||||
### Cases
|
||||
|
||||
```bash
|
||||
# ASMR-style whisper (caption-driven)
|
||||
python examples/offline_inference/text_to_speech/ming_flash_omni_tts/end2end.py --case style
|
||||
@@ -281,6 +311,7 @@ python examples/offline_inference/text_to_speech/ming_flash_omni_tts/end2end.py
|
||||
Override the default text per case with `--text`, write to a custom path with `--output`.
|
||||
|
||||
### Notes
|
||||
|
||||
- Talker-only deployment — for the multimodal Ming-flash-omni example, see [`examples/offline_inference/ming_flash_omni/`](../../ming_flash_omni/).
|
||||
- Deploy config: `vllm_omni/deploy/ming_flash_omni_tts.yaml` (single GPU, `enforce_eager`, `max_num_seqs: 1`).
|
||||
- Decode defaults from the Ming cookbook: `max_decode_steps=200`, `cfg=2.0`, `sigma=0.25`, `temperature=0.0`, `use_zero_spk_emb=True`.
|
||||
@@ -294,6 +325,7 @@ Single-stage 0.1B AR LM + MOSS-Audio-Tokenizer-Nano codec at 48 kHz mono (mixed
|
||||
> **No built-in speaker presets.** `--ref-audio` is required on every call. Default `--mode voice_clone` matches upstream's recommended workflow; `--mode continuation` is exposed for completeness but upstream's continuation-with-prompt path emits very short / near-silent output, so it is rarely useful in practice. Sample reference clips ship in the upstream repo under [`assets/audio/`](https://github.com/OpenMOSS/MOSS-TTS-Nano/tree/main/assets/audio) (e.g. `zh_1.wav`, `en_2.wav`, `jp_2.wav`).
|
||||
|
||||
### Quick start
|
||||
|
||||
```bash
|
||||
# Fetch a sample reference clip (one-off, user-scoped cache).
|
||||
REF_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/moss-tts-nano"
|
||||
@@ -305,9 +337,11 @@ python examples/offline_inference/text_to_speech/moss_tts_nano/end2end.py \
|
||||
--text "你好，这是MOSS-TTS-Nano的语音合成演示。" \
|
||||
--ref-audio "$REF_DIR/zh_1.wav"
|
||||
```
|
||||
|
||||
The first run downloads `OpenMOSS-Team/MOSS-TTS-Nano` and `OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano` from Hugging Face.
|
||||
|
||||
### Reproducible runs
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/moss_tts_nano/end2end.py \
|
||||
--text "Deterministic test." \
|
||||
@@ -316,6 +350,7 @@ python examples/offline_inference/text_to_speech/moss_tts_nano/end2end.py \
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 48 kHz mono WAV (the tokenizer is internally stereo at 48 kHz; the wrapper averages to mono before reaching the engine).
|
||||
- Deploy config: `vllm_omni/deploy/moss_tts_nano.yaml` (auto-loaded; override with `--deploy-config`).
|
||||
- Default `--max-new-frames 375` ≈ 14 s of audio; raise for longer outputs.
|
||||
@@ -329,12 +364,15 @@ python examples/offline_inference/text_to_speech/moss_tts_nano/end2end.py \
|
||||
Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design).
|
||||
|
||||
### Prerequisites
|
||||
|
||||
```bash
|
||||
huggingface-cli download k2-fsa/OmniVoice
|
||||
```
|
||||
|
||||
Voice cloning requires `transformers>=5.3.0`. Auto and design modes work with `transformers>=4.57.0`.
|
||||
|
||||
### Quick start (auto voice)
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
--model k2-fsa/OmniVoice \
|
||||
@@ -342,6 +380,7 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
```
|
||||
|
||||
### Voice cloning
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
--model k2-fsa/OmniVoice \
|
||||
@@ -351,6 +390,7 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
```
|
||||
|
||||
### Voice design
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
--model k2-fsa/OmniVoice \
|
||||
@@ -359,6 +399,7 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
```
|
||||
|
||||
### Language hint
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
--model k2-fsa/OmniVoice \
|
||||
@@ -367,6 +408,7 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
```
|
||||
|
||||
### Seed for Reproducibility
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
--model k2-fsa/OmniVoice \
|
||||
@@ -375,6 +417,7 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking.
|
||||
- Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz.
|
||||
|
||||
@@ -385,13 +428,16 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
|
||||
3-task-variant TTS with 24 kHz output. Has its own argparse surface (this script does not follow the common `--text` / `--ref-audio` shape).
|
||||
|
||||
### Prerequisites
|
||||
|
||||
For ROCm builds, replace `onnxruntime` with `onnxruntime-rocm`:
|
||||
|
||||
```bash
|
||||
pip uninstall onnxruntime
|
||||
pip install onnxruntime-rocm
|
||||
```
|
||||
|
||||
### Task variants
|
||||
|
||||
- `CustomVoice`: predefined speaker (speaker ID) with optional style instruction.
|
||||
- `VoiceDesign`: text + descriptive instruction designs a new voice.
|
||||
- `Base`: voice cloning from reference audio + transcript.
|
||||
@@ -415,22 +461,27 @@ python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-ty
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
|
||||
--query-type CustomVoice \
|
||||
--streaming \
|
||||
--output-dir /tmp/out_stream
|
||||
```
|
||||
|
||||
Streaming requires `async_chunk: true` in the deploy config.
|
||||
|
||||
### Word Timestamps
|
||||
|
||||
Word-level timestamps are currently a serving-path feature: launch
|
||||
`vllm-omni serve` with `--forced-aligner` and request `word_timestamps`
|
||||
(see `examples/online_serving/text_to_speech/README.md`). An offline
|
||||
example is not provided in this release.
|
||||
|
||||
### Batched decoding
|
||||
|
||||
The Code2Wav stage supports batched decoding through the SpeechTokenizer. Pass multiple prompts via `--txt-prompts` and set `--batch-size` accordingly. To raise `max_num_seqs` on either stage, create an overlay based on `vllm_omni/deploy/qwen3_tts.yaml` and pass it with `--deploy-config`:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
|
||||
--query-type CustomVoice \
|
||||
@@ -438,9 +489,11 @@ python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
|
||||
--batch-size 4 \
|
||||
--deploy-config /path/to/qwen3_tts_batched.yaml
|
||||
```
|
||||
|
||||
`--batch-size` must match a CUDA-graph capture size (1, 2, 4, 8, 16…).
|
||||
|
||||
### Notes
|
||||
|
||||
- Run `--help` for the full argument surface.
|
||||
- See `qwen3_tts/end2end.py` for the prompt-length-estimation logic the Talker uses.
|
||||
|
||||
@@ -451,6 +504,7 @@ python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
|
||||
Single-stage native AR TTS at 22.05 kHz. Pipeline: `Qwen3.5 backbone → 32 FSQ codebook heads (one frame/step) → NeMo NanoCodec`. Backbone runs under vLLM paged attention; the 32-head sampling + learned embedding feedback ride the native-AR runner hooks. `enforce_eager` (CUDA graph is a perf follow-up).
|
||||
|
||||
### Prerequisites
|
||||
|
||||
The NanoCodec decoder needs NeMo (NVIDIA Open Model License), installed separately:
|
||||
|
||||
```bash
|
||||
@@ -480,15 +534,18 @@ failure rather than an environment one. The PyTorch sampler it falls back to cos
|
||||
here.
|
||||
|
||||
### Quick start (zero-shot, default voice)
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/gepard/end2end.py \
|
||||
--text "Hello, this is a Gepard demo."
|
||||
```
|
||||
|
||||
### Voice cloning
|
||||
|
||||
Not yet — PR1 is zero-shot only (the learned `null_prefix` default voice). Reference-audio cloning (`ref_compressor` speaker prefix) is a follow-up PR.
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 22.05 kHz mono WAV.
|
||||
- First run downloads two checkpoints, not one: the model itself, and the NanoCodec decoder it names in `codec_id` (`nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps` by default). Both are fetched automatically; expect the first startup to be correspondingly slower.
|
||||
- Offline and online: `examples/offline_inference/text_to_speech/gepard/end2end.py` and `/v1/audio/speech` (see the [online Gepard section](../../online_serving/text_to_speech/README.md#gepard-10)).
|
||||
@@ -502,6 +559,7 @@ Not yet — PR1 is zero-shot only (the learned `null_prefix` default voice). Ref
|
||||
Single-stage native AR TTS at 48 kHz. Pipeline: `feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE`.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
```bash
|
||||
pip install voxcpm
|
||||
# or, for a local source checkout:
|
||||
@@ -509,6 +567,7 @@ export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm
|
||||
```
|
||||
|
||||
### Quick start
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
|
||||
--model openbmb/VoxCPM2 \
|
||||
@@ -516,7 +575,9 @@ python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
|
||||
```
|
||||
|
||||
### Voice cloning
|
||||
|
||||
Pass a reference audio for isolated cloning, or both `--ref-audio` + `--ref-text` for prompt continuation:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
|
||||
--text "Hello, this is a voice clone demo." \
|
||||
@@ -525,39 +586,16 @@ python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
Streaming is exposed through the online OpenAI Speech API (`stream=true`). See [`examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py`](../../online_serving/text_to_speech/voxcpm2/gradio_demo.py) for an AudioWorklet-based gapless streaming player; the offline `end2end.py` script does not expose a streaming path.
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 48 kHz mono WAV.
|
||||
- Deploy config: `vllm_omni/deploy/voxcpm2.yaml` (auto-loaded by HF `model_type`).
|
||||
|
||||
---
|
||||
|
||||
## dots.tts
|
||||
|
||||
Single-stage native AR TTS at 48 kHz (rednote-hilab). Pipeline: `Qwen2.5-1.5B base LM → DiT (10-step Euler flow matching) → patch_encoder AR loopback → AudioVAE (streaming decode)`. Same "vLLM-native base LM + side-path computation" pattern as VoxCPM2, with a plain Qwen2 backbone instead of MiniCPM4 and no FSQ / residual LM stage.
|
||||
|
||||
### Quick start
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/dots_tts/end2end.py \
|
||||
--model dots-studio/dots.tts-soar \
|
||||
--text "Hello, this is a test of dots TTS running on vLLM Omni."
|
||||
```
|
||||
|
||||
### Voice cloning
|
||||
Not wired in this release. Generation is text-only and does not consume reference audio, a named voice, or a precomputed speaker embedding. The CAM++ x-vector speaker encoder weights load, but `end2end.py` has no `--ref-audio`/`--ref-text` flags yet.
|
||||
|
||||
### Streaming
|
||||
The AudioVAE decoder emits incremental audio through its internal streaming path (`init_stream_state` / `stream_step` / `stream_flush`). Online streaming is exposed through the OpenAI-compatible `/v1/audio/speech` endpoint with `stream=true`; see the [online serving guide](../../online_serving/text_to_speech/README.md#dotstts). The offline `end2end.py` script returns the consolidated waveform.
|
||||
|
||||
### Notes
|
||||
- Output: 48 kHz mono WAV.
|
||||
- Deploy config: `vllm_omni/deploy/dots_tts.yaml` (auto-loaded by HF `model_type`).
|
||||
- Checkpoints: `dots-studio/dots.tts-soar` is the validated default. `dots.tts-base` shares the same architecture but is unvalidated in this repo. `dots.tts-mf` (MeanFlow, 2-4 step) is not supported yet.
|
||||
- Known limitation: no CUDA graph capture and no batched side-path computation yet, so concurrent requests do not currently scale (each request's DiT Euler steps run serially). See `recipes/rednote-hilab/dots.tts.md` for details and the roadmap.
|
||||
|
||||
---
|
||||
|
||||
## IndexTTS-2 and IndexTTS-2.5
|
||||
|
||||
Both versions use a 2-stage TTS pipeline and produce 22.05 kHz mono audio. Stage 0 is a GPT AR talker. IndexTTS-2 Stage 1 uses RepCodec semantic embeddings, GPT latent, S2Mel CFM/DiT, and BigVGAN. IndexTTS-2.5 replaces RepCodec with EnhancedCodec and uses the official code-only path (`use_gpt_latent=false`). Every request requires reference audio for zero-shot voice cloning.
|
||||
@@ -592,6 +630,7 @@ python examples/offline_inference/text_to_speech/indextts2/end2end.py \
|
||||
```
|
||||
|
||||
### Emotion control
|
||||
|
||||
```bash
|
||||
# Emotion from reference audio
|
||||
python examples/offline_inference/text_to_speech/indextts2/end2end.py \
|
||||
@@ -616,6 +655,7 @@ python examples/offline_inference/text_to_speech/indextts2/end2end.py \
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- `--ref-audio` is **required** — neither version provides text-only synthesis.
|
||||
- IndexTTS-2.5 Stage 0 (AR Talker) generates mel codes from text + reference audio using plain vLLM sampling. It does not reproduce the official default `num_beams=3` beam search. For parity comparisons, run upstream with `num_beams=1`; output quality can differ from the official beam-search result.
|
||||
- IndexTTS-2.5 accepts language codes such as `zh`, `en`, `zhen` (mixed Chinese/English), `ja`, and `yue`. `Mandarin` is a vLLM-Omni convenience alias for `zh`. Japanese (`ja`) uses `fugashi` tokenization and produces audio, but it does not automatically expand numbers, dates, or percentages; write those inputs as readable Japanese text before calling the model. Use `--no-text-normalization` only when the input is already normalized.
|
||||
@@ -631,12 +671,15 @@ python examples/offline_inference/text_to_speech/indextts2/end2end.py \
|
||||
Voxtral-4B-TTS (Mistral). Has its own argparse surface; uses voice presets and the `mistral_common` `SpeechRequest` protocol.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Latest `mistral_common` with `SpeechRequest` support:
|
||||
|
||||
```bash
|
||||
pip install -e /path/to/mistral-common # or upgrade from PyPI when available
|
||||
```
|
||||
|
||||
### Quick start (voice preset)
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
|
||||
--write-audio --voice cheerful_female \
|
||||
@@ -645,6 +688,7 @@ python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
|
||||
```
|
||||
|
||||
### Voice cloning (capability gated upstream)
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
|
||||
--write-audio \
|
||||
@@ -654,12 +698,14 @@ python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
|
||||
```
|
||||
|
||||
### Streaming + concurrency
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
|
||||
--num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
|
||||
--model mistralai/Voxtral-4B-TTS-2603 \
|
||||
--text "..."
|
||||
```
|
||||
|
||||
Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`).
|
||||
|
||||
### Notes
|
||||
|
||||
@@ -1,125 +0,0 @@
|
||||
"""Offline dots.tts inference example (single-stage native AR pipeline).
|
||||
|
||||
Uses the single-stage native AR config (vllm_omni/deploy/dots_tts.yaml).
|
||||
Reference checkpoint: dots-studio/dots.tts-soar.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import soundfile as sf
|
||||
import torch
|
||||
|
||||
from vllm_omni import Omni
|
||||
from vllm_omni.utils.tracking_parser import TrackingArgumentParser
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[4]
|
||||
SAMPLE_RATE = 48_000
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = TrackingArgumentParser(description="Offline dots.tts native AR inference")
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
type=str,
|
||||
default="dots-studio/dots.tts-soar",
|
||||
help="dots.tts model path or HuggingFace repo ID.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--text",
|
||||
type=str,
|
||||
default="Hello, this is a test of dots TTS running on vLLM Omni.",
|
||||
help="Text to synthesize.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=str,
|
||||
default="output_audio",
|
||||
help="Directory for output WAV files.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--deploy-config",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Override the deploy config path. If unset, auto-loads "
|
||||
"vllm_omni/deploy/dots_tts.yaml based on the HF model_type.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def extract_audio(multimodal_output: dict) -> torch.Tensor:
|
||||
"""Extract the final complete audio tensor from multimodal output.
|
||||
|
||||
The output processor concatenates per-step delta tensors under
|
||||
``model_outputs``. Falls back to ``audio`` for backwards compat.
|
||||
"""
|
||||
audio = multimodal_output.get("model_outputs")
|
||||
if audio is None:
|
||||
audio = multimodal_output.get("audio")
|
||||
if audio is None:
|
||||
raise ValueError(f"No audio key in multimodal_output: {list(multimodal_output.keys())}")
|
||||
|
||||
if isinstance(audio, list):
|
||||
valid = [torch.as_tensor(a).float().cpu().reshape(-1) for a in audio if a is not None]
|
||||
if not valid:
|
||||
raise ValueError("Audio list is empty or all elements are None.")
|
||||
return torch.cat(valid, dim=0) if len(valid) > 1 else valid[0]
|
||||
|
||||
return torch.as_tensor(audio).float().cpu().reshape(-1)
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
output_dir = Path(args.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
engine = Omni(
|
||||
model=args.model,
|
||||
deploy_config=args.deploy_config,
|
||||
)
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
from vllm_omni.model_executor.models.dots_tts.dots_tts_prompt import (
|
||||
build_dots_tts_prompt,
|
||||
)
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
|
||||
|
||||
prompt = build_dots_tts_prompt(
|
||||
tokenizer=tokenizer,
|
||||
text=args.text,
|
||||
)
|
||||
|
||||
print(f"Model : {args.model}")
|
||||
print(f"Text : {args.text}")
|
||||
print(f"Output dir : {output_dir}")
|
||||
print(f"Prompt len : {len(prompt['prompt_token_ids'])} tokens")
|
||||
|
||||
t_start = time.perf_counter()
|
||||
outputs = engine.generate([prompt])
|
||||
elapsed = time.perf_counter() - t_start
|
||||
|
||||
request_output = outputs[0]
|
||||
mm = request_output.outputs[0].multimodal_output
|
||||
audio = extract_audio(mm)
|
||||
|
||||
duration = audio.numel() / SAMPLE_RATE
|
||||
rtf = elapsed / duration if duration > 0 else float("inf")
|
||||
|
||||
output_path = output_dir / "output.wav"
|
||||
sf.write(str(output_path), audio.numpy(), SAMPLE_RATE, format="WAV")
|
||||
|
||||
print(f"Saved : {output_path}")
|
||||
print(f"Duration : {duration:.2f}s")
|
||||
print(f"Inference : {elapsed:.2f}s")
|
||||
print(f"RTF : {rtf:.3f}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
|
||||
main()
|
||||
@@ -1,97 +0,0 @@
|
||||
# Dynin-Omni: Online serving
|
||||
|
||||
## Installation
|
||||
|
||||
Please refer to [README.md](../../../README.md).
|
||||
|
||||
## Launch the Server
|
||||
|
||||
First, find the `transformers_modules` path:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
from transformers.utils.hub import HF_MODULES_CACHE
|
||||
print(HF_MODULES_CACHE)
|
||||
PY
|
||||
```
|
||||
|
||||
Then export it for both `PYTHONPATH` and `HF_MODULES_CACHE`:
|
||||
|
||||
```bash
|
||||
export PYTHONPATH=<transformers_modules_path>:$PYTHONPATH
|
||||
export HF_MODULES_CACHE=<transformers_modules_path>
|
||||
```
|
||||
|
||||
Run from repository root. The 3-stage pipeline is resolved automatically from
|
||||
the model type (`vllm_omni/deploy/dynin_omni.yaml`), so no config path is
|
||||
required. Pass `--deploy-config` only to override the defaults:
|
||||
|
||||
```bash
|
||||
vllm-omni serve snu-aidas/Dynin-Omni \
|
||||
--omni \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
If `vllm-omni` is not in PATH, run:
|
||||
|
||||
```bash
|
||||
PYTHONPATH="$(pwd)" python -m vllm_omni.entrypoints.cli.main serve snu-aidas/Dynin-Omni \
|
||||
--omni \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
Wait until the server logs show both `All stages initialized successfully` and
|
||||
`Application startup complete.` before sending requests.
|
||||
|
||||
## Send Requests via Python Client
|
||||
|
||||
Move to the example directory:
|
||||
|
||||
```bash
|
||||
cd examples/online_serving/dynin_omni
|
||||
```
|
||||
|
||||
### Text -> Image
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type t2i \
|
||||
--prompt "A realistic indoor living room with natural daylight."
|
||||
```
|
||||
|
||||
### Image -> Image
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type i2i \
|
||||
--image-path ../../offline_inference/dynin_omni/data/image/sofa_under_water.jpg \
|
||||
--prompt "Transform this surreal underwater setting into a realistic indoor living room while preserving the sofa layout."
|
||||
```
|
||||
|
||||
### Text -> Speech
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type t2s \
|
||||
--prompt "Hello. This is Dynin-omni."
|
||||
```
|
||||
|
||||
## CLI Arguments
|
||||
|
||||
- `--query-type` (`t2i|t2s|i2i`)
|
||||
- `--model` (default: `snu-aidas/Dynin-Omni`)
|
||||
- `--host` / `--port` (OpenAI-compatible vLLM endpoint)
|
||||
- `--prompt` (custom text)
|
||||
- `--image-path` (required for `i2i`)
|
||||
- `--modalities` (optional output modalities override)
|
||||
- `--output-dir` (default: `/tmp/dynin_online_outputs`)
|
||||
|
||||
## Notes
|
||||
|
||||
- This client currently supports only `t2i`, `t2s`, and `i2i`.
|
||||
- `t2t` is intentionally not exposed in this online example.
|
||||
- This example intentionally uses the OpenAI-compatible chat completion endpoint.
|
||||
- Task routing for non-text outputs relies on Dynin task trigger tokens (`<|t2i|>`, `<|i2i|>`, `<|t2s|>`) injected by the client.
|
||||
- Outputs are saved under `/tmp/dynin_online_outputs` by default.
|
||||
- Dynin stage-0 warmup can take a while on first startup; do not send requests before startup completes.
|
||||
- Dynin itself can execute text-returning tasks such as `t2t`, `s2t`, `i2t`, and `v2t`, but this online serving example currently runs stage-0 in `generation` mode. In that path, the generation worker does not surface the final text as `output.text`, so OpenAI chat responses for those text-output tasks may complete internally but still return empty text.
|
||||
-342
@@ -1,342 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import mimetypes
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
DEFAULT_MODEL = "snu-aidas/Dynin-Omni"
|
||||
DEFAULT_OUTPUT_DIR = "/tmp/dynin_online_outputs"
|
||||
|
||||
QUERY_CHOICES = ("t2i", "t2s", "i2i")
|
||||
DEFAULT_PROMPT_BY_QUERY = {
|
||||
"t2i": "A high quality detailed living room interior photo.",
|
||||
"t2s": "Please read this sentence naturally: Hello from Dynin-Omni online serving.",
|
||||
"i2i": "Transform this image into a realistic indoor living room while preserving layout.",
|
||||
}
|
||||
DEFAULT_MODALITIES_BY_QUERY = {
|
||||
"t2i": ["image"],
|
||||
"t2s": ["audio"],
|
||||
"i2i": ["image"],
|
||||
}
|
||||
OFFLINE_PARITY_STAGE_COUNT = 3
|
||||
OFFLINE_PARITY_STAGE_SAMPLING = {
|
||||
"max_tokens": 1,
|
||||
"temperature": 0.0,
|
||||
"top_p": 1.0,
|
||||
"detokenize": False,
|
||||
}
|
||||
|
||||
|
||||
def _infer_mime_type(path: Path) -> str:
|
||||
mime_type, _ = mimetypes.guess_type(str(path))
|
||||
return mime_type or "application/octet-stream"
|
||||
|
||||
|
||||
def _encode_file_as_data_url(path: Path) -> str:
|
||||
mime_type = _infer_mime_type(path)
|
||||
raw = path.read_bytes()
|
||||
encoded = base64.b64encode(raw).decode("utf-8")
|
||||
return f"data:{mime_type};base64,{encoded}"
|
||||
|
||||
|
||||
def _to_image_url(path_or_url: str) -> str:
|
||||
value = str(path_or_url)
|
||||
if value.startswith(("http://", "https://", "data:image/")):
|
||||
return value
|
||||
path = Path(value).expanduser().resolve()
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"Image file not found: {path}")
|
||||
return _encode_file_as_data_url(path)
|
||||
|
||||
|
||||
def _build_user_content(query_type: str, prompt: str, image_path: str | None) -> list[dict[str, Any]]:
|
||||
if query_type == "t2i":
|
||||
return [{"type": "text", "text": f"<|t2i|> {prompt}"}]
|
||||
|
||||
if query_type == "t2s":
|
||||
return [{"type": "text", "text": f"<|t2s|> {prompt}"}]
|
||||
|
||||
if query_type == "i2i":
|
||||
if not image_path:
|
||||
raise ValueError("--image-path is required for query type i2i")
|
||||
return [
|
||||
{"type": "text", "text": f"<|i2i|> {prompt}"},
|
||||
{"type": "image_url", "image_url": {"url": _to_image_url(image_path)}},
|
||||
]
|
||||
|
||||
raise ValueError(f"Unsupported query_type: {query_type}")
|
||||
|
||||
|
||||
def _collect_text_from_content(content: Any) -> list[str]:
|
||||
texts: list[str] = []
|
||||
if isinstance(content, str):
|
||||
stripped = content.strip()
|
||||
if stripped:
|
||||
texts.append(stripped)
|
||||
return texts
|
||||
|
||||
if isinstance(content, dict):
|
||||
for key in ("text", "content", "value", "output_text"):
|
||||
text_value = content.get(key)
|
||||
if isinstance(text_value, str) and text_value.strip():
|
||||
texts.append(text_value.strip())
|
||||
return texts
|
||||
|
||||
if isinstance(content, list):
|
||||
for item in content:
|
||||
texts.extend(_collect_text_from_content(item))
|
||||
return texts
|
||||
|
||||
content_text = getattr(content, "text", None)
|
||||
if isinstance(content_text, str) and content_text.strip():
|
||||
texts.append(content_text.strip())
|
||||
content_value = getattr(content, "content", None)
|
||||
if isinstance(content_value, str) and content_value.strip():
|
||||
texts.append(content_value.strip())
|
||||
output_text = getattr(content, "output_text", None)
|
||||
if isinstance(output_text, str) and output_text.strip():
|
||||
texts.append(output_text.strip())
|
||||
return texts
|
||||
|
||||
|
||||
def _extract_text_outputs(chat_completion: Any) -> list[str]:
|
||||
texts: list[str] = []
|
||||
for choice in getattr(chat_completion, "choices", []) or []:
|
||||
message = getattr(choice, "message", None)
|
||||
if message is None:
|
||||
continue
|
||||
content = getattr(message, "content", None)
|
||||
texts.extend(_collect_text_from_content(content))
|
||||
reasoning_content = getattr(message, "reasoning_content", None)
|
||||
if isinstance(reasoning_content, str) and reasoning_content.strip():
|
||||
texts.append(reasoning_content.strip())
|
||||
choice_text = getattr(choice, "text", None)
|
||||
if isinstance(choice_text, str) and choice_text.strip():
|
||||
texts.append(choice_text.strip())
|
||||
top_level_output_text = getattr(chat_completion, "output_text", None)
|
||||
if isinstance(top_level_output_text, str) and top_level_output_text.strip():
|
||||
texts.append(top_level_output_text.strip())
|
||||
return texts
|
||||
|
||||
|
||||
def _extract_image_data_urls(chat_completion: Any) -> list[str]:
|
||||
urls: list[str] = []
|
||||
for choice in getattr(chat_completion, "choices", []) or []:
|
||||
message = getattr(choice, "message", None)
|
||||
if message is None:
|
||||
continue
|
||||
content = getattr(message, "content", None)
|
||||
if not isinstance(content, list):
|
||||
continue
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if item.get("type") != "image_url":
|
||||
continue
|
||||
image_url = (item.get("image_url") or {}).get("url")
|
||||
if isinstance(image_url, str) and image_url.startswith("data:image"):
|
||||
urls.append(image_url)
|
||||
return urls
|
||||
|
||||
|
||||
def _extract_audio_payloads(chat_completion: Any) -> list[bytes]:
|
||||
payloads: list[bytes] = []
|
||||
for choice in getattr(chat_completion, "choices", []) or []:
|
||||
message = getattr(choice, "message", None)
|
||||
if message is None:
|
||||
continue
|
||||
message_audio = getattr(message, "audio", None)
|
||||
if message_audio is None:
|
||||
continue
|
||||
data_b64 = getattr(message_audio, "data", None)
|
||||
if isinstance(data_b64, str) and data_b64:
|
||||
try:
|
||||
payloads.append(base64.b64decode(data_b64))
|
||||
except Exception:
|
||||
continue
|
||||
return payloads
|
||||
|
||||
|
||||
def _decode_data_url(data_url: str) -> tuple[bytes, str]:
|
||||
header, data = data_url.split(",", 1)
|
||||
mime_type = "image/png"
|
||||
if ";" in header and ":" in header:
|
||||
mime_type = header.split(":", 1)[1].split(";", 1)[0]
|
||||
return base64.b64decode(data), mime_type
|
||||
|
||||
|
||||
def _image_extension_from_mime(mime_type: str) -> str:
|
||||
if mime_type == "image/jpeg":
|
||||
return ".jpg"
|
||||
if mime_type == "image/webp":
|
||||
return ".webp"
|
||||
if mime_type == "image/gif":
|
||||
return ".gif"
|
||||
return ".png"
|
||||
|
||||
|
||||
def _save_outputs(
|
||||
*,
|
||||
query_type: str,
|
||||
chat_completion: Any,
|
||||
output_dir: Path,
|
||||
) -> None:
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
stamp = time.strftime("%Y%m%d_%H%M%S")
|
||||
|
||||
text_outputs = _extract_text_outputs(chat_completion)
|
||||
image_data_urls = _extract_image_data_urls(chat_completion)
|
||||
audio_payloads = _extract_audio_payloads(chat_completion)
|
||||
|
||||
if text_outputs:
|
||||
text_path = output_dir / f"{query_type}_{stamp}.txt"
|
||||
text_path.write_text("\n\n".join(text_outputs) + "\n", encoding="utf-8")
|
||||
print(f"[dynin-online] text saved: {text_path}")
|
||||
print(text_outputs[0])
|
||||
|
||||
for idx, image_url in enumerate(image_data_urls):
|
||||
image_bytes, mime_type = _decode_data_url(image_url)
|
||||
ext = _image_extension_from_mime(mime_type)
|
||||
image_path = output_dir / f"{query_type}_{stamp}_{idx}{ext}"
|
||||
image_path.write_bytes(image_bytes)
|
||||
print(f"[dynin-online] image saved: {image_path}")
|
||||
|
||||
for idx, audio_bytes in enumerate(audio_payloads):
|
||||
audio_path = output_dir / f"{query_type}_{stamp}_{idx}.wav"
|
||||
audio_path.write_bytes(audio_bytes)
|
||||
print(f"[dynin-online] audio saved: {audio_path}")
|
||||
|
||||
if not text_outputs and not image_data_urls and not audio_payloads:
|
||||
print("[dynin-online] no output extracted from response")
|
||||
raw_path = output_dir / f"{query_type}_{stamp}_raw_response.json"
|
||||
try:
|
||||
if hasattr(chat_completion, "model_dump_json"):
|
||||
serialized = chat_completion.model_dump_json(indent=2)
|
||||
else:
|
||||
if hasattr(chat_completion, "model_dump"):
|
||||
raw_payload: Any = chat_completion.model_dump(mode="json")
|
||||
else:
|
||||
raw_payload = chat_completion
|
||||
try:
|
||||
serialized = json.dumps(raw_payload, ensure_ascii=False, indent=2)
|
||||
except Exception:
|
||||
serialized = json.dumps({"repr": repr(raw_payload)}, ensure_ascii=False, indent=2)
|
||||
raw_path.write_text(serialized + "\n", encoding="utf-8")
|
||||
print(f"[dynin-online] raw response saved: {raw_path}")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _build_offline_parity_sampling_params_list() -> list[dict[str, Any]]:
|
||||
return [dict(OFFLINE_PARITY_STAGE_SAMPLING) for _ in range(OFFLINE_PARITY_STAGE_COUNT)]
|
||||
|
||||
|
||||
def run_request(args: argparse.Namespace) -> None:
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url=f"http://{args.host}:{args.port}/v1",
|
||||
)
|
||||
prompt = args.prompt.strip() if args.prompt else DEFAULT_PROMPT_BY_QUERY[args.query_type]
|
||||
user_content = _build_user_content(
|
||||
query_type=args.query_type,
|
||||
prompt=prompt,
|
||||
image_path=args.image_path,
|
||||
)
|
||||
if args.modalities:
|
||||
modalities = [item.strip() for item in args.modalities.split(",") if item.strip()]
|
||||
else:
|
||||
modalities = DEFAULT_MODALITIES_BY_QUERY[args.query_type]
|
||||
|
||||
extra_body = {
|
||||
"sampling_params_list": _build_offline_parity_sampling_params_list(),
|
||||
}
|
||||
chat_completion = client.chat.completions.create(
|
||||
model=args.model,
|
||||
messages=[{"role": "user", "content": user_content}],
|
||||
modalities=modalities,
|
||||
extra_body=extra_body,
|
||||
)
|
||||
_save_outputs(
|
||||
query_type=args.query_type,
|
||||
chat_completion=chat_completion,
|
||||
output_dir=Path(args.output_dir).expanduser(),
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Dynin-Omni online chat completion client")
|
||||
parser.add_argument(
|
||||
"--query-type",
|
||||
"-q",
|
||||
type=str,
|
||||
default="t2i",
|
||||
choices=QUERY_CHOICES,
|
||||
help="Dynin query type",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
"-m",
|
||||
type=str,
|
||||
default=DEFAULT_MODEL,
|
||||
help="Model name/path",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--host",
|
||||
type=str,
|
||||
default="localhost",
|
||||
help="Host/IP of the vLLM Omni API server",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--port",
|
||||
type=int,
|
||||
default=8091,
|
||||
help="Port of the vLLM Omni API server",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt",
|
||||
"-p",
|
||||
type=str,
|
||||
default="",
|
||||
help="Custom prompt text",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--image-path",
|
||||
"-i",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Image path/URL for i2i",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--modalities",
|
||||
type=str,
|
||||
default="",
|
||||
help="Comma-separated output modalities override (e.g., text,image,audio)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
"-o",
|
||||
type=str,
|
||||
default=DEFAULT_OUTPUT_DIR,
|
||||
help="Directory to save outputs",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
os.environ.setdefault("HF_HUB_DISABLE_XET", "1")
|
||||
run_request(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -27,7 +27,6 @@ For the full list of supported architectures across all modalities, see
|
||||
| OmniVoice | `k2-fsa/OmniVoice` | ✓ | — | — | — |
|
||||
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | ✓ (Base) | ✓ (PCM + WebSocket) | ✓ (presets + `/v1/audio/voices` upload) | ✓ (standard + FastRTC) |
|
||||
| VoxCPM2 | `openbmb/VoxCPM2` | ✓ | ✓ (AudioWorklet via gradio) | — | ✓ |
|
||||
| dots.tts | `dots-studio/dots.tts-soar` | — (text-only) | ✓ (PCM stream) | — (default placeholder only) | — |
|
||||
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✓ (gated upstream) | ✓ | ✓ (presets) | ✓ |
|
||||
|
||||
CosyVoice3 is intentionally absent: no online example exists for it yet. See its [offline section](../../offline_inference/text_to_speech/README.md#cosyvoice3) instead.
|
||||
@@ -143,42 +142,6 @@ python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
- Concurrent requests at `max_num_seqs: 4` are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below `max_num_seqs` and treat preemption as out of scope until the platform fix lands.
|
||||
- Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.
|
||||
|
||||
## dots.tts
|
||||
|
||||
Single-stage, text-only TTS at 48 kHz. The current integration supports
|
||||
no-reference synthesis; voice cloning, reference audio, named voices, and
|
||||
precomputed speaker embeddings are not supported yet.
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
vllm serve dots-studio/dots.tts-soar --omni --trust-remote-code --port 8091
|
||||
```
|
||||
|
||||
### Sending requests
|
||||
|
||||
```bash
|
||||
# Non-streaming WAV
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is a dots TTS online-serving test.",
|
||||
"voice": "default",
|
||||
"response_format": "wav"
|
||||
}' --output output.wav
|
||||
|
||||
# Streaming 48 kHz mono PCM
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is a streaming dots TTS test.",
|
||||
"voice": "default",
|
||||
"stream": true,
|
||||
"stream_format": "audio",
|
||||
"response_format": "pcm"
|
||||
}' --no-buffer | play -t raw -r 48000 -e signed -b 16 -c 1 -
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GLM-TTS
|
||||
|
||||
Reference in new issue
Block a user