anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Docs] Consolidate per-model TTS examples into a single hub (#3234)

Browse Source 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
Yueqian Lin authored and GitHub committed 2026-05-05 08:41:29 +12:00
1 parent e1b30061b6
commit 703e31fc47
58 files changed
+1694 -2675
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/.nav.yml
+2 -6
View File
@@ -16,8 +16,6 @@ nav:
- examples/README.md
- Offline Inference:
- BAGEL-7B-MoT: user_guide/examples/offline_inference/bagel.md
- CosyVoice3: user_guide/examples/offline_inference/cosyvoice3.md
- Fish Speech S2 Pro: user_guide/examples/offline_inference/fish_speech.md
- GLM-Image Multistage End-to-End Inference: user_guide/examples/offline_inference/glm_image.md
- Helios Video Generation: user_guide/examples/offline_inference/helios.md
- HunyuanImage-3.0 Image-to-Text Inference: user_guide/examples/offline_inference/hunyuan_image3.md
@@ -28,24 +26,22 @@ nav:
- MiMo-Audio Offline Inference: user_guide/examples/offline_inference/mimo_audio.md
- Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md
- Qwen3-Omni: user_guide/examples/offline_inference/qwen3_omni.md
- Qwen3-TTS: user_guide/examples/offline_inference/qwen3_tts.md
- Text-To-Audio: user_guide/examples/offline_inference/text_to_audio.md
- Text-To-Image: user_guide/examples/offline_inference/text_to_image.md
- Text-To-Speech: user_guide/examples/offline_inference/text_to_speech.md
- Text-To-Video: user_guide/examples/offline_inference/text_to_video.md
- Voxtral TTS Offline Inference: user_guide/examples/offline_inference/voxtral_tts.md
- X-To-Video-Audio: user_guide/examples/offline_inference/x_to_video_audio.md
- Online Serving:
- BAGEL-7B-MoT: user_guide/examples/online_serving/bagel.md
- vLLM-Omni Helm Chart: user_guide/examples/online_serving/chart-helm.md
- Diffusers Backend Adapter Example: user_guide/examples/online_serving/diffusers_pipeline_adapter.md
- Fish Speech S2 Pro: user_guide/examples/online_serving/fish_speech.md
- GLM-Image Online Serving: user_guide/examples/online_serving/glm_image.md
- Image-To-Image: user_guide/examples/online_serving/image_to_image.md
- Image-To-Video: user_guide/examples/online_serving/image_to_video.md
- Online serving Example of vLLM-Omni for MiMo-Audio: user_guide/examples/online_serving/mimo_audio.md
- Qwen2.5-Omni: user_guide/examples/online_serving/qwen2_5_omni.md
- Qwen3-Omni: user_guide/examples/online_serving/qwen3_omni.md
- Qwen3-TTS: user_guide/examples/online_serving/qwen3_tts.md
- Text-To-Speech: user_guide/examples/online_serving/text_to_speech.md
- Text-To-Image: user_guide/examples/online_serving/text_to_image.md
- Text-To-Video: user_guide/examples/online_serving/text_to_video.md
- General:
docs/serving/speech_api.md
+1 -1
View File
@@ -535,7 +535,7 @@ The bundled `qwen3_tts.yaml` uses `max_num_seqs: 1` (single request) on both sta
|-------|-------------|
| `fishaudio/s2-pro` | 4B dual-AR TTS with DAC codec (44.1 kHz). Supports text-to-speech and voice cloning. |
Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` needed). The `voice` field should be set to `"default"`. See the [Fish Speech online serving example](../user_guide/examples/online_serving/fish_speech.md) for details.
Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` needed). The `voice` field should be set to `"default"`. See the [Fish Speech section of the online TTS hub](../user_guide/examples/online_serving/text_to_speech.md#fish-speech-s2-pro) for details.
### Voxtral TTS
docs/user_guide/examples/offline_inference/cosyvoice3.md
-81
View File
@@ -1,81 +0,0 @@
# CosyVoice3
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/cosyvoice3>.
## Setup
Install dependencies:
```
uv pip install -e .
```
> **Note:** This includes required libraries such as `soundfile`,
> `onnxruntime`, `x-transformers`, and `einops` via
> `requirements/common.txt` and platform-specific requirements files.
Download the model snapshot:
```
from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
```
Add `config.json` in `pretrained_models/Fun-CosyVoice3-0.5B/`:
```json
{
"model_type": "cosyvoice3",
"architectures": [
"CosyVoice3Model"
]
}
```
> **Why `config.json` is required:**
> `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers a class mapping.
> The loader still needs `model_type: "cosyvoice3"` from `config.json` to select that class.
> If no `config.json` is present, model type cannot be inferred automatically.
> If your downloaded checkpoint already includes a valid `config.json` with
> `model_type: "cosyvoice3"`, this manual step can be skipped.
Run the offline verification script:
```
python examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py \
--model pretrained_models/Fun-CosyVoice3-0.5B \
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN
```
## Implementation Overview
CosyVoice3 runs as a **2-stage Omni pipeline**.
- **Stage 0** (`talker`) converts text + prompt audio to speech tokens.
- **Stage 1** (`code2wav`) converts speech tokens + prompt features to waveform via flow matching and HiFiGAN.
Key components live in `vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py`.
- `CosyVoice3MultiModalProcessor` builds the multimodal inputs:
- Tokenizes `prompt` and `prompt_text`.
- Extracts speech tokens and mel features from the prompt audio.
- Extracts a speaker embedding.
- `CosyVoice3Model` implements both stages:
- Stage 0 uses `CosyVoice3LM` and outputs speech tokens + conditioning features.
- Stage 1 runs the flow model (DiT-based CFM) and HiFiGAN to synthesize waveform.
Pipeline topology lives in `vllm_omni/model_executor/models/cosyvoice3/pipeline.py`;
runtime tunables (batch size, memory limits, sampling) live in
`vllm_omni/deploy/cosyvoice3.yaml`. The deploy config auto-loads by
HF `model_type` and defaults to `async_chunk: true` (shared-memory
streaming). Pass `--no-async-chunk` on `vllm serve` to switch to the
legacy sync path where stage 1 runs `text2flow` over the full
speech-token sequence.
- Stage 0 emits latent speech tokens.
- Stage 1 consumes them via `sync_process_input_func` (sync mode) or the
shared-memory connector (async-chunk mode) and outputs audio.
## Example materials
??? abstract "verify_e2e_cosyvoice.py"
``````py
--8<-- "examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py"
``````
docs/user_guide/examples/offline_inference/fish_speech.md
-85
View File
@@ -1,85 +0,0 @@
# Fish Speech S2 Pro
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/fish_speech>.
This directory contains an offline demo for running Fish Speech S2 Pro (`fishaudio/s2-pro`) with vLLM Omni. It supports text-only synthesis and voice cloning with reference audio.
## Model Overview
[Fish Speech S2 Pro](https://huggingface.co/fishaudio/s2-pro) is a 4B dual-AR text-to-speech model by FishAudio. It uses:
- **Slow AR**: Generates semantic tokens autoregressively (Qwen3-based backbone)
- **Fast AR**: Predicts residual codebook tokens from semantic tokens
- **DAC Decoder**: Converts codec codes to 44.1 kHz audio waveform
The model produces high-quality speech with voice cloning capabilities.
## Prerequisites
Install the `fish-speech` package for the DAC codec:
```bash
pip install fish-speech
```
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
## Quick Start
### Text-Only Synthesis
```bash
python end2end.py --text "Hello, this is a test of the Fish Speech text to speech system."
```
Generated audio files are saved to `output_audio/` by default.
### Voice Cloning
Provide a reference audio file and its transcript to clone a voice:
```bash
python end2end.py \
--text "Hello, this is a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
## Streaming Mode
Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config):
```bash
python end2end.py --text "Hello, this is a test." --streaming
```
Each DAC decoder chunk is logged as it arrives. The final WAV file is written once generation completes.
## Configuration
| Argument | Default | Description |
|----------|---------|-------------|
| `--text` | `"Hello, this is a test..."` | Text to synthesize |
| `--ref-audio` | None | Path to reference audio for voice cloning |
| `--ref-text` | None | Transcript of the reference audio |
| `--model` | `fishaudio/s2-pro` | HuggingFace model path |
| `--stage-configs-path` | `fish_speech_s2_pro.yaml` | Path to stage configs YAML |
| `--output-dir` | `output_audio` | Output directory for WAV files |
| `--streaming` | False | Enable streaming via AsyncOmni |
| `--stage-init-timeout` | 600 | Stage initialization timeout (seconds) |
## Notes
- Output audio is 44.1 kHz mono WAV.
- The DAC codec weights (`codec.pth`) are loaded lazily from the model directory.
- Voice cloning encodes the reference audio using the DAC codec to extract semantic codes, then prepends them as a system message.
## Example materials
??? abstract "end2end.py"
``````py
--8<-- "examples/offline_inference/fish_speech/end2end.py"
``````
docs/user_guide/examples/offline_inference/qwen3_tts.md
-172
View File
@@ -1,172 +0,0 @@
# Qwen3-TTS
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen3_tts>.
This directory contains an offline demo for running Qwen3 TTS models with vLLM Omni. It builds task-specific inputs and generates WAV files locally.
## Model Overview
Qwen3 TTS provides multiple task variants for speech generation:
- **CustomVoice**: Generate speech with a known speaker identity (speaker ID) and optional instruction.
- **VoiceDesign**: Generate speech from text plus a descriptive instruction that designs a new voice.
- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. The `ref_audio` field accepts a local file path, HTTP URL, or base64 data URL.
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
### ROCm Dependencies
You will need to install the dependency `onnxruntime-rocm`.
```
pip uninstall onnxruntime # should be removed before we can install onnxruntime-rocm
pip install onnxruntime-rocm
```
## Quick Start
Run a single sample for a task:
```
python end2end.py --query-type CustomVoice
```
Generated audio files are saved to `output_audio/` by default.
## Task Usage
### CustomVoice
Single sample:
```
python end2end.py --query-type CustomVoice
```
Batch sample (multiple prompts in one run):
```
python end2end.py --query-type CustomVoice --use-batch-sample
```
### VoiceDesign
Single sample:
```
python end2end.py --query-type VoiceDesign
```
Batch sample:
```
python end2end.py --query-type VoiceDesign --use-batch-sample
```
### Base (Voice Clone)
Single sample:
```
python end2end.py --query-type Base
```
Batch sample:
```
python end2end.py --query-type Base --use-batch-sample
```
Mode selection for Base:
- `--mode-tag icl` (default): standard mode
- `--mode-tag xvec_only`: enable `x_vector_only_mode` in the request
Examples:
```
python end2end.py --query-type Base --mode-tag icl
```
## Voice and Language Control
### Supported Voices (CustomVoice)
Predefined speaker voices are set via the `speaker` (or `voice_type`) field in `additional_information`. Available speakers depend on the loaded checkpoint; check `talker_config.spk_id` in the model config for the full list. Common voices include `vivian`, `ryan`, `aiden`, `ethan`, `serena` (case-insensitive).
Pass the speaker name in your request:
```python
additional_information = {
"text": ["你好，我是通义千问"],
"task_type": ["CustomVoice"],
"speaker": ["Vivian"],
"language": ["Chinese"],
}
```
### Supported Languages
The `language` field controls the codec-level language tag. Use `"Auto"` (default) for automatic detection.
Supported values: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`.
```python
additional_information = {
"text": ["Hello, nice to meet you."],
"task_type": ["CustomVoice"],
"speaker": ["Aiden"],
"language": ["English"],
}
```
### VoiceDesign and Base
- **VoiceDesign**: Use `instruct` for natural-language voice description; no `speaker` needed.
- **Base**: Use `ref_audio` and `ref_text` for voice cloning; `language` is optional.
## Streaming Mode
Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config):
```bash
python end2end.py --query-type CustomVoice --streaming --output-dir /tmp/out_stream
```
Each Code2Wav chunk is logged as it arrives (default 25 frames; configurable via `codec_chunk_frames`
in the stage config). The initial chunk size is dynamically selected based on server load for reduced
TTFA, and can be overridden per-request via the `initial_codec_chunk_frames` API field. The final WAV file is written once generation
completes. This demonstrates that audio data is available progressively rather than only at the end.
> **Note:** Streaming uses `AsyncOmni` internally. The non-streaming path (`Omni`) is unchanged.
## Batched Decoding
The Code2Wav stage (stage 1) supports batched decoding, where multiple requests are decoded in a single forward pass through the SpeechTokenizer. To use it, set `max_num_seqs > 1` on both stages via `--stage-overrides` and pass multiple prompts via `--txt-prompts` with a matching `--batch-size`.
```
python end2end.py --query-type CustomVoice \
--txt-prompts benchmark_prompts.txt \
--batch-size 4 \
--stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}'
```
**Important:** `--batch-size` must match a CUDA graph capture size (1, 2, 4, 8, 16...) because the Talker's code predictor KV cache is sized to `max_num_seqs`, and CUDA graphs pad the batch to the next capture size. Both stages need `max_num_seqs >= batch_size` in the stage config for batching to take effect. If only stage 1 has a higher `max_num_seqs`, it won't help — stage 1 can only batch chunks from requests that are in-flight simultaneously, which requires stage 0 to also process multiple requests concurrently.
## Notes
- The script uses the model paths embedded in `end2end.py`. Update them if your local cache path differs.
- Use `--output-dir` to change the output folder.
## Example materials
??? abstract "benchmark_prompts.txt"
``````txt
--8<-- "examples/offline_inference/qwen3_tts/benchmark_prompts.txt"
``````
??? abstract "end2end.py"
``````py
--8<-- "examples/offline_inference/qwen3_tts/end2end.py"
``````
docs/user_guide/examples/offline_inference/text_to_speech.md
+395
View File
@@ -0,0 +1,395 @@
# Text-To-Speech
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech>.
vLLM-Omni supports several autoregressive TTS models. They share a common
CLI shape (`--text`, `--ref-audio`, `--ref-text`, `--output-dir`) and live
together in this hub. Each model has its own subdirectory containing a
single `end2end.py` script; this README is the single doc entry point.
For online serving, see `examples/online_serving/<model>/`. For the full
list of supported architectures across all modalities, see
[Supported Models](https://github.com/vllm-project/vllm-omni/tree/main/docs/models/supported_models.md).
## Supported Models
| Model | HuggingFace repo | Stages | Voice cloning | Streaming | Special modes | Sample rate |
|---|---|---|---|---|---|---|
| VoxCPM2 | `openbmb/VoxCPM2` | single (native AR) | ✓ | — | continuation (`--ref-audio` + `--ref-text`) | 48 kHz |
| VoxCPM | local model dir | split | ✓ | ✓ (`voxcpm_async_chunk.yaml`) | — | 24 kHz |
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ (`async_chunk: true` default) | — | 22.05 kHz |
| Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ (`--streaming`) | — | 44.1 kHz |
| OmniVoice | `k2-fsa/OmniVoice` | 2 (gen + dec) | ✓ | — | voice design (`--instruct`), language (`--lang`) | 24 kHz |
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | 2 (talker + code2wav) | ✓ (Base) | ✓ | 3 task variants (`--query-type`) | 24 kHz |
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | varies | ✓ | ✓ | voice presets (`--voice`) | 24 kHz |
## Common Quick Start
Most models share this invocation shape:
```bash
python examples/offline_inference/text_to_speech/<model>/end2end.py \
--text "Hello, this is a test." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
`--ref-audio` and `--ref-text` are optional (text-only synthesis works without them) and must be provided together for voice cloning. The exotic scripts — Qwen3-TTS, Voxtral TTS, CosyVoice3 — accept additional model-specific flags documented in their per-model section below.
---
## VoxCPM2
Single-stage native AR TTS at 48 kHz. Pipeline: `feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE`.
### Prerequisites
```bash
pip install voxcpm
# or, for a local source checkout:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
--model openbmb/VoxCPM2 \
--text "Hello, this is a VoxCPM2 demo."
```
### Voice cloning
Pass a reference audio for isolated cloning, or both `--ref-audio` + `--ref-text` for prompt continuation:
```bash
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
--text "Hello, this is a voice clone demo." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
### Notes
- Output: 48 kHz mono WAV.
- Stage config: `vllm_omni/model_executor/stage_configs/voxcpm2.yaml` (default).
---
## VoxCPM
Split-stage TTS. The hub example covers single TTS, single voice cloning, and streaming. Use `benchmarks/voxcpm/` for warmup, batch JSONL prompts, profiler injection, and offline TTFP / RTF measurement.
### Prerequisites
```bash
pip install voxcpm soundfile
# or use a local source tree:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src
```
If the native VoxCPM `config.json` does not contain HF metadata such as `model_type`, prepare a persistent HF-compatible config directory and point the stage configs to it via `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH`:
```bash
export VOXCPM_MODEL=/path/to/voxcpm-model
export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config
mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"
cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json"
cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true
python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)'
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--text "This is a split-stage VoxCPM synthesis example running on vLLM Omni."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--text "This sentence is synthesized with a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript spoken in reference.wav."
```
### Streaming
Pass the async-chunk stage config:
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
--text "This is a split-stage VoxCPM streaming example running on vLLM Omni."
```
### Notes
- `voxcpm.yaml` is the default non-streaming stage config; `voxcpm_async_chunk.yaml` enables streaming.
- Streaming is currently single-request oriented.
- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades quality.
- For online serving, see [`examples/online_serving/voxcpm`](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/voxcpm/README.md). For benchmark reporting, see [`benchmarks/voxcpm`](https://github.com/vllm-project/vllm-omni/tree/main/benchmarks/voxcpm/README.md).
---
## CosyVoice3
2-stage TTS pipeline (`talker` + `code2wav`) at 22.05 kHz.
### Prerequisites
```bash
uv pip install -e .
# Includes soundfile, onnxruntime, x-transformers, einops via requirements.
```
Download the model snapshot:
```python
from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512',
local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
```
If your downloaded checkpoint lacks `config.json`, add it:
```json
{
"model_type": "cosyvoice3",
"architectures": ["CosyVoice3Model"]
}
```
This is required because `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers the class mapping; the loader still reads `model_type` from `config.json` to select the class.
### Quick start
```bash
python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
--model pretrained_models/Fun-CosyVoice3-0.5B \
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN
```
### Voice cloning
Pass a reference audio. Note that CosyVoice3's `--prompt-text` is a system-style prompt for the GPT stage, not a reference transcript:
```bash
python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
--model pretrained_models/Fun-CosyVoice3-0.5B \
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN \
--ref-audio prompt.wav \
--prompt-text "You are a helpful assistant.<|endofprompt|>Testing my voices. Why should I not?"
```
### Notes
- Stage 0 (`talker`) emits speech tokens; stage 1 (`code2wav`) runs flow matching + HiFiGAN to synthesize waveform.
- Deploy config auto-loads from `vllm_omni/deploy/cosyvoice3.yaml` based on HF `model_type`. Pass `--deploy-config <path>` to override.
- `async_chunk: true` is the default; pass `--no-async-chunk` to switch to the legacy synchronous path.
---
## Fish Speech S2 Pro
4B dual-AR text-to-speech model from FishAudio with the DAC codec at 44.1 kHz.
### Prerequisites
```bash
pip install fish-speech
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a test of the Fish Speech text to speech system."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
### Streaming
```bash
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a streaming test." \
--streaming
```
Streaming requires `async_chunk: true` in the stage config.
### Notes
- Output: 44.1 kHz mono WAV.
- DAC codec weights (`codec.pth`) are loaded lazily from the model directory.
---
## OmniVoice
Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design).
### Prerequisites
```bash
huggingface-cli download k2-fsa/OmniVoice
```
Voice cloning requires `transformers>=5.3.0`. Auto and design modes work with `transformers>=4.57.0`.
### Quick start (auto voice)
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "Hello, this is a test."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "Hello, this is a test." \
--ref-audio ref.wav \
--ref-text "This is the reference transcription."
```
### Voice design
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "Hello, this is a test." \
--instruct "female, low pitch, british accent"
```
### Language hint
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "你好，这是一个测试。" \
--lang zh
```
### Notes
- Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking.
- Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz.
---
## Qwen3-TTS
3-task-variant TTS with 24 kHz output. Has its own argparse surface (this script does not follow the common `--text` / `--ref-audio` shape).
### Prerequisites
For ROCm builds, replace `onnxruntime` with `onnxruntime-rocm`:
```bash
pip uninstall onnxruntime
pip install onnxruntime-rocm
```
### Task variants
- `CustomVoice`: predefined speaker (speaker ID) with optional style instruction.
- `VoiceDesign`: text + descriptive instruction designs a new voice.
- `Base`: voice cloning from reference audio + transcript.
```bash
# Single sample
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type VoiceDesign
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base
# Base variant has an additional mode flag:
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag icl # default
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag xvec_only # x_vector_only_mode
# Batch (multiple prompts in one run)
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice --use-batch-sample
```
### Streaming
```bash
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
--query-type CustomVoice \
--streaming \
--output-dir /tmp/out_stream
```
Streaming requires `async_chunk: true` in the stage config.
### Batched decoding
The Code2Wav stage supports batched decoding through the SpeechTokenizer. Configure both stages with `max_num_seqs > 1` via `--stage-overrides` and pass multiple prompts via `--txt-prompts`:
```bash
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
--query-type CustomVoice \
--txt-prompts examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt \
--batch-size 4 \
--stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}'
```
`--batch-size` must match a CUDA-graph capture size (1, 2, 4, 8, 16…).
### Notes
- Run `--help` for the full argument surface.
- See `qwen3_tts/end2end.py` for the prompt-length-estimation logic the Talker uses.
---
## Voxtral TTS
Voxtral-4B-TTS (Mistral). Has its own argparse surface; uses voice presets and the `mistral_common` `SpeechRequest` protocol.
### Prerequisites
Latest `mistral_common` with `SpeechRequest` support:
```bash
pip install -e /path/to/mistral-common # or upgrade from PyPI when available
```
### Quick start (voice preset)
```bash
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--write-audio --voice cheerful_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
```
### Voice cloning (capability gated upstream)
```bash
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--write-audio \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "This is a test message." \
--ref-audio path/to/reference_audio.wav
```
### Streaming + concurrency
```bash
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "..."
```
Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`).
### Notes
- `--num-prompts N` replicates the prompt for performance measurement.
- `--concurrency M` requires `--streaming` and must evenly divide `--num-prompts`.
- Run `--help` for the full argument surface.
## Example materials
??? abstract "cosyvoice3/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/cosyvoice3/end2end.py"
``````
??? abstract "fish_speech/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/fish_speech/end2end.py"
``````
??? abstract "omnivoice/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/omnivoice/end2end.py"
``````
??? abstract "qwen3_tts/benchmark_prompts.txt"
``````txt
--8<-- "examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt"
``````
??? abstract "qwen3_tts/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/qwen3_tts/end2end.py"
``````
??? abstract "voxcpm/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/voxcpm/end2end.py"
``````
??? abstract "voxcpm2/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/voxcpm2/end2end.py"
``````
??? abstract "voxtral_tts/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/voxtral_tts/end2end.py"
``````
docs/user_guide/examples/offline_inference/voxtral_tts.md
-69
View File
@@ -1,69 +0,0 @@
# Voxtral TTS Offline Inference
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/voxtral_tts>.
`end2end.py` runs Voxtral TTS end-to-end offline inference using vLLM. It supports both blocking (`Omni`) and streaming (`AsyncOmni`) generation, batched prompts with configurable concurrency, and voice selection via preset name or reference audio file.
When `mistral_common` has `SpeechRequest` support, prompt token IDs are built via `encode_speech_request`. Otherwise, the script falls back to manual token construction.
## Usage Examples
```bash
# Basic single-prompt with cheerful_female voice preset
python3 examples/offline_inference/voxtral_tts/end2end.py \
--write-audio --voice cheerful_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# 32 replicate prompts with cheerful_female voice preset
python3 examples/offline_inference/voxtral_tts/end2end.py \
--num-prompts 32 --write-audio --voice cheerful_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# Streaming with neutral_female voice preset
python3 examples/offline_inference/voxtral_tts/end2end.py \
--streaming --write-audio --voice neutral_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# 32 prompts, 8 concurrent requests per wave, streaming with neutral_female voice
python3 examples/offline_inference/voxtral_tts/end2end.py \
--num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# Short debug prompt with reference audio
# Note: Reference audio capability is not yet released.
python3 examples/offline_inference/voxtral_tts/end2end.py \
--write-audio \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "This is a test message." \
--audio-path path/to/reference_audio.wav
```
## Arguments
| Argument | Description |
|---|---|
| `--model PATH` | HuggingFace repo ID or local directory path (default: `mistralai/Voxtral-4B-TTS-2603`) |
| `--text TEXT` | Text to synthesize (default: `"This is a test message."`) |
| `--audio-path PATH` | Path to reference audio file for voice cloning |
| `--output-dir DIR` | Directory to write output WAV files (default: `output_audio`) |
| `--deploy-config PATH` | Override the deploy config path. If unset, auto-loads `vllm_omni/deploy/voxtral_tts.yaml` from the HF `model_type`. |
| `--num-prompts N` | Number of replicate prompts to run for measuring performance (default: 1) |
| `--streaming` | Use streaming generation via `AsyncOmni` (default: blocking `Omni`) |
| `--concurrency N` | Max concurrent requests per wave (must be used with `--streaming`, must evenly divide `--num-prompts`) |
| `--voice NAME` | Voice preset to use instead of reference audio file (e.g., casual_female, casual_male, cheerful_female, neutral_female, neutral_male) |
| `--write-audio` | Write generated audio to WAV files |
| `--profiling-mode` | Enable profiling mode (reduces max tokens to 50) |
| `--log-stats` | Enable detailed statistics logging |
## Example materials
??? abstract "end2end.py"
``````py
--8<-- "examples/offline_inference/voxtral_tts/end2end.py"
``````
docs/user_guide/examples/online_serving/fish_speech.md
-199
View File
@@ -1,199 +0,0 @@
# Fish Speech S2 Pro
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/fish_speech>.
## Model
| Model | Description |
|-------|-------------|
| `fishaudio/s2-pro` | Fish Speech S2 Pro -- 4B dual-AR TTS model with DAC codec (44.1 kHz) |
## Gradio Demo
!!! note "Gradio is an optional dependency"
The Gradio demo requires the `[demo]` extras. Install them first:
```bash
pip install 'vllm-omni[demo]'
```
Or, if installing from source: `pip install -e '.[demo]'`
An interactive Gradio demo is available with text-to-speech synthesis, voice cloning, and streaming support.
```bash
# Option 1: Launch server + Gradio together
./run_gradio_demo.sh
# Option 2: If server is already running
python gradio_demo.py --api-base http://localhost:8091
```
Then open http://localhost:7860 in your browser.
Features:
- Text-to-speech synthesis
- Voice cloning from uploaded audio or URL
- Streaming mode (progressive PCM playback)
## Launch the Server
```bash
vllm serve fishaudio/s2-pro --omni --port 8091
```
The deploy config is auto-loaded from `vllm_omni/deploy/fish_qwen3_omni.yaml`.
Or use the convenience script:
```bash
./run_server.sh
```
## Send TTS Request
### Using curl
```bash
# Basic TTS
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav"
}' --output output.wav
```
### Voice Cloning
Provide a reference audio (URL or base64 data URL) and its transcript:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, this is a cloned voice.",
"voice": "default",
"ref_audio": "https://example.com/reference.wav",
"ref_text": "Transcript of the reference audio."
}' --output cloned.wav
```
### Using Python
```python
import httpx
# Basic TTS
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav",
},
timeout=300.0,
)
with open("output.wav", "wb") as f:
f.write(response.content)
```
### Using the CLI Client
```bash
cd examples/online_serving/fish_speech
# Basic TTS
python speech_client.py --text "Hello, how are you?"
# Voice cloning
python speech_client.py \
--text "Hello, this is a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
# Streaming PCM output
python speech_client.py --text "Hello world" --stream --output output.pcm
```
The CLI client supports:
- `--api-base`: API base URL (default: `http://localhost:8091`)
- `--model` (or `-m`): Model name (default: `fishaudio/s2-pro`)
- `--text`: Text to synthesize (required)
- `--ref-audio`: Reference audio for voice cloning (local path or URL)
- `--ref-text`: Transcript of the reference audio
- `--stream`: Enable streaming (PCM output)
- `--response-format`: Audio format: wav, mp3, flac, pcm, aac, opus (default: wav)
- `--output` (or `-o`): Output file path
## Streaming
Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "default",
"stream": true,
"response_format": "pcm"
}' --no-buffer | play -t raw -r 44100 -e signed -b 16 -c 1 -
```
**Note:** Fish Speech outputs at 44.1 kHz (unlike Qwen3-TTS which outputs at 24 kHz).
## API Parameters
Fish Speech uses the same `/v1/audio/speech` endpoint as Qwen3-TTS. See the [Speech API reference](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/speech_api/) for full parameter documentation.
Key parameters for Fish Speech:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `input` | string | **required** | Text to synthesize |
| `voice` | string | "default" | Voice name (use "default" for Fish Speech) |
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus |
| `ref_audio` | string | null | Reference audio URL or base64 data URL for voice cloning |
| `ref_text` | string | null | Transcript of reference audio (required for voice cloning) |
| `max_new_tokens` | int | 4096 | Maximum tokens to generate |
| `stream` | bool | false | Stream raw PCM chunks |
## Prerequisites
Install the `fish-speech` package for the DAC codec:
```bash
pip install fish-speech
```
## Troubleshooting
1. **No audio output**: Make sure the `fish-speech` package is installed for the DAC decoder
2. **Connection refused**: Ensure the server is running on the correct port
3. **Flashinfer version mismatch**: Set `FLASHINFER_DISABLE_VERSION_CHECK=1` if you see version warnings
4. **Out of memory**: Reduce `--gpu-memory-utilization` or use a GPU with more VRAM
## Example materials
??? abstract "gradio_demo.py"
``````py
--8<-- "examples/online_serving/fish_speech/gradio_demo.py"
``````
??? abstract "run_gradio_demo.sh"
``````sh
--8<-- "examples/online_serving/fish_speech/run_gradio_demo.sh"
``````
??? abstract "run_server.sh"
``````sh
--8<-- "examples/online_serving/fish_speech/run_server.sh"
``````
??? abstract "speech_client.py"
``````py
--8<-- "examples/online_serving/fish_speech/speech_client.py"
``````
docs/user_guide/examples/online_serving/qwen3_tts.md
-451
View File
@@ -1,451 +0,0 @@
# Qwen3-TTS
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_tts>.
## 🛠️ Installation
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
## Supported Models
| Model | Task Type | Description |
| -------------------------------------- | ----------- | ----------------------------------------------------- |
| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | CustomVoice | Predefined speaker voices with optional style control |
| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description |
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio |
| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant |
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning |
## Gradio Demo
!!! note "Gradio is an optional dependency"
The Gradio demo requires the `[demo]` extras. Install them first:
```bash
pip install 'vllm-omni[demo]'
```
Or, if installing from source: `pip install -e '.[demo]'`
Two interactive Gradio demos are available, both supporting all 3 task types:
| Demo | File | Transport | Streaming Quality |
| -------- | ------------------------ | ------------ | -------------------------------------------------- |
| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks |
| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) |
```bash
# Option 1: Launch server + Standard Gradio together
./run_gradio_demo.sh # CustomVoice (default)
./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign
./run_gradio_demo.sh --task-type Base # Voice cloning
# Option 2: If server is already running
python gradio_demo.py --api-base http://localhost:8000
# Option 3: FastRTC demo (gapless streaming)
pip install fastrtc
python gradio_fastrtc_demo.py --api-base http://localhost:8000
```
Then open http://localhost:7860 in your browser.
## Run examples (Qwen3-TTS)
### Launch the Server
```bash
# CustomVoice model (predefined speakers)
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--deploy-config vllm_omni/deploy/qwen3_tts.yaml \
--omni \
--port 8091 \
--trust-remote-code \
--enforce-eager
# VoiceDesign model
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
--deploy-config vllm_omni/deploy/qwen3_tts.yaml \
--omni \
--port 8091 \
--trust-remote-code \
--enforce-eager
# Base model (voice cloning)
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--deploy-config vllm_omni/deploy/qwen3_tts.yaml \
--omni \
--port 8091 \
--trust-remote-code \
--enforce-eager
```
If you have custom stage configs file, launch the server with command below
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--stage-configs-path /path/to/stage_configs_file \
--omni \
--port 8091 \
--trust-remote-code \
--enforce-eager
```
Alternatively, use the convenience script:
```bash
./run_server.sh # Default: CustomVoice model
./run_server.sh CustomVoice # CustomVoice model
./run_server.sh VoiceDesign # VoiceDesign model
./run_server.sh Base # Base (voice clone) model
```
### Send TTS Request
Get into the example folder
```bash
cd examples/online_serving/qwen3_tts
```
#### Send request via python
```bash
# CustomVoice: Use predefined speaker
python openai_speech_client.py \
--text "你好，我是通义千问" \
--speaker vivian \
--language Chinese
# CustomVoice with style instruction
python openai_speech_client.py \
--text "今天天气真好" \
--speaker ryan \
--instructions "用开心的语气说"
# VoiceDesign: Describe the voice style
python openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
--task-type VoiceDesign \
--text "哥哥，你回来啦" \
--instructions "体现撒娇稚嫩的萝莉女声，音调偏高"
# Base: Voice cloning
python openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--task-type Base \
--text "Hello, this is a cloned voice" \
--ref-audio /path/to/reference.wav \
--ref-text "Original transcript of the reference audio"
```
The Python client supports the following command-line arguments:
- `--api-base`: API base URL (default: `http://localhost:8091`)
- `--model` (or `-m`): Model name/path (default: `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`)
- `--task-type` (or `-t`): TTS task type. Options: `CustomVoice`, `VoiceDesign`, `Base`
- `--text`: Text to synthesize (required)
- `--speaker`: Speaker name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc.
- `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`
- `--instructions`: Voice style/emotion instructions
- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server.
- `--ref-text`: Reference audio transcript for voice cloning (Base task).
- `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus`
- `--output` (or `-o`): Output audio file path (default: `tts_output.wav`)
#### Send request via curl
```bash
# Simple TTS request
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English"
}' --output output.wav
# With style instruction
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "I am so excited!",
"voice": "vivian",
"instructions": "Speak with great enthusiasm"
}' --output excited.wav
# List available voices in CustomVoice models
curl http://localhost:8091/v1/audio/voices
```
### Using OpenAI SDK
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
voice="vivian",
input="Hello, how are you?",
)
response.stream_to_file("output.wav")
```
### Using Python httpx
```python
import httpx
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English",
},
timeout=300.0,
)
with open("output.wav", "wb") as f:
f.write(response.content)
```
## API Reference
### Voices Endpoint
#### GET /v1/audio/voices
List all available voices/speakers from the loaded model, including both built-in model voices and uploaded custom voices.
**Response Example:**
```json
{
"voices": ["vivian", "ryan", "custom_voice_1"],
"uploaded_voices": [
{
"name": "custom_voice_1",
"consent": "user_consent_id",
"created_at": 1738660000,
"file_size": 1024000,
"mime_type": "audio/wav",
"ref_text": "The exact transcript of the audio sample.",
"speaker_description": "warm narrator"
}
]
}
```
Fields `ref_text` and `speaker_description` are omitted per-entry when not provided at upload time.
#### POST /v1/audio/voices
Upload a new voice sample for voice cloning in Base task TTS requests.
**Form Parameters:**
- `audio_sample` (required): Audio file (max 10MB, supported formats: wav, mp3, flac, ogg, aac, webm, mp4)
- `consent` (required): Consent recording ID
- `name` (required): Name for the new voice
- `ref_text` (optional): Transcript of the audio. Enables in-context voice cloning (higher quality).
- `speaker_description` (optional): Free-form description of the voice (e.g. "warm narrator", "energetic presenter"). Stored as metadata.
**Response Example:**
```json
{
"success": true,
"voice": {
"name": "custom_voice_1",
"consent": "user_consent_id",
"created_at": 1738660000,
"mime_type": "audio/wav",
"file_size": 1024000,
"ref_text": "The exact transcript of the audio sample.",
"speaker_description": "warm narrator"
}
}
```
Fields `ref_text` and `speaker_description` are omitted when not provided at upload time.
**Usage Example:**
```bash
curl -X POST http://localhost:8091/v1/audio/voices \
-F "audio_sample=@/path/to/voice_sample.wav" \
-F "consent=user_consent_id" \
-F "name=custom_voice_1" \
-F "ref_text=The exact transcript of the audio sample." \
-F "speaker_description=warm narrator"
```
### Endpoint
```
POST /v1/audio/speech
Content-Type: application/json
```
This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters.
### Request Body
```json
{
"input": "Text to synthesize",
"voice": "vivian",
"response_format": "wav",
"task_type": "CustomVoice",
"language": "Auto",
"instructions": "Optional style instructions",
"ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning",
"ref_text": "Reference audio transcript",
"x_vector_only_mode": false,
"max_new_tokens": 2048
}
```
> **Note:** The `model` field is optional when serving a single model, as the server already knows which model is loaded.
### Response
Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/wav`).
### Voice and language (summary)
- **Speaker**: Use the `voice` request field to select the speaker (e.g., `vivian`, `ryan`, `aiden`). List available speakers with `GET /v1/audio/voices`.
- **Language**: Use the `language` field for the codec language tag (`Auto`, `Chinese`, `English`, etc.). Default is `Auto` for automatic detection.
- **CustomVoice**: Requires a valid `voice` from the model’s speaker set. **VoiceDesign**: Use `instructions` to describe the voice. **Base**: Use `ref_audio` and `ref_text` for voice cloning.
## Parameters
### OpenAI Standard Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------ | -------------- | ----------------------------------------------------------- |
| `input` | string | **required** | Text to synthesize |
| `model` | string | server's model | Model to use (optional, should match server if specified) |
| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) |
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus |
| `speed` | float | 1.0 | Playback speed (0.25-4.0, not supported with `stream=true`) |
### vLLM-Omni Extension Parameters
| Parameter | Type | Default | Description |
| ---------------------------- | ------ | ------------- | -------------------------------------------------------------------------------------------------------------------- |
| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base |
| `language` | string | "Auto" | Language (see supported languages below) |
| `instructions` | string | "" | Voice style/emotion instructions |
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. |
| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
**Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
### Voice Clone Parameters (Base task)
| Parameter | Type | Required | Description |
| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------- |
| `ref_audio` | string | **Yes** | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) |
| `ref_text` | string | No | Transcript of reference audio (for ICL mode) |
| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) |
## Streaming
Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded
(one chunk per Code2Wav window, default 25 frames; configurable in the stage config):
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English",
"stream": true,
"response_format": "pcm"
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
```
**Constraints:**
- `stream=true` requires `response_format="pcm"` (raw 16-bit signed PCM, 24 kHz mono).
- `speed` adjustment is not supported when streaming.
- Requires the server stage config to have `async_chunk: true` (default in `qwen3_tts.yaml`).
## Streaming Text Input (WebSocket)
The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally, buffers and splits it at sentence boundaries, and generates audio per sentence.
When `stream_audio=true`, each sentence is emitted as `audio.start`, one or more binary PCM frames, and `audio.done`.
### Quick Start
```bash
python streaming_speech_client.py \
--text "Hello world. How are you? I am fine."
python streaming_speech_client.py \
--text "Hello world. How are you? I am fine." \
--simulate-stt --stt-delay 0.1
```
### WebSocket Protocol
Client -> Server:
```jsonc
{"type": "session.config", "voice": "Vivian", "task_type": "CustomVoice", "language": "Auto", "split_granularity": "sentence", "stream_audio": true, "response_format": "pcm"}
{"type": "input.text", "text": "Hello, how are you? "}
{"type": "input.done"}
```
Server -> Client:
```jsonc
{"type": "audio.start", "sentence_index": 0, "sentence_text": "Hello, how are you?", "format": "pcm", "sample_rate": 24000}
// binary PCM frame(s)
{"type": "audio.done", "sentence_index": 0, "total_bytes": 96000, "error": false}
{"type": "session.done", "total_sentences": 1}
```
## Limitations
- **Single request**: Batch processing is not yet optimized for online serving.
## Troubleshooting
1. **TTS model did not produce audio output**: Ensure you're using the correct model variant for your task type (CustomVoice task → CustomVoice model, etc.)
2. **Connection refused**: Make sure the server is running on the correct port
3. **Out of memory**: Use smaller model variant (`Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice`) or reduce `--gpu-memory-utilization`
4. **Unsupported speaker**: Use `/v1/audio/voices` to list available voices for the loaded model
5. **Voice clone fails**: Ensure you're using the Base model variant for voice cloning
## Example materials
??? abstract "gradio_demo.py"
``````py
--8<-- "examples/online_serving/qwen3_tts/gradio_demo.py"
``````
??? abstract "gradio_fastrtc_demo.py"
``````py
--8<-- "examples/online_serving/qwen3_tts/gradio_fastrtc_demo.py"
``````
??? abstract "openai_speech_client.py"
``````py
--8<-- "examples/online_serving/qwen3_tts/openai_speech_client.py"
``````
??? abstract "run_gradio_demo.sh"
``````sh
--8<-- "examples/online_serving/qwen3_tts/run_gradio_demo.sh"
``````
??? abstract "run_server.sh"
``````sh
--8<-- "examples/online_serving/qwen3_tts/run_server.sh"
``````
??? abstract "streaming_speech_client.py"
``````py
--8<-- "examples/online_serving/qwen3_tts/streaming_speech_client.py"
``````
??? abstract "tts_common.py"
``````py
--8<-- "examples/online_serving/qwen3_tts/tts_common.py"
``````
docs/user_guide/examples/online_serving/text_to_speech.md
+489
View File
@@ -0,0 +1,489 @@
# Text-To-Speech (Online Serving)
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech>.
vLLM-Omni exposes TTS models through the OpenAI-compatible
[`POST /v1/audio/speech`](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md) endpoint,
launched with `vllm serve <model> --omni`. Each TTS model has its own
subdirectory containing client snippets, gradio demos, and helper
scripts; this README is the single doc entry point for all of them.
For offline inference, see [`examples/offline_inference/text_to_speech`](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md).
For the full list of supported architectures across all modalities, see
[Supported Models](https://github.com/vllm-project/vllm-omni/tree/main/docs/models/supported_models.md).
## Supported Models
| Model | HuggingFace repo | Voice cloning | Streaming | Voice presets / upload | Gradio demo |
|---|---|---|---|---|---|
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | ✓ (Base) | ✓ (PCM + WebSocket) | ✓ (presets + `/v1/audio/voices` upload) | ✓ (standard + FastRTC) |
| VoxCPM | local model dir | ✓ | ✓ (PCM stream) | — | — |
| VoxCPM2 | `openbmb/VoxCPM2` | ✓ | ✓ (AudioWorklet via gradio) | — | ✓ |
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✓ (gated upstream) | ✓ | ✓ (presets) | ✓ |
CosyVoice3 is intentionally absent: no online example exists for it yet. See its [offline section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#cosyvoice3) instead.
## Common Quick Start
Launch the server (defaults shown — adjust `--port`, `--gpu-memory-utilization`, etc. as needed):
```bash
vllm serve <hf-repo-or-local-path> --omni --port 8091
```
Send a TTS request via curl:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav"
}' --output output.wav
```
Or via Python httpx:
```python
import httpx
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav",
},
timeout=300.0,
)
open("output.wav", "wb").write(response.content)
```
Or via the OpenAI SDK:
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
model="<hf-repo>",
voice="default",
input="Hello, how are you?",
)
response.stream_to_file("output.wav")
```
Streaming PCM output (where supported) — set `stream=true` with `response_format="pcm"`:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "default",
"stream": true,
"response_format": "pcm"
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
```
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others).
For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md).
---
## Fish Speech S2 Pro
4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec.
### Prerequisites
```bash
pip install fish-speech
```
### Launch
```bash
vllm serve fishaudio/s2-pro --omni --port 8091
# or:
./fish_speech/run_server.sh
```
The deploy config auto-loads from `vllm_omni/deploy/fish_qwen3_omni.yaml` (the HF `model_type` on the fishaudio checkpoint is `fish_qwen3_omni`).
### Voice cloning
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, this is a cloned voice.",
"voice": "default",
"ref_audio": "https://example.com/reference.wav",
"ref_text": "Transcript of the reference audio."
}' --output cloned.wav
```
### CLI client
```bash
cd examples/online_serving/text_to_speech/fish_speech
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Hello world" --stream --output output.pcm
```
### Gradio demo
```bash
./fish_speech/run_gradio_demo.sh # launches server + Gradio
python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server already running
```
### Notes
- Output: 44.1 kHz mono.
- Streaming PCM player command must use `-r 44100`.
---
## OmniVoice
Zero-shot multilingual TTS (600+ languages). Online serving currently exposes **auto voice** only; voice cloning and voice design are available offline.
### Prerequisites
```bash
huggingface-cli download k2-fsa/OmniVoice
```
Voice cloning (offline) needs `transformers>=5.3.0`; auto voice works with `transformers>=4.57.0`.
### Launch
```bash
vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code
# or:
./omnivoice/run_server.sh
```
### CLI client
```bash
cd examples/online_serving/text_to_speech/omnivoice
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Bonjour, comment allez-vous?" --language French
```
The client supports `--api-base`, `--model`, `--text`, `--response-format`, `--language`, `--output`.
### Notes
- Voice cloning and voice design require offline inference; see the [offline OmniVoice section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#omnivoice).
---
## Qwen3-TTS
Three model variants exposed via separate checkpoints:
| Variant | HF repo | Use |
|---|---|---|
| CustomVoice | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | Predefined speakers (`vivian`, `ryan`, …) with optional style instructions |
| VoiceDesign | `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | Natural-language voice style description |
| Base | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Voice cloning from a reference audio |
Each variant ships smaller `0.6B` companions where available.
### Launch
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
# or:
./qwen3_tts/run_server.sh # default: CustomVoice
./qwen3_tts/run_server.sh VoiceDesign
./qwen3_tts/run_server.sh Base
```
### Choosing an executor backend (uniproc vs mp)
Stage configs ship with the chunked-streaming default. To use the uniproc executor (lower IPC overhead for the Base cloning task), pass `--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_uniproc.yaml`. See [#2603](https://github.com/vllm-project/vllm-omni/issues/2603) and [#2604](https://github.com/vllm-project/vllm-omni/pull/2604) for the full investigation.
To opt out of chunked streaming, pass `--no-async-chunk` instead — the pipeline auto-dispatches to the end-to-end codec processor.
### Sending requests
```bash
# CustomVoice with a predefined speaker
python qwen3_tts/openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--text "今天天气真好" \
--voice ryan \
--instructions "用开心的语气说"
# VoiceDesign with a style description
python qwen3_tts/openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
--task-type VoiceDesign \
--text "哥哥，你回来啦" \
--instructions "体现撒娇稚嫩的萝莉女声，音调偏高"
# Base voice cloning
python qwen3_tts/openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--task-type Base \
--text "Hello, this is a cloned voice" \
--ref-audio /path/to/reference.wav \
--ref-text "Original transcript of the reference audio"
```
### Voices endpoint
List available voices, or upload a custom one for Base cloning:
```bash
# List
curl http://localhost:8091/v1/audio/voices
# Upload
curl -X POST http://localhost:8091/v1/audio/voices \
-F "audio_sample=@/path/to/voice_sample.wav" \
-F "consent=user_consent_id" \
-F "name=custom_voice_1" \
-F "ref_text=The exact transcript of the audio sample." \
-F "speaker_description=warm narrator"
```
Uploaded voices are then usable as `voice="custom_voice_1"` on subsequent requests.
### Streaming PCM
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English",
"stream": true,
"response_format": "pcm"
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
```
Streaming requires `response_format="pcm"` and `async_chunk: true` on the stage config (default in `qwen3_tts.yaml`). `speed` is not supported when streaming.
### Streaming WebSocket
The `/v1/audio/speech/stream` endpoint accepts text incrementally, splits it at sentence boundaries, and emits one PCM stream per sentence:
```bash
python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you? I am fine."
python qwen3_tts/streaming_speech_client.py --text "..." --simulate-stt --stt-delay 0.1
```
### Gradio demos
```bash
./qwen3_tts/run_gradio_demo.sh # CustomVoice (default)
./qwen3_tts/run_gradio_demo.sh --task-type VoiceDesign
./qwen3_tts/run_gradio_demo.sh --task-type Base
# FastRTC variant (gapless WebRTC streaming):
pip install fastrtc
python qwen3_tts/gradio_fastrtc_demo.py --api-base http://localhost:8000
```
### Speaker embedding interpolation
`qwen3_tts/speaker_embedding_interpolation.py` blends two predefined speakers' embeddings to produce intermediate voices. See the script for usage.
### Batch client
`qwen3_tts/batch_speech_client.py` issues many concurrent requests for throughput measurement.
### Notes
- Base voice cloning has uniproc-vs-mp tradeoffs depending on per-request reference audio cost; see the executor-backend section above.
- `vllm_omni/deploy/qwen3_tts.yaml` is the default deploy config (loaded by HF `model_type`); per-stage runtime overrides are available via `--stage-N-<field> <value>`.
---
## VoxCPM
Split-stage TTS at 24 kHz.
### Prerequisites
```bash
pip install voxcpm
# or use a local source tree:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src
```
If the native VoxCPM `config.json` lacks HF `model_type`, set up an HF-compatible config dir:
```bash
export VOXCPM_MODEL=/path/to/voxcpm-model
export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config
mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"
cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json"
cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true
python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)'
```
### Launch
```bash
export VOXCPM_MODEL=/path/to/voxcpm-model
./voxcpm/run_server.sh # async-chunk streaming (default)
./voxcpm/run_server.sh sync # non-streaming
```
Or directly:
```bash
vllm serve "$VOXCPM_MODEL" \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
--trust-remote-code --enforce-eager --omni --port 8091
```
### Sending requests
```bash
# Basic TTS
python voxcpm/openai_speech_client.py \
--model "$VOXCPM_MODEL" \
--text "This is a VoxCPM online text-to-speech example."
# Voice cloning
python voxcpm/openai_speech_client.py \
--model "$VOXCPM_MODEL" \
--text "This sentence is synthesized with a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript spoken in reference.wav."
# Streaming PCM
python voxcpm/openai_speech_client.py \
--model "$VOXCPM_MODEL" \
--text "This is a streaming VoxCPM request." \
--stream --output voxcpm_stream.pcm
```
### Notes
- `voxcpm.yaml` for one-shot decode; `voxcpm_async_chunk.yaml` for single-request streaming. Do not use the async-chunk config for concurrent requests or `/v1/audio/speech/batch`.
- Generic TTS fields not supported by VoxCPM: `voice`, `instructions`, `language`, `speaker_embedding`, `x_vector_only_mode`.
- For benchmark measurement, see [`benchmarks/voxcpm`](https://github.com/vllm-project/vllm-omni/tree/main/benchmarks/voxcpm/README.md).
---
## VoxCPM2
Single-stage native AR TTS at 48 kHz.
### Launch
```bash
vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000
```
Deploy config auto-loads from `vllm_omni/deploy/voxcpm2.yaml`. Pass `--deploy-config <path>` to override or `--stage-N-<field> <value>` for per-stage runtime tweaks.
### Sending requests
```bash
# Zero-shot synthesis
python voxcpm2/openai_speech_client.py --text "Hello, this is VoxCPM2."
# Voice cloning
python voxcpm2/openai_speech_client.py \
--text "This should sound like the reference speaker." \
--ref-audio /path/to/reference.wav
```
The `ref_audio` field accepts local file paths (auto-base64), HTTP URLs, or `data:audio/wav;base64,...` data URIs.
### Gradio demo (gapless streaming via AudioWorklet)
```bash
python voxcpm2/gradio_demo.py
```
Uses an AudioWorklet-based player adapted from the Qwen3-TTS demo for gap-free playback. Audio is streamed from the OpenAI Speech endpoint with `stream=true`.
---
## Voxtral TTS
Voxtral-4B-TTS (Mistral). Uses the `mistral_common` `SpeechRequest` protocol; voice presets are model-specific.
### Prerequisites
Latest `mistral_common` with `SpeechRequest` support:
```bash
pip install -e /path/to/mistral-common # or upgrade from PyPI when available
```
### Launch
```bash
vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091
```
Deploy config auto-loads from `vllm_omni/deploy/voxtral_tts.yaml`.
### Gradio demo
```bash
python voxtral_tts/gradio_demo.py
```
The demo handles voice-preset selection and reference-audio upload. `voxtral_tts/text_preprocess.py` provides the text-normalization helpers used by the demo (also available for other clients).
### Notes
- Voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`).
- Voice cloning is gated upstream and may require a recent `mistral_common`.
- A standalone CLI client is not yet shipped; the gradio demo is the canonical reference for now.
## Example materials
??? abstract "fish_speech/gradio_demo.py"
``````py
--8<-- "examples/online_serving/text_to_speech/fish_speech/gradio_demo.py"
``````
??? abstract "fish_speech/run_gradio_demo.sh"
``````sh
--8<-- "examples/online_serving/text_to_speech/fish_speech/run_gradio_demo.sh"
``````
??? abstract "fish_speech/run_server.sh"
``````sh
--8<-- "examples/online_serving/text_to_speech/fish_speech/run_server.sh"
``````
??? abstract "fish_speech/speech_client.py"
``````py
--8<-- "examples/online_serving/text_to_speech/fish_speech/speech_client.py"
``````
??? abstract "omnivoice/run_server.sh"
``````sh
--8<-- "examples/online_serving/text_to_speech/omnivoice/run_server.sh"
``````
??? abstract "omnivoice/speech_client.py"
``````py
--8<-- "examples/online_serving/text_to_speech/omnivoice/speech_client.py"
``````
??? abstract "qwen3_tts/batch_speech_client.py"
``````py
--8<-- "examples/online_serving/text_to_speech/qwen3_tts/batch_speech_client.py"
``````
??? abstract "qwen3_tts/gradio_demo.py"
``````py
--8<-- "examples/online_serving/text_to_speech/qwen3_tts/gradio_demo.py"
``````
??? abstract "qwen3_tts/openai_speech_client.py"
``````py
--8<-- "examples/online_serving/text_to_speech/qwen3_tts/openai_speech_client.py"
``````
??? abstract "qwen3_tts/run_gradio_demo.sh"
``````sh
--8<-- "examples/online_serving/text_to_speech/qwen3_tts/run_gradio_demo.sh"
``````
??? abstract "qwen3_tts/run_server.sh"
``````sh
--8<-- "examples/online_serving/text_to_speech/qwen3_tts/run_server.sh"
``````
??? abstract "qwen3_tts/speaker_embedding_interpolation.py"
``````py
--8<-- "examples/online_serving/text_to_speech/qwen3_tts/speaker_embedding_interpolation.py"
``````
??? abstract "qwen3_tts/streaming_speech_client.py"
``````py
--8<-- "examples/online_serving/text_to_speech/qwen3_tts/streaming_speech_client.py"
``````
??? abstract "qwen3_tts/tts_common.py"
``````py
--8<-- "examples/online_serving/text_to_speech/qwen3_tts/tts_common.py"
``````
??? abstract "voxcpm/openai_speech_client.py"
``````py
--8<-- "examples/online_serving/text_to_speech/voxcpm/openai_speech_client.py"
``````
??? abstract "voxcpm/run_server.sh"
``````sh
--8<-- "examples/online_serving/text_to_speech/voxcpm/run_server.sh"
``````
??? abstract "voxcpm2/gradio_demo.py"
``````py
--8<-- "examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py"
``````
??? abstract "voxcpm2/openai_speech_client.py"
``````py
--8<-- "examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py"
``````
??? abstract "voxtral_tts/gradio_demo.py"
``````py
--8<-- "examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py"
``````
??? abstract "voxtral_tts/text_preprocess.py"
``````py
--8<-- "examples/online_serving/text_to_speech/voxtral_tts/text_preprocess.py"
``````
examples/offline_inference/cosyvoice3/README.md
-71
View File
@@ -1,71 +0,0 @@
# CosyVoice3
## Setup
Install dependencies:
```
uv pip install -e .
```
> **Note:** This includes required libraries such as `soundfile`,
> `onnxruntime`, `x-transformers`, and `einops` via
> `requirements/common.txt` and platform-specific requirements files.
Download the model snapshot:
```
from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
```
Add `config.json` in `pretrained_models/Fun-CosyVoice3-0.5B/`:
```json
{
"model_type": "cosyvoice3",
"architectures": [
"CosyVoice3Model"
]
}
```
> **Why `config.json` is required:**
> `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers a class mapping.
> The loader still needs `model_type: "cosyvoice3"` from `config.json` to select that class.
> If no `config.json` is present, model type cannot be inferred automatically.
> If your downloaded checkpoint already includes a valid `config.json` with
> `model_type: "cosyvoice3"`, this manual step can be skipped.
Run the offline verification script:
```
python examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py \
--model pretrained_models/Fun-CosyVoice3-0.5B \
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN
```
## Implementation Overview
CosyVoice3 runs as a **2-stage Omni pipeline**.
- **Stage 0** (`talker`) converts text + prompt audio to speech tokens.
- **Stage 1** (`code2wav`) converts speech tokens + prompt features to waveform via flow matching and HiFiGAN.
Key components live in `vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py`.
- `CosyVoice3MultiModalProcessor` builds the multimodal inputs:
- Tokenizes `prompt` and `prompt_text`.
- Extracts speech tokens and mel features from the prompt audio.
- Extracts a speaker embedding.
- `CosyVoice3Model` implements both stages:
- Stage 0 uses `CosyVoice3LM` and outputs speech tokens + conditioning features.
- Stage 1 runs the flow model (DiT-based CFM) and HiFiGAN to synthesize waveform.
Pipeline topology lives in `vllm_omni/model_executor/models/cosyvoice3/pipeline.py`;
runtime tunables (batch size, memory limits, sampling) live in
`vllm_omni/deploy/cosyvoice3.yaml`. The deploy config auto-loads by
HF `model_type` and defaults to `async_chunk: true` (shared-memory
streaming). Pass `--no-async-chunk` on `vllm serve` to switch to the
legacy sync path where stage 1 runs `text2flow` over the full
speech-token sequence.
- Stage 0 emits latent speech tokens.
- Stage 1 consumes them via `sync_process_input_func` (sync mode) or the
shared-memory connector (async-chunk mode) and outputs audio.
examples/offline_inference/fish_speech/README.md
-75
View File
@@ -1,75 +0,0 @@
# Fish Speech S2 Pro
This directory contains an offline demo for running Fish Speech S2 Pro (`fishaudio/s2-pro`) with vLLM Omni. It supports text-only synthesis and voice cloning with reference audio.
## Model Overview
[Fish Speech S2 Pro](https://huggingface.co/fishaudio/s2-pro) is a 4B dual-AR text-to-speech model by FishAudio. It uses:
- **Slow AR**: Generates semantic tokens autoregressively (Qwen3-based backbone)
- **Fast AR**: Predicts residual codebook tokens from semantic tokens
- **DAC Decoder**: Converts codec codes to 44.1 kHz audio waveform
The model produces high-quality speech with voice cloning capabilities.
## Prerequisites
Install the `fish-speech` package for the DAC codec:
```bash
pip install fish-speech
```
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
## Quick Start
### Text-Only Synthesis
```bash
python end2end.py --text "Hello, this is a test of the Fish Speech text to speech system."
```
Generated audio files are saved to `output_audio/` by default.
### Voice Cloning
Provide a reference audio file and its transcript to clone a voice:
```bash
python end2end.py \
--text "Hello, this is a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
## Streaming Mode
Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config):
```bash
python end2end.py --text "Hello, this is a test." --streaming
```
Each DAC decoder chunk is logged as it arrives. The final WAV file is written once generation completes.
## Configuration
| Argument | Default | Description |
|----------|---------|-------------|
| `--text` | `"Hello, this is a test..."` | Text to synthesize |
| `--ref-audio` | None | Path to reference audio for voice cloning |
| `--ref-text` | None | Transcript of the reference audio |
| `--model` | `fishaudio/s2-pro` | HuggingFace model path |
| `--stage-configs-path` | `fish_speech_s2_pro.yaml` | Path to stage configs YAML |
| `--output-dir` | `output_audio` | Output directory for WAV files |
| `--streaming` | False | Enable streaming via AsyncOmni |
| `--stage-init-timeout` | 600 | Stage initialization timeout (seconds) |
## Notes
- Output audio is 44.1 kHz mono WAV.
- The DAC codec weights (`codec.pth`) are loaded lazily from the model directory.
- Voice cloning encodes the reference audio using the DAC codec to extract semantic codes, then prepends them as a system message.
examples/offline_inference/omnivoice/README.md
-73
View File
@@ -1,73 +0,0 @@
# OmniVoice
This directory contains an offline demo for running OmniVoice TTS models with vLLM Omni. It generates speech from text and saves WAV files locally.
## Model Overview
[OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) is a zero-shot multilingual TTS model supporting 600+ languages. It uses a diffusion language model (Qwen3-0.6B backbone) with iterative masked unmasking to generate speech.
Three inference modes are supported:
- **Auto Voice**: Generate speech without any reference — the model picks a voice automatically.
- **Voice Clone**: Clone a voice from a reference audio + transcription.
- **Voice Design**: Control voice style via natural language instruction (e.g., "female, low pitch, british accent").
## Setup
Ensure the model is downloaded:
```bash
huggingface-cli download k2-fsa/OmniVoice
```
> **Note:** Voice cloning requires `transformers>=5.3.0` for `HiggsAudioV2TokenizerModel`. Auto voice and voice design modes work with `transformers>=4.57.0`.
## Quick Start
Auto voice (text only):
```bash
python end2end.py --model k2-fsa/OmniVoice --text "Hello, this is a test."
```
Voice design (with style instruction):
```bash
python end2end.py --model k2-fsa/OmniVoice \
--text "Hello, this is a test." \
--instruct "female, low pitch, british accent"
```
Voice clone (with reference audio):
```bash
python end2end.py --model k2-fsa/OmniVoice \
--text "Hello, this is a test." \
--ref-audio ref.wav \
--ref-text "This is the reference transcription."
```
## Language Support
Specify a language for improved quality:
```bash
python end2end.py --model k2-fsa/OmniVoice \
--text "你好，这是一个测试。" \
--lang zh
```
## Architecture
OmniVoice uses a two-stage pipeline:
- **Stage 0 (Generator)**: Qwen3-0.6B transformer with 32-step iterative unmasking and classifier-free guidance. Generates 8-codebook audio tokens from text.
- **Stage 1 (Decoder)**: HiggsAudioV2 RVQ quantizer + DAC acoustic decoder. Converts tokens to 24kHz waveform.
Both stages use `GPUGenerationWorker` with `OmniGenerationScheduler`.
## Notes
- Output audio is saved to `output.wav` by default. Use `--output` to change the path.
- The model estimates duration from text automatically via `RuleDurationEstimator`.
- Use `--stage-init-timeout` to increase the stage initialization timeout for first-time model downloads.
examples/offline_inference/qwen3_tts/README.md
-121
View File
@@ -1,121 +0,0 @@
# Qwen3-TTS
This directory contains an offline demo for running Qwen3 TTS models with vLLM Omni. It builds task-specific inputs and generates WAV files locally.
## Model Overview
Qwen3 TTS provides multiple task variants for speech generation:
- **CustomVoice**: Generate speech with a known speaker identity (speaker ID) and optional instruction.
- **VoiceDesign**: Generate speech from text plus a descriptive instruction that designs a new voice.
- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. The `ref_audio` field accepts a local file path, HTTP URL, or base64 data URL.
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
### ROCm Dependencies
You will need to install the dependency `onnxruntime-rocm`.
```
pip uninstall onnxruntime # should be removed before we can install onnxruntime-rocm
pip install onnxruntime-rocm
```
## Quick Start
Run a single sample for a task:
```
python end2end.py --query-type CustomVoice
```
Generated audio files are saved to `output_audio/` by default.
## Task Usage
### CustomVoice
Single sample:
```
python end2end.py --query-type CustomVoice
```
Batch sample (multiple prompts in one run):
```
python end2end.py --query-type CustomVoice --use-batch-sample
```
### VoiceDesign
Single sample:
```
python end2end.py --query-type VoiceDesign
```
Batch sample:
```
python end2end.py --query-type VoiceDesign --use-batch-sample
```
### Base (Voice Clone)
Single sample:
```
python end2end.py --query-type Base
```
Batch sample:
```
python end2end.py --query-type Base --use-batch-sample
```
Mode selection for Base:
- `--mode-tag icl` (default): standard mode
- `--mode-tag xvec_only`: enable `x_vector_only_mode` in the request
Examples:
```
python end2end.py --query-type Base --mode-tag icl
```
## Streaming Mode
Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config):
```bash
python end2end.py --query-type CustomVoice --streaming --output-dir /tmp/out_stream
```
Each Code2Wav chunk is logged as it arrives (default 25 frames; configurable via `codec_chunk_frames`
in the stage config). The initial chunk size is dynamically selected based on server load for reduced
TTFA, and can be overridden per-request via the `initial_codec_chunk_frames` API field. The final WAV file is written once generation
completes. This demonstrates that audio data is available progressively rather than only at the end.
> **Note:** Streaming uses `AsyncOmni` internally. The non-streaming path (`Omni`) is unchanged.
## Batched Decoding
The Code2Wav stage (stage 1) supports batched decoding, where multiple requests are decoded in a single forward pass through the SpeechTokenizer. To use it, set `max_num_seqs > 1` on both stages via `--stage-overrides` and pass multiple prompts via `--txt-prompts` with a matching `--batch-size`.
```
python end2end.py --query-type CustomVoice \
--txt-prompts benchmark_prompts.txt \
--batch-size 4 \
--stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}'
```
**Important:** `--batch-size` must match a CUDA graph capture size (1, 2, 4, 8, 16...) because the Talker's code predictor KV cache is sized to `max_num_seqs`, and CUDA graphs pad the batch to the next capture size. Both stages need `max_num_seqs >= batch_size` in the stage config for batching to take effect. If only stage 1 has a higher `max_num_seqs`, it won't help — stage 1 can only batch chunks from requests that are in-flight simultaneously, which requires stage 0 to also process multiple requests concurrently.
## Notes
- The script uses the model paths embedded in `end2end.py`. Update them if your local cache path differs.
- Use `--output-dir` to change the output folder.
examples/offline_inference/text_to_speech/README.md
+367
View File
@@ -0,0 +1,367 @@
# Text-To-Speech
vLLM-Omni supports several autoregressive TTS models. They share a mostly
common CLI shape (`--text`, `--ref-audio`, `--ref-text`, plus an
output-path flag — `--output-dir` for most, `--output` for OmniVoice) and
live together in this hub. Each model has its own subdirectory containing
a single `end2end.py` script; this README is the single doc entry point.
For online serving, see [`examples/online_serving/text_to_speech/`](../../online_serving/text_to_speech/README.md). For the full
list of supported architectures across all modalities, see
[Supported Models](../../../docs/models/supported_models.md).
## Supported Models
| Model | HuggingFace repo | Stages | Voice cloning | Streaming | Special modes | Sample rate |
|---|---|---|---|---|---|---|
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ | — | 22.05 kHz |
| Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ | — | 44.1 kHz |
| OmniVoice | `k2-fsa/OmniVoice` | 2 (gen + dec) | ✓ | — | voice design, language hint | 24 kHz |
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | 2 (talker + code2wav) | ✓ (Base) | ✓ | 3 task variants | 24 kHz |
| VoxCPM | local model dir | split | ✓ | ✓ | — | 24 kHz |
| VoxCPM2 | `openbmb/VoxCPM2` | single (native AR) | ✓ | ✓ (online) | continuation | 48 kHz |
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | varies | ✓ | ✓ | voice presets | 24 kHz |
## Common Quick Start
Most models share this invocation shape:
```bash
python examples/offline_inference/text_to_speech/<model>/end2end.py \
--text "Hello, this is a test." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
`--ref-audio` and `--ref-text` are optional (text-only synthesis works without them) and must be provided together for voice cloning. The exotic scripts — Qwen3-TTS, Voxtral TTS, CosyVoice3 — accept additional model-specific flags documented in their per-model section below. Qwen3-TTS in particular uses its own argparse surface (`--query-type`, `--audio-path`, etc.) and does not follow the common shape; see its section.
---
## CosyVoice3
2-stage TTS pipeline (`talker` + `code2wav`) at 22.05 kHz.
### Prerequisites
```bash
uv pip install -e .
# Includes soundfile, onnxruntime, x-transformers, einops via requirements.
```
Download the model snapshot:
```python
from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512',
local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
```
If your downloaded checkpoint lacks `config.json`, add it:
```json
{
"model_type": "cosyvoice3",
"architectures": ["CosyVoice3Model"]
}
```
This is required because `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers the class mapping; the loader still reads `model_type` from `config.json` to select the class.
### Quick start
```bash
python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
--model pretrained_models/Fun-CosyVoice3-0.5B \
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN
```
### Voice cloning
Pass a reference audio. Note that CosyVoice3's `--prompt-text` is a system-style prompt for the GPT stage, not a reference transcript:
```bash
python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
--model pretrained_models/Fun-CosyVoice3-0.5B \
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN \
--ref-audio prompt.wav \
--prompt-text "You are a helpful assistant.<|endofprompt|>Testing my voices. Why should I not?"
```
### Streaming
Streaming is enabled by default via `async_chunk: true` in `vllm_omni/deploy/cosyvoice3.yaml`. Pass `--no-async-chunk` on `vllm serve` to switch to the legacy synchronous path.
### Notes
- Stage 0 (`talker`) emits speech tokens; stage 1 (`code2wav`) runs flow matching + HiFiGAN to synthesize waveform.
- Deploy config auto-loads from `vllm_omni/deploy/cosyvoice3.yaml` based on HF `model_type`. Pass `--deploy-config <path>` to override.
---
## Fish Speech S2 Pro
4B dual-AR text-to-speech model from FishAudio with the DAC codec at 44.1 kHz.
### Prerequisites
```bash
pip install fish-speech
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a test of the Fish Speech text to speech system."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
### Streaming
```bash
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a streaming test." \
--streaming
```
Streaming requires `async_chunk: true` in the stage config.
### Notes
- Output: 44.1 kHz mono WAV.
- DAC codec weights (`codec.pth`) are loaded lazily from the model directory.
---
## OmniVoice
Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design).
### Prerequisites
```bash
huggingface-cli download k2-fsa/OmniVoice
```
Voice cloning requires `transformers>=5.3.0`. Auto and design modes work with `transformers>=4.57.0`.
### Quick start (auto voice)
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "Hello, this is a test."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "Hello, this is a test." \
--ref-audio ref.wav \
--ref-text "This is the reference transcription."
```
### Voice design
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "Hello, this is a test." \
--instruct "female, low pitch, british accent"
```
### Language hint
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "你好，这是一个测试。" \
--lang zh
```
### Notes
- Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking.
- Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz.
---
## Qwen3-TTS
3-task-variant TTS with 24 kHz output. Has its own argparse surface (this script does not follow the common `--text` / `--ref-audio` shape).
### Prerequisites
For ROCm builds, replace `onnxruntime` with `onnxruntime-rocm`:
```bash
pip uninstall onnxruntime
pip install onnxruntime-rocm
```
### Task variants
- `CustomVoice`: predefined speaker (speaker ID) with optional style instruction.
- `VoiceDesign`: text + descriptive instruction designs a new voice.
- `Base`: voice cloning from reference audio + transcript.
```bash
# Single sample
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type VoiceDesign
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base
# Base with a custom reference audio (Qwen3-TTS uses --audio-path, not --ref-audio):
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
--query-type Base --audio-path /path/to/reference.wav
# Base variant has an additional mode flag:
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag icl # default
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag xvec_only # x_vector_only_mode
# Batch (multiple prompts in one run)
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice --use-batch-sample
```
### Streaming
```bash
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
--query-type CustomVoice \
--streaming \
--output-dir /tmp/out_stream
```
Streaming requires `async_chunk: true` in the stage config.
### Batched decoding
The Code2Wav stage supports batched decoding through the SpeechTokenizer. Pass multiple prompts via `--txt-prompts` and set `--batch-size` accordingly. To raise `max_num_seqs` on either stage, point `--stage-configs-path` at a stage configs YAML with the desired values (see `vllm_omni/model_executor/stage_configs/` for templates):
```bash
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
--query-type CustomVoice \
--txt-prompts examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt \
--batch-size 4 \
--stage-configs-path /path/to/qwen3_tts_batched.yaml
```
`--batch-size` must match a CUDA-graph capture size (1, 2, 4, 8, 16…).
### Notes
- Run `--help` for the full argument surface.
- See `qwen3_tts/end2end.py` for the prompt-length-estimation logic the Talker uses.
---
## VoxCPM
Split-stage TTS. The hub example covers single TTS, single voice cloning, and streaming. Use `benchmarks/voxcpm/` for warmup, batch JSONL prompts, profiler injection, and offline TTFP / RTF measurement.
### Prerequisites
```bash
pip install voxcpm soundfile
# or use a local source tree:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src
```
If the native VoxCPM `config.json` does not contain HF metadata such as `model_type`, prepare a persistent HF-compatible config directory and point the stage configs to it via `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH`:
```bash
export VOXCPM_MODEL=/path/to/voxcpm-model
export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config
mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"
cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json"
cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true
python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)'
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--text "This is a split-stage VoxCPM synthesis example running on vLLM Omni."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--text "This sentence is synthesized with a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript spoken in reference.wav."
```
### Streaming
Pass the async-chunk stage config:
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
--text "This is a split-stage VoxCPM streaming example running on vLLM Omni."
```
### Notes
- `voxcpm.yaml` is the default non-streaming stage config; `voxcpm_async_chunk.yaml` enables streaming.
- Streaming is currently single-request oriented.
- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades quality.
- For online serving, see the [VoxCPM section in the online hub](../../online_serving/text_to_speech/README.md#voxcpm). For benchmark reporting, see [`benchmarks/voxcpm`](../../../benchmarks/voxcpm/README.md).
---
## VoxCPM2
Single-stage native AR TTS at 48 kHz. Pipeline: `feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE`.
### Prerequisites
```bash
pip install voxcpm
# or, for a local source checkout:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
--model openbmb/VoxCPM2 \
--text "Hello, this is a VoxCPM2 demo."
```
### Voice cloning
Pass a reference audio for isolated cloning, or both `--ref-audio` + `--ref-text` for prompt continuation:
```bash
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
--text "Hello, this is a voice clone demo." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
### Streaming
Streaming is exposed through the online OpenAI Speech API (`stream=true`). See [`examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py`](../../online_serving/text_to_speech/voxcpm2/gradio_demo.py) for an AudioWorklet-based gapless streaming player; the offline `end2end.py` script does not expose a streaming path.
### Notes
- Output: 48 kHz mono WAV.
- Deploy config: `vllm_omni/deploy/voxcpm2.yaml` (auto-loaded by HF `model_type`).
---
## Voxtral TTS
Voxtral-4B-TTS (Mistral). Has its own argparse surface; uses voice presets and the `mistral_common` `SpeechRequest` protocol.
### Prerequisites
Latest `mistral_common` with `SpeechRequest` support:
```bash
pip install -e /path/to/mistral-common # or upgrade from PyPI when available
```
### Quick start (voice preset)
```bash
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--write-audio --voice cheerful_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
```
### Voice cloning (capability gated upstream)
```bash
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--write-audio \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "This is a test message." \
--ref-audio path/to/reference_audio.wav
```
### Streaming + concurrency
```bash
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "..."
```
Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`).
### Notes
- `--num-prompts N` replicates the prompt for performance measurement.
- `--concurrency M` requires `--streaming` and must evenly divide `--num-prompts`.
- Run `--help` for the full argument surface.
examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py → examples/offline_inference/text_to_speech/cosyvoice3/end2end.py
+9 -9
View File
@@ -31,13 +31,13 @@ def run_e2e():
help="Override the deploy config path. If unset, auto-loads "
"vllm_omni/deploy/cosyvoice3.yaml based on the HF model_type.",
)
parser.add_argument("--prompt", type=str, default="Hello, this is a test of the CosyVoice system capability.")
parser.add_argument("--text", type=str, default="Hello, this is a test of the CosyVoice system capability.")
parser.add_argument(
"--prompt-text",
type=str,
default="You are a helpful assistant.<|endofprompt|>Testing my voices. Why should I not?",
)
parser.add_argument("--audio-path", type=str, default="prompt.wav")
parser.add_argument("--ref-audio", type=str, default="prompt.wav")
parser.add_argument(
"--tokenizer",
type=str,
@@ -64,11 +64,11 @@ def run_e2e():
sampling_cfg = {"top_p": 0.8, "top_k": 25, "eos_token_id": 6561 + 1}
print("Model initialized. Preparing inputs...")
if args.audio_path:
if not os.path.exists(args.audio_path):
raise FileNotFoundError(f"Audio file not found: {args.audio_path}")
if args.ref_audio:
if not os.path.exists(args.ref_audio):
raise FileNotFoundError(f"Audio file not found: {args.ref_audio}")
# Load at native sample rate
audio_signal, sr = load_audio(args.audio_path, sr=None)
audio_signal, sr = load_audio(args.ref_audio, sr=None)
# Validate sample rate before processing (similar to original CosyVoice)
min_sr = 16000
@@ -84,7 +84,7 @@ def run_e2e():
audio_data = AudioAsset("mary_had_lamb").audio_and_sample_rate
prompts = {
"prompt": args.prompt,
"prompt": args.text,
"multi_modal_data": {
"audio": audio_data,
},
@@ -94,7 +94,7 @@ def run_e2e():
},
}
print(f"Generating for prompt: {args.prompt}")
print(f"Generating for prompt: {args.text}")
config = CosyVoice3Config()
tokenizer = get_qwen_tokenizer(
@@ -102,7 +102,7 @@ def run_e2e():
skip_special_tokens=config.skip_special_tokens,
version=config.version,
)
_, text_token_len = extract_text_token(args.prompt, tokenizer, config.allowed_special)
_, text_token_len = extract_text_token(args.text, tokenizer, config.allowed_special)
base_len = int(text_token_len)
min_len = int(base_len * config.min_token_text_ratio)
max_len = int(base_len * config.max_token_text_ratio)
examples/offline_inference/fish_speech/end2end.py → examples/offline_inference/text_to_speech/fish_speech/end2end.py
+1
View File
@@ -44,6 +44,7 @@ DEFAULT_STAGE_CONFIG = os.path.join(
"..",
"..",
"..",
"..",
"vllm_omni",
"model_executor",
"stage_configs",
examples/offline_inference/omnivoice/end2end.py → examples/offline_inference/text_to_speech/omnivoice/end2end.py
View File
File renamed without changes.
examples/offline_inference/qwen3_tts/benchmark_prompts.txt → examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt
View File
File renamed without changes.
examples/offline_inference/qwen3_tts/end2end.py → examples/offline_inference/text_to_speech/qwen3_tts/end2end.py
View File
File renamed without changes.
examples/offline_inference/voxcpm/end2end.py → examples/offline_inference/text_to_speech/voxcpm/end2end.py
+1 -1
View File
@@ -13,7 +13,7 @@ from vllm.utils.argparse_utils import FlexibleArgumentParser
from vllm_omni import AsyncOmni, Omni
REPO_ROOT = Path(__file__).resolve().parents[3]
REPO_ROOT = Path(__file__).resolve().parents[4]
DEFAULT_SYNC_STAGE_CONFIG = REPO_ROOT / "vllm_omni" / "model_executor" / "stage_configs" / "voxcpm.yaml"
examples/offline_inference/voxcpm2/end2end.py → examples/offline_inference/text_to_speech/voxcpm2/end2end.py
+6 -18
View File
@@ -16,7 +16,7 @@ from vllm.utils.argparse_utils import FlexibleArgumentParser
from vllm_omni import Omni
REPO_ROOT = Path(__file__).resolve().parents[3]
REPO_ROOT = Path(__file__).resolve().parents[4]
DEFAULT_STAGE_CONFIGS_PATH = str(REPO_ROOT / "vllm_omni" / "model_executor" / "stage_configs" / "voxcpm2.yaml")
SAMPLE_RATE = 48_000
@@ -48,28 +48,16 @@ def parse_args():
help="Path to the stage config YAML file.",
)
parser.add_argument(
"--reference-audio",
"--ref-audio",
type=str,
default=None,
help="Path to reference audio for voice cloning (isolated ref mode).",
)
parser.add_argument(
"--prompt-audio",
type=str,
default=None,
help="Path to prompt audio for continuation mode (requires --prompt-text).",
)
parser.add_argument(
"--prompt-text",
type=str,
default=None,
help="Text matching --prompt-audio for continuation mode.",
help="Path to reference audio for voice cloning.",
)
parser.add_argument(
"--ref-text",
type=str,
default=None,
help="Optional transcript of --reference-audio (enables ref_continuation mode).",
help="Optional transcript of --ref-audio (enables continuation mode).",
)
return parser.parse_args()
@@ -120,8 +108,8 @@ def main():
split_map = build_cjk_split_map(tokenizer)
hf_config = engine.engine.stage_vllm_configs[0].model_config.hf_config
ref_audio_arg = args.reference_audio or args.prompt_audio
ref_text_arg = args.ref_text or args.prompt_text
ref_audio_arg = args.ref_audio
ref_text_arg = args.ref_text
ref_wav, ref_sr = (None, None)
if ref_audio_arg:
ref_wav_arr, ref_sr = sf.read(ref_audio_arg)
examples/offline_inference/voxtral_tts/end2end.py → examples/offline_inference/text_to_speech/voxtral_tts/end2end.py
+4 -4
View File
@@ -241,7 +241,7 @@ def parse_args() -> Namespace:
help="Text to synthesize.",
)
parser.add_argument(
"--audio-path",
"--ref-audio",
type=str,
default=None,
help="Path to reference audio file for voice cloning.",
@@ -344,10 +344,10 @@ def main(args: Any) -> None:
model_name = args.model
output_dir = args.output_dir
if args.voice is None and args.audio_path is None:
raise ValueError("Either --voice or --audio-path must be provided.")
if args.voice is None and args.ref_audio is None:
raise ValueError("Either --voice or --ref-audio must be provided.")
audio_prompt_file = args.audio_path
audio_prompt_file = args.ref_audio
text_chunk = TextChunk(text=args.text)
if args.write_audio:
examples/offline_inference/voxcpm/README.md
-123
View File
@@ -1,123 +0,0 @@
# VoxCPM Offline Example
This directory contains the minimal offline VoxCPM example for vLLM Omni.
`end2end.py` is intentionally small and only covers:
- single text-to-speech
- single voice cloning with `ref_audio` + `ref_text`
- non-streaming with `vllm_omni/model_executor/stage_configs/voxcpm.yaml`
- streaming with `vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml`
Advanced workflows were moved out of the getting-started example:
- `benchmarks/voxcpm/vllm_omni/bench_tts_offline.py`: warmup, batch prompts, profiler, offline TTFP / RTF
- `benchmarks/voxcpm/vllm_omni/run_offline_matrix.py`: fixed offline smoke matrix
- `benchmarks/voxcpm/`: benchmark scripts and benchmark docs
## Prerequisites
Install VoxCPM in one of these ways:
```bash
pip install voxcpm
```
or point vLLM Omni to the local VoxCPM source tree:
```bash
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src
```
The example writes WAV files with `soundfile`:
```bash
pip install soundfile
```
## Model Path
Pass the native VoxCPM model directory directly:
```bash
export VOXCPM_MODEL=/path/to/voxcpm-model
```
If the native VoxCPM `config.json` does not contain HuggingFace metadata such as
`model_type`, prepare a persistent HF-compatible config directory and point the
stage configs to it with `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH`:
```bash
export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config
mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"
cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json"
cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true
python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)'
```
If the model directory itself already has `model_type`, this extra directory is
not required.
## Quick Start
Single text-to-speech, non-streaming:
```bash
python examples/offline_inference/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--text "This is a split-stage VoxCPM synthesis example running on vLLM Omni."
```
Single voice cloning, non-streaming:
```bash
python examples/offline_inference/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--text "This sentence is synthesized with a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript spoken in reference.wav."
```
Streaming:
```bash
python examples/offline_inference/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
--text "This is a split-stage VoxCPM streaming example running on vLLM Omni."
```
By default, `end2end.py` writes to `output_audio/` for non-streaming and
`output_audio_streaming/` for streaming.
## Advanced Workflows
Use `benchmarks/voxcpm/vllm_omni/bench_tts_offline.py` when you need:
- warmup runs
- prompt files
- batch JSONL inputs
- profiler injection
- offline TTFP / RTF emission
Use `benchmarks/voxcpm/vllm_omni/run_offline_matrix.py` when you need the fixed offline smoke matrix that previously lived in `test.py`.
Full matrix benchmark example:
```bash
python benchmarks/voxcpm/vllm_omni/run_offline_matrix.py \
--model "$VOXCPM_MODEL" \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript spoken in reference.wav."
```
For online serving examples, see [examples/online_serving/voxcpm](../../online_serving/voxcpm/README.md).
For benchmark reporting, see [benchmarks/voxcpm](../../../benchmarks/voxcpm/README.md).
## Notes
- `voxcpm.yaml` is the default non-streaming stage config.
- `voxcpm_async_chunk.yaml` is the streaming stage config.
- Streaming is currently single-request oriented; the fixed smoke matrix now lives in `benchmarks/voxcpm/vllm_omni/run_offline_matrix.py`.
- `ref_text` must be the real transcript of the reference audio. Mismatched text usually causes obvious quality degradation.
examples/offline_inference/voxcpm2/README.md
-83
View File
@@ -1,83 +0,0 @@
# VoxCPM2 Offline Inference (Native AR)
VoxCPM2 is a 2B-parameter tokenizer-free diffusion AR TTS model. It produces 48kHz audio and supports 30+ languages with a single-stage native AR pipeline backed by MiniCPM4.
## Prerequisites
Install the `voxcpm` package, or set the environment variable pointing to the source tree:
```bash
# Option A: install package
pip install voxcpm
# Option B: use source checkout
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm
```
## Quick Start
Zero-shot synthesis:
```bash
python examples/offline_inference/voxcpm2/end2end.py \
--model openbmb/VoxCPM2 \
--text "Hello, this is a VoxCPM2 demo." \
--output-dir output_audio
```
Voice cloning with a reference audio:
```bash
python examples/offline_inference/voxcpm2/end2end.py \
--text "Hello, this is a voice clone demo." \
--reference-audio /path/to/reference.wav \
--output-dir output_clone
```
Prompt continuation (matched audio + text prefix):
```bash
python examples/offline_inference/voxcpm2/end2end.py \
--text "Continuation target sentence." \
--prompt-audio /path/to/prompt.wav \
--prompt-text "Transcript of the prompt audio." \
--output-dir output_cont
```
The script accepts the following arguments:
| Argument | Default | Description |
|---|---|---|
| `--model` | `openbmb/VoxCPM2` | HuggingFace repo ID or local path |
| `--text` | (example sentence) | Text to synthesize |
| `--output-dir` | `output_audio` | Directory for output WAV files |
| `--stage-configs-path` | `voxcpm2.yaml` | Stage config YAML path |
| `--reference-audio` | `None` | Reference audio for voice cloning (isolated) |
| `--prompt-audio` | `None` | Prompt audio for continuation mode |
| `--prompt-text` | `None` | Transcript matching `--prompt-audio` |
## Performance
Measured on a single H20 GPU (80 GB):
| Input length | RTF | Sample rate |
|---|---|---|
| Short (~10 tokens) | ~0.28 | 48 kHz |
| Long (~100 tokens) | ~0.34 | 48 kHz |
RTF < 1.0 means faster than real time.
## Architecture
VoxCPM2 uses a single-stage native AR pipeline:
```
feat_encoder
└─► MiniCPM4 (base LM)
└─► FSQ (finite scalar quantization)
└─► residual_lm (residual AR)
└─► LocDiT (local diffusion transformer)
└─► AudioVAE → 48 kHz waveform
```
All stages are fused into one vllm-native execution graph via `voxcpm2.yaml`, eliminating inter-stage coordination overhead and enabling true end-to-end batching.
examples/offline_inference/voxtral_tts/README.md
-59
View File
@@ -1,59 +0,0 @@
# Voxtral TTS Offline Inference
`end2end.py` runs Voxtral TTS end-to-end offline inference using vLLM. It supports both blocking (`Omni`) and streaming (`AsyncOmni`) generation, batched prompts with configurable concurrency, and voice selection via preset name or reference audio file.
When `mistral_common` has `SpeechRequest` support, prompt token IDs are built via `encode_speech_request`. Otherwise, the script falls back to manual token construction.
## Usage Examples
```bash
# Basic single-prompt with cheerful_female voice preset
python3 examples/offline_inference/voxtral_tts/end2end.py \
--write-audio --voice cheerful_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# 32 replicate prompts with cheerful_female voice preset
python3 examples/offline_inference/voxtral_tts/end2end.py \
--num-prompts 32 --write-audio --voice cheerful_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# Streaming with neutral_female voice preset
python3 examples/offline_inference/voxtral_tts/end2end.py \
--streaming --write-audio --voice neutral_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# 32 prompts, 8 concurrent requests per wave, streaming with neutral_female voice
python3 examples/offline_inference/voxtral_tts/end2end.py \
--num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# Short debug prompt with reference audio
# Note: Reference audio capability is not yet released.
python3 examples/offline_inference/voxtral_tts/end2end.py \
--write-audio \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "This is a test message." \
--audio-path path/to/reference_audio.wav
```
## Arguments
| Argument | Description |
|---|---|
| `--model PATH` | HuggingFace repo ID or local directory path (default: `mistralai/Voxtral-4B-TTS-2603`) |
| `--text TEXT` | Text to synthesize (default: `"This is a test message."`) |
| `--audio-path PATH` | Path to reference audio file for voice cloning |
| `--output-dir DIR` | Directory to write output WAV files (default: `output_audio`) |
| `--deploy-config PATH` | Override the deploy config path. If unset, auto-loads `vllm_omni/deploy/voxtral_tts.yaml` from the HF `model_type`. |
| `--num-prompts N` | Number of replicate prompts to run for measuring performance (default: 1) |
| `--streaming` | Use streaming generation via `AsyncOmni` (default: blocking `Omni`) |
| `--concurrency N` | Max concurrent requests per wave (must be used with `--streaming`, must evenly divide `--num-prompts`) |
| `--voice NAME` | Voice preset to use instead of reference audio file. Check Huggingface `mistralai/Voxtral-4B-TTS-2603` to get the list of available voices |
| `--write-audio` | Write generated audio to WAV files |
| `--profiling-mode` | Enable profiling mode (reduces max tokens to 50) |
| `--log-stats` | Enable detailed statistics logging |
examples/online_serving/fish_speech/README.md
-169
View File
@@ -1,169 +0,0 @@
# Fish Speech S2 Pro
## Model
| Model | Description |
|-------|-------------|
| `fishaudio/s2-pro` | Fish Speech S2 Pro -- 4B dual-AR TTS model with DAC codec (44.1 kHz) |
## Gradio Demo
An interactive Gradio demo is available with text-to-speech synthesis, voice cloning, and streaming support.
```bash
# Option 1: Launch server + Gradio together
./run_gradio_demo.sh
# Option 2: If server is already running
python gradio_demo.py --api-base http://localhost:8091
```
Then open http://localhost:7860 in your browser.
Features:
- Text-to-speech synthesis
- Voice cloning from uploaded audio or URL
- Streaming mode (progressive PCM playback)
## Launch the Server
```bash
vllm serve fishaudio/s2-pro --omni --port 8091
```
The deploy config is auto-loaded from `vllm_omni/deploy/fish_qwen3_omni.yaml`
(the HF `model_type` on the fishaudio checkpoints is `fish_qwen3_omni`).
Or use the convenience script:
```bash
./run_server.sh
```
## Send TTS Request
### Using curl
```bash
# Basic TTS
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav"
}' --output output.wav
```
### Voice Cloning
Provide a reference audio (URL or base64 data URL) and its transcript:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, this is a cloned voice.",
"voice": "default",
"ref_audio": "https://example.com/reference.wav",
"ref_text": "Transcript of the reference audio."
}' --output cloned.wav
```
### Using Python
```python
import httpx
# Basic TTS
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav",
},
timeout=300.0,
)
with open("output.wav", "wb") as f:
f.write(response.content)
```
### Using the CLI Client
```bash
cd examples/online_serving/fish_speech
# Basic TTS
python speech_client.py --text "Hello, how are you?"
# Voice cloning
python speech_client.py \
--text "Hello, this is a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
# Streaming PCM output
python speech_client.py --text "Hello world" --stream --output output.pcm
```
The CLI client supports:
- `--api-base`: API base URL (default: `http://localhost:8091`)
- `--model` (or `-m`): Model name (default: `fishaudio/s2-pro`)
- `--text`: Text to synthesize (required)
- `--ref-audio`: Reference audio for voice cloning (local path or URL)
- `--ref-text`: Transcript of the reference audio
- `--stream`: Enable streaming (PCM output)
- `--response-format`: Audio format: wav, mp3, flac, pcm, aac, opus (default: wav)
- `--output` (or `-o`): Output file path
## Streaming
Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "default",
"stream": true,
"response_format": "pcm"
}' --no-buffer | play -t raw -r 44100 -e signed -b 16 -c 1 -
```
**Note:** Fish Speech outputs at 44.1 kHz (unlike Qwen3-TTS which outputs at 24 kHz).
## API Parameters
Fish Speech uses the same `/v1/audio/speech` endpoint as Qwen3-TTS. See the [Speech API reference](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/speech_api/) for full parameter documentation.
Key parameters for Fish Speech:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `input` | string | **required** | Text to synthesize |
| `voice` | string | "default" | Voice name (use "default" for Fish Speech) |
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus |
| `ref_audio` | string | null | Reference audio URL or base64 data URL for voice cloning |
| `ref_text` | string | null | Transcript of reference audio (required for voice cloning) |
| `max_new_tokens` | int | 4096 | Maximum tokens to generate |
| `stream` | bool | false | Stream raw PCM chunks |
## Prerequisites
Install the `fish-speech` package for the DAC codec:
```bash
pip install fish-speech
```
## Troubleshooting
1. **No audio output**: Make sure the `fish-speech` package is installed for the DAC decoder
2. **Connection refused**: Ensure the server is running on the correct port
3. **Flashinfer version mismatch**: Set `FLASHINFER_DISABLE_VERSION_CHECK=1` if you see version warnings
4. **Out of memory**: Reduce `--gpu-memory-utilization` or use a GPU with more VRAM
examples/online_serving/omnivoice/README.md
-131
View File
@@ -1,131 +0,0 @@
# OmniVoice
## Model Overview
| Model | Description |
|-------|-------------|
| `k2-fsa/OmniVoice` | Zero-shot multilingual TTS (600+ languages) with diffusion language model (Qwen3-0.6B backbone) |
> **Note:** Requires `transformers>=5.3.0` for voice cloning (HiggsAudioV2 tokenizer). Auto voice and voice design work with `transformers>=4.57.0`.
## Launch the Server
```bash
vllm serve k2-fsa/OmniVoice \
--omni \
--port 8091 \
--trust-remote-code
```
Or use the convenience script:
```bash
./run_server.sh
```
## Send TTS Request
### Using curl
```bash
# Basic TTS (auto voice)
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav"
}' --output output.wav
```
### Using Python
```python
import httpx
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav",
},
timeout=300.0,
)
with open("output.wav", "wb") as f:
f.write(response.content)
```
### Using OpenAI SDK
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
model="k2-fsa/OmniVoice",
voice="default",
input="Hello, how are you?",
)
response.stream_to_file("output.wav")
```
### Using the CLI Client
```bash
cd examples/online_serving/omnivoice
# Basic TTS
python speech_client.py --text "Hello, how are you?"
# Specify language for improved quality
python speech_client.py --text "Bonjour, comment allez-vous?" --language French
```
The CLI client supports:
- `--api-base`: API base URL (default: `http://localhost:8091`)
- `--model` (or `-m`): Model name (default: `k2-fsa/OmniVoice`)
- `--text`: Text to synthesize (required)
- `--response-format`: Audio format: wav, mp3, flac, pcm, aac, opus (default: wav)
- `--language`: Language hint (default: Auto)
- `--output` (or `-o`): Output file path (default: `omnivoice_output.wav`)
## Inference Modes
OmniVoice supports three inference modes. Currently, **auto voice** is supported through the online Speech API. Voice cloning and voice design are available via offline inference (see `examples/offline_inference/omnivoice/`).
| Mode | Description | Online API | Offline |
|------|-------------|:----------:|:-------:|
| Auto Voice | Generate speech without reference | Yes | Yes |
| Voice Clone | Clone from reference audio + transcript | - | Yes |
| Voice Design | Control style via natural language instruction | - | Yes |
## Architecture
OmniVoice uses a single-stage diffusion pipeline:
- **Stage 0 (Generator)**: Qwen3-0.6B transformer with 32-step iterative masked unmasking and classifier-free guidance. Generates 8-codebook audio tokens from text, then decodes to 24kHz waveform via HiggsAudioV2 RVQ quantizer + DAC acoustic decoder.
## API Parameters
OmniVoice uses the standard `/v1/audio/speech` endpoint. See the [Speech API reference](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/speech_api/) for full documentation.
Key parameters:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `input` | string | **required** | Text to synthesize |
| `voice` | string | "default" | Voice name |
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus |
| `speed` | float | 1.0 | Playback speed (0.25-4.0) |
## Troubleshooting
1. **TTS model did not produce audio output**: Ensure the model is fully downloaded (`huggingface-cli download k2-fsa/OmniVoice`)
2. **Connection refused**: Make sure the server is running on the correct port
3. **Out of memory**: Reduce `--gpu-memory-utilization` (default stage config uses 0.5)
4. **Slow first request**: The model performs warmup on first inference; subsequent requests are faster
examples/online_serving/qwen3_omni/README.md
+3 -2
View File
@@ -122,8 +122,9 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--no-async-chunk
```
For the TTS counterpart (synchronous codec variant), see
[qwen3_tts README](../qwen3_tts/README.md#sync-vs-async-chunk-mode).
For the TTS counterpart (synchronous codec variant), see the Qwen3-TTS
section of the online TTS hub:
[examples/online_serving/text_to_speech/README.md#qwen3-tts](../text_to_speech/README.md#qwen3-tts).
Explicit CLI flags **override** the deploy YAML (which itself overrides the
parser defaults). If you don't pass a flag, the YAML value wins.
examples/online_serving/qwen3_tts/README.md
-455
View File
@@ -1,455 +0,0 @@
# Qwen3-TTS
## 🛠️ Installation
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
## Supported Models
| Model | Task Type | Description |
| -------------------------------------- | ----------- | ----------------------------------------------------- |
| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | CustomVoice | Predefined speaker voices with optional style control |
| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description |
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio |
| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant |
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning |
## Gradio Demo
Two interactive Gradio demos are available, both supporting all 3 task types:
| Demo | File | Transport | Streaming Quality |
| -------- | ------------------------ | ------------ | -------------------------------------------------- |
| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks |
| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) |
```bash
# Option 1: Launch server + Standard Gradio together
./run_gradio_demo.sh # CustomVoice (default)
./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign
./run_gradio_demo.sh --task-type Base # Voice cloning
# Option 2: If server is already running
python gradio_demo.py --api-base http://localhost:8000
# Option 3: FastRTC demo (gapless streaming)
pip install fastrtc
python gradio_fastrtc_demo.py --api-base http://localhost:8000
```
Then open http://localhost:7860 in your browser.
## Run examples (Qwen3-TTS)
### Launch the Server
The default deploy config is located at `vllm_omni/deploy/qwen3_tts.yaml` and is loaded automatically by the model registry — no `--deploy-config` flag needed for default use. Platform-specific deltas (NPU, ROCm, XPU) are merged in automatically from the `platforms:` block of the same YAML based on the detected runtime.
```bash
# CustomVoice model (predefined speakers)
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--omni \
--port 8091
# VoiceDesign model
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
--omni \
--port 8091
# Base model (voice cloning)
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--omni \
--port 8091
```
If you have custom stage configs file, launch the server with command below
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--stage-configs-path /path/to/stage_configs_file \
--omni \
--port 8091
```
#### Sync vs async-chunk mode
Qwen3-TTS supports both **chunked streaming** (default, lower latency) and
**synchronous end-to-end** modes from the same deploy YAML. The bundled
`qwen3_tts.yaml` ships with `async_chunk: true`; flip with `--no-async-chunk`
and the pipeline automatically dispatches to the end-to-end codec processor:
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091 \
--no-async-chunk
```
No variant YAML or extra flag is needed — the `StagePipelineConfig` on each
stage declares both processor functions and the runtime picks based on the
`async_chunk:` bool.
Alternatively, use the convenience script:
```bash
./run_server.sh # Default: CustomVoice model
./run_server.sh CustomVoice # CustomVoice model
./run_server.sh VoiceDesign # VoiceDesign model
./run_server.sh Base # Base (voice clone) model
```
### Send TTS Request
Get into the example folder
```bash
cd examples/online_serving/qwen3_tts
```
#### Send request via python
```bash
# CustomVoice: Use predefined speaker
python openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--text "你好，我是通义千问" \
--voice vivian \
--language Chinese
# CustomVoice with style instruction
python openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--text "今天天气真好" \
--voice ryan \
--instructions "用开心的语气说"
# VoiceDesign: Describe the voice style
python openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
--task-type VoiceDesign \
--text "哥哥，你回来啦" \
--instructions "体现撒娇稚嫩的萝莉女声，音调偏高"
# Base: Voice cloning
python openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--task-type Base \
--text "Hello, this is a cloned voice" \
--ref-audio /path/to/reference.wav \
--ref-text "Original transcript of the reference audio"
```
The Python client supports the following command-line arguments:
- `--api-base`: API base URL (default: `http://localhost:8091`)
- `--model` (or `-m`): Model name/path (default: `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`)
- `--task-type` (or `-t`): TTS task type. Options: `CustomVoice`, `VoiceDesign`, `Base`
- `--text`: Text to synthesize (required)
- `--voice`: Speaker/voice name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc.
- `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`
- `--instructions`: Voice style/emotion instructions
- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server.
- `--ref-text`: Reference audio transcript for voice cloning (Base task).
- `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus`
- `--output` (or `-o`): Output audio file path (default: `tts_output.wav`)
#### Send request via curl
```bash
# Simple TTS request
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English"
}' --output output.wav
# With style instruction
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "I am so excited!",
"voice": "vivian",
"instructions": "Speak with great enthusiasm"
}' --output excited.wav
# List available voices in CustomVoice models
curl http://localhost:8091/v1/audio/voices
```
### Using OpenAI SDK
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
voice="vivian",
input="Hello, how are you?",
)
response.stream_to_file("output.wav")
```
### Using Python httpx
```python
import httpx
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English",
},
timeout=300.0,
)
with open("output.wav", "wb") as f:
f.write(response.content)
```
## API Reference
### Voices Endpoint
#### GET /v1/audio/voices
List all available voices/speakers from the loaded model, including both built-in model voices and uploaded custom voices.
**Response Example:**
```json
{
"voices": ["vivian", "ryan", "custom_voice_1"],
"uploaded_voices": [
{
"name": "custom_voice_1",
"consent": "user_consent_id",
"created_at": 1738660000,
"file_size": 1024000,
"mime_type": "audio/wav"
}
]
}
```
#### POST /v1/audio/voices
Upload a new voice sample for voice cloning in Base task TTS requests.
**Form Parameters:**
- `audio_sample` (required): Audio file (max 10MB, supported formats: wav, mp3, flac, ogg, aac, webm, mp4)
- `consent` (required): Consent recording ID
- `name` (required): Name for the new voice
- `ref_text` (optional): Transcript of the audio. Enables in-context voice cloning (higher quality).
- `speaker_description` (optional): Free-form description of the voice (e.g. "warm narrator", "energetic presenter").
**Response Example:**
```json
{
"success": true,
"voice": {
"name": "custom_voice_1",
"consent": "user_consent_id",
"created_at": 1738660000,
"mime_type": "audio/wav",
"file_size": 1024000,
"ref_text": "The exact transcript of the audio sample.",
"speaker_description": "warm narrator"
}
}
```
Fields `ref_text` and `speaker_description` are omitted when not provided at upload time.
**Usage Example:**
```bash
curl -X POST http://localhost:8000/v1/audio/voices \
-F "audio_sample=@/path/to/voice_sample.wav" \
-F "consent=user_consent_id" \
-F "name=custom_voice_1" \
-F "ref_text=The exact transcript of the audio sample." \
-F "speaker_description=warm narrator"
```
### Endpoint
```
POST /v1/audio/speech
Content-Type: application/json
```
This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters.
### Request Body
```json
{
"input": "Text to synthesize",
"voice": "vivian",
"response_format": "wav",
"task_type": "CustomVoice",
"language": "Auto",
"instructions": "Optional style instructions",
"ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning",
"ref_text": "Reference audio transcript",
"x_vector_only_mode": false,
"max_new_tokens": 2048
}
```
> **Note:** The `model` field is optional when serving a single model, as the server already knows which model is loaded.
### Response
Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/wav`).
## Parameters
### OpenAI Standard Parameters
| Parameter | Type | Default | Description |
| ----------------- | ------ | -------------- | ----------------------------------------------------------- |
| `input` | string | **required** | Text to synthesize |
| `model` | string | server's model | Model to use (optional, should match server if specified) |
| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) |
| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus |
| `speed` | float | 1.0 | Playback speed (0.25-4.0, not supported with `stream=true`) |
### vLLM-Omni Extension Parameters
| Parameter | Type | Default | Description |
| ---------------------------- | ------ | ------------- | -------------------------------------------------------------------------------------------------------------------- |
| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base |
| `language` | string | "Auto" | Language (see supported languages below) |
| `instructions` | string | "" | Voice style/emotion instructions |
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. |
| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
**Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
### Voice Clone Parameters (Base task)
| Parameter | Type | Required | Description |
| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------- |
| `ref_audio` | string | **Yes** | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) |
| `ref_text` | string | No | Transcript of reference audio (for ICL mode) |
| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) |
## Streaming
Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded
(one chunk per Code2Wav window, default 25 frames; configurable in the stage config):
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English",
"stream": true,
"response_format": "pcm"
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
```
**Constraints:**
- `stream=true` requires `response_format="pcm"` (raw 16-bit signed PCM, 24 kHz mono).
- `speed` adjustment is not supported when streaming.
- Requires the server stage config to have `async_chunk: true` (default in `qwen3_tts.yaml`).
## Streaming Text Input (WebSocket)
The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally, buffers and splits it at sentence boundaries, and generates audio per sentence.
When `stream_audio=true`, each sentence is emitted as `audio.start`, one or more binary PCM frames, and `audio.done`.
### Quick Start
```bash
python streaming_speech_client.py \
--text "Hello world. How are you? I am fine."
python streaming_speech_client.py \
--text "Hello world. How are you? I am fine." \
--simulate-stt --stt-delay 0.1
```
### WebSocket Protocol
Client -> Server:
```jsonc
{"type": "session.config", "voice": "Vivian", "task_type": "CustomVoice", "language": "Auto", "split_granularity": "sentence", "stream_audio": true, "response_format": "pcm"}
{"type": "input.text", "text": "Hello, how are you? "}
{"type": "input.done"}
```
Server -> Client:
```jsonc
{"type": "audio.start", "sentence_index": 0, "sentence_text": "Hello, how are you?", "format": "pcm", "sample_rate": 24000}
// binary PCM frame(s)
{"type": "audio.done", "sentence_index": 0, "total_bytes": 96000, "error": false}
{"type": "session.done", "total_sentences": 1}
```
## Choosing an Execution Backend: Uniproc vs Multiprocessing
Qwen3-TTS stage configs support two execution backends controlled by the
`distributed_executor_backend` engine arg. The performance tradeoff between
them is **both hardware- and task-dependent**, so there is no single best
default (see [#2603](https://github.com/vllm-project/vllm-omni/issues/2603),
[#2604](https://github.com/vllm-project/vllm-omni/pull/2604) for the full
investigation).
| Backend | Stage config setting | Behaviour |
| ------- | -------------------- | --------- |
| **Uniproc** (default, world_size=1) | `distributed_executor_backend` omitted | Both stages run inside the orchestrator process. Avoids IPC serialisation, D2H copies, and msgpack overhead between stages. |
| **Multiprocessing** | `distributed_executor_backend: "mp"` | Each stage runs in its own subprocess. The Talker can continue decoding while Code2Wav runs the vocoder in parallel, improving pipeline utilisation under concurrency. |
> **Note:** When `distributed_executor_backend` is omitted and `world_size=1`,
> vLLM [automatically uses the uniproc executor](https://github.com/vllm-project/vllm/blob/main/vllm/config/parallel.py#L825).
> When `world_size > 1`, it defaults to `mp`.
### When uniproc wins
The uniproc path eliminates inter-process data transfer (D2H copies,
msgpack serialisation/deserialisation, tensor detaching). This matters most
when per-request processing is heavy relative to autoregressive decode.
The Base cloning task involves reference-audio encoding on every request, making IPC
overhead a larger fraction of total cost. Qwen3-Omni shows a similar pattern.
### When multiprocessing (`mp`) wins
For lighter per-request workloads, process-level parallelism between the
Talker and Code2Wav stages dominates.
CustomVoice is lighter per-request (no reference audio encoding), so the
process-level parallelism of `mp` outweighs its serialisation cost at
concurrency ≥ 4.
### How to switch
To use the uniproc executor on a single-GPU setup, pass the
`qwen3_tts_uniproc.yaml` stage config:
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--omni \
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_uniproc.yaml \
--port 8091
```
## Limitations
- **Single request**: Batch processing is not yet optimized for online serving.
## Troubleshooting
1. **TTS model did not produce audio output**: Ensure you're using the correct model variant for your task type (CustomVoice task → CustomVoice model, etc.)
2. **Connection refused**: Make sure the server is running on the correct port
3. **Out of memory**: Use smaller model variant (`Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice`) or reduce `--gpu-memory-utilization`
4. **Unsupported speaker**: Use `/v1/audio/voices` to list available voices for the loaded model
5. **Voice clone fails**: Ensure you're using the Base model variant for voice cloning
examples/online_serving/text_to_speech/README.md
+399
View File
@@ -0,0 +1,399 @@
# Text-To-Speech (Online Serving)
vLLM-Omni exposes TTS models through the OpenAI-compatible
[`POST /v1/audio/speech`](../../../docs/serving/speech_api.md) endpoint,
launched with `vllm serve <model> --omni`. Each TTS model has its own
subdirectory containing client snippets, gradio demos, and helper
scripts; this README is the single doc entry point for all of them.
For offline inference, see [`examples/offline_inference/text_to_speech`](../../offline_inference/text_to_speech/README.md).
For the full list of supported architectures across all modalities, see
[Supported Models](../../../docs/models/supported_models.md).
## Supported Models
| Model | HuggingFace repo | Voice cloning | Streaming | Voice presets / upload | Gradio demo |
|---|---|---|---|---|---|
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | ✓ (Base) | ✓ (PCM + WebSocket) | ✓ (presets + `/v1/audio/voices` upload) | ✓ (standard + FastRTC) |
| VoxCPM | local model dir | ✓ | ✓ (PCM stream) | — | — |
| VoxCPM2 | `openbmb/VoxCPM2` | ✓ | ✓ (AudioWorklet via gradio) | — | ✓ |
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✓ (gated upstream) | ✓ | ✓ (presets) | ✓ |
CosyVoice3 is intentionally absent: no online example exists for it yet. See its [offline section](../../offline_inference/text_to_speech/README.md#cosyvoice3) instead.
## Common Quick Start
Launch the server (defaults shown — adjust `--port`, `--gpu-memory-utilization`, etc. as needed):
```bash
vllm serve <hf-repo-or-local-path> --omni --port 8091
```
Send a TTS request via curl:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav"
}' --output output.wav
```
Or via Python httpx:
```python
import httpx
response = httpx.post(
"http://localhost:8091/v1/audio/speech",
json={
"input": "Hello, how are you?",
"voice": "default",
"response_format": "wav",
},
timeout=300.0,
)
open("output.wav", "wb").write(response.content)
```
Or via the OpenAI SDK:
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8091/v1", api_key="none")
response = client.audio.speech.create(
model="<hf-repo>",
voice="default",
input="Hello, how are you?",
)
response.stream_to_file("output.wav")
```
Streaming PCM output (where supported) — set `stream=true` with `response_format="pcm"`:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "default",
"stream": true,
"response_format": "pcm"
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
```
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others).
For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](../../../docs/serving/speech_api.md).
---
## Fish Speech S2 Pro
4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec.
### Prerequisites
```bash
pip install fish-speech
```
### Launch
```bash
vllm serve fishaudio/s2-pro --omni --port 8091
# or:
./fish_speech/run_server.sh
```
The deploy config auto-loads from `vllm_omni/deploy/fish_qwen3_omni.yaml` (the HF `model_type` on the fishaudio checkpoint is `fish_qwen3_omni`).
### Voice cloning
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, this is a cloned voice.",
"voice": "default",
"ref_audio": "https://example.com/reference.wav",
"ref_text": "Transcript of the reference audio."
}' --output cloned.wav
```
### CLI client
```bash
cd examples/online_serving/text_to_speech/fish_speech
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Hello world" --stream --output output.pcm
```
### Gradio demo
```bash
./fish_speech/run_gradio_demo.sh # launches server + Gradio
python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server already running
```
### Notes
- Output: 44.1 kHz mono.
- Streaming PCM player command must use `-r 44100`.
---
## OmniVoice
Zero-shot multilingual TTS (600+ languages). Online serving currently exposes **auto voice** only; voice cloning and voice design are available offline.
### Prerequisites
```bash
huggingface-cli download k2-fsa/OmniVoice
```
Voice cloning (offline) needs `transformers>=5.3.0`; auto voice works with `transformers>=4.57.0`.
### Launch
```bash
vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code
# or:
./omnivoice/run_server.sh
```
### CLI client
```bash
cd examples/online_serving/text_to_speech/omnivoice
python speech_client.py --text "Hello, how are you?"
python speech_client.py --text "Bonjour, comment allez-vous?" --language French
```
The client supports `--api-base`, `--model`, `--text`, `--response-format`, `--language`, `--output`.
### Notes
- Voice cloning and voice design require offline inference; see the [offline OmniVoice section](../../offline_inference/text_to_speech/README.md#omnivoice).
---
## Qwen3-TTS
Three model variants exposed via separate checkpoints:
| Variant | HF repo | Use |
|---|---|---|
| CustomVoice | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | Predefined speakers (`vivian`, `ryan`, …) with optional style instructions |
| VoiceDesign | `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | Natural-language voice style description |
| Base | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Voice cloning from a reference audio |
Each variant ships smaller `0.6B` companions where available.
### Launch
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
# or:
./qwen3_tts/run_server.sh # default: CustomVoice
./qwen3_tts/run_server.sh VoiceDesign
./qwen3_tts/run_server.sh Base
```
### Choosing an executor backend (uniproc vs mp)
Stage configs ship with the chunked-streaming default. To use the uniproc executor (lower IPC overhead for the Base cloning task), pass `--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_uniproc.yaml`. See [#2603](https://github.com/vllm-project/vllm-omni/issues/2603) and [#2604](https://github.com/vllm-project/vllm-omni/pull/2604) for the full investigation.
To opt out of chunked streaming, pass `--no-async-chunk` instead — the pipeline auto-dispatches to the end-to-end codec processor.
### Sending requests
```bash
# CustomVoice with a predefined speaker
python qwen3_tts/openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--text "今天天气真好" \
--speaker ryan \
--instructions "用开心的语气说"
# VoiceDesign with a style description
python qwen3_tts/openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
--task-type VoiceDesign \
--text "哥哥，你回来啦" \
--instructions "体现撒娇稚嫩的萝莉女声，音调偏高"
# Base voice cloning
python qwen3_tts/openai_speech_client.py \
--model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
--task-type Base \
--text "Hello, this is a cloned voice" \
--ref-audio /path/to/reference.wav \
--ref-text "Original transcript of the reference audio"
```
### Voices endpoint
List available voices, or upload a custom one for Base cloning:
```bash
# List
curl http://localhost:8091/v1/audio/voices
# Upload
curl -X POST http://localhost:8091/v1/audio/voices \
-F "audio_sample=@/path/to/voice_sample.wav" \
-F "consent=user_consent_id" \
-F "name=custom_voice_1" \
-F "ref_text=The exact transcript of the audio sample." \
-F "speaker_description=warm narrator"
```
Uploaded voices are then usable as `voice="custom_voice_1"` on subsequent requests.
### Streaming PCM
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, how are you?",
"voice": "vivian",
"language": "English",
"stream": true,
"response_format": "pcm"
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
```
Streaming requires `response_format="pcm"` and `async_chunk: true` on the stage config (default in `qwen3_tts.yaml`). `speed` is not supported when streaming.
### Streaming WebSocket
The `/v1/audio/speech/stream` endpoint accepts text incrementally, splits it at sentence boundaries, and emits one PCM stream per sentence:
```bash
python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you? I am fine."
python qwen3_tts/streaming_speech_client.py --text "..." --simulate-stt --stt-delay 0.1
```
### Gradio demos
```bash
./qwen3_tts/run_gradio_demo.sh # CustomVoice (default)
./qwen3_tts/run_gradio_demo.sh --task-type VoiceDesign
./qwen3_tts/run_gradio_demo.sh --task-type Base
```
### Speaker embedding interpolation
`qwen3_tts/speaker_embedding_interpolation.py` blends two predefined speakers' embeddings to produce intermediate voices. See the script for usage.
### Batch client
`qwen3_tts/batch_speech_client.py` issues many concurrent requests for throughput measurement.
### Notes
- Base voice cloning has uniproc-vs-mp tradeoffs depending on per-request reference audio cost; see the executor-backend section above.
- `vllm_omni/deploy/qwen3_tts.yaml` is the default deploy config (loaded by HF `model_type`); per-stage runtime overrides are available via `--stage-N-<field> <value>`.
---
## VoxCPM
Split-stage TTS at 24 kHz.
### Prerequisites
```bash
pip install voxcpm
# or use a local source tree:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src
```
If the native VoxCPM `config.json` lacks HF `model_type`, set up an HF-compatible config dir:
```bash
export VOXCPM_MODEL=/path/to/voxcpm-model
export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config
mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"
cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json"
cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true
python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)'
```
### Launch
```bash
export VOXCPM_MODEL=/path/to/voxcpm-model
./voxcpm/run_server.sh # async-chunk streaming (default)
./voxcpm/run_server.sh sync # non-streaming
```
Or directly:
```bash
vllm serve "$VOXCPM_MODEL" \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
--trust-remote-code --enforce-eager --omni --port 8091
```
### Sending requests
```bash
# Basic TTS
python voxcpm/openai_speech_client.py \
--model "$VOXCPM_MODEL" \
--text "This is a VoxCPM online text-to-speech example."
# Voice cloning
python voxcpm/openai_speech_client.py \
--model "$VOXCPM_MODEL" \
--text "This sentence is synthesized with a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript spoken in reference.wav."
# Streaming PCM
python voxcpm/openai_speech_client.py \
--model "$VOXCPM_MODEL" \
--text "This is a streaming VoxCPM request." \
--stream --output voxcpm_stream.pcm
```
### Notes
- `voxcpm.yaml` for one-shot decode; `voxcpm_async_chunk.yaml` for single-request streaming. Do not use the async-chunk config for concurrent requests or `/v1/audio/speech/batch`.
- Generic TTS fields not supported by VoxCPM: `voice`, `instructions`, `language`, `speaker_embedding`, `x_vector_only_mode`.
- For benchmark measurement, see [`benchmarks/voxcpm`](../../../benchmarks/voxcpm/README.md).
---
## VoxCPM2
Single-stage native AR TTS at 48 kHz.
### Launch
```bash
vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000
```
Deploy config auto-loads from `vllm_omni/deploy/voxcpm2.yaml`. Pass `--deploy-config <path>` to override or `--stage-N-<field> <value>` for per-stage runtime tweaks.
### Sending requests
```bash
# Zero-shot synthesis
python voxcpm2/openai_speech_client.py --text "Hello, this is VoxCPM2."
# Voice cloning
python voxcpm2/openai_speech_client.py \
--text "This should sound like the reference speaker." \
--ref-audio /path/to/reference.wav
```
The `ref_audio` field accepts local file paths (auto-base64), HTTP URLs, or `data:audio/wav;base64,...` data URIs.
### Gradio demo (gapless streaming via AudioWorklet)
```bash
python voxcpm2/gradio_demo.py
```
Uses an AudioWorklet-based player adapted from the Qwen3-TTS demo for gap-free playback. Audio is streamed from the OpenAI Speech endpoint with `stream=true`.
---
## Voxtral TTS
Voxtral-4B-TTS (Mistral). Uses the `mistral_common` `SpeechRequest` protocol; voice presets are model-specific.
### Prerequisites
Latest `mistral_common` with `SpeechRequest` support:
```bash
pip install -e /path/to/mistral-common # or upgrade from PyPI when available
```
### Launch
```bash
vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091
```
Deploy config auto-loads from `vllm_omni/deploy/voxtral_tts.yaml`.
### Gradio demo
```bash
python voxtral_tts/gradio_demo.py
```
The demo handles voice-preset selection and reference-audio upload. `voxtral_tts/text_preprocess.py` provides the text-normalization helpers used by the demo (also available for other clients).
### Notes
- Voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`).
- Voice cloning is gated upstream and may require a recent `mistral_common`.
- A standalone CLI client is not yet shipped; the gradio demo is the canonical reference for now.
examples/online_serving/fish_speech/gradio_demo.py → examples/online_serving/text_to_speech/fish_speech/gradio_demo.py
View File
File renamed without changes.
examples/online_serving/fish_speech/run_gradio_demo.sh → examples/online_serving/text_to_speech/fish_speech/run_gradio_demo.sh
View File
File renamed without changes.
examples/online_serving/fish_speech/run_server.sh → examples/online_serving/text_to_speech/fish_speech/run_server.sh
View File
File renamed without changes.
examples/online_serving/fish_speech/speech_client.py → examples/online_serving/text_to_speech/fish_speech/speech_client.py
View File
File renamed without changes.
examples/online_serving/omnivoice/run_server.sh → examples/online_serving/text_to_speech/omnivoice/run_server.sh
View File
File renamed without changes.
examples/online_serving/omnivoice/speech_client.py → examples/online_serving/text_to_speech/omnivoice/speech_client.py
View File
File renamed without changes.
examples/online_serving/qwen3_tts/batch_speech_client.py → examples/online_serving/text_to_speech/qwen3_tts/batch_speech_client.py
View File
File renamed without changes.
examples/online_serving/qwen3_tts/gradio_demo.py → examples/online_serving/text_to_speech/qwen3_tts/gradio_demo.py
View File
File renamed without changes.
examples/online_serving/qwen3_tts/openai_speech_client.py → examples/online_serving/text_to_speech/qwen3_tts/openai_speech_client.py
View File
File renamed without changes.
examples/online_serving/qwen3_tts/run_gradio_demo.sh → examples/online_serving/text_to_speech/qwen3_tts/run_gradio_demo.sh
View File
File renamed without changes.
examples/online_serving/qwen3_tts/run_server.sh → examples/online_serving/text_to_speech/qwen3_tts/run_server.sh
View File
File renamed without changes.
examples/online_serving/qwen3_tts/speaker_embedding_interpolation.py → examples/online_serving/text_to_speech/qwen3_tts/speaker_embedding_interpolation.py
View File
File renamed without changes.
examples/online_serving/qwen3_tts/streaming_speech_client.py → examples/online_serving/text_to_speech/qwen3_tts/streaming_speech_client.py
View File
File renamed without changes.
examples/online_serving/qwen3_tts/tts_common.py → examples/online_serving/text_to_speech/qwen3_tts/tts_common.py
View File
File renamed without changes.
examples/online_serving/voxcpm/openai_speech_client.py → examples/online_serving/text_to_speech/voxcpm/openai_speech_client.py
View File
File renamed without changes.
examples/online_serving/voxcpm/run_server.sh → examples/online_serving/text_to_speech/voxcpm/run_server.sh
View File
File renamed without changes.
examples/online_serving/voxcpm2/gradio_demo.py → examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py
View File
File renamed without changes.
examples/online_serving/voxcpm2/openai_speech_client.py → examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py
View File
File renamed without changes.
examples/online_serving/voxtral_tts/gradio_demo.py → examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py
+1 -1
View File
@@ -5,7 +5,7 @@
Example use case:
python examples/online_serving/voxtral_tts/gradio_demo.py --host slurm-199-077 --port 8000
python examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py --host slurm-199-077 --port 8000
"""
examples/online_serving/voxtral_tts/text_preprocess.py → examples/online_serving/text_to_speech/voxtral_tts/text_preprocess.py
View File
File renamed without changes.
examples/online_serving/voxcpm/README.md
-166
View File
@@ -1,166 +0,0 @@
# VoxCPM
## Prerequisites
Install VoxCPM in one of these ways:
```bash
pip install voxcpm
```
or point vLLM-Omni to a local VoxCPM source tree:
```bash
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src
```
If the native VoxCPM `config.json` lacks HF metadata such as `model_type`,
prepare a persistent HF-compatible config directory and export:
```bash
export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config
mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"
cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json"
cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true
python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)'
```
The VoxCPM stage configs read `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH` directly. The `python3 -c` form above avoids heredoc/indentation issues in interactive shells.
## Launch the Server
Use the async-chunk stage config by default:
```bash
export VOXCPM_MODEL=/path/to/voxcpm-model
cd examples/online_serving/voxcpm
./run_server.sh
```
Use the non-streaming stage config:
```bash
./run_server.sh sync
```
You can also launch the server directly:
```bash
vllm serve "$VOXCPM_MODEL" \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
--trust-remote-code \
--enforce-eager \
--omni \
--port 8091
```
## Send Requests
### Basic text-to-speech
```bash
python openai_speech_client.py \
--model "$VOXCPM_MODEL" \
--text "This is a VoxCPM online text-to-speech example."
```
### Voice cloning
```bash
python openai_speech_client.py \
--model "$VOXCPM_MODEL" \
--text "This sentence is synthesized with a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript spoken in reference.wav."
```
`ref_text` must be the real transcript of the reference audio. Placeholder text or mismatched text will usually degrade quality badly.
### Streaming PCM output
```bash
python openai_speech_client.py \
--model "$VOXCPM_MODEL" \
--text "This is a streaming VoxCPM request." \
--stream \
--output voxcpm_stream.pcm
```
### Using curl
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "OpenBMB/VoxCPM1.5",
"input": "Hello from VoxCPM online serving.",
"response_format": "wav"
}' --output output.wav
```
Voice cloning:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "OpenBMB/VoxCPM1.5",
"input": "This sentence uses a cloned voice.",
"ref_audio": "https://example.com/reference.wav",
"ref_text": "The exact transcript spoken in the reference audio.",
"response_format": "wav"
}' --output cloned.wav
```
Streaming PCM:
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "OpenBMB/VoxCPM1.5",
"input": "This is a streaming VoxCPM request.",
"stream": true,
"response_format": "pcm"
}' --output output.pcm
```
## Supported Request Shape
VoxCPM online serving currently supports:
- plain text-to-speech
- voice cloning with `ref_audio` + `ref_text`
- `stream=true` with `response_format=pcm` or `wav`
VoxCPM online serving does not use these generic TTS fields:
- `voice`
- `instructions`
- `language`
- `speaker_embedding`
- `x_vector_only_mode`
## Streaming vs Non-Streaming
- `voxcpm_async_chunk.yaml` enables async-chunk streaming and is best for single-request streaming latency.
- `voxcpm.yaml` performs one-shot latent generation then VAE decode.
Like native VoxCPM, the async streaming path should be treated as single-request. If you need stable throughput benchmarking, prefer `voxcpm.yaml`.
Do not use `voxcpm_async_chunk.yaml` for concurrent online streaming or `/v1/audio/speech/batch`. For multiple requests, prefer `voxcpm.yaml`.
## Benchmark
The serving benchmark reports TTFP and RTF:
```bash
python benchmarks/voxcpm/vllm_omni/bench_tts_serve.py \
--host 127.0.0.1 \
--port 8091 \
--num-prompts 10 \
--max-concurrency 1 \
--result-dir /tmp/voxcpm_bench
```
For the async-chunk server, keep `--max-concurrency 1`.
examples/online_serving/voxcpm2/README.md
-43
View File
@@ -1,43 +0,0 @@
# VoxCPM2 Online Serving
Serve VoxCPM2 TTS via the OpenAI-compatible `/v1/audio/speech` endpoint.
## Start the Server
```bash
vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000
```
The deploy config is auto-loaded from `vllm_omni/deploy/voxcpm2.yaml`. Pass
`--deploy-config <path>` to override, or `--stage-N-<field> <value>` (e.g.
`--stage-0-max-num-seqs 8`) for per-stage runtime overrides.
## Zero-shot Synthesis
```bash
python openai_speech_client.py --text "Hello, this is VoxCPM2."
```
Or with curl:
```bash
curl -X POST http://localhost:8000/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"model": "voxcpm2", "input": "Hello, this is VoxCPM2.", "voice": "default"}' \
--output output.wav
```
## Voice Cloning
Clone a speaker's voice using a reference audio file:
```bash
python openai_speech_client.py \
--text "This should sound like the reference speaker." \
--ref-audio /path/to/reference.wav
```
The `--ref-audio` parameter accepts:
- Local file path (auto-encoded to base64)
- URL (`https://...`)
- Base64 data URI (`data:audio/wav;base64,...`)
tests/e2e/offline_inference/test_voxtral_tts.py
+1 -1
View File
@@ -7,7 +7,7 @@ End-to-end offline inference tests for Voxtral TTS model.
Tests both synchronous (Omni) and async streaming (AsyncOmni) paths.
Equivalent to running:
python3 examples/offline_inference/voxtral_tts/end2end.py \
python3 examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--model mistralai/Voxtral-4B-TTS-2603 \
--voice casual_female \
--text "Hello, how are you?" \
tests/examples/offline_inference/test_qwen3_tts_estimator.py
+4 -2
View File
@@ -2,7 +2,7 @@
Offline inference regression tests: Qwen3-TTS _estimate_prompt_len.
Covers the four ref_audio input forms handled by the estimator in
examples/offline_inference/qwen3_tts/end2end.py:
examples/offline_inference/text_to_speech/qwen3_tts/end2end.py:
- bare local path
- file:// URI
- http:// URL
@@ -20,7 +20,9 @@ from unittest.mock import MagicMock
import numpy as np
import pytest
_examples_dir = str(Path(__file__).parent.parent.parent.parent / "examples" / "offline_inference" / "qwen3_tts")
_examples_dir = str(
Path(__file__).parent.parent.parent.parent / "examples" / "offline_inference" / "text_to_speech" / "qwen3_tts"
)
sys.path.insert(0, _examples_dir)
from end2end import _estimate_prompt_len # noqa: E402
tests/examples/test_slerp_interpolation.py
+3 -1
View File
@@ -9,7 +9,9 @@ import numpy as np
import pytest
# Import the slerp function from the example script.
_examples_dir = str(Path(__file__).parent.parent.parent / "examples" / "online_serving" / "qwen3_tts")
_examples_dir = str(
Path(__file__).parent.parent.parent / "examples" / "online_serving" / "text_to_speech" / "qwen3_tts"
)
sys.path.insert(0, _examples_dir)
from speaker_embedding_interpolation import slerp # noqa: E402
tests/helpers/runtime.py
+1 -1
View File
@@ -1240,7 +1240,7 @@ class OmniRunner:
if isinstance(prompts, str):
prompts = [prompts]
# Qwen-TTS: follow examples/offline_inference/qwen3_tts/end2end.py style.
# Qwen-TTS: follow examples/offline_inference/text_to_speech/qwen3_tts/end2end.py style.
# Stage 0 expects token placeholders + additional_information (text/speaker/task_type/...),
# and Talker replaces embeddings in preprocess based on additional_information only.
is_tts_model = "Qwen3-TTS" in self.model_name or "qwen3_tts" in self.model_name.lower()
tests/model_executor/models/voxtral_tts/test_text_preprocess.py
+7 -2
View File
@@ -3,7 +3,7 @@
"""Unit tests for the Voxtral TTS demo text preprocessing helper.
These tests intentionally cover only behavior ported into
examples/online_serving/voxtral_tts/text_preprocess.py.
examples/online_serving/text_to_speech/voxtral_tts/text_preprocess.py.
"""
import importlib.util
@@ -13,7 +13,12 @@ from pathlib import Path
import pytest
_HELPER_PATH = (
Path(__file__).resolve().parents[4] / "examples" / "online_serving" / "voxtral_tts" / "text_preprocess.py"
Path(__file__).resolve().parents[4]
/ "examples"
/ "online_serving"
/ "text_to_speech"
/ "voxtral_tts"
/ "text_preprocess.py"
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]