anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Docs] Consolidate per-model TTS examples into a single hub (#3234)

Browse Source 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
Yueqian Lin authored and GitHub committed 2026-05-05 08:41:29 +12:00
1 parent e1b30061b6
commit 703e31fc47
58 files changed
+1694 -2675
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/examples/offline_inference/cosyvoice3.md
-81
View File
@@ -1,81 +0,0 @@
# CosyVoice3
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/cosyvoice3>.
## Setup
Install dependencies:
```
uv pip install -e .
```
> **Note:** This includes required libraries such as `soundfile`,
> `onnxruntime`, `x-transformers`, and `einops` via
> `requirements/common.txt` and platform-specific requirements files.
Download the model snapshot:
```
from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
```
Add `config.json` in `pretrained_models/Fun-CosyVoice3-0.5B/`:
```json
{
"model_type": "cosyvoice3",
"architectures": [
"CosyVoice3Model"
]
}
```
> **Why `config.json` is required:**
> `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers a class mapping.
> The loader still needs `model_type: "cosyvoice3"` from `config.json` to select that class.
> If no `config.json` is present, model type cannot be inferred automatically.
> If your downloaded checkpoint already includes a valid `config.json` with
> `model_type: "cosyvoice3"`, this manual step can be skipped.
Run the offline verification script:
```
python examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py \
--model pretrained_models/Fun-CosyVoice3-0.5B \
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN
```
## Implementation Overview
CosyVoice3 runs as a **2-stage Omni pipeline**.
- **Stage 0** (`talker`) converts text + prompt audio to speech tokens.
- **Stage 1** (`code2wav`) converts speech tokens + prompt features to waveform via flow matching and HiFiGAN.
Key components live in `vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py`.
- `CosyVoice3MultiModalProcessor` builds the multimodal inputs:
- Tokenizes `prompt` and `prompt_text`.
- Extracts speech tokens and mel features from the prompt audio.
- Extracts a speaker embedding.
- `CosyVoice3Model` implements both stages:
- Stage 0 uses `CosyVoice3LM` and outputs speech tokens + conditioning features.
- Stage 1 runs the flow model (DiT-based CFM) and HiFiGAN to synthesize waveform.
Pipeline topology lives in `vllm_omni/model_executor/models/cosyvoice3/pipeline.py`;
runtime tunables (batch size, memory limits, sampling) live in
`vllm_omni/deploy/cosyvoice3.yaml`. The deploy config auto-loads by
HF `model_type` and defaults to `async_chunk: true` (shared-memory
streaming). Pass `--no-async-chunk` on `vllm serve` to switch to the
legacy sync path where stage 1 runs `text2flow` over the full
speech-token sequence.
- Stage 0 emits latent speech tokens.
- Stage 1 consumes them via `sync_process_input_func` (sync mode) or the
shared-memory connector (async-chunk mode) and outputs audio.
## Example materials
??? abstract "verify_e2e_cosyvoice.py"
``````py
--8<-- "examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py"
``````
docs/user_guide/examples/offline_inference/fish_speech.md
-85
View File
@@ -1,85 +0,0 @@
# Fish Speech S2 Pro
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/fish_speech>.
This directory contains an offline demo for running Fish Speech S2 Pro (`fishaudio/s2-pro`) with vLLM Omni. It supports text-only synthesis and voice cloning with reference audio.
## Model Overview
[Fish Speech S2 Pro](https://huggingface.co/fishaudio/s2-pro) is a 4B dual-AR text-to-speech model by FishAudio. It uses:
- **Slow AR**: Generates semantic tokens autoregressively (Qwen3-based backbone)
- **Fast AR**: Predicts residual codebook tokens from semantic tokens
- **DAC Decoder**: Converts codec codes to 44.1 kHz audio waveform
The model produces high-quality speech with voice cloning capabilities.
## Prerequisites
Install the `fish-speech` package for the DAC codec:
```bash
pip install fish-speech
```
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
## Quick Start
### Text-Only Synthesis
```bash
python end2end.py --text "Hello, this is a test of the Fish Speech text to speech system."
```
Generated audio files are saved to `output_audio/` by default.
### Voice Cloning
Provide a reference audio file and its transcript to clone a voice:
```bash
python end2end.py \
--text "Hello, this is a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
## Streaming Mode
Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config):
```bash
python end2end.py --text "Hello, this is a test." --streaming
```
Each DAC decoder chunk is logged as it arrives. The final WAV file is written once generation completes.
## Configuration
| Argument | Default | Description |
|----------|---------|-------------|
| `--text` | `"Hello, this is a test..."` | Text to synthesize |
| `--ref-audio` | None | Path to reference audio for voice cloning |
| `--ref-text` | None | Transcript of the reference audio |
| `--model` | `fishaudio/s2-pro` | HuggingFace model path |
| `--stage-configs-path` | `fish_speech_s2_pro.yaml` | Path to stage configs YAML |
| `--output-dir` | `output_audio` | Output directory for WAV files |
| `--streaming` | False | Enable streaming via AsyncOmni |
| `--stage-init-timeout` | 600 | Stage initialization timeout (seconds) |
## Notes
- Output audio is 44.1 kHz mono WAV.
- The DAC codec weights (`codec.pth`) are loaded lazily from the model directory.
- Voice cloning encodes the reference audio using the DAC codec to extract semantic codes, then prepends them as a system message.
## Example materials
??? abstract "end2end.py"
``````py
--8<-- "examples/offline_inference/fish_speech/end2end.py"
``````
docs/user_guide/examples/offline_inference/qwen3_tts.md
-172
View File
@@ -1,172 +0,0 @@
# Qwen3-TTS
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen3_tts>.
This directory contains an offline demo for running Qwen3 TTS models with vLLM Omni. It builds task-specific inputs and generates WAV files locally.
## Model Overview
Qwen3 TTS provides multiple task variants for speech generation:
- **CustomVoice**: Generate speech with a known speaker identity (speaker ID) and optional instruction.
- **VoiceDesign**: Generate speech from text plus a descriptive instruction that designs a new voice.
- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. The `ref_audio` field accepts a local file path, HTTP URL, or base64 data URL.
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
### ROCm Dependencies
You will need to install the dependency `onnxruntime-rocm`.
```
pip uninstall onnxruntime # should be removed before we can install onnxruntime-rocm
pip install onnxruntime-rocm
```
## Quick Start
Run a single sample for a task:
```
python end2end.py --query-type CustomVoice
```
Generated audio files are saved to `output_audio/` by default.
## Task Usage
### CustomVoice
Single sample:
```
python end2end.py --query-type CustomVoice
```
Batch sample (multiple prompts in one run):
```
python end2end.py --query-type CustomVoice --use-batch-sample
```
### VoiceDesign
Single sample:
```
python end2end.py --query-type VoiceDesign
```
Batch sample:
```
python end2end.py --query-type VoiceDesign --use-batch-sample
```
### Base (Voice Clone)
Single sample:
```
python end2end.py --query-type Base
```
Batch sample:
```
python end2end.py --query-type Base --use-batch-sample
```
Mode selection for Base:
- `--mode-tag icl` (default): standard mode
- `--mode-tag xvec_only`: enable `x_vector_only_mode` in the request
Examples:
```
python end2end.py --query-type Base --mode-tag icl
```
## Voice and Language Control
### Supported Voices (CustomVoice)
Predefined speaker voices are set via the `speaker` (or `voice_type`) field in `additional_information`. Available speakers depend on the loaded checkpoint; check `talker_config.spk_id` in the model config for the full list. Common voices include `vivian`, `ryan`, `aiden`, `ethan`, `serena` (case-insensitive).
Pass the speaker name in your request:
```python
additional_information = {
"text": ["你好，我是通义千问"],
"task_type": ["CustomVoice"],
"speaker": ["Vivian"],
"language": ["Chinese"],
}
```
### Supported Languages
The `language` field controls the codec-level language tag. Use `"Auto"` (default) for automatic detection.
Supported values: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`.
```python
additional_information = {
"text": ["Hello, nice to meet you."],
"task_type": ["CustomVoice"],
"speaker": ["Aiden"],
"language": ["English"],
}
```
### VoiceDesign and Base
- **VoiceDesign**: Use `instruct` for natural-language voice description; no `speaker` needed.
- **Base**: Use `ref_audio` and `ref_text` for voice cloning; `language` is optional.
## Streaming Mode
Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config):
```bash
python end2end.py --query-type CustomVoice --streaming --output-dir /tmp/out_stream
```
Each Code2Wav chunk is logged as it arrives (default 25 frames; configurable via `codec_chunk_frames`
in the stage config). The initial chunk size is dynamically selected based on server load for reduced
TTFA, and can be overridden per-request via the `initial_codec_chunk_frames` API field. The final WAV file is written once generation
completes. This demonstrates that audio data is available progressively rather than only at the end.
> **Note:** Streaming uses `AsyncOmni` internally. The non-streaming path (`Omni`) is unchanged.
## Batched Decoding
The Code2Wav stage (stage 1) supports batched decoding, where multiple requests are decoded in a single forward pass through the SpeechTokenizer. To use it, set `max_num_seqs > 1` on both stages via `--stage-overrides` and pass multiple prompts via `--txt-prompts` with a matching `--batch-size`.
```
python end2end.py --query-type CustomVoice \
--txt-prompts benchmark_prompts.txt \
--batch-size 4 \
--stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}'
```
**Important:** `--batch-size` must match a CUDA graph capture size (1, 2, 4, 8, 16...) because the Talker's code predictor KV cache is sized to `max_num_seqs`, and CUDA graphs pad the batch to the next capture size. Both stages need `max_num_seqs >= batch_size` in the stage config for batching to take effect. If only stage 1 has a higher `max_num_seqs`, it won't help — stage 1 can only batch chunks from requests that are in-flight simultaneously, which requires stage 0 to also process multiple requests concurrently.
## Notes
- The script uses the model paths embedded in `end2end.py`. Update them if your local cache path differs.
- Use `--output-dir` to change the output folder.
## Example materials
??? abstract "benchmark_prompts.txt"
``````txt
--8<-- "examples/offline_inference/qwen3_tts/benchmark_prompts.txt"
``````
??? abstract "end2end.py"
``````py
--8<-- "examples/offline_inference/qwen3_tts/end2end.py"
``````
docs/user_guide/examples/offline_inference/text_to_speech.md
+395
View File
@@ -0,0 +1,395 @@
# Text-To-Speech
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech>.
vLLM-Omni supports several autoregressive TTS models. They share a common
CLI shape (`--text`, `--ref-audio`, `--ref-text`, `--output-dir`) and live
together in this hub. Each model has its own subdirectory containing a
single `end2end.py` script; this README is the single doc entry point.
For online serving, see `examples/online_serving/<model>/`. For the full
list of supported architectures across all modalities, see
[Supported Models](https://github.com/vllm-project/vllm-omni/tree/main/docs/models/supported_models.md).
## Supported Models
| Model | HuggingFace repo | Stages | Voice cloning | Streaming | Special modes | Sample rate |
|---|---|---|---|---|---|---|
| VoxCPM2 | `openbmb/VoxCPM2` | single (native AR) | ✓ | — | continuation (`--ref-audio` + `--ref-text`) | 48 kHz |
| VoxCPM | local model dir | split | ✓ | ✓ (`voxcpm_async_chunk.yaml`) | — | 24 kHz |
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ (`async_chunk: true` default) | — | 22.05 kHz |
| Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ (`--streaming`) | — | 44.1 kHz |
| OmniVoice | `k2-fsa/OmniVoice` | 2 (gen + dec) | ✓ | — | voice design (`--instruct`), language (`--lang`) | 24 kHz |
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | 2 (talker + code2wav) | ✓ (Base) | ✓ | 3 task variants (`--query-type`) | 24 kHz |
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | varies | ✓ | ✓ | voice presets (`--voice`) | 24 kHz |
## Common Quick Start
Most models share this invocation shape:
```bash
python examples/offline_inference/text_to_speech/<model>/end2end.py \
--text "Hello, this is a test." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
`--ref-audio` and `--ref-text` are optional (text-only synthesis works without them) and must be provided together for voice cloning. The exotic scripts — Qwen3-TTS, Voxtral TTS, CosyVoice3 — accept additional model-specific flags documented in their per-model section below.
---
## VoxCPM2
Single-stage native AR TTS at 48 kHz. Pipeline: `feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE`.
### Prerequisites
```bash
pip install voxcpm
# or, for a local source checkout:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
--model openbmb/VoxCPM2 \
--text "Hello, this is a VoxCPM2 demo."
```
### Voice cloning
Pass a reference audio for isolated cloning, or both `--ref-audio` + `--ref-text` for prompt continuation:
```bash
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
--text "Hello, this is a voice clone demo." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
### Notes
- Output: 48 kHz mono WAV.
- Stage config: `vllm_omni/model_executor/stage_configs/voxcpm2.yaml` (default).
---
## VoxCPM
Split-stage TTS. The hub example covers single TTS, single voice cloning, and streaming. Use `benchmarks/voxcpm/` for warmup, batch JSONL prompts, profiler injection, and offline TTFP / RTF measurement.
### Prerequisites
```bash
pip install voxcpm soundfile
# or use a local source tree:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src
```
If the native VoxCPM `config.json` does not contain HF metadata such as `model_type`, prepare a persistent HF-compatible config directory and point the stage configs to it via `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH`:
```bash
export VOXCPM_MODEL=/path/to/voxcpm-model
export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config
mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"
cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json"
cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true
python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)'
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--text "This is a split-stage VoxCPM synthesis example running on vLLM Omni."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--text "This sentence is synthesized with a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript spoken in reference.wav."
```
### Streaming
Pass the async-chunk stage config:
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model "$VOXCPM_MODEL" \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
--text "This is a split-stage VoxCPM streaming example running on vLLM Omni."
```
### Notes
- `voxcpm.yaml` is the default non-streaming stage config; `voxcpm_async_chunk.yaml` enables streaming.
- Streaming is currently single-request oriented.
- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades quality.
- For online serving, see [`examples/online_serving/voxcpm`](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/voxcpm/README.md). For benchmark reporting, see [`benchmarks/voxcpm`](https://github.com/vllm-project/vllm-omni/tree/main/benchmarks/voxcpm/README.md).
---
## CosyVoice3
2-stage TTS pipeline (`talker` + `code2wav`) at 22.05 kHz.
### Prerequisites
```bash
uv pip install -e .
# Includes soundfile, onnxruntime, x-transformers, einops via requirements.
```
Download the model snapshot:
```python
from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512',
local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
```
If your downloaded checkpoint lacks `config.json`, add it:
```json
{
"model_type": "cosyvoice3",
"architectures": ["CosyVoice3Model"]
}
```
This is required because `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers the class mapping; the loader still reads `model_type` from `config.json` to select the class.
### Quick start
```bash
python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
--model pretrained_models/Fun-CosyVoice3-0.5B \
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN
```
### Voice cloning
Pass a reference audio. Note that CosyVoice3's `--prompt-text` is a system-style prompt for the GPT stage, not a reference transcript:
```bash
python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
--model pretrained_models/Fun-CosyVoice3-0.5B \
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN \
--ref-audio prompt.wav \
--prompt-text "You are a helpful assistant.<|endofprompt|>Testing my voices. Why should I not?"
```
### Notes
- Stage 0 (`talker`) emits speech tokens; stage 1 (`code2wav`) runs flow matching + HiFiGAN to synthesize waveform.
- Deploy config auto-loads from `vllm_omni/deploy/cosyvoice3.yaml` based on HF `model_type`. Pass `--deploy-config <path>` to override.
- `async_chunk: true` is the default; pass `--no-async-chunk` to switch to the legacy synchronous path.
---
## Fish Speech S2 Pro
4B dual-AR text-to-speech model from FishAudio with the DAC codec at 44.1 kHz.
### Prerequisites
```bash
pip install fish-speech
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a test of the Fish Speech text to speech system."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "Transcript of the reference audio."
```
### Streaming
```bash
python examples/offline_inference/text_to_speech/fish_speech/end2end.py \
--text "Hello, this is a streaming test." \
--streaming
```
Streaming requires `async_chunk: true` in the stage config.
### Notes
- Output: 44.1 kHz mono WAV.
- DAC codec weights (`codec.pth`) are loaded lazily from the model directory.
---
## OmniVoice
Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design).
### Prerequisites
```bash
huggingface-cli download k2-fsa/OmniVoice
```
Voice cloning requires `transformers>=5.3.0`. Auto and design modes work with `transformers>=4.57.0`.
### Quick start (auto voice)
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "Hello, this is a test."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "Hello, this is a test." \
--ref-audio ref.wav \
--ref-text "This is the reference transcription."
```
### Voice design
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "Hello, this is a test." \
--instruct "female, low pitch, british accent"
```
### Language hint
```bash
python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
--model k2-fsa/OmniVoice \
--text "你好，这是一个测试。" \
--lang zh
```
### Notes
- Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking.
- Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz.
---
## Qwen3-TTS
3-task-variant TTS with 24 kHz output. Has its own argparse surface (this script does not follow the common `--text` / `--ref-audio` shape).
### Prerequisites
For ROCm builds, replace `onnxruntime` with `onnxruntime-rocm`:
```bash
pip uninstall onnxruntime
pip install onnxruntime-rocm
```
### Task variants
- `CustomVoice`: predefined speaker (speaker ID) with optional style instruction.
- `VoiceDesign`: text + descriptive instruction designs a new voice.
- `Base`: voice cloning from reference audio + transcript.
```bash
# Single sample
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type VoiceDesign
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base
# Base variant has an additional mode flag:
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag icl # default
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag xvec_only # x_vector_only_mode
# Batch (multiple prompts in one run)
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice --use-batch-sample
```
### Streaming
```bash
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
--query-type CustomVoice \
--streaming \
--output-dir /tmp/out_stream
```
Streaming requires `async_chunk: true` in the stage config.
### Batched decoding
The Code2Wav stage supports batched decoding through the SpeechTokenizer. Configure both stages with `max_num_seqs > 1` via `--stage-overrides` and pass multiple prompts via `--txt-prompts`:
```bash
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
--query-type CustomVoice \
--txt-prompts examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt \
--batch-size 4 \
--stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}'
```
`--batch-size` must match a CUDA-graph capture size (1, 2, 4, 8, 16…).
### Notes
- Run `--help` for the full argument surface.
- See `qwen3_tts/end2end.py` for the prompt-length-estimation logic the Talker uses.
---
## Voxtral TTS
Voxtral-4B-TTS (Mistral). Has its own argparse surface; uses voice presets and the `mistral_common` `SpeechRequest` protocol.
### Prerequisites
Latest `mistral_common` with `SpeechRequest` support:
```bash
pip install -e /path/to/mistral-common # or upgrade from PyPI when available
```
### Quick start (voice preset)
```bash
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--write-audio --voice cheerful_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
```
### Voice cloning (capability gated upstream)
```bash
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--write-audio \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "This is a test message." \
--ref-audio path/to/reference_audio.wav
```
### Streaming + concurrency
```bash
python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \
--num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "..."
```
Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`).
### Notes
- `--num-prompts N` replicates the prompt for performance measurement.
- `--concurrency M` requires `--streaming` and must evenly divide `--num-prompts`.
- Run `--help` for the full argument surface.
## Example materials
??? abstract "cosyvoice3/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/cosyvoice3/end2end.py"
``````
??? abstract "fish_speech/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/fish_speech/end2end.py"
``````
??? abstract "omnivoice/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/omnivoice/end2end.py"
``````
??? abstract "qwen3_tts/benchmark_prompts.txt"
``````txt
--8<-- "examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt"
``````
??? abstract "qwen3_tts/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/qwen3_tts/end2end.py"
``````
??? abstract "voxcpm/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/voxcpm/end2end.py"
``````
??? abstract "voxcpm2/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/voxcpm2/end2end.py"
``````
??? abstract "voxtral_tts/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/voxtral_tts/end2end.py"
``````
docs/user_guide/examples/offline_inference/voxtral_tts.md
-69
View File
@@ -1,69 +0,0 @@
# Voxtral TTS Offline Inference
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/voxtral_tts>.
`end2end.py` runs Voxtral TTS end-to-end offline inference using vLLM. It supports both blocking (`Omni`) and streaming (`AsyncOmni`) generation, batched prompts with configurable concurrency, and voice selection via preset name or reference audio file.
When `mistral_common` has `SpeechRequest` support, prompt token IDs are built via `encode_speech_request`. Otherwise, the script falls back to manual token construction.
## Usage Examples
```bash
# Basic single-prompt with cheerful_female voice preset
python3 examples/offline_inference/voxtral_tts/end2end.py \
--write-audio --voice cheerful_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# 32 replicate prompts with cheerful_female voice preset
python3 examples/offline_inference/voxtral_tts/end2end.py \
--num-prompts 32 --write-audio --voice cheerful_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# Streaming with neutral_female voice preset
python3 examples/offline_inference/voxtral_tts/end2end.py \
--streaming --write-audio --voice neutral_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# 32 prompts, 8 concurrent requests per wave, streaming with neutral_female voice
python3 examples/offline_inference/voxtral_tts/end2end.py \
--num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?"
# Short debug prompt with reference audio
# Note: Reference audio capability is not yet released.
python3 examples/offline_inference/voxtral_tts/end2end.py \
--write-audio \
--model mistralai/Voxtral-4B-TTS-2603 \
--text "This is a test message." \
--audio-path path/to/reference_audio.wav
```
## Arguments
| Argument | Description |
|---|---|
| `--model PATH` | HuggingFace repo ID or local directory path (default: `mistralai/Voxtral-4B-TTS-2603`) |
| `--text TEXT` | Text to synthesize (default: `"This is a test message."`) |
| `--audio-path PATH` | Path to reference audio file for voice cloning |
| `--output-dir DIR` | Directory to write output WAV files (default: `output_audio`) |
| `--deploy-config PATH` | Override the deploy config path. If unset, auto-loads `vllm_omni/deploy/voxtral_tts.yaml` from the HF `model_type`. |
| `--num-prompts N` | Number of replicate prompts to run for measuring performance (default: 1) |
| `--streaming` | Use streaming generation via `AsyncOmni` (default: blocking `Omni`) |
| `--concurrency N` | Max concurrent requests per wave (must be used with `--streaming`, must evenly divide `--num-prompts`) |
| `--voice NAME` | Voice preset to use instead of reference audio file (e.g., casual_female, casual_male, cheerful_female, neutral_female, neutral_male) |
| `--write-audio` | Write generated audio to WAV files |
| `--profiling-mode` | Enable profiling mode (reduces max tokens to 50) |
| `--log-stats` | Enable detailed statistics logging |
## Example materials
??? abstract "end2end.py"
``````py
--8<-- "examples/offline_inference/voxtral_tts/end2end.py"
``````