diff --git a/docs/.nav.yml b/docs/.nav.yml index 2604f1e98..3ad99b5d4 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -16,8 +16,6 @@ nav: - examples/README.md - Offline Inference: - BAGEL-7B-MoT: user_guide/examples/offline_inference/bagel.md - - CosyVoice3: user_guide/examples/offline_inference/cosyvoice3.md - - Fish Speech S2 Pro: user_guide/examples/offline_inference/fish_speech.md - GLM-Image Multistage End-to-End Inference: user_guide/examples/offline_inference/glm_image.md - Helios Video Generation: user_guide/examples/offline_inference/helios.md - HunyuanImage-3.0 Image-to-Text Inference: user_guide/examples/offline_inference/hunyuan_image3.md @@ -28,24 +26,22 @@ nav: - MiMo-Audio Offline Inference: user_guide/examples/offline_inference/mimo_audio.md - Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md - Qwen3-Omni: user_guide/examples/offline_inference/qwen3_omni.md - - Qwen3-TTS: user_guide/examples/offline_inference/qwen3_tts.md - Text-To-Audio: user_guide/examples/offline_inference/text_to_audio.md - Text-To-Image: user_guide/examples/offline_inference/text_to_image.md + - Text-To-Speech: user_guide/examples/offline_inference/text_to_speech.md - Text-To-Video: user_guide/examples/offline_inference/text_to_video.md - - Voxtral TTS Offline Inference: user_guide/examples/offline_inference/voxtral_tts.md - X-To-Video-Audio: user_guide/examples/offline_inference/x_to_video_audio.md - Online Serving: - BAGEL-7B-MoT: user_guide/examples/online_serving/bagel.md - vLLM-Omni Helm Chart: user_guide/examples/online_serving/chart-helm.md - Diffusers Backend Adapter Example: user_guide/examples/online_serving/diffusers_pipeline_adapter.md - - Fish Speech S2 Pro: user_guide/examples/online_serving/fish_speech.md - GLM-Image Online Serving: user_guide/examples/online_serving/glm_image.md - Image-To-Image: user_guide/examples/online_serving/image_to_image.md - Image-To-Video: user_guide/examples/online_serving/image_to_video.md - Online serving Example of vLLM-Omni for MiMo-Audio: user_guide/examples/online_serving/mimo_audio.md - Qwen2.5-Omni: user_guide/examples/online_serving/qwen2_5_omni.md - Qwen3-Omni: user_guide/examples/online_serving/qwen3_omni.md - - Qwen3-TTS: user_guide/examples/online_serving/qwen3_tts.md + - Text-To-Speech: user_guide/examples/online_serving/text_to_speech.md - Text-To-Image: user_guide/examples/online_serving/text_to_image.md - Text-To-Video: user_guide/examples/online_serving/text_to_video.md - General: diff --git a/docs/serving/speech_api.md b/docs/serving/speech_api.md index 8f78d6a20..407aa2f6e 100644 --- a/docs/serving/speech_api.md +++ b/docs/serving/speech_api.md @@ -535,7 +535,7 @@ The bundled `qwen3_tts.yaml` uses `max_num_seqs: 1` (single request) on both sta |-------|-------------| | `fishaudio/s2-pro` | 4B dual-AR TTS with DAC codec (44.1 kHz). Supports text-to-speech and voice cloning. | -Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` needed). The `voice` field should be set to `"default"`. See the [Fish Speech online serving example](../user_guide/examples/online_serving/fish_speech.md) for details. +Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` needed). The `voice` field should be set to `"default"`. See the [Fish Speech section of the online TTS hub](../user_guide/examples/online_serving/text_to_speech.md#fish-speech-s2-pro) for details. ### Voxtral TTS diff --git a/docs/user_guide/examples/offline_inference/cosyvoice3.md b/docs/user_guide/examples/offline_inference/cosyvoice3.md deleted file mode 100644 index d0638f414..000000000 --- a/docs/user_guide/examples/offline_inference/cosyvoice3.md +++ /dev/null @@ -1,81 +0,0 @@ -# CosyVoice3 - -Source . - - -## Setup - -Install dependencies: -``` -uv pip install -e . -``` - -> **Note:** This includes required libraries such as `soundfile`, -> `onnxruntime`, `x-transformers`, and `einops` via -> `requirements/common.txt` and platform-specific requirements files. - -Download the model snapshot: -``` -from huggingface_hub import snapshot_download -snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B') -``` - -Add `config.json` in `pretrained_models/Fun-CosyVoice3-0.5B/`: -```json -{ - "model_type": "cosyvoice3", - "architectures": [ - "CosyVoice3Model" - ] -} -``` - -> **Why `config.json` is required:** -> `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers a class mapping. -> The loader still needs `model_type: "cosyvoice3"` from `config.json` to select that class. -> If no `config.json` is present, model type cannot be inferred automatically. -> If your downloaded checkpoint already includes a valid `config.json` with -> `model_type: "cosyvoice3"`, this manual step can be skipped. - -Run the offline verification script: -``` -python examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py \ - --model pretrained_models/Fun-CosyVoice3-0.5B \ - --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN -``` - -## Implementation Overview - -CosyVoice3 runs as a **2-stage Omni pipeline**. - -- **Stage 0** (`talker`) converts text + prompt audio to speech tokens. -- **Stage 1** (`code2wav`) converts speech tokens + prompt features to waveform via flow matching and HiFiGAN. - -Key components live in `vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py`. - -- `CosyVoice3MultiModalProcessor` builds the multimodal inputs: - - Tokenizes `prompt` and `prompt_text`. - - Extracts speech tokens and mel features from the prompt audio. - - Extracts a speaker embedding. -- `CosyVoice3Model` implements both stages: - - Stage 0 uses `CosyVoice3LM` and outputs speech tokens + conditioning features. - - Stage 1 runs the flow model (DiT-based CFM) and HiFiGAN to synthesize waveform. - -Pipeline topology lives in `vllm_omni/model_executor/models/cosyvoice3/pipeline.py`; -runtime tunables (batch size, memory limits, sampling) live in -`vllm_omni/deploy/cosyvoice3.yaml`. The deploy config auto-loads by -HF `model_type` and defaults to `async_chunk: true` (shared-memory -streaming). Pass `--no-async-chunk` on `vllm serve` to switch to the -legacy sync path where stage 1 runs `text2flow` over the full -speech-token sequence. - -- Stage 0 emits latent speech tokens. -- Stage 1 consumes them via `sync_process_input_func` (sync mode) or the - shared-memory connector (async-chunk mode) and outputs audio. - -## Example materials - -??? abstract "verify_e2e_cosyvoice.py" - ``````py - --8<-- "examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py" - `````` diff --git a/docs/user_guide/examples/offline_inference/fish_speech.md b/docs/user_guide/examples/offline_inference/fish_speech.md deleted file mode 100644 index 0d5c538be..000000000 --- a/docs/user_guide/examples/offline_inference/fish_speech.md +++ /dev/null @@ -1,85 +0,0 @@ -# Fish Speech S2 Pro - -Source . - - -This directory contains an offline demo for running Fish Speech S2 Pro (`fishaudio/s2-pro`) with vLLM Omni. It supports text-only synthesis and voice cloning with reference audio. - -## Model Overview - -[Fish Speech S2 Pro](https://huggingface.co/fishaudio/s2-pro) is a 4B dual-AR text-to-speech model by FishAudio. It uses: - -- **Slow AR**: Generates semantic tokens autoregressively (Qwen3-based backbone) -- **Fast AR**: Predicts residual codebook tokens from semantic tokens -- **DAC Decoder**: Converts codec codes to 44.1 kHz audio waveform - -The model produces high-quality speech with voice cloning capabilities. - -## Prerequisites - -Install the `fish-speech` package for the DAC codec: - -```bash -pip install fish-speech -``` - -## Setup - -Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. - -## Quick Start - -### Text-Only Synthesis - -```bash -python end2end.py --text "Hello, this is a test of the Fish Speech text to speech system." -``` - -Generated audio files are saved to `output_audio/` by default. - -### Voice Cloning - -Provide a reference audio file and its transcript to clone a voice: - -```bash -python end2end.py \ - --text "Hello, this is a cloned voice." \ - --ref-audio /path/to/reference.wav \ - --ref-text "Transcript of the reference audio." -``` - -## Streaming Mode - -Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config): - -```bash -python end2end.py --text "Hello, this is a test." --streaming -``` - -Each DAC decoder chunk is logged as it arrives. The final WAV file is written once generation completes. - -## Configuration - -| Argument | Default | Description | -|----------|---------|-------------| -| `--text` | `"Hello, this is a test..."` | Text to synthesize | -| `--ref-audio` | None | Path to reference audio for voice cloning | -| `--ref-text` | None | Transcript of the reference audio | -| `--model` | `fishaudio/s2-pro` | HuggingFace model path | -| `--stage-configs-path` | `fish_speech_s2_pro.yaml` | Path to stage configs YAML | -| `--output-dir` | `output_audio` | Output directory for WAV files | -| `--streaming` | False | Enable streaming via AsyncOmni | -| `--stage-init-timeout` | 600 | Stage initialization timeout (seconds) | - -## Notes - -- Output audio is 44.1 kHz mono WAV. -- The DAC codec weights (`codec.pth`) are loaded lazily from the model directory. -- Voice cloning encodes the reference audio using the DAC codec to extract semantic codes, then prepends them as a system message. - -## Example materials - -??? abstract "end2end.py" - ``````py - --8<-- "examples/offline_inference/fish_speech/end2end.py" - `````` diff --git a/docs/user_guide/examples/offline_inference/qwen3_tts.md b/docs/user_guide/examples/offline_inference/qwen3_tts.md deleted file mode 100644 index 7226ac1fe..000000000 --- a/docs/user_guide/examples/offline_inference/qwen3_tts.md +++ /dev/null @@ -1,172 +0,0 @@ -# Qwen3-TTS - -Source . - - -This directory contains an offline demo for running Qwen3 TTS models with vLLM Omni. It builds task-specific inputs and generates WAV files locally. - -## Model Overview - -Qwen3 TTS provides multiple task variants for speech generation: - -- **CustomVoice**: Generate speech with a known speaker identity (speaker ID) and optional instruction. -- **VoiceDesign**: Generate speech from text plus a descriptive instruction that designs a new voice. -- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. The `ref_audio` field accepts a local file path, HTTP URL, or base64 data URL. - -## Setup -Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. - -### ROCm Dependencies - -You will need to install the dependency `onnxruntime-rocm`. - -``` -pip uninstall onnxruntime # should be removed before we can install onnxruntime-rocm -pip install onnxruntime-rocm -``` - -## Quick Start - -Run a single sample for a task: - -``` -python end2end.py --query-type CustomVoice -``` - -Generated audio files are saved to `output_audio/` by default. - -## Task Usage - -### CustomVoice - -Single sample: - -``` -python end2end.py --query-type CustomVoice -``` - -Batch sample (multiple prompts in one run): - -``` -python end2end.py --query-type CustomVoice --use-batch-sample -``` - -### VoiceDesign - -Single sample: - -``` -python end2end.py --query-type VoiceDesign -``` - -Batch sample: - -``` -python end2end.py --query-type VoiceDesign --use-batch-sample -``` - -### Base (Voice Clone) - -Single sample: - -``` -python end2end.py --query-type Base -``` - -Batch sample: - -``` -python end2end.py --query-type Base --use-batch-sample -``` - -Mode selection for Base: - -- `--mode-tag icl` (default): standard mode -- `--mode-tag xvec_only`: enable `x_vector_only_mode` in the request - -Examples: - -``` -python end2end.py --query-type Base --mode-tag icl -``` - -## Voice and Language Control - -### Supported Voices (CustomVoice) - -Predefined speaker voices are set via the `speaker` (or `voice_type`) field in `additional_information`. Available speakers depend on the loaded checkpoint; check `talker_config.spk_id` in the model config for the full list. Common voices include `vivian`, `ryan`, `aiden`, `ethan`, `serena` (case-insensitive). - -Pass the speaker name in your request: - -```python -additional_information = { - "text": ["你好，我是通义千问"], - "task_type": ["CustomVoice"], - "speaker": ["Vivian"], - "language": ["Chinese"], -} -``` - -### Supported Languages - -The `language` field controls the codec-level language tag. Use `"Auto"` (default) for automatic detection. - -Supported values: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`. - -```python -additional_information = { - "text": ["Hello, nice to meet you."], - "task_type": ["CustomVoice"], - "speaker": ["Aiden"], - "language": ["English"], -} -``` - -### VoiceDesign and Base - -- **VoiceDesign**: Use `instruct` for natural-language voice description; no `speaker` needed. -- **Base**: Use `ref_audio` and `ref_text` for voice cloning; `language` is optional. - -## Streaming Mode - -Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config): - -```bash -python end2end.py --query-type CustomVoice --streaming --output-dir /tmp/out_stream -``` - -Each Code2Wav chunk is logged as it arrives (default 25 frames; configurable via `codec_chunk_frames` -in the stage config). The initial chunk size is dynamically selected based on server load for reduced -TTFA, and can be overridden per-request via the `initial_codec_chunk_frames` API field. The final WAV file is written once generation -completes. This demonstrates that audio data is available progressively rather than only at the end. - -> **Note:** Streaming uses `AsyncOmni` internally. The non-streaming path (`Omni`) is unchanged. - -## Batched Decoding - -The Code2Wav stage (stage 1) supports batched decoding, where multiple requests are decoded in a single forward pass through the SpeechTokenizer. To use it, set `max_num_seqs > 1` on both stages via `--stage-overrides` and pass multiple prompts via `--txt-prompts` with a matching `--batch-size`. - -``` -python end2end.py --query-type CustomVoice \ - --txt-prompts benchmark_prompts.txt \ - --batch-size 4 \ - --stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}' -``` - -**Important:** `--batch-size` must match a CUDA graph capture size (1, 2, 4, 8, 16...) because the Talker's code predictor KV cache is sized to `max_num_seqs`, and CUDA graphs pad the batch to the next capture size. Both stages need `max_num_seqs >= batch_size` in the stage config for batching to take effect. If only stage 1 has a higher `max_num_seqs`, it won't help — stage 1 can only batch chunks from requests that are in-flight simultaneously, which requires stage 0 to also process multiple requests concurrently. - -## Notes - -- The script uses the model paths embedded in `end2end.py`. Update them if your local cache path differs. -- Use `--output-dir` to change the output folder. - -## Example materials - -??? abstract "benchmark_prompts.txt" - ``````txt - --8<-- "examples/offline_inference/qwen3_tts/benchmark_prompts.txt" - `````` -??? abstract "end2end.py" - ``````py - --8<-- "examples/offline_inference/qwen3_tts/end2end.py" - `````` diff --git a/docs/user_guide/examples/offline_inference/text_to_speech.md b/docs/user_guide/examples/offline_inference/text_to_speech.md new file mode 100644 index 000000000..e88a68220 --- /dev/null +++ b/docs/user_guide/examples/offline_inference/text_to_speech.md @@ -0,0 +1,395 @@ +# Text-To-Speech + +Source . + + +vLLM-Omni supports several autoregressive TTS models. They share a common +CLI shape (`--text`, `--ref-audio`, `--ref-text`, `--output-dir`) and live +together in this hub. Each model has its own subdirectory containing a +single `end2end.py` script; this README is the single doc entry point. + +For online serving, see `examples/online_serving//`. For the full +list of supported architectures across all modalities, see +[Supported Models](https://github.com/vllm-project/vllm-omni/tree/main/docs/models/supported_models.md). + +## Supported Models + +| Model | HuggingFace repo | Stages | Voice cloning | Streaming | Special modes | Sample rate | +|---|---|---|---|---|---|---| +| VoxCPM2 | `openbmb/VoxCPM2` | single (native AR) | ✓ | — | continuation (`--ref-audio` + `--ref-text`) | 48 kHz | +| VoxCPM | local model dir | split | ✓ | ✓ (`voxcpm_async_chunk.yaml`) | — | 24 kHz | +| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ (`async_chunk: true` default) | — | 22.05 kHz | +| Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ (`--streaming`) | — | 44.1 kHz | +| OmniVoice | `k2-fsa/OmniVoice` | 2 (gen + dec) | ✓ | — | voice design (`--instruct`), language (`--lang`) | 24 kHz | +| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | 2 (talker + code2wav) | ✓ (Base) | ✓ | 3 task variants (`--query-type`) | 24 kHz | +| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | varies | ✓ | ✓ | voice presets (`--voice`) | 24 kHz | + +## Common Quick Start + +Most models share this invocation shape: + +```bash +python examples/offline_inference/text_to_speech//end2end.py \ + --text "Hello, this is a test." \ + --ref-audio /path/to/reference.wav \ + --ref-text "Transcript of the reference audio." +``` + +`--ref-audio` and `--ref-text` are optional (text-only synthesis works without them) and must be provided together for voice cloning. The exotic scripts — Qwen3-TTS, Voxtral TTS, CosyVoice3 — accept additional model-specific flags documented in their per-model section below. + +--- + +## VoxCPM2 + +Single-stage native AR TTS at 48 kHz. Pipeline: `feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE`. + +### Prerequisites +```bash +pip install voxcpm +# or, for a local source checkout: +export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm +``` + +### Quick start +```bash +python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \ + --model openbmb/VoxCPM2 \ + --text "Hello, this is a VoxCPM2 demo." +``` + +### Voice cloning +Pass a reference audio for isolated cloning, or both `--ref-audio` + `--ref-text` for prompt continuation: +```bash +python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \ + --text "Hello, this is a voice clone demo." \ + --ref-audio /path/to/reference.wav \ + --ref-text "Transcript of the reference audio." +``` + +### Notes +- Output: 48 kHz mono WAV. +- Stage config: `vllm_omni/model_executor/stage_configs/voxcpm2.yaml` (default). + +--- + +## VoxCPM + +Split-stage TTS. The hub example covers single TTS, single voice cloning, and streaming. Use `benchmarks/voxcpm/` for warmup, batch JSONL prompts, profiler injection, and offline TTFP / RTF measurement. + +### Prerequisites +```bash +pip install voxcpm soundfile +# or use a local source tree: +export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src +``` + +If the native VoxCPM `config.json` does not contain HF metadata such as `model_type`, prepare a persistent HF-compatible config directory and point the stage configs to it via `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH`: + +```bash +export VOXCPM_MODEL=/path/to/voxcpm-model +export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config +mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH" +cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json" +cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true +python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)' +``` + +### Quick start +```bash +python examples/offline_inference/text_to_speech/voxcpm/end2end.py \ + --model "$VOXCPM_MODEL" \ + --text "This is a split-stage VoxCPM synthesis example running on vLLM Omni." +``` + +### Voice cloning +```bash +python examples/offline_inference/text_to_speech/voxcpm/end2end.py \ + --model "$VOXCPM_MODEL" \ + --text "This sentence is synthesized with a cloned voice." \ + --ref-audio /path/to/reference.wav \ + --ref-text "The exact transcript spoken in reference.wav." +``` + +### Streaming +Pass the async-chunk stage config: +```bash +python examples/offline_inference/text_to_speech/voxcpm/end2end.py \ + --model "$VOXCPM_MODEL" \ + --stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \ + --text "This is a split-stage VoxCPM streaming example running on vLLM Omni." +``` + +### Notes +- `voxcpm.yaml` is the default non-streaming stage config; `voxcpm_async_chunk.yaml` enables streaming. +- Streaming is currently single-request oriented. +- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades quality. +- For online serving, see [`examples/online_serving/voxcpm`](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/voxcpm/README.md). For benchmark reporting, see [`benchmarks/voxcpm`](https://github.com/vllm-project/vllm-omni/tree/main/benchmarks/voxcpm/README.md). + +--- + +## CosyVoice3 + +2-stage TTS pipeline (`talker` + `code2wav`) at 22.05 kHz. + +### Prerequisites +```bash +uv pip install -e . +# Includes soundfile, onnxruntime, x-transformers, einops via requirements. +``` + +Download the model snapshot: +```python +from huggingface_hub import snapshot_download +snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', + local_dir='pretrained_models/Fun-CosyVoice3-0.5B') +``` + +If your downloaded checkpoint lacks `config.json`, add it: +```json +{ + "model_type": "cosyvoice3", + "architectures": ["CosyVoice3Model"] +} +``` +This is required because `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers the class mapping; the loader still reads `model_type` from `config.json` to select the class. + +### Quick start +```bash +python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \ + --model pretrained_models/Fun-CosyVoice3-0.5B \ + --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN +``` + +### Voice cloning +Pass a reference audio. Note that CosyVoice3's `--prompt-text` is a system-style prompt for the GPT stage, not a reference transcript: +```bash +python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \ + --model pretrained_models/Fun-CosyVoice3-0.5B \ + --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN \ + --ref-audio prompt.wav \ + --prompt-text "You are a helpful assistant.<|endofprompt|>Testing my voices. Why should I not?" +``` + +### Notes +- Stage 0 (`talker`) emits speech tokens; stage 1 (`code2wav`) runs flow matching + HiFiGAN to synthesize waveform. +- Deploy config auto-loads from `vllm_omni/deploy/cosyvoice3.yaml` based on HF `model_type`. Pass `--deploy-config ` to override. +- `async_chunk: true` is the default; pass `--no-async-chunk` to switch to the legacy synchronous path. + +--- + +## Fish Speech S2 Pro + +4B dual-AR text-to-speech model from FishAudio with the DAC codec at 44.1 kHz. + +### Prerequisites +```bash +pip install fish-speech +``` + +### Quick start +```bash +python examples/offline_inference/text_to_speech/fish_speech/end2end.py \ + --text "Hello, this is a test of the Fish Speech text to speech system." +``` + +### Voice cloning +```bash +python examples/offline_inference/text_to_speech/fish_speech/end2end.py \ + --text "Hello, this is a cloned voice." \ + --ref-audio /path/to/reference.wav \ + --ref-text "Transcript of the reference audio." +``` + +### Streaming +```bash +python examples/offline_inference/text_to_speech/fish_speech/end2end.py \ + --text "Hello, this is a streaming test." \ + --streaming +``` +Streaming requires `async_chunk: true` in the stage config. + +### Notes +- Output: 44.1 kHz mono WAV. +- DAC codec weights (`codec.pth`) are loaded lazily from the model directory. + +--- + +## OmniVoice + +Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design). + +### Prerequisites +```bash +huggingface-cli download k2-fsa/OmniVoice +``` +Voice cloning requires `transformers>=5.3.0`. Auto and design modes work with `transformers>=4.57.0`. + +### Quick start (auto voice) +```bash +python examples/offline_inference/text_to_speech/omnivoice/end2end.py \ + --model k2-fsa/OmniVoice \ + --text "Hello, this is a test." +``` + +### Voice cloning +```bash +python examples/offline_inference/text_to_speech/omnivoice/end2end.py \ + --model k2-fsa/OmniVoice \ + --text "Hello, this is a test." \ + --ref-audio ref.wav \ + --ref-text "This is the reference transcription." +``` + +### Voice design +```bash +python examples/offline_inference/text_to_speech/omnivoice/end2end.py \ + --model k2-fsa/OmniVoice \ + --text "Hello, this is a test." \ + --instruct "female, low pitch, british accent" +``` + +### Language hint +```bash +python examples/offline_inference/text_to_speech/omnivoice/end2end.py \ + --model k2-fsa/OmniVoice \ + --text "你好，这是一个测试。" \ + --lang zh +``` + +### Notes +- Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking. +- Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz. + +--- + +## Qwen3-TTS + +3-task-variant TTS with 24 kHz output. Has its own argparse surface (this script does not follow the common `--text` / `--ref-audio` shape). + +### Prerequisites +For ROCm builds, replace `onnxruntime` with `onnxruntime-rocm`: +```bash +pip uninstall onnxruntime +pip install onnxruntime-rocm +``` + +### Task variants +- `CustomVoice`: predefined speaker (speaker ID) with optional style instruction. +- `VoiceDesign`: text + descriptive instruction designs a new voice. +- `Base`: voice cloning from reference audio + transcript. + +```bash +# Single sample +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type VoiceDesign +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base + +# Base variant has an additional mode flag: +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag icl # default +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag xvec_only # x_vector_only_mode + +# Batch (multiple prompts in one run) +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice --use-batch-sample +``` + +### Streaming +```bash +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \ + --query-type CustomVoice \ + --streaming \ + --output-dir /tmp/out_stream +``` +Streaming requires `async_chunk: true` in the stage config. + +### Batched decoding +The Code2Wav stage supports batched decoding through the SpeechTokenizer. Configure both stages with `max_num_seqs > 1` via `--stage-overrides` and pass multiple prompts via `--txt-prompts`: +```bash +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \ + --query-type CustomVoice \ + --txt-prompts examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt \ + --batch-size 4 \ + --stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}' +``` +`--batch-size` must match a CUDA-graph capture size (1, 2, 4, 8, 16…). + +### Notes +- Run `--help` for the full argument surface. +- See `qwen3_tts/end2end.py` for the prompt-length-estimation logic the Talker uses. + +--- + +## Voxtral TTS + +Voxtral-4B-TTS (Mistral). Has its own argparse surface; uses voice presets and the `mistral_common` `SpeechRequest` protocol. + +### Prerequisites +Latest `mistral_common` with `SpeechRequest` support: +```bash +pip install -e /path/to/mistral-common # or upgrade from PyPI when available +``` + +### Quick start (voice preset) +```bash +python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \ + --write-audio --voice cheerful_female \ + --model mistralai/Voxtral-4B-TTS-2603 \ + --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" +``` + +### Voice cloning (capability gated upstream) +```bash +python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \ + --write-audio \ + --model mistralai/Voxtral-4B-TTS-2603 \ + --text "This is a test message." \ + --ref-audio path/to/reference_audio.wav +``` + +### Streaming + concurrency +```bash +python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \ + --num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \ + --model mistralai/Voxtral-4B-TTS-2603 \ + --text "..." +``` +Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`). + +### Notes +- `--num-prompts N` replicates the prompt for performance measurement. +- `--concurrency M` requires `--streaming` and must evenly divide `--num-prompts`. +- Run `--help` for the full argument surface. + +## Example materials + +??? abstract "cosyvoice3/end2end.py" + ``````py + --8<-- "examples/offline_inference/text_to_speech/cosyvoice3/end2end.py" + `````` +??? abstract "fish_speech/end2end.py" + ``````py + --8<-- "examples/offline_inference/text_to_speech/fish_speech/end2end.py" + `````` +??? abstract "omnivoice/end2end.py" + ``````py + --8<-- "examples/offline_inference/text_to_speech/omnivoice/end2end.py" + `````` +??? abstract "qwen3_tts/benchmark_prompts.txt" + ``````txt + --8<-- "examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt" + `````` +??? abstract "qwen3_tts/end2end.py" + ``````py + --8<-- "examples/offline_inference/text_to_speech/qwen3_tts/end2end.py" + `````` +??? abstract "voxcpm/end2end.py" + ``````py + --8<-- "examples/offline_inference/text_to_speech/voxcpm/end2end.py" + `````` +??? abstract "voxcpm2/end2end.py" + ``````py + --8<-- "examples/offline_inference/text_to_speech/voxcpm2/end2end.py" + `````` +??? abstract "voxtral_tts/end2end.py" + ``````py + --8<-- "examples/offline_inference/text_to_speech/voxtral_tts/end2end.py" + `````` diff --git a/docs/user_guide/examples/offline_inference/voxtral_tts.md b/docs/user_guide/examples/offline_inference/voxtral_tts.md deleted file mode 100644 index 1a4f2c21b..000000000 --- a/docs/user_guide/examples/offline_inference/voxtral_tts.md +++ /dev/null @@ -1,69 +0,0 @@ -# Voxtral TTS Offline Inference - -Source . - - -`end2end.py` runs Voxtral TTS end-to-end offline inference using vLLM. It supports both blocking (`Omni`) and streaming (`AsyncOmni`) generation, batched prompts with configurable concurrency, and voice selection via preset name or reference audio file. - -When `mistral_common` has `SpeechRequest` support, prompt token IDs are built via `encode_speech_request`. Otherwise, the script falls back to manual token construction. - -## Usage Examples - - -```bash -# Basic single-prompt with cheerful_female voice preset -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --write-audio --voice cheerful_female \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" - -# 32 replicate prompts with cheerful_female voice preset -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --num-prompts 32 --write-audio --voice cheerful_female \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" - -# Streaming with neutral_female voice preset -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --streaming --write-audio --voice neutral_female \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" - -# 32 prompts, 8 concurrent requests per wave, streaming with neutral_female voice -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" - -# Short debug prompt with reference audio -# Note: Reference audio capability is not yet released. -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --write-audio \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "This is a test message." \ - --audio-path path/to/reference_audio.wav -``` - -## Arguments - -| Argument | Description | -|---|---| -| `--model PATH` | HuggingFace repo ID or local directory path (default: `mistralai/Voxtral-4B-TTS-2603`) | -| `--text TEXT` | Text to synthesize (default: `"This is a test message."`) | -| `--audio-path PATH` | Path to reference audio file for voice cloning | -| `--output-dir DIR` | Directory to write output WAV files (default: `output_audio`) | -| `--deploy-config PATH` | Override the deploy config path. If unset, auto-loads `vllm_omni/deploy/voxtral_tts.yaml` from the HF `model_type`. | -| `--num-prompts N` | Number of replicate prompts to run for measuring performance (default: 1) | -| `--streaming` | Use streaming generation via `AsyncOmni` (default: blocking `Omni`) | -| `--concurrency N` | Max concurrent requests per wave (must be used with `--streaming`, must evenly divide `--num-prompts`) | -| `--voice NAME` | Voice preset to use instead of reference audio file (e.g., casual_female, casual_male, cheerful_female, neutral_female, neutral_male) | -| `--write-audio` | Write generated audio to WAV files | -| `--profiling-mode` | Enable profiling mode (reduces max tokens to 50) | -| `--log-stats` | Enable detailed statistics logging | - -## Example materials - -??? abstract "end2end.py" - ``````py - --8<-- "examples/offline_inference/voxtral_tts/end2end.py" - `````` diff --git a/docs/user_guide/examples/online_serving/fish_speech.md b/docs/user_guide/examples/online_serving/fish_speech.md deleted file mode 100644 index 2a15ef44a..000000000 --- a/docs/user_guide/examples/online_serving/fish_speech.md +++ /dev/null @@ -1,199 +0,0 @@ -# Fish Speech S2 Pro - -Source . - - -## Model - -| Model | Description | -|-------|-------------| -| `fishaudio/s2-pro` | Fish Speech S2 Pro -- 4B dual-AR TTS model with DAC codec (44.1 kHz) | - -## Gradio Demo - -!!! note "Gradio is an optional dependency" - The Gradio demo requires the `[demo]` extras. Install them first: - - ```bash - pip install 'vllm-omni[demo]' - ``` - - Or, if installing from source: `pip install -e '.[demo]'` - -An interactive Gradio demo is available with text-to-speech synthesis, voice cloning, and streaming support. - -```bash -# Option 1: Launch server + Gradio together -./run_gradio_demo.sh - -# Option 2: If server is already running -python gradio_demo.py --api-base http://localhost:8091 -``` - -Then open http://localhost:7860 in your browser. - -Features: - -- Text-to-speech synthesis -- Voice cloning from uploaded audio or URL -- Streaming mode (progressive PCM playback) - -## Launch the Server - -```bash -vllm serve fishaudio/s2-pro --omni --port 8091 -``` - -The deploy config is auto-loaded from `vllm_omni/deploy/fish_qwen3_omni.yaml`. - -Or use the convenience script: - -```bash -./run_server.sh -``` - -## Send TTS Request - -### Using curl - -```bash -# Basic TTS -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, how are you?", - "voice": "default", - "response_format": "wav" - }' --output output.wav -``` - -### Voice Cloning - -Provide a reference audio (URL or base64 data URL) and its transcript: - -```bash -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, this is a cloned voice.", - "voice": "default", - "ref_audio": "https://example.com/reference.wav", - "ref_text": "Transcript of the reference audio." - }' --output cloned.wav -``` - -### Using Python - -```python -import httpx - -# Basic TTS -response = httpx.post( - "http://localhost:8091/v1/audio/speech", - json={ - "input": "Hello, how are you?", - "voice": "default", - "response_format": "wav", - }, - timeout=300.0, -) - -with open("output.wav", "wb") as f: - f.write(response.content) -``` - -### Using the CLI Client - -```bash -cd examples/online_serving/fish_speech - -# Basic TTS -python speech_client.py --text "Hello, how are you?" - -# Voice cloning -python speech_client.py \ - --text "Hello, this is a cloned voice." \ - --ref-audio /path/to/reference.wav \ - --ref-text "Transcript of the reference audio." - -# Streaming PCM output -python speech_client.py --text "Hello world" --stream --output output.pcm -``` - -The CLI client supports: - -- `--api-base`: API base URL (default: `http://localhost:8091`) -- `--model` (or `-m`): Model name (default: `fishaudio/s2-pro`) -- `--text`: Text to synthesize (required) -- `--ref-audio`: Reference audio for voice cloning (local path or URL) -- `--ref-text`: Transcript of the reference audio -- `--stream`: Enable streaming (PCM output) -- `--response-format`: Audio format: wav, mp3, flac, pcm, aac, opus (default: wav) -- `--output` (or `-o`): Output file path - -## Streaming - -Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded: - -```bash -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, how are you?", - "voice": "default", - "stream": true, - "response_format": "pcm" - }' --no-buffer | play -t raw -r 44100 -e signed -b 16 -c 1 - -``` - -**Note:** Fish Speech outputs at 44.1 kHz (unlike Qwen3-TTS which outputs at 24 kHz). - -## API Parameters - -Fish Speech uses the same `/v1/audio/speech` endpoint as Qwen3-TTS. See the [Speech API reference](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/speech_api/) for full parameter documentation. - -Key parameters for Fish Speech: - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `input` | string | **required** | Text to synthesize | -| `voice` | string | "default" | Voice name (use "default" for Fish Speech) | -| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus | -| `ref_audio` | string | null | Reference audio URL or base64 data URL for voice cloning | -| `ref_text` | string | null | Transcript of reference audio (required for voice cloning) | -| `max_new_tokens` | int | 4096 | Maximum tokens to generate | -| `stream` | bool | false | Stream raw PCM chunks | - -## Prerequisites - -Install the `fish-speech` package for the DAC codec: - -```bash -pip install fish-speech -``` - -## Troubleshooting - -1. **No audio output**: Make sure the `fish-speech` package is installed for the DAC decoder -2. **Connection refused**: Ensure the server is running on the correct port -3. **Flashinfer version mismatch**: Set `FLASHINFER_DISABLE_VERSION_CHECK=1` if you see version warnings -4. **Out of memory**: Reduce `--gpu-memory-utilization` or use a GPU with more VRAM - -## Example materials - -??? abstract "gradio_demo.py" - ``````py - --8<-- "examples/online_serving/fish_speech/gradio_demo.py" - `````` -??? abstract "run_gradio_demo.sh" - ``````sh - --8<-- "examples/online_serving/fish_speech/run_gradio_demo.sh" - `````` -??? abstract "run_server.sh" - ``````sh - --8<-- "examples/online_serving/fish_speech/run_server.sh" - `````` -??? abstract "speech_client.py" - ``````py - --8<-- "examples/online_serving/fish_speech/speech_client.py" - `````` diff --git a/docs/user_guide/examples/online_serving/qwen3_tts.md b/docs/user_guide/examples/online_serving/qwen3_tts.md deleted file mode 100644 index 95f234f02..000000000 --- a/docs/user_guide/examples/online_serving/qwen3_tts.md +++ /dev/null @@ -1,451 +0,0 @@ -# Qwen3-TTS - -Source . - - -## 🛠️ Installation - -Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md) - -## Supported Models - -| Model | Task Type | Description | -| -------------------------------------- | ----------- | ----------------------------------------------------- | -| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | CustomVoice | Predefined speaker voices with optional style control | -| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description | -| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio | -| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant | -| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning | - -## Gradio Demo - -!!! note "Gradio is an optional dependency" - The Gradio demo requires the `[demo]` extras. Install them first: - - ```bash - pip install 'vllm-omni[demo]' - ``` - - Or, if installing from source: `pip install -e '.[demo]'` - -Two interactive Gradio demos are available, both supporting all 3 task types: - -| Demo | File | Transport | Streaming Quality | -| -------- | ------------------------ | ------------ | -------------------------------------------------- | -| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks | -| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) | - -```bash -# Option 1: Launch server + Standard Gradio together -./run_gradio_demo.sh # CustomVoice (default) -./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign -./run_gradio_demo.sh --task-type Base # Voice cloning - -# Option 2: If server is already running -python gradio_demo.py --api-base http://localhost:8000 - -# Option 3: FastRTC demo (gapless streaming) -pip install fastrtc -python gradio_fastrtc_demo.py --api-base http://localhost:8000 -``` - -Then open http://localhost:7860 in your browser. - -## Run examples (Qwen3-TTS) - -### Launch the Server - -```bash -# CustomVoice model (predefined speakers) -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ - --deploy-config vllm_omni/deploy/qwen3_tts.yaml \ - --omni \ - --port 8091 \ - --trust-remote-code \ - --enforce-eager - -# VoiceDesign model -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \ - --deploy-config vllm_omni/deploy/qwen3_tts.yaml \ - --omni \ - --port 8091 \ - --trust-remote-code \ - --enforce-eager - -# Base model (voice cloning) -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \ - --deploy-config vllm_omni/deploy/qwen3_tts.yaml \ - --omni \ - --port 8091 \ - --trust-remote-code \ - --enforce-eager -``` - -If you have custom stage configs file, launch the server with command below -```bash -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ - --stage-configs-path /path/to/stage_configs_file \ - --omni \ - --port 8091 \ - --trust-remote-code \ - --enforce-eager -``` - -Alternatively, use the convenience script: -```bash -./run_server.sh # Default: CustomVoice model -./run_server.sh CustomVoice # CustomVoice model -./run_server.sh VoiceDesign # VoiceDesign model -./run_server.sh Base # Base (voice clone) model -``` - -### Send TTS Request - -Get into the example folder -```bash -cd examples/online_serving/qwen3_tts -``` - -#### Send request via python - -```bash -# CustomVoice: Use predefined speaker -python openai_speech_client.py \ - --text "你好，我是通义千问" \ - --speaker vivian \ - --language Chinese - -# CustomVoice with style instruction -python openai_speech_client.py \ - --text "今天天气真好" \ - --speaker ryan \ - --instructions "用开心的语气说" - -# VoiceDesign: Describe the voice style -python openai_speech_client.py \ - --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \ - --task-type VoiceDesign \ - --text "哥哥，你回来啦" \ - --instructions "体现撒娇稚嫩的萝莉女声，音调偏高" - -# Base: Voice cloning -python openai_speech_client.py \ - --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \ - --task-type Base \ - --text "Hello, this is a cloned voice" \ - --ref-audio /path/to/reference.wav \ - --ref-text "Original transcript of the reference audio" -``` - -The Python client supports the following command-line arguments: - -- `--api-base`: API base URL (default: `http://localhost:8091`) -- `--model` (or `-m`): Model name/path (default: `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`) -- `--task-type` (or `-t`): TTS task type. Options: `CustomVoice`, `VoiceDesign`, `Base` -- `--text`: Text to synthesize (required) -- `--speaker`: Speaker name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc. -- `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian` -- `--instructions`: Voice style/emotion instructions -- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server. -- `--ref-text`: Reference audio transcript for voice cloning (Base task). -- `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus` -- `--output` (or `-o`): Output audio file path (default: `tts_output.wav`) - -#### Send request via curl - -```bash -# Simple TTS request -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, how are you?", - "voice": "vivian", - "language": "English" - }' --output output.wav - -# With style instruction -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "I am so excited!", - "voice": "vivian", - "instructions": "Speak with great enthusiasm" - }' --output excited.wav - -# List available voices in CustomVoice models -curl http://localhost:8091/v1/audio/voices -``` - -### Using OpenAI SDK - -```python -from openai import OpenAI - -client = OpenAI(base_url="http://localhost:8091/v1", api_key="none") - -response = client.audio.speech.create( - model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", - voice="vivian", - input="Hello, how are you?", -) - -response.stream_to_file("output.wav") -``` - -### Using Python httpx - -```python -import httpx - -response = httpx.post( - "http://localhost:8091/v1/audio/speech", - json={ - "input": "Hello, how are you?", - "voice": "vivian", - "language": "English", - }, - timeout=300.0, -) - -with open("output.wav", "wb") as f: - f.write(response.content) -``` - -## API Reference - -### Voices Endpoint - -#### GET /v1/audio/voices - -List all available voices/speakers from the loaded model, including both built-in model voices and uploaded custom voices. - -**Response Example:** -```json -{ - "voices": ["vivian", "ryan", "custom_voice_1"], - "uploaded_voices": [ - { - "name": "custom_voice_1", - "consent": "user_consent_id", - "created_at": 1738660000, - "file_size": 1024000, - "mime_type": "audio/wav", - "ref_text": "The exact transcript of the audio sample.", - "speaker_description": "warm narrator" - } - ] -} -``` - -Fields `ref_text` and `speaker_description` are omitted per-entry when not provided at upload time. - -#### POST /v1/audio/voices - -Upload a new voice sample for voice cloning in Base task TTS requests. - -**Form Parameters:** -- `audio_sample` (required): Audio file (max 10MB, supported formats: wav, mp3, flac, ogg, aac, webm, mp4) -- `consent` (required): Consent recording ID -- `name` (required): Name for the new voice -- `ref_text` (optional): Transcript of the audio. Enables in-context voice cloning (higher quality). -- `speaker_description` (optional): Free-form description of the voice (e.g. "warm narrator", "energetic presenter"). Stored as metadata. - -**Response Example:** -```json -{ - "success": true, - "voice": { - "name": "custom_voice_1", - "consent": "user_consent_id", - "created_at": 1738660000, - "mime_type": "audio/wav", - "file_size": 1024000, - "ref_text": "The exact transcript of the audio sample.", - "speaker_description": "warm narrator" - } -} -``` - -Fields `ref_text` and `speaker_description` are omitted when not provided at upload time. - -**Usage Example:** -```bash -curl -X POST http://localhost:8091/v1/audio/voices \ - -F "audio_sample=@/path/to/voice_sample.wav" \ - -F "consent=user_consent_id" \ - -F "name=custom_voice_1" \ - -F "ref_text=The exact transcript of the audio sample." \ - -F "speaker_description=warm narrator" -``` - -### Endpoint - -``` -POST /v1/audio/speech -Content-Type: application/json -``` - -This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters. - -### Request Body - -```json -{ - "input": "Text to synthesize", - "voice": "vivian", - "response_format": "wav", - "task_type": "CustomVoice", - "language": "Auto", - "instructions": "Optional style instructions", - "ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning", - "ref_text": "Reference audio transcript", - "x_vector_only_mode": false, - "max_new_tokens": 2048 -} -``` - -> **Note:** The `model` field is optional when serving a single model, as the server already knows which model is loaded. - -### Response - -Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/wav`). - -### Voice and language (summary) - -- **Speaker**: Use the `voice` request field to select the speaker (e.g., `vivian`, `ryan`, `aiden`). List available speakers with `GET /v1/audio/voices`. -- **Language**: Use the `language` field for the codec language tag (`Auto`, `Chinese`, `English`, etc.). Default is `Auto` for automatic detection. -- **CustomVoice**: Requires a valid `voice` from the model’s speaker set. **VoiceDesign**: Use `instructions` to describe the voice. **Base**: Use `ref_audio` and `ref_text` for voice cloning. - -## Parameters - -### OpenAI Standard Parameters - -| Parameter | Type | Default | Description | -| ----------------- | ------ | -------------- | ----------------------------------------------------------- | -| `input` | string | **required** | Text to synthesize | -| `model` | string | server's model | Model to use (optional, should match server if specified) | -| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) | -| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus | -| `speed` | float | 1.0 | Playback speed (0.25-4.0, not supported with `stream=true`) | - -### vLLM-Omni Extension Parameters - -| Parameter | Type | Default | Description | -| ---------------------------- | ------ | ------------- | -------------------------------------------------------------------------------------------------------------------- | -| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | -| `language` | string | "Auto" | Language (see supported languages below) | -| `instructions` | string | "" | Voice style/emotion instructions | -| `max_new_tokens` | int | 2048 | Maximum tokens to generate | -| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. | -| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | - -**Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian - -### Voice Clone Parameters (Base task) - -| Parameter | Type | Required | Description | -| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------- | -| `ref_audio` | string | **Yes** | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) | -| `ref_text` | string | No | Transcript of reference audio (for ICL mode) | -| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | - -## Streaming - -Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded -(one chunk per Code2Wav window, default 25 frames; configurable in the stage config): - -```bash -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, how are you?", - "voice": "vivian", - "language": "English", - "stream": true, - "response_format": "pcm" - }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 - -``` - -**Constraints:** -- `stream=true` requires `response_format="pcm"` (raw 16-bit signed PCM, 24 kHz mono). -- `speed` adjustment is not supported when streaming. -- Requires the server stage config to have `async_chunk: true` (default in `qwen3_tts.yaml`). - -## Streaming Text Input (WebSocket) - -The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally, buffers and splits it at sentence boundaries, and generates audio per sentence. - -When `stream_audio=true`, each sentence is emitted as `audio.start`, one or more binary PCM frames, and `audio.done`. - -### Quick Start - -```bash -python streaming_speech_client.py \ - --text "Hello world. How are you? I am fine." - -python streaming_speech_client.py \ - --text "Hello world. How are you? I am fine." \ - --simulate-stt --stt-delay 0.1 -``` - -### WebSocket Protocol - -Client -> Server: - -```jsonc -{"type": "session.config", "voice": "Vivian", "task_type": "CustomVoice", "language": "Auto", "split_granularity": "sentence", "stream_audio": true, "response_format": "pcm"} -{"type": "input.text", "text": "Hello, how are you? "} -{"type": "input.done"} -``` - -Server -> Client: - -```jsonc -{"type": "audio.start", "sentence_index": 0, "sentence_text": "Hello, how are you?", "format": "pcm", "sample_rate": 24000} -// binary PCM frame(s) -{"type": "audio.done", "sentence_index": 0, "total_bytes": 96000, "error": false} -{"type": "session.done", "total_sentences": 1} -``` - -## Limitations - -- **Single request**: Batch processing is not yet optimized for online serving. - -## Troubleshooting - -1. **TTS model did not produce audio output**: Ensure you're using the correct model variant for your task type (CustomVoice task → CustomVoice model, etc.) -2. **Connection refused**: Make sure the server is running on the correct port -3. **Out of memory**: Use smaller model variant (`Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice`) or reduce `--gpu-memory-utilization` -4. **Unsupported speaker**: Use `/v1/audio/voices` to list available voices for the loaded model -5. **Voice clone fails**: Ensure you're using the Base model variant for voice cloning - -## Example materials - -??? abstract "gradio_demo.py" - ``````py - --8<-- "examples/online_serving/qwen3_tts/gradio_demo.py" - `````` -??? abstract "gradio_fastrtc_demo.py" - ``````py - --8<-- "examples/online_serving/qwen3_tts/gradio_fastrtc_demo.py" - `````` -??? abstract "openai_speech_client.py" - ``````py - --8<-- "examples/online_serving/qwen3_tts/openai_speech_client.py" - `````` -??? abstract "run_gradio_demo.sh" - ``````sh - --8<-- "examples/online_serving/qwen3_tts/run_gradio_demo.sh" - `````` -??? abstract "run_server.sh" - ``````sh - --8<-- "examples/online_serving/qwen3_tts/run_server.sh" - `````` -??? abstract "streaming_speech_client.py" - ``````py - --8<-- "examples/online_serving/qwen3_tts/streaming_speech_client.py" - `````` -??? abstract "tts_common.py" - ``````py - --8<-- "examples/online_serving/qwen3_tts/tts_common.py" - `````` diff --git a/docs/user_guide/examples/online_serving/text_to_speech.md b/docs/user_guide/examples/online_serving/text_to_speech.md new file mode 100644 index 000000000..703b7cf7a --- /dev/null +++ b/docs/user_guide/examples/online_serving/text_to_speech.md @@ -0,0 +1,489 @@ +# Text-To-Speech (Online Serving) + +Source . + + +vLLM-Omni exposes TTS models through the OpenAI-compatible +[`POST /v1/audio/speech`](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md) endpoint, +launched with `vllm serve --omni`. Each TTS model has its own +subdirectory containing client snippets, gradio demos, and helper +scripts; this README is the single doc entry point for all of them. + +For offline inference, see [`examples/offline_inference/text_to_speech`](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md). +For the full list of supported architectures across all modalities, see +[Supported Models](https://github.com/vllm-project/vllm-omni/tree/main/docs/models/supported_models.md). + +## Supported Models + +| Model | HuggingFace repo | Voice cloning | Streaming | Voice presets / upload | Gradio demo | +|---|---|---|---|---|---| +| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ | +| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — | +| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | ✓ (Base) | ✓ (PCM + WebSocket) | ✓ (presets + `/v1/audio/voices` upload) | ✓ (standard + FastRTC) | +| VoxCPM | local model dir | ✓ | ✓ (PCM stream) | — | — | +| VoxCPM2 | `openbmb/VoxCPM2` | ✓ | ✓ (AudioWorklet via gradio) | — | ✓ | +| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✓ (gated upstream) | ✓ | ✓ (presets) | ✓ | + +CosyVoice3 is intentionally absent: no online example exists for it yet. See its [offline section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#cosyvoice3) instead. + +## Common Quick Start + +Launch the server (defaults shown — adjust `--port`, `--gpu-memory-utilization`, etc. as needed): + +```bash +vllm serve --omni --port 8091 +``` + +Send a TTS request via curl: + +```bash +curl -X POST http://localhost:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello, how are you?", + "voice": "default", + "response_format": "wav" + }' --output output.wav +``` + +Or via Python httpx: + +```python +import httpx + +response = httpx.post( + "http://localhost:8091/v1/audio/speech", + json={ + "input": "Hello, how are you?", + "voice": "default", + "response_format": "wav", + }, + timeout=300.0, +) +open("output.wav", "wb").write(response.content) +``` + +Or via the OpenAI SDK: + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8091/v1", api_key="none") +response = client.audio.speech.create( + model="", + voice="default", + input="Hello, how are you?", +) +response.stream_to_file("output.wav") +``` + +Streaming PCM output (where supported) — set `stream=true` with `response_format="pcm"`: + +```bash +curl -X POST http://localhost:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello, how are you?", + "voice": "default", + "stream": true, + "response_format": "pcm" + }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 - +``` + +Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others). + +For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md). + +--- + +## Fish Speech S2 Pro + +4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec. + +### Prerequisites +```bash +pip install fish-speech +``` + +### Launch +```bash +vllm serve fishaudio/s2-pro --omni --port 8091 +# or: +./fish_speech/run_server.sh +``` +The deploy config auto-loads from `vllm_omni/deploy/fish_qwen3_omni.yaml` (the HF `model_type` on the fishaudio checkpoint is `fish_qwen3_omni`). + +### Voice cloning +```bash +curl -X POST http://localhost:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello, this is a cloned voice.", + "voice": "default", + "ref_audio": "https://example.com/reference.wav", + "ref_text": "Transcript of the reference audio." + }' --output cloned.wav +``` + +### CLI client +```bash +cd examples/online_serving/text_to_speech/fish_speech +python speech_client.py --text "Hello, how are you?" +python speech_client.py --text "Hello world" --stream --output output.pcm +``` + +### Gradio demo +```bash +./fish_speech/run_gradio_demo.sh # launches server + Gradio +python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server already running +``` + +### Notes +- Output: 44.1 kHz mono. +- Streaming PCM player command must use `-r 44100`. + +--- + +## OmniVoice + +Zero-shot multilingual TTS (600+ languages). Online serving currently exposes **auto voice** only; voice cloning and voice design are available offline. + +### Prerequisites +```bash +huggingface-cli download k2-fsa/OmniVoice +``` +Voice cloning (offline) needs `transformers>=5.3.0`; auto voice works with `transformers>=4.57.0`. + +### Launch +```bash +vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code +# or: +./omnivoice/run_server.sh +``` + +### CLI client +```bash +cd examples/online_serving/text_to_speech/omnivoice +python speech_client.py --text "Hello, how are you?" +python speech_client.py --text "Bonjour, comment allez-vous?" --language French +``` + +The client supports `--api-base`, `--model`, `--text`, `--response-format`, `--language`, `--output`. + +### Notes +- Voice cloning and voice design require offline inference; see the [offline OmniVoice section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#omnivoice). + +--- + +## Qwen3-TTS + +Three model variants exposed via separate checkpoints: + +| Variant | HF repo | Use | +|---|---|---| +| CustomVoice | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | Predefined speakers (`vivian`, `ryan`, …) with optional style instructions | +| VoiceDesign | `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | Natural-language voice style description | +| Base | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Voice cloning from a reference audio | + +Each variant ships smaller `0.6B` companions where available. + +### Launch +```bash +vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091 +# or: +./qwen3_tts/run_server.sh # default: CustomVoice +./qwen3_tts/run_server.sh VoiceDesign +./qwen3_tts/run_server.sh Base +``` + +### Choosing an executor backend (uniproc vs mp) +Stage configs ship with the chunked-streaming default. To use the uniproc executor (lower IPC overhead for the Base cloning task), pass `--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_uniproc.yaml`. See [#2603](https://github.com/vllm-project/vllm-omni/issues/2603) and [#2604](https://github.com/vllm-project/vllm-omni/pull/2604) for the full investigation. + +To opt out of chunked streaming, pass `--no-async-chunk` instead — the pipeline auto-dispatches to the end-to-end codec processor. + +### Sending requests +```bash +# CustomVoice with a predefined speaker +python qwen3_tts/openai_speech_client.py \ + --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ + --text "今天天气真好" \ + --voice ryan \ + --instructions "用开心的语气说" + +# VoiceDesign with a style description +python qwen3_tts/openai_speech_client.py \ + --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \ + --task-type VoiceDesign \ + --text "哥哥，你回来啦" \ + --instructions "体现撒娇稚嫩的萝莉女声，音调偏高" + +# Base voice cloning +python qwen3_tts/openai_speech_client.py \ + --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \ + --task-type Base \ + --text "Hello, this is a cloned voice" \ + --ref-audio /path/to/reference.wav \ + --ref-text "Original transcript of the reference audio" +``` + +### Voices endpoint +List available voices, or upload a custom one for Base cloning: +```bash +# List +curl http://localhost:8091/v1/audio/voices + +# Upload +curl -X POST http://localhost:8091/v1/audio/voices \ + -F "audio_sample=@/path/to/voice_sample.wav" \ + -F "consent=user_consent_id" \ + -F "name=custom_voice_1" \ + -F "ref_text=The exact transcript of the audio sample." \ + -F "speaker_description=warm narrator" +``` +Uploaded voices are then usable as `voice="custom_voice_1"` on subsequent requests. + +### Streaming PCM +```bash +curl -X POST http://localhost:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello, how are you?", + "voice": "vivian", + "language": "English", + "stream": true, + "response_format": "pcm" + }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 - +``` +Streaming requires `response_format="pcm"` and `async_chunk: true` on the stage config (default in `qwen3_tts.yaml`). `speed` is not supported when streaming. + +### Streaming WebSocket +The `/v1/audio/speech/stream` endpoint accepts text incrementally, splits it at sentence boundaries, and emits one PCM stream per sentence: +```bash +python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you? I am fine." +python qwen3_tts/streaming_speech_client.py --text "..." --simulate-stt --stt-delay 0.1 +``` + +### Gradio demos +```bash +./qwen3_tts/run_gradio_demo.sh # CustomVoice (default) +./qwen3_tts/run_gradio_demo.sh --task-type VoiceDesign +./qwen3_tts/run_gradio_demo.sh --task-type Base + +# FastRTC variant (gapless WebRTC streaming): +pip install fastrtc +python qwen3_tts/gradio_fastrtc_demo.py --api-base http://localhost:8000 +``` + +### Speaker embedding interpolation +`qwen3_tts/speaker_embedding_interpolation.py` blends two predefined speakers' embeddings to produce intermediate voices. See the script for usage. + +### Batch client +`qwen3_tts/batch_speech_client.py` issues many concurrent requests for throughput measurement. + +### Notes +- Base voice cloning has uniproc-vs-mp tradeoffs depending on per-request reference audio cost; see the executor-backend section above. +- `vllm_omni/deploy/qwen3_tts.yaml` is the default deploy config (loaded by HF `model_type`); per-stage runtime overrides are available via `--stage-N- `. + +--- + +## VoxCPM + +Split-stage TTS at 24 kHz. + +### Prerequisites +```bash +pip install voxcpm +# or use a local source tree: +export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src +``` + +If the native VoxCPM `config.json` lacks HF `model_type`, set up an HF-compatible config dir: +```bash +export VOXCPM_MODEL=/path/to/voxcpm-model +export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config +mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH" +cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json" +cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true +python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)' +``` + +### Launch +```bash +export VOXCPM_MODEL=/path/to/voxcpm-model +./voxcpm/run_server.sh # async-chunk streaming (default) +./voxcpm/run_server.sh sync # non-streaming +``` +Or directly: +```bash +vllm serve "$VOXCPM_MODEL" \ + --stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \ + --trust-remote-code --enforce-eager --omni --port 8091 +``` + +### Sending requests +```bash +# Basic TTS +python voxcpm/openai_speech_client.py \ + --model "$VOXCPM_MODEL" \ + --text "This is a VoxCPM online text-to-speech example." + +# Voice cloning +python voxcpm/openai_speech_client.py \ + --model "$VOXCPM_MODEL" \ + --text "This sentence is synthesized with a cloned voice." \ + --ref-audio /path/to/reference.wav \ + --ref-text "The exact transcript spoken in reference.wav." + +# Streaming PCM +python voxcpm/openai_speech_client.py \ + --model "$VOXCPM_MODEL" \ + --text "This is a streaming VoxCPM request." \ + --stream --output voxcpm_stream.pcm +``` + +### Notes +- `voxcpm.yaml` for one-shot decode; `voxcpm_async_chunk.yaml` for single-request streaming. Do not use the async-chunk config for concurrent requests or `/v1/audio/speech/batch`. +- Generic TTS fields not supported by VoxCPM: `voice`, `instructions`, `language`, `speaker_embedding`, `x_vector_only_mode`. +- For benchmark measurement, see [`benchmarks/voxcpm`](https://github.com/vllm-project/vllm-omni/tree/main/benchmarks/voxcpm/README.md). + +--- + +## VoxCPM2 + +Single-stage native AR TTS at 48 kHz. + +### Launch +```bash +vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000 +``` +Deploy config auto-loads from `vllm_omni/deploy/voxcpm2.yaml`. Pass `--deploy-config ` to override or `--stage-N- ` for per-stage runtime tweaks. + +### Sending requests +```bash +# Zero-shot synthesis +python voxcpm2/openai_speech_client.py --text "Hello, this is VoxCPM2." + +# Voice cloning +python voxcpm2/openai_speech_client.py \ + --text "This should sound like the reference speaker." \ + --ref-audio /path/to/reference.wav +``` +The `ref_audio` field accepts local file paths (auto-base64), HTTP URLs, or `data:audio/wav;base64,...` data URIs. + +### Gradio demo (gapless streaming via AudioWorklet) +```bash +python voxcpm2/gradio_demo.py +``` +Uses an AudioWorklet-based player adapted from the Qwen3-TTS demo for gap-free playback. Audio is streamed from the OpenAI Speech endpoint with `stream=true`. + +--- + +## Voxtral TTS + +Voxtral-4B-TTS (Mistral). Uses the `mistral_common` `SpeechRequest` protocol; voice presets are model-specific. + +### Prerequisites +Latest `mistral_common` with `SpeechRequest` support: +```bash +pip install -e /path/to/mistral-common # or upgrade from PyPI when available +``` + +### Launch +```bash +vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091 +``` +Deploy config auto-loads from `vllm_omni/deploy/voxtral_tts.yaml`. + +### Gradio demo +```bash +python voxtral_tts/gradio_demo.py +``` +The demo handles voice-preset selection and reference-audio upload. `voxtral_tts/text_preprocess.py` provides the text-normalization helpers used by the demo (also available for other clients). + +### Notes +- Voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`). +- Voice cloning is gated upstream and may require a recent `mistral_common`. +- A standalone CLI client is not yet shipped; the gradio demo is the canonical reference for now. + +## Example materials + +??? abstract "fish_speech/gradio_demo.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/fish_speech/gradio_demo.py" + `````` +??? abstract "fish_speech/run_gradio_demo.sh" + ``````sh + --8<-- "examples/online_serving/text_to_speech/fish_speech/run_gradio_demo.sh" + `````` +??? abstract "fish_speech/run_server.sh" + ``````sh + --8<-- "examples/online_serving/text_to_speech/fish_speech/run_server.sh" + `````` +??? abstract "fish_speech/speech_client.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/fish_speech/speech_client.py" + `````` +??? abstract "omnivoice/run_server.sh" + ``````sh + --8<-- "examples/online_serving/text_to_speech/omnivoice/run_server.sh" + `````` +??? abstract "omnivoice/speech_client.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/omnivoice/speech_client.py" + `````` +??? abstract "qwen3_tts/batch_speech_client.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/qwen3_tts/batch_speech_client.py" + `````` +??? abstract "qwen3_tts/gradio_demo.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/qwen3_tts/gradio_demo.py" + `````` +??? abstract "qwen3_tts/openai_speech_client.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/qwen3_tts/openai_speech_client.py" + `````` +??? abstract "qwen3_tts/run_gradio_demo.sh" + ``````sh + --8<-- "examples/online_serving/text_to_speech/qwen3_tts/run_gradio_demo.sh" + `````` +??? abstract "qwen3_tts/run_server.sh" + ``````sh + --8<-- "examples/online_serving/text_to_speech/qwen3_tts/run_server.sh" + `````` +??? abstract "qwen3_tts/speaker_embedding_interpolation.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/qwen3_tts/speaker_embedding_interpolation.py" + `````` +??? abstract "qwen3_tts/streaming_speech_client.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/qwen3_tts/streaming_speech_client.py" + `````` +??? abstract "qwen3_tts/tts_common.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/qwen3_tts/tts_common.py" + `````` +??? abstract "voxcpm/openai_speech_client.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/voxcpm/openai_speech_client.py" + `````` +??? abstract "voxcpm/run_server.sh" + ``````sh + --8<-- "examples/online_serving/text_to_speech/voxcpm/run_server.sh" + `````` +??? abstract "voxcpm2/gradio_demo.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py" + `````` +??? abstract "voxcpm2/openai_speech_client.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py" + `````` +??? abstract "voxtral_tts/gradio_demo.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py" + `````` +??? abstract "voxtral_tts/text_preprocess.py" + ``````py + --8<-- "examples/online_serving/text_to_speech/voxtral_tts/text_preprocess.py" + `````` diff --git a/examples/offline_inference/cosyvoice3/README.md b/examples/offline_inference/cosyvoice3/README.md deleted file mode 100644 index 704b49614..000000000 --- a/examples/offline_inference/cosyvoice3/README.md +++ /dev/null @@ -1,71 +0,0 @@ -# CosyVoice3 - -## Setup - -Install dependencies: -``` -uv pip install -e . -``` - -> **Note:** This includes required libraries such as `soundfile`, -> `onnxruntime`, `x-transformers`, and `einops` via -> `requirements/common.txt` and platform-specific requirements files. - -Download the model snapshot: -``` -from huggingface_hub import snapshot_download -snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B') -``` - -Add `config.json` in `pretrained_models/Fun-CosyVoice3-0.5B/`: -```json -{ - "model_type": "cosyvoice3", - "architectures": [ - "CosyVoice3Model" - ] -} -``` - -> **Why `config.json` is required:** -> `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers a class mapping. -> The loader still needs `model_type: "cosyvoice3"` from `config.json` to select that class. -> If no `config.json` is present, model type cannot be inferred automatically. -> If your downloaded checkpoint already includes a valid `config.json` with -> `model_type: "cosyvoice3"`, this manual step can be skipped. - -Run the offline verification script: -``` -python examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py \ - --model pretrained_models/Fun-CosyVoice3-0.5B \ - --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN -``` - -## Implementation Overview - -CosyVoice3 runs as a **2-stage Omni pipeline**. - -- **Stage 0** (`talker`) converts text + prompt audio to speech tokens. -- **Stage 1** (`code2wav`) converts speech tokens + prompt features to waveform via flow matching and HiFiGAN. - -Key components live in `vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py`. - -- `CosyVoice3MultiModalProcessor` builds the multimodal inputs: - - Tokenizes `prompt` and `prompt_text`. - - Extracts speech tokens and mel features from the prompt audio. - - Extracts a speaker embedding. -- `CosyVoice3Model` implements both stages: - - Stage 0 uses `CosyVoice3LM` and outputs speech tokens + conditioning features. - - Stage 1 runs the flow model (DiT-based CFM) and HiFiGAN to synthesize waveform. - -Pipeline topology lives in `vllm_omni/model_executor/models/cosyvoice3/pipeline.py`; -runtime tunables (batch size, memory limits, sampling) live in -`vllm_omni/deploy/cosyvoice3.yaml`. The deploy config auto-loads by -HF `model_type` and defaults to `async_chunk: true` (shared-memory -streaming). Pass `--no-async-chunk` on `vllm serve` to switch to the -legacy sync path where stage 1 runs `text2flow` over the full -speech-token sequence. - -- Stage 0 emits latent speech tokens. -- Stage 1 consumes them via `sync_process_input_func` (sync mode) or the - shared-memory connector (async-chunk mode) and outputs audio. diff --git a/examples/offline_inference/fish_speech/README.md b/examples/offline_inference/fish_speech/README.md deleted file mode 100644 index 9af853096..000000000 --- a/examples/offline_inference/fish_speech/README.md +++ /dev/null @@ -1,75 +0,0 @@ -# Fish Speech S2 Pro - -This directory contains an offline demo for running Fish Speech S2 Pro (`fishaudio/s2-pro`) with vLLM Omni. It supports text-only synthesis and voice cloning with reference audio. - -## Model Overview - -[Fish Speech S2 Pro](https://huggingface.co/fishaudio/s2-pro) is a 4B dual-AR text-to-speech model by FishAudio. It uses: - -- **Slow AR**: Generates semantic tokens autoregressively (Qwen3-based backbone) -- **Fast AR**: Predicts residual codebook tokens from semantic tokens -- **DAC Decoder**: Converts codec codes to 44.1 kHz audio waveform - -The model produces high-quality speech with voice cloning capabilities. - -## Prerequisites - -Install the `fish-speech` package for the DAC codec: - -```bash -pip install fish-speech -``` - -## Setup - -Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. - -## Quick Start - -### Text-Only Synthesis - -```bash -python end2end.py --text "Hello, this is a test of the Fish Speech text to speech system." -``` - -Generated audio files are saved to `output_audio/` by default. - -### Voice Cloning - -Provide a reference audio file and its transcript to clone a voice: - -```bash -python end2end.py \ - --text "Hello, this is a cloned voice." \ - --ref-audio /path/to/reference.wav \ - --ref-text "Transcript of the reference audio." -``` - -## Streaming Mode - -Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config): - -```bash -python end2end.py --text "Hello, this is a test." --streaming -``` - -Each DAC decoder chunk is logged as it arrives. The final WAV file is written once generation completes. - -## Configuration - -| Argument | Default | Description | -|----------|---------|-------------| -| `--text` | `"Hello, this is a test..."` | Text to synthesize | -| `--ref-audio` | None | Path to reference audio for voice cloning | -| `--ref-text` | None | Transcript of the reference audio | -| `--model` | `fishaudio/s2-pro` | HuggingFace model path | -| `--stage-configs-path` | `fish_speech_s2_pro.yaml` | Path to stage configs YAML | -| `--output-dir` | `output_audio` | Output directory for WAV files | -| `--streaming` | False | Enable streaming via AsyncOmni | -| `--stage-init-timeout` | 600 | Stage initialization timeout (seconds) | - -## Notes - -- Output audio is 44.1 kHz mono WAV. -- The DAC codec weights (`codec.pth`) are loaded lazily from the model directory. -- Voice cloning encodes the reference audio using the DAC codec to extract semantic codes, then prepends them as a system message. diff --git a/examples/offline_inference/omnivoice/README.md b/examples/offline_inference/omnivoice/README.md deleted file mode 100644 index d804b61b5..000000000 --- a/examples/offline_inference/omnivoice/README.md +++ /dev/null @@ -1,73 +0,0 @@ -# OmniVoice - -This directory contains an offline demo for running OmniVoice TTS models with vLLM Omni. It generates speech from text and saves WAV files locally. - -## Model Overview - -[OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) is a zero-shot multilingual TTS model supporting 600+ languages. It uses a diffusion language model (Qwen3-0.6B backbone) with iterative masked unmasking to generate speech. - -Three inference modes are supported: - -- **Auto Voice**: Generate speech without any reference — the model picks a voice automatically. -- **Voice Clone**: Clone a voice from a reference audio + transcription. -- **Voice Design**: Control voice style via natural language instruction (e.g., "female, low pitch, british accent"). - -## Setup - -Ensure the model is downloaded: - -```bash -huggingface-cli download k2-fsa/OmniVoice -``` - -> **Note:** Voice cloning requires `transformers>=5.3.0` for `HiggsAudioV2TokenizerModel`. Auto voice and voice design modes work with `transformers>=4.57.0`. - -## Quick Start - -Auto voice (text only): - -```bash -python end2end.py --model k2-fsa/OmniVoice --text "Hello, this is a test." -``` - -Voice design (with style instruction): - -```bash -python end2end.py --model k2-fsa/OmniVoice \ - --text "Hello, this is a test." \ - --instruct "female, low pitch, british accent" -``` - -Voice clone (with reference audio): - -```bash -python end2end.py --model k2-fsa/OmniVoice \ - --text "Hello, this is a test." \ - --ref-audio ref.wav \ - --ref-text "This is the reference transcription." -``` - -## Language Support - -Specify a language for improved quality: - -```bash -python end2end.py --model k2-fsa/OmniVoice \ - --text "你好，这是一个测试。" \ - --lang zh -``` - -## Architecture - -OmniVoice uses a two-stage pipeline: - -- **Stage 0 (Generator)**: Qwen3-0.6B transformer with 32-step iterative unmasking and classifier-free guidance. Generates 8-codebook audio tokens from text. -- **Stage 1 (Decoder)**: HiggsAudioV2 RVQ quantizer + DAC acoustic decoder. Converts tokens to 24kHz waveform. - -Both stages use `GPUGenerationWorker` with `OmniGenerationScheduler`. - -## Notes - -- Output audio is saved to `output.wav` by default. Use `--output` to change the path. -- The model estimates duration from text automatically via `RuleDurationEstimator`. -- Use `--stage-init-timeout` to increase the stage initialization timeout for first-time model downloads. diff --git a/examples/offline_inference/qwen3_tts/README.md b/examples/offline_inference/qwen3_tts/README.md deleted file mode 100644 index 2971ad716..000000000 --- a/examples/offline_inference/qwen3_tts/README.md +++ /dev/null @@ -1,121 +0,0 @@ -# Qwen3-TTS - -This directory contains an offline demo for running Qwen3 TTS models with vLLM Omni. It builds task-specific inputs and generates WAV files locally. - -## Model Overview - -Qwen3 TTS provides multiple task variants for speech generation: - -- **CustomVoice**: Generate speech with a known speaker identity (speaker ID) and optional instruction. -- **VoiceDesign**: Generate speech from text plus a descriptive instruction that designs a new voice. -- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. The `ref_audio` field accepts a local file path, HTTP URL, or base64 data URL. - -## Setup -Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. - -### ROCm Dependencies - -You will need to install the dependency `onnxruntime-rocm`. - -``` -pip uninstall onnxruntime # should be removed before we can install onnxruntime-rocm -pip install onnxruntime-rocm -``` - -## Quick Start - -Run a single sample for a task: - -``` -python end2end.py --query-type CustomVoice -``` - -Generated audio files are saved to `output_audio/` by default. - -## Task Usage - -### CustomVoice - -Single sample: - -``` -python end2end.py --query-type CustomVoice -``` - -Batch sample (multiple prompts in one run): - -``` -python end2end.py --query-type CustomVoice --use-batch-sample -``` - -### VoiceDesign - -Single sample: - -``` -python end2end.py --query-type VoiceDesign -``` - -Batch sample: - -``` -python end2end.py --query-type VoiceDesign --use-batch-sample -``` - -### Base (Voice Clone) - -Single sample: - -``` -python end2end.py --query-type Base -``` - -Batch sample: - -``` -python end2end.py --query-type Base --use-batch-sample -``` - -Mode selection for Base: - -- `--mode-tag icl` (default): standard mode -- `--mode-tag xvec_only`: enable `x_vector_only_mode` in the request - -Examples: - -``` -python end2end.py --query-type Base --mode-tag icl -``` - -## Streaming Mode - -Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config): - -```bash -python end2end.py --query-type CustomVoice --streaming --output-dir /tmp/out_stream -``` - -Each Code2Wav chunk is logged as it arrives (default 25 frames; configurable via `codec_chunk_frames` -in the stage config). The initial chunk size is dynamically selected based on server load for reduced -TTFA, and can be overridden per-request via the `initial_codec_chunk_frames` API field. The final WAV file is written once generation -completes. This demonstrates that audio data is available progressively rather than only at the end. - -> **Note:** Streaming uses `AsyncOmni` internally. The non-streaming path (`Omni`) is unchanged. - -## Batched Decoding - -The Code2Wav stage (stage 1) supports batched decoding, where multiple requests are decoded in a single forward pass through the SpeechTokenizer. To use it, set `max_num_seqs > 1` on both stages via `--stage-overrides` and pass multiple prompts via `--txt-prompts` with a matching `--batch-size`. - -``` -python end2end.py --query-type CustomVoice \ - --txt-prompts benchmark_prompts.txt \ - --batch-size 4 \ - --stage-overrides '{"0":{"max_num_seqs":4,"gpu_memory_utilization":0.2},"1":{"max_num_seqs":4,"gpu_memory_utilization":0.2}}' -``` - -**Important:** `--batch-size` must match a CUDA graph capture size (1, 2, 4, 8, 16...) because the Talker's code predictor KV cache is sized to `max_num_seqs`, and CUDA graphs pad the batch to the next capture size. Both stages need `max_num_seqs >= batch_size` in the stage config for batching to take effect. If only stage 1 has a higher `max_num_seqs`, it won't help — stage 1 can only batch chunks from requests that are in-flight simultaneously, which requires stage 0 to also process multiple requests concurrently. - -## Notes - -- The script uses the model paths embedded in `end2end.py`. Update them if your local cache path differs. -- Use `--output-dir` to change the output folder. diff --git a/examples/offline_inference/text_to_speech/README.md b/examples/offline_inference/text_to_speech/README.md new file mode 100644 index 000000000..a457c6c0a --- /dev/null +++ b/examples/offline_inference/text_to_speech/README.md @@ -0,0 +1,367 @@ +# Text-To-Speech + +vLLM-Omni supports several autoregressive TTS models. They share a mostly +common CLI shape (`--text`, `--ref-audio`, `--ref-text`, plus an +output-path flag — `--output-dir` for most, `--output` for OmniVoice) and +live together in this hub. Each model has its own subdirectory containing +a single `end2end.py` script; this README is the single doc entry point. + +For online serving, see [`examples/online_serving/text_to_speech/`](../../online_serving/text_to_speech/README.md). For the full +list of supported architectures across all modalities, see +[Supported Models](../../../docs/models/supported_models.md). + +## Supported Models + +| Model | HuggingFace repo | Stages | Voice cloning | Streaming | Special modes | Sample rate | +|---|---|---|---|---|---|---| +| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ | — | 22.05 kHz | +| Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ | — | 44.1 kHz | +| OmniVoice | `k2-fsa/OmniVoice` | 2 (gen + dec) | ✓ | — | voice design, language hint | 24 kHz | +| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | 2 (talker + code2wav) | ✓ (Base) | ✓ | 3 task variants | 24 kHz | +| VoxCPM | local model dir | split | ✓ | ✓ | — | 24 kHz | +| VoxCPM2 | `openbmb/VoxCPM2` | single (native AR) | ✓ | ✓ (online) | continuation | 48 kHz | +| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | varies | ✓ | ✓ | voice presets | 24 kHz | + +## Common Quick Start + +Most models share this invocation shape: + +```bash +python examples/offline_inference/text_to_speech//end2end.py \ + --text "Hello, this is a test." \ + --ref-audio /path/to/reference.wav \ + --ref-text "Transcript of the reference audio." +``` + +`--ref-audio` and `--ref-text` are optional (text-only synthesis works without them) and must be provided together for voice cloning. The exotic scripts — Qwen3-TTS, Voxtral TTS, CosyVoice3 — accept additional model-specific flags documented in their per-model section below. Qwen3-TTS in particular uses its own argparse surface (`--query-type`, `--audio-path`, etc.) and does not follow the common shape; see its section. + +--- + +## CosyVoice3 + +2-stage TTS pipeline (`talker` + `code2wav`) at 22.05 kHz. + +### Prerequisites +```bash +uv pip install -e . +# Includes soundfile, onnxruntime, x-transformers, einops via requirements. +``` + +Download the model snapshot: +```python +from huggingface_hub import snapshot_download +snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', + local_dir='pretrained_models/Fun-CosyVoice3-0.5B') +``` + +If your downloaded checkpoint lacks `config.json`, add it: +```json +{ + "model_type": "cosyvoice3", + "architectures": ["CosyVoice3Model"] +} +``` +This is required because `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers the class mapping; the loader still reads `model_type` from `config.json` to select the class. + +### Quick start +```bash +python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \ + --model pretrained_models/Fun-CosyVoice3-0.5B \ + --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN +``` + +### Voice cloning +Pass a reference audio. Note that CosyVoice3's `--prompt-text` is a system-style prompt for the GPT stage, not a reference transcript: +```bash +python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \ + --model pretrained_models/Fun-CosyVoice3-0.5B \ + --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN \ + --ref-audio prompt.wav \ + --prompt-text "You are a helpful assistant.<|endofprompt|>Testing my voices. Why should I not?" +``` + +### Streaming +Streaming is enabled by default via `async_chunk: true` in `vllm_omni/deploy/cosyvoice3.yaml`. Pass `--no-async-chunk` on `vllm serve` to switch to the legacy synchronous path. + +### Notes +- Stage 0 (`talker`) emits speech tokens; stage 1 (`code2wav`) runs flow matching + HiFiGAN to synthesize waveform. +- Deploy config auto-loads from `vllm_omni/deploy/cosyvoice3.yaml` based on HF `model_type`. Pass `--deploy-config ` to override. + +--- + +## Fish Speech S2 Pro + +4B dual-AR text-to-speech model from FishAudio with the DAC codec at 44.1 kHz. + +### Prerequisites +```bash +pip install fish-speech +``` + +### Quick start +```bash +python examples/offline_inference/text_to_speech/fish_speech/end2end.py \ + --text "Hello, this is a test of the Fish Speech text to speech system." +``` + +### Voice cloning +```bash +python examples/offline_inference/text_to_speech/fish_speech/end2end.py \ + --text "Hello, this is a cloned voice." \ + --ref-audio /path/to/reference.wav \ + --ref-text "Transcript of the reference audio." +``` + +### Streaming +```bash +python examples/offline_inference/text_to_speech/fish_speech/end2end.py \ + --text "Hello, this is a streaming test." \ + --streaming +``` +Streaming requires `async_chunk: true` in the stage config. + +### Notes +- Output: 44.1 kHz mono WAV. +- DAC codec weights (`codec.pth`) are loaded lazily from the model directory. + +--- + +## OmniVoice + +Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design). + +### Prerequisites +```bash +huggingface-cli download k2-fsa/OmniVoice +``` +Voice cloning requires `transformers>=5.3.0`. Auto and design modes work with `transformers>=4.57.0`. + +### Quick start (auto voice) +```bash +python examples/offline_inference/text_to_speech/omnivoice/end2end.py \ + --model k2-fsa/OmniVoice \ + --text "Hello, this is a test." +``` + +### Voice cloning +```bash +python examples/offline_inference/text_to_speech/omnivoice/end2end.py \ + --model k2-fsa/OmniVoice \ + --text "Hello, this is a test." \ + --ref-audio ref.wav \ + --ref-text "This is the reference transcription." +``` + +### Voice design +```bash +python examples/offline_inference/text_to_speech/omnivoice/end2end.py \ + --model k2-fsa/OmniVoice \ + --text "Hello, this is a test." \ + --instruct "female, low pitch, british accent" +``` + +### Language hint +```bash +python examples/offline_inference/text_to_speech/omnivoice/end2end.py \ + --model k2-fsa/OmniVoice \ + --text "你好，这是一个测试。" \ + --lang zh +``` + +### Notes +- Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking. +- Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz. + +--- + +## Qwen3-TTS + +3-task-variant TTS with 24 kHz output. Has its own argparse surface (this script does not follow the common `--text` / `--ref-audio` shape). + +### Prerequisites +For ROCm builds, replace `onnxruntime` with `onnxruntime-rocm`: +```bash +pip uninstall onnxruntime +pip install onnxruntime-rocm +``` + +### Task variants +- `CustomVoice`: predefined speaker (speaker ID) with optional style instruction. +- `VoiceDesign`: text + descriptive instruction designs a new voice. +- `Base`: voice cloning from reference audio + transcript. + +```bash +# Single sample +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type VoiceDesign +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base + +# Base with a custom reference audio (Qwen3-TTS uses --audio-path, not --ref-audio): +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \ + --query-type Base --audio-path /path/to/reference.wav + +# Base variant has an additional mode flag: +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag icl # default +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --mode-tag xvec_only # x_vector_only_mode + +# Batch (multiple prompts in one run) +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice --use-batch-sample +``` + +### Streaming +```bash +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \ + --query-type CustomVoice \ + --streaming \ + --output-dir /tmp/out_stream +``` +Streaming requires `async_chunk: true` in the stage config. + +### Batched decoding +The Code2Wav stage supports batched decoding through the SpeechTokenizer. Pass multiple prompts via `--txt-prompts` and set `--batch-size` accordingly. To raise `max_num_seqs` on either stage, point `--stage-configs-path` at a stage configs YAML with the desired values (see `vllm_omni/model_executor/stage_configs/` for templates): +```bash +python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \ + --query-type CustomVoice \ + --txt-prompts examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt \ + --batch-size 4 \ + --stage-configs-path /path/to/qwen3_tts_batched.yaml +``` +`--batch-size` must match a CUDA-graph capture size (1, 2, 4, 8, 16…). + +### Notes +- Run `--help` for the full argument surface. +- See `qwen3_tts/end2end.py` for the prompt-length-estimation logic the Talker uses. + +--- + +## VoxCPM + +Split-stage TTS. The hub example covers single TTS, single voice cloning, and streaming. Use `benchmarks/voxcpm/` for warmup, batch JSONL prompts, profiler injection, and offline TTFP / RTF measurement. + +### Prerequisites +```bash +pip install voxcpm soundfile +# or use a local source tree: +export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src +``` + +If the native VoxCPM `config.json` does not contain HF metadata such as `model_type`, prepare a persistent HF-compatible config directory and point the stage configs to it via `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH`: + +```bash +export VOXCPM_MODEL=/path/to/voxcpm-model +export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config +mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH" +cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json" +cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true +python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)' +``` + +### Quick start +```bash +python examples/offline_inference/text_to_speech/voxcpm/end2end.py \ + --model "$VOXCPM_MODEL" \ + --text "This is a split-stage VoxCPM synthesis example running on vLLM Omni." +``` + +### Voice cloning +```bash +python examples/offline_inference/text_to_speech/voxcpm/end2end.py \ + --model "$VOXCPM_MODEL" \ + --text "This sentence is synthesized with a cloned voice." \ + --ref-audio /path/to/reference.wav \ + --ref-text "The exact transcript spoken in reference.wav." +``` + +### Streaming +Pass the async-chunk stage config: +```bash +python examples/offline_inference/text_to_speech/voxcpm/end2end.py \ + --model "$VOXCPM_MODEL" \ + --stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \ + --text "This is a split-stage VoxCPM streaming example running on vLLM Omni." +``` + +### Notes +- `voxcpm.yaml` is the default non-streaming stage config; `voxcpm_async_chunk.yaml` enables streaming. +- Streaming is currently single-request oriented. +- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades quality. +- For online serving, see the [VoxCPM section in the online hub](../../online_serving/text_to_speech/README.md#voxcpm). For benchmark reporting, see [`benchmarks/voxcpm`](../../../benchmarks/voxcpm/README.md). + +--- + +## VoxCPM2 + +Single-stage native AR TTS at 48 kHz. Pipeline: `feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE`. + +### Prerequisites +```bash +pip install voxcpm +# or, for a local source checkout: +export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm +``` + +### Quick start +```bash +python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \ + --model openbmb/VoxCPM2 \ + --text "Hello, this is a VoxCPM2 demo." +``` + +### Voice cloning +Pass a reference audio for isolated cloning, or both `--ref-audio` + `--ref-text` for prompt continuation: +```bash +python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \ + --text "Hello, this is a voice clone demo." \ + --ref-audio /path/to/reference.wav \ + --ref-text "Transcript of the reference audio." +``` + +### Streaming +Streaming is exposed through the online OpenAI Speech API (`stream=true`). See [`examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py`](../../online_serving/text_to_speech/voxcpm2/gradio_demo.py) for an AudioWorklet-based gapless streaming player; the offline `end2end.py` script does not expose a streaming path. + +### Notes +- Output: 48 kHz mono WAV. +- Deploy config: `vllm_omni/deploy/voxcpm2.yaml` (auto-loaded by HF `model_type`). + +--- + +## Voxtral TTS + +Voxtral-4B-TTS (Mistral). Has its own argparse surface; uses voice presets and the `mistral_common` `SpeechRequest` protocol. + +### Prerequisites +Latest `mistral_common` with `SpeechRequest` support: +```bash +pip install -e /path/to/mistral-common # or upgrade from PyPI when available +``` + +### Quick start (voice preset) +```bash +python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \ + --write-audio --voice cheerful_female \ + --model mistralai/Voxtral-4B-TTS-2603 \ + --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" +``` + +### Voice cloning (capability gated upstream) +```bash +python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \ + --write-audio \ + --model mistralai/Voxtral-4B-TTS-2603 \ + --text "This is a test message." \ + --ref-audio path/to/reference_audio.wav +``` + +### Streaming + concurrency +```bash +python examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \ + --num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \ + --model mistralai/Voxtral-4B-TTS-2603 \ + --text "..." +``` +Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`). + +### Notes +- `--num-prompts N` replicates the prompt for performance measurement. +- `--concurrency M` requires `--streaming` and must evenly divide `--num-prompts`. +- Run `--help` for the full argument surface. diff --git a/examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py b/examples/offline_inference/text_to_speech/cosyvoice3/end2end.py similarity index 91% rename from examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py rename to examples/offline_inference/text_to_speech/cosyvoice3/end2end.py index a5dc564ec..c3b80152f 100644 --- a/examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py +++ b/examples/offline_inference/text_to_speech/cosyvoice3/end2end.py @@ -31,13 +31,13 @@ def run_e2e(): help="Override the deploy config path. If unset, auto-loads " "vllm_omni/deploy/cosyvoice3.yaml based on the HF model_type.", ) - parser.add_argument("--prompt", type=str, default="Hello, this is a test of the CosyVoice system capability.") + parser.add_argument("--text", type=str, default="Hello, this is a test of the CosyVoice system capability.") parser.add_argument( "--prompt-text", type=str, default="You are a helpful assistant.<|endofprompt|>Testing my voices. Why should I not?", ) - parser.add_argument("--audio-path", type=str, default="prompt.wav") + parser.add_argument("--ref-audio", type=str, default="prompt.wav") parser.add_argument( "--tokenizer", type=str, @@ -64,11 +64,11 @@ def run_e2e(): sampling_cfg = {"top_p": 0.8, "top_k": 25, "eos_token_id": 6561 + 1} print("Model initialized. Preparing inputs...") - if args.audio_path: - if not os.path.exists(args.audio_path): - raise FileNotFoundError(f"Audio file not found: {args.audio_path}") + if args.ref_audio: + if not os.path.exists(args.ref_audio): + raise FileNotFoundError(f"Audio file not found: {args.ref_audio}") # Load at native sample rate - audio_signal, sr = load_audio(args.audio_path, sr=None) + audio_signal, sr = load_audio(args.ref_audio, sr=None) # Validate sample rate before processing (similar to original CosyVoice) min_sr = 16000 @@ -84,7 +84,7 @@ def run_e2e(): audio_data = AudioAsset("mary_had_lamb").audio_and_sample_rate prompts = { - "prompt": args.prompt, + "prompt": args.text, "multi_modal_data": { "audio": audio_data, }, @@ -94,7 +94,7 @@ def run_e2e(): }, } - print(f"Generating for prompt: {args.prompt}") + print(f"Generating for prompt: {args.text}") config = CosyVoice3Config() tokenizer = get_qwen_tokenizer( @@ -102,7 +102,7 @@ def run_e2e(): skip_special_tokens=config.skip_special_tokens, version=config.version, ) - _, text_token_len = extract_text_token(args.prompt, tokenizer, config.allowed_special) + _, text_token_len = extract_text_token(args.text, tokenizer, config.allowed_special) base_len = int(text_token_len) min_len = int(base_len * config.min_token_text_ratio) max_len = int(base_len * config.max_token_text_ratio) diff --git a/examples/offline_inference/fish_speech/end2end.py b/examples/offline_inference/text_to_speech/fish_speech/end2end.py similarity index 99% rename from examples/offline_inference/fish_speech/end2end.py rename to examples/offline_inference/text_to_speech/fish_speech/end2end.py index 60830d06b..237aa8351 100644 --- a/examples/offline_inference/fish_speech/end2end.py +++ b/examples/offline_inference/text_to_speech/fish_speech/end2end.py @@ -44,6 +44,7 @@ DEFAULT_STAGE_CONFIG = os.path.join( "..", "..", "..", + "..", "vllm_omni", "model_executor", "stage_configs", diff --git a/examples/offline_inference/omnivoice/end2end.py b/examples/offline_inference/text_to_speech/omnivoice/end2end.py similarity index 100% rename from examples/offline_inference/omnivoice/end2end.py rename to examples/offline_inference/text_to_speech/omnivoice/end2end.py diff --git a/examples/offline_inference/qwen3_tts/benchmark_prompts.txt b/examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt similarity index 100% rename from examples/offline_inference/qwen3_tts/benchmark_prompts.txt rename to examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt diff --git a/examples/offline_inference/qwen3_tts/end2end.py b/examples/offline_inference/text_to_speech/qwen3_tts/end2end.py similarity index 100% rename from examples/offline_inference/qwen3_tts/end2end.py rename to examples/offline_inference/text_to_speech/qwen3_tts/end2end.py diff --git a/examples/offline_inference/voxcpm/end2end.py b/examples/offline_inference/text_to_speech/voxcpm/end2end.py similarity index 99% rename from examples/offline_inference/voxcpm/end2end.py rename to examples/offline_inference/text_to_speech/voxcpm/end2end.py index 980410fea..d92fbc55c 100644 --- a/examples/offline_inference/voxcpm/end2end.py +++ b/examples/offline_inference/text_to_speech/voxcpm/end2end.py @@ -13,7 +13,7 @@ from vllm.utils.argparse_utils import FlexibleArgumentParser from vllm_omni import AsyncOmni, Omni -REPO_ROOT = Path(__file__).resolve().parents[3] +REPO_ROOT = Path(__file__).resolve().parents[4] DEFAULT_SYNC_STAGE_CONFIG = REPO_ROOT / "vllm_omni" / "model_executor" / "stage_configs" / "voxcpm.yaml" diff --git a/examples/offline_inference/voxcpm2/end2end.py b/examples/offline_inference/text_to_speech/voxcpm2/end2end.py similarity index 87% rename from examples/offline_inference/voxcpm2/end2end.py rename to examples/offline_inference/text_to_speech/voxcpm2/end2end.py index 6b6bf78dd..a08510faf 100644 --- a/examples/offline_inference/voxcpm2/end2end.py +++ b/examples/offline_inference/text_to_speech/voxcpm2/end2end.py @@ -16,7 +16,7 @@ from vllm.utils.argparse_utils import FlexibleArgumentParser from vllm_omni import Omni -REPO_ROOT = Path(__file__).resolve().parents[3] +REPO_ROOT = Path(__file__).resolve().parents[4] DEFAULT_STAGE_CONFIGS_PATH = str(REPO_ROOT / "vllm_omni" / "model_executor" / "stage_configs" / "voxcpm2.yaml") SAMPLE_RATE = 48_000 @@ -48,28 +48,16 @@ def parse_args(): help="Path to the stage config YAML file.", ) parser.add_argument( - "--reference-audio", + "--ref-audio", type=str, default=None, - help="Path to reference audio for voice cloning (isolated ref mode).", - ) - parser.add_argument( - "--prompt-audio", - type=str, - default=None, - help="Path to prompt audio for continuation mode (requires --prompt-text).", - ) - parser.add_argument( - "--prompt-text", - type=str, - default=None, - help="Text matching --prompt-audio for continuation mode.", + help="Path to reference audio for voice cloning.", ) parser.add_argument( "--ref-text", type=str, default=None, - help="Optional transcript of --reference-audio (enables ref_continuation mode).", + help="Optional transcript of --ref-audio (enables continuation mode).", ) return parser.parse_args() @@ -120,8 +108,8 @@ def main(): split_map = build_cjk_split_map(tokenizer) hf_config = engine.engine.stage_vllm_configs[0].model_config.hf_config - ref_audio_arg = args.reference_audio or args.prompt_audio - ref_text_arg = args.ref_text or args.prompt_text + ref_audio_arg = args.ref_audio + ref_text_arg = args.ref_text ref_wav, ref_sr = (None, None) if ref_audio_arg: ref_wav_arr, ref_sr = sf.read(ref_audio_arg) diff --git a/examples/offline_inference/voxtral_tts/end2end.py b/examples/offline_inference/text_to_speech/voxtral_tts/end2end.py similarity index 98% rename from examples/offline_inference/voxtral_tts/end2end.py rename to examples/offline_inference/text_to_speech/voxtral_tts/end2end.py index 0a6f88715..a193d9920 100644 --- a/examples/offline_inference/voxtral_tts/end2end.py +++ b/examples/offline_inference/text_to_speech/voxtral_tts/end2end.py @@ -241,7 +241,7 @@ def parse_args() -> Namespace: help="Text to synthesize.", ) parser.add_argument( - "--audio-path", + "--ref-audio", type=str, default=None, help="Path to reference audio file for voice cloning.", @@ -344,10 +344,10 @@ def main(args: Any) -> None: model_name = args.model output_dir = args.output_dir - if args.voice is None and args.audio_path is None: - raise ValueError("Either --voice or --audio-path must be provided.") + if args.voice is None and args.ref_audio is None: + raise ValueError("Either --voice or --ref-audio must be provided.") - audio_prompt_file = args.audio_path + audio_prompt_file = args.ref_audio text_chunk = TextChunk(text=args.text) if args.write_audio: diff --git a/examples/offline_inference/voxcpm/README.md b/examples/offline_inference/voxcpm/README.md deleted file mode 100644 index 1eaea9b0d..000000000 --- a/examples/offline_inference/voxcpm/README.md +++ /dev/null @@ -1,123 +0,0 @@ -# VoxCPM Offline Example - -This directory contains the minimal offline VoxCPM example for vLLM Omni. - -`end2end.py` is intentionally small and only covers: - -- single text-to-speech -- single voice cloning with `ref_audio` + `ref_text` -- non-streaming with `vllm_omni/model_executor/stage_configs/voxcpm.yaml` -- streaming with `vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml` - -Advanced workflows were moved out of the getting-started example: - -- `benchmarks/voxcpm/vllm_omni/bench_tts_offline.py`: warmup, batch prompts, profiler, offline TTFP / RTF -- `benchmarks/voxcpm/vllm_omni/run_offline_matrix.py`: fixed offline smoke matrix -- `benchmarks/voxcpm/`: benchmark scripts and benchmark docs - -## Prerequisites - -Install VoxCPM in one of these ways: - -```bash -pip install voxcpm -``` - -or point vLLM Omni to the local VoxCPM source tree: - -```bash -export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src -``` - -The example writes WAV files with `soundfile`: - -```bash -pip install soundfile -``` - -## Model Path - -Pass the native VoxCPM model directory directly: - -```bash -export VOXCPM_MODEL=/path/to/voxcpm-model -``` - -If the native VoxCPM `config.json` does not contain HuggingFace metadata such as -`model_type`, prepare a persistent HF-compatible config directory and point the -stage configs to it with `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH`: - -```bash -export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config -mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH" -cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json" -cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true -python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)' -``` - -If the model directory itself already has `model_type`, this extra directory is -not required. - -## Quick Start - -Single text-to-speech, non-streaming: - -```bash -python examples/offline_inference/voxcpm/end2end.py \ - --model "$VOXCPM_MODEL" \ - --text "This is a split-stage VoxCPM synthesis example running on vLLM Omni." -``` - -Single voice cloning, non-streaming: - -```bash -python examples/offline_inference/voxcpm/end2end.py \ - --model "$VOXCPM_MODEL" \ - --text "This sentence is synthesized with a cloned voice." \ - --ref-audio /path/to/reference.wav \ - --ref-text "The exact transcript spoken in reference.wav." -``` - -Streaming: - -```bash -python examples/offline_inference/voxcpm/end2end.py \ - --model "$VOXCPM_MODEL" \ - --stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \ - --text "This is a split-stage VoxCPM streaming example running on vLLM Omni." -``` - -By default, `end2end.py` writes to `output_audio/` for non-streaming and -`output_audio_streaming/` for streaming. - -## Advanced Workflows - -Use `benchmarks/voxcpm/vllm_omni/bench_tts_offline.py` when you need: - -- warmup runs -- prompt files -- batch JSONL inputs -- profiler injection -- offline TTFP / RTF emission - -Use `benchmarks/voxcpm/vllm_omni/run_offline_matrix.py` when you need the fixed offline smoke matrix that previously lived in `test.py`. - -Full matrix benchmark example: - -```bash -python benchmarks/voxcpm/vllm_omni/run_offline_matrix.py \ - --model "$VOXCPM_MODEL" \ - --ref-audio /path/to/reference.wav \ - --ref-text "The exact transcript spoken in reference.wav." -``` - -For online serving examples, see [examples/online_serving/voxcpm](../../online_serving/voxcpm/README.md). - -For benchmark reporting, see [benchmarks/voxcpm](../../../benchmarks/voxcpm/README.md). - -## Notes - -- `voxcpm.yaml` is the default non-streaming stage config. -- `voxcpm_async_chunk.yaml` is the streaming stage config. -- Streaming is currently single-request oriented; the fixed smoke matrix now lives in `benchmarks/voxcpm/vllm_omni/run_offline_matrix.py`. -- `ref_text` must be the real transcript of the reference audio. Mismatched text usually causes obvious quality degradation. diff --git a/examples/offline_inference/voxcpm2/README.md b/examples/offline_inference/voxcpm2/README.md deleted file mode 100644 index e98273079..000000000 --- a/examples/offline_inference/voxcpm2/README.md +++ /dev/null @@ -1,83 +0,0 @@ -# VoxCPM2 Offline Inference (Native AR) - -VoxCPM2 is a 2B-parameter tokenizer-free diffusion AR TTS model. It produces 48kHz audio and supports 30+ languages with a single-stage native AR pipeline backed by MiniCPM4. - -## Prerequisites - -Install the `voxcpm` package, or set the environment variable pointing to the source tree: - -```bash -# Option A: install package -pip install voxcpm - -# Option B: use source checkout -export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/voxcpm -``` - -## Quick Start - -Zero-shot synthesis: - -```bash -python examples/offline_inference/voxcpm2/end2end.py \ - --model openbmb/VoxCPM2 \ - --text "Hello, this is a VoxCPM2 demo." \ - --output-dir output_audio -``` - -Voice cloning with a reference audio: - -```bash -python examples/offline_inference/voxcpm2/end2end.py \ - --text "Hello, this is a voice clone demo." \ - --reference-audio /path/to/reference.wav \ - --output-dir output_clone -``` - -Prompt continuation (matched audio + text prefix): - -```bash -python examples/offline_inference/voxcpm2/end2end.py \ - --text "Continuation target sentence." \ - --prompt-audio /path/to/prompt.wav \ - --prompt-text "Transcript of the prompt audio." \ - --output-dir output_cont -``` - -The script accepts the following arguments: - -| Argument | Default | Description | -|---|---|---| -| `--model` | `openbmb/VoxCPM2` | HuggingFace repo ID or local path | -| `--text` | (example sentence) | Text to synthesize | -| `--output-dir` | `output_audio` | Directory for output WAV files | -| `--stage-configs-path` | `voxcpm2.yaml` | Stage config YAML path | -| `--reference-audio` | `None` | Reference audio for voice cloning (isolated) | -| `--prompt-audio` | `None` | Prompt audio for continuation mode | -| `--prompt-text` | `None` | Transcript matching `--prompt-audio` | - -## Performance - -Measured on a single H20 GPU (80 GB): - -| Input length | RTF | Sample rate | -|---|---|---| -| Short (~10 tokens) | ~0.28 | 48 kHz | -| Long (~100 tokens) | ~0.34 | 48 kHz | - -RTF < 1.0 means faster than real time. - -## Architecture - -VoxCPM2 uses a single-stage native AR pipeline: - -``` -feat_encoder -└─► MiniCPM4 (base LM) - └─► FSQ (finite scalar quantization) - └─► residual_lm (residual AR) - └─► LocDiT (local diffusion transformer) - └─► AudioVAE → 48 kHz waveform -``` - -All stages are fused into one vllm-native execution graph via `voxcpm2.yaml`, eliminating inter-stage coordination overhead and enabling true end-to-end batching. diff --git a/examples/offline_inference/voxtral_tts/README.md b/examples/offline_inference/voxtral_tts/README.md deleted file mode 100644 index a55ce8830..000000000 --- a/examples/offline_inference/voxtral_tts/README.md +++ /dev/null @@ -1,59 +0,0 @@ -# Voxtral TTS Offline Inference - -`end2end.py` runs Voxtral TTS end-to-end offline inference using vLLM. It supports both blocking (`Omni`) and streaming (`AsyncOmni`) generation, batched prompts with configurable concurrency, and voice selection via preset name or reference audio file. - -When `mistral_common` has `SpeechRequest` support, prompt token IDs are built via `encode_speech_request`. Otherwise, the script falls back to manual token construction. - -## Usage Examples - - -```bash -# Basic single-prompt with cheerful_female voice preset -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --write-audio --voice cheerful_female \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" - -# 32 replicate prompts with cheerful_female voice preset -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --num-prompts 32 --write-audio --voice cheerful_female \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" - -# Streaming with neutral_female voice preset -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --streaming --write-audio --voice neutral_female \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" - -# 32 prompts, 8 concurrent requests per wave, streaming with neutral_female voice -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --num-prompts 32 --concurrency 8 --streaming --write-audio --voice neutral_female \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "That eerie silence after the first storm was just the calm before another round of chaos, wasn't it?" - -# Short debug prompt with reference audio -# Note: Reference audio capability is not yet released. -python3 examples/offline_inference/voxtral_tts/end2end.py \ - --write-audio \ - --model mistralai/Voxtral-4B-TTS-2603 \ - --text "This is a test message." \ - --audio-path path/to/reference_audio.wav -``` - -## Arguments - -| Argument | Description | -|---|---| -| `--model PATH` | HuggingFace repo ID or local directory path (default: `mistralai/Voxtral-4B-TTS-2603`) | -| `--text TEXT` | Text to synthesize (default: `"This is a test message."`) | -| `--audio-path PATH` | Path to reference audio file for voice cloning | -| `--output-dir DIR` | Directory to write output WAV files (default: `output_audio`) | -| `--deploy-config PATH` | Override the deploy config path. If unset, auto-loads `vllm_omni/deploy/voxtral_tts.yaml` from the HF `model_type`. | -| `--num-prompts N` | Number of replicate prompts to run for measuring performance (default: 1) | -| `--streaming` | Use streaming generation via `AsyncOmni` (default: blocking `Omni`) | -| `--concurrency N` | Max concurrent requests per wave (must be used with `--streaming`, must evenly divide `--num-prompts`) | -| `--voice NAME` | Voice preset to use instead of reference audio file. Check Huggingface `mistralai/Voxtral-4B-TTS-2603` to get the list of available voices | -| `--write-audio` | Write generated audio to WAV files | -| `--profiling-mode` | Enable profiling mode (reduces max tokens to 50) | -| `--log-stats` | Enable detailed statistics logging | diff --git a/examples/online_serving/fish_speech/README.md b/examples/online_serving/fish_speech/README.md deleted file mode 100644 index 9b4e3cc40..000000000 --- a/examples/online_serving/fish_speech/README.md +++ /dev/null @@ -1,169 +0,0 @@ -# Fish Speech S2 Pro - -## Model - -| Model | Description | -|-------|-------------| -| `fishaudio/s2-pro` | Fish Speech S2 Pro -- 4B dual-AR TTS model with DAC codec (44.1 kHz) | - -## Gradio Demo - -An interactive Gradio demo is available with text-to-speech synthesis, voice cloning, and streaming support. - -```bash -# Option 1: Launch server + Gradio together -./run_gradio_demo.sh - -# Option 2: If server is already running -python gradio_demo.py --api-base http://localhost:8091 -``` - -Then open http://localhost:7860 in your browser. - -Features: - -- Text-to-speech synthesis -- Voice cloning from uploaded audio or URL -- Streaming mode (progressive PCM playback) - -## Launch the Server - -```bash -vllm serve fishaudio/s2-pro --omni --port 8091 -``` - -The deploy config is auto-loaded from `vllm_omni/deploy/fish_qwen3_omni.yaml` -(the HF `model_type` on the fishaudio checkpoints is `fish_qwen3_omni`). - -Or use the convenience script: - -```bash -./run_server.sh -``` - -## Send TTS Request - -### Using curl - -```bash -# Basic TTS -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, how are you?", - "voice": "default", - "response_format": "wav" - }' --output output.wav -``` - -### Voice Cloning - -Provide a reference audio (URL or base64 data URL) and its transcript: - -```bash -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, this is a cloned voice.", - "voice": "default", - "ref_audio": "https://example.com/reference.wav", - "ref_text": "Transcript of the reference audio." - }' --output cloned.wav -``` - -### Using Python - -```python -import httpx - -# Basic TTS -response = httpx.post( - "http://localhost:8091/v1/audio/speech", - json={ - "input": "Hello, how are you?", - "voice": "default", - "response_format": "wav", - }, - timeout=300.0, -) - -with open("output.wav", "wb") as f: - f.write(response.content) -``` - -### Using the CLI Client - -```bash -cd examples/online_serving/fish_speech - -# Basic TTS -python speech_client.py --text "Hello, how are you?" - -# Voice cloning -python speech_client.py \ - --text "Hello, this is a cloned voice." \ - --ref-audio /path/to/reference.wav \ - --ref-text "Transcript of the reference audio." - -# Streaming PCM output -python speech_client.py --text "Hello world" --stream --output output.pcm -``` - -The CLI client supports: - -- `--api-base`: API base URL (default: `http://localhost:8091`) -- `--model` (or `-m`): Model name (default: `fishaudio/s2-pro`) -- `--text`: Text to synthesize (required) -- `--ref-audio`: Reference audio for voice cloning (local path or URL) -- `--ref-text`: Transcript of the reference audio -- `--stream`: Enable streaming (PCM output) -- `--response-format`: Audio format: wav, mp3, flac, pcm, aac, opus (default: wav) -- `--output` (or `-o`): Output file path - -## Streaming - -Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded: - -```bash -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, how are you?", - "voice": "default", - "stream": true, - "response_format": "pcm" - }' --no-buffer | play -t raw -r 44100 -e signed -b 16 -c 1 - -``` - -**Note:** Fish Speech outputs at 44.1 kHz (unlike Qwen3-TTS which outputs at 24 kHz). - -## API Parameters - -Fish Speech uses the same `/v1/audio/speech` endpoint as Qwen3-TTS. See the [Speech API reference](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/speech_api/) for full parameter documentation. - -Key parameters for Fish Speech: - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `input` | string | **required** | Text to synthesize | -| `voice` | string | "default" | Voice name (use "default" for Fish Speech) | -| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus | -| `ref_audio` | string | null | Reference audio URL or base64 data URL for voice cloning | -| `ref_text` | string | null | Transcript of reference audio (required for voice cloning) | -| `max_new_tokens` | int | 4096 | Maximum tokens to generate | -| `stream` | bool | false | Stream raw PCM chunks | - -## Prerequisites - -Install the `fish-speech` package for the DAC codec: - -```bash -pip install fish-speech -``` - -## Troubleshooting - -1. **No audio output**: Make sure the `fish-speech` package is installed for the DAC decoder -2. **Connection refused**: Ensure the server is running on the correct port -3. **Flashinfer version mismatch**: Set `FLASHINFER_DISABLE_VERSION_CHECK=1` if you see version warnings -4. **Out of memory**: Reduce `--gpu-memory-utilization` or use a GPU with more VRAM diff --git a/examples/online_serving/omnivoice/README.md b/examples/online_serving/omnivoice/README.md deleted file mode 100644 index 1d8f00421..000000000 --- a/examples/online_serving/omnivoice/README.md +++ /dev/null @@ -1,131 +0,0 @@ -# OmniVoice - -## Model Overview - -| Model | Description | -|-------|-------------| -| `k2-fsa/OmniVoice` | Zero-shot multilingual TTS (600+ languages) with diffusion language model (Qwen3-0.6B backbone) | - -> **Note:** Requires `transformers>=5.3.0` for voice cloning (HiggsAudioV2 tokenizer). Auto voice and voice design work with `transformers>=4.57.0`. - -## Launch the Server - -```bash -vllm serve k2-fsa/OmniVoice \ - --omni \ - --port 8091 \ - --trust-remote-code -``` - -Or use the convenience script: - -```bash -./run_server.sh -``` - -## Send TTS Request - -### Using curl - -```bash -# Basic TTS (auto voice) -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, how are you?", - "voice": "default", - "response_format": "wav" - }' --output output.wav -``` - -### Using Python - -```python -import httpx - -response = httpx.post( - "http://localhost:8091/v1/audio/speech", - json={ - "input": "Hello, how are you?", - "voice": "default", - "response_format": "wav", - }, - timeout=300.0, -) - -with open("output.wav", "wb") as f: - f.write(response.content) -``` - -### Using OpenAI SDK - -```python -from openai import OpenAI - -client = OpenAI(base_url="http://localhost:8091/v1", api_key="none") - -response = client.audio.speech.create( - model="k2-fsa/OmniVoice", - voice="default", - input="Hello, how are you?", -) - -response.stream_to_file("output.wav") -``` - -### Using the CLI Client - -```bash -cd examples/online_serving/omnivoice - -# Basic TTS -python speech_client.py --text "Hello, how are you?" - -# Specify language for improved quality -python speech_client.py --text "Bonjour, comment allez-vous?" --language French -``` - -The CLI client supports: - -- `--api-base`: API base URL (default: `http://localhost:8091`) -- `--model` (or `-m`): Model name (default: `k2-fsa/OmniVoice`) -- `--text`: Text to synthesize (required) -- `--response-format`: Audio format: wav, mp3, flac, pcm, aac, opus (default: wav) -- `--language`: Language hint (default: Auto) -- `--output` (or `-o`): Output file path (default: `omnivoice_output.wav`) - -## Inference Modes - -OmniVoice supports three inference modes. Currently, **auto voice** is supported through the online Speech API. Voice cloning and voice design are available via offline inference (see `examples/offline_inference/omnivoice/`). - -| Mode | Description | Online API | Offline | -|------|-------------|:----------:|:-------:| -| Auto Voice | Generate speech without reference | Yes | Yes | -| Voice Clone | Clone from reference audio + transcript | - | Yes | -| Voice Design | Control style via natural language instruction | - | Yes | - -## Architecture - -OmniVoice uses a single-stage diffusion pipeline: - -- **Stage 0 (Generator)**: Qwen3-0.6B transformer with 32-step iterative masked unmasking and classifier-free guidance. Generates 8-codebook audio tokens from text, then decodes to 24kHz waveform via HiggsAudioV2 RVQ quantizer + DAC acoustic decoder. - -## API Parameters - -OmniVoice uses the standard `/v1/audio/speech` endpoint. See the [Speech API reference](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/speech_api/) for full documentation. - -Key parameters: - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `input` | string | **required** | Text to synthesize | -| `voice` | string | "default" | Voice name | -| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus | -| `speed` | float | 1.0 | Playback speed (0.25-4.0) | - -## Troubleshooting - -1. **TTS model did not produce audio output**: Ensure the model is fully downloaded (`huggingface-cli download k2-fsa/OmniVoice`) -2. **Connection refused**: Make sure the server is running on the correct port -3. **Out of memory**: Reduce `--gpu-memory-utilization` (default stage config uses 0.5) -4. **Slow first request**: The model performs warmup on first inference; subsequent requests are faster diff --git a/examples/online_serving/qwen3_omni/README.md b/examples/online_serving/qwen3_omni/README.md index 2f40b8347..b34b83e73 100644 --- a/examples/online_serving/qwen3_omni/README.md +++ b/examples/online_serving/qwen3_omni/README.md @@ -122,8 +122,9 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ --no-async-chunk ``` -For the TTS counterpart (synchronous codec variant), see -[qwen3_tts README](../qwen3_tts/README.md#sync-vs-async-chunk-mode). +For the TTS counterpart (synchronous codec variant), see the Qwen3-TTS +section of the online TTS hub: +[examples/online_serving/text_to_speech/README.md#qwen3-tts](../text_to_speech/README.md#qwen3-tts). Explicit CLI flags **override** the deploy YAML (which itself overrides the parser defaults). If you don't pass a flag, the YAML value wins. diff --git a/examples/online_serving/qwen3_tts/README.md b/examples/online_serving/qwen3_tts/README.md deleted file mode 100644 index 350fcb71c..000000000 --- a/examples/online_serving/qwen3_tts/README.md +++ /dev/null @@ -1,455 +0,0 @@ -# Qwen3-TTS - -## 🛠️ Installation - -Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md) - -## Supported Models - -| Model | Task Type | Description | -| -------------------------------------- | ----------- | ----------------------------------------------------- | -| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | CustomVoice | Predefined speaker voices with optional style control | -| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description | -| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio | -| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant | -| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning | - -## Gradio Demo - -Two interactive Gradio demos are available, both supporting all 3 task types: - -| Demo | File | Transport | Streaming Quality | -| -------- | ------------------------ | ------------ | -------------------------------------------------- | -| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks | -| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) | - -```bash -# Option 1: Launch server + Standard Gradio together -./run_gradio_demo.sh # CustomVoice (default) -./run_gradio_demo.sh --task-type VoiceDesign # VoiceDesign -./run_gradio_demo.sh --task-type Base # Voice cloning - -# Option 2: If server is already running -python gradio_demo.py --api-base http://localhost:8000 - -# Option 3: FastRTC demo (gapless streaming) -pip install fastrtc -python gradio_fastrtc_demo.py --api-base http://localhost:8000 -``` - -Then open http://localhost:7860 in your browser. - -## Run examples (Qwen3-TTS) - -### Launch the Server - -The default deploy config is located at `vllm_omni/deploy/qwen3_tts.yaml` and is loaded automatically by the model registry — no `--deploy-config` flag needed for default use. Platform-specific deltas (NPU, ROCm, XPU) are merged in automatically from the `platforms:` block of the same YAML based on the detected runtime. - -```bash -# CustomVoice model (predefined speakers) -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ - --omni \ - --port 8091 - -# VoiceDesign model -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \ - --omni \ - --port 8091 - -# Base model (voice cloning) -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \ - --omni \ - --port 8091 -``` - -If you have custom stage configs file, launch the server with command below -```bash -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ - --stage-configs-path /path/to/stage_configs_file \ - --omni \ - --port 8091 -``` - -#### Sync vs async-chunk mode - -Qwen3-TTS supports both **chunked streaming** (default, lower latency) and -**synchronous end-to-end** modes from the same deploy YAML. The bundled -`qwen3_tts.yaml` ships with `async_chunk: true`; flip with `--no-async-chunk` -and the pipeline automatically dispatches to the end-to-end codec processor: - -```bash -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091 \ - --no-async-chunk -``` - -No variant YAML or extra flag is needed — the `StagePipelineConfig` on each -stage declares both processor functions and the runtime picks based on the -`async_chunk:` bool. - -Alternatively, use the convenience script: -```bash -./run_server.sh # Default: CustomVoice model -./run_server.sh CustomVoice # CustomVoice model -./run_server.sh VoiceDesign # VoiceDesign model -./run_server.sh Base # Base (voice clone) model -``` - -### Send TTS Request - -Get into the example folder -```bash -cd examples/online_serving/qwen3_tts -``` - -#### Send request via python - -```bash -# CustomVoice: Use predefined speaker -python openai_speech_client.py \ - --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ - --text "你好，我是通义千问" \ - --voice vivian \ - --language Chinese - -# CustomVoice with style instruction -python openai_speech_client.py \ - --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ - --text "今天天气真好" \ - --voice ryan \ - --instructions "用开心的语气说" - -# VoiceDesign: Describe the voice style -python openai_speech_client.py \ - --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \ - --task-type VoiceDesign \ - --text "哥哥，你回来啦" \ - --instructions "体现撒娇稚嫩的萝莉女声，音调偏高" - -# Base: Voice cloning -python openai_speech_client.py \ - --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \ - --task-type Base \ - --text "Hello, this is a cloned voice" \ - --ref-audio /path/to/reference.wav \ - --ref-text "Original transcript of the reference audio" -``` - -The Python client supports the following command-line arguments: - -- `--api-base`: API base URL (default: `http://localhost:8091`) -- `--model` (or `-m`): Model name/path (default: `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`) -- `--task-type` (or `-t`): TTS task type. Options: `CustomVoice`, `VoiceDesign`, `Base` -- `--text`: Text to synthesize (required) -- `--voice`: Speaker/voice name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc. -- `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian` -- `--instructions`: Voice style/emotion instructions -- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server. -- `--ref-text`: Reference audio transcript for voice cloning (Base task). -- `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus` -- `--output` (or `-o`): Output audio file path (default: `tts_output.wav`) - -#### Send request via curl - -```bash -# Simple TTS request -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, how are you?", - "voice": "vivian", - "language": "English" - }' --output output.wav - -# With style instruction -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "I am so excited!", - "voice": "vivian", - "instructions": "Speak with great enthusiasm" - }' --output excited.wav - -# List available voices in CustomVoice models -curl http://localhost:8091/v1/audio/voices -``` - -### Using OpenAI SDK - -```python -from openai import OpenAI - -client = OpenAI(base_url="http://localhost:8091/v1", api_key="none") - -response = client.audio.speech.create( - model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", - voice="vivian", - input="Hello, how are you?", -) - -response.stream_to_file("output.wav") -``` - -### Using Python httpx - -```python -import httpx - -response = httpx.post( - "http://localhost:8091/v1/audio/speech", - json={ - "input": "Hello, how are you?", - "voice": "vivian", - "language": "English", - }, - timeout=300.0, -) - -with open("output.wav", "wb") as f: - f.write(response.content) -``` - -## API Reference - -### Voices Endpoint - -#### GET /v1/audio/voices - -List all available voices/speakers from the loaded model, including both built-in model voices and uploaded custom voices. - -**Response Example:** -```json -{ - "voices": ["vivian", "ryan", "custom_voice_1"], - "uploaded_voices": [ - { - "name": "custom_voice_1", - "consent": "user_consent_id", - "created_at": 1738660000, - "file_size": 1024000, - "mime_type": "audio/wav" - } - ] -} -``` - -#### POST /v1/audio/voices - -Upload a new voice sample for voice cloning in Base task TTS requests. - -**Form Parameters:** -- `audio_sample` (required): Audio file (max 10MB, supported formats: wav, mp3, flac, ogg, aac, webm, mp4) -- `consent` (required): Consent recording ID -- `name` (required): Name for the new voice -- `ref_text` (optional): Transcript of the audio. Enables in-context voice cloning (higher quality). -- `speaker_description` (optional): Free-form description of the voice (e.g. "warm narrator", "energetic presenter"). - -**Response Example:** -```json -{ - "success": true, - "voice": { - "name": "custom_voice_1", - "consent": "user_consent_id", - "created_at": 1738660000, - "mime_type": "audio/wav", - "file_size": 1024000, - "ref_text": "The exact transcript of the audio sample.", - "speaker_description": "warm narrator" - } -} -``` - -Fields `ref_text` and `speaker_description` are omitted when not provided at upload time. - -**Usage Example:** -```bash -curl -X POST http://localhost:8000/v1/audio/voices \ - -F "audio_sample=@/path/to/voice_sample.wav" \ - -F "consent=user_consent_id" \ - -F "name=custom_voice_1" \ - -F "ref_text=The exact transcript of the audio sample." \ - -F "speaker_description=warm narrator" -``` - -### Endpoint - -``` -POST /v1/audio/speech -Content-Type: application/json -``` - -This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters. - -### Request Body - -```json -{ - "input": "Text to synthesize", - "voice": "vivian", - "response_format": "wav", - "task_type": "CustomVoice", - "language": "Auto", - "instructions": "Optional style instructions", - "ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning", - "ref_text": "Reference audio transcript", - "x_vector_only_mode": false, - "max_new_tokens": 2048 -} -``` - -> **Note:** The `model` field is optional when serving a single model, as the server already knows which model is loaded. - -### Response - -Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/wav`). - -## Parameters - -### OpenAI Standard Parameters - -| Parameter | Type | Default | Description | -| ----------------- | ------ | -------------- | ----------------------------------------------------------- | -| `input` | string | **required** | Text to synthesize | -| `model` | string | server's model | Model to use (optional, should match server if specified) | -| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) | -| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus | -| `speed` | float | 1.0 | Playback speed (0.25-4.0, not supported with `stream=true`) | - -### vLLM-Omni Extension Parameters - -| Parameter | Type | Default | Description | -| ---------------------------- | ------ | ------------- | -------------------------------------------------------------------------------------------------------------------- | -| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | -| `language` | string | "Auto" | Language (see supported languages below) | -| `instructions` | string | "" | Voice style/emotion instructions | -| `max_new_tokens` | int | 2048 | Maximum tokens to generate | -| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. | -| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | - -**Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian - -### Voice Clone Parameters (Base task) - -| Parameter | Type | Required | Description | -| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------- | -| `ref_audio` | string | **Yes** | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) | -| `ref_text` | string | No | Transcript of reference audio (for ICL mode) | -| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | - -## Streaming - -Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded -(one chunk per Code2Wav window, default 25 frames; configurable in the stage config): - -```bash -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "input": "Hello, how are you?", - "voice": "vivian", - "language": "English", - "stream": true, - "response_format": "pcm" - }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 - -``` - -**Constraints:** -- `stream=true` requires `response_format="pcm"` (raw 16-bit signed PCM, 24 kHz mono). -- `speed` adjustment is not supported when streaming. -- Requires the server stage config to have `async_chunk: true` (default in `qwen3_tts.yaml`). - -## Streaming Text Input (WebSocket) - -The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally, buffers and splits it at sentence boundaries, and generates audio per sentence. - -When `stream_audio=true`, each sentence is emitted as `audio.start`, one or more binary PCM frames, and `audio.done`. - -### Quick Start - -```bash -python streaming_speech_client.py \ - --text "Hello world. How are you? I am fine." - -python streaming_speech_client.py \ - --text "Hello world. How are you? I am fine." \ - --simulate-stt --stt-delay 0.1 -``` - -### WebSocket Protocol - -Client -> Server: - -```jsonc -{"type": "session.config", "voice": "Vivian", "task_type": "CustomVoice", "language": "Auto", "split_granularity": "sentence", "stream_audio": true, "response_format": "pcm"} -{"type": "input.text", "text": "Hello, how are you? "} -{"type": "input.done"} -``` - -Server -> Client: - -```jsonc -{"type": "audio.start", "sentence_index": 0, "sentence_text": "Hello, how are you?", "format": "pcm", "sample_rate": 24000} -// binary PCM frame(s) -{"type": "audio.done", "sentence_index": 0, "total_bytes": 96000, "error": false} -{"type": "session.done", "total_sentences": 1} -``` - -## Choosing an Execution Backend: Uniproc vs Multiprocessing - -Qwen3-TTS stage configs support two execution backends controlled by the -`distributed_executor_backend` engine arg. The performance tradeoff between -them is **both hardware- and task-dependent**, so there is no single best -default (see [#2603](https://github.com/vllm-project/vllm-omni/issues/2603), -[#2604](https://github.com/vllm-project/vllm-omni/pull/2604) for the full -investigation). - -| Backend | Stage config setting | Behaviour | -| ------- | -------------------- | --------- | -| **Uniproc** (default, world_size=1) | `distributed_executor_backend` omitted | Both stages run inside the orchestrator process. Avoids IPC serialisation, D2H copies, and msgpack overhead between stages. | -| **Multiprocessing** | `distributed_executor_backend: "mp"` | Each stage runs in its own subprocess. The Talker can continue decoding while Code2Wav runs the vocoder in parallel, improving pipeline utilisation under concurrency. | - -> **Note:** When `distributed_executor_backend` is omitted and `world_size=1`, -> vLLM [automatically uses the uniproc executor](https://github.com/vllm-project/vllm/blob/main/vllm/config/parallel.py#L825). -> When `world_size > 1`, it defaults to `mp`. - -### When uniproc wins - -The uniproc path eliminates inter-process data transfer (D2H copies, -msgpack serialisation/deserialisation, tensor detaching). This matters most -when per-request processing is heavy relative to autoregressive decode. - -The Base cloning task involves reference-audio encoding on every request, making IPC -overhead a larger fraction of total cost. Qwen3-Omni shows a similar pattern. - -### When multiprocessing (`mp`) wins - -For lighter per-request workloads, process-level parallelism between the -Talker and Code2Wav stages dominates. - -CustomVoice is lighter per-request (no reference audio encoding), so the -process-level parallelism of `mp` outweighs its serialisation cost at -concurrency ≥ 4. - -### How to switch - -To use the uniproc executor on a single-GPU setup, pass the -`qwen3_tts_uniproc.yaml` stage config: - -```bash -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \ - --omni \ - --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_uniproc.yaml \ - --port 8091 -``` - -## Limitations - -- **Single request**: Batch processing is not yet optimized for online serving. - -## Troubleshooting - -1. **TTS model did not produce audio output**: Ensure you're using the correct model variant for your task type (CustomVoice task → CustomVoice model, etc.) -2. **Connection refused**: Make sure the server is running on the correct port -3. **Out of memory**: Use smaller model variant (`Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice`) or reduce `--gpu-memory-utilization` -4. **Unsupported speaker**: Use `/v1/audio/voices` to list available voices for the loaded model -5. **Voice clone fails**: Ensure you're using the Base model variant for voice cloning diff --git a/examples/online_serving/text_to_speech/README.md b/examples/online_serving/text_to_speech/README.md new file mode 100644 index 000000000..836b8ed7a --- /dev/null +++ b/examples/online_serving/text_to_speech/README.md @@ -0,0 +1,399 @@ +# Text-To-Speech (Online Serving) + +vLLM-Omni exposes TTS models through the OpenAI-compatible +[`POST /v1/audio/speech`](../../../docs/serving/speech_api.md) endpoint, +launched with `vllm serve --omni`. Each TTS model has its own +subdirectory containing client snippets, gradio demos, and helper +scripts; this README is the single doc entry point for all of them. + +For offline inference, see [`examples/offline_inference/text_to_speech`](../../offline_inference/text_to_speech/README.md). +For the full list of supported architectures across all modalities, see +[Supported Models](../../../docs/models/supported_models.md). + +## Supported Models + +| Model | HuggingFace repo | Voice cloning | Streaming | Voice presets / upload | Gradio demo | +|---|---|---|---|---|---| +| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ | +| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — | +| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | ✓ (Base) | ✓ (PCM + WebSocket) | ✓ (presets + `/v1/audio/voices` upload) | ✓ (standard + FastRTC) | +| VoxCPM | local model dir | ✓ | ✓ (PCM stream) | — | — | +| VoxCPM2 | `openbmb/VoxCPM2` | ✓ | ✓ (AudioWorklet via gradio) | — | ✓ | +| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✓ (gated upstream) | ✓ | ✓ (presets) | ✓ | + +CosyVoice3 is intentionally absent: no online example exists for it yet. See its [offline section](../../offline_inference/text_to_speech/README.md#cosyvoice3) instead. + +## Common Quick Start + +Launch the server (defaults shown — adjust `--port`, `--gpu-memory-utilization`, etc. as needed): + +```bash +vllm serve --omni --port 8091 +``` + +Send a TTS request via curl: + +```bash +curl -X POST http://localhost:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello, how are you?", + "voice": "default", + "response_format": "wav" + }' --output output.wav +``` + +Or via Python httpx: + +```python +import httpx + +response = httpx.post( + "http://localhost:8091/v1/audio/speech", + json={ + "input": "Hello, how are you?", + "voice": "default", + "response_format": "wav", + }, + timeout=300.0, +) +open("output.wav", "wb").write(response.content) +``` + +Or via the OpenAI SDK: + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8091/v1", api_key="none") +response = client.audio.speech.create( + model="", + voice="default", + input="Hello, how are you?", +) +response.stream_to_file("output.wav") +``` + +Streaming PCM output (where supported) — set `stream=true` with `response_format="pcm"`: + +```bash +curl -X POST http://localhost:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello, how are you?", + "voice": "default", + "stream": true, + "response_format": "pcm" + }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 - +``` + +Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others). + +For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](../../../docs/serving/speech_api.md). + +--- + +## Fish Speech S2 Pro + +4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec. + +### Prerequisites +```bash +pip install fish-speech +``` + +### Launch +```bash +vllm serve fishaudio/s2-pro --omni --port 8091 +# or: +./fish_speech/run_server.sh +``` +The deploy config auto-loads from `vllm_omni/deploy/fish_qwen3_omni.yaml` (the HF `model_type` on the fishaudio checkpoint is `fish_qwen3_omni`). + +### Voice cloning +```bash +curl -X POST http://localhost:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello, this is a cloned voice.", + "voice": "default", + "ref_audio": "https://example.com/reference.wav", + "ref_text": "Transcript of the reference audio." + }' --output cloned.wav +``` + +### CLI client +```bash +cd examples/online_serving/text_to_speech/fish_speech +python speech_client.py --text "Hello, how are you?" +python speech_client.py --text "Hello world" --stream --output output.pcm +``` + +### Gradio demo +```bash +./fish_speech/run_gradio_demo.sh # launches server + Gradio +python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server already running +``` + +### Notes +- Output: 44.1 kHz mono. +- Streaming PCM player command must use `-r 44100`. + +--- + +## OmniVoice + +Zero-shot multilingual TTS (600+ languages). Online serving currently exposes **auto voice** only; voice cloning and voice design are available offline. + +### Prerequisites +```bash +huggingface-cli download k2-fsa/OmniVoice +``` +Voice cloning (offline) needs `transformers>=5.3.0`; auto voice works with `transformers>=4.57.0`. + +### Launch +```bash +vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code +# or: +./omnivoice/run_server.sh +``` + +### CLI client +```bash +cd examples/online_serving/text_to_speech/omnivoice +python speech_client.py --text "Hello, how are you?" +python speech_client.py --text "Bonjour, comment allez-vous?" --language French +``` + +The client supports `--api-base`, `--model`, `--text`, `--response-format`, `--language`, `--output`. + +### Notes +- Voice cloning and voice design require offline inference; see the [offline OmniVoice section](../../offline_inference/text_to_speech/README.md#omnivoice). + +--- + +## Qwen3-TTS + +Three model variants exposed via separate checkpoints: + +| Variant | HF repo | Use | +|---|---|---| +| CustomVoice | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | Predefined speakers (`vivian`, `ryan`, …) with optional style instructions | +| VoiceDesign | `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | Natural-language voice style description | +| Base | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Voice cloning from a reference audio | + +Each variant ships smaller `0.6B` companions where available. + +### Launch +```bash +vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091 +# or: +./qwen3_tts/run_server.sh # default: CustomVoice +./qwen3_tts/run_server.sh VoiceDesign +./qwen3_tts/run_server.sh Base +``` + +### Choosing an executor backend (uniproc vs mp) +Stage configs ship with the chunked-streaming default. To use the uniproc executor (lower IPC overhead for the Base cloning task), pass `--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_uniproc.yaml`. See [#2603](https://github.com/vllm-project/vllm-omni/issues/2603) and [#2604](https://github.com/vllm-project/vllm-omni/pull/2604) for the full investigation. + +To opt out of chunked streaming, pass `--no-async-chunk` instead — the pipeline auto-dispatches to the end-to-end codec processor. + +### Sending requests +```bash +# CustomVoice with a predefined speaker +python qwen3_tts/openai_speech_client.py \ + --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ + --text "今天天气真好" \ + --speaker ryan \ + --instructions "用开心的语气说" + +# VoiceDesign with a style description +python qwen3_tts/openai_speech_client.py \ + --model Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \ + --task-type VoiceDesign \ + --text "哥哥，你回来啦" \ + --instructions "体现撒娇稚嫩的萝莉女声，音调偏高" + +# Base voice cloning +python qwen3_tts/openai_speech_client.py \ + --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \ + --task-type Base \ + --text "Hello, this is a cloned voice" \ + --ref-audio /path/to/reference.wav \ + --ref-text "Original transcript of the reference audio" +``` + +### Voices endpoint +List available voices, or upload a custom one for Base cloning: +```bash +# List +curl http://localhost:8091/v1/audio/voices + +# Upload +curl -X POST http://localhost:8091/v1/audio/voices \ + -F "audio_sample=@/path/to/voice_sample.wav" \ + -F "consent=user_consent_id" \ + -F "name=custom_voice_1" \ + -F "ref_text=The exact transcript of the audio sample." \ + -F "speaker_description=warm narrator" +``` +Uploaded voices are then usable as `voice="custom_voice_1"` on subsequent requests. + +### Streaming PCM +```bash +curl -X POST http://localhost:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Hello, how are you?", + "voice": "vivian", + "language": "English", + "stream": true, + "response_format": "pcm" + }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 - +``` +Streaming requires `response_format="pcm"` and `async_chunk: true` on the stage config (default in `qwen3_tts.yaml`). `speed` is not supported when streaming. + +### Streaming WebSocket +The `/v1/audio/speech/stream` endpoint accepts text incrementally, splits it at sentence boundaries, and emits one PCM stream per sentence: +```bash +python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you? I am fine." +python qwen3_tts/streaming_speech_client.py --text "..." --simulate-stt --stt-delay 0.1 +``` + +### Gradio demos +```bash +./qwen3_tts/run_gradio_demo.sh # CustomVoice (default) +./qwen3_tts/run_gradio_demo.sh --task-type VoiceDesign +./qwen3_tts/run_gradio_demo.sh --task-type Base +``` + +### Speaker embedding interpolation +`qwen3_tts/speaker_embedding_interpolation.py` blends two predefined speakers' embeddings to produce intermediate voices. See the script for usage. + +### Batch client +`qwen3_tts/batch_speech_client.py` issues many concurrent requests for throughput measurement. + +### Notes +- Base voice cloning has uniproc-vs-mp tradeoffs depending on per-request reference audio cost; see the executor-backend section above. +- `vllm_omni/deploy/qwen3_tts.yaml` is the default deploy config (loaded by HF `model_type`); per-stage runtime overrides are available via `--stage-N- `. + +--- + +## VoxCPM + +Split-stage TTS at 24 kHz. + +### Prerequisites +```bash +pip install voxcpm +# or use a local source tree: +export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src +``` + +If the native VoxCPM `config.json` lacks HF `model_type`, set up an HF-compatible config dir: +```bash +export VOXCPM_MODEL=/path/to/voxcpm-model +export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config +mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH" +cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json" +cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true +python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)' +``` + +### Launch +```bash +export VOXCPM_MODEL=/path/to/voxcpm-model +./voxcpm/run_server.sh # async-chunk streaming (default) +./voxcpm/run_server.sh sync # non-streaming +``` +Or directly: +```bash +vllm serve "$VOXCPM_MODEL" \ + --stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \ + --trust-remote-code --enforce-eager --omni --port 8091 +``` + +### Sending requests +```bash +# Basic TTS +python voxcpm/openai_speech_client.py \ + --model "$VOXCPM_MODEL" \ + --text "This is a VoxCPM online text-to-speech example." + +# Voice cloning +python voxcpm/openai_speech_client.py \ + --model "$VOXCPM_MODEL" \ + --text "This sentence is synthesized with a cloned voice." \ + --ref-audio /path/to/reference.wav \ + --ref-text "The exact transcript spoken in reference.wav." + +# Streaming PCM +python voxcpm/openai_speech_client.py \ + --model "$VOXCPM_MODEL" \ + --text "This is a streaming VoxCPM request." \ + --stream --output voxcpm_stream.pcm +``` + +### Notes +- `voxcpm.yaml` for one-shot decode; `voxcpm_async_chunk.yaml` for single-request streaming. Do not use the async-chunk config for concurrent requests or `/v1/audio/speech/batch`. +- Generic TTS fields not supported by VoxCPM: `voice`, `instructions`, `language`, `speaker_embedding`, `x_vector_only_mode`. +- For benchmark measurement, see [`benchmarks/voxcpm`](../../../benchmarks/voxcpm/README.md). + +--- + +## VoxCPM2 + +Single-stage native AR TTS at 48 kHz. + +### Launch +```bash +vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000 +``` +Deploy config auto-loads from `vllm_omni/deploy/voxcpm2.yaml`. Pass `--deploy-config ` to override or `--stage-N- ` for per-stage runtime tweaks. + +### Sending requests +```bash +# Zero-shot synthesis +python voxcpm2/openai_speech_client.py --text "Hello, this is VoxCPM2." + +# Voice cloning +python voxcpm2/openai_speech_client.py \ + --text "This should sound like the reference speaker." \ + --ref-audio /path/to/reference.wav +``` +The `ref_audio` field accepts local file paths (auto-base64), HTTP URLs, or `data:audio/wav;base64,...` data URIs. + +### Gradio demo (gapless streaming via AudioWorklet) +```bash +python voxcpm2/gradio_demo.py +``` +Uses an AudioWorklet-based player adapted from the Qwen3-TTS demo for gap-free playback. Audio is streamed from the OpenAI Speech endpoint with `stream=true`. + +--- + +## Voxtral TTS + +Voxtral-4B-TTS (Mistral). Uses the `mistral_common` `SpeechRequest` protocol; voice presets are model-specific. + +### Prerequisites +Latest `mistral_common` with `SpeechRequest` support: +```bash +pip install -e /path/to/mistral-common # or upgrade from PyPI when available +``` + +### Launch +```bash +vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091 +``` +Deploy config auto-loads from `vllm_omni/deploy/voxtral_tts.yaml`. + +### Gradio demo +```bash +python voxtral_tts/gradio_demo.py +``` +The demo handles voice-preset selection and reference-audio upload. `voxtral_tts/text_preprocess.py` provides the text-normalization helpers used by the demo (also available for other clients). + +### Notes +- Voice presets are listed on the HF model card (`mistralai/Voxtral-4B-TTS-2603`). +- Voice cloning is gated upstream and may require a recent `mistral_common`. +- A standalone CLI client is not yet shipped; the gradio demo is the canonical reference for now. diff --git a/examples/online_serving/fish_speech/gradio_demo.py b/examples/online_serving/text_to_speech/fish_speech/gradio_demo.py similarity index 100% rename from examples/online_serving/fish_speech/gradio_demo.py rename to examples/online_serving/text_to_speech/fish_speech/gradio_demo.py diff --git a/examples/online_serving/fish_speech/run_gradio_demo.sh b/examples/online_serving/text_to_speech/fish_speech/run_gradio_demo.sh similarity index 100% rename from examples/online_serving/fish_speech/run_gradio_demo.sh rename to examples/online_serving/text_to_speech/fish_speech/run_gradio_demo.sh diff --git a/examples/online_serving/fish_speech/run_server.sh b/examples/online_serving/text_to_speech/fish_speech/run_server.sh similarity index 100% rename from examples/online_serving/fish_speech/run_server.sh rename to examples/online_serving/text_to_speech/fish_speech/run_server.sh diff --git a/examples/online_serving/fish_speech/speech_client.py b/examples/online_serving/text_to_speech/fish_speech/speech_client.py similarity index 100% rename from examples/online_serving/fish_speech/speech_client.py rename to examples/online_serving/text_to_speech/fish_speech/speech_client.py diff --git a/examples/online_serving/omnivoice/run_server.sh b/examples/online_serving/text_to_speech/omnivoice/run_server.sh similarity index 100% rename from examples/online_serving/omnivoice/run_server.sh rename to examples/online_serving/text_to_speech/omnivoice/run_server.sh diff --git a/examples/online_serving/omnivoice/speech_client.py b/examples/online_serving/text_to_speech/omnivoice/speech_client.py similarity index 100% rename from examples/online_serving/omnivoice/speech_client.py rename to examples/online_serving/text_to_speech/omnivoice/speech_client.py diff --git a/examples/online_serving/qwen3_tts/batch_speech_client.py b/examples/online_serving/text_to_speech/qwen3_tts/batch_speech_client.py similarity index 100% rename from examples/online_serving/qwen3_tts/batch_speech_client.py rename to examples/online_serving/text_to_speech/qwen3_tts/batch_speech_client.py diff --git a/examples/online_serving/qwen3_tts/gradio_demo.py b/examples/online_serving/text_to_speech/qwen3_tts/gradio_demo.py similarity index 100% rename from examples/online_serving/qwen3_tts/gradio_demo.py rename to examples/online_serving/text_to_speech/qwen3_tts/gradio_demo.py diff --git a/examples/online_serving/qwen3_tts/openai_speech_client.py b/examples/online_serving/text_to_speech/qwen3_tts/openai_speech_client.py similarity index 100% rename from examples/online_serving/qwen3_tts/openai_speech_client.py rename to examples/online_serving/text_to_speech/qwen3_tts/openai_speech_client.py diff --git a/examples/online_serving/qwen3_tts/run_gradio_demo.sh b/examples/online_serving/text_to_speech/qwen3_tts/run_gradio_demo.sh similarity index 100% rename from examples/online_serving/qwen3_tts/run_gradio_demo.sh rename to examples/online_serving/text_to_speech/qwen3_tts/run_gradio_demo.sh diff --git a/examples/online_serving/qwen3_tts/run_server.sh b/examples/online_serving/text_to_speech/qwen3_tts/run_server.sh similarity index 100% rename from examples/online_serving/qwen3_tts/run_server.sh rename to examples/online_serving/text_to_speech/qwen3_tts/run_server.sh diff --git a/examples/online_serving/qwen3_tts/speaker_embedding_interpolation.py b/examples/online_serving/text_to_speech/qwen3_tts/speaker_embedding_interpolation.py similarity index 100% rename from examples/online_serving/qwen3_tts/speaker_embedding_interpolation.py rename to examples/online_serving/text_to_speech/qwen3_tts/speaker_embedding_interpolation.py diff --git a/examples/online_serving/qwen3_tts/streaming_speech_client.py b/examples/online_serving/text_to_speech/qwen3_tts/streaming_speech_client.py similarity index 100% rename from examples/online_serving/qwen3_tts/streaming_speech_client.py rename to examples/online_serving/text_to_speech/qwen3_tts/streaming_speech_client.py diff --git a/examples/online_serving/qwen3_tts/tts_common.py b/examples/online_serving/text_to_speech/qwen3_tts/tts_common.py similarity index 100% rename from examples/online_serving/qwen3_tts/tts_common.py rename to examples/online_serving/text_to_speech/qwen3_tts/tts_common.py diff --git a/examples/online_serving/voxcpm/openai_speech_client.py b/examples/online_serving/text_to_speech/voxcpm/openai_speech_client.py similarity index 100% rename from examples/online_serving/voxcpm/openai_speech_client.py rename to examples/online_serving/text_to_speech/voxcpm/openai_speech_client.py diff --git a/examples/online_serving/voxcpm/run_server.sh b/examples/online_serving/text_to_speech/voxcpm/run_server.sh similarity index 100% rename from examples/online_serving/voxcpm/run_server.sh rename to examples/online_serving/text_to_speech/voxcpm/run_server.sh diff --git a/examples/online_serving/voxcpm2/gradio_demo.py b/examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py similarity index 100% rename from examples/online_serving/voxcpm2/gradio_demo.py rename to examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py diff --git a/examples/online_serving/voxcpm2/openai_speech_client.py b/examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py similarity index 100% rename from examples/online_serving/voxcpm2/openai_speech_client.py rename to examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py diff --git a/examples/online_serving/voxtral_tts/gradio_demo.py b/examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py similarity index 99% rename from examples/online_serving/voxtral_tts/gradio_demo.py rename to examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py index 7905c6261..88f8cc0d4 100644 --- a/examples/online_serving/voxtral_tts/gradio_demo.py +++ b/examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py @@ -5,7 +5,7 @@ Example use case: -python examples/online_serving/voxtral_tts/gradio_demo.py --host slurm-199-077 --port 8000 +python examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py --host slurm-199-077 --port 8000 """ diff --git a/examples/online_serving/voxtral_tts/text_preprocess.py b/examples/online_serving/text_to_speech/voxtral_tts/text_preprocess.py similarity index 100% rename from examples/online_serving/voxtral_tts/text_preprocess.py rename to examples/online_serving/text_to_speech/voxtral_tts/text_preprocess.py diff --git a/examples/online_serving/voxcpm/README.md b/examples/online_serving/voxcpm/README.md deleted file mode 100644 index 78e1bf4aa..000000000 --- a/examples/online_serving/voxcpm/README.md +++ /dev/null @@ -1,166 +0,0 @@ -# VoxCPM - -## Prerequisites - -Install VoxCPM in one of these ways: - -```bash -pip install voxcpm -``` - -or point vLLM-Omni to a local VoxCPM source tree: - -```bash -export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src -``` - -If the native VoxCPM `config.json` lacks HF metadata such as `model_type`, -prepare a persistent HF-compatible config directory and export: - -```bash -export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config -mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH" -cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json" -cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true -python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)' -``` - -The VoxCPM stage configs read `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH` directly. The `python3 -c` form above avoids heredoc/indentation issues in interactive shells. - -## Launch the Server - -Use the async-chunk stage config by default: - -```bash -export VOXCPM_MODEL=/path/to/voxcpm-model -cd examples/online_serving/voxcpm -./run_server.sh -``` - -Use the non-streaming stage config: - -```bash -./run_server.sh sync -``` - -You can also launch the server directly: - -```bash -vllm serve "$VOXCPM_MODEL" \ - --stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \ - --trust-remote-code \ - --enforce-eager \ - --omni \ - --port 8091 -``` - -## Send Requests - -### Basic text-to-speech - -```bash -python openai_speech_client.py \ - --model "$VOXCPM_MODEL" \ - --text "This is a VoxCPM online text-to-speech example." -``` - -### Voice cloning - -```bash -python openai_speech_client.py \ - --model "$VOXCPM_MODEL" \ - --text "This sentence is synthesized with a cloned voice." \ - --ref-audio /path/to/reference.wav \ - --ref-text "The exact transcript spoken in reference.wav." -``` - -`ref_text` must be the real transcript of the reference audio. Placeholder text or mismatched text will usually degrade quality badly. - -### Streaming PCM output - -```bash -python openai_speech_client.py \ - --model "$VOXCPM_MODEL" \ - --text "This is a streaming VoxCPM request." \ - --stream \ - --output voxcpm_stream.pcm -``` - -### Using curl - -```bash -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "model": "OpenBMB/VoxCPM1.5", - "input": "Hello from VoxCPM online serving.", - "response_format": "wav" - }' --output output.wav -``` - -Voice cloning: - -```bash -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "model": "OpenBMB/VoxCPM1.5", - "input": "This sentence uses a cloned voice.", - "ref_audio": "https://example.com/reference.wav", - "ref_text": "The exact transcript spoken in the reference audio.", - "response_format": "wav" - }' --output cloned.wav -``` - -Streaming PCM: - -```bash -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{ - "model": "OpenBMB/VoxCPM1.5", - "input": "This is a streaming VoxCPM request.", - "stream": true, - "response_format": "pcm" - }' --output output.pcm -``` - -## Supported Request Shape - -VoxCPM online serving currently supports: - -- plain text-to-speech -- voice cloning with `ref_audio` + `ref_text` -- `stream=true` with `response_format=pcm` or `wav` - -VoxCPM online serving does not use these generic TTS fields: - -- `voice` -- `instructions` -- `language` -- `speaker_embedding` -- `x_vector_only_mode` - -## Streaming vs Non-Streaming - -- `voxcpm_async_chunk.yaml` enables async-chunk streaming and is best for single-request streaming latency. -- `voxcpm.yaml` performs one-shot latent generation then VAE decode. - -Like native VoxCPM, the async streaming path should be treated as single-request. If you need stable throughput benchmarking, prefer `voxcpm.yaml`. - -Do not use `voxcpm_async_chunk.yaml` for concurrent online streaming or `/v1/audio/speech/batch`. For multiple requests, prefer `voxcpm.yaml`. - -## Benchmark - -The serving benchmark reports TTFP and RTF: - -```bash -python benchmarks/voxcpm/vllm_omni/bench_tts_serve.py \ - --host 127.0.0.1 \ - --port 8091 \ - --num-prompts 10 \ - --max-concurrency 1 \ - --result-dir /tmp/voxcpm_bench -``` - -For the async-chunk server, keep `--max-concurrency 1`. diff --git a/examples/online_serving/voxcpm2/README.md b/examples/online_serving/voxcpm2/README.md deleted file mode 100644 index 9ca2ae708..000000000 --- a/examples/online_serving/voxcpm2/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# VoxCPM2 Online Serving - -Serve VoxCPM2 TTS via the OpenAI-compatible `/v1/audio/speech` endpoint. - -## Start the Server - -```bash -vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000 -``` - -The deploy config is auto-loaded from `vllm_omni/deploy/voxcpm2.yaml`. Pass -`--deploy-config ` to override, or `--stage-N- ` (e.g. -`--stage-0-max-num-seqs 8`) for per-stage runtime overrides. - -## Zero-shot Synthesis - -```bash -python openai_speech_client.py --text "Hello, this is VoxCPM2." -``` - -Or with curl: - -```bash -curl -X POST http://localhost:8000/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{"model": "voxcpm2", "input": "Hello, this is VoxCPM2.", "voice": "default"}' \ - --output output.wav -``` - -## Voice Cloning - -Clone a speaker's voice using a reference audio file: - -```bash -python openai_speech_client.py \ - --text "This should sound like the reference speaker." \ - --ref-audio /path/to/reference.wav -``` - -The `--ref-audio` parameter accepts: -- Local file path (auto-encoded to base64) -- URL (`https://...`) -- Base64 data URI (`data:audio/wav;base64,...`) diff --git a/tests/e2e/offline_inference/test_voxtral_tts.py b/tests/e2e/offline_inference/test_voxtral_tts.py index 5386e2db4..de88dc220 100644 --- a/tests/e2e/offline_inference/test_voxtral_tts.py +++ b/tests/e2e/offline_inference/test_voxtral_tts.py @@ -7,7 +7,7 @@ End-to-end offline inference tests for Voxtral TTS model. Tests both synchronous (Omni) and async streaming (AsyncOmni) paths. Equivalent to running: - python3 examples/offline_inference/voxtral_tts/end2end.py \ + python3 examples/offline_inference/text_to_speech/voxtral_tts/end2end.py \ --model mistralai/Voxtral-4B-TTS-2603 \ --voice casual_female \ --text "Hello, how are you?" \ diff --git a/tests/examples/offline_inference/test_qwen3_tts_estimator.py b/tests/examples/offline_inference/test_qwen3_tts_estimator.py index f993d4ed3..fde638ba5 100644 --- a/tests/examples/offline_inference/test_qwen3_tts_estimator.py +++ b/tests/examples/offline_inference/test_qwen3_tts_estimator.py @@ -2,7 +2,7 @@ Offline inference regression tests: Qwen3-TTS _estimate_prompt_len. Covers the four ref_audio input forms handled by the estimator in -examples/offline_inference/qwen3_tts/end2end.py: +examples/offline_inference/text_to_speech/qwen3_tts/end2end.py: - bare local path - file:// URI - http:// URL @@ -20,7 +20,9 @@ from unittest.mock import MagicMock import numpy as np import pytest -_examples_dir = str(Path(__file__).parent.parent.parent.parent / "examples" / "offline_inference" / "qwen3_tts") +_examples_dir = str( + Path(__file__).parent.parent.parent.parent / "examples" / "offline_inference" / "text_to_speech" / "qwen3_tts" +) sys.path.insert(0, _examples_dir) from end2end import _estimate_prompt_len # noqa: E402 diff --git a/tests/examples/test_slerp_interpolation.py b/tests/examples/test_slerp_interpolation.py index fe7db74cc..d35323625 100644 --- a/tests/examples/test_slerp_interpolation.py +++ b/tests/examples/test_slerp_interpolation.py @@ -9,7 +9,9 @@ import numpy as np import pytest # Import the slerp function from the example script. -_examples_dir = str(Path(__file__).parent.parent.parent / "examples" / "online_serving" / "qwen3_tts") +_examples_dir = str( + Path(__file__).parent.parent.parent / "examples" / "online_serving" / "text_to_speech" / "qwen3_tts" +) sys.path.insert(0, _examples_dir) from speaker_embedding_interpolation import slerp # noqa: E402 diff --git a/tests/helpers/runtime.py b/tests/helpers/runtime.py index 45d86548f..79ca9ee5c 100644 --- a/tests/helpers/runtime.py +++ b/tests/helpers/runtime.py @@ -1240,7 +1240,7 @@ class OmniRunner: if isinstance(prompts, str): prompts = [prompts] - # Qwen-TTS: follow examples/offline_inference/qwen3_tts/end2end.py style. + # Qwen-TTS: follow examples/offline_inference/text_to_speech/qwen3_tts/end2end.py style. # Stage 0 expects token placeholders + additional_information (text/speaker/task_type/...), # and Talker replaces embeddings in preprocess based on additional_information only. is_tts_model = "Qwen3-TTS" in self.model_name or "qwen3_tts" in self.model_name.lower() diff --git a/tests/model_executor/models/voxtral_tts/test_text_preprocess.py b/tests/model_executor/models/voxtral_tts/test_text_preprocess.py index 2e7b71091..31195ab56 100644 --- a/tests/model_executor/models/voxtral_tts/test_text_preprocess.py +++ b/tests/model_executor/models/voxtral_tts/test_text_preprocess.py @@ -3,7 +3,7 @@ """Unit tests for the Voxtral TTS demo text preprocessing helper. These tests intentionally cover only behavior ported into -examples/online_serving/voxtral_tts/text_preprocess.py. +examples/online_serving/text_to_speech/voxtral_tts/text_preprocess.py. """ import importlib.util @@ -13,7 +13,12 @@ from pathlib import Path import pytest _HELPER_PATH = ( - Path(__file__).resolve().parents[4] / "examples" / "online_serving" / "voxtral_tts" / "text_preprocess.py" + Path(__file__).resolve().parents[4] + / "examples" + / "online_serving" + / "text_to_speech" + / "voxtral_tts" + / "text_preprocess.py" ) pytestmark = [pytest.mark.core_model, pytest.mark.cpu]