diff --git a/recipes/README.md b/recipes/README.md index ef11edeb3..20248b59c 100644 --- a/recipes/README.md +++ b/recipes/README.md @@ -34,6 +34,7 @@ recipes/ | [`inclusionAI/Ming-flash-omni-2.0.md`](./inclusionAI/Ming-flash-omni-2.0.md) | Online serving for multimodal chat + standalone TTS | 4x H100 / 1x H100 80GB | | [`LTX/LTX-2.md`](./LTX/LTX-2.md) | Text-to-video and image-to-video serving | 1x H200 141GB | | [`LTX/LTX-2.3.md`](./LTX/LTX-2.3.md) | Text-to-video with audio generation (22B) | 1x GPU (96GB VRAM) | +| [`mistralai/Voxtral-TTS.md`](./mistralai/Voxtral-TTS.md) | Online serving for TTS | 1x RTX 4090 24GB | | [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt mixed FP8/NVFP4 | 1x A100 80GB / 2x B200 | | [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay | 1x A100 80GB | | [`Qwen/Qwen-Image-Edit.md`](./Qwen/Qwen-Image-Edit.md) | Text-guided single-image editing | 1x or 2x H200 141GB | diff --git a/recipes/mistralai/Voxtral-TTS.md b/recipes/mistralai/Voxtral-TTS.md new file mode 100644 index 000000000..595d5c29e --- /dev/null +++ b/recipes/mistralai/Voxtral-TTS.md @@ -0,0 +1,107 @@ +# Voxtral TTS for text-to-speech + +## Summary + +- Vendor: Mistral AI +- Model: `mistralai/Voxtral-4B-TTS-2603` +- Task: Text-to-speech with model-provided voice presets or voice cloning +- Mode: Online serving with the OpenAI-compatible `/v1/audio/speech` API +- Maintainer: Community + +## When to use this recipe + +Use this recipe when you want a known-good starting point for serving Voxtral TTS +models with vLLM-Omni and validate the deployment with the existing TTS client +examples in this repository. + +## References + +- Upstream model card: +- Related examples under `examples/`: + [`examples/offline_inference/text_to_speech/voxtral_tts/end2end.py`](../../examples/offline_inference/text_to_speech/voxtral_tts/end2end.py) + [`examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py`](../../examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py) +- Related issue or discussion: + [RFC: add recipes folder](https://github.com/vllm-project/vllm-omni/issues/2645) + +## Hardware Support + +## GPU + +### 1x RTX 4090 24GB + +#### Environment + +- OS: Ubuntu 22.04.5 LTS +- Python: 3.12.11 +- PyTorch: 2.11.0+cu130 +- Driver / runtime: CUDA 13.0 +- GPU: NVIDIA GeForce RTX 4090, 24 GB +- vLLM version: 0.20.0 +- vLLM-Omni version or commit: 0.20.0rc2.dev99+g857356d5b +- Required package: mistral_common >= 1.10.0 + +#### Command + +Online: + +```bash +vllm serve mistralai/Voxtral-4B-TTS-2603 \ + --omni \ + --port 8091 +``` + +The deploy config is auto-loaded from `vllm_omni/deploy/voxtral_tts.yaml` + +#### Verification + +**Quick smoke test with curl:** + +```bash +curl -X POST http://127.0.0.1:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "model": "mistralai/Voxtral-4B-TTS-2603", + "input": "Hello, this is Voxtral TTS running with vLLM-Omni.", + "voice": "casual_female", + "language": "English", + "response_format": "wav" + }' \ + --output voxtral.wav +``` + +**Streaming audio:** + +```bash +curl -X POST http://127.0.0.1:8091/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{ + "model": "mistralai/Voxtral-4B-TTS-2603", + "input": "Hello, this is Voxtral TTS streaming PCM.", + "voice": "casual_female", + "language": "English", + "stream": true, + "response_format": "pcm" + }' \ + --output voxtral_stream.pcm +``` +Convert the raw PCM output to WAV +```bash +ffmpeg -f s16le -ar 24000 -ac 1 -i voxtral_stream.pcm voxtral_stream.wav -y +ffprobe -hide_banner voxtral_stream.wav +``` + +#### Notes + +- Memory usage: Stage 0 (`audio_generation`) ~18.95 GiB, Stage 1 + (`audio_tokenizer`) ~1.55 GiB. The observed server-startup peak was + `21006 MiB / 24564 MiB` (`20.51 GiB / 23.99 GiB`, about 85.5%). + Both stages share GPU 0 via the deploy config (`gpu_memory_utilization: 0.8` + for Stage 0, `0.1` for Stage 1). +- Key flags: `--omni` is required. `--port 8091` is optional and only selects + the HTTP port. +- Known limitation: Voxtral TTS outputs mono audio at 24 kHz. And with the public +mistralai/Voxtral-4B-TTS-2603 checkpoint. running the +voice-cloning e2e test ([`Voice cloning (capability gated upstream)`](../../examples/offline_inference/text_to_speech/README.md#voice-cloning-capability-gated-upstream) +) or adding a `ref_audio` request parameter will fail +because the public checkpoint does not provide the encoder weights needed to +convert reference audio into conditioning features. The expected failure is: `RuntimeError: encode_waveforms requires encoder weights which are not available in the open-source checkpoint.`