[Recipe] add mistralai voxtral tts recipe (#3498)
Signed-off-by: Dmaner <2663515256@qq.com>
This commit is contained in:
2 files changed
+108
No files matched your search
@@ -34,6 +34,7 @@ recipes/
|
|||||||
| [`inclusionAI/Ming-flash-omni-2.0.md`](./inclusionAI/Ming-flash-omni-2.0.md) | Online serving for multimodal chat + standalone TTS | 4x H100 / 1x H100 80GB |
|
| [`inclusionAI/Ming-flash-omni-2.0.md`](./inclusionAI/Ming-flash-omni-2.0.md) | Online serving for multimodal chat + standalone TTS | 4x H100 / 1x H100 80GB |
|
||||||
| [`LTX/LTX-2.md`](./LTX/LTX-2.md) | Text-to-video and image-to-video serving | 1x H200 141GB |
|
| [`LTX/LTX-2.md`](./LTX/LTX-2.md) | Text-to-video and image-to-video serving | 1x H200 141GB |
|
||||||
| [`LTX/LTX-2.3.md`](./LTX/LTX-2.3.md) | Text-to-video with audio generation (22B) | 1x GPU (96GB VRAM) |
|
| [`LTX/LTX-2.3.md`](./LTX/LTX-2.3.md) | Text-to-video with audio generation (22B) | 1x GPU (96GB VRAM) |
|
||||||
|
| [`mistralai/Voxtral-TTS.md`](./mistralai/Voxtral-TTS.md) | Online serving for TTS | 1x RTX 4090 24GB |
|
||||||
| [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt mixed FP8/NVFP4 | 1x A100 80GB / 2x B200 |
|
| [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt mixed FP8/NVFP4 | 1x A100 80GB / 2x B200 |
|
||||||
| [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay | 1x A100 80GB |
|
| [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay | 1x A100 80GB |
|
||||||
| [`Qwen/Qwen-Image-Edit.md`](./Qwen/Qwen-Image-Edit.md) | Text-guided single-image editing | 1x or 2x H200 141GB |
|
| [`Qwen/Qwen-Image-Edit.md`](./Qwen/Qwen-Image-Edit.md) | Text-guided single-image editing | 1x or 2x H200 141GB |
|
||||||
|
|||||||
@@ -0,0 +1,107 @@
|
|||||||
|
# Voxtral TTS for text-to-speech
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
- Vendor: Mistral AI
|
||||||
|
- Model: `mistralai/Voxtral-4B-TTS-2603`
|
||||||
|
- Task: Text-to-speech with model-provided voice presets or voice cloning
|
||||||
|
- Mode: Online serving with the OpenAI-compatible `/v1/audio/speech` API
|
||||||
|
- Maintainer: Community
|
||||||
|
|
||||||
|
## When to use this recipe
|
||||||
|
|
||||||
|
Use this recipe when you want a known-good starting point for serving Voxtral TTS
|
||||||
|
models with vLLM-Omni and validate the deployment with the existing TTS client
|
||||||
|
examples in this repository.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- Upstream model card: <https://huggingface.co/mistralai/Voxtral-4B-TTS-2603>
|
||||||
|
- Related examples under `examples/`:
|
||||||
|
[`examples/offline_inference/text_to_speech/voxtral_tts/end2end.py`](../../examples/offline_inference/text_to_speech/voxtral_tts/end2end.py)
|
||||||
|
[`examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py`](../../examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py)
|
||||||
|
- Related issue or discussion:
|
||||||
|
[RFC: add recipes folder](https://github.com/vllm-project/vllm-omni/issues/2645)
|
||||||
|
|
||||||
|
## Hardware Support
|
||||||
|
|
||||||
|
## GPU
|
||||||
|
|
||||||
|
### 1x RTX 4090 24GB
|
||||||
|
|
||||||
|
#### Environment
|
||||||
|
|
||||||
|
- OS: Ubuntu 22.04.5 LTS
|
||||||
|
- Python: 3.12.11
|
||||||
|
- PyTorch: 2.11.0+cu130
|
||||||
|
- Driver / runtime: CUDA 13.0
|
||||||
|
- GPU: NVIDIA GeForce RTX 4090, 24 GB
|
||||||
|
- vLLM version: 0.20.0
|
||||||
|
- vLLM-Omni version or commit: 0.20.0rc2.dev99+g857356d5b
|
||||||
|
- Required package: mistral_common >= 1.10.0
|
||||||
|
|
||||||
|
#### Command
|
||||||
|
|
||||||
|
Online:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
vllm serve mistralai/Voxtral-4B-TTS-2603 \
|
||||||
|
--omni \
|
||||||
|
--port 8091
|
||||||
|
```
|
||||||
|
|
||||||
|
The deploy config is auto-loaded from `vllm_omni/deploy/voxtral_tts.yaml`
|
||||||
|
|
||||||
|
#### Verification
|
||||||
|
|
||||||
|
**Quick smoke test with curl:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://127.0.0.1:8091/v1/audio/speech \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "mistralai/Voxtral-4B-TTS-2603",
|
||||||
|
"input": "Hello, this is Voxtral TTS running with vLLM-Omni.",
|
||||||
|
"voice": "casual_female",
|
||||||
|
"language": "English",
|
||||||
|
"response_format": "wav"
|
||||||
|
}' \
|
||||||
|
--output voxtral.wav
|
||||||
|
```
|
||||||
|
|
||||||
|
**Streaming audio:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://127.0.0.1:8091/v1/audio/speech \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "mistralai/Voxtral-4B-TTS-2603",
|
||||||
|
"input": "Hello, this is Voxtral TTS streaming PCM.",
|
||||||
|
"voice": "casual_female",
|
||||||
|
"language": "English",
|
||||||
|
"stream": true,
|
||||||
|
"response_format": "pcm"
|
||||||
|
}' \
|
||||||
|
--output voxtral_stream.pcm
|
||||||
|
```
|
||||||
|
Convert the raw PCM output to WAV
|
||||||
|
```bash
|
||||||
|
ffmpeg -f s16le -ar 24000 -ac 1 -i voxtral_stream.pcm voxtral_stream.wav -y
|
||||||
|
ffprobe -hide_banner voxtral_stream.wav
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Notes
|
||||||
|
|
||||||
|
- Memory usage: Stage 0 (`audio_generation`) ~18.95 GiB, Stage 1
|
||||||
|
(`audio_tokenizer`) ~1.55 GiB. The observed server-startup peak was
|
||||||
|
`21006 MiB / 24564 MiB` (`20.51 GiB / 23.99 GiB`, about 85.5%).
|
||||||
|
Both stages share GPU 0 via the deploy config (`gpu_memory_utilization: 0.8`
|
||||||
|
for Stage 0, `0.1` for Stage 1).
|
||||||
|
- Key flags: `--omni` is required. `--port 8091` is optional and only selects
|
||||||
|
the HTTP port.
|
||||||
|
- Known limitation: Voxtral TTS outputs mono audio at 24 kHz. And with the public
|
||||||
|
mistralai/Voxtral-4B-TTS-2603 checkpoint. running the
|
||||||
|
voice-cloning e2e test ([`Voice cloning (capability gated upstream)`](../../examples/offline_inference/text_to_speech/README.md#voice-cloning-capability-gated-upstream)
|
||||||
|
) or adding a `ref_audio` request parameter will fail
|
||||||
|
because the public checkpoint does not provide the encoder weights needed to
|
||||||
|
convert reference audio into conditioning features. The expected failure is: `RuntimeError: encode_waveforms requires encoder weights which are not available in the open-source checkpoint.`
|
||||||
Reference in new issue
Block a user