Files
vllm-omni/recipes/mistralai/Voxtral-TTS.md
T
2026-05-31 00:18:52 +08:00

3.5 KiB

Voxtral TTS for text-to-speech

Summary

  • Vendor: Mistral AI
  • Model: mistralai/Voxtral-4B-TTS-2603
  • Task: Text-to-speech with model-provided voice presets or voice cloning
  • Mode: Online serving with the OpenAI-compatible /v1/audio/speech API
  • Maintainer: Community

When to use this recipe

Use this recipe when you want a known-good starting point for serving Voxtral TTS models with vLLM-Omni and validate the deployment with the existing TTS client examples in this repository.

References

Hardware Support

GPU

1x RTX 4090 24GB

Environment

  • OS: Ubuntu 22.04.5 LTS
  • Python: 3.12.11
  • PyTorch: 2.11.0+cu130
  • Driver / runtime: CUDA 13.0
  • GPU: NVIDIA GeForce RTX 4090, 24 GB
  • vLLM version: 0.20.0
  • vLLM-Omni version or commit: 0.20.0rc2.dev99+g857356d5b
  • Required package: mistral_common >= 1.10.0

Command

Online:

vllm serve mistralai/Voxtral-4B-TTS-2603 \
    --omni \
    --port 8091

The deploy config is auto-loaded from vllm_omni/deploy/voxtral_tts.yaml

Verification

Quick smoke test with curl:

curl -X POST http://127.0.0.1:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Voxtral-4B-TTS-2603",
    "input": "Hello, this is Voxtral TTS running with vLLM-Omni.",
    "voice": "casual_female",
    "language": "English",
    "response_format": "wav"
  }' \
  --output voxtral.wav

Streaming audio:

curl -X POST http://127.0.0.1:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Voxtral-4B-TTS-2603",
    "input": "Hello, this is Voxtral TTS streaming PCM.",
    "voice": "casual_female",
    "language": "English",
    "stream": true,
    "response_format": "pcm"
  }' \
  --output voxtral_stream.pcm

Convert the raw PCM output to WAV

ffmpeg -f s16le -ar 24000 -ac 1 -i voxtral_stream.pcm voxtral_stream.wav -y
ffprobe -hide_banner voxtral_stream.wav

Notes

  • Memory usage: Stage 0 (audio_generation) ~18.95 GiB, Stage 1 (audio_tokenizer) ~1.55 GiB. The observed server-startup peak was 21006 MiB / 24564 MiB (20.51 GiB / 23.99 GiB, about 85.5%). Both stages share GPU 0 via the deploy config (gpu_memory_utilization: 0.8 for Stage 0, 0.1 for Stage 1).
  • Key flags: --omni is required. --port 8091 is optional and only selects the HTTP port.
  • Known limitation: Voxtral TTS outputs mono audio at 24 kHz. And with the public mistralai/Voxtral-4B-TTS-2603 checkpoint. running the voice-cloning e2e test (Voice cloning (capability gated upstream) ) or adding a ref_audio request parameter will fail because the public checkpoint does not provide the encoder weights needed to convert reference audio into conditioning features. The expected failure is: RuntimeError: encode_waveforms requires encoder weights which are not available in the open-source checkpoint.