[Model] Add Voxtral TTS model (#1803)

Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
This commit is contained in:
Chen-Yo Sun authored and GitHub committed 2026-03-19 16:02:11 -04:00
1 parent b766c4702c
commit e9fe78e28c
29 files changed
+5135 -51

No files matched your search

+1
View File
@@ -141,6 +141,7 @@ def scan_package(package_name: str = "vllm_omni") -> dict[str, list[str]]:
"vllm_omni.diffusion.models.qwen_image",
"vllm_omni.entrypoints.async_diffusion",
"vllm_omni.entrypoints.openai",
"vllm_omni.model_executor.models.voxtral_tts.configuration_voxtral_tts",
]
if any(module_path.startswith(prefix) for prefix in excluded_prefixes):
continue
+1
View File
@@ -52,6 +52,7 @@ th {
|`Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` |
|`FishSpeechSlowARForConditionalGeneration` | Fish Speech S2 Pro | `fishaudio/s2-pro` |
|`DreamIDOmniPipeline`| DreamID-Omni | `XuGuo699/DreamID-Omni` |
|`VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/tts-model` |
## List of Supported Models for NPU
+15
View File
@@ -4,6 +4,7 @@ vLLM-Omni provides an OpenAI-compatible API for text-to-speech (TTS) generation.
- **Qwen3-TTS** (`Qwen/Qwen3-TTS-12Hz-*`) -- Qwen3-based TTS with CustomVoice, VoiceDesign, and Base (voice cloning) task types. Output: 24 kHz.
- **Fish Speech S2 Pro** (`fishaudio/s2-pro`) -- Dual-AR TTS with DAC codec. Supports text-to-speech and voice cloning via reference audio. Output: 44.1 kHz.
- **Voxtral TTS** (`mistralai/tts-model`) -- AR + FlowMatching TTS with preset voices. Output: 24 kHz.
Each server instance runs a single model (specified at startup via `vllm serve <model> --omni`).
@@ -28,6 +29,14 @@ vllm-omni serve fishaudio/s2-pro \
--trust-remote-code \
--enforce-eager \
--gpu-memory-utilization 0.9
# Voxtral TTS
vllm serve mistralai/tts-model \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxtral_tts.yaml \
--omni \
--port 8091 \
--trust-remote-code \
--enforce-eager
```
### Generate Speech
@@ -347,6 +356,12 @@ curl -X POST http://localhost:8091/v1/audio/speech \
Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` needed). The `voice` field should be set to `"default"`. See the [Fish Speech online serving example](../user_guide/examples/online_serving/fish_speech.md) for details.
### Voxtral TTS
| Model | Description |
|-------|-------------|
| `mistralai/tts-model` | 3B AR + FlowMatching TTS. Supports text-to-speech with preset voices. |
## Error Responses
### 400 Bad Request