[Model] Add Gepard-1.0 OpenAI /v1/audio/speech serving (#7499)
Signed-off-by: Mengjie Zhao <zmj0129@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
1 parent
78934753ff
commit
c17126c219
17 files changed
+1329
-25
No files matched your search
@@ -6,6 +6,7 @@ vLLM-Omni provides an OpenAI-compatible API for text-to-speech (TTS) generation.
|
||||
- **Fish Speech S2 Pro** (`fishaudio/s2-pro`) -- Dual-AR TTS with DAC codec. Supports text-to-speech and voice cloning via reference audio. Output: 44.1 kHz.
|
||||
- **Voxtral TTS** (`mistralai/Voxtral-4B-TTS-2603`) -- AR + FlowMatching TTS with preset voices. Output: 24 kHz.
|
||||
- **CosyVoice3** (`FunAudioLLM/Fun-CosyVoice3-0.5B-2512`) -- 2-stage talker + flow-matching code2wav. Voice cloning via `ref_audio` + `ref_text` (no presets). Output: 24 kHz.
|
||||
- **Gepard-1.0** (`nineninesix/gepard-1.0`) -- single-stage native-AR TTS with a 22.05 kHz NanoCodec. Zero-shot default voice only.
|
||||
|
||||
See the [Supported Models](#supported-models) section below for the full list, including OmniVoice, VoxCPM2, MOSS-TTS-Nano, and Breeze-TTS-2.
|
||||
|
||||
@@ -40,11 +41,16 @@ vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8091
|
||||
# CosyVoice3 (voice cloning only — supply ref_audio + ref_text per request)
|
||||
vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 \
|
||||
--omni --port 8091 --trust-remote-code
|
||||
|
||||
# Gepard-1.0 (zero-shot default voice; packaged gepard.yaml)
|
||||
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
|
||||
--stage-init-timeout 900 \
|
||||
--deploy-config vllm_omni/deploy/gepard.yaml
|
||||
```
|
||||
|
||||
### Generate Speech
|
||||
|
||||
**Using curl:**
|
||||
**Qwen3-TTS CustomVoice, using curl:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
@@ -56,7 +62,24 @@ curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
**Using Python:**
|
||||
**Gepard-1.0, using curl:**
|
||||
|
||||
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`,
|
||||
`stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields
|
||||
include `speed`, `language`, `instructions`, `task_type`, `ref_audio`,
|
||||
`ref_text`, `extra_params`, and `word_timestamps`.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is Gepard speaking.",
|
||||
"voice": "default",
|
||||
"seed": 7
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
**Qwen3-TTS CustomVoice, using Python:**
|
||||
|
||||
```python
|
||||
import httpx
|
||||
@@ -761,6 +784,28 @@ Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` ne
|
||||
| ------- | ------------- |
|
||||
| `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | Voice cloning from `ref_audio` + `ref_text`. No built-in voice presets — upload a voice or pass `ref_audio`/`ref_text` per request. |
|
||||
|
||||
### Gepard-1.0
|
||||
|
||||
| Model | Description |
|
||||
| ----- | ----------- |
|
||||
| `nineninesix/gepard-1.0` | Zero-shot native-AR TTS. 22.05 kHz mono. `voice` must be omitted or `"default"`. |
|
||||
|
||||
Gepard request fields:
|
||||
|
||||
| Field | Behavior |
|
||||
| ----- | -------- |
|
||||
| `input` | Required. Empty/whitespace-only returns 400. |
|
||||
| `voice` | Omitted or `"default"` only. Other values 400. |
|
||||
| `response_format` | `wav` default. Non-streaming: `wav`/`pcm`/`flac`/`mp3`. `opus` returns 400 (22.05 kHz is not an Opus sample rate). Streaming: `pcm`/`wav` only. |
|
||||
| `stream` / `stream_format` | SSE (`speech.audio.*`) and raw `audio` byte streaming. |
|
||||
| `speed` | Must be `1.0`. Gepard has no native speed control. |
|
||||
| `max_new_tokens` | Frame budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms). Default 1000 from deploy YAML; adapter bounds 1..4096. |
|
||||
| `seed` | Optional. Reaches the in-model 32-head sampler. The packaged YAML currently pins `seed: 42`, so serving is deterministic by default until that pin is removed. |
|
||||
| `extra_params` | Any key returns 400, including `temperature`/`top_p`/`top_k`. |
|
||||
| Cloning / style fields | `ref_audio`, `ref_text`, `speaker_embedding`, `instructions`, `language`, `task_type`, `word_timestamps`, and similar declared-but-unsupported fields return 400. |
|
||||
|
||||
See the [Gepard section of the online TTS hub](../user_guide/examples/online_serving/text_to_speech.md#gepard-10) for launch commands. Native-AR recompute preemption is a known limitation under concurrency.
|
||||
|
||||
### OmniVoice
|
||||
|
||||
| Model | Description |
|
||||
|
||||
@@ -18,6 +18,7 @@ For the full list of supported architectures across all modalities, see
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | — |
|
||||
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
|
||||
| Gepard-1.0 | `nineninesix/gepard-1.0` | — (zero-shot default voice) | ✓ (PCM / WAV stream) | `default` only | — |
|
||||
| higgs-audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✓ (`ref_audio`+`ref_text`) | ✓ (codec_streaming) | — | — |
|
||||
| GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
|
||||
| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
|
||||
@@ -92,7 +93,7 @@ curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
|
||||
```
|
||||
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others).
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for Gepard, 24 kHz for the others).
|
||||
|
||||
For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md).
|
||||
|
||||
@@ -212,6 +213,61 @@ python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server
|
||||
|
||||
---
|
||||
|
||||
## Gepard-1.0
|
||||
|
||||
Single-stage native AR TTS at 22.05 kHz mono. Zero-shot only: omit `voice` or pass `"default"`. Voice cloning from reference audio is not available yet.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Same NeMo NanoCodec install as the [offline Gepard section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#gepard-10). On a host whose CUDA toolkit cannot build kernels — no `nvcc`/`ninja`, or a consumer Blackwell (`sm_120`) card — also `export VLLM_USE_FLASHINFER_SAMPLER=0` before launch. That env is not a deploy-YAML field.
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
|
||||
--stage-init-timeout 900 \
|
||||
--deploy-config vllm_omni/deploy/gepard.yaml
|
||||
# or:
|
||||
./gepard/run_server.sh
|
||||
```
|
||||
|
||||
`--stage-init-timeout 900` matches the online e2e fixture and the offline example. Serve defaults to 300s, which is often too short for a cold download of the talker plus NanoCodec.
|
||||
|
||||
The packaged `vllm_omni/deploy/gepard.yaml` must be passed with `--deploy-config`. The checkpoint self-identifies as `qwen3_5_text`, so omitting the YAML launches a diffusion fallback instead of the Gepard pipeline. The YAML sets `async_chunk: false`, `max_num_seqs: 4`, and currently pins `seed: 42`, so serving is deterministic by default until that YAML seed is removed. Pass an explicit per-request `seed` in tests and clients rather than depending on either default.
|
||||
|
||||
### Sending requests
|
||||
|
||||
```bash
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking."
|
||||
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking." --seed 7 --stream --output output.pcm
|
||||
```
|
||||
|
||||
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`, `stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields include `speed`, `language`, `instructions`, `task_type`, `ref_audio`, `ref_text`, `extra_params`, and `word_timestamps`.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is Gepard speaking.",
|
||||
"voice": "default",
|
||||
"seed": 7
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 22.05 kHz mono. `max_new_tokens` is a **frame** budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms at 21.5 fps; adapter bounds 1..4096).
|
||||
- Supported request fields: `input` (required), `voice` (`default` only), `response_format` (`wav` default; `wav/pcm/flac/mp3` non-streaming; `opus` 400 because 22.05 kHz is not an Opus sample rate; streaming `pcm/wav` only), `stream` / `stream_format`, `max_new_tokens`, `seed`.
|
||||
- Unsupported: `speed`, `extra_params` (including `temperature`/`top_p`/`top_k`), `ref_audio`, `ref_text`, `speaker_embedding`, `task_type`, `instructions`, `language`, and `word_timestamps`.
|
||||
- Concurrent requests at `max_num_seqs: 4` are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below `max_num_seqs` and treat preemption as out of scope until the platform fix lands.
|
||||
- Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.
|
||||
- For the request-field table, see the [Speech API Gepard section](https://github.com/vllm-project/vllm-omni/blob/main/docs/serving/speech_api.md#gepard-10).
|
||||
|
||||
---
|
||||
|
||||
## higgs-audio v2
|
||||
|
||||
2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) that streams chunks back to the client.
|
||||
|
||||
Reference in new issue
Block a user