[Model] Add GLM-TTS text-to-speech model support (#3141)

Signed-off-by: BeatSeat <wendavid552@gmail.com>
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
This commit is contained in:
BeatSeatandYueqian Lin authored and GitHub committed 2026-05-24 05:42:26 +08:00
1 parent 748470c1fb
commit 28ce618f0e
38 files changed
+7568 -7

No files matched your search

@@ -19,6 +19,7 @@ list of supported architectures across all modalities, see
| VoxCPM2 | `openbmb/VoxCPM2` | single (native AR) | ✓ | — | continuation (`--ref-audio` + `--ref-text`) | 48 kHz |
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ (`async_chunk: true` default) | — | 22.05 kHz |
| Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ (`--streaming`) | — | 44.1 kHz |
| GLM-TTS | `zai-org/GLM-TTS` | 2 (AR + DiT) | ✓ (required) | ✓ | — | 24 kHz |
| OmniVoice | `k2-fsa/OmniVoice` | 2 (gen + dec) | ✓ | — | voice design (`--instruct`), language (`--lang`) | 24 kHz |
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | 2 (talker + code2wav) | ✓ (Base) | ✓ | 3 task variants (`--query-type`) | 24 kHz |
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | varies | ✓ | ✓ | voice presets (`--voice`) | 24 kHz |
@@ -158,6 +159,35 @@ Streaming requires `async_chunk: true` in the stage config.
---
## GLM-TTS
2-stage TTS pipeline (AR + DiT flow-matching) at 24 kHz. Every request requires reference audio and its transcript for zero-shot voice cloning.
### Quick start
```bash
python examples/offline_inference/text_to_speech/glm_tts/end2end.py \
--model zai-org/GLM-TTS \
--text "你好,这是语音合成测试。" \
--ref-audio /path/to/reference.wav \
--ref-text "这是参考音频的文本内容。" \
--output-dir ./output
```
### Architecture
```
Text → [Stage 0: AR] → Speech Tokens → [Stage 1: DiT + HiFT] → Audio (24 kHz)
(Llama-based) (32k vocab) (Flow Matching)
```
### Notes
- `--ref-audio` and `--ref-text` are **required** together; GLM-TTS does not support text-only synthesis.
- Reference audio should be 3-10 seconds.
- First run may be slow due to lazy loading of WhisperVQ tokenizer and CampPlus ONNX speaker embedder.
- Default sampling: temperature=1.0, top_k=25, top_p=0.8 (RAS method).
- The `--model` path should point to the repository root (not `llm/` subdirectory).
---
## OmniVoice
Zero-shot multilingual TTS supporting 600+ languages, with three modes (auto / clone / design).
@@ -313,6 +343,10 @@ Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-T
``````py
--8<-- "examples/offline_inference/text_to_speech/fish_speech/end2end.py"
``````
??? abstract "glm_tts/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/glm_tts/end2end.py"
``````
??? abstract "omnivoice/end2end.py"
``````py
--8<-- "examples/offline_inference/text_to_speech/omnivoice/end2end.py"