anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
926e7fbfa2340d033016be79bc5b57dff9152a20
vllm-omni/recipes/zai-org/GLM-TTS.md
T

8.4 KiB
Raw Blame History

GLM-TTS for Chinese/English TTS

Summary

  • Vendor: zai-org
  • Model: zai-org/GLM-TTS
  • Task: Zero-shot voice-cloned text-to-speech synthesis
  • Mode: Online serving with the OpenAI-compatible /v1/audio/speech API
  • Maintainer: Community

When to use this recipe

Use this recipe to serve GLM-TTS as a two-stage TTS system (AR + DiT flow-matching) for Chinese and English speech synthesis. Every request is conditioned on reference audio and its transcript.

References

Hardware Support

GPU

1x A40 48GB

Environment

  • OS: Linux
  • Python: 3.10+
  • Driver / runtime: NVIDIA CUDA environment with A40 48GB GPU
  • vLLM version: Match the repository requirements for your checkout
  • vLLM-Omni version or commit: Use the commit you are deploying from

Command

Start the server from the repository root:

vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091

Async chunking is enabled by default in the bundled deployment config. For the sync (non-streaming) path:

vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091 --no-async-chunk

Use a custom deploy config for advanced cases:

vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091 \
  --deploy-config /path/to/your_glm_tts_overrides.yaml

Verification

Run the bundled OpenAI-compatible client with reference audio:

python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
  --text "你好，这是一个语音合成测试。" \
  --ref-audio file:///path/to/ref.wav \
  --ref-text "这是参考音频的文本内容。"

For a quick API smoke test:

curl http://localhost:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-TTS",
    "input": "你好，这是一个语音合成测试。",
    "response_format": "wav",
    "ref_audio": "file:///path/to/ref.wav",
    "ref_text": "这是参考音频的文本内容。"
  }' --output test.wav

Voice cloning with reference audio:

python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
  --text "你好，这是语音克隆测试。" \
  --ref-audio file:///path/to/ref.wav \
  --ref-text "这是参考音频的文本内容。"

Notes

  • Hardware scope: the default bundled config is CUDA-only and verified on 1x A40 48GB (~16.6 GiB peak); fits 24GB cards. Split stages across GPUs for higher concurrency.
  • Memory usage: ~18-20GB total (AR ~10GB, DiT ~8GB); both stages share GPU 0 by default.
  • Audio output: 24kHz mono WAV via HiFT vocoder (Vocos2D 32kHz fallback with resampling).
  • Key flags: --omni is required; --trust-remote-code is needed for the GLM-TTS phoneme tokenizer; the DiT stage enables bucketed model-internal CUDA graphs with eager fallback.
  • Voice cloning: requires ref_audio + ref_text together. Reference audio should be 3-10 seconds. Feature extraction (WhisperVQ tokenizer, CampPlus ONNX, mel) runs on the model side.
  • Known limitations: First request may be slow due to lazy model loading (WhisperVQ, CampPlus ONNX). Warm-cache RTF is approximately 0.6-0.7x on A40.

2x non-standard RTX 4090 48GB

This configuration was personally validated on two non-standard RTX 4090 cards reporting 49,140 MiB each in nvidia-smi. Stock 24GB RTX 4090 cards were not validated.

Environment

  • OS: Ubuntu 22.04.1 LTS
  • NVIDIA driver: 580.76.05
  • Python: 3.12.13
  • PyTorch: 2.11.0+cu130
  • CUDA runtime and JIT toolkit: 13.0
  • vLLM: 0.26.0
  • vLLM-Omni: commit 66a9b0c8, version 0.26.0rc2.dev5+g66a9b0c84
  • FlashInfer: 0.6.14

The host's default nvcc was 11.8. The working CUDA toolkit came from the Python environment, so CUDA_HOME was set to $VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cu13.

Command

From the repository root, start the async-chunk server with:

export CUDA_HOME="$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cu13"
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib:${LD_LIBRARY_PATH:-}"

CUDA_VISIBLE_DEVICES=0,1 vllm-omni serve zai-org/GLM-TTS \
  --omni \
  --trust-remote-code \
  --port 8091 \
  --deploy-config vllm_omni/deploy/glm_tts.yaml \
  --stage-overrides '{"1":{"devices":"1"}}' \
  --allowed-local-media-path "${PWD}/tests/assets"

Stage 0 runs on GPU 0, while the override places stage 1 on GPU 1. For the sync variant, add --no-async-chunk to the same command.

Verification

Run the Chinese voice-cloning check:

python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
  --model zai-org/GLM-TTS \
  --text "今天天气很好，我们一起学习语音模型推理。" \
  --ref-audio "file://${PWD}/tests/assets/glm_tts/jiayan_zh.wav" \
  --ref-text "他当时还跟线下其他的站姐吵架，然后，打架进局子了。" \
  --response-format wav \
  --output glm_tts_zh.wav

Run the English voice-cloning check with the same Chinese reference:

python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
  --model zai-org/GLM-TTS \
  --text "Hello, this is a voice cloning test with GLM-TTS." \
  --ref-audio "file://${PWD}/tests/assets/glm_tts/jiayan_zh.wav" \
  --ref-text "他当时还跟线下其他的站姐吵架，然后，打架进局子了。" \
  --response-format wav \
  --output glm_tts_en.wav

Inspect either output:

ffprobe -v error -select_streams a:0 \
  -show_entries stream=codec_name,sample_rate,channels \
  -of default=noprint_wrappers=1 glm_tts_zh.wav

Expected output:

codec_name=pcm_s16le
sample_rate=24000
channels=1

The English smoke test produced a valid WAV with an actual duration of 5.020 seconds.

Observed warm-cache results

These observations are from one controlled run, not a hardware or performance guarantee. Cold-start and JIT costs were excluded. Values are mean +/- standard deviation.

Measurement procedure
  • For the single-GPU server, use the documented command with CUDA_VISIBLE_DEVICES=0 and omit --stage-overrides; both bundled stages remain on GPU 0. For the dual-GPU server, use the command as shown.
  • Async chunking is the default. Add --no-async-chunk for sync mode.
  • For non-streaming measurements, use the exact Chinese client command above. For streaming, add --stream to that command and change the output filename to glm_tts_zh.pcm; streaming output is raw PCM, not WAV.
  • The fixed Chinese prompt and reference were exactly those shown in Verification and yielded 3.900 seconds of audio. Each case used one warm-up followed by three measured requests at concurrency 1.
  • Wall time starts immediately before the HTTP POST and stops after the complete response body. TTFA runs from immediately before the POST to the first non-empty audio bytes. RTF is wall time divided by audio duration.
  • Peak total device memory used was sampled every 0.2 seconds and reported by NVML via nvmlDeviceGetMemoryInfo.used. The readings include background and baseline use and are not baseline-subtracted.
GPUs Mode Response Wall time (s) TTFA (s) RTF
1 Async Non-streaming 2.144 +/- 0.076 -- 0.550 +/- 0.019
2 Async Non-streaming 1.969 +/- 0.130 -- 0.505 +/- 0.033
1 Async Streaming 2.135 +/- 0.093 1.245 +/- 0.178 0.548 +/- 0.024
2 Async Streaming 1.891 +/- 0.006 1.170 +/- 0.042 0.485 +/- 0.002
1 Sync Non-streaming 1.822 +/- 0.016 -- 0.467 +/- 0.004
2 Sync Non-streaming 1.792 +/- 0.023 -- 0.459 +/- 0.006

Moving stage 1 to GPU 1 reduced async non-streaming wall time by 8.2%. It reduced async streaming wall time by 11.4% and TTFA by 6.0%, and reduced sync wall time by 1.6%. Single- and dual-GPU outputs were bitwise identical only within matching execution and response modes.

Configuration GPU 0 GPU 1
Single-GPU async 31.52 GiB --
Dual-GPU async 29.23 GiB 2.91 GiB
Single-GPU sync 31.47 GiB --
Dual-GPU sync 29.23 GiB 2.91 GiB
Reference in new issue View Git Blame Copy Permalink