From 926e7fbfa2340d033016be79bc5b57dff9152a20 Mon Sep 17 00:00:00 2001 From: harley Date: Wed, 26 Aug 2026 22:16:49 +0800 Subject: [PATCH] [Recipe] zai-org/GLM-TTS (2x non-standard RTX 4090 48GB) (#5769) Signed-off-by: harley <480497125@qq.com> --- recipes/README.md | 2 +- recipes/zai-org/GLM-TTS.md | 136 ++++++++++++++++++++++++++++++++++++- 2 files changed, 136 insertions(+), 2 deletions(-) diff --git a/recipes/README.md b/recipes/README.md index 5003e0d6e..f82a5955d 100644 --- a/recipes/README.md +++ b/recipes/README.md @@ -87,7 +87,7 @@ recipes/ | [`Wan-AI/Wan2.1-VACE.md`](./Wan-AI/Wan2.1-VACE.md) | Unified T2V, I2V, V2LF, FLF2V, inpaint, and R2V | 1x RTX 5090 (1.3B) / 1x L40S 48GB with layerwise offload (14B) | | [`XiaomiMiMo/MiMo-Audio.md`](./XiaomiMiMo/MiMo-Audio.md) | Offline + online omni audio (TTS, ASR, dialogue) | 1x RTX 5090 / 5090D 32GB | | [`StabilityAI/Stable-Diffusion-3.5.md`](./StabilityAI/Stable-Diffusion-3.5.md) | Text-to-image serving (SD 3.5-medium and SD 3.5-large) | 1x RTX A6000 48GB | -| [`zai-org/GLM-TTS.md`](./zai-org/GLM-TTS.md) | Online serving for Chinese/English zero-shot voice-cloned TTS | 1x A40 48GB | +| [`zai-org/GLM-TTS.md`](./zai-org/GLM-TTS.md) | Online serving for Chinese/English zero-shot voice-cloned TTS | 1x A40 48GB / 2x non-standard RTX 4090 48GB (49,140 MiB each) | | [`GLM/GLM-Image.md`](./GLM/GLM-Image.md) | Online serving for image generation | 1x A800 80GB / 2x A800 80GB | | [`JD/JoyAI-VL-Interaction.md`](./JD/JoyAI-VL-Interaction.md) | Real-time streaming video-language interaction (proactive speak/silence/delegate) | 1x GPU 24GB+ | diff --git a/recipes/zai-org/GLM-TTS.md b/recipes/zai-org/GLM-TTS.md index c7f331d96..79bb85c47 100644 --- a/recipes/zai-org/GLM-TTS.md +++ b/recipes/zai-org/GLM-TTS.md @@ -1,4 +1,4 @@ -# GLM-TTS for Chinese/English TTS on 1x GPU +# GLM-TTS for Chinese/English TTS ## Summary @@ -103,3 +103,137 @@ python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \ - Key flags: `--omni` is required; `--trust-remote-code` is needed for the GLM-TTS phoneme tokenizer; the DiT stage enables bucketed model-internal CUDA graphs with eager fallback. - Voice cloning: requires `ref_audio` + `ref_text` together. Reference audio should be 3-10 seconds. Feature extraction (WhisperVQ tokenizer, CampPlus ONNX, mel) runs on the model side. - Known limitations: First request may be slow due to lazy model loading (WhisperVQ, CampPlus ONNX). Warm-cache RTF is approximately 0.6-0.7x on A40. + +### 2x non-standard RTX 4090 48GB + +This configuration was personally validated on two non-standard RTX 4090 cards +reporting 49,140 MiB each in `nvidia-smi`. Stock 24GB RTX 4090 cards were not +validated. + +#### Environment + +- OS: Ubuntu 22.04.1 LTS +- NVIDIA driver: 580.76.05 +- Python: 3.12.13 +- PyTorch: 2.11.0+cu130 +- CUDA runtime and JIT toolkit: 13.0 +- vLLM: 0.26.0 +- vLLM-Omni: commit `66a9b0c8`, version + `0.26.0rc2.dev5+g66a9b0c84` +- FlashInfer: 0.6.14 + +The host's default `nvcc` was 11.8. The working CUDA toolkit came from the +Python environment, so `CUDA_HOME` was set to +`$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cu13`. + +#### Command + +From the repository root, start the async-chunk server with: + +```bash +export CUDA_HOME="$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cu13" +export PATH="$CUDA_HOME/bin:$PATH" +export LD_LIBRARY_PATH="$CUDA_HOME/lib:${LD_LIBRARY_PATH:-}" + +CUDA_VISIBLE_DEVICES=0,1 vllm-omni serve zai-org/GLM-TTS \ + --omni \ + --trust-remote-code \ + --port 8091 \ + --deploy-config vllm_omni/deploy/glm_tts.yaml \ + --stage-overrides '{"1":{"devices":"1"}}' \ + --allowed-local-media-path "${PWD}/tests/assets" +``` + +Stage 0 runs on GPU 0, while the override places stage 1 on GPU 1. For the sync +variant, add `--no-async-chunk` to the same command. + +#### Verification + +Run the Chinese voice-cloning check: + +```bash +python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \ + --model zai-org/GLM-TTS \ + --text "今天天气很好，我们一起学习语音模型推理。" \ + --ref-audio "file://${PWD}/tests/assets/glm_tts/jiayan_zh.wav" \ + --ref-text "他当时还跟线下其他的站姐吵架，然后，打架进局子了。" \ + --response-format wav \ + --output glm_tts_zh.wav +``` + +Run the English voice-cloning check with the same Chinese reference: + +```bash +python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \ + --model zai-org/GLM-TTS \ + --text "Hello, this is a voice cloning test with GLM-TTS." \ + --ref-audio "file://${PWD}/tests/assets/glm_tts/jiayan_zh.wav" \ + --ref-text "他当时还跟线下其他的站姐吵架，然后，打架进局子了。" \ + --response-format wav \ + --output glm_tts_en.wav +``` + +Inspect either output: + +```bash +ffprobe -v error -select_streams a:0 \ + -show_entries stream=codec_name,sample_rate,channels \ + -of default=noprint_wrappers=1 glm_tts_zh.wav +``` + +Expected output: + +```text +codec_name=pcm_s16le +sample_rate=24000 +channels=1 +``` + +The English smoke test produced a valid WAV with an actual duration of 5.020 +seconds. + +#### Observed warm-cache results + +These observations are from one controlled run, not a hardware or performance +guarantee. Cold-start and JIT costs were excluded. Values are mean +/- standard +deviation. + +##### Measurement procedure + +- For the single-GPU server, use the documented command with + `CUDA_VISIBLE_DEVICES=0` and omit `--stage-overrides`; both bundled stages + remain on GPU 0. For the dual-GPU server, use the command as shown. +- Async chunking is the default. Add `--no-async-chunk` for sync mode. +- For non-streaming measurements, use the exact Chinese client command above. + For streaming, add `--stream` to that command and change the output filename + to `glm_tts_zh.pcm`; streaming output is raw PCM, not WAV. +- The fixed Chinese prompt and reference were exactly those shown in + Verification and yielded 3.900 seconds of audio. Each case used one warm-up + followed by three measured requests at concurrency 1. +- Wall time starts immediately before the HTTP POST and stops after the complete + response body. TTFA runs from immediately before the POST to the first + non-empty audio bytes. RTF is wall time divided by audio duration. +- Peak total device memory used was sampled every 0.2 seconds and reported by + NVML via `nvmlDeviceGetMemoryInfo.used`. The readings include background and + baseline use and are not baseline-subtracted. + +| GPUs | Mode | Response | Wall time (s) | TTFA (s) | RTF | +|---:|---|---|---:|---:|---:| +| 1 | Async | Non-streaming | 2.144 +/- 0.076 | -- | 0.550 +/- 0.019 | +| 2 | Async | Non-streaming | 1.969 +/- 0.130 | -- | 0.505 +/- 0.033 | +| 1 | Async | Streaming | 2.135 +/- 0.093 | 1.245 +/- 0.178 | 0.548 +/- 0.024 | +| 2 | Async | Streaming | 1.891 +/- 0.006 | 1.170 +/- 0.042 | 0.485 +/- 0.002 | +| 1 | Sync | Non-streaming | 1.822 +/- 0.016 | -- | 0.467 +/- 0.004 | +| 2 | Sync | Non-streaming | 1.792 +/- 0.023 | -- | 0.459 +/- 0.006 | + +Moving stage 1 to GPU 1 reduced async non-streaming wall time by 8.2%. It +reduced async streaming wall time by 11.4% and TTFA by 6.0%, and reduced sync +wall time by 1.6%. Single- and dual-GPU outputs were bitwise identical only +within matching execution and response modes. + +| Configuration | GPU 0 | GPU 1 | +|---|---:|---:| +| Single-GPU async | 31.52 GiB | -- | +| Dual-GPU async | 29.23 GiB | 2.91 GiB | +| Single-GPU sync | 31.47 GiB | -- | +| Dual-GPU sync | 29.23 GiB | 2.91 GiB |