212 lines
6.4 KiB
Markdown
212 lines
6.4 KiB
Markdown
# Stable-Audio-Open Text-To-Audio Generation on 1x GPU
|
|
|
|
> Text-to-audio recipe for Stable Audio Open with offline inference and
|
|
> OpenAI-compatible online serving on 1x RTX 4090 24GB.
|
|
|
|
## Summary
|
|
|
|
- Vendor: Stability AI
|
|
- Model: `stabilityai/stable-audio-open-1.0`
|
|
- Task: Text-to-audio generation
|
|
- Mode: Offline inference and online serving
|
|
- Maintainer: Community
|
|
|
|
## When to use this recipe
|
|
|
|
Use this recipe when you want to run Stable Audio Open on a single RTX 4090
|
|
24GB GPU for music or sound-effect generation. The recipe covers a 10-second
|
|
offline validation sample with TeaCache and online serving through the
|
|
`/v1/audio/generate` endpoint.
|
|
|
|
## References
|
|
|
|
- Model: <https://huggingface.co/stabilityai/stable-audio-open-1.0>
|
|
- Offline example:
|
|
[`examples/offline_inference/text_to_audio`](../../examples/offline_inference/text_to_audio)
|
|
- Audio generation API:
|
|
[`docs/serving/audio_generate_api.md`](../../docs/serving/audio_generate_api.md)
|
|
- Related issue:
|
|
[#2645](https://github.com/vllm-project/vllm-omni/issues/2645)
|
|
|
|
## Hardware Support
|
|
|
|
### GPU
|
|
|
|
#### 1x RTX 4090 24GB
|
|
|
|
##### Environment
|
|
|
|
- OS: Ubuntu 22.04.5
|
|
- Python: 3.12
|
|
- GPU: NVIDIA GeForce RTX 4090, 24564 MiB VRAM
|
|
- Driver / runtime: NVIDIA driver 595.80, CUDA-capable runtime matching the
|
|
repository build
|
|
- vLLM version: 0.22.0
|
|
- vLLM-Omni version: source checkout
|
|
- PyTorch: 2.11.0+cu130
|
|
- Model path used in the commands below: `/path/to/stable-audio-open-1.0`
|
|
|
|
Stable Audio Open is a gated Hugging Face model. Accept the model license on
|
|
the Hugging Face model card before downloading the checkpoint.
|
|
|
|
```bash
|
|
hf auth login
|
|
|
|
hf download stabilityai/stable-audio-open-1.0 \
|
|
--local-dir /path/to/stable-audio-open-1.0
|
|
```
|
|
|
|
##### Commands
|
|
|
|
Run a 10-second offline validation sample from the repository root:
|
|
|
|
```bash
|
|
python examples/offline_inference/text_to_audio/text_to_audio.py \
|
|
--model /path/to/stable-audio-open-1.0 \
|
|
--prompt "A gentle piano melody with soft room ambience" \
|
|
--negative-prompt "Low quality, distorted, noisy" \
|
|
--seed 42 \
|
|
--guidance-scale 7.0 \
|
|
--audio-length 10.0 \
|
|
--num-inference-steps 50 \
|
|
--cache-backend tea_cache \
|
|
--output examples/offline_inference/text_to_audio/stable_audio_10s.wav
|
|
```
|
|
|
|
Start the online serving endpoint:
|
|
|
|
```bash
|
|
vllm serve /path/to/stable-audio-open-1.0 \
|
|
--host 0.0.0.0 \
|
|
--port 8091 \
|
|
--gpu-memory-utilization 0.9 \
|
|
--trust-remote-code \
|
|
--enforce-eager \
|
|
--omni
|
|
```
|
|
|
|
Generate a 10-second WAV file from the repository root in another terminal:
|
|
|
|
```bash
|
|
curl http://localhost:8091/health
|
|
|
|
curl -X POST http://localhost:8091/v1/audio/generate \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"input": "A gentle piano melody with soft room ambience",
|
|
"audio_length": 10.0,
|
|
"num_inference_steps": 50,
|
|
"guidance_scale": 7.0,
|
|
"negative_prompt": "Low quality, distorted, noisy",
|
|
"seed": 42,
|
|
"response_format": "wav"
|
|
}' \
|
|
--output piano_10s.wav
|
|
```
|
|
|
|
##### Verification
|
|
|
|
Check that:
|
|
|
|
- The offline command writes a valid WAV file.
|
|
- The server responds on `http://localhost:8091/health`.
|
|
- The online request writes a valid WAV file.
|
|
- The generated audio sample rate is 44.1 kHz.
|
|
- The generated duration is approximately 10 seconds.
|
|
- Peak sampled GPU memory is within the 24GB RTX 4090 budget. In the validated
|
|
run, offline and online generation each peaked at about 12.6 GiB.
|
|
|
|
Validate the offline outputs:
|
|
|
|
```bash
|
|
ls -lh examples/offline_inference/text_to_audio/stable_audio_10s.wav
|
|
|
|
python - <<'PY'
|
|
import soundfile as sf
|
|
|
|
path = "examples/offline_inference/text_to_audio/stable_audio_10s.wav"
|
|
audio, sample_rate = sf.read(path)
|
|
print("sample_rate:", sample_rate)
|
|
print("shape:", audio.shape)
|
|
print("duration:", len(audio) / sample_rate)
|
|
PY
|
|
```
|
|
|
|
Validate the online output:
|
|
|
|
```bash
|
|
ls -lh piano_10s.wav
|
|
|
|
python - <<'PY'
|
|
import soundfile as sf
|
|
|
|
path = "piano_10s.wav"
|
|
audio, sample_rate = sf.read(path)
|
|
print("sample_rate:", sample_rate)
|
|
print("shape:", audio.shape)
|
|
print("duration:", len(audio) / sample_rate)
|
|
PY
|
|
```
|
|
|
|
##### Notes
|
|
|
|
- `stable-audio-open-1.0` can generate up to about 47 seconds of 44.1 kHz
|
|
stereo audio. This recipe validates 10-second WAV outputs.
|
|
- `--cache-backend tea_cache` is supported and was used for the 10-second
|
|
offline validation command.
|
|
- The model is gated on Hugging Face and requires license acceptance before
|
|
download.
|
|
- If online serving fails while importing `torchaudio`, make sure the
|
|
`torchaudio` wheel matches the installed PyTorch and CUDA build. The
|
|
validated environment used `torch==2.11.0+cu130` and
|
|
`torchaudio==2.11.0+cu130`.
|
|
- The `NIXL is not available`, `GLOO_SOCKET_IFNAME`, and `torchsde` boundary
|
|
warnings observed during validation did not prevent successful generation.
|
|
- The RTX 4090 entry was validated on one 24 GB GPU. The MI300X entry below covers one 192 GB GPU.
|
|
- Long generations, higher inference-step counts, and non-WAV response formats
|
|
were not benchmarked in this recipe.
|
|
|
|
#### 1x AMD MI300X 192GB
|
|
|
|
##### Environment
|
|
|
|
- OS: Linux 6.8.0-134-generic, x86_64
|
|
- Container: official ROCm image built from `docker/Dockerfile.rocm`
|
|
- Python: 3.12.13
|
|
- PyTorch: 2.11.0+gitd0c8b1f
|
|
- Driver / runtime: AMD 6.19.14.31400000 / ROCm 7.2.53211
|
|
- GPU: one AMD Instinct MI300X, `gfx942:sramecc+:xnack-`, 191.69 GiB visible HBM
|
|
- vLLM version: 0.27.0+rocm723
|
|
- vLLM Omni version or commit: `73e1368c7bb940efe1a025859c9d6c8eeeb2e3f0`
|
|
- Installed vLLM Omni package metadata: `0.27.0rc2.dev44+g55abdade9.rocm`
|
|
|
|
##### Commands
|
|
|
|
```bash
|
|
python3 examples/offline_inference/text_to_audio/text_to_audio.py \
|
|
--model stabilityai/stable-audio-open-1.0 \
|
|
--prompt "A gentle piano melody with soft room ambience" \
|
|
--negative-prompt "Low quality, distorted, noisy" \
|
|
--seed 42 \
|
|
--guidance-scale 7.0 \
|
|
--audio-length 10.0 \
|
|
--num-inference-steps 50 \
|
|
--cache-backend tea_cache \
|
|
--enable-diffusion-pipeline-profiler \
|
|
--output stable_audio_10s.wav
|
|
```
|
|
|
|
##### Verification
|
|
|
|
The command completed and wrote a valid 44.1 kHz stereo WAV with 10.00 seconds of audio.
|
|
|
|
##### Notes
|
|
|
|
- TeaCache ran with `rel_l1_thresh=0.2`.
|
|
- Model loading used 2.7891 GiB and took 3.706 seconds.
|
|
- Generation took 4.750 seconds, which gives a real time factor of 0.475 for the 10.00 second output.
|
|
- The internal profiler recorded 15.65 GB reserved and 9.68 GB allocated for the request.
|
|
- The highest one second whole device memory sample was 19.61 GiB.
|
|
- The output RMS was 0.0887, and its peak absolute amplitude was 0.5761.
|
|
- The full process took 384 seconds, including startup and compilation.
|