251 lines
11 KiB
Markdown
251 lines
11 KiB
Markdown
# VoxCPM2
|
||
|
||
> Online + offline TTS with native AR pipeline (48 kHz, 30+ languages)
|
||
|
||
## Summary
|
||
|
||
- Vendor: OpenBMB
|
||
- Model: `openbmb/VoxCPM2`
|
||
- Task: Text-to-speech (zero-shot synthesis and reference-audio voice cloning)
|
||
- Mode: Online serving via the OpenAI-compatible `/v1/audio/speech` API, plus
|
||
an offline end-to-end example
|
||
- Maintainer: Community
|
||
|
||
## When to use this recipe
|
||
|
||
Use this recipe as a known-good starting point for serving `openbmb/VoxCPM2`
|
||
on vLLM-Omni on consumer-class GPUs. VoxCPM2 is a 2B-parameter native AR
|
||
TTS model that emits 48 kHz mono audio and supports 30+ languages out of
|
||
the box. It runs as a single-stage pipeline (`MiniCPM4 base LM → FSQ →
|
||
MiniCPM4 residual LM → LocDiT CFM solver → AudioVAE`) and supports both
|
||
zero-shot synthesis and reference-audio voice cloning, with optional
|
||
gapless streaming via the bundled Gradio AudioWorklet demo.
|
||
|
||
Because the model is only 2B params, a single 24 GB consumer GPU is
|
||
sufficient — no tensor parallelism is required.
|
||
|
||
## References
|
||
|
||
- Example guide (online):
|
||
[`examples/online_serving/text_to_speech/README.md`](../../examples/online_serving/text_to_speech/README.md#voxcpm2)
|
||
- Online client / Gradio demo:
|
||
[`examples/online_serving/text_to_speech/voxcpm2/`](../../examples/online_serving/text_to_speech/voxcpm2/)
|
||
- Offline end-to-end script:
|
||
[`examples/offline_inference/text_to_speech/voxcpm2/end2end.py`](../../examples/offline_inference/text_to_speech/voxcpm2/end2end.py)
|
||
- Default deploy config:
|
||
[`vllm_omni/deploy/voxcpm2.yaml`](../../vllm_omni/deploy/voxcpm2.yaml)
|
||
- Talker / pipeline source:
|
||
[`vllm_omni/model_executor/models/voxcpm2/`](../../vllm_omni/model_executor/models/voxcpm2/)
|
||
- Speech API reference:
|
||
[`docs/serving/speech_api.md`](../../docs/serving/speech_api.md#voxcpm2)
|
||
- Related issue or discussion:
|
||
[RFC: add recipes folder](https://github.com/vllm-project/vllm-omni/issues/2645)
|
||
|
||
## Hardware Support
|
||
|
||
This recipe documents one tested 24 GB consumer-GPU configuration.
|
||
Larger-VRAM (H20 / H100 / A100) and other vendor sections (ROCm, NPU) are
|
||
welcome as community validation lands.
|
||
|
||
## GPU
|
||
|
||
### 1 x RTX 4090 24GB (Single GPU, Minimum Recommended)
|
||
|
||
VoxCPM2 (2B params in bfloat16) fits comfortably on a single 24 GB GPU.
|
||
The bundled default config at
|
||
[`vllm_omni/deploy/voxcpm2.yaml`](../../vllm_omni/deploy/voxcpm2.yaml)
|
||
(`gpu_memory_utilization: 0.9`, `max_num_seqs: 4`, `enforce_eager: true`)
|
||
loads cleanly on a 4090 with ~4.9 GiB for model weights, ~15.2 GiB for
|
||
KV cache, and ~2 GiB for the talker's CUDA-Graph / CFM / VAE buffers,
|
||
for a total resident footprint of about **22 GiB / 24 GiB** in steady
|
||
state.
|
||
|
||
#### Environment
|
||
|
||
- OS: Linux (Ubuntu 22.04)
|
||
- Python: 3.12
|
||
- Driver / runtime: NVIDIA driver 575.57.08, CUDA 12.9 runtime
|
||
- torch: 2.11.0+cu128
|
||
- vLLM: 0.21.0
|
||
- vLLM-Omni: 0.20.1.dev (current `main`)
|
||
- Native VoxCPM2 source: `voxcpm` >= 2.0
|
||
|
||
#### Command
|
||
|
||
Start the server from the repository root. On a clean 4090 with no other
|
||
tenant on the GPU, the defaults from `voxcpm2.yaml` work as-is:
|
||
|
||
```bash
|
||
vllm serve openbmb/VoxCPM2 --omni \
|
||
--host 0.0.0.0 --port 8000
|
||
```
|
||
|
||
The deploy config at
|
||
[`vllm_omni/deploy/voxcpm2.yaml`](../../vllm_omni/deploy/voxcpm2.yaml)
|
||
is loaded automatically by the model registry (HF `model_type=voxcpm2`).
|
||
Pass `--deploy-config <path>` to override.
|
||
|
||
#### Verification
|
||
|
||
**Server cold-start**: ~60 s from `vllm serve` to `Application startup
|
||
complete` (subprocess fork + vLLM 0.21 init + model load + flashinfer JIT
|
||
+ torch.compile of LocDiT / feat_encoder / AudioVAE + CUDA-Graph warmup).
|
||
The first request after startup pays a small additional cost; steady-state
|
||
requests are much faster.
|
||
|
||
**T1 — zero-shot synthesis (curl, WAV)**:
|
||
|
||
```bash
|
||
curl -X POST http://127.0.0.1:8000/v1/audio/speech \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"model": "voxcpm2",
|
||
"input": "Hello, this is VoxCPM2 served by vLLM-Omni on a single RTX 4090.",
|
||
"voice": "default",
|
||
"response_format": "wav"
|
||
}' --output output.wav
|
||
```
|
||
|
||
Observed: `output.wav` 491 KB, 5.12 s @ 48 kHz mono, peak amplitude 0.996.
|
||
Server-to-client wall time: 2.6 s (server-side RTF ~0.5 after warmup).
|
||
|
||
**T2 — bundled Python client (zero-shot)**:
|
||
|
||
```bash
|
||
python examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py \
|
||
--text "This audio was generated by the bundled openai_speech_client.py script against a vLLM-Omni server." \
|
||
--api-base http://127.0.0.1:8000 \
|
||
--output output.wav
|
||
```
|
||
|
||
Observed: `output.wav` 968 KB, 10.08 s @ 48 kHz mono, peak amplitude 0.949.
|
||
|
||
**T3 — voice cloning from a reference WAV**:
|
||
|
||
```bash
|
||
python examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py \
|
||
--text "This should sound like the reference speaker, but saying a different sentence on a single RTX 4090." \
|
||
--ref-audio /path/to/reference.wav \
|
||
--api-base http://127.0.0.1:8000 \
|
||
--output cloned.wav
|
||
```
|
||
|
||
Observed (using the T2 output `output.wav` as `--ref-audio`):
|
||
`cloned.wav` 568 KB, 5.92 s @ 48 kHz mono, peak amplitude 0.990;
|
||
end-to-end wall time 3.16 s.
|
||
|
||
`--ref-audio` accepts local file paths (auto-base64), HTTP/HTTPS URLs, or
|
||
`data:audio/wav;base64,...` data URIs. The `voice` field is required by
|
||
the OpenAI schema but ignored by VoxCPM2 unless it refers to an
|
||
uploaded voice — voice cloning is driven entirely by `ref_audio`.
|
||
|
||
**T4 — streaming PCM (raw, 48 kHz s16le)**:
|
||
|
||
```bash
|
||
curl -X POST http://127.0.0.1:8000/v1/audio/speech \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"model": "voxcpm2",
|
||
"input": "Streaming PCM test on a single RTX 4090 with VoxCPM2.",
|
||
"voice": "default",
|
||
"stream": true,
|
||
"stream_format": "audio",
|
||
"response_format": "pcm"
|
||
}' --no-buffer | play -t raw -r 48000 -e signed -b 16 -c 1 -
|
||
```
|
||
|
||
Observed: 230400 int16 samples = 4.80 s of PCM at 48 kHz, peak 32525 /
|
||
32767 (~99 % full scale). The player sample rate is **48 kHz**, not
|
||
24 kHz.
|
||
|
||
**T5 — gapless streaming (Gradio AudioWorklet demo)**:
|
||
|
||
```bash
|
||
python examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py \
|
||
--api-base http://127.0.0.1:8000
|
||
```
|
||
|
||
Observed: the demo binds to `http://0.0.0.0:7860` in ~4 s after launch.
|
||
The UI ("VoxCPM2 Streaming Demo") exposes three modes — **Voice Design**
|
||
(control instruction only), **Controllable Cloning** (ref audio +
|
||
optional style control), and **Ultimate Cloning** (ref audio +
|
||
transcript for audio continuation) — plus a `Stream (gapless)`
|
||
checkbox (default on) that drives the AudioWorklet streaming path. The
|
||
right pane includes runnable example prompts and a `Ready` status
|
||
indicator. End-to-end browser playback through `Generate Speech` was
|
||
verified working on the same vLLM-Omni server backing T1–T4 above.
|
||
|
||
**T6 — offline inference (no server, single process per request)**:
|
||
|
||
```bash
|
||
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
|
||
--text "Hello, this is VoxCPM2 native AR synthesis running on vLLM-Omni and a single RTX 4090." \
|
||
--output-dir output_audio
|
||
```
|
||
|
||
`end2end.py` is a one-shot script (init → one `generate()` → exit), so
|
||
its reported RTF includes per-process compile + CUDA-Graph capture +
|
||
first-request runtime warmup. Note that the default `voxcpm2.yaml` sets
|
||
`enforce_eager: true`, so vLLM's engine-level CUDA graphs are off by
|
||
design; the "CUDA-Graph capture" referenced throughout this recipe is the
|
||
talker's own `_CapturedGraph` over the CFM solver / AudioVAE, not the LM
|
||
decode loop. Flipping `enforce_eager=false` does **not** remove this cost.
|
||
To separate the warmup cost from the
|
||
talker's true per-step throughput, the table below reports **5
|
||
consecutive `engine.generate([prompt])` calls in the same process**
|
||
after a one-off ~28 s engine init:
|
||
|
||
| call | duration | inference | RTF | notes |
|
||
| ---- | -------- | --------- | ----- | ------------------------------------------------ |
|
||
| #1 | 6.72 s | 11.97 s | 1.782 | cold: torch.compile + CUDA-Graph capture |
|
||
| #2 | 6.24 s | 11.43 s | 1.831 | still runtime warmup |
|
||
| #3 | 6.88 s | 0.82 s | 0.120 | ⚡ steady-state |
|
||
| #4 | 6.56 s | 0.78 s | 0.119 | steady-state |
|
||
| #5 | 5.76 s | 0.70 s | 0.121 | steady-state |
|
||
|
||
**Steady-state RTF (mean of calls #3–#5): 0.120**, i.e. ~8× real-time on
|
||
a single 4090. The high call-#1 / call-#2 numbers are one-time warmup
|
||
costs that **amortise to zero after the second request** — they are not
|
||
the model's per-step throughput.
|
||
|
||
A literal one-shot `python end2end.py` invocation in a fresh process
|
||
only sees call #1 (~RTF 1.8). For repeated workloads, either reuse the
|
||
same engine across calls (as in the table above) or use the online
|
||
serving path (T1–T5); a long-lived server amortises the warmup across
|
||
all subsequent requests (T1 wall-time RTF was 0.51 with a warm server).
|
||
|
||
Voice cloning + continuation mode (`ref_audio` + transcript):
|
||
|
||
```bash
|
||
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
|
||
--text "Hello, this is the offline voice cloning test on a single RTX 4090." \
|
||
--ref-audio /path/to/reference.wav \
|
||
--ref-text "Original transcript of the reference audio."
|
||
```
|
||
|
||
Same 5-call methodology as the zero-shot table, this time with a
|
||
10.08 s reference WAV supplied via `--ref-audio` (and matching
|
||
`--ref-text`):
|
||
|
||
| call | duration | inference | RTF | notes |
|
||
| ---- | -------- | --------- | ----- | ------------------------------------------------ |
|
||
| #1 | 5.44 s | 12.38 s | 2.276 | cold: compile + CUDA-Graph capture + ref encode |
|
||
| #2 | 5.12 s | 2.47 s | 0.482 | most warmup done |
|
||
| #3 | 5.44 s | 0.76 s | 0.139 | ⚡ steady-state |
|
||
| #4 | 4.96 s | 0.68 s | 0.137 | steady-state |
|
||
| #5 | 5.28 s | 0.71 s | 0.134 | steady-state |
|
||
|
||
**Steady-state cloning RTF (mean of calls #3–#5): 0.137**, i.e. only
|
||
**+0.017 RTF** above the zero-shot steady-state of 0.120. That delta is
|
||
the per-request `AudioVAE.encode` over the reference clip (~85 ms for a
|
||
10 s ref WAV) plus the ref-continuation prompt build; the AR decode
|
||
steps themselves run at the same per-step rate as zero-shot.
|
||
|
||
#### Notes
|
||
|
||
- Prerequisites: `pip install voxcpm soundfile httpx ninja`. `ninja` must be on `PATH` because `flashinfer` JIT-compiles a top-k sampling kernel during the first profile pass (without it, startup dies with `FileNotFoundError: 'ninja'`).
|
||
- Memory usage: ~4.9 GiB model weights + ~15.2 GiB KV cache ≈ 22 GiB / 24 GiB resident with the default `voxcpm2.yaml`. On a shared GPU, pass `--gpu-memory-utilization 0.75` (or lower) if startup fails the free-memory check.
|
||
- Key flags: `--omni` is required. `--trust-remote-code` is not needed — the HF config registers under `model_type=voxcpm2`.
|
||
- Output: 48 kHz mono. When streaming PCM to a player, use `-r 48000`.
|
||
- Cold start: ~60 s server boot + ~25 s first-inference overhead (torch.compile + flashinfer JIT + the talker's own `_CapturedGraph` capture over CFM / AudioVAE — not vLLM engine CUDA graphs, which `enforce_eager: true` disables). Steady-state inference RTF is **~0.12** (~8× real-time on a single 4090; see T6 table for the per-call breakdown); online server requests run at ~RTF 0.5 wall-time (includes HTTP round-trip). Keep the server warm for interactive use.
|