vllm-omni/recipes/OpenBMB/VoxCPM2.md

# VoxCPM2

> Online + offline TTS with native AR pipeline (48 kHz, 30+ languages)

## Summary

- Vendor: OpenBMB
- Model: `openbmb/VoxCPM2`
- Task: Text-to-speech (zero-shot synthesis and reference-audio voice cloning)
- Mode: Online serving via the OpenAI-compatible `/v1/audio/speech` API, plus
  an offline end-to-end example
- Maintainer: Community

## When to use this recipe

Use this recipe as a known-good starting point for serving `openbmb/VoxCPM2`
on vLLM-Omni on consumer-class GPUs. VoxCPM2 is a 2B-parameter native AR
TTS model that emits 48 kHz mono audio and supports 30+ languages out of
the box. It runs as a single-stage pipeline (`MiniCPM4 base LM → FSQ →
MiniCPM4 residual LM → LocDiT CFM solver → AudioVAE`) and supports both
zero-shot synthesis and reference-audio voice cloning, with optional
gapless streaming via the bundled Gradio AudioWorklet demo.

Because the model is only 2B params, a single 24 GB consumer GPU is
sufficient — no tensor parallelism is required.

## References

- Example guide (online):
  [`examples/online_serving/text_to_speech/README.md`](../../examples/online_serving/text_to_speech/README.md#voxcpm2)
- Online client / Gradio demo:
  [`examples/online_serving/text_to_speech/voxcpm2/`](../../examples/online_serving/text_to_speech/voxcpm2/)
- Offline end-to-end script:
  [`examples/offline_inference/text_to_speech/voxcpm2/end2end.py`](../../examples/offline_inference/text_to_speech/voxcpm2/end2end.py)
- Default deploy config:
  [`vllm_omni/deploy/voxcpm2.yaml`](../../vllm_omni/deploy/voxcpm2.yaml)
- Talker / pipeline source:
  [`vllm_omni/model_executor/models/voxcpm2/`](../../vllm_omni/model_executor/models/voxcpm2/)
- Speech API reference:
  [`docs/serving/speech_api.md`](../../docs/serving/speech_api.md#voxcpm2)
- Related issue or discussion:
  [RFC: add recipes folder](https://github.com/vllm-project/vllm-omni/issues/2645)

## Hardware Support

This recipe documents one tested 24 GB consumer-GPU configuration.
Larger-VRAM (H20 / H100 / A100) and other vendor sections (ROCm, NPU) are
welcome as community validation lands.

## GPU

### 1 x RTX 4090 24GB (Single GPU, Minimum Recommended)

VoxCPM2 (2B params in bfloat16) fits comfortably on a single 24 GB GPU.
The bundled default config at
[`vllm_omni/deploy/voxcpm2.yaml`](../../vllm_omni/deploy/voxcpm2.yaml)
(`gpu_memory_utilization: 0.9`, `max_num_seqs: 4`, `enforce_eager: true`)
loads cleanly on a 4090 with ~4.9 GiB for model weights, ~15.2 GiB for
KV cache, and ~2 GiB for the talker's CUDA-Graph / CFM / VAE buffers,
for a total resident footprint of about **22 GiB / 24 GiB** in steady
state.

#### Environment

- OS: Linux (Ubuntu 22.04)
- Python: 3.12
- Driver / runtime: NVIDIA driver 575.57.08, CUDA 12.9 runtime
- torch: 2.11.0+cu128
- vLLM: 0.21.0
- vLLM-Omni: 0.20.1.dev (current `main`)
- Native VoxCPM2 source: `voxcpm` >= 2.0

#### Command

Start the server from the repository root. On a clean 4090 with no other
tenant on the GPU, the defaults from `voxcpm2.yaml` work as-is:

```bash
vllm serve openbmb/VoxCPM2 --omni \
    --host 0.0.0.0 --port 8000
```

The deploy config at
[`vllm_omni/deploy/voxcpm2.yaml`](../../vllm_omni/deploy/voxcpm2.yaml)
is loaded automatically by the model registry (HF `model_type=voxcpm2`).
Pass `--deploy-config <path>` to override.

#### Verification

**Server cold-start**: ~60 s from `vllm serve` to `Application startup
complete` (subprocess fork + vLLM 0.21 init + model load + flashinfer JIT
+ torch.compile of LocDiT / feat_encoder / AudioVAE + CUDA-Graph warmup).
The first request after startup pays a small additional cost; steady-state
requests are much faster.

**T1 — zero-shot synthesis (curl, WAV)**:

```bash
curl -X POST http://127.0.0.1:8000/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "model": "voxcpm2",
        "input": "Hello, this is VoxCPM2 served by vLLM-Omni on a single RTX 4090.",
        "voice": "default",
        "response_format": "wav"
    }' --output output.wav
```

Observed: `output.wav` 491 KB, 5.12 s @ 48 kHz mono, peak amplitude 0.996.
Server-to-client wall time: 2.6 s (server-side RTF ~0.5 after warmup).

**T2 — bundled Python client (zero-shot)**:

```bash
python examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py \
    --text "This audio was generated by the bundled openai_speech_client.py script against a vLLM-Omni server." \
    --api-base http://127.0.0.1:8000 \
    --output output.wav
```

Observed: `output.wav` 968 KB, 10.08 s @ 48 kHz mono, peak amplitude 0.949.

**T3 — voice cloning from a reference WAV**:

```bash
python examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py \
    --text "This should sound like the reference speaker, but saying a different sentence on a single RTX 4090." \
    --ref-audio /path/to/reference.wav \
    --api-base http://127.0.0.1:8000 \
    --output cloned.wav
```

Observed (using the T2 output `output.wav` as `--ref-audio`):
`cloned.wav` 568 KB, 5.92 s @ 48 kHz mono, peak amplitude 0.990;
end-to-end wall time 3.16 s.

`--ref-audio` accepts local file paths (auto-base64), HTTP/HTTPS URLs, or
`data:audio/wav;base64,...` data URIs. The `voice` field is required by
the OpenAI schema but ignored by VoxCPM2 unless it refers to an
uploaded voice — voice cloning is driven entirely by `ref_audio`.

**T4 — streaming PCM (raw, 48 kHz s16le)**:

```bash
curl -X POST http://127.0.0.1:8000/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{
        "model": "voxcpm2",
        "input": "Streaming PCM test on a single RTX 4090 with VoxCPM2.",
        "voice": "default",
        "stream": true,
        "response_format": "pcm"
    }' --no-buffer | play -t raw -r 48000 -e signed -b 16 -c 1 -
```

Observed: 230400 int16 samples = 4.80 s of PCM at 48 kHz, peak 32525 /
32767 (~99 % full scale). The player sample rate is **48 kHz**, not
24 kHz.

**T5 — gapless streaming (Gradio AudioWorklet demo)**:

```bash
python examples/online_serving/text_to_speech/voxcpm2/gradio_demo.py \
    --api-base http://127.0.0.1:8000
```

Observed: the demo binds to `http://0.0.0.0:7860` in ~4 s after launch.
The UI ("VoxCPM2 Streaming Demo") exposes three modes — **Voice Design**
(control instruction only), **Controllable Cloning** (ref audio +
optional style control), and **Ultimate Cloning** (ref audio +
transcript for audio continuation) — plus a `Stream (gapless)`
checkbox (default on) that drives the AudioWorklet streaming path. The
right pane includes runnable example prompts and a `Ready` status
indicator. End-to-end browser playback through `Generate Speech` was
verified working on the same vLLM-Omni server backing T1–T4 above.

**T6 — offline inference (no server, single process per request)**:

```bash
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
    --text "Hello, this is VoxCPM2 native AR synthesis running on vLLM-Omni and a single RTX 4090." \
    --output-dir output_audio
```

`end2end.py` is a one-shot script (init → one `generate()` → exit), so
its reported RTF includes per-process compile + CUDA-Graph capture +
first-request runtime warmup. Note that the default `voxcpm2.yaml` sets
`enforce_eager: true`, so vLLM's engine-level CUDA graphs are off by
design; the "CUDA-Graph capture" referenced throughout this recipe is the
talker's own `_CapturedGraph` over the CFM solver / AudioVAE, not the LM
decode loop. Flipping `enforce_eager=false` does **not** remove this cost.
To separate the warmup cost from the
talker's true per-step throughput, the table below reports **5
consecutive `engine.generate([prompt])` calls in the same process**
after a one-off ~28 s engine init:

| call | duration | inference | RTF   | notes                                            |
| ---- | -------- | --------- | ----- | ------------------------------------------------ |
| #1   | 6.72 s   | 11.97 s   | 1.782 | cold: torch.compile + CUDA-Graph capture         |
| #2   | 6.24 s   | 11.43 s   | 1.831 | still runtime warmup                             |
| #3   | 6.88 s   | 0.82 s    | 0.120 | ⚡ steady-state                                   |
| #4   | 6.56 s   | 0.78 s    | 0.119 | steady-state                                     |
| #5   | 5.76 s   | 0.70 s    | 0.121 | steady-state                                     |

**Steady-state RTF (mean of calls #3–#5): 0.120**, i.e. ~8× real-time on
a single 4090. The high call-#1 / call-#2 numbers are one-time warmup
costs that **amortise to zero after the second request** — they are not
the model's per-step throughput.

A literal one-shot `python end2end.py` invocation in a fresh process
only sees call #1 (~RTF 1.8). For repeated workloads, either reuse the
same engine across calls (as in the table above) or use the online
serving path (T1–T5); a long-lived server amortises the warmup across
all subsequent requests (T1 wall-time RTF was 0.51 with a warm server).

Voice cloning + continuation mode (`ref_audio` + transcript):

```bash
python examples/offline_inference/text_to_speech/voxcpm2/end2end.py \
    --text "Hello, this is the offline voice cloning test on a single RTX 4090." \
    --ref-audio /path/to/reference.wav \
    --ref-text "Original transcript of the reference audio."
```

Same 5-call methodology as the zero-shot table, this time with a
10.08 s reference WAV supplied via `--ref-audio` (and matching
`--ref-text`):

| call | duration | inference | RTF   | notes                                            |
| ---- | -------- | --------- | ----- | ------------------------------------------------ |
| #1   | 5.44 s   | 12.38 s   | 2.276 | cold: compile + CUDA-Graph capture + ref encode  |
| #2   | 5.12 s   | 2.47 s    | 0.482 | most warmup done                                 |
| #3   | 5.44 s   | 0.76 s    | 0.139 | ⚡ steady-state                                   |
| #4   | 4.96 s   | 0.68 s    | 0.137 | steady-state                                     |
| #5   | 5.28 s   | 0.71 s    | 0.134 | steady-state                                     |

**Steady-state cloning RTF (mean of calls #3–#5): 0.137**, i.e. only
**+0.017 RTF** above the zero-shot steady-state of 0.120. That delta is
the per-request `AudioVAE.encode` over the reference clip (~85 ms for a
10 s ref WAV) plus the ref-continuation prompt build; the AR decode
steps themselves run at the same per-step rate as zero-shot.

#### Notes

- Prerequisites: `pip install voxcpm soundfile httpx ninja`. `ninja` must be on `PATH` because `flashinfer` JIT-compiles a top-k sampling kernel during the first profile pass (without it, startup dies with `FileNotFoundError: 'ninja'`).
- Memory usage: ~4.9 GiB model weights + ~15.2 GiB KV cache ≈ 22 GiB / 24 GiB resident with the default `voxcpm2.yaml`. On a shared GPU, pass `--gpu-memory-utilization 0.75` (or lower) if startup fails the free-memory check.
- Key flags: `--omni` is required. `--trust-remote-code` is not needed — the HF config registers under `model_type=voxcpm2`.
- Output: 48 kHz mono. When streaming PCM to a player, use `-r 48000`.
- Cold start: ~60 s server boot + ~25 s first-inference overhead (torch.compile + flashinfer JIT + the talker's own `_CapturedGraph` capture over CFM / AudioVAE — not vLLM engine CUDA graphs, which `enforce_eager: true` disables). Steady-state inference RTF is **~0.12** (~8× real-time on a single 4090; see T6 table for the per-call breakdown); online server requests run at ~RTF 0.5 wall-time (includes HTTP round-trip). Keep the server warm for interactive use.