Add voxcpm model support. (#2467)

Signed-off-by: Celeste-jq <591998922@qq.com> Signed-off-by: lyj-jjj <liuyingjun5@huawei.com> Signed-off-by: IsleOfDawnlight <stellamou@qq.com> Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: Celeste-jq <591998922@qq.com> Co-authored-by: lyj-jjj <liuyingjun5@huawei.com> Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
2026-04-15 15:04:58 +08:00
1 parent 38d5f2d530
commit 4bf4c63147
@@ -0,0 +1,166 @@
+# VoxCPM
+
+## Prerequisites
+
+Install VoxCPM in one of these ways:
+
+```bash
+pip install voxcpm
+```
+
+or point vLLM-Omni to a local VoxCPM source tree:
+
+```bash
+export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src
+```
+
+If the native VoxCPM `config.json` lacks HF metadata such as `model_type`,
+prepare a persistent HF-compatible config directory and export:
+
+```bash
+export VLLM_OMNI_VOXCPM_HF_CONFIG_PATH=/tmp/voxcpm_hf_config
+mkdir -p "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"
+cp "$VOXCPM_MODEL/config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/config.json"
+cp "$VOXCPM_MODEL/generation_config.json" "$VLLM_OMNI_VOXCPM_HF_CONFIG_PATH/generation_config.json" 2>/dev/null || true
+python3 -c 'import json, os; p=os.path.join(os.environ["VLLM_OMNI_VOXCPM_HF_CONFIG_PATH"], "config.json"); cfg=json.load(open(p, "r", encoding="utf-8")); cfg["model_type"]="voxcpm"; cfg.setdefault("architectures", ["VoxCPMForConditionalGeneration"]); json.dump(cfg, open(p, "w", encoding="utf-8"), indent=2, ensure_ascii=False)'
+```
+
+The VoxCPM stage configs read `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH` directly. The `python3 -c` form above avoids heredoc/indentation issues in interactive shells.
+
+## Launch the Server
+
+Use the async-chunk stage config by default:
+
+```bash
+export VOXCPM_MODEL=/path/to/voxcpm-model
+cd examples/online_serving/voxcpm
+./run_server.sh
+```
+
+Use the non-streaming stage config:
+
+```bash
+./run_server.sh sync
+```
+
+You can also launch the server directly:
+
+```bash
+vllm serve "$VOXCPM_MODEL" \
+    --stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
+    --trust-remote-code \
+    --enforce-eager \
+    --omni \
+    --port 8091
+```
+
+## Send Requests
+
+### Basic text-to-speech
+
+```bash
+python openai_speech_client.py \
+    --model "$VOXCPM_MODEL" \
+    --text "This is a VoxCPM online text-to-speech example."
+```
+
+### Voice cloning
+
+```bash
+python openai_speech_client.py \
+    --model "$VOXCPM_MODEL" \
+    --text "This sentence is synthesized with a cloned voice." \
+    --ref-audio /path/to/reference.wav \
+    --ref-text "The exact transcript spoken in reference.wav."
+```
+
+`ref_text` must be the real transcript of the reference audio. Placeholder text or mismatched text will usually degrade quality badly.
+
+### Streaming PCM output
+
+```bash
+python openai_speech_client.py \
+    --model "$VOXCPM_MODEL" \
+    --text "This is a streaming VoxCPM request." \
+    --stream \
+    --output voxcpm_stream.pcm
+```
+
+### Using curl
+
+```bash
+curl -X POST http://localhost:8091/v1/audio/speech \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "OpenBMB/VoxCPM1.5",
+        "input": "Hello from VoxCPM online serving.",
+        "response_format": "wav"
+    }' --output output.wav
+```
+
+Voice cloning:
+
+```bash
+curl -X POST http://localhost:8091/v1/audio/speech \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "OpenBMB/VoxCPM1.5",
+        "input": "This sentence uses a cloned voice.",
+        "ref_audio": "https://example.com/reference.wav",
+        "ref_text": "The exact transcript spoken in the reference audio.",
+        "response_format": "wav"
+    }' --output cloned.wav
+```
+
+Streaming PCM:
+
+```bash
+curl -X POST http://localhost:8091/v1/audio/speech \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "OpenBMB/VoxCPM1.5",
+        "input": "This is a streaming VoxCPM request.",
+        "stream": true,
+        "response_format": "pcm"
+    }' --output output.pcm
+```
+
+## Supported Request Shape
+
+VoxCPM online serving currently supports:
+
+- plain text-to-speech
+- voice cloning with `ref_audio` + `ref_text`
+- `stream=true` with `response_format=pcm` or `wav`
+
+VoxCPM online serving does not use these generic TTS fields:
+
+- `voice`
+- `instructions`
+- `language`
+- `speaker_embedding`
+- `x_vector_only_mode`
+
+## Streaming vs Non-Streaming
+
+- `voxcpm_async_chunk.yaml` enables async-chunk streaming and is best for single-request streaming latency.
+- `voxcpm.yaml` performs one-shot latent generation then VAE decode.
+
+Like native VoxCPM, the async streaming path should be treated as single-request. If you need stable throughput benchmarking, prefer `voxcpm.yaml`.
+
+Do not use `voxcpm_async_chunk.yaml` for concurrent online streaming or `/v1/audio/speech/batch`. For multiple requests, prefer `voxcpm.yaml`.
+
+## Benchmark
+
+The serving benchmark reports TTFP and RTF:
+
+```bash
+python benchmarks/voxcpm/vllm_omni/bench_tts_serve.py \
+    --host 127.0.0.1 \
+    --port 8091 \
+    --num-prompts 10 \
+    --max-concurrency 1 \
+    --result-dir /tmp/voxcpm_bench
+```
+
+For the async-chunk server, keep `--max-concurrency 1`.
@@ -0,0 +1,155 @@
+"""OpenAI-compatible client for VoxCPM via /v1/audio/speech.
+
+Examples:
+    # Basic text-to-speech
+    python openai_speech_client.py --text "Hello from VoxCPM"
+
+    # Voice cloning
+    python openai_speech_client.py \
+        --text "This sentence uses the cloned voice." \
+        --ref-audio /path/to/reference.wav \
+        --ref-text "The exact transcript spoken in the reference audio."
+
+    # Streaming PCM output
+    python openai_speech_client.py \
+        --text "This is a streaming VoxCPM request." \
+        --stream \
+        --output output.pcm
+"""
+
+import argparse
+import base64
+import os
+
+import httpx
+
+DEFAULT_API_BASE = "http://localhost:8091"
+DEFAULT_API_KEY = "EMPTY"
+DEFAULT_MODEL = "OpenBMB/VoxCPM1.5"
+
+
+def encode_audio_to_base64(audio_path: str) -> str:
+    """Encode a local audio file to base64 data URL."""
+    if not os.path.exists(audio_path):
+        raise FileNotFoundError(f"Audio file not found: {audio_path}")
+
+    ext = audio_path.lower().rsplit(".", 1)[-1]
+    mime_map = {
+        "wav": "audio/wav",
+        "mp3": "audio/mpeg",
+        "flac": "audio/flac",
+        "ogg": "audio/ogg",
+    }
+    mime_type = mime_map.get(ext, "audio/wav")
+
+    with open(audio_path, "rb") as f:
+        audio_b64 = base64.b64encode(f.read()).decode("utf-8")
+    return f"data:{mime_type};base64,{audio_b64}"
+
+
+def build_payload(args) -> dict[str, object]:
+    payload: dict[str, object] = {
+        "model": args.model,
+        "input": args.text,
+        "response_format": "pcm" if args.stream else args.response_format,
+    }
+
+    if args.ref_audio:
+        if args.ref_audio.startswith(("http://", "https://", "data:")):
+            payload["ref_audio"] = args.ref_audio
+        else:
+            payload["ref_audio"] = encode_audio_to_base64(args.ref_audio)
+    if args.ref_text:
+        payload["ref_text"] = args.ref_text
+    if args.max_new_tokens is not None:
+        payload["max_new_tokens"] = args.max_new_tokens
+    if args.stream:
+        payload["stream"] = True
+
+    return payload
+
+
+def run_tts(args) -> None:
+    payload = build_payload(args)
+    api_url = f"{args.api_base}/v1/audio/speech"
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": f"Bearer {args.api_key}",
+    }
+
+    print(f"Model: {args.model}")
+    print(f"Text: {args.text}")
+    if args.ref_audio:
+        print("Mode: voice cloning")
+        print(f"Reference audio: {args.ref_audio}")
+    else:
+        print("Mode: text-to-speech")
+
+    if args.stream:
+        output_path = args.output or "voxcpm_output.pcm"
+        with httpx.Client(timeout=300.0) as client:
+            with client.stream("POST", api_url, json=payload, headers=headers) as response:
+                if response.status_code != 200:
+                    print(f"Error: {response.status_code}")
+                    print(response.read().decode("utf-8", errors="ignore"))
+                    return
+
+                total_bytes = 0
+                with open(output_path, "wb") as f:
+                    for chunk in response.iter_bytes():
+                        if not chunk:
+                            continue
+                        f.write(chunk)
+                        total_bytes += len(chunk)
+        print(f"Streamed {total_bytes} bytes to: {output_path}")
+        return
+
+    with httpx.Client(timeout=300.0) as client:
+        response = client.post(api_url, json=payload, headers=headers)
+
+    if response.status_code != 200:
+        print(f"Error: {response.status_code}")
+        print(response.text)
+        return
+
+    try:
+        text = response.content.decode("utf-8")
+        if text.startswith('{"error"'):
+            print(f"Error: {text}")
+            return
+    except UnicodeDecodeError:
+        pass
+
+    output_path = args.output or "voxcpm_output.wav"
+    with open(output_path, "wb") as f:
+        f.write(response.content)
+    print(f"Audio saved to: {output_path}")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="VoxCPM OpenAI-compatible speech client")
+    parser.add_argument("--api-base", default=DEFAULT_API_BASE, help="API base URL")
+    parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="API key")
+    parser.add_argument("--model", "-m", default=DEFAULT_MODEL, help="Model name or path")
+    parser.add_argument("--text", required=True, help="Text to synthesize")
+    parser.add_argument("--ref-audio", default=None, help="Reference audio path, URL, or data URL")
+    parser.add_argument(
+        "--ref-text",
+        default=None,
+        help="The exact transcript spoken in the reference audio",
+    )
+    parser.add_argument("--stream", action="store_true", help="Enable streaming PCM output")
+    parser.add_argument(
+        "--response-format",
+        default="wav",
+        choices=["wav", "pcm", "flac", "mp3", "aac", "opus"],
+        help="Audio format for non-streaming mode (default: wav)",
+    )
+    parser.add_argument("--max-new-tokens", type=int, default=None, help="Maximum tokens to generate")
+    parser.add_argument("--output", "-o", default=None, help="Output file path")
+    args = parser.parse_args()
+    run_tts(args)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,38 @@
+#!/bin/bash
+# Launch vLLM-Omni server for VoxCPM online speech serving.
+#
+# Usage:
+#   ./run_server.sh                 # default: async_chunk stage config
+#   ./run_server.sh async           # async_chunk stage config
+#   ./run_server.sh sync            # no-async-chunk stage config
+#   VOXCPM_MODEL=/path/to/model ./run_server.sh
+
+set -e
+
+MODE="${1:-async}"
+MODEL="${VOXCPM_MODEL:-OpenBMB/VoxCPM1.5}"
+
+case "$MODE" in
+    async)
+        STAGE_CONFIG="vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml"
+        ;;
+    sync)
+        STAGE_CONFIG="vllm_omni/model_executor/stage_configs/voxcpm.yaml"
+        ;;
+    *)
+        echo "Unknown mode: $MODE"
+        echo "Supported: async, sync"
+        exit 1
+        ;;
+esac
+
+echo "Starting VoxCPM server with model: $MODEL"
+echo "Stage config: $STAGE_CONFIG"
+
+vllm serve "$MODEL" \
+    --stage-configs-path "$STAGE_CONFIG" \
+    --host 0.0.0.0 \
+    --port 8091 \
+    --trust-remote-code \
+    --enforce-eager \
+    --omni