anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Doc][TTS] CosyVoice3 online docs + residual TTS yaml cleanup + remove VoxCPM v1 (#3748)

Browse Source 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
Yueqian Lin authored and GitHub committed 2026-05-20 15:24:06 +08:00
1 parent 1c530d984d
commit fc8486cf20
35 files changed
+248 -4147
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/offline_inference/text_to_speech/README.md
+2 -53
View File
@@ -14,13 +14,12 @@ list of supported architectures across all modalities, see
| Model | HuggingFace repo | Stages | Voice cloning | Streaming | Special modes | Sample rate |
|---|---|---|---|---|---|---|
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ | — | 22.05 kHz |
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ | — | 24 kHz |
| Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ | — | 44.1 kHz |
| Ming-flash-omni-TTS | `Jonathan1909/Ming-flash-omni-2.0` | single (talker only) | — (caption-controlled) | — | style / IP / basic captions | 44.1 kHz |
| MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | single (AR + codec) | ✓ (required) | ✓ | voice_clone, continuation | 48 kHz |
| OmniVoice | `k2-fsa/OmniVoice` | 2 (gen + dec) | ✓ | — | voice design, language hint | 24 kHz |
| Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | 2 (talker + code2wav) | ✓ (Base) | ✓ | 3 task variants | 24 kHz |
| VoxCPM | `openbmb/VoxCPM-0.5B` | split | ✓ | ✓ | — | 24 kHz |
| VoxCPM2 | `openbmb/VoxCPM2` | single (native AR) | ✓ | ✓ (online) | continuation | 48 kHz |
| Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | varies | ✓ | ✓ | voice presets | 24 kHz |
@@ -41,7 +40,7 @@ python examples/offline_inference/text_to_speech/<model>/end2end.py \
## CosyVoice3
2-stage TTS pipeline (`talker` + `code2wav`) at 22.05 kHz.
2-stage TTS pipeline (`talker` + `code2wav`) at 24 kHz.
### Prerequisites
```bash
@@ -308,56 +307,6 @@ python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \
---
## VoxCPM
Split-stage TTS. The hub example covers single TTS, single voice cloning, and streaming. Use `benchmarks/voxcpm/` for warmup, batch JSONL prompts, profiler injection, and offline TTFP / RTF measurement.
### Prerequisites
```bash
pip install voxcpm soundfile
# or use a local source tree:
export VLLM_OMNI_VOXCPM_CODE_PATH=/path/to/VoxCPM/src
```
### Quick start
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model openbmb/VoxCPM-0.5B \
--text "This is a split-stage VoxCPM synthesis example running on vLLM Omni."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model openbmb/VoxCPM-0.5B \
--text "This sentence is synthesized with a cloned voice." \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript spoken in reference.wav."
```
### Streaming
Pass `--streaming` together with the legacy async-chunk stage config:
```bash
python examples/offline_inference/text_to_speech/voxcpm/end2end.py \
--model openbmb/VoxCPM-0.5B \
--streaming \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml \
--text "This is a split-stage VoxCPM streaming example running on vLLM Omni."
```
### Persistent HF config (optional)
The engine auto-patches a missing `model_type` in the checkpoint's `config.json` in-memory at startup (see `vllm_omni/engine/arg_utils.py`), so no manual setup is required.
If you'd rather maintain a persistent patched config dir (e.g. to share across runs or to add fields the auto-patcher doesn't set), export `VLLM_OMNI_VOXCPM_HF_CONFIG_PATH` to its location — both the bundled deploy and async-chunk yamls read it via `${oc.env:VLLM_OMNI_VOXCPM_HF_CONFIG_PATH,}`.
### Notes
- Non-streaming auto-loads `vllm_omni/deploy/voxcpm.yaml` from the model directory name / HF `model_type`. Streaming uses the legacy `voxcpm_async_chunk.yaml` passed via `--stage-configs-path`.
- Streaming is currently single-request oriented.
- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades quality.
- For online serving, see the [VoxCPM section in the online hub](../../online_serving/text_to_speech/README.md#voxcpm). For benchmark reporting, see [`benchmarks/voxcpm`](../../../benchmarks/voxcpm/README.md).
---
## VoxCPM2
Single-stage native AR TTS at 48 kHz. Pipeline: `feat_encoder → MiniCPM4 → FSQ → residual_lm → LocDiT → AudioVAE`.
examples/offline_inference/text_to_speech/voxcpm/end2end.py
-220
View File
@@ -1,220 +0,0 @@
"""Minimal offline VoxCPM example for vLLM Omni."""
from __future__ import annotations
import asyncio
import time
from pathlib import Path
from typing import Any
import soundfile as sf
import torch
from vllm.utils.argparse_utils import FlexibleArgumentParser
from vllm_omni import AsyncOmni, Omni
from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
def _build_prompt(args) -> dict[str, Any]:
additional_information: dict[str, list[Any]] = {
"text": [args.text],
"cfg_value": [args.cfg_value],
"inference_timesteps": [args.inference_timesteps],
"min_len": [args.min_len],
"max_new_tokens": [args.max_new_tokens],
}
if args.streaming_prefix_len is not None:
additional_information["streaming_prefix_len"] = [args.streaming_prefix_len]
if args.ref_audio is not None:
additional_information["ref_audio"] = [args.ref_audio]
if args.ref_text is not None:
additional_information["ref_text"] = [args.ref_text]
return {
"prompt_token_ids": [1],
"additional_information": additional_information,
}
def _extract_audio_tensor(mm: dict[str, Any]) -> torch.Tensor:
audio = mm.get("audio", mm.get("model_outputs"))
if audio is None:
raise ValueError("No audio output found in multimodal output.")
if isinstance(audio, list):
parts = [torch.as_tensor(item).float().cpu().reshape(-1) for item in audio]
audio = torch.cat(parts, dim=-1) if parts else torch.zeros(0)
if not isinstance(audio, torch.Tensor):
audio = torch.as_tensor(audio)
return audio.float().cpu().reshape(-1)
def _extract_sample_rate(mm: dict[str, Any]) -> int:
sr_raw = mm.get("sr", 24000)
if isinstance(sr_raw, list) and sr_raw:
sr_raw = sr_raw[-1]
if hasattr(sr_raw, "item"):
return int(sr_raw.item())
return int(sr_raw)
def _save_audio(audio: torch.Tensor, sample_rate: int, output_dir: Path, request_id: str) -> Path:
output_dir.mkdir(parents=True, exist_ok=True)
output_path = output_dir / f"output_{request_id}.wav"
sf.write(
output_path,
audio.float().cpu().clamp(-1.0, 1.0).numpy(),
sample_rate,
format="WAV",
subtype="PCM_16",
)
return output_path
async def _run_streaming(args) -> Path:
prompt = _build_prompt(args)
output_dir = Path(args.output_dir) if args.output_dir is not None else Path("output_audio_streaming")
request_id = "streaming_example"
sample_rate = 24000
buffered_samples = 0
chunks: list[torch.Tensor] = []
started = time.perf_counter()
omni = AsyncOmni(
model=args.model,
stage_configs_path=args.stage_configs_path,
deploy_config=args.deploy_config,
log_stats=args.log_stats,
stage_init_timeout=args.stage_init_timeout,
)
try:
async for stage_output in omni.generate(prompt, request_id=request_id):
mm = getattr(stage_output, "multimodal_output", None)
if not isinstance(mm, dict):
request_output = getattr(stage_output, "request_output", None)
if request_output is None:
continue
mm = getattr(request_output, "multimodal_output", None)
if not isinstance(mm, dict) and getattr(request_output, "outputs", None):
mm = getattr(request_output.outputs[0], "multimodal_output", None)
if not isinstance(mm, dict):
continue
audio = _extract_audio_tensor(mm)
if audio.numel() == 0:
continue
sample_rate = _extract_sample_rate(mm)
if audio.numel() > buffered_samples:
delta = audio[buffered_samples:]
buffered_samples = int(audio.numel())
else:
delta = audio
buffered_samples += int(delta.numel())
if delta.numel() > 0:
chunks.append(delta)
if not chunks:
raise RuntimeError("No streaming audio chunks received from VoxCPM.")
output_audio = torch.cat(chunks, dim=0)
output_path = _save_audio(output_audio, sample_rate, output_dir, request_id)
print(f"Saved streaming audio to: {output_path} ({time.perf_counter() - started:.2f}s)")
return output_path
finally:
omni.shutdown()
def _run_sync(args) -> Path:
prompt = _build_prompt(args)
output_dir = Path(args.output_dir) if args.output_dir is not None else Path("output_audio")
request_id = "sync_example"
started = time.perf_counter()
last_mm: dict[str, Any] | None = None
omni = Omni(
model=args.model,
stage_configs_path=args.stage_configs_path,
deploy_config=args.deploy_config,
log_stats=args.log_stats,
stage_init_timeout=args.stage_init_timeout,
)
for stage_outputs in omni.generate(prompt):
request_output = getattr(stage_outputs, "request_output", None)
if request_output is None:
continue
outputs = getattr(request_output, "outputs", None)
if outputs:
for output in outputs:
mm = getattr(output, "multimodal_output", None)
if isinstance(mm, dict):
last_mm = mm
mm = getattr(request_output, "multimodal_output", None)
if isinstance(mm, dict):
last_mm = mm
if last_mm is None:
raise RuntimeError("No audio output received from VoxCPM.")
output_path = _save_audio(
_extract_audio_tensor(last_mm),
_extract_sample_rate(last_mm),
output_dir,
request_id,
)
print(f"Saved audio to: {output_path} ({time.perf_counter() - started:.2f}s)")
return output_path
def parse_args():
parser = FlexibleArgumentParser(description="Minimal offline VoxCPM example for vLLM Omni.")
parser.add_argument("--model", type=str, required=True, help="Local VoxCPM model directory.")
parser.add_argument(
"--deploy-config",
type=str,
default=None,
help=("Override the deploy config path."),
)
parser.add_argument(
"--stage-configs-path",
type=str,
default=None,
help=(
"Legacy stage_args yaml path. Required for streaming "
"(vllm_omni/model_executor/stage_configs/voxcpm_async_chunk.yaml); "
"leave unset for non-streaming to use the auto-resolved deploy config."
),
)
parser.add_argument(
"--streaming",
action="store_true",
default=False,
help="Stream audio chunks as they arrive via AsyncOmni.",
)
parser.add_argument("--text", type=str, required=True, help="Input text for synthesis.")
parser.add_argument("--ref-audio", type=str, default=None, help="Reference audio path for voice cloning.")
parser.add_argument("--ref-text", type=str, default=None, help="Transcript of the reference audio.")
parser.add_argument("--output-dir", type=str, default=None, help="Output directory for generated wav files.")
parser.add_argument("--cfg-value", type=float, default=2.0, help="Guidance value passed to VoxCPM.")
parser.add_argument("--inference-timesteps", type=int, default=10, help="Number of diffusion timesteps.")
parser.add_argument("--min-len", type=int, default=2, help="Minimum latent length.")
parser.add_argument("--max-new-tokens", type=int, default=4096, help="Maximum latent length.")
parser.add_argument(
"--streaming-prefix-len",
type=int,
default=3,
help="Streaming prefix length used by voxcpm_async_chunk.yaml.",
)
parser.add_argument("--stage-init-timeout", type=int, default=600, help="Stage initialization timeout in seconds.")
parser.add_argument("--log-stats", action="store_true", help="Enable vLLM Omni stats logging.")
nullify_stage_engine_defaults(parser)
args = parser.parse_args()
if (args.ref_audio is None) != (args.ref_text is None):
raise ValueError("Voice cloning requires --ref-audio and --ref-text together.")
return args
def main(args) -> None:
route = "streaming" if args.streaming else "sync"
print(f"Model: {args.model}")
print(f"Deploy config: {args.deploy_config or '<auto from HF model_type>'}")
print(f"Stage configs path: {args.stage_configs_path or '<unused>'}")
print(f"Route: {route}")
if route == "streaming":
asyncio.run(_run_streaming(args))
else:
_run_sync(args)
if __name__ == "__main__":
main(parse_args())