anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[CI]Remove SoulX-Singer support (#6362)

Browse Source 
Signed-off-by: wangyu <410167048@qq.com>
This commit is contained in:
wangyu authored and GitHub committed 2026-08-23 07:16:32 +08:00
1 parent b2c9376539
commit 2e096788a6
51 files changed
+1 -9575
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

examples/offline_inference/text_to_speech/README.md
+1 -99
View File
@@ -420,8 +420,7 @@ against transformers 5.x, so nothing is lost by holding it there.
(The same clash is why this is documented here rather than declared in
`pyproject.toml`: an extra resolves together with the base dependencies, so
`vllm-omni[gepard]` would be unsatisfiable rather than opt-in. SoulX-Singer English
SVS documents `nemo_toolkit[asr]` the same way.)
`vllm-omni[gepard]` would be unsatisfiable rather than opt-in.)
On a host whose CUDA toolkit cannot build kernels for the GPU — no `nvcc`/`ninja`, or a
consumer Blackwell (`sm_120`) card — also set `VLLM_USE_FLASHINFER_SAMPLER=0`. vLLM's
@@ -617,100 +616,3 @@ Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-T
- `--num-prompts N` replicates the prompt for performance measurement.
- `--concurrency M` requires `--streaming` and must evenly divide `--num-prompts`.
- Run `--help` for the full argument surface.
---
## SoulX-Singer
Singing voice synthesis (SVS) and conversion (SVC) at 24 kHz. Script: `soulxsinger/end2end.py`. Deploy: `vllm_omni/deploy/soulxsinger_svs.yaml` or `soulxsinger_svc.yaml`.
### Prerequisites
Download DiT and preprocess weights, then set up separate SVS / SVC view directories. Copy `soulxsinger/utils/phoneme/phone_set.json` from upstream [SoulX-Singer](https://github.com/Soul-AILab/SoulX-Singer) into the model weights dir as `phoneme/phone_set.json` — HuggingFace does not ship it.
```bash
# 1. DiT weights
export BASE=path/to/SoulX-Singer
export PREPROCESS=path/to/SoulX-Singer-Preprocess
export SVC_DIR=path/to/SoulX-Singer-svc
huggingface-cli download Soul-AILab/SoulX-Singer --local-dir "$BASE"
# 2. Preprocess weights (required)
huggingface-cli download Soul-AILab/SoulX-Singer-Preprocess --local-dir "$PREPROCESS"
export SOULX_PREPROCESS_WEIGHTS_DIR="$PREPROCESS"
# 3. SVS / SVC view directories
mkdir -p "$SVC_DIR"
cp $BASE/{config.yaml,README.md,assets} $SVC_DIR
mv $BASE/model-svc.pt $SVC_DIR/model-svc.pt
cat > "$BASE/config.json" <<'EOF'
{
"model_type": "soulxsinger",
"architectures": ["SoulXSingerPipeline"],
"max_num_seqs": 1
}
EOF
cat > "$SVC_DIR/config.json" <<'EOF'
{
"model_type": "soulxsinger",
"architectures": ["SoulXSingerSVCPipeline"],
"max_num_seqs": 1
}
EOF
```
`config.yaml` hyper-parameters live under `$BASE`; each view's `config.json` `architectures` field is the single source of truth for SVS vs SVC. Point `--model` at the matching directory (`$BASE` for SVS, `$SVC_DIR` for SVC). Deploy YAML is chosen automatically from `config.json`; optional `--svs` / `--svc` only assert the mode matches.
**Online preprocess** is the default: pass `--prompt-audio` and `--target-audio`, and the worker runs vocal separation, F0, and (for SVS) lyrics/MIDI before DiT. Install only what your run needs:
```bash
pip install "BS-RoFormer" # vocal sep + F0 on GPU — SVS and SVC
```
Mandarin SVS also needs FunASR and Chinese G2P; `ffmpeg` must be on `PATH`:
```bash
# install optional dependencies:
pip install -e ".[soulx-svs]"
```
English SVS adds NeMo ASR and NLTK data; pass `--language English`:
```bash
pip install "nemo_toolkit[asr]==2.6.1" lhotse==1.32.2
python -c "import nltk; nltk.download('cmudict'); nltk.download('averaged_perceptron_tagger_eng')"
```
**Precomputed metadata** is the alternative: pass both `--prompt-metadata-path` and `--target-metadata-path` and skip online ASR/ROSVOT — none of the packages above are required. JSON can be produced by integrated preprocess on a prior run, or by upstream [SoulX-Singer](https://github.com/Soul-AILab/SoulX-Singer) `preprocess/` scripts if you prefer to run that outside vLLM-Omni.
### Quick start
```bash
# SVS — default demo audio: tests/assets/soulxsinger/zh_prompt.mp3 + music.mp3
python examples/offline_inference/text_to_speech/soulxsinger/end2end.py \
--model "$BASE" \
--preprocess-weights-dir "$PREPROCESS" \
--control score \
--num-inference-steps 32 \
-o output.wav
python examples/offline_inference/text_to_speech/soulxsinger/end2end.py \
--model "$SVC_DIR" \
--preprocess-weights-dir "$PREPROCESS" \
--svc \
--num-inference-steps 32 \
-o output_svc.wav
```
`SOULX_PREPROCESS_WEIGHTS_DIR` makes `--preprocess-weights-dir` optional. Long SVS targets are handled in one request. See `end2end.py --help` for `--pitch-shift`, `--vocal-sep`, `--auto-shift`, and language/control options.
### Notes
- Output: 24 kHz mono WAV; batch only.
- Defaults match upstream: `--guidance-scale 3.0`, `--seed 42`, `--auto-shift` on.
- SVS `--control`: `score` or `melody`. MIDI / lyric QC: upstream `midi_editor` only.
---
examples/offline_inference/text_to_speech/soulxsinger/benchmark.py
-162
View File
@@ -1,162 +0,0 @@
"""Repeatable SoulX-Singer offline benchmark (preprocess + DiT, RTF + stage timings).
Usage:
python benchmark.py --model /path/to/SoulX-Singer --svs \\
--prompt-audio ../SoulX-Singer/example/audio/zh_prompt.mp3 \\
--target-audio ../SoulX-Singer/example/audio/music.mp3 \\
--preprocess-weights-dir /path/to/SoulX-Singer-Preprocess \\
-o benchmark.wav
python benchmark.py ... --enable-diffusion-pipeline-profiler
"""
import argparse
import statistics
import time
from pathlib import Path
import soundfile as sf
from end2end import (
SVC_DEPLOY_CONFIG,
SVS_DEPLOY_CONFIG,
add_inference_args,
build_sampling,
extract_audio,
resolve_preprocess_weights_dir,
)
from vllm_omni.entrypoints.omni import Omni
def _audio_duration_sec(outputs) -> float:
audio_np, sr = extract_audio(outputs)
return audio_np.size / sr
def _stage_durations_from_output(outputs) -> dict[str, float]:
omni_out = outputs[0]
durations = getattr(omni_out, "stage_durations", None) or {}
if durations:
return dict(durations)
ro = omni_out
if ro is not None:
return dict(getattr(ro, "stage_durations", None) or {})
return {}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="SoulX-Singer offline benchmark")
add_inference_args(parser)
parser.add_argument("--warmup", type=int, default=1, help="Warmup runs (excluded from stats)")
parser.add_argument("--runs", type=int, default=3, help="Measured runs")
parser.add_argument(
"--enable-diffusion-pipeline-profiler",
action="store_true",
help="Enable low-overhead stage timing in DiffusionOutput.stage_durations",
)
parser.add_argument(
"--enforce-eager",
action="store_true",
help="Disable torch.compile on diff_estimator (eager baseline for A/B)",
)
parser.add_argument(
"--dtype",
type=str,
default=None,
choices=["bfloat16", "bf16", "float16", "fp16", "half"],
help="DiT trunk dtype (default from deploy YAML). Use float16 to match upstream --fp16.",
)
parser.add_argument(
"--output",
"-o",
type=str,
default=None,
help="Save WAV from the last measured run (after warmup)",
)
parser.add_argument(
"--save-each-run",
action="store_true",
help="With --output, also write measured runs as <stem>_run1.wav, _run2.wav, ...",
)
return parser.parse_args()
def main() -> None:
args = parse_args()
preprocess_dir = resolve_preprocess_weights_dir(args.preprocess_weights_dir)
deploy_config = args.deploy_config or str(SVS_DEPLOY_CONFIG if args.svs else SVC_DEPLOY_CONFIG)
mode = "SVS" if args.svs else "SVC"
omni_kwargs: dict = {
"model": args.model,
"deploy_config": deploy_config,
"async_chunk": False, # SoulX-Singer currently supports only batch mode (pseudo-streaming was stashed)
}
if args.enable_diffusion_pipeline_profiler:
omni_kwargs["enable_diffusion_pipeline_profiler"] = True
if args.enforce_eager:
omni_kwargs["enforce_eager"] = True
if args.dtype is not None:
omni_kwargs["dtype"] = args.dtype
compile_mode = "eager" if args.enforce_eager else "torch.compile"
dtype_label = args.dtype or "from deploy YAML"
print(f"Loading SoulX-Singer {mode} from {args.model} [{compile_mode}, dtype={dtype_label}]")
omni = Omni(**omni_kwargs)
kind = "svs" if args.svs else "svc"
sampling = build_sampling(args, preprocess_weights_dir=preprocess_dir, kind=kind)
prompt = {"prompt_token_ids": [0]}
latencies_ms: list[float] = []
rtfs: list[float] = []
last_stages: dict[str, float] = {}
last_measured_outputs = None
total_runs = args.warmup + args.runs
for run_idx in range(total_runs):
is_warmup = run_idx < args.warmup
label = "warmup" if is_warmup else f"run {run_idx - args.warmup + 1}/{args.runs}"
t0 = time.perf_counter()
outputs = list(omni.generate([prompt], sampling))
elapsed_ms = (time.perf_counter() - t0) * 1000.0
audio_sec = _audio_duration_sec(outputs)
rtf = (elapsed_ms / 1000.0) / audio_sec if audio_sec > 0 else float("inf")
print(f"[{label}] client={elapsed_ms:.1f} ms, audio={audio_sec:.2f}s, RTF={rtf:.3f}")
if not is_warmup:
latencies_ms.append(elapsed_ms)
rtfs.append(rtf)
last_stages = _stage_durations_from_output(outputs)
last_measured_outputs = outputs
if args.output and args.save_each_run:
measured_idx = run_idx - args.warmup + 1
out_path = Path(args.output)
run_path = out_path.with_name(f"{out_path.stem}_run{measured_idx}{out_path.suffix}")
audio_np, sr = extract_audio(outputs)
sf.write(str(run_path), audio_np, sr)
print(f" saved {run_path} ({sr} Hz, {audio_np.size / sr:.2f}s)")
omni.close()
if args.output and last_measured_outputs is not None:
audio_np, sr = extract_audio(last_measured_outputs)
out_path = Path(args.output)
out_path.parent.mkdir(parents=True, exist_ok=True)
sf.write(str(out_path), audio_np, sr)
print(f"\nSaved last measured run → {out_path} ({sr} Hz, {audio_np.size / sr:.2f}s)")
if latencies_ms:
print("\n=== Summary (measured runs) ===")
print(f"client_ms: mean={statistics.mean(latencies_ms):.1f}, stdev={statistics.pstdev(latencies_ms):.1f}")
print(f"RTF: mean={statistics.mean(rtfs):.3f}, stdev={statistics.pstdev(rtfs):.3f}")
if last_stages:
print("\nStage durations (last measured run):")
for name, value in sorted(last_stages.items()):
if name.endswith("_ms"):
print(f" {name}: {value:.1f} ms")
else:
print(f" {name}: {value * 1000.0:.1f} ms")
if __name__ == "__main__":
main()
examples/offline_inference/text_to_speech/soulxsinger/end2end.py
-285
View File
@@ -1,285 +0,0 @@
"""Offline SoulX-Singer SVS / SVC: single-stage DiT (preprocess inline).
Pass raw prompt/target audio; vLLM-Omni runs integrated preprocess then inference.
Usage:
# SVS (default)
python end2end.py --model /path/to/SoulX-Singer \\
--prompt-audio path/to/prompt/audio.mp3 \\
--target-audio path/to/target/audio.mp3 \\
--preprocess-weights-dir /path/to/SoulX-Singer-Preprocess \\
-o output.wav
# SVC (config.json must declare SoulXSingerSVCPipeline; --model points at SVC view)
python end2end.py --model /path/to/SoulX-Singer-svc \\
--prompt-audio path/to/prompt/audio.mp3 \\
--target-audio path/to/target/audio.mp3 \\
--preprocess-weights-dir /path/to/SoulX-Singer-Preprocess \\
-o output_svc.wav
"""
import argparse
import os
from pathlib import Path
import numpy as np
import soundfile as sf
from vllm_omni.diffusion.models.soulx_singer.utils import resolve_soulx_kind, validate_soulx_extra_args
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
REPO_ROOT = Path(__file__).resolve().parents[4]
SVS_DEPLOY_CONFIG = REPO_ROOT / "vllm_omni" / "deploy" / "soulxsinger_svs.yaml"
SVC_DEPLOY_CONFIG = REPO_ROOT / "vllm_omni" / "deploy" / "soulxsinger_svc.yaml"
DEFAULT_PROMPT_AUDIO = REPO_ROOT / "tests" / "assets" / "soulxsinger" / "zh_prompt.mp3"
DEFAULT_TARGET_AUDIO = REPO_ROOT / "tests" / "assets" / "soulxsinger" / "music.mp3"
_SAMPLE_RATE = 24000
def _require_paths(paths: dict[str, str | None]) -> None:
missing = [name for name, p in paths.items() if not p or not os.path.isfile(p)]
if missing:
raise FileNotFoundError("Missing file(s): " + ", ".join(f"{k}={paths[k]!r}" for k in missing))
def resolve_preprocess_weights_dir(explicit: str | None) -> str:
if explicit:
path = Path(explicit).expanduser().resolve()
if not (path / "rmvpe" / "rmvpe.pt").is_file():
raise FileNotFoundError(f"--preprocess-weights-dir must contain rmvpe/rmvpe.pt: {path}")
return str(path)
env = os.environ.get("SOULX_PREPROCESS_WEIGHTS_DIR")
if env:
path = Path(env).expanduser().resolve()
if (path / "rmvpe" / "rmvpe.pt").is_file():
return str(path)
raise FileNotFoundError(
"SoulX preprocess weights not found. Pass --preprocess-weights-dir or set "
"SOULX_PREPROCESS_WEIGHTS_DIR to a directory containing rmvpe/rmvpe.pt "
"(huggingface-cli download Soul-AILab/SoulX-Singer-Preprocess)."
)
def build_sampling(
args: argparse.Namespace,
*,
preprocess_weights_dir: str,
kind: str,
) -> OmniDiffusionSamplingParams:
extra: dict = {
"prompt_audio": os.path.abspath(args.prompt_audio),
"target_audio": os.path.abspath(args.target_audio),
"preprocess_weights_dir": preprocess_weights_dir,
"vocal_sep": args.vocal_sep,
"auto_shift": args.auto_shift,
"pitch_shift": args.pitch_shift,
}
if kind == "svs":
extra["language"] = args.language
extra["control"] = args.control
if args.prompt_metadata_path:
extra["prompt_metadata_path"] = os.path.abspath(args.prompt_metadata_path)
if args.target_metadata_path:
extra["target_metadata_path"] = os.path.abspath(args.target_metadata_path)
if args.prompt_metadata_path or args.target_metadata_path:
extra["audio_path"] = os.path.abspath(args.prompt_wav_path or args.prompt_audio)
extra = validate_soulx_extra_args(kind, extra)
return OmniDiffusionSamplingParams(
num_inference_steps=args.num_inference_steps,
guidance_scale=args.guidance_scale,
seed=args.seed,
extra_args=extra,
)
def extract_audio(outputs) -> tuple[np.ndarray, int]:
out = outputs[0]
mm = getattr(out, "multimodal_output", None)
if mm is None:
ro = out
if ro is not None:
mm = getattr(ro, "multimodal_output", None)
if mm is None and ro.outputs:
mm = getattr(ro.outputs[0], "multimodal_output", None)
if not mm or "audio" not in mm:
raise RuntimeError("No audio in multimodal_output")
audio_val = mm["audio"]
if isinstance(audio_val, list):
chunks = [
np.asarray(chunk.detach().cpu().float().numpy() if hasattr(chunk, "detach") else chunk).reshape(-1)
for chunk in audio_val
if chunk is not None
]
audio_np = np.concatenate(chunks, axis=0) if chunks else np.array([], dtype=np.float32)
elif hasattr(audio_val, "cpu"):
audio_np = audio_val.detach().cpu().numpy().squeeze()
else:
audio_np = np.asarray(audio_val, dtype=np.float32).squeeze()
sr_val = mm.get("sr", mm.get("audio_sample_rate", _SAMPLE_RATE))
if isinstance(sr_val, list) and sr_val:
sr_val = sr_val[-1]
if hasattr(sr_val, "item"):
sr_val = sr_val.item()
return audio_np.astype(np.float32, copy=False), int(sr_val)
def add_inference_args(parser: argparse.ArgumentParser) -> None:
mode = parser.add_mutually_exclusive_group()
mode.add_argument(
"--svc",
action="store_true",
help="Assert SVC mode (must match config.json architectures)",
)
parser.add_argument(
"--model",
type=str,
required=True,
help="SoulX-Singer weight directory ($BASE for SVS, $SVC_DIR for SVC)",
)
parser.add_argument(
"--deploy-config",
type=str,
default=None,
help="Override deploy YAML (default: soulxsinger_svs.yaml or soulxsinger_svc.yaml)",
)
parser.add_argument(
"--prompt-audio",
type=str,
default=str(DEFAULT_PROMPT_AUDIO),
help="Prompt / reference audio (mp3 or wav)",
)
parser.add_argument(
"--target-audio",
type=str,
default=str(DEFAULT_TARGET_AUDIO),
help="Target content audio (mp3 or wav)",
)
parser.add_argument(
"--prompt-metadata-path",
type=str,
default=None,
help="SVS precomputed prompt metadata.json (skips online ASR/ROSVOT for prompt)",
)
parser.add_argument(
"--target-metadata-path",
type=str,
default=None,
help="SVS precomputed target metadata.json (skips online ASR/ROSVOT for target)",
)
parser.add_argument(
"--prompt-wav-path",
type=str,
default=None,
help="SVS prompt vocal wav for metadata processor when using --prompt-metadata-path (default: --prompt-audio)",
)
parser.add_argument(
"--preprocess-weights-dir",
type=str,
default=None,
help="SoulX-Singer-Preprocess weights (or set SOULX_PREPROCESS_WEIGHTS_DIR)",
)
parser.add_argument(
"--language",
type=str,
default="Mandarin",
help="SVS lyrics language (Mandarin, Cantonese, English)",
)
parser.add_argument(
"--control",
type=str,
default="melody",
choices=["score", "melody"],
help="SVS control mode",
)
parser.add_argument(
"--vocal-sep",
action=argparse.BooleanOptionalAction,
default=False,
help="Run vocal separation during preprocess",
)
parser.add_argument(
"--auto-shift",
action=argparse.BooleanOptionalAction,
default=True,
help="Auto pitch shift to align target pitch with prompt",
)
parser.add_argument("--pitch-shift", type=int, default=0)
parser.add_argument("--num-inference-steps", type=int, default=32)
parser.add_argument("--guidance-scale", type=float, default=3.0)
parser.add_argument(
"--seed",
type=int,
default=42,
help="Optional CFM RNG seed. Omit for non-deterministic sampling.",
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="SoulX-Singer offline SVS / SVC (preprocess + DiT)")
add_inference_args(parser)
parser.add_argument("--output", "-o", type=str, default="output.wav")
args = parser.parse_args()
return args
def main() -> None:
args = parse_args()
kind = resolve_soulx_kind(args.model)
if not (args.svc ^ (kind == "svs")):
raise ValueError(f"mode mismatch: {args.model}/config.json (declares {kind!r}), --svc={args.svc}")
required = {
"prompt_audio": args.prompt_audio,
"target_audio": args.target_audio,
}
if kind == "svs" and bool(args.prompt_metadata_path) ^ bool(args.target_metadata_path):
raise ValueError(
"SVS precomputed metadata requires both --prompt-metadata-path and "
"--target-metadata-path (or neither, for online preprocess)."
)
if args.prompt_metadata_path:
required["prompt_metadata_path"] = args.prompt_metadata_path
required["target_metadata_path"] = args.target_metadata_path
if args.prompt_wav_path:
required["prompt_wav_path"] = args.prompt_wav_path
_require_paths(required)
preprocess_dir = resolve_preprocess_weights_dir(args.preprocess_weights_dir)
deploy_config = args.deploy_config or str(SVC_DEPLOY_CONFIG if kind == "svc" else SVS_DEPLOY_CONFIG)
mode = kind.upper()
print(f"Loading SoulX-Singer {mode} from {args.model}")
print(f" deploy: {deploy_config}")
print(f" prompt: {args.prompt_audio}")
print(f" target: {args.target_audio}")
omni = Omni(model=args.model, deploy_config=deploy_config, async_chunk=False) # SoulX-Singer is batch-only
sampling = build_sampling(
args,
preprocess_weights_dir=preprocess_dir,
kind=kind,
)
prompt = {"prompt_token_ids": [0], "multi_modal_data": {}}
print("Running generate (inline preprocess + DiT)...")
outputs = list(omni.generate([prompt], sampling))
if not outputs:
raise RuntimeError("No output from omni.generate")
err = getattr(outputs[0], "error", None)
if err is not None:
raise RuntimeError(f"Pipeline error: {err}")
audio_np, sr = extract_audio(outputs)
sf.write(args.output, audio_np, sr)
print(f"Saved {args.output} ({sr} Hz, {len(audio_np) / sr:.2f}s)")
omni.close()
if __name__ == "__main__":
main()