vllm-omni/examples/offline_inference/cosyvoice3

CosyVoice3

Setup

Install dependencies:

uv pip install -e .

Note: This includes required libraries such as soundfile, onnxruntime, x-transformers, and einops via requirements/common.txt and platform-specific requirements files.

Download the model snapshot:

from huggingface_hub import snapshot_download
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')

Add config.json in pretrained_models/Fun-CosyVoice3-0.5B/:

{
    "model_type": "cosyvoice3",
    "architectures": [
        "CosyVoice3Model"
    ]
}

Why config.json is required: AutoConfig.register("cosyvoice3", CosyVoice3Config) only registers a class mapping. The loader still needs model_type: "cosyvoice3" from config.json to select that class. If no config.json is present, model type cannot be inferred automatically. If your downloaded checkpoint already includes a valid config.json with model_type: "cosyvoice3", this manual step can be skipped.

Run the offline verification script:

python examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py \
  --model pretrained_models/Fun-CosyVoice3-0.5B \
  --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN

Implementation Overview

CosyVoice3 runs as a 2-stage Omni pipeline.

• Stage 0 (talker) converts text + prompt audio to speech tokens.
• Stage 1 (code2wav) converts speech tokens + prompt features to waveform via flow matching and HiFiGAN.

Key components live in vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py.

• CosyVoice3MultiModalProcessor builds the multimodal inputs:
  • Tokenizes prompt and prompt_text.
  • Extracts speech tokens and mel features from the prompt audio.
  • Extracts a speaker embedding.
• CosyVoice3Model implements both stages:
  • Stage 0 uses CosyVoice3LM and outputs speech tokens + conditioning features.
  • Stage 1 runs the flow model (DiT-based CFM) and HiFiGAN to synthesize waveform.

Pipeline topology lives in vllm_omni/model_executor/models/cosyvoice3/pipeline.py; runtime tunables (batch size, memory limits, sampling) live in vllm_omni/deploy/cosyvoice3.yaml. The deploy config auto-loads by HF model_type and defaults to async_chunk: true (shared-memory streaming). Pass --no-async-chunk on vllm serve to switch to the legacy sync path where stage 1 runs text2flow over the full speech-token sequence.

• Stage 0 emits latent speech tokens.
• Stage 1 consumes them via sync_process_input_func (sync mode) or the shared-memory connector (async-chunk mode) and outputs audio.