Yueqian LinandSYLAR
3f504b45d2
Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
82 lines
2.9 KiB
Markdown
82 lines
2.9 KiB
Markdown
# CosyVoice3
|
|
|
|
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/cosyvoice3>.
|
|
|
|
|
|
## Setup
|
|
|
|
Install dependencies:
|
|
```
|
|
uv pip install -e .
|
|
```
|
|
|
|
> **Note:** This includes required libraries such as `soundfile`,
|
|
> `onnxruntime`, `x-transformers`, and `einops` via
|
|
> `requirements/common.txt` and platform-specific requirements files.
|
|
|
|
Download the model snapshot:
|
|
```
|
|
from huggingface_hub import snapshot_download
|
|
snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B')
|
|
```
|
|
|
|
Add `config.json` in `pretrained_models/Fun-CosyVoice3-0.5B/`:
|
|
```json
|
|
{
|
|
"model_type": "cosyvoice3",
|
|
"architectures": [
|
|
"CosyVoice3Model"
|
|
]
|
|
}
|
|
```
|
|
|
|
> **Why `config.json` is required:**
|
|
> `AutoConfig.register("cosyvoice3", CosyVoice3Config)` only registers a class mapping.
|
|
> The loader still needs `model_type: "cosyvoice3"` from `config.json` to select that class.
|
|
> If no `config.json` is present, model type cannot be inferred automatically.
|
|
> If your downloaded checkpoint already includes a valid `config.json` with
|
|
> `model_type: "cosyvoice3"`, this manual step can be skipped.
|
|
|
|
Run the offline verification script:
|
|
```
|
|
python examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py \
|
|
--model pretrained_models/Fun-CosyVoice3-0.5B \
|
|
--tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN
|
|
```
|
|
|
|
## Implementation Overview
|
|
|
|
CosyVoice3 runs as a **2-stage Omni pipeline**.
|
|
|
|
- **Stage 0** (`talker`) converts text + prompt audio to speech tokens.
|
|
- **Stage 1** (`code2wav`) converts speech tokens + prompt features to waveform via flow matching and HiFiGAN.
|
|
|
|
Key components live in `vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py`.
|
|
|
|
- `CosyVoice3MultiModalProcessor` builds the multimodal inputs:
|
|
- Tokenizes `prompt` and `prompt_text`.
|
|
- Extracts speech tokens and mel features from the prompt audio.
|
|
- Extracts a speaker embedding.
|
|
- `CosyVoice3Model` implements both stages:
|
|
- Stage 0 uses `CosyVoice3LM` and outputs speech tokens + conditioning features.
|
|
- Stage 1 runs the flow model (DiT-based CFM) and HiFiGAN to synthesize waveform.
|
|
|
|
Pipeline topology lives in `vllm_omni/model_executor/models/cosyvoice3/pipeline.py`;
|
|
runtime tunables (batch size, memory limits, sampling) live in
|
|
`vllm_omni/deploy/cosyvoice3.yaml`. The deploy config auto-loads by
|
|
HF `model_type` and defaults to `async_chunk: true` (shared-memory
|
|
streaming). Pass `--no-async-chunk` on `vllm serve` to switch to the
|
|
legacy sync path where stage 1 runs `text2flow` over the full
|
|
speech-token sequence.
|
|
|
|
- Stage 0 emits latent speech tokens.
|
|
- Stage 1 consumes them via `sync_process_input_func` (sync mode) or the
|
|
shared-memory connector (async-chunk mode) and outputs audio.
|
|
|
|
## Example materials
|
|
|
|
??? abstract "verify_e2e_cosyvoice.py"
|
|
``````py
|
|
--8<-- "examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py"
|
|
``````
|