78 lines
2.4 KiB
Markdown
78 lines
2.4 KiB
Markdown
# Covo-Audio-Chat: Offline inference
|
|
|
|
## Setup
|
|
|
|
Use `--deploy-config` for deployment overrides such as stage memory allocation.
|
|
See the [pipeline and deploy configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/).
|
|
|
|
> **Note**
|
|
> Covo-Audio code2wav requires `torchdiffeq`. Install it with: `pip install torchdiffeq`
|
|
|
|
## Run examples
|
|
|
|
Get into the example folder:
|
|
```bash
|
|
cd examples/offline_inference/covo_audio
|
|
```
|
|
|
|
### Audio input chat
|
|
|
|
Using the default audio asset:
|
|
```bash
|
|
python end2end.py
|
|
```
|
|
|
|
Using a custom audio file:
|
|
```bash
|
|
python end2end.py --audio-path /path/to/audio.wav
|
|
```
|
|
|
|
Using a local model:
|
|
```bash
|
|
python end2end.py -m /path/to/Covo-Audio-Chat --output-dir ./my_output
|
|
```
|
|
|
|
### Command-line Arguments
|
|
|
|
| Argument | Short | Default | Description |
|
|
|----------|-------|---------|-------------|
|
|
| `--model-name` | `-m` | `tencent/Covo-Audio-Chat` | Model path or HuggingFace model ID |
|
|
| `--text` | `-t` | `请回答这段音频里的问题。` | Text prompt / question for the audio |
|
|
| `--audio-path` | `-a` | default audio asset | Path to local audio file |
|
|
| `--sampling-rate` | | `16000` | Sampling rate for audio loading (Hz) |
|
|
| `--output-dir` | | `./output_audio` | Output directory for generated files |
|
|
| `--num-prompts` | | `1` | Number of prompts to generate |
|
|
| `--deploy-config` | | (auto) | Optional path to a deploy YAML override |
|
|
| `--log-stats` | | `false` | Enable detailed statistics logging |
|
|
| `--stage-init-timeout` | | `300` | Stage initialization timeout (seconds) |
|
|
| `--batch-timeout` | | `5` | Batching timeout (seconds) |
|
|
| `--init-timeout` | | `300` | Overall initialization timeout (seconds) |
|
|
| `--shm-threshold-bytes` | | `65536` | Shared memory threshold (bytes) |
|
|
|
|
## Pipeline
|
|
|
|
Covo-Audio-Chat uses a 2-stage pipeline:
|
|
|
|
- **Stage 0 (fused_thinker_talker):** The 7B LLM generates interleaved text and audio tokens in a single autoregressive pass.
|
|
- **Stage 1 (code2wav):** A BigVGAN-based vocoder converts the extracted audio codes into a 24kHz WAV waveform.
|
|
|
|
## Output
|
|
|
|
The script generates two files per request in the output directory:
|
|
|
|
- `{request_id}.txt` -- prompt and generated text
|
|
- `{request_id}.wav` -- generated audio (24kHz WAV)
|
|
|
|
## FAQ
|
|
|
|
If you encounter `ModuleNotFoundError: No module named 'librosa'`, install it with:
|
|
```bash
|
|
pip install librosa
|
|
```
|
|
|
|
## Environment
|
|
|
|
- GPU: 1x A100 (80 GiB)
|
|
- Stage 0 (7B LLM): ~16 GiB VRAM
|
|
- Stage 1 (BigVGAN vocoder): ~2 GiB VRAM
|