[New Model] Add support for tencent/Covo-Audio-Chat (#2293)
Signed-off-by: Dnoob <dxpouo@gmail.com> Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
25 files changed
+2844
No files matched your search
@@ -29,6 +29,8 @@ recipes/
|
||||
Qwen-Image on `1x A100 80GB`, including optional step-wise continuous batching replay
|
||||
- [`Qwen/Qwen3-Omni.md`](./Qwen/Qwen3-Omni.md): online serving recipe for
|
||||
multimodal chat on `1x A100 80GB`
|
||||
- [`Tencent/Covo-Audio-Chat.md`](./Tencent/Covo-Audio-Chat.md): online
|
||||
serving recipe for audio chat on `1x A100 80GB`
|
||||
- [`Qwen/Qwen3-TTS.md`](./Qwen/Qwen3-TTS.md): text-to-speech serving recipe
|
||||
for Qwen3-TTS (CustomVoice / VoiceDesign / Base) on `1x H100/A100 80GB`
|
||||
- [`LTX/LTX-2.md`](./LTX/LTX-2.md): text-to-video and image-to-video serving
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
# Covo-Audio-Chat for audio chat on 1x A100 80GB
|
||||
|
||||
## Summary
|
||||
|
||||
- Vendor: Tencent
|
||||
- Model: `tencent/Covo-Audio-Chat`
|
||||
- Task: Audio chat — audio (and optional text) input, interleaved text + audio output
|
||||
- Mode: Online serving with the OpenAI-compatible API (offline inference also supported)
|
||||
- Maintainer: Community
|
||||
|
||||
## When to use this recipe
|
||||
|
||||
Use this recipe when you want a known-good starting point for serving
|
||||
`tencent/Covo-Audio-Chat` with vLLM-Omni on a single 80 GB-class GPU and
|
||||
validate the deployment with the bundled client example.
|
||||
|
||||
## References
|
||||
|
||||
- Related example under `examples/`:
|
||||
[`examples/online_serving/covo_audio/README.md`](../../examples/online_serving/covo_audio/README.md)
|
||||
- Offline end-to-end script:
|
||||
[`examples/offline_inference/covo_audio/end2end.py`](../../examples/offline_inference/covo_audio/end2end.py)
|
||||
- Deploy config:
|
||||
[`vllm_omni/deploy/covo_audio.yaml`](../../vllm_omni/deploy/covo_audio.yaml)
|
||||
- Pipeline topology:
|
||||
[`vllm_omni/model_executor/models/covo_audio/pipeline.py`](../../vllm_omni/model_executor/models/covo_audio/pipeline.py)
|
||||
|
||||
## Hardware Support
|
||||
|
||||
This recipe currently documents one tested reference configuration for
|
||||
CUDA GPU serving. The fused thinker+talker stage and the BigVGAN code2wav
|
||||
stage both run on the same physical device.
|
||||
|
||||
## GPU
|
||||
|
||||
### 1x A100 80GB
|
||||
|
||||
#### Environment
|
||||
|
||||
- OS: Linux
|
||||
- Python: 3.12
|
||||
- Driver / runtime: NVIDIA CUDA 12.x with an A100 80 GB GPU
|
||||
- vLLM version: Match the repository requirements for your checkout
|
||||
- vLLM-Omni version or commit: Use the commit you are deploying from
|
||||
- Extra Python deps: `torchdiffeq`, `librosa`
|
||||
|
||||
#### Command
|
||||
|
||||
Install the two runtime-only deps that are not pulled in by vLLM-Omni itself:
|
||||
|
||||
```bash
|
||||
pip install torchdiffeq librosa
|
||||
```
|
||||
|
||||
Start the server from the repository root:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 vllm serve tencent/Covo-Audio-Chat --omni \
|
||||
--trust-remote-code --port 18091
|
||||
```
|
||||
|
||||
The deploy config at `vllm_omni/deploy/covo_audio.yaml` is loaded
|
||||
automatically by the model registry.
|
||||
|
||||
#### Verification
|
||||
|
||||
With the server ready, send a test request via the bundled client:
|
||||
|
||||
```bash
|
||||
cd examples/online_serving/covo_audio
|
||||
python openai_chat_completion_client.py
|
||||
```
|
||||
|
||||
Expected: the client prints the generated text and writes an output WAV
|
||||
file to the current directory.
|
||||
|
||||
#### Notes
|
||||
|
||||
- Memory usage: Stage 0 (7B LLM) ~16 GiB + stage 1 (BigVGAN vocoder)
|
||||
~2 GiB, both resident on the same device.
|
||||
- Key flags: `--omni` and `--trust-remote-code` are both required.
|
||||
- System prompt: the bundled `openai_chat_completion_client.py` and
|
||||
`end2end.py` already include `COVO_AUDIO_SYSTEM_PROMPT` (defined in
|
||||
`vllm_omni/model_executor/models/covo_audio/prompt_utils.py`) in the
|
||||
system message by default. When integrating your own client, make sure
|
||||
to include it — without it, stage 0 emits text-only tokens and the
|
||||
output audio is silent.
|
||||
- Stage 1 runs with `enforce_eager: true` and `dtype: float32` — see the
|
||||
header of `deploy/covo_audio.yaml` for the specific reasons.
|
||||
- Known limitations: this starter recipe covers the single-GPU online-serving
|
||||
path only; multi-GPU layouts and platform-specific overrides are not
|
||||
documented here.
|
||||
Reference in new issue
Block a user