Yueqian LinandSYLAR
3f504b45d2
Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
62 lines
2.4 KiB
Markdown
62 lines
2.4 KiB
Markdown
# Online serving Example of vLLM-Omni for MiMo-Audio
|
|
|
|
## 🛠️ Installation
|
|
|
|
Please refer to [README.md](../../../README.md)
|
|
|
|
> ⚠️ **Important (audio generation)**
|
|
> For **audio generation** (TTS, responses that include synthesized audio, etc.), install **`flash-attn`** for your **CUDA** and **PyTorch** stack. Without it on GPU, **output audio may be noise-only or unusable**. See the [FlashAttention](https://github.com/Dao-AILab/flash-attention) repository for compatible builds.
|
|
|
|
## Run examples (MiMo-Audio)
|
|
|
|
### Launch the Server
|
|
```bash
|
|
export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
|
|
|
|
vllm serve XiaomiMiMo/MiMo-Audio-7B-Instruct --omni \
|
|
--served-model-name "MiMo-Audio-7B-Instruct" \
|
|
--port 18091 \
|
|
--chat-template ./examples/online_serving/mimo_audio/chat_template.jinja
|
|
```
|
|
> ⚠️ **Important**
|
|
> **MiMo-Audio is not compatible with the default chat template.**
|
|
> The provided `chat_template.jinja` implements MiMo-specific role, audio token, and instruction formatting and **must be used for all inference**.
|
|
|
|
|
|
### Send Multi-modal Request
|
|
|
|
Get into the example folder
|
|
```bash
|
|
cd examples/online_serving/mimo_audio
|
|
```
|
|
|
|
#### Send request via python
|
|
|
|
```bash
|
|
# Audio dialogue task
|
|
python openai_chat_completion_client_for_multimodal_generation.py \
|
|
--query-type multi_audios \
|
|
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
|
|
```
|
|
|
|
The Python client supports the following command-line arguments:
|
|
|
|
- `--query-type` (or `-q`): Query type (default: `multi_audios`)
|
|
- Options: `multi_audios`, `text`
|
|
- `--message-json` (or `-m`): Path to `base64` multi rounds audio messages json file
|
|
- Do not pass any value for "text" query type
|
|
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs, only for "Are these two audio clips the same?" task
|
|
- Example: `---message-json ./examples/offline_inference/mimo_audio/message_base64_wav.json`
|
|
- `--prompt` (or `-p`): Custom text prompt/question, only for query type is "text"(TTS task)
|
|
- Attention! Do not pass any value for "multi_audios" query type
|
|
- Example: `--prompt "What are the main activities shown in this video?"`
|
|
|
|
|
|
For example, to use multi rounds audios with local files:
|
|
|
|
```bash
|
|
python openai_chat_completion_client_for_multimodal_generation.py \
|
|
--query-type multi_audios \
|
|
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
|
|
```
|