+13
b19c227fb3
b19c227fb3
Signed-off-by: wangyu31577 <wangyu31577@hundsun.com> Signed-off-by: 齐保元 <qibaoyuan@xiaomi.com> Signed-off-by: hsliu <liuhongsheng4@huawei.com> Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Signed-off-by: GG-li <3226868735@qq.com> Signed-off-by: Sihao Li <111170255+GG-li@users.noreply.github.com> Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com> Signed-off-by: mxuax <mxuax@connect.ust.hk> Signed-off-by: Baoyuan Qi <qibaoyuan@126.com> Signed-off-by: gcanlin <canlinguosdu@gmail.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: wuzhongjian <wuzhongjian_yewu@cmss.chinamobile.com> Signed-off-by: dongbo910220 <1275604947@qq.com> Signed-off-by: dongbo910220 <32610838+dongbo910220@users.noreply.github.com> Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com> Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com> Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com> Signed-off-by: baoyuan qi <qibaoyuan@126.com> Signed-off-by: tzhouam <tzhouam@connect.ust.hk> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: Prajwal A <prajwalanagani@gmail.com> Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com> Signed-off-by: 丁宁 <nndding@gmail.com> Signed-off-by: SHIJIN ZHANG <75300765+Dovis01@users.noreply.github.com> Signed-off-by: dingning<dingning7@xiaomi.com> Signed-off-by: dingning <dingning7@xiaomi.com> Signed-off-by: dingning <dingning@xiaomi.com> Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com> Co-authored-by: wangyu31577 <wangyu31577@hundsun.com> Co-authored-by: Zhang Shijin <zhangshijin@xiaomi.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: Sihao Li <111170255+GG-li@users.noreply.github.com> Co-authored-by: XU Mingshi <91017482+mxuax@users.noreply.github.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: JohnJan <wuzhongjian_yewu@cmss.chinamobile.com> Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com> Co-authored-by: dongbo910220 <32610838+dongbo910220@users.noreply.github.com> Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com> Co-authored-by: Junhong Liu <ljh_lbj@163.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com> Co-authored-by: shijin zhang <zsj1364226740@gmail.com> Co-authored-by: Zhou Taichang <tzhouam@connect.ust.hk> Co-authored-by: root <root@hk01dgx028.cm.cluster> Co-authored-by: Prajwal A <34590600+LawJarp-A@users.noreply.github.com> Co-authored-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com> Co-authored-by: dingning <dingning7@xiaomi.com> Co-authored-by: ning ding <nndding@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
59 lines
2.1 KiB
Markdown
59 lines
2.1 KiB
Markdown
# Online serving Example of vLLM-Omni for MiMo-Audio
|
|
|
|
## 🛠️ Installation
|
|
|
|
Please refer to [README.md](../../../README.md)
|
|
|
|
## Run examples (MiMo-Audio)
|
|
|
|
### Launch the Server
|
|
```bash
|
|
export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
|
|
|
|
vllm-omni serve XiaomiMiMo/MiMo-Audio-7B-Instruct --omni \
|
|
--served-model-name "MiMo-Audio-7B-Instruct" \
|
|
--port 18091 --stage-configs-path ./vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
|
--chat-template ./examples/online_serving/mimo_audio/chat_template.jinja
|
|
```
|
|
> ⚠️ **Important**
|
|
> **MiMo-Audio is not compatible with the default chat template.**
|
|
> The provided `chat_template.jinja` implements MiMo-specific role, audio token, and instruction formatting and **must be used for all inference**.
|
|
|
|
|
|
### Send Multi-modal Request
|
|
|
|
Get into the example folder
|
|
```bash
|
|
cd examples/online_serving/mimo_audio
|
|
```
|
|
|
|
#### Send request via python
|
|
|
|
```bash
|
|
# Audio dialogue task
|
|
python openai_chat_completion_client_for_multimodal_generation.py \
|
|
--query-type multi_audios \
|
|
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
|
|
```
|
|
|
|
The Python client supports the following command-line arguments:
|
|
|
|
- `--query-type` (or `-q`): Query type (default: `multi_audios`)
|
|
- Options: `multi_audios`, `text`
|
|
- `--message-json` (or `-m`): Path to `base64` multi rounds audio messages json file
|
|
- Do not pass any value for "text" query type
|
|
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs, only for "Are these two audio clips the same?" task
|
|
- Example: `---message-json ./examples/offline_inference/mimo_audio/message_base64_wav.json`
|
|
- `--prompt` (or `-p`): Custom text prompt/question, only for query type is "text"(TTS task)
|
|
- Attention! Do not pass any value for "multi_audios" query type
|
|
- Example: `--prompt "What are the main activities shown in this video?"`
|
|
|
|
|
|
For example, to use multi rounds audios with local files:
|
|
|
|
```bash
|
|
python openai_chat_completion_client_for_multimodal_generation.py \
|
|
--query-type multi_audios \
|
|
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
|
|
```
|