[Doc] Sync and fix. (#2110)
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
This commit is contained in:
1 parent
9bf8f8ce79
commit
b0efb6ede0
23 files changed
+1382
-141
No files matched your search
@@ -0,0 +1,72 @@
|
||||
# Online serving Example of vLLM-Omni for MiMo-Audio
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/mimo_audio>.
|
||||
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
|
||||
|
||||
## Run examples (MiMo-Audio)
|
||||
|
||||
### Launch the Server
|
||||
```bash
|
||||
export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
|
||||
|
||||
vllm-omni serve XiaomiMiMo/MiMo-Audio-7B-Instruct --omni \
|
||||
--served-model-name "MiMo-Audio-7B-Instruct" \
|
||||
--port 18091 --stage-configs-path ./vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
|
||||
--chat-template ./examples/online_serving/mimo_audio/chat_template.jinja
|
||||
```
|
||||
> ⚠️ **Important**
|
||||
> **MiMo-Audio is not compatible with the default chat template.**
|
||||
> The provided `chat_template.jinja` implements MiMo-specific role, audio token, and instruction formatting and **must be used for all inference**.
|
||||
|
||||
|
||||
### Send Multi-modal Request
|
||||
|
||||
Get into the example folder
|
||||
```bash
|
||||
cd examples/online_serving/mimo_audio
|
||||
```
|
||||
|
||||
#### Send request via python
|
||||
|
||||
```bash
|
||||
# Audio dialogue task
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type multi_audios \
|
||||
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
|
||||
```
|
||||
|
||||
The Python client supports the following command-line arguments:
|
||||
|
||||
- `--query-type` (or `-q`): Query type (default: `multi_audios`)
|
||||
- Options: `multi_audios`, `text`
|
||||
- `--message-json` (or `-m`): Path to `base64` multi rounds audio messages json file
|
||||
- Do not pass any value for "text" query type
|
||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs, only for "Are these two audio clips the same?" task
|
||||
- Example: `---message-json ./examples/offline_inference/mimo_audio/message_base64_wav.json`
|
||||
- `--prompt` (or `-p`): Custom text prompt/question, only for query type is "text"(TTS task)
|
||||
- Attention! Do not pass any value for "multi_audios" query type
|
||||
- Example: `--prompt "What are the main activities shown in this video?"`
|
||||
|
||||
|
||||
For example, to use multi rounds audios with local files:
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type multi_audios \
|
||||
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
|
||||
```
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "chat_template.jinja"
|
||||
``````jinja
|
||||
--8<-- "examples/online_serving/mimo_audio/chat_template.jinja"
|
||||
``````
|
||||
??? abstract "openai_chat_completion_client_for_multimodal_generation.py"
|
||||
``````py
|
||||
--8<-- "examples/online_serving/mimo_audio/openai_chat_completion_client_for_multimodal_generation.py"
|
||||
``````
|
||||
Reference in new issue
Block a user