41 lines
870 B
Markdown
41 lines
870 B
Markdown
# Online serving Example of vLLM-omni for Qwen2.5-omni
|
|
|
|
## 🛠️ Installation
|
|
|
|
Please refer to [README.md](../../README.md)
|
|
|
|
## Run examples (Qwen2.5-omni)
|
|
|
|
Launch the server
|
|
```bash
|
|
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
|
|
```
|
|
|
|
If you have custom stage configs file, launch the server with command below
|
|
```bash
|
|
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
|
|
```
|
|
|
|
Get into the example folder
|
|
```bash
|
|
cd examples/online_serving
|
|
```
|
|
|
|
Send request via python
|
|
```bash
|
|
python openai_chat_completion_client_for_multimodal_generation.py --query-type mixed_modalities
|
|
```
|
|
|
|
Send request via curl
|
|
```bash
|
|
bash run_curl_multimodal_generation.sh mixed_modalities
|
|
```
|
|
|
|
### FAQ
|
|
|
|
If you encounter error about backend of librosa, try to install ffmpeg with command below.
|
|
```
|
|
sudo apt update
|
|
sudo apt install ffmpeg
|
|
```
|