[Feature] Support online inference (#64)

Signed-off-by: Gaohan123 <gaohan19@huawei.com>
This commit is contained in:
Gao Han authored and GitHub committed 2025-11-17 14:44:10 +08:00
1 parent bf82e18374
commit c5a89302f1
52 files changed
+3666 -685

No files matched your search

@@ -0,0 +1,35 @@
# Online serving Example of vLLM-omni for Qwen2.5-omni
Source <https://github.com/vllm-project/vllm-omni/blob/main/examples\online_serving\README.md>.
## 🛠️ Installation
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/blob/main/README.md)
## Run examples (Qwen2.5-omni)
Launch the server
```bash
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
```
If you have custom stage configs file, launch the server with command below
```bash
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
```
Get into the example folder
```bash
cd examples/online_serving
```
Send request via python
```bash
python openai_chat_completion_client_for_multimodal_generation.py
```
Send request via curl
```bash
bash run_curl_multimodal_generation.sh
```
@@ -0,0 +1,7 @@
# OpenAI Chat Completion Client For Multimodal Generation
Source <https://github.com/vllm-project/vllm-omni/blob/main/examples\online_serving\openai_chat_completion_client_for_multimodal_generation.py>.
``````py
--8<-- "examples\online_serving\openai_chat_completion_client_for_multimodal_generation.py"
``````