[Feature] Support online inference (#64)
Signed-off-by: Gaohan123 <gaohan19@huawei.com>
This commit is contained in:
1 parent
bf82e18374
commit
c5a89302f1
52 files changed
+3666
-685
No files matched your search
@@ -0,0 +1,35 @@
|
||||
# Online serving Example of vLLM-omni for Qwen2.5-omni
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/blob/main/examples\online_serving\README.md>.
|
||||
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/blob/main/README.md)
|
||||
|
||||
## Run examples (Qwen2.5-omni)
|
||||
|
||||
Launch the server
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
|
||||
```
|
||||
|
||||
If you have custom stage configs file, launch the server with command below
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
|
||||
```
|
||||
|
||||
Get into the example folder
|
||||
```bash
|
||||
cd examples/online_serving
|
||||
```
|
||||
|
||||
Send request via python
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py
|
||||
```
|
||||
|
||||
Send request via curl
|
||||
```bash
|
||||
bash run_curl_multimodal_generation.sh
|
||||
```
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
# OpenAI Chat Completion Client For Multimodal Generation
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/blob/main/examples\online_serving\openai_chat_completion_client_for_multimodal_generation.py>.
|
||||
|
||||
``````py
|
||||
--8<-- "examples\online_serving\openai_chat_completion_client_for_multimodal_generation.py"
|
||||
``````
|
||||
Reference in new issue
Block a user