Compare commits
15
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Roger Wang
|
9fe730a44b | ||
|
Mengqing Cao
|
00b1a9a1a0 | ||
|
Canlin Guo
|
7e60c49ce4 | ||
|
Samit
|
98d94e0e3d | ||
|
Roger Wang
|
510aeb7da4 | ||
|
Roger Wang
|
5939089c98 | ||
|
Samit
|
1a23d70a97 | ||
|
Roger Wang
|
a6c5a6aca4 | ||
|
Gao Han
|
d53155f45c | ||
|
youkaichao
|
e8fb85d38c | ||
|
Samit
|
f378f80430 | ||
|
Roger Wang
|
a1914e1176 | ||
|
Roger Wang
|
3b12ed9cf4 | ||
|
Roger Wang
|
c852267a1a | ||
|
Cyrus Leung
|
2c5d0481b3 |
No files matched your search
+1
-1
@@ -1,3 +1,3 @@
|
||||
# Contributing to vLLM-Omni
|
||||
|
||||
You may find information about contributing to vLLM-Omni on [Contributing](docs/contributing/)
|
||||
You may find information about contributing to vLLM-Omni on [Contributing](https://vllm-omni.readthedocs.io/en/latest/contributing/)
|
||||
@@ -1,7 +1,7 @@
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="./docs/source/logos/vllm-omni-logo-text-dark.png">
|
||||
<img alt="vllm-omni" src="./docs/source/logos/vllm-omni-logo-text-dark.png" width=55%>
|
||||
<source media="(prefers-color-scheme: dark)" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/logos/vllm-omni-logo.png">
|
||||
<img alt="vllm-omni" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/logos/vllm-omni-logo.png" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
<h3 align="center">
|
||||
@@ -9,7 +9,7 @@ Easy, fast, and cheap omni-modality model serving for everyone
|
||||
</h3>
|
||||
|
||||
<p align="center">
|
||||
| <a href="https://vllm-omni.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href="https://tinyurl.com/vllm-omni-meeting"><b>Weekly Meeting</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> |
|
||||
| <a href="https://vllm-omni.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> |
|
||||
</p>
|
||||
|
||||
---
|
||||
@@ -30,7 +30,7 @@ Easy, fast, and cheap omni-modality model serving for everyone
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<img alt="vllm-omni" src="./docs/source/architecture/omni-modality-model-architecture.png" width=55%>
|
||||
<img alt="vllm-omni" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/omni-modality-model-architecture.png" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
@@ -66,10 +66,8 @@ Visit our [documentation](https://vllm-omni.readthedocs.io/en/latest/) to learn
|
||||
We welcome and value any contributions and collaborations.
|
||||
Please check out [Contributing to vLLM-Omni](https://vllm-omni.readthedocs.io/en/latest/contributing/) for how to get involved.
|
||||
|
||||
## Weekly Meeting
|
||||
|
||||
- vLLM Omni Weekly Meeting: https://tinyurl.com/vllm-omni-meeting
|
||||
- Wednesday, 11:30 - 12:30 (UTC+8, [Convert to your timezone](https://dateful.com/convert/gmt8?t=15))
|
||||
## Join the Community
|
||||
Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in `#sig-omni` slack channel at [slack.vllm.ai](https://slack.vllm.ai) or vLLM user forum at [discuss.vllm.ai](https://discuss.vllm.ai).
|
||||
|
||||
## License
|
||||
|
||||
|
||||
+3
-4
@@ -7,11 +7,11 @@ nav:
|
||||
- Examples:
|
||||
- examples/README.md
|
||||
- Offline Inference:
|
||||
- Offline Example of vLLM-Omni for Qwen2.5-omni: user_guide/examples/offline_inference/qwen2_5_omni.md
|
||||
- Offline Example of vLLM-Omni for Qwen3-omni: user_guide/examples/offline_inference/qwen3_omni.md
|
||||
- Offline Example of vLLM-Omni for Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md
|
||||
- Offline Example of vLLM-Omni for Qwen3-Omni: user_guide/examples/offline_inference/qwen3_omni.md
|
||||
- Qwen-Image Offline Inference: user_guide/examples/offline_inference/qwen_image.md
|
||||
- Online Serving:
|
||||
- Online serving Example of vLLM-Omni for Qwen2.5-omni: user_guide/examples/online_serving/qwen2_5_omni.md
|
||||
- Online serving Example of vLLM-Omni for Qwen2.5-Omni: user_guide/examples/online_serving/qwen2_5_omni.md
|
||||
- Online serving Example of vLLM-Omni for Qwen3-omni: user_guide/examples/online_serving/qwen3_omni.md
|
||||
- General:
|
||||
- usage/*
|
||||
@@ -33,7 +33,6 @@ nav:
|
||||
- design/architecture_overview.md
|
||||
- design/vllm_omni_design.md
|
||||
- design/mrs_design.md
|
||||
- design/api_design_doc.md
|
||||
- Docs Guide: contributing/DOCS_GUIDE.md
|
||||
- API Reference:
|
||||
- api/README.md
|
||||
|
||||
+5
-4
@@ -8,8 +8,8 @@ hide:
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="./source/logos/vllm-omni-logo-text-dark.png">
|
||||
<img alt="vllm-omni" src="./source/logos/vllm-omni-logo-text-dark.png" width=55%>
|
||||
<source media="(prefers-color-scheme: dark)" src="./source/logos/vllm-omni-logo.png">
|
||||
<img alt="vllm-omni" src="./source/logos/vllm-omni-logo.png" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
<h3 align="center">
|
||||
@@ -34,7 +34,8 @@ Easy, fast, and cheap omni-modality model serving for everyone
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<img alt="vllm-omni" src="./source/architecture/omni-modality model architecture.png" width=55%>
|
||||
<source media="(prefers-color-scheme: dark)" src="./source/architecture/omni-modality-model-architecture.png">
|
||||
<img alt="vllm-omni-arch" src="./source/architecture/omni-modality-model-architecture.png" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
@@ -58,4 +59,4 @@ vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, in
|
||||
- Multi-modality generation models (e.g. Qwen-Image)
|
||||
|
||||
For more information, checkout the following:
|
||||
- [Placeholder for vllm-omni blogpost]()
|
||||
- [vllm-omni announcement blogpost](https://blog.vllm.ai/2025/11/30/vllm-omni.html)
|
||||
@@ -102,6 +102,10 @@ Configuration classes.
|
||||
|
||||
Worker classes and model runners for distributed inference.
|
||||
|
||||
- [vllm_omni.diffusion.worker.gpu_worker.GPUWorker][]
|
||||
- [vllm_omni.diffusion.worker.gpu_worker.WorkerProc][]
|
||||
- [vllm_omni.diffusion.worker.npu.npu_worker.NPUWorker][]
|
||||
- [vllm_omni.diffusion.worker.npu.npu_worker.NPUWorkerProc][]
|
||||
- [vllm_omni.worker.gpu_ar_model_runner.GPUARModelRunner][]
|
||||
- [vllm_omni.worker.gpu_ar_worker.GPUARWorker][]
|
||||
- [vllm_omni.worker.gpu_diffusion_model_runner.GPUDiffusionModelRunner][]
|
||||
|
||||
@@ -2,6 +2,9 @@
|
||||
|
||||
In vLLM-Omni, the target model is separated into multiple stages, which are processed by different LLMEngines, DiffusionEngines or other types of engines. Depending on different types of stages, such as Autoregressive (AR) stage or Diffusion transformer (DiT) stage, each can choose corresponding schedulers, model workers to load with the Engines in a plug-in fashion.
|
||||
|
||||
!!! note
|
||||
Default stage config YAMLs (for example, `vllm_omni/model_executor/stage_configs/qwen2_5_omni.yaml` and `vllm_omni/model_executor/stage_configs/qwen3_omni_moe.yaml`) are bundled and loaded automatically when `stage_configs_path` is not provided. They have been verified to work on 1xH100 for Qwen2.5-Omni and 2xH100 for Qwen3-Omni.
|
||||
|
||||
Therefore, as a core part of vLLM-Omni, the stage configs for a model have several main functions:
|
||||
|
||||
- Claim partition of stages and their corresponding class implementation in `model_executor/models`.
|
||||
|
||||
@@ -2,6 +2,9 @@
|
||||
|
||||
Thank you for your interest in contributing to vLLM-Omni! This document provides guidelines and instructions for contributing.
|
||||
|
||||
!!! note
|
||||
We host weekly developer-facing online meetings to discuss milestones and updates **every Tuesday at 19:30 PDT**. Meeting link as well as the past meeting notes can be found [here](https://tinyurl.com/vllm-omni-meeting).
|
||||
|
||||
## Getting Started
|
||||
|
||||
vLLM-Omni uses `uv` as the environment manager, to create and manage Python environments. Please follow the documentation to install `uv`. After installing `uv`, you can create a new Python environment using the following commands:
|
||||
|
||||
@@ -7,7 +7,3 @@ This section contains design documents and architecture specifications for vLLM-
|
||||
- [Architecture Overview](architecture_overview.md)
|
||||
- [vLLM-Omni Software Design Document](vllm_omni_design.md)
|
||||
- [Multi-Request Streaming](mrs_design.md)
|
||||
|
||||
## API Design Documentation
|
||||
|
||||
- [vLLM-Omni API Design Documentation](api_design_doc.md)
|
||||
@@ -19,6 +19,7 @@ export DEVICE1=/dev/davinci1
|
||||
export IMAGE=quay.io/ascend/vllm-ascend:v0.11.0rc2
|
||||
docker run --rm \
|
||||
--name vllm-omni-npu \
|
||||
--shm-size=1g \
|
||||
--device $DEVICE0 \
|
||||
--device $DEVICE1 \
|
||||
--device /dev/davinci_manager \
|
||||
|
||||
@@ -192,7 +192,8 @@ class Example:
|
||||
code_fence = "``````"
|
||||
|
||||
if self.is_code:
|
||||
content += f'{code_fence}{self.main_file.suffix[1:]}\n--8<-- "{self.main_file}"\n{code_fence}\n'
|
||||
main_file_rel = self.main_file.relative_to(ROOT_DIR)
|
||||
content += f'{code_fence}{self.main_file.suffix[1:]}\n--8<-- "{main_file_rel}"\n{code_fence}\n'
|
||||
else:
|
||||
with open(self.main_file) as f:
|
||||
# Skip the title from md snippets as it's been included above
|
||||
@@ -208,7 +209,7 @@ class Example:
|
||||
content += f'??? abstract "{file.relative_to(self.path)}"\n'
|
||||
if file.suffix != ".md":
|
||||
content += f" {code_fence}{file.suffix[1:]}\n"
|
||||
content += f' --8<-- "{file}"\n'
|
||||
content += f' --8<-- "{file.relative_to(ROOT_DIR)}"\n'
|
||||
if file.suffix != ".md":
|
||||
content += f" {code_fence}\n"
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
@@ -1,4 +1,4 @@
|
||||
# Offline Example of vLLM-Omni for Qwen2.5-omni
|
||||
# Offline Example of vLLM-Omni for Qwen2.5-Omni
|
||||
|
||||
Source <https://github.com/vllm-project/vllm/tree/main/examples/offline_inference/qwen2_5_omni>.
|
||||
|
||||
@@ -7,7 +7,7 @@ Source <https://github.com/vllm-project/vllm/tree/main/examples/offline_inferenc
|
||||
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm/tree/main/README.md)
|
||||
|
||||
## Run examples (Qwen2.5-omni)
|
||||
## Run examples (Qwen2.5-Omni)
|
||||
### Multiple Prompts
|
||||
Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). To get the prompt, you can:
|
||||
```bash
|
||||
@@ -48,17 +48,17 @@ sudo apt install ffmpeg
|
||||
|
||||
??? abstract "end2end.py"
|
||||
``````py
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen2_5_omni/end2end.py"
|
||||
--8<-- "examples/offline_inference/qwen2_5_omni/end2end.py"
|
||||
``````
|
||||
??? abstract "extract_prompts.py"
|
||||
``````py
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen2_5_omni/extract_prompts.py"
|
||||
--8<-- "examples/offline_inference/qwen2_5_omni/extract_prompts.py"
|
||||
``````
|
||||
??? abstract "run_multiple_prompts.sh"
|
||||
``````sh
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen2_5_omni/run_multiple_prompts.sh"
|
||||
--8<-- "examples/offline_inference/qwen2_5_omni/run_multiple_prompts.sh"
|
||||
``````
|
||||
??? abstract "run_single_prompt.sh"
|
||||
``````sh
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen2_5_omni/run_single_prompt.sh"
|
||||
--8<-- "examples/offline_inference/qwen2_5_omni/run_single_prompt.sh"
|
||||
``````
|
||||
@@ -1,4 +1,4 @@
|
||||
# Offline Example of vLLM-Omni for Qwen3-omni
|
||||
# Offline Example of vLLM-Omni for Qwen3-Omni
|
||||
|
||||
Source <https://github.com/vllm-project/vllm/tree/main/examples/offline_inference/qwen3_omni>.
|
||||
|
||||
@@ -7,9 +7,9 @@ Source <https://github.com/vllm-project/vllm/tree/main/examples/offline_inferenc
|
||||
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm/tree/main/README.md)
|
||||
|
||||
## Run examples (Qwen3-omni)
|
||||
## Run examples (Qwen3-Omni)
|
||||
### Multiple Prompts
|
||||
Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). For processing dataset please refer to [Qwen2.5-omni README.md](https://github.com/vllm-project/vllm/tree/main/examples/offline_inference/qwen2_5_omni/README.md)
|
||||
Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). For processing dataset please refer to [Qwen2.5-Omni README.md](https://github.com/vllm-project/vllm/tree/main/examples/offline_inference/qwen2_5_omni/README.md)
|
||||
Get into the example folder
|
||||
```bash
|
||||
cd examples/offline_inference/qwen3_omni
|
||||
@@ -44,21 +44,17 @@ sudo apt install ffmpeg
|
||||
|
||||
??? abstract "end2end.py"
|
||||
``````py
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/end2end.py"
|
||||
``````
|
||||
??? abstract "qwen3_omni_moe_tp.yaml"
|
||||
``````yaml
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/qwen3_omni_moe_tp.yaml"
|
||||
--8<-- "examples/offline_inference/qwen3_omni/end2end.py"
|
||||
``````
|
||||
??? abstract "run_multiple_prompts.sh"
|
||||
``````sh
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/run_multiple_prompts.sh"
|
||||
--8<-- "examples/offline_inference/qwen3_omni/run_multiple_prompts.sh"
|
||||
``````
|
||||
??? abstract "run_single_prompt.sh"
|
||||
``````sh
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/run_single_prompt.sh"
|
||||
--8<-- "examples/offline_inference/qwen3_omni/run_single_prompt.sh"
|
||||
``````
|
||||
??? abstract "run_single_prompt_tp.sh"
|
||||
``````sh
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/run_single_prompt_tp.sh"
|
||||
--8<-- "examples/offline_inference/qwen3_omni/run_single_prompt_tp.sh"
|
||||
``````
|
||||
@@ -49,9 +49,9 @@ Then open `http://localhost:7862/` on your local browser to interact with the we
|
||||
|
||||
??? abstract "gradio_demo.py"
|
||||
``````py
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen_image/gradio_demo.py"
|
||||
--8<-- "examples/offline_inference/qwen_image/gradio_demo.py"
|
||||
``````
|
||||
??? abstract "text_to_image.py"
|
||||
``````py
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen_image/text_to_image.py"
|
||||
--8<-- "examples/offline_inference/qwen_image/text_to_image.py"
|
||||
``````
|
||||
@@ -1,13 +1,13 @@
|
||||
# Online serving Example of vLLM-Omni for Qwen2.5-omni
|
||||
# Online serving Example of vLLM-Omni for Qwen2.5-Omni
|
||||
|
||||
Source <https://github.com/vllm-project/vllm/tree/main/examples/online_serving/qwen2_5_omni>.
|
||||
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm/tree/main/examples/README.md)
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm/tree/main/README.md)
|
||||
|
||||
## Run examples (Qwen2.5-omni)
|
||||
## Run examples (Qwen2.5-Omni)
|
||||
|
||||
Launch the server
|
||||
```bash
|
||||
@@ -21,7 +21,7 @@ vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to
|
||||
|
||||
Get into the example folder
|
||||
```bash
|
||||
cd examples/online_serving
|
||||
cd examples/online_serving/qwen2_5_omni
|
||||
```
|
||||
|
||||
Send request via python
|
||||
@@ -79,13 +79,13 @@ python gradio_demo.py \
|
||||
|
||||
??? abstract "gradio_demo.py"
|
||||
``````py
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/online_serving/qwen2_5_omni/gradio_demo.py"
|
||||
--8<-- "examples/online_serving/qwen2_5_omni/gradio_demo.py"
|
||||
``````
|
||||
??? abstract "openai_chat_completion_client_for_multimodal_generation.py"
|
||||
``````py
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/online_serving/qwen2_5_omni/openai_chat_completion_client_for_multimodal_generation.py"
|
||||
--8<-- "examples/online_serving/qwen2_5_omni/openai_chat_completion_client_for_multimodal_generation.py"
|
||||
``````
|
||||
??? abstract "run_curl_multimodal_generation.sh"
|
||||
``````sh
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/online_serving/qwen2_5_omni/run_curl_multimodal_generation.sh"
|
||||
--8<-- "examples/online_serving/qwen2_5_omni/run_curl_multimodal_generation.sh"
|
||||
``````
|
||||
@@ -5,7 +5,7 @@ Source <https://github.com/vllm-project/vllm/tree/main/examples/online_serving/q
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm/tree/main/examples/README.md)
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm/tree/main/README.md)
|
||||
|
||||
## Run examples (Qwen3-Omni)
|
||||
|
||||
@@ -21,7 +21,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --stage-configs-p
|
||||
|
||||
Get into the example folder
|
||||
```bash
|
||||
cd examples/online_serving
|
||||
cd examples/online_serving/qwen3_omni
|
||||
```
|
||||
|
||||
Send request via python
|
||||
@@ -83,13 +83,13 @@ python gradio_demo.py \
|
||||
|
||||
??? abstract "gradio_demo.py"
|
||||
``````py
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/online_serving/qwen3_omni/gradio_demo.py"
|
||||
--8<-- "examples/online_serving/qwen3_omni/gradio_demo.py"
|
||||
``````
|
||||
??? abstract "openai_chat_completion_client_for_multimodal_generation.py"
|
||||
``````py
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/online_serving/qwen3_omni/openai_chat_completion_client_for_multimodal_generation.py"
|
||||
--8<-- "examples/online_serving/qwen3_omni/openai_chat_completion_client_for_multimodal_generation.py"
|
||||
``````
|
||||
??? abstract "run_curl_multimodal_generation.sh"
|
||||
``````sh
|
||||
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/online_serving/qwen3_omni/run_curl_multimodal_generation.sh"
|
||||
--8<-- "examples/online_serving/qwen3_omni/run_curl_multimodal_generation.sh"
|
||||
``````
|
||||
@@ -1,94 +0,0 @@
|
||||
# Stage config for running Qwen3-Omni-MoE with 3-stage architecture
|
||||
# Stage 0: Thinker (multimodal understanding + text generation)
|
||||
# Stage 1: Talker (text embeddings → 8-layer RVQ codec codes)
|
||||
# Stage 2: Code2Wav (8-layer RVQ codes → audio waveform)
|
||||
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
runtime:
|
||||
devices: "0,1"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: thinker
|
||||
model_arch: Qwen3OmniMoeForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler
|
||||
gpu_memory_utilization: 0.8
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent # Output hidden states for talker
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
hf_config_name: thinker_config
|
||||
tensor_parallel_size: 2
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
is_comprehension: true
|
||||
default_sampling_params:
|
||||
temperature: 0.4
|
||||
top_p: 0.9
|
||||
top_k: 1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.05
|
||||
|
||||
- stage_id: 1
|
||||
runtime:
|
||||
devices: "1"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: talker
|
||||
model_arch: Qwen3OmniMoeForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler
|
||||
gpu_memory_utilization: 0.5
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent # Output codec codes for code2wav
|
||||
# tensor_parallel_size: 2
|
||||
enable_prefix_caching: false
|
||||
distributed_executor_backend: "mp"
|
||||
hf_config_name: talker_config
|
||||
engine_input_source: [0]
|
||||
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker
|
||||
# final_output: true
|
||||
# final_output_type: text
|
||||
default_sampling_params:
|
||||
temperature: 0.9
|
||||
top_k: 50
|
||||
max_tokens: 4096
|
||||
seed: 42
|
||||
detokenize: False
|
||||
repetition_penalty: 1.05
|
||||
stop_token_ids: [2150]
|
||||
|
||||
- stage_id: 2
|
||||
runtime:
|
||||
devices: "1"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: code2wav
|
||||
model_arch: Qwen3OmniMoeForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_generation_worker.GPUGenerationWorker
|
||||
scheduler_cls: vllm_omni.core.sched.generation_scheduler.GenerationScheduler
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
enable_prefix_caching: false
|
||||
engine_output_type: audio # Final output: audio waveform
|
||||
gpu_memory_utilization: 0.1
|
||||
distributed_executor_backend: "mp"
|
||||
max_num_batched_tokens: 1000000
|
||||
hf_config_name: thinker_config
|
||||
engine_input_source: [1]
|
||||
custom_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav
|
||||
final_output: true
|
||||
final_output_type: audio
|
||||
default_sampling_params:
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: -1
|
||||
max_tokens: 65536
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.1
|
||||
@@ -1,6 +1,5 @@
|
||||
python end2end.py --output-wav output_audio \
|
||||
--query-type use_audio \
|
||||
--init-sleep-seconds 90 \
|
||||
--stage-configs-path qwen3_omni_moe_tp.yaml
|
||||
--init-sleep-seconds 90
|
||||
|
||||
# init-sleep-seconds works to avoid two vLLM stages initialized at the same time within a card.
|
||||
@@ -5,7 +5,7 @@ import gradio as gr
|
||||
import torch
|
||||
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.utils.platform_utils import detect_device_type
|
||||
from vllm_omni.utils.platform_utils import detect_device_type, is_npu
|
||||
|
||||
ASPECT_RATIOS: dict[str, tuple[int, int]] = {
|
||||
"1:1": (1328, 1328),
|
||||
@@ -59,7 +59,14 @@ def parse_args() -> argparse.Namespace:
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_omni(model_name: str) -> Omni:
|
||||
return Omni(model=model_name)
|
||||
# Enable VAE memory optimizations on NPU
|
||||
vae_use_slicing = is_npu()
|
||||
vae_use_tiling = is_npu()
|
||||
return Omni(
|
||||
model=model_name,
|
||||
vae_use_slicing=vae_use_slicing,
|
||||
vae_use_tiling=vae_use_tiling,
|
||||
)
|
||||
|
||||
|
||||
def build_demo(args: argparse.Namespace) -> gr.Blocks:
|
||||
|
||||
@@ -7,7 +7,7 @@ from pathlib import Path
|
||||
import torch
|
||||
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.utils.platform_utils import detect_device_type
|
||||
from vllm_omni.utils.platform_utils import detect_device_type, is_npu
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
@@ -49,7 +49,15 @@ def main():
|
||||
device = detect_device_type()
|
||||
generator = torch.Generator(device=device).manual_seed(args.seed)
|
||||
|
||||
omni = Omni(model=args.model)
|
||||
# Enable VAE memory optimizations on NPU
|
||||
vae_use_slicing = is_npu()
|
||||
vae_use_tiling = is_npu()
|
||||
|
||||
omni = Omni(
|
||||
model=args.model,
|
||||
vae_use_slicing=vae_use_slicing,
|
||||
vae_use_tiling=vae_use_tiling,
|
||||
)
|
||||
images = omni.generate(
|
||||
args.prompt,
|
||||
height=args.height,
|
||||
|
||||
@@ -6,7 +6,8 @@ Please refer to [README.md](../../../README.md)
|
||||
|
||||
## Run examples (Qwen2.5-Omni)
|
||||
|
||||
Launch the server
|
||||
### Launch the Server
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
|
||||
```
|
||||
@@ -16,29 +17,59 @@ If you have custom stage configs file, launch the server with command below
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
|
||||
```
|
||||
|
||||
### Send Multi-modal Request
|
||||
|
||||
Get into the example folder
|
||||
```bash
|
||||
cd examples/online_serving/qwen2_5_omni
|
||||
```
|
||||
|
||||
Send request via python
|
||||
#### Send request via python
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py --query-type mixed_modalities
|
||||
```
|
||||
|
||||
Send request via curl
|
||||
The Python client supports the following command-line arguments:
|
||||
|
||||
- `--query-type` (or `-q`): Query type (default: `mixed_modalities`)
|
||||
- Options: `mixed_modalities`, `use_audio_in_video`, `multi_audios`, `text`
|
||||
- `--video-path` (or `-v`): Path to local video file or URL
|
||||
- If not provided and query-type uses video, uses default video URL
|
||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
||||
- Example: `--video-path /path/to/video.mp4` or `--video-path https://example.com/video.mp4`
|
||||
- `--image-path` (or `-i`): Path to local image file or URL
|
||||
- If not provided and query-type uses image, uses default image URL
|
||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
||||
- Supports common image formats: JPEG, PNG, GIF, WebP
|
||||
- Example: `--image-path /path/to/image.jpg` or `--image-path https://example.com/image.png`
|
||||
- `--audio-path` (or `-a`): Path to local audio file or URL
|
||||
- If not provided and query-type uses audio, uses default audio URL
|
||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
||||
- Supports common audio formats: MP3, WAV, OGG, FLAC, M4A
|
||||
- Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
|
||||
- `--prompt` (or `-p`): Custom text prompt/question
|
||||
- If not provided, uses default prompt for the selected query type
|
||||
- Example: `--prompt "What are the main activities shown in this video?"`
|
||||
|
||||
|
||||
For example, to use mixed modalities with all local files:
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type mixed_modalities \
|
||||
--video-path /path/to/your/video.mp4 \
|
||||
--image-path /path/to/your/image.jpg \
|
||||
--audio-path /path/to/your/audio.wav \
|
||||
--prompt "Analyze all the media content and provide a comprehensive summary."
|
||||
```
|
||||
|
||||
#### Send request via curl
|
||||
|
||||
```bash
|
||||
bash run_curl_multimodal_generation.sh mixed_modalities
|
||||
```
|
||||
|
||||
### FAQ
|
||||
|
||||
If you encounter error about backend of librosa, try to install ffmpeg with command below.
|
||||
```
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
## Run Local Web UI Demo
|
||||
|
||||
This Web UI demo allows users to interact with the model through a web browser.
|
||||
@@ -53,21 +84,18 @@ python gradio_demo.py --model Qwen/Qwen2.5-Omni-7B --port 7861
|
||||
|
||||
Then open `http://localhost:7861/` on your local browser to interact with the web UI.
|
||||
|
||||
The gradio script supports the following arguments:
|
||||
|
||||
### Options
|
||||
- `--model`: Model name
|
||||
- `--ip`: Host/IP for Gradio server (default: 127.0.0.1)
|
||||
- `--port`: Port for Gradio server (default: 7861)
|
||||
- `--stage-configs-path`: Path to custom stage configs YAML file (optional)
|
||||
- `--share`: Share the Gradio demo publicly (creates a public link)
|
||||
|
||||
You can customize its basic launch parameters:
|
||||
### FAQ
|
||||
|
||||
```bash
|
||||
python gradio_demo.py \
|
||||
--model Qwen/Qwen2.5-Omni-7B \
|
||||
--ip 127.0.0.1 \
|
||||
--port 7861 \
|
||||
--stage-configs-path /path/to/stage_configs.yaml
|
||||
If you encounter error about backend of librosa, try to install ffmpeg with command below.
|
||||
```
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
- `--model`: Local model checkpoint to load (default `Qwen/Qwen2.5-Omni-7B`).
|
||||
- `--ip`: Host/IP for the Gradio server (default `127.0.0.1`).
|
||||
- `--port`: Port for the Gradio server (default `7861`).
|
||||
- `--stage-configs-path`: Optional path to custom stage configs YAML.
|
||||
- `--share`: Set to expose a temporary public link via Gradio.
|
||||
+201
-18
@@ -1,4 +1,6 @@
|
||||
import base64
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
from openai import OpenAI
|
||||
@@ -28,6 +30,129 @@ def encode_base64_content_from_url(content_url: str) -> str:
|
||||
return result
|
||||
|
||||
|
||||
def encode_base64_content_from_file(file_path: str) -> str:
|
||||
"""Encode a local file to base64 format."""
|
||||
with open(file_path, "rb") as f:
|
||||
content = f.read()
|
||||
result = base64.b64encode(content).decode("utf-8")
|
||||
return result
|
||||
|
||||
|
||||
def get_video_url_from_path(video_path: Optional[str]) -> str:
|
||||
"""Convert a video path (local file or URL) to a video URL format for the API.
|
||||
|
||||
If video_path is None or empty, returns the default URL.
|
||||
If video_path is a local file path, encodes it to base64 data URL.
|
||||
If video_path is a URL, returns it as-is.
|
||||
"""
|
||||
if not video_path:
|
||||
# Default video URL
|
||||
return "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4"
|
||||
|
||||
# Check if it's a URL (starts with http:// or https://)
|
||||
if video_path.startswith(("http://", "https://")):
|
||||
return video_path
|
||||
|
||||
# Otherwise, treat it as a local file path
|
||||
if not os.path.exists(video_path):
|
||||
raise FileNotFoundError(f"Video file not found: {video_path}")
|
||||
|
||||
# Detect video MIME type from file extension
|
||||
video_path_lower = video_path.lower()
|
||||
if video_path_lower.endswith(".mp4"):
|
||||
mime_type = "video/mp4"
|
||||
elif video_path_lower.endswith(".webm"):
|
||||
mime_type = "video/webm"
|
||||
elif video_path_lower.endswith(".mov"):
|
||||
mime_type = "video/quicktime"
|
||||
elif video_path_lower.endswith(".avi"):
|
||||
mime_type = "video/x-msvideo"
|
||||
elif video_path_lower.endswith(".mkv"):
|
||||
mime_type = "video/x-matroska"
|
||||
else:
|
||||
# Default to mp4 if extension is unknown
|
||||
mime_type = "video/mp4"
|
||||
|
||||
video_base64 = encode_base64_content_from_file(video_path)
|
||||
return f"data:{mime_type};base64,{video_base64}"
|
||||
|
||||
|
||||
def get_image_url_from_path(image_path: Optional[str]) -> str:
|
||||
"""Convert an image path (local file or URL) to an image URL format for the API.
|
||||
|
||||
If image_path is None or empty, returns the default URL.
|
||||
If image_path is a local file path, encodes it to base64 data URL.
|
||||
If image_path is a URL, returns it as-is.
|
||||
"""
|
||||
if not image_path:
|
||||
# Default image URL
|
||||
return "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg"
|
||||
|
||||
# Check if it's a URL (starts with http:// or https://)
|
||||
if image_path.startswith(("http://", "https://")):
|
||||
return image_path
|
||||
|
||||
# Otherwise, treat it as a local file path
|
||||
if not os.path.exists(image_path):
|
||||
raise FileNotFoundError(f"Image file not found: {image_path}")
|
||||
|
||||
# Detect image MIME type from file extension
|
||||
image_path_lower = image_path.lower()
|
||||
if image_path_lower.endswith((".jpg", ".jpeg")):
|
||||
mime_type = "image/jpeg"
|
||||
elif image_path_lower.endswith(".png"):
|
||||
mime_type = "image/png"
|
||||
elif image_path_lower.endswith(".gif"):
|
||||
mime_type = "image/gif"
|
||||
elif image_path_lower.endswith(".webp"):
|
||||
mime_type = "image/webp"
|
||||
else:
|
||||
# Default to jpeg if extension is unknown
|
||||
mime_type = "image/jpeg"
|
||||
|
||||
image_base64 = encode_base64_content_from_file(image_path)
|
||||
return f"data:{mime_type};base64,{image_base64}"
|
||||
|
||||
|
||||
def get_audio_url_from_path(audio_path: Optional[str]) -> str:
|
||||
"""Convert an audio path (local file or URL) to an audio URL format for the API.
|
||||
|
||||
If audio_path is None or empty, returns the default URL.
|
||||
If audio_path is a local file path, encodes it to base64 data URL.
|
||||
If audio_path is a URL, returns it as-is.
|
||||
"""
|
||||
if not audio_path:
|
||||
# Default audio URL
|
||||
return AudioAsset("mary_had_lamb").url
|
||||
|
||||
# Check if it's a URL (starts with http:// or https://)
|
||||
if audio_path.startswith(("http://", "https://")):
|
||||
return audio_path
|
||||
|
||||
# Otherwise, treat it as a local file path
|
||||
if not os.path.exists(audio_path):
|
||||
raise FileNotFoundError(f"Audio file not found: {audio_path}")
|
||||
|
||||
# Detect audio MIME type from file extension
|
||||
audio_path_lower = audio_path.lower()
|
||||
if audio_path_lower.endswith((".mp3", ".mpeg")):
|
||||
mime_type = "audio/mpeg"
|
||||
elif audio_path_lower.endswith(".wav"):
|
||||
mime_type = "audio/wav"
|
||||
elif audio_path_lower.endswith(".ogg"):
|
||||
mime_type = "audio/ogg"
|
||||
elif audio_path_lower.endswith(".flac"):
|
||||
mime_type = "audio/flac"
|
||||
elif audio_path_lower.endswith(".m4a"):
|
||||
mime_type = "audio/mp4"
|
||||
else:
|
||||
# Default to wav if extension is unknown
|
||||
mime_type = "audio/wav"
|
||||
|
||||
audio_base64 = encode_base64_content_from_file(audio_path)
|
||||
return f"data:{mime_type};base64,{audio_base64}"
|
||||
|
||||
|
||||
def get_system_prompt():
|
||||
return {
|
||||
"role": "system",
|
||||
@@ -44,8 +169,10 @@ def get_system_prompt():
|
||||
}
|
||||
|
||||
|
||||
def get_text_query():
|
||||
question = "Explain the system architecture for a scalable audio generation pipeline. Answer in 15 words."
|
||||
def get_text_query(custom_prompt: Optional[str] = None):
|
||||
question = (
|
||||
custom_prompt or "Explain the system architecture for a scalable audio generation pipeline. Answer in 15 words."
|
||||
)
|
||||
prompt = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
@@ -58,26 +185,32 @@ def get_text_query():
|
||||
return prompt
|
||||
|
||||
|
||||
def get_mixed_modalities_query():
|
||||
question = "What is recited in the audio? What is the content of this image? Why is this video funny?"
|
||||
def get_mixed_modalities_query(
|
||||
video_path: Optional[str] = None,
|
||||
image_path: Optional[str] = None,
|
||||
audio_path: Optional[str] = None,
|
||||
custom_prompt: Optional[str] = None,
|
||||
):
|
||||
question = (
|
||||
custom_prompt or "What is recited in the audio? What is the content of this image? Why is this video funny?"
|
||||
)
|
||||
video_url = get_video_url_from_path(video_path)
|
||||
image_url = get_image_url_from_path(image_path)
|
||||
audio_url = get_audio_url_from_path(audio_path)
|
||||
prompt = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {"url": AudioAsset("mary_had_lamb").url},
|
||||
"audio_url": {"url": audio_url},
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg"
|
||||
},
|
||||
"image_url": {"url": image_url},
|
||||
},
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4"
|
||||
},
|
||||
"video_url": {"url": video_url},
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
@@ -89,8 +222,9 @@ def get_mixed_modalities_query():
|
||||
return prompt
|
||||
|
||||
|
||||
def get_use_audio_in_video_query():
|
||||
question = "Describe the content of the video, then convert what the baby say into text."
|
||||
def get_use_audio_in_video_query(video_path: Optional[str] = None, custom_prompt: Optional[str] = None):
|
||||
question = custom_prompt or "Describe the content of the video, then convert what the baby say into text."
|
||||
video_url = get_video_url_from_path(video_path)
|
||||
|
||||
prompt = {
|
||||
"role": "user",
|
||||
@@ -98,7 +232,7 @@ def get_use_audio_in_video_query():
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4",
|
||||
"url": video_url,
|
||||
"num_frames": 16,
|
||||
},
|
||||
},
|
||||
@@ -112,14 +246,15 @@ def get_use_audio_in_video_query():
|
||||
return prompt
|
||||
|
||||
|
||||
def get_multi_audios_query():
|
||||
question = "Are these two audio clips the same?"
|
||||
def get_multi_audios_query(audio_path: Optional[str] = None, custom_prompt: Optional[str] = None):
|
||||
question = custom_prompt or "Are these two audio clips the same?"
|
||||
audio_url = get_audio_url_from_path(audio_path)
|
||||
prompt = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {"url": AudioAsset("mary_had_lamb").url},
|
||||
"audio_url": {"url": audio_url},
|
||||
},
|
||||
{
|
||||
"type": "audio_url",
|
||||
@@ -179,7 +314,27 @@ def run_multimodal_generation(args) -> None:
|
||||
code2wav_sampling_params,
|
||||
]
|
||||
|
||||
prompt = query_map[args.query_type]()
|
||||
# Get paths and custom prompt from args
|
||||
video_path = getattr(args, "video_path", None)
|
||||
image_path = getattr(args, "image_path", None)
|
||||
audio_path = getattr(args, "audio_path", None)
|
||||
custom_prompt = getattr(args, "prompt", None)
|
||||
|
||||
# Get the query function and call it with appropriate parameters
|
||||
query_func = query_map[args.query_type]
|
||||
if args.query_type == "mixed_modalities":
|
||||
prompt = query_func(
|
||||
video_path=video_path, image_path=image_path, audio_path=audio_path, custom_prompt=custom_prompt
|
||||
)
|
||||
elif args.query_type == "use_audio_in_video":
|
||||
prompt = query_func(video_path=video_path, custom_prompt=custom_prompt)
|
||||
elif args.query_type == "multi_audios":
|
||||
prompt = query_func(audio_path=audio_path, custom_prompt=custom_prompt)
|
||||
elif args.query_type == "text":
|
||||
prompt = query_func(custom_prompt=custom_prompt)
|
||||
else:
|
||||
prompt = query_func()
|
||||
|
||||
extra_body = {
|
||||
"sampling_params_list": sampling_params_list # Optional, it has a default setting in stage_configs of the corresponding model.
|
||||
}
|
||||
@@ -219,6 +374,34 @@ def parse_args():
|
||||
choices=query_map.keys(),
|
||||
help="Query type.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--video-path",
|
||||
"-v",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to local video file or URL. If not provided and query-type uses video, uses default video URL.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--image-path",
|
||||
"-i",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to local image file or URL. If not provided and query-type uses image, uses default image URL.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--audio-path",
|
||||
"-a",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to local audio file or URL. If not provided and query-type uses audio, uses default audio URL.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt",
|
||||
"-p",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Custom text prompt/question to use instead of the default prompt for the selected query type.",
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
@@ -6,7 +6,8 @@ Please refer to [README.md](../../../README.md)
|
||||
|
||||
## Run examples (Qwen3-Omni)
|
||||
|
||||
Launch the server
|
||||
### Launch the Server
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
@@ -16,21 +17,59 @@ If you have custom stage configs file, launch the server with command below
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
|
||||
```
|
||||
|
||||
### Send Multi-modal Request
|
||||
|
||||
Get into the example folder
|
||||
```bash
|
||||
cd examples/online_serving/qwen3_omni
|
||||
```
|
||||
|
||||
Send request via python
|
||||
#### Send request via python
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py --query-type use_image
|
||||
```
|
||||
|
||||
Send request via curl
|
||||
The Python client supports the following command-line arguments:
|
||||
|
||||
- `--query-type` (or `-q`): Query type (default: `use_video`)
|
||||
- Options: `text`, `use_audio`, `use_image`, `use_video`
|
||||
- `--model` (or `-m`): Model name/path (default: `Qwen/Qwen3-Omni-30B-A3B-Instruct`)
|
||||
- `--video-path` (or `-v`): Path to local video file or URL
|
||||
- If not provided and query-type is `use_video`, uses default video URL
|
||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
||||
- Example: `--video-path /path/to/video.mp4` or `--video-path https://example.com/video.mp4`
|
||||
- `--image-path` (or `-i`): Path to local image file or URL
|
||||
- If not provided and query-type is `use_image`, uses default image URL
|
||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
||||
- Supports common image formats: JPEG, PNG, GIF, WebP
|
||||
- Example: `--image-path /path/to/image.jpg` or `--image-path https://example.com/image.png`
|
||||
- `--audio-path` (or `-a`): Path to local audio file or URL
|
||||
- If not provided and query-type is `use_audio`, uses default audio URL
|
||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
||||
- Supports common audio formats: MP3, WAV, OGG, FLAC, M4A
|
||||
- Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
|
||||
- `--prompt` (or `-p`): Custom text prompt/question
|
||||
- If not provided, uses default prompt for the selected query type
|
||||
- Example: `--prompt "What are the main activities shown in this video?"`
|
||||
|
||||
|
||||
For example, to use a local video file with custom prompt:
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type use_video \
|
||||
--video-path /path/to/your/video.mp4 \
|
||||
--prompt "What are the main activities shown in this video?"
|
||||
```
|
||||
|
||||
#### Send request via curl
|
||||
|
||||
```bash
|
||||
bash run_curl_multimodal_generation.sh use_image
|
||||
```
|
||||
|
||||
|
||||
### FAQ
|
||||
|
||||
If you encounter error about backend of librosa, try to install ffmpeg with command below.
|
||||
@@ -54,23 +93,9 @@ python gradio_demo.py --model Qwen/Qwen3-Omni-30B-A3B-Instruct --port 7861
|
||||
Then open `http://localhost:7861/` on your local browser to interact with the web UI.
|
||||
|
||||
|
||||
### Options
|
||||
|
||||
The gradio demo also supports running with an existing API server and can be customized with the following arguments.
|
||||
|
||||
|
||||
```bash
|
||||
python gradio_demo.py \
|
||||
--model Qwen/Qwen3-Omni-30B-A3B-Instruct \
|
||||
--use-api-server \
|
||||
--api-base http://localhost:8091/v1 \
|
||||
--ip 127.0.0.1 \
|
||||
--port 7861
|
||||
```
|
||||
The gradio script supports the following arguments:
|
||||
|
||||
- `--model`: Model name
|
||||
- `--use-api-server`: If set, connect to an existing vLLM HTTP API server instead of running AsyncOmni locally.
|
||||
- `--api-base`: Base URL for vllm serve (only used when `use-api-server` is set, default: http://localhost:8091/v1)
|
||||
- `--ip`: Host/IP for Gradio server (default: 127.0.0.1)
|
||||
- `--port`: Port for Gradio server (default: 7861)
|
||||
- `--stage-configs-path`: Path to custom stage configs YAML file (optional)
|
||||
|
||||
@@ -264,7 +264,7 @@ async def run_inference_async_omni(
|
||||
# Process audio
|
||||
audio_data = process_audio_file(audio_file)
|
||||
if audio_data is not None:
|
||||
multi_modal_data["audio"] = audio_data
|
||||
multi_modal_data["audio"] = [audio_data] if not isinstance(audio_data, list) else audio_data
|
||||
|
||||
# Process image
|
||||
image_data = process_image_file(image_file)
|
||||
@@ -282,7 +282,9 @@ async def run_inference_async_omni(
|
||||
video_entry = video_frames
|
||||
multi_modal_data["video"] = video_entry
|
||||
if use_audio_in_video and extracted_audio is not None and "audio" not in multi_modal_data:
|
||||
multi_modal_data["audio"] = extracted_audio
|
||||
multi_modal_data["audio"] = (
|
||||
[extracted_audio] if not isinstance(extracted_audio, list) else extracted_audio
|
||||
)
|
||||
mm_processor_kwargs["use_audio_in_video"] = True
|
||||
|
||||
# Build the prompt input
|
||||
|
||||
+188
-17
@@ -1,5 +1,6 @@
|
||||
import base64
|
||||
from typing import NamedTuple
|
||||
import os
|
||||
from typing import NamedTuple, Optional
|
||||
|
||||
import requests
|
||||
from openai import OpenAI
|
||||
@@ -34,6 +35,129 @@ def encode_base64_content_from_url(content_url: str) -> str:
|
||||
return result
|
||||
|
||||
|
||||
def encode_base64_content_from_file(file_path: str) -> str:
|
||||
"""Encode a local file to base64 format."""
|
||||
with open(file_path, "rb") as f:
|
||||
content = f.read()
|
||||
result = base64.b64encode(content).decode("utf-8")
|
||||
return result
|
||||
|
||||
|
||||
def get_video_url_from_path(video_path: Optional[str]) -> str:
|
||||
"""Convert a video path (local file or URL) to a video URL format for the API.
|
||||
|
||||
If video_path is None or empty, returns the default URL.
|
||||
If video_path is a local file path, encodes it to base64 data URL.
|
||||
If video_path is a URL, returns it as-is.
|
||||
"""
|
||||
if not video_path:
|
||||
# Default video URL
|
||||
return "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4"
|
||||
|
||||
# Check if it's a URL (starts with http:// or https://)
|
||||
if video_path.startswith(("http://", "https://")):
|
||||
return video_path
|
||||
|
||||
# Otherwise, treat it as a local file path
|
||||
if not os.path.exists(video_path):
|
||||
raise FileNotFoundError(f"Video file not found: {video_path}")
|
||||
|
||||
# Detect video MIME type from file extension
|
||||
video_path_lower = video_path.lower()
|
||||
if video_path_lower.endswith(".mp4"):
|
||||
mime_type = "video/mp4"
|
||||
elif video_path_lower.endswith(".webm"):
|
||||
mime_type = "video/webm"
|
||||
elif video_path_lower.endswith(".mov"):
|
||||
mime_type = "video/quicktime"
|
||||
elif video_path_lower.endswith(".avi"):
|
||||
mime_type = "video/x-msvideo"
|
||||
elif video_path_lower.endswith(".mkv"):
|
||||
mime_type = "video/x-matroska"
|
||||
else:
|
||||
# Default to mp4 if extension is unknown
|
||||
mime_type = "video/mp4"
|
||||
|
||||
video_base64 = encode_base64_content_from_file(video_path)
|
||||
return f"data:{mime_type};base64,{video_base64}"
|
||||
|
||||
|
||||
def get_image_url_from_path(image_path: Optional[str]) -> str:
|
||||
"""Convert an image path (local file or URL) to an image URL format for the API.
|
||||
|
||||
If image_path is None or empty, returns the default URL.
|
||||
If image_path is a local file path, encodes it to base64 data URL.
|
||||
If image_path is a URL, returns it as-is.
|
||||
"""
|
||||
if not image_path:
|
||||
# Default image URL
|
||||
return "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg"
|
||||
|
||||
# Check if it's a URL (starts with http:// or https://)
|
||||
if image_path.startswith(("http://", "https://")):
|
||||
return image_path
|
||||
|
||||
# Otherwise, treat it as a local file path
|
||||
if not os.path.exists(image_path):
|
||||
raise FileNotFoundError(f"Image file not found: {image_path}")
|
||||
|
||||
# Detect image MIME type from file extension
|
||||
image_path_lower = image_path.lower()
|
||||
if image_path_lower.endswith((".jpg", ".jpeg")):
|
||||
mime_type = "image/jpeg"
|
||||
elif image_path_lower.endswith(".png"):
|
||||
mime_type = "image/png"
|
||||
elif image_path_lower.endswith(".gif"):
|
||||
mime_type = "image/gif"
|
||||
elif image_path_lower.endswith(".webp"):
|
||||
mime_type = "image/webp"
|
||||
else:
|
||||
# Default to jpeg if extension is unknown
|
||||
mime_type = "image/jpeg"
|
||||
|
||||
image_base64 = encode_base64_content_from_file(image_path)
|
||||
return f"data:{mime_type};base64,{image_base64}"
|
||||
|
||||
|
||||
def get_audio_url_from_path(audio_path: Optional[str]) -> str:
|
||||
"""Convert an audio path (local file or URL) to an audio URL format for the API.
|
||||
|
||||
If audio_path is None or empty, returns the default URL.
|
||||
If audio_path is a local file path, encodes it to base64 data URL.
|
||||
If audio_path is a URL, returns it as-is.
|
||||
"""
|
||||
if not audio_path:
|
||||
# Default audio URL
|
||||
return AudioAsset("mary_had_lamb").url
|
||||
|
||||
# Check if it's a URL (starts with http:// or https://)
|
||||
if audio_path.startswith(("http://", "https://")):
|
||||
return audio_path
|
||||
|
||||
# Otherwise, treat it as a local file path
|
||||
if not os.path.exists(audio_path):
|
||||
raise FileNotFoundError(f"Audio file not found: {audio_path}")
|
||||
|
||||
# Detect audio MIME type from file extension
|
||||
audio_path_lower = audio_path.lower()
|
||||
if audio_path_lower.endswith((".mp3", ".mpeg")):
|
||||
mime_type = "audio/mpeg"
|
||||
elif audio_path_lower.endswith(".wav"):
|
||||
mime_type = "audio/wav"
|
||||
elif audio_path_lower.endswith(".ogg"):
|
||||
mime_type = "audio/ogg"
|
||||
elif audio_path_lower.endswith(".flac"):
|
||||
mime_type = "audio/flac"
|
||||
elif audio_path_lower.endswith(".m4a"):
|
||||
mime_type = "audio/mp4"
|
||||
else:
|
||||
# Default to wav if extension is unknown
|
||||
mime_type = "audio/wav"
|
||||
|
||||
audio_base64 = encode_base64_content_from_file(audio_path)
|
||||
return f"data:{mime_type};base64,{audio_base64}"
|
||||
|
||||
|
||||
def get_system_prompt():
|
||||
return {
|
||||
"role": "system",
|
||||
@@ -50,8 +174,10 @@ def get_system_prompt():
|
||||
}
|
||||
|
||||
|
||||
def get_text_query():
|
||||
question = "Explain the system architecture for a scalable audio generation pipeline. Answer in 15 words."
|
||||
def get_text_query(custom_prompt: Optional[str] = None):
|
||||
question = (
|
||||
custom_prompt or "Explain the system architecture for a scalable audio generation pipeline. Answer in 15 words."
|
||||
)
|
||||
prompt = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
@@ -71,16 +197,15 @@ default_system = (
|
||||
)
|
||||
|
||||
|
||||
def get_video_query() -> QueryResult:
|
||||
question = "Why is this video funny?"
|
||||
def get_video_query(video_path: Optional[str] = None, custom_prompt: Optional[str] = None):
|
||||
question = custom_prompt or "Why is this video funny?"
|
||||
video_url = get_video_url_from_path(video_path)
|
||||
prompt = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4"
|
||||
},
|
||||
"video_url": {"url": video_url},
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
@@ -91,16 +216,15 @@ def get_video_query() -> QueryResult:
|
||||
return prompt
|
||||
|
||||
|
||||
def get_image_query() -> QueryResult:
|
||||
question = "What is the content of this image?"
|
||||
def get_image_query(image_path: Optional[str] = None, custom_prompt: Optional[str] = None):
|
||||
question = custom_prompt or "What is the content of this image?"
|
||||
image_url = get_image_url_from_path(image_path)
|
||||
prompt = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg"
|
||||
},
|
||||
"image_url": {"url": image_url},
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
@@ -111,14 +235,15 @@ def get_image_query() -> QueryResult:
|
||||
return prompt
|
||||
|
||||
|
||||
def get_audio_query() -> QueryResult:
|
||||
question = "What is the content of this audio?"
|
||||
def get_audio_query(audio_path: Optional[str] = None, custom_prompt: Optional[str] = None):
|
||||
question = custom_prompt or "What is the content of this audio?"
|
||||
audio_url = get_audio_url_from_path(audio_path)
|
||||
prompt = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {"url": AudioAsset("mary_had_lamb").url},
|
||||
"audio_url": {"url": audio_url},
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
@@ -178,7 +303,25 @@ def run_multimodal_generation(args) -> None:
|
||||
code2wav_sampling_params,
|
||||
]
|
||||
|
||||
prompt = query_map[args.query_type]()
|
||||
# Get paths and custom prompt from args
|
||||
video_path = getattr(args, "video_path", None)
|
||||
image_path = getattr(args, "image_path", None)
|
||||
audio_path = getattr(args, "audio_path", None)
|
||||
custom_prompt = getattr(args, "prompt", None)
|
||||
|
||||
# Get the query function and call it with appropriate parameters
|
||||
query_func = query_map[args.query_type]
|
||||
if args.query_type == "use_video":
|
||||
prompt = query_func(video_path=video_path, custom_prompt=custom_prompt)
|
||||
elif args.query_type == "use_image":
|
||||
prompt = query_func(image_path=image_path, custom_prompt=custom_prompt)
|
||||
elif args.query_type == "use_audio":
|
||||
prompt = query_func(audio_path=audio_path, custom_prompt=custom_prompt)
|
||||
elif args.query_type == "text":
|
||||
prompt = query_func(custom_prompt=custom_prompt)
|
||||
else:
|
||||
prompt = query_func()
|
||||
|
||||
extra_body = {
|
||||
"sampling_params_list": sampling_params_list # Optional, it has a default setting in stage_configs of the corresponding model.
|
||||
}
|
||||
@@ -225,6 +368,34 @@ def parse_args():
|
||||
default="Qwen/Qwen3-Omni-30B-A3B-Instruct",
|
||||
help="Model Name / Path",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--video-path",
|
||||
"-v",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to local video file or URL. If not provided and query-type is 'use_video', uses default video URL.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--image-path",
|
||||
"-i",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to local image file or URL. If not provided and query-type is 'use_image', uses default image URL.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--audio-path",
|
||||
"-a",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to local audio file or URL. If not provided and query-type is 'use_audio', uses default audio URL.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--prompt",
|
||||
"-p",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Custom text prompt/question to use instead of the default prompt for the selected query type.",
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
@@ -41,7 +41,7 @@ code2wav_sampling_params='{
|
||||
"temperature": 0.0,
|
||||
"top_p": 1.0,
|
||||
"top_k": -1,
|
||||
"max_tokens": 4096 * 16,
|
||||
"max_tokens": 65536,
|
||||
"seed": 42,
|
||||
"detokenize": true,
|
||||
"repetition_penalty": 1.1
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
site_name: vLLM-Omni
|
||||
site_description: Multi-modality models inference and serving with non-autoregressive structures
|
||||
site_description: Efficient omni-modality model serving for everyone
|
||||
site_author: vLLM-Omni Team
|
||||
site_url: https://vllm-project.github.io/vllm-omni/
|
||||
|
||||
|
||||
+3
-2
@@ -5,12 +5,12 @@ build-backend = "setuptools.build_meta"
|
||||
[project]
|
||||
name = "vllm-omni"
|
||||
version = "0.11.0rc1"
|
||||
description = "vLLM-Omni: Multi-modality models inference and serving with non-autoregressive structures"
|
||||
description = "A framework for efficient model inference with omni-modality models"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.9,<3.14" # Align with vLLM v0.11 supported Python versions
|
||||
license = {text = "Apache-2.0"}
|
||||
authors = [
|
||||
{name = "vLLM-Omni Team", email = "vllm-project@gmail.com"}
|
||||
{name = "vLLM-Omni Team"}
|
||||
]
|
||||
keywords = ["vllm", "multimodal", "diffusion", "transformer", "inference", "serving"]
|
||||
classifiers = [
|
||||
@@ -27,6 +27,7 @@ classifiers = [
|
||||
"Topic :: Software Development :: Libraries :: Python Modules",
|
||||
]
|
||||
|
||||
|
||||
dependencies = [
|
||||
# Core runtime dependencies (required for actual usage)
|
||||
"omegaconf>=2.3.0",
|
||||
|
||||
@@ -80,6 +80,10 @@ class OmniDiffusionConfig:
|
||||
vae_cpu_offload: bool = True
|
||||
pin_cpu_memory: bool = True
|
||||
|
||||
# VAE memory optimization parameters
|
||||
vae_use_slicing: bool = False
|
||||
vae_use_tiling: bool = False
|
||||
|
||||
# STA (Sliding Tile Attention) parameters
|
||||
mask_strategy_file_path: str | None = None
|
||||
# STA_mode: STA_Mode = STA_Mode.STA_INFERENCE
|
||||
|
||||
@@ -24,6 +24,13 @@ def initialize_model(
|
||||
module = importlib.import_module(module_name)
|
||||
model_class = getattr(module, cls_name)
|
||||
model = model_class(od_config=od_config, prefix=mod_relname)
|
||||
|
||||
# Configure VAE memory optimization settings from config
|
||||
if hasattr(model.vae, "use_slicing"):
|
||||
model.vae.use_slicing = od_config.vae_use_slicing
|
||||
if hasattr(model.vae, "use_tiling"):
|
||||
model.vae.use_tiling = od_config.vae_use_tiling
|
||||
|
||||
return model
|
||||
else:
|
||||
raise ValueError(f"Model class {od_config.model_class_name} not found in diffusion model registry.")
|
||||
|
||||
@@ -129,7 +129,7 @@ class WorkerProc:
|
||||
"""
|
||||
Receive requests from broadcast queue
|
||||
"""
|
||||
return self.mq.dequeue()
|
||||
return self.mq.dequeue(indefinite=True)
|
||||
|
||||
# TODO: queueing, cancellation
|
||||
def worker_busy_loop(self) -> None:
|
||||
|
||||
@@ -5,7 +5,8 @@ import os
|
||||
|
||||
import torch
|
||||
import zmq
|
||||
from vllm.config import VllmConfig, set_current_vllm_config
|
||||
from transformers import PretrainedConfig
|
||||
from vllm.config import ModelConfig, VllmConfig, set_current_vllm_config
|
||||
from vllm.distributed.device_communicators.shm_broadcast import MessageQueue
|
||||
from vllm.distributed.parallel_state import (
|
||||
init_distributed_environment,
|
||||
@@ -53,7 +54,12 @@ class NPUWorker:
|
||||
torch.npu.set_device(device)
|
||||
|
||||
# hack
|
||||
vllm_config = VllmConfig()
|
||||
# set hf_config to a fake one to avolid get attr error
|
||||
class _FakePretrainedConfig(PretrainedConfig):
|
||||
def __getattr__(self, name):
|
||||
return "fake"
|
||||
|
||||
vllm_config = VllmConfig(model_config=ModelConfig(hf_config=_FakePretrainedConfig()))
|
||||
vllm_config.parallel_config.tensor_parallel_size = self.od_config.num_gpus
|
||||
set_current_vllm_config(vllm_config)
|
||||
|
||||
@@ -129,7 +135,7 @@ class NPUWorkerProc:
|
||||
"""
|
||||
Receive requests from broadcast queue
|
||||
"""
|
||||
return self.mq.dequeue()
|
||||
return self.mq.dequeue(indefinite=True)
|
||||
|
||||
# TODO: queueing, cancellation
|
||||
def worker_busy_loop(self) -> None:
|
||||
|
||||
@@ -90,7 +90,7 @@ class AsyncOmni(EngineClient):
|
||||
stage_configs_path: Optional[str] = None,
|
||||
log_stats: bool = False,
|
||||
log_file: Optional[str] = None,
|
||||
init_sleep_seconds: int = 20,
|
||||
init_sleep_seconds: int = 30,
|
||||
shm_threshold_bytes: int = 65536,
|
||||
batch_timeout: int = 10,
|
||||
init_timeout: int = 60000,
|
||||
|
||||
@@ -1,16 +1,38 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
import os
|
||||
|
||||
from vllm_omni.diffusion.omni_diffusion import OmniDiffusion
|
||||
from vllm_omni.diffusion.utils.hf_utils import is_diffusion_model
|
||||
from vllm_omni.entrypoints.omni_llm import OmniLLM
|
||||
|
||||
|
||||
def _dummy_snapshot_download(model_id):
|
||||
return model_id
|
||||
|
||||
|
||||
def omni_snapshot_download(model_id) -> str:
|
||||
# TODO: this is just a workaround for quickly use modelscope, we should support
|
||||
# modelscope in weight loading feature instead of using `snapshot_download`
|
||||
if os.environ.get("VLLM_USE_MODELSCOPE", False):
|
||||
from modelscope.hub.snapshot_download import snapshot_download
|
||||
|
||||
return snapshot_download(model_id)
|
||||
else:
|
||||
return _dummy_snapshot_download(model_id)
|
||||
|
||||
|
||||
class Omni:
|
||||
"""Unified entrypoint for both LLM and Diffusion models for better usability."""
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
model = args[0] if args else kwargs.get("model", "")
|
||||
assert model != "", "Null model id detected, please specify a model id."
|
||||
model = omni_snapshot_download(model)
|
||||
if args:
|
||||
args[0] = model
|
||||
elif kwargs.get("model", "") != "":
|
||||
kwargs["model"] = model
|
||||
if is_diffusion_model(model):
|
||||
self.instance: OmniLLM | OmniDiffusion = OmniDiffusion(*args, **kwargs)
|
||||
else:
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
# stage config for running qwen2.5-omni with architecture of OmniLLM.
|
||||
|
||||
# The following config has been verified on 1x H100-80G GPU.
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
runtime:
|
||||
|
||||
@@ -3,24 +3,25 @@
|
||||
# Stage 1: Talker (text embeddings → 8-layer RVQ codec codes)
|
||||
# Stage 2: Code2Wav (8-layer RVQ codes → audio waveform)
|
||||
|
||||
# The following config has been verified on 2x H100-80G GPUs.
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
runtime:
|
||||
devices: "0"
|
||||
devices: "0,1"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: thinker
|
||||
model_arch: Qwen3OmniMoeForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler
|
||||
gpu_memory_utilization: 0.8
|
||||
gpu_memory_utilization: 0.6
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent # Output hidden states for talker
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
hf_config_name: thinker_config
|
||||
# tensor_parallel_size: 2 # Commented out - using single GPU for thinker
|
||||
tensor_parallel_size: 2
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
is_comprehension: true
|
||||
@@ -42,7 +43,7 @@ stage_args:
|
||||
model_arch: Qwen3OmniMoeForConditionalGeneration
|
||||
worker_cls: vllm_omni.worker.gpu_ar_worker.GPUARWorker
|
||||
scheduler_cls: vllm_omni.core.sched.scheduler.OmniScheduler
|
||||
gpu_memory_utilization: 0.5
|
||||
gpu_memory_utilization: 0.3
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent # Output codec codes for code2wav
|
||||
@@ -65,7 +66,7 @@ stage_args:
|
||||
|
||||
- stage_id: 2
|
||||
runtime:
|
||||
devices: "1"
|
||||
devices: "0"
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
model_stage: code2wav
|
||||
|
||||
@@ -15,6 +15,9 @@ from vllm_omni.model_executor.layers.mrope import MRotaryEmbedding
|
||||
from vllm_omni.request import OmniRequest
|
||||
|
||||
for module_name, module in sys.modules.items():
|
||||
# only do patch on module of vllm, pass others
|
||||
if "vllm" not in module_name:
|
||||
continue
|
||||
if hasattr(module, "EngineCoreOutput") and module.EngineCoreOutput == _OriginalEngineCoreOutput:
|
||||
module.EngineCoreOutput = OmniEngineCoreOutput
|
||||
if hasattr(module, "EngineCoreOutputs") and module.EngineCoreOutputs == _OriginalEngineCoreOutputs:
|
||||
|
||||
Reference in new issue
Block a user