anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Bugfix] Remove mandatory flash-attn dependency and optimzie docs (#113)

Browse Source 
Signed-off-by: Gaohan123 <gaohan19@huawei.com>
This commit is contained in:
Gao Han authored and GitHub committed 2025-11-30 18:20:44 +08:00
1 parent 228f4e5070
commit 88c00fb271
29 files changed
+857 -176
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/examples/offline_inference/qwen2_5_omni.md
+38 -34
View File
@@ -1,55 +1,40 @@
# Offline Inference Example of vLLM-Omni for Qwen2.5-Omni
# Offline Example of vLLM-Omni for Qwen2.5-omni
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen2_5_omni>.
Source <https://github.com/vllm-project/vllm/tree/main/examples/offline_inference/qwen2_5_omni>.
## 🛠️ Installation
Please refer to [installation](../../../getting_started/installation/README.md).
## Run Offline inference with Qwen2.5-Omni.
First, navigate to the example folder
```bash
cd examples/offline_inference/qwen2_5_omni
```
Inside the directory, `end2end.py` is a comprehensive demo suite for initializing a model instance of `Qwen/Qwen2.5-Omni-7B` and use it for various offline inference tasks.
??? abstract "end2end.py"
``````py
--8<-- "examples/offline_inference/qwen2_5_omni/end2end.py"
``````
Below we also provide simple bash scripts to execute this file.
### Single Prompt
```bash
bash run_single_prompt.sh
```
??? abstract "run_single_prompt.sh"
``````sh
--8<-- "examples/offline_inference/qwen2_5_omni/run_single_prompt.sh"
``````
Please refer to [README.md](https://github.com/vllm-project/vllm/tree/main/README.md)
## Run examples (Qwen2.5-omni)
### Multiple Prompts
Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit) and extract the prompts with `extract_prompts.py`
??? abstract "extract_prompts.py"
``````py
--8<-- "examples/offline_inference/qwen2_5_omni/extract_prompts.py"
``````
Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). To get the prompt, you can:
```bash
tar -xf <Your Download Path>/seedtts_testset.tar
cp seedtts_testset/en/meta.lst examples/offline_inference/qwen2_5_omni/meta.lst
python3 examples/offline_inference/qwen2_5_omni/extract_prompts.py \
--input examples/offline_inference/qwen2_5_omni/meta.lst \
--output examples/offline_inference/qwen2_5_omni/top100.txt \
--topk 10
--topk 100
```
Get into the example folder
```bash
cd examples/offline_inference/qwen2_5_omni
```
Then run the command below.
```bash
bash run_multiple_prompts.sh
```
??? abstract "run_multiple_prompts.sh"
``````sh
--8<-- "examples/offline_inference/qwen2_5_omni/run_multiple_prompts.sh"
``````
### Single Prompt
Get into the example folder
```bash
cd examples/offline_inference/qwen2_5_omni
```
Then run the command below.
```bash
bash run_single_prompt.sh
```
### FAQ
@@ -58,3 +43,22 @@ If you encounter error about backend of librosa, try to install ffmpeg with comm
sudo apt update
sudo apt install ffmpeg
```
## Example materials
??? abstract "end2end.py"
``````py
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen2_5_omni/end2end.py"
``````
??? abstract "extract_prompts.py"
``````py
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen2_5_omni/extract_prompts.py"
``````
??? abstract "run_multiple_prompts.sh"
``````sh
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen2_5_omni/run_multiple_prompts.sh"
``````
??? abstract "run_single_prompt.sh"
``````sh
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen2_5_omni/run_single_prompt.sh"
``````
docs/user_guide/examples/offline_inference/qwen3_omni.md
+64
View File
@@ -0,0 +1,64 @@
# Offline Example of vLLM-Omni for Qwen3-omni
Source <https://github.com/vllm-project/vllm/tree/main/examples/offline_inference/qwen3_omni>.
## 🛠️ Installation
Please refer to [README.md](https://github.com/vllm-project/vllm/tree/main/README.md)
## Run examples (Qwen3-omni)
### Multiple Prompts
Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). For processing dataset please refer to [Qwen2.5-omni README.md](https://github.com/vllm-project/vllm/tree/main/examples/offline_inference/qwen2_5_omni/README.md)
Get into the example folder
```bash
cd examples/offline_inference/qwen3_omni
```
Then run the command below.
```bash
bash run_multiple_prompts.sh
```
### Single Prompt
Get into the example folder
```bash
cd examples/offline_inference/qwen3_omni
```
Then run the command below.
```bash
bash run_single_prompt.sh
```
If you have not enough memory, you can set thinker with tensor parallel. Just run the command below.
```bash
bash run_single_prompt_tp.sh
```
### FAQ
If you encounter error about backend of librosa, try to install ffmpeg with command below.
```
sudo apt update
sudo apt install ffmpeg
```
## Example materials
??? abstract "end2end.py"
``````py
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/end2end.py"
``````
??? abstract "qwen3_omni_moe_tp.yaml"
``````yaml
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/qwen3_omni_moe_tp.yaml"
``````
??? abstract "run_multiple_prompts.sh"
``````sh
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/run_multiple_prompts.sh"
``````
??? abstract "run_single_prompt.sh"
``````sh
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/run_single_prompt.sh"
``````
??? abstract "run_single_prompt_tp.sh"
``````sh
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen3_omni/run_single_prompt_tp.sh"
``````
docs/user_guide/examples/offline_inference/qwen_image.md
+13 -22
View File
@@ -1,25 +1,15 @@
# Offline Inference Example of vLLM-Omni for Qwen-Image
# Qwen-Image Offline Inference
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen_image>.
Source <https://github.com/vllm-project/vllm/tree/main/examples/offline_inference/qwen_image>.
## 🛠️ Installation
Please refer to [installation](../../../getting_started/installation/README.md).
## Run Offline inference with Qwen-Image
First, navigate to the example folder
```bash
cd examples/offline_inference/qwen_image
```
This folder provides two simple entrypoints for experimenting with `Qwen/Qwen-Image` using vLLM-Omni:
- `text_to_image.py`: command-line script for single image generation.
- `gradio_demo.py`: lightweight Gradio UI for interactive prompt/seed/CFG exploration.
- `web_demo.py`: lightweight Gradio UI for interactive prompt/seed/CFG exploration.
### Command-line Usage
## Local CLI Usage
```bash
python text_to_image.py \
@@ -43,14 +33,9 @@ Key arguments:
- `--height/--width`: output resolution (defaults 1024x1024).
- `--output`: path to save the generated PNG.
??? abstract "text_to_image.py"
``````py
--8<-- "examples/offline_inference/qwen_image/text_to_image.py"
``````
> ℹ️ Qwen-Image currently publishes best-effort presets at `1328x1328`, `1664x928`, `928x1664`, `1472x1140`, `1140x1472`, `1584x1056`, and `1056x1584`. Adjust `--height/--width` accordingly for the most reliable outcomes.
### Web UI Demo
## Web UI Demo
Launch the gradio demo:
@@ -60,7 +45,13 @@ python gradio_demo.py --port 7862
Then open `http://localhost:7862/` on your local browser to interact with the web UI.
## Example materials
??? abstract "gradio_demo.py"
``````py
--8<-- "examples/offline_inference/qwen_image/gradio_demo.py"
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen_image/gradio_demo.py"
``````
??? abstract "text_to_image.py"
``````py
--8<-- "/mnt/vllm_open_release/vllm-omni-cursor/vllm-omni/examples/offline_inference/qwen_image/text_to_image.py"
``````