1 parent
1ae0c64fe0
commit
51c0303945
11 files changed
+25
-74
No files matched your search
@@ -13,25 +13,18 @@ vLLM-Omni uses `uv` as the environment manager, to create and manage Python envi
|
|||||||
uv venv --python 3.12 --seed
|
uv venv --python 3.12 --seed
|
||||||
source .venv/bin/activate
|
source .venv/bin/activate
|
||||||
```
|
```
|
||||||
### Development Environment for vLLM
|
|
||||||
|
|
||||||
vLLM-Omni is built on top of the stable release 0.12.0 of vLLM. You may install `vllm==0.12.0` directly or install the library from source if you need to check, modify or debug with source code of vLLM.
|
### Development Environment for vLLM and vLLM-Omni
|
||||||
|
|
||||||
The release 0.12.0 of vLLM is based on PyTorch 2.9.0 which requires CUDA 12.9 environment.
|
vLLM-Omni is quickly evolving, please see the [installation guide](../getting_started/installation/README.md) for details. It's recommended to build from source to provide the latest development environment.
|
||||||
|
|
||||||
### Development Environment for vLLM-Omni
|
|
||||||
|
|
||||||
Install vLLM-Omni from source with development dependencies:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/vllm-project/vllm-omni.git
|
|
||||||
cd vllm-omni
|
|
||||||
uv pip install -e ".[dev]"
|
|
||||||
```
|
|
||||||
|
|
||||||
!!! tip
|
!!! tip
|
||||||
vLLM-Omni is compatible with Python versions 3.10 to 3.12. However, we recommend developing with Python 3.12 to minimize the chance of your local environment clashing with our CI environment.
|
vLLM-Omni is compatible with Python versions 3.10 to 3.12. However, we recommend developing with Python 3.12 to minimize the chance of your local environment clashing with our CI environment.
|
||||||
|
|
||||||
|
### Adding a new model to vLLM-Omni
|
||||||
|
|
||||||
|
Please check [model implementation](model/README.md) for how to add diffusion and omni-modality models to vLLM-Omni.
|
||||||
|
|
||||||
### Linting
|
### Linting
|
||||||
|
|
||||||
vLLM-Omni uses `pre-commit` to lint and format the codebase. See [pre-commit documentation](https://pre-commit.com/#usage) if `pre-commit` is new to you. Setting up `pre-commit` is as easy as:
|
vLLM-Omni uses `pre-commit` to lint and format the codebase. See [pre-commit documentation](https://pre-commit.com/#usage) if `pre-commit` is new to you. Setting up `pre-commit` is as easy as:
|
||||||
|
|||||||
@@ -26,4 +26,4 @@ The guide covers:
|
|||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
For a quick reference, see the [Adding a New Omni Model Guide](adding_multi_stage_model.md) and [Adding a New Diffusion Model Guide](adding_diffusion_model.md).
|
For a quick reference, see the [Adding a New Multi-Stage Model Guide](adding_multi_stage_model.md) and [Adding a New Diffusion Model Guide](adding_diffusion_model.md).
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
# Adding a New Diffusion Model to vLLM-Omni
|
# Adding a Diffusion Model
|
||||||
This guide walks through the process of adding a new Diffusion model to vLLM-Omni, using Qwen/Qwen-Image-Edit as a comprehensive example.
|
This guide walks through the process of adding a new Diffusion model to vLLM-Omni, using Qwen/Qwen-Image-Edit as a comprehensive example.
|
||||||
|
|
||||||
# Table of Contents
|
# Table of Contents
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Adding a New Multi-Stage Model to vLLM-Omni
|
# Adding a Multi-Stage Model
|
||||||
|
|
||||||
This guide walks through the process of adding a new model to vLLM-Omni, using **Qwen3-Omni** as a comprehensive example. Qwen3-Omni is a multi-stage omni-modality model that demonstrates the full capabilities of vLLM-Omni's architecture.
|
This guide walks through the process of adding a new multi-stage model to vLLM-Omni, using **Qwen3-Omni** as a comprehensive example. Qwen3-Omni is a multi-stage omni-modality model that demonstrates the full capabilities of vLLM-Omni's architecture.
|
||||||
|
|
||||||
## Table of Contents
|
## Table of Contents
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
# Profiling vLLM-Omni (update soon)
|
# Profiling vLLM-Omni
|
||||||
|
|
||||||
Profiling is only intended for vLLM-Omni developers and maintainers to understand the proportion of time spent in different parts of the codebase. **vLLM-Omni end-users should never turn on profiling** as it will significantly slow down the inference.
|
Profiling is only intended for vLLM-Omni developers and maintainers to understand the proportion of time spent in different parts of the codebase. **vLLM-Omni end-users should never turn on profiling** as it will significantly slow down the inference.
|
||||||
@@ -42,8 +42,10 @@ If you do not need to modify source code of vLLM, you can directly install the s
|
|||||||
uv pip install vllm==0.12.0 --torch-backend=auto
|
uv pip install vllm==0.12.0 --torch-backend=auto
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The release 0.12.0 of vLLM is based on PyTorch 2.9.0 which requires CUDA 12.9 environment.
|
||||||
|
|
||||||
#### Installation of vLLM-Omni
|
#### Installation of vLLM-Omni
|
||||||
Install additional requirements for vLLM-Omni
|
Since vllm-omni is rapidly evolving, it's recommended to install it from source
|
||||||
```bash
|
```bash
|
||||||
git clone https://github.com/vllm-project/vllm-omni.git
|
git clone https://github.com/vllm-project/vllm-omni.git
|
||||||
cd vllm-omni
|
cd vllm-omni
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ cd vllm-omni
|
|||||||
uv pip install -e .
|
uv pip install -e .
|
||||||
```
|
```
|
||||||
|
|
||||||
For additional details—including alternative installation methods, installation on NPU and other platforms — please see the installation guide in [installation](installation/README.md)
|
For additional installation methods — please see the [installation guide](installation/README.md).
|
||||||
|
|
||||||
## Offline Inference
|
## Offline Inference
|
||||||
|
|
||||||
|
|||||||
@@ -70,13 +70,6 @@ If media file paths are not provided, the script will use default assets. Suppor
|
|||||||
- `use_audio_in_video`: Extract audio from video
|
- `use_audio_in_video`: Extract audio from video
|
||||||
- `text`: Text-only query
|
- `text`: Text-only query
|
||||||
|
|
||||||
### FAQ
|
|
||||||
|
|
||||||
If you encounter error about backend of librosa, try to install ffmpeg with command below.
|
|
||||||
```
|
|
||||||
sudo apt update
|
|
||||||
sudo apt install ffmpeg
|
|
||||||
```
|
|
||||||
|
|
||||||
## Example materials
|
## Example materials
|
||||||
|
|
||||||
|
|||||||
@@ -57,13 +57,6 @@ If media file paths are not provided, the script will use default assets. Suppor
|
|||||||
- `use_audio`: Audio input
|
- `use_audio`: Audio input
|
||||||
- `text`: Text-only query
|
- `text`: Text-only query
|
||||||
|
|
||||||
### FAQ
|
|
||||||
|
|
||||||
If you encounter error about backend of librosa, try to install ffmpeg with command below.
|
|
||||||
```
|
|
||||||
sudo apt update
|
|
||||||
sudo apt install ffmpeg
|
|
||||||
```
|
|
||||||
|
|
||||||
## Example materials
|
## Example materials
|
||||||
|
|
||||||
|
|||||||
@@ -35,26 +35,11 @@ python openai_chat_completion_client_for_multimodal_generation.py --query-type m
|
|||||||
|
|
||||||
The Python client supports the following command-line arguments:
|
The Python client supports the following command-line arguments:
|
||||||
|
|
||||||
- `--query-type` (or `-q`): Query type (default: `mixed_modalities`)
|
- `--query-type` (or `-q`): Query type (default: `mixed_modalities`). Options: `mixed_modalities`, `use_audio_in_video`, `multi_audios`, `text`
|
||||||
- Options: `mixed_modalities`, `use_audio_in_video`, `multi_audios`, `text`
|
- `--video-path` (or `-v`): Path to local video file or URL. If not provided and query-type uses video, uses default video URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs. Example: `--video-path /path/to/video.mp4` or `--video-path https://example.com/video.mp4`
|
||||||
- `--video-path` (or `-v`): Path to local video file or URL
|
- `--image-path` (or `-i`): Path to local image file or URL. If not provided and query-type uses image, uses default image URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common image formats: JPEG, PNG, GIF. Example: `--image-path /path/to/image.jpg` or `--image-path https://example.com/image.png`
|
||||||
- If not provided and query-type uses video, uses default video URL
|
- `--audio-path` (or `-a`): Path to local audio file or URL. If not provided and query-type uses audio, uses default audio URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common audio formats: MP3, WAV, OGG, FLAC, M4A. Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
|
||||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
- `--prompt` (or `-p`): Custom text prompt/question. If not provided, uses default prompt for the selected query type. Example: `--prompt "What are the main activities shown in this video?"`
|
||||||
- Example: `--video-path /path/to/video.mp4` or `--video-path https://example.com/video.mp4`
|
|
||||||
- `--image-path` (or `-i`): Path to local image file or URL
|
|
||||||
- If not provided and query-type uses image, uses default image URL
|
|
||||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
|
||||||
- Supports common image formats: JPEG, PNG, GIF, WebP
|
|
||||||
- Example: `--image-path /path/to/image.jpg` or `--image-path https://example.com/image.png`
|
|
||||||
- `--audio-path` (or `-a`): Path to local audio file or URL
|
|
||||||
- If not provided and query-type uses audio, uses default audio URL
|
|
||||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
|
||||||
- Supports common audio formats: MP3, WAV, OGG, FLAC, M4A
|
|
||||||
- Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
|
|
||||||
- `--prompt` (or `-p`): Custom text prompt/question
|
|
||||||
- If not provided, uses default prompt for the selected query type
|
|
||||||
- Example: `--prompt "What are the main activities shown in this video?"`
|
|
||||||
|
|
||||||
|
|
||||||
For example, to use mixed modalities with all local files:
|
For example, to use mixed modalities with all local files:
|
||||||
|
|
||||||
|
|||||||
@@ -35,27 +35,12 @@ python openai_chat_completion_client_for_multimodal_generation.py --query-type u
|
|||||||
|
|
||||||
The Python client supports the following command-line arguments:
|
The Python client supports the following command-line arguments:
|
||||||
|
|
||||||
- `--query-type` (or `-q`): Query type (default: `use_video`)
|
- `--query-type` (or `-q`): Query type (default: `use_video`). Options: `text`, `use_audio`, `use_image`, `use_video`
|
||||||
- Options: `text`, `use_audio`, `use_image`, `use_video`
|
|
||||||
- `--model` (or `-m`): Model name/path (default: `Qwen/Qwen3-Omni-30B-A3B-Instruct`)
|
- `--model` (or `-m`): Model name/path (default: `Qwen/Qwen3-Omni-30B-A3B-Instruct`)
|
||||||
- `--video-path` (or `-v`): Path to local video file or URL
|
- `--video-path` (or `-v`): Path to local video file or URL. If not provided and query-type is `use_video`, uses default video URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs. Example: `--video-path /path/to/video.mp4` or `--video-path https://example.com/video.mp4`
|
||||||
- If not provided and query-type is `use_video`, uses default video URL
|
- `--image-path` (or `-i`): Path to local image file or URL. If not provided and query-type is `use_image`, uses default image URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common image formats: JPEG, PNG, GIF. Example: `--image-path /path/to/image.jpg` or `--image-path https://example.com/image.png`.
|
||||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
- `--audio-path` (or `-a`): Path to local audio file or URL. If not provided and query-type is `use_audio`, uses default audio URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common audio formats: MP3, WAV, OGG, FLAC, M4A. Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
|
||||||
- Example: `--video-path /path/to/video.mp4` or `--video-path https://example.com/video.mp4`
|
- `--prompt` (or `-p`): Custom text prompt/question. If not provided, uses default prompt for the selected query type. Example: `--prompt "What are the main activities shown in this video?"`
|
||||||
- `--image-path` (or `-i`): Path to local image file or URL
|
|
||||||
- If not provided and query-type is `use_image`, uses default image URL
|
|
||||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
|
||||||
- Supports common image formats: JPEG, PNG, GIF, WebP
|
|
||||||
- Example: `--image-path /path/to/image.jpg` or `--image-path https://example.com/image.png`
|
|
||||||
- `--audio-path` (or `-a`): Path to local audio file or URL
|
|
||||||
- If not provided and query-type is `use_audio`, uses default audio URL
|
|
||||||
- Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs
|
|
||||||
- Supports common audio formats: MP3, WAV, OGG, FLAC, M4A
|
|
||||||
- Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
|
|
||||||
- `--prompt` (or `-p`): Custom text prompt/question
|
|
||||||
- If not provided, uses default prompt for the selected query type
|
|
||||||
- Example: `--prompt "What are the main activities shown in this video?"`
|
|
||||||
|
|
||||||
|
|
||||||
For example, to use a local video file with custom prompt:
|
For example, to use a local video file with custom prompt:
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user