diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index ff0770175..ef02d5747 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -2,4 +2,4 @@ blank_issues_enabled: false contact_links: - name: Questions url: https://github.com/vllm-project/vllm-omni/discussions - about: Ask questions and discuss with other vLLM-omni community members + about: Ask questions and discuss with other vLLM-Omni community members diff --git a/.gitignore b/.gitignore index 59a839811..e38e2dc74 100644 --- a/.gitignore +++ b/.gitignore @@ -175,7 +175,7 @@ tests/ # *.iws # .idea/ -# vLLM-omni specific +# vLLM-Omni specific # Model files and checkpoints checkpoints/ *.bin diff --git a/README.md b/README.md index c1d840081..6201c4b44 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Easy, fast, and cheap omni-modality model serving for everyone ## About -[vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni extends its support for omni-modality model inference and serving: +[vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving: - **Omni-modality**: Text, image, video, and audio data processing - **Non-autoregressive Architectures**: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models @@ -34,24 +34,24 @@ Easy, fast, and cheap omni-modality model serving for everyone

-vLLM-omni is fast with: +vLLM-Omni is fast with: -- Seamless AR support for efficient KV Cache management with vLLM -- Pipelined stage execution overlapping -- Fully disaggregation based on omniConnector and dynamic resource allocation across stages +- State-of-the-art AR support by leveraging efficient KV cache management from vLLM +- Pipelined stage execution overlapping for high throughput performance +- Fully disaggregation based on OmniConnector and dynamic resource allocation across stages -vLLM-omni is flexible and easy to use with: +vLLM-Omni is flexible and easy to use with: +- Heterogeneous pipeline abstraction to manage complex model workflows - Seamless integration with popular Hugging Face models -- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more - Tensor, pipeline, data and expert parallelism support for distributed inference - Streaming outputs - OpenAI-compatible API server -vLLM seamlessly supports most popular open-source models on HuggingFace, including: +vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including: -- Multi-modality generation models (e.g. Qwen-image) -- Omni-modality models (e.g. Qwen-omni) +- Omni-modality models (e.g. Qwen-Omni) +- Multi-modality generation models (e.g. Qwen-Image) ## Getting Started @@ -66,7 +66,7 @@ Visit our documentation to learn more. ## Contributing We welcome and value any contributions and collaborations. -Please check out [Contributing to vLLM-omni](https://vllm-omni.readthedocs.io/en/latest/contributing/) for how to get involved. +Please check out [Contributing to vLLM-Omni](https://vllm-omni.readthedocs.io/en/latest/contributing/) for how to get involved. ## Weekly Meeting diff --git a/docs/.nav.yml b/docs/.nav.yml index caf2b7736..a9cd65d33 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -8,8 +8,9 @@ nav: - examples/README.md - Offline Inference: - Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md + - Online Serving: + - Qwen2.5-Omni: user_guide/examples/online_serving/qwen2_5_omni.md - Qwen2.5-Image: user_guide/examples/offline_inference/qwen_image.md - - Online Serving: user_guide/examples/online_serving/README.md - General: - usage/* - Configuration: @@ -17,16 +18,20 @@ nav: - configuration/* - Models: - models/supported_models.md - - Features: features - Developer Guide: - - contributing/README.md - General: + - contributing/README.md - glob: contributing/* flatten_single_child_sections: true - Model Implementation: - contributing/model/README.md - CI: contributing/ci - - Design Documents: design + - Design Documents: + - design/index.md + - design/architecture_overview.md + - design/vllm_omni_design.md + - design/mrs_design.md + - design/api_design_doc.md - Docs Guide: contributing/DOCS_GUIDE.md - API Reference: - api/README.md @@ -34,6 +39,6 @@ nav: - CLI Reference: cli - Community: - community/* + - Slack: https://slack.vllm.ai - Blog: https://blog.vllm.ai - Forum: https://discuss.vllm.ai - - Slack: https://slack.vllm.ai diff --git a/docs/README.md b/docs/README.md index f3e0af3b5..27bf83f39 100644 --- a/docs/README.md +++ b/docs/README.md @@ -26,7 +26,7 @@ Easy, fast, and cheap omni-modality model serving for everyone ## About -[vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni extends its support for omni-modality model inference and serving: +[vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving: - **Omni-modality**: Text, image, video, and audio data processing - **Non-autoregressive Architectures**: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models @@ -38,23 +38,24 @@ Easy, fast, and cheap omni-modality model serving for everyone

-vLLM-omni is fast with: +vLLM-Omni is fast with: -- Seamless AR support for efficient KV Cache management with vLLM -- Pipelined stage execution overlapping -- Fully disaggregation based on omniConnector and dynamic resource allocation across stages +- State-of-the-art AR support by leveraging efficient KV cache management from vLLM +- Pipelined stage execution overlapping for high throughput performance +- Fully disaggregation based on OmniConnector and dynamic resource allocation across stages -vLLM-omni is flexible and easy to use with: +vLLM-Omni is flexible and easy to use with: +- Heterogeneous pipeline abstraction to manage complex model workflows - Seamless integration with popular Hugging Face models -- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more - Tensor, pipeline, data and expert parallelism support for distributed inference - Streaming outputs - OpenAI-compatible API server vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including: -- Multi-modality generation models (e.g. Qwen-image) -- Omni-modality models (e.g. Qwen-omni) +- Omni-modality models (e.g. Qwen2.5-Omni, Qwen3-Omni) +- Multi-modality generation models (e.g. Qwen-Image) For more information, checkout the following: +- [Placeholder for vllm-omni blogpost]() diff --git a/docs/api/README.md b/docs/api/README.md index 40844c442..100e17ad4 100644 --- a/docs/api/README.md +++ b/docs/api/README.md @@ -2,7 +2,7 @@ ## Entry Points -Main entry points for vLLM-omni inference and serving. +Main entry points for vLLM-Omni inference and serving. - [vllm_omni.entrypoints.async_omni_llm.AsyncOmniLLM][] - [vllm_omni.entrypoints.async_omni_llm.AsyncOmniStageLLM][] diff --git a/docs/community/contact_us.md b/docs/community/contact_us.md index d1a0fbccf..8edc2b957 100644 --- a/docs/community/contact_us.md +++ b/docs/community/contact_us.md @@ -1,6 +1,5 @@ # Contact Us - For technical questions and feature requests, please use GitHub [Issues](https://github.com/vllm-project/vllm-omni/issues) -- For discussing with fellow users, please use the [vLLM Forum](https://discuss.vllm.ai/) -- For coordinating contributions and development, please use [Slack](https://slack.vllm.ai/) +- For coordinating contributions and development and discussing with other users and developers, please join `sig-omni` channel in our [Slack](https://slack.vllm.ai/) or use the [vLLM Forum](https://discuss.vllm.ai/) - For security disclosures, please use GitHub's [Security Advisories](https://github.com/vllm-project/vllm/security/advisories) feature diff --git a/docs/configuration/stage_configs.md b/docs/configuration/stage_configs.md index 48ceac159..2ed46e93d 100644 --- a/docs/configuration/stage_configs.md +++ b/docs/configuration/stage_configs.md @@ -1,10 +1,10 @@ # Stage configs for vLLM-Omni -In vLLM-Omni, it divides models in vLLM-omni into multiple stages, which are processed by different LLMEngines, DiffusionEngines or other kinds of engines. According to different types of stages, such as AR stage or DiT stage, each stage can choose corresponding schedulers, model workers to load into Engines with a plug-in way. +In vLLM-Omni, the target model is separated into multiple stages, which are processed by different LLMEngines, DiffusionEngines or other types of engines. Depending on different types of stages, such as Autoregressive (AR) stage or Diffusion transformer (DiT) stage, each can choose corresponding schedulers, model workers to load with the Engines in a plug-in fashion. -Therefore, the stage configs for a model have several main functions: +Therefore, as a core part of vLLM-Omni, the stage configs for a model have several main functions: -- Claim partition of stages and corresponding class implementation in the folder model_executor/models. +- Claim partition of stages and their corresponding class implementation in `model_executor/models`. - The disaggregated configuration for each stage and the communication topology among them. - Engine arguments for each engine within the stage. - Input and output dependencies for each stage. @@ -22,6 +22,8 @@ For online serving: ```bash vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file ``` +!!! important + We are actively iterating on the definition of stage configs, and we welcome all feedbacks from both community users and developers to help us shape the development! Below is a specific example of stage_configs.yaml in Qwen2.5-omni. ```python diff --git a/docs/contributing/DOCS_GUIDE.md b/docs/contributing/DOCS_GUIDE.md index f4c4265ab..dcbe2d7b6 100644 --- a/docs/contributing/DOCS_GUIDE.md +++ b/docs/contributing/DOCS_GUIDE.md @@ -1,6 +1,6 @@ # Documentation Build Guide -This directory contains the source files for the vLLM-omni documentation. +This directory contains the source files for the vLLM-Omni documentation. ## Building Documentation Locally @@ -39,7 +39,7 @@ Example docstring: ```python class Omni: - """Main entry point for vLLM-omni inference. + """Main entry point for vLLM-Omni inference. This class provides a high-level interface for running multi-modal inference with non-autoregressive models. diff --git a/docs/contributing/README.md b/docs/contributing/README.md index c27b7f32c..5f69a3170 100644 --- a/docs/contributing/README.md +++ b/docs/contributing/README.md @@ -1,64 +1,42 @@ -# Contributing to vLLM-omni +# Contributing to vLLM-Omni -Thank you for your interest in contributing to vLLM-omni! This document provides guidelines and instructions for contributing. +Thank you for your interest in contributing to vLLM-Omni! This document provides guidelines and instructions for contributing. ## Getting Started -### Development Environment for vLLM - -vLLM-omni uses `uv` as the environment manager, to create and manage Python environments. Please follow the documentation to install `uv`. After installing `uv`, you can create a new Python environment using the following commands: +vLLM-Omni uses `uv` as the environment manager, to create and manage Python environments. Please follow the documentation to install `uv`. After installing `uv`, you can create a new Python environment using the following commands: ```bash uv venv --python 3.12 --seed source .venv/bin/activate ``` +### Development Environment for vLLM -Install vLLM from source: - -```bash -git clone https://github.com/vllm-project/vllm.git -cd vllm -git checkout v0.11.0 -``` -Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set VLLM_PRECOMPILED_WHEEL_LOCATION as your local absolute path of whl file. -```bash -export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.11.0/vllm-0.11.0-cp38-abi3-manylinux1_x86_64.whl -``` -Install vllm with command below (If you have no existing PyTorch). -```bash -uv pip install --editable . -``` - -Install vllm with command below (If you already have PyTorch). -```bash -python use_existing_torch.py -uv pip install -r requirements/build.txt -uv pip install --no-build-isolation --editable . -``` +vLLM-Omni is built on top of the stable release 0.11.0 of vLLM. You may install `vllm==0.11.0` directly or install the library from source if you need to check, modify or debug with source code of vLLM. ### Development Environment for vLLM-Omni -Install vLLM-omni from source: +Install vLLM-Omni from source with development dependencies: ```bash git clone https://github.com/vllm-project/vllm-omni.git cd vllm_omni -uv pip install -e . +uv pip install -e ".[dev]" ``` !!! tip - vLLM-omni is compatible with Python versions 3.10 to 3.12. However, we recommend developing with Python 3.12 to minimize the chance of your local environment clashing with our CI environment. + vLLM-Omni is compatible with Python versions 3.10 to 3.12. However, we recommend developing with Python 3.12 to minimize the chance of your local environment clashing with our CI environment. ### Linting -vLLM-omni uses `pre-commit` to lint and format the codebase. See [pre-commit documentation](https://pre-commit.com/#usage) if `pre-commit` is new to you. Setting up `pre-commit` is as easy as: +vLLM-Omni uses `pre-commit` to lint and format the codebase. See [pre-commit documentation](https://pre-commit.com/#usage) if `pre-commit` is new to you. Setting up `pre-commit` is as easy as: ```bash uv pip install pre-commit pre-commit install ``` -vLLM-omni's `pre-commit` hooks will now run automatically every time you commit. +vLLM-Omni's `pre-commit` hooks will now run automatically every time you commit. !!! tip You can manually run the `pre-commit` hooks using: @@ -94,20 +72,14 @@ For additional features and advanced configurations, refer to the: ### Testing -vLLM-omni uses `pytest` to test the codebase. +vLLM-Omni uses `pytest` to test the codebase. ```bash -# Install the test dependencies used in CI (CUDA only) -uv pip install -e ".[dev]" --torch-backend=auto - -# Install some common test dependencies (hardware agnostic) -uv pip install pytest pytest-asyncio - # Run all tests pytest tests/ # Run tests for a single test file with detailed output -pytest -s -v tests/test_logger.py +pytest -s -v tests/test_omni_llm.py ``` !!! warning @@ -122,7 +94,7 @@ If you encounter a bug or have a feature request, please search existing issues ## Pull Requests & Code Reviews -Thank you for your contribution to vLLM-omni! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM-omni maintain the code quality and improve the efficiency of the review process. +Thank you for your contribution to vLLM-Omni! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM-Omni maintain the code quality and improve the efficiency of the review process. ### DCO and Signed-off-by @@ -144,9 +116,9 @@ Only specific types of PRs will be reviewed. The PR title is prefixed appropriat - `[CI/Build]` for build or continuous integration improvements. - `[Doc]` for documentation fixes and improvements. - `[Model]` for adding a new model or improving an existing model. Model name should appear in the title. -- `[Frontend]` For changes on the vLLM-omni frontend (e.g., OpenAI API server, `OmniLLM` class, etc.) +- `[Frontend]` For changes on the vLLM-Omni frontend (e.g., OpenAI API server, `OmniLLM` class, etc.) - `[Kernel]` for changes affecting CUDA kernels or other compute kernels. -- `[Core]` for changes in the core vLLM-omni logic (e.g., `OmniProcessor`, `OmniScheduler`, etc.) +- `[Core]` for changes in the core vLLM-Omni logic (e.g., `OmniProcessor`, `OmniScheduler`, etc.) - `[Hardware][Vendor]` for hardware-specific changes. Vendor name should appear in the prefix, such as [Ascend] for Ascend NPUs. - `[Misc]` for PRs that do not fit the above categories. Please use this sparingly. @@ -161,7 +133,7 @@ The PR needs to meet the following code quality standards: - Pass all linter checks. - The code needs to be well-documented to ensure future contributors can easily understand the code. - Include sufficient tests to ensure the project stays correct and robust. This includes both unit tests and integration tests. -- Please add documentation to `docs/` if the PR modifies the user-facing behaviors of vLLM-omni. It helps vLLM-omni users understand and utilize the new features or changes. +- Please add documentation to `docs/` if the PR modifies the user-facing behaviors of vLLM-Omni. It helps vLLM-Omni users understand and utilize the new features or changes. ### Notes for Large Changes @@ -169,13 +141,12 @@ Please keep the changes as concise as possible. For major architectural changes ### What to Expect for the Reviews -The goal of the vLLM-omni team is to be a _transparent reviewing machine_. We would like to make the review process transparent and efficient and make sure no contributor feels confused or frustrated. However, the vLLM-omni team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process: +The goal of the vLLM-Omni team is to be a _transparent reviewing machine_. We would like to make the review process transparent and efficient and make sure no contributor feels confused or frustrated. However, the vLLM-Omni team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process: - After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability. -- After the PR is assigned, the reviewer will provide status updates every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM-omni team. +- After the PR is assigned, the reviewer will provide status updates every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM-Omni team. - After the review, the reviewer will put an `action-required` label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR. - Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion. -- Note that not all CI checks will be executed due to limited computational resources. The reviewer will add `ready` label to the PR when the PR is ready to merge or a full CI run is needed. ## Additional Resources @@ -183,4 +154,4 @@ The goal of the vLLM-omni team is to be a _transparent reviewing machine_. We wo ## Thank You -Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM-omni. All of your contributions help make vLLM-omni a great tool and community for everyone! +Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM-Omni. All of your contributions help make vLLM-Omni a great tool and community for everyone! diff --git a/docs/contributing/model/README.md b/docs/contributing/model/README.md index ac9323cbc..2e7c3a529 100644 --- a/docs/contributing/model/README.md +++ b/docs/contributing/model/README.md @@ -1 +1,2 @@ -# Summary +# Adding a New Model +Stay tuned for guidance on how to add a new model on vLLM-Omni! diff --git a/docs/design/api_design_doc.md b/docs/design/api_design_doc.md index f67f325fe..b29c44ab5 100644 --- a/docs/design/api_design_doc.md +++ b/docs/design/api_design_doc.md @@ -1,6 +1,6 @@ # API Design Documentation -This document contains comprehensive API design documentation for all core modules in vLLM-omni. These templates provide a standardized structure for designing and implementing the core, engine, executor, and worker modules. +This document contains comprehensive API design documentation for all core modules in vLLM-Omni. These templates provide a standardized structure for designing and implementing the core, engine, executor, and worker modules. ## 📋 Module API diff --git a/docs/design/api_design_template.md b/docs/design/api_design_template.md index f9b0478dd..2fe2b233f 100644 --- a/docs/design/api_design_template.md +++ b/docs/design/api_design_template.md @@ -1,6 +1,6 @@ -# API Design Template for vLLM-omni Modules +# API Design Template for vLLM-Omni Modules -This template provides a standardized structure for designing APIs for core, engine, executor, and worker modules in vLLM-omni. Use this template to ensure consistency and completeness across all modules. +This template provides a standardized structure for designing APIs for core, engine, executor, and worker modules in vLLM-Omni. Use this template to ensure consistency and completeness across all modules. ## Template Structure @@ -44,7 +44,7 @@ This template provides a standardized structure for designing APIs for core, eng ### 1. Module Overview -**Purpose**: The core module provides fundamental scheduling and caching functionality for vLLM-omni. +**Purpose**: The core module provides fundamental scheduling and caching functionality for vLLM-Omni. **Responsibilities**: - Request scheduling and prioritization @@ -333,4 +333,4 @@ print(f"Cache hit rate: {status['cache_hit_rate']:.2%}") 6. Include error handling strategies 7. Submit for review before implementation -This template ensures consistency and completeness across all vLLM-omni modules. +This template ensures consistency and completeness across all vLLM-Omni modules. diff --git a/docs/design/architecture_overview.md b/docs/design/architecture_overview.md index 54319bce0..3a219a564 100644 --- a/docs/design/architecture_overview.md +++ b/docs/design/architecture_overview.md @@ -2,11 +2,11 @@ # Introduction -This document outlines the architectural design for vLLM-omni, a new repository developed under the vLLM project. vLLM-omni supports multi-modality models with non-autoregressive structures and non-textual outputs, extending vLLM beyond traditional text-based, autoregressive generation. +This document outlines the architectural design for vLLM-Omni, a new repository developed under the vLLM project. vLLM-Omni supports multi-modality models with non-autoregressive structures and non-textual outputs, extending vLLM beyond traditional text-based, autoregressive generation. # Goals -The primary goals for vLLM-omni are: +The primary goals for vLLM-Omni are: * **Support for Non-textual Output:** Enables the integration, efficient processing and output of various data types, including but not limited to, images, audio, and video, alongside text. * **Non-Autoregressive Structure:** Support model structure beyond autoregressive, especially Diffusion Transformer (DiT), which is widely used in visual and audio generation. @@ -57,12 +57,12 @@ According to analysis for current popular open-source models, most of them have # Architectural Overview -vLLM-omni will import core modules from existing vLLM framework and introduce several new components and modifications to achieve its goals. The core architecture will be based on a modular design, allowing for independent development and integration of different modality handlers and output generators. +vLLM-Omni will import core modules from existing vLLM framework and introduce several new components and modifications to achieve its goals. The core architecture will be based on a modular design, allowing for independent development and integration of different modality handlers and output generators. ![][image6] ## General ideas -* It divides models in vLLM-omni into multiple stages, which are processed by different Engine Cores. Each stage can choose AR Engine Core for AR model part, and Diffusion Engine Core for diffusion model part. +* It divides models in vLLM-Omni into multiple stages, which are processed by different Engine Cores. Each stage can choose AR Engine Core for AR model part, and Diffusion Engine Core for diffusion model part. * Designed a specialized diffusion engine core for DiT or dLLM. * Plug-in with external DiT acceleration modules such as xDiT, Cache-DiT. * Output linked with Higress API, Gradio for online demo, ComfyUI for less-code process. @@ -89,32 +89,6 @@ vLLM-omni will import core modules from existing vLLM framework and introduce se ![][image11] -# Roadmap - -* AsyncLLM: -- [ ] Initializer: resource allocation for different stages (2025 H2) -- [ ] Hidden state Output Processor (2025 H2) -- [ ] Multimodal Output Processor (2025 H2) -- [ ] Interface for API, Gradio, ComfyUI -* OmniEngine: -- [ ] Basic Diffusion Engine core: Diffusion step manager, Cache manager (2025 H2) -- [ ] Pipeline of different Engine Cores (2025 H2) -- [ ] Communication between Engine Cores via AsyncLLM (2025 H2) -* Model support (Included but not limited to): -- [ ] DiT: Qwen-Image (2025 H2), Flux-Kontext -- [ ] Unified-MLLM: BAGEL (2025 H2), OmniGen2, Janus-Pro -- [ ] Extensions of AR+DiT: Qwen-omni (2025 H2) -* Optimization -- [ ] Integration of other DiT acceleration modules, such as xDiT, Cache-DiT -- [ ] Disaggregation for AR and DiT -- [ ] Integration with vLLM's existing KV cache for potential future hybrid models. -- [ ] Support for streaming multi-modal inputs and outputs - -# Contact - -For any questions or feedback regarding this design document, please contact -Han Gao: [hgaoaf@connect.ust.hk](mailto:hgaoaf@connect.ust.hk), -Hongsheng Liu: [vllm-project@gmail.com](mailto:vllm-project@gmail.com) [image1]: diff --git a/docs/design/index.md b/docs/design/index.md index 42e360ed4..aefefa0c8 100644 --- a/docs/design/index.md +++ b/docs/design/index.md @@ -1,12 +1,13 @@ # Design Documents -This section contains design documents and architecture specifications for vLLM-omni. +This section contains design documents and architecture specifications for vLLM-Omni. ## Architecture Documents -- [vLLM-omni Software Design Document](vllm_omni_design.md) - [Architecture Overview](architecture_overview.md) +- [vLLM-Omni Software Design Document](vllm_omni_design.md) +- [Multi-Request Streaming](mrs_design.md) ## API Design Documentation -- [vLLM-omni API Documentation](api_design_doc.md) +- [vLLM-Omni API Design Documentation](api_design_doc.md) diff --git a/docs/design/vllm_omni_design.md b/docs/design/vllm_omni_design.md index 27b62f691..8fe328878 100644 --- a/docs/design/vllm_omni_design.md +++ b/docs/design/vllm_omni_design.md @@ -2,7 +2,7 @@ ## Overview -vLLM-omni is a multi-modality extension for vLLM that supports non-autoregressive structures and non-textual outputs. This document outlines the key software abstractions, APIs, and dependencies for the system, designed to maximize reuse of vLLM's proven architecture. +vLLM-Omni is a multi-modality extension for vLLM that supports non-autoregressive structures and non-textual outputs. This document outlines the key software abstractions, APIs, and dependencies for the system, designed to maximize reuse of vLLM's proven architecture. ## Architecture Principles @@ -36,7 +36,7 @@ vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8000 #### Design Architecture ```mermaid graph TD - A[vllm serve model --omni] --> B[vLLM-omni CLI Wrapper] + A[vllm serve model --omni] --> B[vLLM-Omni CLI Wrapper] B --> C{Detect --omni flag} C -->|Yes| D[Parse OmniConfig] C -->|No| E[Forward to vLLM CLI] @@ -45,7 +45,7 @@ graph TD G --> H[Multi-stage Processing] E --> I[Standard vLLM Pipeline] - subgraph "vLLM-omni Components" + subgraph "vLLM-Omni Components" F G H @@ -68,7 +68,7 @@ def main(): """Main CLI entry point that intercepts vLLM commands""" # Check if --omni flag is present if "--omni" in sys.argv: - # Remove --omni flag and process with vLLM-omni + # Remove --omni flag and process with vLLM-Omni omni_args = [arg for arg in sys.argv[1:] if arg != "--omni"] omni_serve = OmniServeCommand() omni_serve.run(omni_args) diff --git a/docs/features/README.md b/docs/features/README.md deleted file mode 100644 index c18fb056e..000000000 --- a/docs/features/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# Features - -vLLM-Omni introduces new features to support multimodal comprehension and generation models inference and serving. diff --git a/docs/getting_started/installation/README.md b/docs/getting_started/installation/README.md index 17f1b1f5c..c301e7a12 100644 --- a/docs/getting_started/installation/README.md +++ b/docs/getting_started/installation/README.md @@ -3,5 +3,5 @@ vLLM supports the following hardware platforms: - [GPU](gpu.md) - - [NVIDIA CUDA](gpu.md#nvidia-cuda) + - [NVIDIA CUDA](gpu.md) - [Ascend NPU](npu.md) diff --git a/docs/getting_started/installation/gpu.md b/docs/getting_started/installation/gpu.md index 5b7d080f7..4926f93f7 100644 --- a/docs/getting_started/installation/gpu.md +++ b/docs/getting_started/installation/gpu.md @@ -1,12 +1,6 @@ # GPU -vLLM-Omni is a Python library that supports the following GPU variants. Select your GPU type to see vendor specific instructions: - -[](){ #nvidia-cuda } - -=== "NVIDIA CUDA" - - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:installation" +vLLM-Omni is a Python library that supports the following GPU variants. The library itself mainly contains python implementations for framework and models. ## Requirements @@ -14,7 +8,7 @@ vLLM-Omni is a Python library that supports the following GPU variants. Select y - Python: 3.12 !!! note - vLLM-Omni does not support Windows natively. + vLLM-Omni is currently not natively supported on Windows. === "NVIDIA CUDA" @@ -26,10 +20,6 @@ vLLM-Omni is a Python library that supports the following GPU variants. Select y --8<-- "docs/getting_started/installation/python_env_setup.inc.md" -=== "NVIDIA CUDA" - - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:set-up-using-python" - ### Pre-built wheels === "NVIDIA CUDA" @@ -47,3 +37,4 @@ vLLM-Omni is a Python library that supports the following GPU variants. Select y ## Set up using Docker ### Pre-built images +To be released... Stay tuned! diff --git a/docs/getting_started/installation/gpu/cuda.inc.md b/docs/getting_started/installation/gpu/cuda.inc.md index 59087cee7..ce394f8a6 100644 --- a/docs/getting_started/installation/gpu/cuda.inc.md +++ b/docs/getting_started/installation/gpu/cuda.inc.md @@ -1,8 +1,3 @@ -# --8<-- [start:installation] - -vLLM-Omni mainly contains python implementations for framework and models. - -# --8<-- [end:installation] # --8<-- [start:requirements] - GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.) @@ -21,14 +16,14 @@ Therefore, it is recommended to install vLLM and vLLM-Omni with a **fresh new** # --8<-- [start:pre-built-wheels] -## Installation of vLLM +#### Installation of vLLM -Now we build it based on vLLM v0.11.0. Please install it with command below. +vLLM-Omni is built based on vLLM v0.11.0. Please install it with command below. ```bash uv pip install vllm==0.11.0 --torch-backend=auto ``` -## Installation of vLLM-Omni +#### Installation of vLLM-Omni ```bash uv pip install vllm-omni @@ -38,15 +33,30 @@ uv pip install vllm-omni # --8<-- [start:build-wheel-from-source] -## Installation of vLLM -If you want to check or debug with source code of vLLM, install stable release version of vllm with 0.11.0 from source with pre-built wheel file. +#### Installation of vLLM +If you do not need to modify source code of vLLM, you can directly install the stable 0.11.0 release version of the library + +```bash +uv pip install vllm==0.11.0 --torch-backend=auto +``` + +#### Installation of vLLM-Omni +Install additional requirements for vLLM-Omni +```bash +git clone https://github.com/vllm-project/vllm-omni.git +cd vllm_omni +uv pip install -e . +``` + +
(Optional) Installation of vLLM from source +If you want to check, modify or debug with source code of vLLM, install the library from source with the following instructions: ```bash git clone https://github.com/vllm-project/vllm.git cd vllm git checkout v0.11.0 ``` -Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set VLLM_PRECOMPILED_WHEEL_LOCATION as your local absolute path of whl file. +Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set `VLLM_PRECOMPILED_WHEEL_LOCATION` as your local absolute path of whl file. ```bash export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.11.0/vllm-0.11.0-cp38-abi3-manylinux1_x86_64.whl ``` @@ -54,20 +64,12 @@ Install vllm with command below (If you have no existing PyTorch). ```bash uv pip install --editable . ``` - Install vllm with command below (If you already have PyTorch). ```bash python use_existing_torch.py uv pip install -r requirements/build.txt uv pip install --no-build-isolation --editable . ``` - -## Installation of vLLM-omni -Install additional requirements for vllm-omni -```bash -git clone https://github.com/vllm-project/vllm-omni.git -cd vllm_omni -uv pip install -e . -``` +
# --8<-- [end:build-wheel-from-source] diff --git a/docs/getting_started/quickstart.md b/docs/getting_started/quickstart.md index c8fdf7665..d2fd88961 100644 --- a/docs/getting_started/quickstart.md +++ b/docs/getting_started/quickstart.md @@ -20,4 +20,4 @@ Please refer to [offline inference](../user_guide/examples/offline_inference/qwe ## Online Serving with OpenAI-Completions API -Please refer to [online serving](../user_guide/examples/online_serving/README.md) +Please refer to [online serving](../user_guide/examples/online_serving/qwen2_5_omni.md) diff --git a/docs/mkdocs/hooks/generate_api_readme.py b/docs/mkdocs/hooks/generate_api_readme.py index f1819cfc8..eb7bbfdbe 100644 --- a/docs/mkdocs/hooks/generate_api_readme.py +++ b/docs/mkdocs/hooks/generate_api_readme.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM-omni project +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project """ Hook to automatically generate docs/api/README.md from the codebase. @@ -20,7 +20,7 @@ API_README_PATH = ROOT_DIR / "docs" / "api" / "README.md" CATEGORIES = { "entrypoints": { "name": "Entry Points", - "description": "Main entry points for vLLM-omni inference and serving.", + "description": "Main entry points for vLLM-Omni inference and serving.", }, "inputs": { "name": "Inputs", diff --git a/docs/mkdocs/hooks/generate_examples.py b/docs/mkdocs/hooks/generate_examples.py deleted file mode 100644 index a1f75983f..000000000 --- a/docs/mkdocs/hooks/generate_examples.py +++ /dev/null @@ -1,233 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM-omni project -import itertools -import logging -from dataclasses import dataclass, field -from pathlib import Path -from typing import Literal - -import regex as re - -logger = logging.getLogger("mkdocs") - -ROOT_DIR = Path(__file__).parent.parent.parent.parent -ROOT_DIR_RELATIVE = "../../../../.." -EXAMPLE_DIR = ROOT_DIR / "examples" -EXAMPLE_DOC_DIR = ROOT_DIR / "docs/user_guide/examples" - - -def fix_case(text: str) -> str: - subs = { - "api": "API", - "cli": "CLI", - "cpu": "CPU", - "llm": "LLM", - "mae": "MAE", - "tpu": "TPU", - "gguf": "GGUF", - "lora": "LoRA", - "rlhf": "RLHF", - "vllm": "vLLM", - "omni": "Omni", - "openai": "OpenAI", - "lmcache": "LMCache", - "multilora": "MultiLoRA", - "mlpspeculator": "MLPSpeculator", - r"fp\d+": lambda x: x.group(0).upper(), # e.g. fp16, fp32 - r"int\d+": lambda x: x.group(0).upper(), # e.g. int8, int16 - } - for pattern, repl in subs.items(): - text = re.sub(rf"\b{pattern}\b", repl, text, flags=re.IGNORECASE) - return text - - -@dataclass -class Example: - """ - Example class for generating documentation content from a given path. - - Attributes: - path (Path): The path to the main directory or file. - category (str): The category of the document. - main_file (Path): The main file in the directory. - other_files (list[Path]): list of other files in the directory. - title (str): The title of the document. - - Methods: - __post_init__(): Initializes the main_file, other_files, and title attributes. - determine_main_file() -> Path: Determines the main file in the given path. - determine_other_files() -> list[Path]: Determines other files in the directory excluding the main file. - determine_title() -> str: Determines the title of the document. - generate() -> str: Generates the documentation content. - """ # noqa: E501 - - path: Path - category: str = None - main_file: Path = field(init=False) - other_files: list[Path] = field(init=False) - title: str = field(init=False) - - def __post_init__(self): - self.main_file = self.determine_main_file() - self.other_files = self.determine_other_files() - self.title = self.determine_title() - - @property - def is_code(self) -> bool: - return self.main_file.suffix != ".md" - - def determine_main_file(self) -> Path: - """ - Determines the main file in the given path. - If the path is a file, it returns the path itself. Otherwise, it searches - for Markdown files (*.md) in the directory and returns the first one found. - Returns: - Path: The main file path, either the original path if it's a file or the first - Markdown file found in the directory. - Raises: - IndexError: If no Markdown files are found in the directory. - """ # noqa: E501 - return self.path if self.path.is_file() else list(self.path.glob("*.md")).pop() - - def determine_other_files(self) -> list[Path]: - """ - Determine other files in the directory excluding the main file. - - This method checks if the given path is a file. If it is, it returns an empty list. - Otherwise, it recursively searches through the directory and returns a list of all - files that are not the main file. - - Returns: - list[Path]: A list of Path objects representing the other files in the directory. - """ # noqa: E501 - if self.path.is_file(): - return [] - # Text file extensions that can be safely included - text_extensions = { - ".py", - ".md", - ".sh", - ".yaml", - ".yml", - ".json", - ".txt", - ".toml", - ".cfg", - ".ini", - } - is_other_file = lambda file: (file.is_file() and file != self.main_file and file.suffix in text_extensions) # noqa: E731 - return [file for file in self.path.rglob("*") if is_other_file(file)] - - def determine_title(self) -> str: - if not self.is_code: - # Specify encoding for building on Windows - with open(self.main_file, encoding="utf-8") as f: - first_line = f.readline().strip() - match = re.match(r"^#\s+(?P.+)$", first_line) - if match: - return match.group("title") - return fix_case(self.path.stem.replace("_", " ").title()) - - def fix_relative_links(self, content: str) -> str: - """ - Fix relative links in markdown content by converting them to gh-file - format. - - Args: - content (str): The markdown content to process - - Returns: - str: Content with relative links converted to gh-file format - """ - # Regex to match markdown links [text](relative_path) - # This matches links that don't start with http, https, ftp, or # - link_pattern = r"\[([^\]]*)\]\((?!(?:https?|ftp)://|#)([^)]+)\)" - - def replace_link(match): - link_text = match.group(1) - relative_path = match.group(2) - - # Make relative to repo root - gh_file = (self.main_file.parent / relative_path).resolve() - gh_file = gh_file.relative_to(ROOT_DIR) - - # Make GitHub URL - url = "https://github.com/vllm-project/vllm-omni/" - url += "tree/main" if self.path.is_dir() else "blob/main" - gh_url = f"{url}/{gh_file}" - - return f"[{link_text}]({gh_url})" - - return re.sub(link_pattern, replace_link, content) - - def generate(self) -> str: - content = f"# {self.title}\n\n" - url = "https://github.com/vllm-project/vllm-omni/" - url += "tree/main" if self.path.is_dir() else "blob/main" - content += f"Source <{url}/{self.path.relative_to(ROOT_DIR)}>.\n\n" - - # Use long code fence to avoid issues with - # included files containing code fences too - code_fence = "``````" - - if self.is_code: - # Use relative path from ROOT_DIR for snippet inclusion - main_file_rel_path = self.main_file.relative_to(ROOT_DIR) - content += f'{code_fence}{self.main_file.suffix[1:]}\n--8<-- "{main_file_rel_path}"\n{code_fence}\n' - else: - with open(self.main_file, encoding="utf-8") as f: - # Skip the title from md snippets as it's been included above - main_content = f.readlines()[1:] - content += self.fix_relative_links("".join(main_content)) - content += "\n" - - if not self.other_files: - return content - - content += "## Example materials\n\n" - for file in sorted(self.other_files): - # Use relative path from ROOT_DIR for snippet inclusion - file_rel_path = file.relative_to(ROOT_DIR) - content += f'??? abstract "{file.relative_to(self.path)}"\n' - if file.suffix != ".md": - content += f" {code_fence}{file.suffix[1:]}\n" - content += f' --8<-- "{file_rel_path}"\n' - if file.suffix != ".md": - content += f" {code_fence}\n" - - return content - - -def on_startup(command: Literal["build", "gh-deploy", "serve"], dirty: bool): - logger.info("Generating example documentation") - logger.debug("Root directory: %s", ROOT_DIR.resolve()) - logger.debug("Example directory: %s", EXAMPLE_DIR.resolve()) - logger.debug("Example document directory: %s", EXAMPLE_DOC_DIR.resolve()) - - # Create the EXAMPLE_DOC_DIR if it doesn't exist - if not EXAMPLE_DOC_DIR.exists(): - EXAMPLE_DOC_DIR.mkdir(parents=True) - - categories = sorted(p for p in EXAMPLE_DIR.iterdir() if p.is_dir()) - - examples = [] - glob_patterns = ["*.py", "*.md"] - # Find categorised examples - for category in categories: - globs = [category.glob(pattern) for pattern in glob_patterns] - for path in itertools.chain(*globs): - examples.append(Example(path, category.stem)) - # Find examples in subdirectories - for path in category.glob("*/*.md"): - examples.append(Example(path.parent, category.stem)) - - # Generate the example documentation - for example in sorted(examples, key=lambda e: e.path.stem): - example_name = f"{example.path.stem}.md" - doc_path = EXAMPLE_DOC_DIR / example.category / example_name - if not doc_path.parent.exists(): - doc_path.parent.mkdir(parents=True) - # Specify encoding for building on Windows - with open(doc_path, "w+", encoding="utf-8") as f: - f.write(example.generate()) - logger.debug("Example generated: %s", doc_path.relative_to(ROOT_DIR)) diff --git a/docs/user_guide/examples/offline_inference/qwen2_5_omni.md b/docs/user_guide/examples/offline_inference/qwen2_5_omni.md index 0e82bb9b0..c11360bcf 100644 --- a/docs/user_guide/examples/offline_inference/qwen2_5_omni.md +++ b/docs/user_guide/examples/offline_inference/qwen2_5_omni.md @@ -1,40 +1,55 @@ -# Offline Example of vLLM-omni for Qwen2.5-omni +# Offline Inference Example of vLLM-Omni for Qwen2.5-Omni Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen2_5_omni>. ## 🛠️ Installation -Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md) +Please refer to [installation](../../../getting_started/installation/README.md). + +## Run Offline inference with Qwen2.5-Omni. +First, navigate to the example folder +```bash +cd examples/offline_inference/qwen2_5_omni +``` +Inside the directory, `end2end.py` is a comprehensive demo suite for initializing a model instance of `Qwen/Qwen2.5-Omni-7B` and use it for various offline inference tasks. +??? abstract "end2end.py" + ``````py + --8<-- "examples/offline_inference/qwen2_5_omni/end2end.py" + `````` + +Below we also provide simple bash scripts to execute this file. +### Single Prompt +```bash +bash run_single_prompt.sh +``` +??? abstract "run_single_prompt.sh" + ``````sh + --8<-- "examples/offline_inference/qwen2_5_omni/run_single_prompt.sh" + `````` -## Run examples (Qwen2.5-omni) ### Multiple Prompts -Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). To get the prompt, you can: +Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit) and extract the prompts with `extract_prompts.py` +??? abstract "extract_prompts.py" + ``````py + --8<-- "examples/offline_inference/qwen2_5_omni/extract_prompts.py" + `````` ```bash tar -xf <Your Download Path>/seedtts_testset.tar cp seedtts_testset/en/meta.lst examples/offline_inference/qwen2_5_omni/meta.lst python3 examples/offline_inference/qwen2_5_omni/extract_prompts.py \ --input examples/offline_inference/qwen2_5_omni/meta.lst \ --output examples/offline_inference/qwen2_5_omni/top100.txt \ - --topk 100 -``` -Get into the example folder -```bash -cd examples/offline_inference/qwen2_5_omni + --topk 10 ``` Then run the command below. ```bash bash run_multiple_prompts.sh ``` -### Single Prompts -Get into the example folder -```bash -cd examples/offline_inference/qwen2_5_omni -``` -Then run the command below. -```bash -bash run_single_prompt.sh -``` +??? abstract "run_multiple_prompts.sh" + ``````sh + --8<-- "examples/offline_inference/qwen2_5_omni/run_multiple_prompts.sh" + `````` ### FAQ @@ -43,22 +58,3 @@ If you encounter error about backend of librosa, try to install ffmpeg with comm sudo apt update sudo apt install ffmpeg ``` - -## Example materials - -??? abstract "end2end.py" - ``````py - --8<-- "examples/offline_inference/qwen2_5_omni/end2end.py" - `````` -??? abstract "extract_prompts.py" - ``````py - --8<-- "examples/offline_inference/qwen2_5_omni/extract_prompts.py" - `````` -??? abstract "run_multiple_prompts.sh" - ``````sh - --8<-- "examples/offline_inference/qwen2_5_omni/run_multiple_prompts.sh" - `````` -??? abstract "run_single_prompt.sh" - ``````sh - --8<-- "examples/offline_inference/qwen2_5_omni/run_single_prompt.sh" - `````` diff --git a/docs/user_guide/examples/online_serving/README.md b/docs/user_guide/examples/online_serving/README.md deleted file mode 100644 index f6312fdae..000000000 --- a/docs/user_guide/examples/online_serving/README.md +++ /dev/null @@ -1,80 +0,0 @@ -# Online serving Example of vLLM-omni for Qwen2.5-omni - -Source <https://github.com/vllm-project/vllm-omni/blob/main/examples/online_serving/README.md>. - - -## 🛠️ Installation - -Please refer to [README.md](https://github.com/vllm-project/vllm-omni/blob/main/README.md) - -## Run examples (Qwen2.5-omni) - -Launch the server -```bash -vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 -``` - -If you have custom stage configs file, launch the server with command below -```bash -vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file -``` - -Get into the example folder -```bash -cd examples/online_serving -``` - -Send request via python -```bash -python openai_chat_completion_client_for_multimodal_generation.py --query-type mixed_modalities -``` - -Send request via curl -```bash -bash run_curl_multimodal_generation.sh mixed_modalities -``` - -### FAQ - -If you encounter error about backend of librosa, try to install ffmpeg with command below. -``` -sudo apt update -sudo apt install ffmpeg -``` - -## Run Local Web UI Demo - -This Web UI demo allows users to interact with the model through a web browser. - -### Running Gradio Demo - -Once vllm and vllm-omni are installed, you can launch the web service built on AsyncOmniLLM by - -```bash -python gradio_demo.py --model Qwen/Qwen2.5-Omni-7B --port 7861 -``` - -Then open `http://localhost:7861/` on your local browser to interact with the web UI. - - -### Options - -The gradio demo also supports running with an existing API server and can be customized with the following arguments. - - -```bash -python gradio_demo.py \ - --model Qwen/Qwen2.5-Omni-7B \ - --use-api-server \ - --api-base http://localhost:8091/v1 \ - --ip 127.0.0.1 \ - --port 7861 -``` - -- `--model`: Model name -- `--use-api-server`: If set, connect to an existing vLLM HTTP API server instead of running AsyncOmniLLM locally. -- `--api-base`: Base URL for vllm serve (only used when `use-api-server` is set, default: http://localhost:8091/v1) -- `--ip`: Host/IP for Gradio server (default: 127.0.0.1) -- `--port`: Port for Gradio server (default: 7861) -- `--stage-configs-path`: Path to custom stage configs YAML file (optional) -- `--share`: Share the Gradio demo publicly (creates a public link) diff --git a/docs/user_guide/examples/online_serving/qwen2_5_omni.md b/docs/user_guide/examples/online_serving/qwen2_5_omni.md new file mode 100644 index 000000000..96c283d16 --- /dev/null +++ b/docs/user_guide/examples/online_serving/qwen2_5_omni.md @@ -0,0 +1,80 @@ +# Online Serving Example of vLLM-Omni for Qwen2.5-Omni + +Source <https://github.com/vllm-project/vllm-omni/blob/main/examples/online_serving/README.md>. + + +## 🛠️ Installation + +Please refer to [installation](../../../getting_started/installation/README.md). + +## Deploy Qwen/Qwen2.5-Omni-7B + +First launch the OpenAI-compatible inference server +```bash +vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 +``` +If you have custom stage configs file, launch the server with command below +```bash +vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file +``` +## Query the model +Navigate to the example folder +```bash +cd examples/online_serving +``` +Query the model server with OpenAI Python API client: +```bash +python openai_chat_completion_client_for_multimodal_generation.py --query-type mixed_modalities +``` +??? abstract "openai_chat_completion_client_for_multimodal_generation.py" + ``````py + --8<-- "examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py" + `````` +You can also query the model with `curl` command: +```bash +bash run_curl_multimodal_generation.sh mixed_modalities +``` +??? abstract "run_curl_multimodal_generation.sh" + ``````py + --8<-- "examples/online_serving/run_curl_multimodal_generation.sh" + `````` + +### FAQ + +If you encounter error about backend of librosa, try to install ffmpeg with command below. +``` +sudo apt update +sudo apt install ffmpeg +``` + +## Run Local Web UI Demo + +You can also deploy a Gradio Web UI that allows users to interact with the model through a web browser. Below is an example on how to do so with `Qwen/Qwen2.5-Omni-7B`. + +### Running Gradio Demo + +Install gradio with `uv pip install "gradio>=5.49.1,<6.0.0"`, then you can launch the web service built on AsyncOmni by + +```bash +python gradio_demo.py --model Qwen/Qwen2.5-Omni-7B --port 7861 +``` +Now you can interact with model via the web UI at `http://localhost:7861/` on your local browser. + + +### Options + +You can customize its basic launch parameters: + +```bash +python gradio_demo.py \ + --model Qwen/Qwen2.5-Omni-7B \ + --ip 127.0.0.1 \ + --port 7861 \ + --stage-configs-path /path/to/stage_configs.yaml +``` + +- `--model`: Local model checkpoint to load (default `Qwen/Qwen2.5-Omni-7B`). +- `--ip`: Host/IP for the Gradio server (default `127.0.0.1`). +- `--port`: Port for the Gradio server (default `7861`). +- `--stage-configs-path`: Optional path to custom stage configs YAML. +- `--share`: Set to expose a temporary public link via Gradio. diff --git a/examples/offline_inference/qwen2_5_omni/README.md b/examples/offline_inference/qwen2_5_omni/README.md index 142039789..afdaee13e 100644 --- a/examples/offline_inference/qwen2_5_omni/README.md +++ b/examples/offline_inference/qwen2_5_omni/README.md @@ -1,4 +1,4 @@ -# Offline Example of vLLM-omni for Qwen2.5-omni +# Offline Example of vLLM-Omni for Qwen2.5-omni ## 🛠️ Installation diff --git a/examples/offline_inference/qwen2_5_omni/end2end.py b/examples/offline_inference/qwen2_5_omni/end2end.py index 9045172ae..10bcf4a20 100644 --- a/examples/offline_inference/qwen2_5_omni/end2end.py +++ b/examples/offline_inference/qwen2_5_omni/end2end.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """ -This example shows how to use vLLM-omni for running offline inference +This example shows how to use vLLM-Omni for running offline inference with the correct prompt format on Qwen2.5-Omni """ diff --git a/examples/online_serving/README.md b/examples/online_serving/README.md index 948c196be..da9570270 100644 --- a/examples/online_serving/README.md +++ b/examples/online_serving/README.md @@ -1,4 +1,4 @@ -# Online serving Example of vLLM-omni for Qwen2.5-omni +# Online serving Example of vLLM-Omni for Qwen2.5-omni ## 🛠️ Installation diff --git a/mkdocs.yml b/mkdocs.yml index f5239388e..73783ab8c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -56,7 +56,6 @@ theme: custom_dir: docs/mkdocs/overrides hooks: - - docs/mkdocs/hooks/generate_examples.py - docs/mkdocs/hooks/generate_api_readme.py - docs/mkdocs/hooks/url_schemes.py diff --git a/pyproject.toml b/pyproject.toml index 037996e16..7f499e0a1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,12 +5,12 @@ build-backend = "setuptools.build_meta" [project] name = "vllm-omni" version = "0.11.0rc1" -description = "vLLM-omni: Multi-modality models inference and serving with non-autoregressive structures" +description = "vLLM-Omni: Multi-modality models inference and serving with non-autoregressive structures" readme = "README.md" requires-python = ">=3.9,<3.14" # Align with vLLM v0.11 supported Python versions license = {text = "Apache-2.0"} authors = [ - {name = "vLLM-omni Team", email = "vllm-project@gmail.com"} + {name = "vLLM-Omni Team", email = "vllm-project@gmail.com"} ] keywords = ["vllm", "multimodal", "diffusion", "transformer", "inference", "serving"] classifiers = [ diff --git a/vllm_omni/__init__.py b/vllm_omni/__init__.py index be1afdf00..668d59b3d 100644 --- a/vllm_omni/__init__.py +++ b/vllm_omni/__init__.py @@ -1,5 +1,5 @@ """ -vLLM-omni: Multi-modality models inference and serving with +vLLM-Omni: Multi-modality models inference and serving with non-autoregressive structures. This package extends vLLM beyond traditional text-based, autoregressive diff --git a/vllm_omni/config/__init__.py b/vllm_omni/config/__init__.py index 0fae07e7a..cc7e8132e 100644 --- a/vllm_omni/config/__init__.py +++ b/vllm_omni/config/__init__.py @@ -1,5 +1,5 @@ """ -Configuration module for vLLM-omni. +Configuration module for vLLM-Omni. """ from vllm_omni.config.model import OmniModelConfig diff --git a/vllm_omni/core/dit_cache_manager.py b/vllm_omni/core/dit_cache_manager.py index 9009638d5..e07c4a167 100644 --- a/vllm_omni/core/dit_cache_manager.py +++ b/vllm_omni/core/dit_cache_manager.py @@ -1,5 +1,5 @@ """ -DiT Cache Manager for vLLM-omni. +DiT Cache Manager for vLLM-Omni. This module provides caching functionality for DiT (Diffusion Transformer) models to optimize inference performance and memory usage. diff --git a/vllm_omni/core/sched/__init__.py b/vllm_omni/core/sched/__init__.py index 4f2b89170..9b3a34ca5 100644 --- a/vllm_omni/core/sched/__init__.py +++ b/vllm_omni/core/sched/__init__.py @@ -1,5 +1,5 @@ """ -Scheduling components for vLLM-omni. +Scheduling components for vLLM-Omni. """ from .diffusion_scheduler import DiffusionScheduler diff --git a/vllm_omni/core/sched/scheduler.py b/vllm_omni/core/sched/scheduler.py index 3a30543c7..8d1fe8520 100644 --- a/vllm_omni/core/sched/scheduler.py +++ b/vllm_omni/core/sched/scheduler.py @@ -7,11 +7,11 @@ from vllm.v1.core.sched.scheduler import Scheduler as VLLMScheduler class OmniScheduler(VLLMScheduler): """ - OmniScheduler: Scheduler for vLLM-omni multimodal processing. + OmniScheduler: Scheduler for vLLM-Omni multimodal processing. This scheduler extends vLLM's scheduler to support multimodal and non-autoregressive processing with additional fields and methods - specific to vLLM-omni. + specific to vLLM-Omni. """ # Ensure scheduled_new_reqs carry omni-specific payloads diff --git a/vllm_omni/engine/__init__.py b/vllm_omni/engine/__init__.py index 6a7f48525..7cc4066c9 100644 --- a/vllm_omni/engine/__init__.py +++ b/vllm_omni/engine/__init__.py @@ -1,5 +1,5 @@ """ -Engine components for vLLM-omni. +Engine components for vLLM-Omni. """ from typing import Any, Optional diff --git a/vllm_omni/entrypoints/async_omni_llm.py b/vllm_omni/entrypoints/async_omni_llm.py index 730001173..e1b74ea6f 100644 --- a/vllm_omni/entrypoints/async_omni_llm.py +++ b/vllm_omni/entrypoints/async_omni_llm.py @@ -53,7 +53,7 @@ logger = init_logger(__name__) class AsyncOmniLLM(EngineClient): - """Async entry point for vLLM-omni inference. + """Async entry point for vLLM-Omni inference. This class provides an asynchronous interface for running multi-modal comprehension and generation models. It orchestrates multiple diff --git a/vllm_omni/entrypoints/cli/__init__.py b/vllm_omni/entrypoints/cli/__init__.py index 2df4defa7..b233a71e6 100644 --- a/vllm_omni/entrypoints/cli/__init__.py +++ b/vllm_omni/entrypoints/cli/__init__.py @@ -1,4 +1,4 @@ -"""CLI helpers for vLLM-omni entrypoints.""" +"""CLI helpers for vLLM-Omni entrypoints.""" from .serve import OmniServeCommand diff --git a/vllm_omni/entrypoints/cli/main.py b/vllm_omni/entrypoints/cli/main.py index 148839d75..9647484f8 100644 --- a/vllm_omni/entrypoints/cli/main.py +++ b/vllm_omni/entrypoints/cli/main.py @@ -1,5 +1,5 @@ """ -CLI entry point for vLLM-omni that intercepts vLLM commands. +CLI entry point for vLLM-Omni that intercepts vLLM commands. """ import importlib.metadata diff --git a/vllm_omni/entrypoints/cli/serve.py b/vllm_omni/entrypoints/cli/serve.py index 25746cb5e..4606029ab 100644 --- a/vllm_omni/entrypoints/cli/serve.py +++ b/vllm_omni/entrypoints/cli/serve.py @@ -1,5 +1,5 @@ """ -Omni serve command for vLLM-omni. +Omni serve command for vLLM-Omni. """ import argparse @@ -52,7 +52,7 @@ class OmniServeCommand(CLISubcommand): serve_parser.add_argument( "--omni", action="store_true", - help="Enable vLLM-omni mode", + help="Enable vLLM-Omni mode", ) serve_parser.add_argument( "--stage-configs-path", diff --git a/vllm_omni/entrypoints/omni_llm.py b/vllm_omni/entrypoints/omni_llm.py index dccc380ae..0a04ab406 100644 --- a/vllm_omni/entrypoints/omni_llm.py +++ b/vllm_omni/entrypoints/omni_llm.py @@ -42,7 +42,7 @@ logger = init_logger(__name__) class OmniLLM: - """Main entry point for vLLM-omni inference. + """Main entry point for vLLM-Omni inference. This class provides a high-level interface for running multi-modal comprehension and generation models. It orchestrates multiple diff --git a/vllm_omni/entrypoints/omni_stage.py b/vllm_omni/entrypoints/omni_stage.py index c52f77043..624c943fe 100644 --- a/vllm_omni/entrypoints/omni_stage.py +++ b/vllm_omni/entrypoints/omni_stage.py @@ -1,5 +1,5 @@ """ -Stage manager for orchestrating multiple engines in vLLM-omni. +Stage manager for orchestrating multiple engines in vLLM-Omni. Enhanced to encapsulate per-stage process lifecycle and worker logic (device setup, LLM init, batching, shared-memory IPC), while preserving diff --git a/vllm_omni/worker/gpu_ar_model_runner.py b/vllm_omni/worker/gpu_ar_model_runner.py index 7b793214b..5830a3adc 100644 --- a/vllm_omni/worker/gpu_ar_model_runner.py +++ b/vllm_omni/worker/gpu_ar_model_runner.py @@ -1,4 +1,4 @@ -"""AR GPU Model Runner for vLLM-omni. +"""AR GPU Model Runner for vLLM-Omni. Exposes per-request hidden representations via ModelRunnerOutput.pooler_output and also outputs sampled tokens. diff --git a/vllm_omni/worker/gpu_diffusion_model_runner.py b/vllm_omni/worker/gpu_diffusion_model_runner.py index d381db8d1..401ff91bd 100644 --- a/vllm_omni/worker/gpu_diffusion_model_runner.py +++ b/vllm_omni/worker/gpu_diffusion_model_runner.py @@ -31,7 +31,7 @@ logger = logging.getLogger(__name__) class GPUDiffusionModelRunner(OmniGPUModelRunner): - """Diffusion model runner for vLLM-omni (non-autoregressive). + """Diffusion model runner for vLLM-Omni (non-autoregressive). - Reuses GPUModelRunner preparation, multimodal handling, and TP/PP/DP glue. - Does not compute logits or perform token sampling.