- Introduced OmniMRotaryEmbedding to extend MRotaryEmbedding with methods for computing input positions across various multimodal scenarios, including image, video, and audio inputs.
- Updated patch.py to replace the original MRotaryEmbedding with OmniMRotaryEmbedding in the module patching logic.
- Created new files for the OmniMRotaryEmbedding implementation and its associated methods.
Signed-off-by: [Your Name] <[Your Email]>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Changed the multimodal_output field to a private attribute _multimodal_output for better encapsulation.
- Added a property method to access multimodal_output, allowing for a clearer interface and improved handling of request outputs.
Signed-off-by: [Your Name] <[Your Email]>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated the audio tensor extraction logic to access the correct output structure in Qwen2.5 Omni, Qwen3 Omni, and Qwen3 TTS implementations.
- Ensured consistency in handling audio outputs for improved functionality.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Eliminated the _process_embeds method and related imports from preprocess.py, streamlining the input preprocessing workflow.
- Updated the main processing logic to focus solely on token inputs, enhancing clarity and maintainability.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Reintroduced the load_chat_template import in api_server.py for better template handling.
- Updated the logic in serving_chat.py to streamline the resolution of chat template content formats, enhancing clarity and functionality.
- Adjusted the application of chat templates to ensure proper handling of tools and chat templates.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Changed the import path for multimodal processing to improve module organization.
- Removed the unused multimodal configuration parameter from the Qwen3OmniMoeThinker initialization, streamlining the class definition.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Removed the unused import of supports_mm_encoder_only from interfaces.
- Updated the dummy run check to utilize the multimodal configuration directly, enhancing clarity and future extensibility.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated import statements in chat_utils.py to replace the internal content format type with the public one for better clarity.
- Refactored api_server.py to streamline the chat template loading process by replacing the previous method with a direct function call.
- Enhanced serving_chat.py by reintroducing safe_apply_chat_template functions for better template handling.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Modified the pip install command for vllm to use the correct URL format for direct installation from the specified commit hash, ensuring a more reliable installation process.
- Removed the fallback option for pre-release installation to streamline the installation command.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated the import path for AnthropicServingMessages in api_server.py to enhance clarity and maintain consistency with the new module structure.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Changed the pip install command for vllm to download the wheel file directly instead of using an extra index URL, ensuring a more reliable installation process.
- Added a fallback option to install the pre-release version of vllm if the direct installation fails.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Refactored import statements in api_server.py to enhance module organization, including changes to the routing and server setup.
- Updated the method for removing routes to ensure compatibility with the new app structure.
- Adjusted the handling of chat completion requests to improve error responses and streamline processing.
- Modified serving modules to align with the new import paths and enhance code clarity.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Modified the pip install command for vllm to include --upgrade, --force-reinstall flags, and an extra index URL, ensuring the latest version is installed from the specified source during testing.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Modified the pip install command for vllm to include --upgrade and --force-reinstall flags, ensuring the latest version is installed during the CI process.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Added a warning in GPUARWorker and GPUGenerationWorker to indicate that the v2 model runner does not include omni hooks, forcing the use of the v1 model runner.
- Updated import statements in gpu_ar_model_runner.py and gpu_generation_model_runner.py to include necessary functions from parallel_state.
- Removed unused imports from both model runner files to streamline the code.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Deleted the MRotaryEmbedding class and its associated methods from mrope.py to streamline the codebase.
- Removed the get_llm_pos_ids_for_vision function from vision.py as it is no longer needed.
- Updated import statements in qwen2_5_omni.py to reflect the removal of the vision module.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated OmniInputProcessor to streamline input processing, including enhanced handling of prompt embeddings and additional information.
- Introduced OmniTextPrompt and OmniEmbedsInputs to support new input structures with optional metadata.
- Modified input preprocessing methods to accommodate new prompt types and ensure compatibility with multimodal inputs.
- Improved error handling and type checking for data parallel configurations.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Changed type hint for stage_connector_spec from 'any' to 'Any' for improved type checking and consistency.
- Ensured compatibility with typing standards in the arg_utils module.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated OmniARScheduler to wrap output in OmniSchedulerOutput, including transfer metadata for finished requests needing KV cache transfer.
- Enhanced OmniGenerationScheduler to handle routed experts and finish reasons more effectively, ensuring proper output structure.
- Introduced performance statistics handling in both schedulers for improved monitoring.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Introduced mm_encoder_only parameter to OmniModelConfig.
- Added a deprecation warning for the `--convert mm_encoder_only` option, advising users to switch to `--mm-encoder-only`.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Modified the constructor to accept torch.Tensor as an option for prompt_embeds.
- Introduced a new static method _maybe_decode_prompt_embeds to handle the conversion from PromptEmbedsPayload to torch.Tensor.
- Preserved the original prompt_embeds payload for reference.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
@@ -15,8 +15,7 @@ Easy, fast, and cheap omni-modality model serving for everyone
---
*Latest News* 🔥
- [2026/02] We released [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0) - This is the first **stable release** of vLLM-Omni that expands Omni’s diffusion / image-video generation and audio / TTS stack, improves distributed execution and memory efficiency, and broadens platform/backend coverage (GPU/ROCm/NPU/XPU). It also brings meaningful upgrades to serving APIs, profiling & benchmarking, and overall stability. Please check our latest [paper](https://arxiv.org/abs/2602.02204) for architecture design and performance results.
- [2026/02] We released [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0) - This is the first **stable release** of vLLM-Omni that expands Omni’s diffusion / image-video generation and audio / TTS stack, improves distributed execution and memory efficiency, and broadens platform/backend coverage (GPU/ROCm/NPU/XPU). It also brings meaningful upgrades to serving APIs, profiling & benchmarking, and overall stability.
- [2026/01] We released [0.12.0rc1](https://github.com/vllm-project/vllm-omni/releases/tag/v0.12.0rc1) - a major RC milestone focused on maturing the diffusion stack, strengthening OpenAI-compatible serving, expanding omni-model coverage, and improving stability across platforms (GPU/NPU/ROCm), please check our latest [design](https://docs.google.com/presentation/d/1qv4qMW1rKAqDREMXiUDLIgqqHQe7TDPj/edit?usp=sharing&ouid=110473603432222024453&rtpof=true&sd=true).
- [2025/11] vLLM community officially released [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) in order to support omni-modality models serving.
@@ -68,19 +67,6 @@ Visit our [documentation](https://vllm-omni.readthedocs.io/en/latest/) to learn
We welcome and value any contributions and collaborations.
Please check out [Contributing to vLLM-Omni](https://vllm-omni.readthedocs.io/en/latest/contributing/) for how to get involved.
## Citation
If you use vLLM-Omni for your research, please cite our [paper](https://arxiv.org/abs/2602.02204):
```bibtex
@article{yin2026vllmomni,
title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models},
Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in `#sig-omni` slack channel at [slack.vllm.ai](https://slack.vllm.ai) or vLLM user forum at [discuss.vllm.ai](https://discuss.vllm.ai).
# Profiling stop might take time to flush files, give it 600s
response=self._out_q.get(timeout=600)
# Profiling stop might take time to flush files, give it 180s
response=self._out_q.get(timeout=60000)
ifisinstance(response,dict):
ifresponse.get("type")=="profiler_result":
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.