Compare commits

..
Author SHA1 Message Date
tzhouam 7f2035634b Merge branch 'main' into dev/rebase-v0.15.0 2026-02-03 05:09:10 +00:00
tzhouam 4cbde7a460 Update Dockerfile.ci to improve installation commands and ensure Python environment consistency
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-02-03 05:06:51 +00:00
Taichang Zhou e1026a6bc0 Update according to Cyrus comments
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
2026-02-02 23:42:55 +08:00
Taichang Zhou 2eda72b5dc Refactor comments and formatting in Qwen3 TTS configuration and modeling files for improved readability.
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
2026-02-02 21:51:01 +08:00
Taichang Zhou 22ca74432b Merge branch 'main' into dev/vllm-align 2026-02-02 21:35:45 +08:00
Taichang Zhou 354777d5e7 Merge branch 'main' of https://github.com/vllm-project/vllm-omni 2026-02-02 21:34:42 +08:00
汪志鹏 b6c846cc95 Add stage_init_timeout=300 (#1153)
Signed-off-by: princepride <wangzhipeng628@gmail.com>
2026-02-02 17:58:48 +08:00
tzhouam 9194c1dd84 Update base image in Dockerfile.rocm from v0.14.0 to v0.15.0
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-02-02 08:41:35 +00:00
tzhouam ce9105267f Enhance error handling in TTS generation and improve audio output compatibility by supporting both "audio" and "model_outputs" keys in response.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-02-02 08:40:56 +00:00
tzhouam 39f7cade27 Fix audio sample rate access in end2end.py for correct output retrieval
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-02-02 08:38:24 +00:00
tzhouam 08fa85a1ad Update audio tensor access in offline inference tests and increase image fetch timeout for CI
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-02-02 06:10:54 +00:00
tzhouam 6568a78a26 Remove unused tokenizer argument from OmniInputProcessor class docstring
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-02-02 03:12:43 +00:00
tzhouam ff8b3dbbc8 update qwen 2.5 Omni
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-02-02 03:01:11 +00:00
tzhouam 1297a46040 Merge remote-tracking branch 'origin/main' into dev/vllm-align
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-02-02 01:54:12 +00:00
Taichang Zhou e033498600 Merge branch 'main' of https://github.com/vllm-project/vllm-omni 2026-01-31 10:59:36 +08:00
tzhouam 7aea8e8d91 Add OmniMRotaryEmbedding for multimodal position computation
- Introduced OmniMRotaryEmbedding to extend MRotaryEmbedding with methods for computing input positions across various multimodal scenarios, including image, video, and audio inputs.
- Updated patch.py to replace the original MRotaryEmbedding with OmniMRotaryEmbedding in the module patching logic.
- Created new files for the OmniMRotaryEmbedding implementation and its associated methods.

Signed-off-by: [Your Name] <[Your Email]>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-30 10:03:02 +00:00
tzhouam 3208afd82b Refactor OmniRequestOutput to encapsulate multimodal output
- Changed the multimodal_output field to a private attribute _multimodal_output for better encapsulation.
- Added a property method to access multimodal_output, allowing for a clearer interface and improved handling of request outputs.

Signed-off-by: [Your Name] <[Your Email]>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-30 10:02:22 +00:00
tzhouam 2b009b3cef Fix audio tensor extraction in end2end.py files across multiple examples
- Updated the audio tensor extraction logic to access the correct output structure in Qwen2.5 Omni, Qwen3 Omni, and Qwen3 TTS implementations.
- Ensured consistency in handling audio outputs for improved functionality.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-30 10:00:14 +00:00
tzhouam 4faf190e28 Merge remote-tracking branch 'origin/main' into dev/vllm-align 2026-01-30 03:49:04 +00:00
tzhouam ca57f510c7 Refactor OmniInputPreprocessor to remove unused embed processing logic
- Eliminated the _process_embeds method and related imports from preprocess.py, streamlining the input preprocessing workflow.
- Updated the main processing logic to focus solely on token inputs, enhancing clarity and maintainability.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-30 03:02:38 +00:00
tzhouam 4b895459e4 Merge branch 'main' into dev/vllm-align
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-30 02:53:03 +00:00
tzhouam 62dbf07fb6 Refactor OpenAI serving modules for improved functionality and organization
- Reintroduced the load_chat_template import in api_server.py for better template handling.
- Updated the logic in serving_chat.py to streamline the resolution of chat template content formats, enhancing clarity and functionality.
- Adjusted the application of chat templates to ensure proper handling of tools and chat templates.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-30 02:44:13 +00:00
tzhouam 104e105c93 Refactor multimodal processing imports and update configuration handling in Qwen3OmniMoeThinker
- Changed the import path for multimodal processing to improve module organization.
- Removed the unused multimodal configuration parameter from the Qwen3OmniMoeThinker initialization, streamlining the class definition.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-30 02:43:41 +00:00
tzhouam ef21879c3a Refactor GPU model runner to streamline multimodal configuration handling
- Removed the unused import of supports_mm_encoder_only from interfaces.
- Updated the dummy run check to utilize the multimodal configuration directly, enhancing clarity and future extensibility.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-30 02:43:26 +00:00
tzhouam 45880a30f0 Refactor chat_utils and OpenAI modules for improved functionality and organization
- Updated import statements in chat_utils.py to replace the internal content format type with the public one for better clarity.
- Refactored api_server.py to streamline the chat template loading process by replacing the previous method with a direct function call.
- Enhanced serving_chat.py by reintroducing safe_apply_chat_template functions for better template handling.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-29 09:00:43 +00:00
tzhouam 2ac33489b5 Update Dockerfile.ci to install vllm package using the correct URL format
- Modified the pip install command for vllm to use the correct URL format for direct installation from the specified commit hash, ensuring a more reliable installation process.
- Removed the fallback option for pre-release installation to streamline the installation command.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-29 08:25:51 +00:00
tzhouam 63fd163efb Refactor import statement in OpenAI API server for improved module organization
- Updated the import path for AnthropicServingMessages in api_server.py to enhance clarity and maintain consistency with the new module structure.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-29 08:23:17 +00:00
tzhouam fea328364a Update Dockerfile.ci to install vllm package directly from wheel file
- Changed the pip install command for vllm to download the wheel file directly instead of using an extra index URL, ensuring a more reliable installation process.
- Added a fallback option to install the pre-release version of vllm if the direct installation fails.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-29 08:05:57 +00:00
tzhouam 478c036a01 [Refactor] Update OpenAI API server and related modules for improved structure and functionality
- Refactored import statements in api_server.py to enhance module organization, including changes to the routing and server setup.
- Updated the method for removing routes to ensure compatibility with the new app structure.
- Adjusted the handling of chat completion requests to improve error responses and streamline processing.
- Modified serving modules to align with the new import paths and enhance code clarity.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-29 15:28:51 +08:00
tzhouam 8407818368 Update simple_test.sh to force reinstall vllm package with extra index URL
- Modified the pip install command for vllm to include --upgrade, --force-reinstall flags, and an extra index URL, ensuring the latest version is installed from the specified source during testing.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 17:34:22 +08:00
tzhouam 6e7cb01afe Update Dockerfile.ci to force reinstall vllm package during CI build
- Modified the pip install command for vllm to include --upgrade and --force-reinstall flags, ensuring the latest version is installed during the CI process.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 17:20:29 +08:00
tzhouam acb28f3d1d Remove commented-out import and related code for MRotaryEmbedding in patch.py to clean up the codebase.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 16:52:10 +08:00
tzhouam bf51318176 debug
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 16:40:06 +08:00
tzhouam bd8c0443cf build image from commit id
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 16:21:29 +08:00
tzhouam 554d14fd28 debug docker file
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 15:57:13 +08:00
tzhouam 49b9d0d6b1 debug docker build
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 15:47:30 +08:00
tzhouam 2c2d06d920 [Enhancement] Update GPUARWorker and GPUGenerationWorker to enforce v1 model runner
- Added a warning in GPUARWorker and GPUGenerationWorker to indicate that the v2 model runner does not include omni hooks, forcing the use of the v1 model runner.
- Updated import statements in gpu_ar_model_runner.py and gpu_generation_model_runner.py to include necessary functions from parallel_state.
- Removed unused imports from both model runner files to streamline the code.

Signed-off-by: [Your Name] <your.email@example.com>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 15:43:27 +08:00
tzhouam 31d127edcd [Refactor] Remove unused MRotaryEmbedding and vision position ID functions
- Deleted the MRotaryEmbedding class and its associated methods from mrope.py to streamline the codebase.
- Removed the get_llm_pos_ids_for_vision function from vision.py as it is no longer needed.
- Updated import statements in qwen2_5_omni.py to reflect the removal of the vision module.

Signed-off-by: [Your Name] <your.email@example.com>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 15:42:35 +08:00
tzhouam 52d0122fb5 [Enhancement] Refactor OmniInputProcessor and related classes for improved prompt handling
- Updated OmniInputProcessor to streamline input processing, including enhanced handling of prompt embeddings and additional information.
- Introduced OmniTextPrompt and OmniEmbedsInputs to support new input structures with optional metadata.
- Modified input preprocessing methods to accommodate new prompt types and ensure compatibility with multimodal inputs.
- Improved error handling and type checking for data parallel configurations.

Signed-off-by: [Your Name] <your.email@example.com>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 15:42:05 +08:00
tzhouam b26a9735e6 [Enhancement] Update type hint for stage_connector_spec in OmniEngineArgs and AsyncOmniEngineArgs
- Changed type hint for stage_connector_spec from 'any' to 'Any' for improved type checking and consistency.
- Ensured compatibility with typing standards in the arg_utils module.

Signed-off-by: [Your Name] <your.email@example.com>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 15:41:20 +08:00
tzhouam d7cfe195ea [Enhancement] Refactor OmniARScheduler and OmniGenerationScheduler to include OmniSchedulerOutput
- Updated OmniARScheduler to wrap output in OmniSchedulerOutput, including transfer metadata for finished requests needing KV cache transfer.
- Enhanced OmniGenerationScheduler to handle routed experts and finish reasons more effectively, ensuring proper output structure.
- Introduced performance statistics handling in both schedulers for improved monitoring.

Signed-off-by: [Your Name] <your.email@example.com>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 15:40:59 +08:00
tzhouam e1c6639f18 [Enhancement] Add mm_encoder_only parameter and deprecation warning for convert option
- Introduced mm_encoder_only parameter to OmniModelConfig.
- Added a deprecation warning for the `--convert mm_encoder_only` option, advising users to switch to `--mm-encoder-only`.

Signed-off-by: [Your Name] <your.email@example.com>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 15:40:29 +08:00
tzhouam ce7430b562 [Enhancement] Update OmniRequest to support torch.Tensor for prompt_embeds and add decoding method
- Modified the constructor to accept torch.Tensor as an option for prompt_embeds.
- Introduced a new static method _maybe_decode_prompt_embeds to handle the conversion from PromptEmbedsPayload to torch.Tensor.
- Preserved the original prompt_embeds payload for reference.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 15:40:17 +08:00
tzhouam ed8b03224b update the docker setup to vllm nightly
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
2026-01-28 11:15:54 +08:00
Taichang Zhou af8501dcc7 debug Qwen TTS
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
2026-01-22 22:46:06 +08:00
5 changed files with 14 additions and 27 deletions

No files matched your search

+1 -15
View File
@@ -15,8 +15,7 @@ Easy, fast, and cheap omni-modality model serving for everyone
---
*Latest News* 🔥
- [2026/02] We released [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0) - This is the first **stable release** of vLLM-Omni that expands Omni’s diffusion / image-video generation and audio / TTS stack, improves distributed execution and memory efficiency, and broadens platform/backend coverage (GPU/ROCm/NPU/XPU). It also brings meaningful upgrades to serving APIs, profiling & benchmarking, and overall stability. Please check our latest [paper](https://arxiv.org/abs/2602.02204) for architecture design and performance results.
- [2026/02] We released [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0) - This is the first **stable release** of vLLM-Omni that expands Omni’s diffusion / image-video generation and audio / TTS stack, improves distributed execution and memory efficiency, and broadens platform/backend coverage (GPU/ROCm/NPU/XPU). It also brings meaningful upgrades to serving APIs, profiling & benchmarking, and overall stability.
- [2026/01] We released [0.12.0rc1](https://github.com/vllm-project/vllm-omni/releases/tag/v0.12.0rc1) - a major RC milestone focused on maturing the diffusion stack, strengthening OpenAI-compatible serving, expanding omni-model coverage, and improving stability across platforms (GPU/NPU/ROCm), please check our latest [design](https://docs.google.com/presentation/d/1qv4qMW1rKAqDREMXiUDLIgqqHQe7TDPj/edit?usp=sharing&ouid=110473603432222024453&rtpof=true&sd=true).
- [2025/11] vLLM community officially released [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) in order to support omni-modality models serving.
@@ -68,19 +67,6 @@ Visit our [documentation](https://vllm-omni.readthedocs.io/en/latest/) to learn
We welcome and value any contributions and collaborations.
Please check out [Contributing to vLLM-Omni](https://vllm-omni.readthedocs.io/en/latest/contributing/) for how to get involved.
## Citation
If you use vLLM-Omni for your research, please cite our [paper](https://arxiv.org/abs/2602.02204):
```bibtex
@article{yin2026vllmomni,
title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models},
author={Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu},
journal={arXiv preprint arXiv:2602.02204},
year={2026}
}
```
## Join the Community
Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in `#sig-omni` slack channel at [slack.vllm.ai](https://slack.vllm.ai) or vLLM user forum at [discuss.vllm.ai](https://discuss.vllm.ai).
+9 -8
View File
@@ -3,18 +3,19 @@ ARG VLLM_BASE_TAG=v0.15.0
FROM ${VLLM_BASE_IMAGE}:${VLLM_BASE_TAG}
ARG APP_DIR=/workspace/vllm-omni
WORKDIR ${APP_DIR}
COPY . .
# Install system dependencies
RUN apt-get update && \
apt-get install -y ffmpeg && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
apt-get install -y ffmpeg && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# Install vLLM wheel for the specific commit
# URL format: https://wheels.vllm.ai/<commit_hash>/
# Current vLLM commit: f176443446f659dbab5315e056e605d8984fd976 (v0.15.0)
RUN uv pip install --system --upgrade --force-reinstall vllm --extra-index-url https://wheels.vllm.ai/f176443446f659dbab5315e056e605d8984fd976
# Install vllm-omni into the same uv-managed Python environment used by the base image.
RUN uv pip install --python "$(python3 -c 'import sys; print(sys.executable)')" --no-cache-dir ".[dev]"
RUN uv pip install --system --no-cache-dir ".[dev]"
RUN ln -sf /usr/bin/python3 /usr/bin/python
ENTRYPOINT []
+1 -1
View File
@@ -253,7 +253,7 @@ class DiffusionEngine:
try:
# Give worker enough time — export + compression + table can be slow
results = self.collective_rpc(method="stop_profile", timeout=600)
results = self.collective_rpc(method="stop_profile", timeout=60000)
except Exception:
logger.error("Failed to stop profiling on workers", exc_info=True)
return {"traces": [], "tables": []}
+1 -1
View File
@@ -97,7 +97,7 @@ class OmniServeCommand(CLISubcommand):
omni_config_group.add_argument(
"--init-timeout",
type=int,
default=600,
default=60000,
help="The timeout for initializing the stages.",
)
omni_config_group.add_argument(
+2 -2
View File
@@ -221,8 +221,8 @@ class OmniStage:
# Wait for result from worker
try:
# Profiling stop might take time to flush files, give it 600s
response = self._out_q.get(timeout=600)
# Profiling stop might take time to flush files, give it 180s
response = self._out_q.get(timeout=60000)
if isinstance(response, dict):
if response.get("type") == "profiler_result":