Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
765caa2a12 | ||
|
|
75fdab6b35 | ||
|
|
e261962661 |
No files matched your search
@@ -16,14 +16,10 @@ Easy, fast, and cheap omni-modality model serving for everyone
|
|||||||
---
|
---
|
||||||
|
|
||||||
*Latest News* 🔥
|
*Latest News* 🔥
|
||||||
|
- [2026/08] We released [0.26.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.26.0) - aligned with the vLLM 0.26 release line, featuring [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) joint video/audio generation, an experimental full-duplex realtime runtime for [MiniCPM-o 4.5](recipes/OpenBMB/MiniCPM-o-4_5.md), distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support.
|
||||||
- [2026/07] We released [0.24.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.24.0) - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
|
- [2026/07] We released [0.24.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.24.0) - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
|
||||||
- [2026/06] We released [0.22.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.22.0) - an **omnimodal world-model** release aligned with vLLM 0.22, featuring [Nvidia Cosmos3](recipes/cosmos3/Cosmos3-Nano.md)/DreamZero world model support, expanded quantization coverage across Blackwell/NPU/XPU, TTS production improvements, new models including MiniCPM-o 4.5, MOSS-TTS, and Lance, plus RL integration with [VeRL-Omni](https://github.com/verl-project/verl-omni).
|
- [2026/06] Starting with [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0), vLLM-Omni publishes a stable release aligned with every even-numbered upstream vLLM minor version. [0.16.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.16.0), [0.18.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.18.0), [0.20.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.20.0), and [0.22.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.22.0) continued this cadence, expanding omni and world-model support with [NVIDIA Cosmos3](recipes/cosmos3/Cosmos3-Nano.md) and DreamZero, adding models such as MiniCPM-o 4.5, MOSS-TTS, and Lance, and advancing TTS, diffusion, distributed execution, quantization, RL integration through [VeRL-Omni](https://github.com/verl-project/verl-omni), and CUDA/ROCm/MUSA/NPU/XPU coverage.
|
||||||
- [2026/05] We released [0.20.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.20.0) - refreshes the serving/runtime stack for large-scale omni workloads, and improves diffusion model performance, quantization, and hardware readiness across CUDA, ROCm, MUSA, NPU, and XPU backends.
|
|
||||||
- [2026/03] We released [0.18.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.18.0) - strengthens the core runtime through a large entrypoint refactor and scheduler/runtime cleanups, expands unified quantization and diffusion execution, broadens multimodal model coverage, and improves production readiness across audio, omni, image, video, RL, and multi-platform deployments.
|
|
||||||
- [2026/03] Check out our first public [project deepdive](https://youtu.be/sgwNfsNnR9I) at the vLLM Hong Kong Meetup!
|
- [2026/03] Check out our first public [project deepdive](https://youtu.be/sgwNfsNnR9I) at the vLLM Hong Kong Meetup!
|
||||||
- [2026/03] **[vllm-omni-skills](https://github.com/hsliuustc0106/vllm-omni-skills)** is a community-driven collection of AI assistant skills that help developers work with vLLM-Omni more effectively. These skills can be used with popular agentic AI coding assistants like **Cursor IDE**, **Claude**, **Codex**, and more.
|
|
||||||
- [2026/02] We released [0.16.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.16.0) - A major alignment + capability release that rebases onto **upstream vLLM v0.16.0** and significantly expands performance, distributed execution, and production readiness across **Qwen3-Omni / Qwen3-TTS**, **Bagel**, **MiMo-Audio**, **GLM-Image** and the **Diffusion (DiT) image/video stack**—while also improving platform coverage (CUDA / ROCm / NPU / XPU), CI quality, and documentation.
|
|
||||||
- [2026/02] We released [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0) - This is the first **stable release** of vLLM-Omni that expands Omni’s diffusion / image-video generation and audio / TTS stack, improves distributed execution and memory efficiency, and broadens platform/backend coverage (GPU/ROCm/NPU/XPU). It also brings meaningful upgrades to serving APIs, profiling & benchmarking, and overall stability. Please check our latest [paper](https://arxiv.org/abs/2602.02204) for architecture design and performance results.
|
|
||||||
- [2025/11] vLLM community officially released [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) in order to support omni-modality models serving.
|
- [2025/11] vLLM community officially released [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) in order to support omni-modality models serving.
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -55,12 +51,13 @@ vLLM-Omni is flexible and easy to use with:
|
|||||||
- Tensor, pipeline, data and expert parallelism support for distributed inference
|
- Tensor, pipeline, data and expert parallelism support for distributed inference
|
||||||
- Streaming outputs
|
- Streaming outputs
|
||||||
- OpenAI-compatible API server
|
- OpenAI-compatible API server
|
||||||
|
- Full-duplex realtime serving with streaming audio input and output (experimental)
|
||||||
|
|
||||||
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
|
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
|
||||||
|
|
||||||
- **Omni-modality models** (e.g. Qwen3-Omni, Cosmos3, HunyuanImage, BAGEL)
|
- **Omni-modality models** (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
|
||||||
- **TTS models** (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
|
- **TTS models** (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
|
||||||
- **Diffusion models** — image, video, and audio generation (e.g. Qwen-Image, Wan2.2, FLUX)
|
- **Diffusion models** — image, video, and audio generation (e.g. MiniMax H3, Qwen-Image, Wan2.2, FLUX)
|
||||||
- **Robot-policy and action models** (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)
|
- **Robot-policy and action models** (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|||||||
+3
-2
@@ -52,12 +52,13 @@ vLLM-Omni is flexible and easy to use with:
|
|||||||
- Tensor, pipeline, data and expert parallelism support for distributed inference
|
- Tensor, pipeline, data and expert parallelism support for distributed inference
|
||||||
- Streaming outputs
|
- Streaming outputs
|
||||||
- OpenAI-compatible API server
|
- OpenAI-compatible API server
|
||||||
|
- Full-duplex realtime serving with streaming audio input and output (experimental)
|
||||||
|
|
||||||
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
|
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
|
||||||
|
|
||||||
- Omni-modality models (e.g. Qwen3-Omni, Cosmos3, HunyuanImage, BAGEL)
|
- Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
|
||||||
- TTS models (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
|
- TTS models (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
|
||||||
- Diffusion models — image, video, and audio generation (e.g. Qwen-Image, Wan2.2, FLUX)
|
- Diffusion models — image, video, and audio generation (e.g. MiniMax H3, Qwen-Image, Wan2.2, FLUX)
|
||||||
- Robot-policy and action models (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)
|
- Robot-policy and action models (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)
|
||||||
|
|
||||||
For more information, checkout the following:
|
For more information, checkout the following:
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
This section lists the most common options for running vLLM-Omni.
|
This section lists the most common options for running vLLM-Omni.
|
||||||
|
|
||||||
For options within a vLLM Engine. Please refer to [vLLM Configuration](https://docs.vllm.ai/en/v0.16.0/configuration/index.html)
|
For options within a vLLM Engine, please refer to the [vLLM 0.26 configuration guide](https://docs.vllm.ai/en/v0.26.0/configuration/index.html).
|
||||||
|
|
||||||
Currently, the main options are maintained by stage configs for each model.
|
Currently, the main options are maintained by stage configs for each model.
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,8 @@
|
|||||||
# Installation
|
# Installation
|
||||||
|
|
||||||
|
!!! important
|
||||||
|
vLLM-Omni is released against the matching upstream vLLM major/minor version. vLLM-Omni 0.26.x requires vLLM 0.26.x; the stable v0.26.0 instructions pin vLLM 0.26.0.
|
||||||
|
|
||||||
vLLM-Omni supports the following hardware platforms:
|
vLLM-Omni supports the following hardware platforms:
|
||||||
|
|
||||||
- [GPU](gpu.md)
|
- [GPU](gpu.md)
|
||||||
|
|||||||
@@ -5,7 +5,7 @@
|
|||||||
# --8<-- [end:requirements]
|
# --8<-- [end:requirements]
|
||||||
# --8<-- [start:set-up-using-python]
|
# --8<-- [start:set-up-using-python]
|
||||||
|
|
||||||
vLLM-Omni depends vLLM. So please follow instructions below mainly for vLLM.
|
vLLM-Omni depends on the matching major/minor release of vLLM. The vLLM-Omni 0.26.x release line uses vLLM 0.26.x.
|
||||||
|
|
||||||
!!! note
|
!!! note
|
||||||
PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
|
PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
|
||||||
@@ -124,7 +124,7 @@ If you want to specify the base vLLM version:
|
|||||||
```bash
|
```bash
|
||||||
DOCKER_BUILDKIT=1 docker build \
|
DOCKER_BUILDKIT=1 docker build \
|
||||||
-f docker/Dockerfile.cuda \
|
-f docker/Dockerfile.cuda \
|
||||||
--build-arg BASE_IMAGE=vllm/vllm-openai:v0.22.1 \
|
--build-arg BASE_IMAGE=vllm/vllm-openai:v0.26.0 \
|
||||||
-t vllm-omni-cuda .
|
-t vllm-omni-cuda .
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -53,16 +53,20 @@ We are keeping [issue #886](https://github.com/vllm-project/vllm-omni/issues/886
|
|||||||
You can also build vLLM-Omni from the latest main branch if you want to use the latest features or bug fixes. (But sometimes it will break for a while. You can check [issue #886](https://github.com/vllm-project/vllm-omni/issues/886) for the status of the latest commit of vLLM-Omni main branch on NPU.)
|
You can also build vLLM-Omni from the latest main branch if you want to use the latest features or bug fixes. (But sometimes it will break for a while. You can check [issue #886](https://github.com/vllm-project/vllm-omni/issues/886) for the status of the latest commit of vLLM-Omni main branch on NPU.)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Pin vLLM version to 0.18.0
|
# Pin vLLM and vLLM-Ascend to the v0.26 release line
|
||||||
git clone -b v0.18.0 https://github.com/vllm-project/vllm.git
|
git clone -b v0.26.0 https://github.com/vllm-project/vllm.git
|
||||||
|
cd vllm
|
||||||
VLLM_TARGET_DEVICE=empty pip install -v -e .
|
VLLM_TARGET_DEVICE=empty pip install -v -e .
|
||||||
|
cd ..
|
||||||
|
|
||||||
git clone -b v0.18.0rc1 https://github.com/vllm-project/vllm-ascend.git
|
git clone -b releases/v0.26.0rc https://github.com/vllm-project/vllm-ascend.git
|
||||||
|
cd vllm-ascend
|
||||||
pip install -v -e .
|
pip install -v -e .
|
||||||
|
cd ..
|
||||||
|
|
||||||
# Install vLLM-Omni from the latest main branch
|
# Install vLLM-Omni from the latest main branch
|
||||||
git clone https://github.com/vllm-project/vllm-omni.git
|
git clone https://github.com/vllm-project/vllm-omni.git
|
||||||
cd /vllm-workspace/vllm-omni
|
cd vllm-omni
|
||||||
pip install -v -e . --no-build-isolation
|
pip install -v -e . --no-build-isolation
|
||||||
# or VLLM_OMNI_TARGET_DEVICE=npu pip install -v -e .
|
# or VLLM_OMNI_TARGET_DEVICE=npu pip install -v -e .
|
||||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||||
|
|||||||
Reference in new issue
Block a user