anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Rebase] Align vLLM-Omni with vLLM v0.31.0 (#8459)

Browse Source 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
This commit is contained in:
Zhou Taichang authored and GitHub committed 2026-10-05 14:11:01 +08:00
1 parent 1f47fa2f29
commit 0b0d2d7009
141 files changed
+2885 -721
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/getting_started/installation/gpu/cuda.inc.md
+18 -11
View File
@@ -1,11 +1,13 @@
# --8<-- [start:requirements]
- GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
- GPU: compute capability 7.5 or higher (e.g., T4, RTX20xx, A100, L4, H100, etc.)
# --8<-- [end:requirements]
# --8<-- [start:set-up-using-python]
vLLM-Omni depends on the matching major/minor release of vLLM. The vLLM-Omni 0.30.x release line uses vLLM 0.30.x.
vLLM-Omni depends on the matching major/minor release of vLLM. This source
checkout targets vLLM 0.31.0. The pre-built Omni wheel instructions below install
the published 0.30.0 release with its matching vLLM dependency.
!!! note
PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
@@ -41,13 +43,17 @@ uv pip install 'vllm-omni[demo]==0.30.0'
# --8<-- [start:build-wheel-from-source]
#### Installation of vLLM
If you do not need to modify source code of vLLM, you can directly install the stable 0.30.0 release version of the library
If you do not need to modify source code of vLLM, install the final 0.31.0
release wheel from its commit-pinned index:
```bash
uv pip install vllm==0.30.0 --torch-backend=auto
uv pip install vllm==0.31.0 --torch-backend=auto \
--extra-index-url https://wheels.vllm.ai/db9527a46873454610df6dbedf79a36d6bf1a7f6
```
The 0.30.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.
The default 0.31.0 wheel contains CUDA 12.9 binaries. If you need a different
CUDA variant or want to reuse an existing PyTorch installation, build vLLM
from source instead.
#### Installation of vLLM-Omni
Since vllm-omni is rapidly evolving, it's recommended to install it from source
@@ -68,16 +74,16 @@ If you want to check, modify or debug with source code of vLLM, install the libr
```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.30.0
git checkout v0.31.0
```
Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set `VLLM_PRECOMPILED_WHEEL_LOCATION` as your local absolute path of whl file.
```bash
#For CUDA 13.0 (the default for v0.30.0; the wheel filename has no `+cu130` suffix)
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.30.0/vllm-0.30.0-cp38-abi3-manylinux_2_28_x86_64.whl
# Default CUDA 12.9 wheel for v0.31.0 on x86_64
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/db9527a46873454610df6dbedf79a36d6bf1a7f6/vllm-0.31.0-cp38-abi3-manylinux_2_28_x86_64.whl
```
Install vllm with command below (If you have no existing PyTorch).
```bash
uv pip install --editable .
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
```
Install vllm with command below (If you already have PyTorch).
```bash
@@ -128,12 +134,13 @@ docker run --runtime nvidia --gpus 2 \
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.cuda -t vllm-omni-cuda .
```
If you want to specify the base vLLM version:
The source build defaults to `vllm/vllm-openai:v0.31.0`. To select that
upstream release image explicitly:
```bash
DOCKER_BUILDKIT=1 docker build \
-f docker/Dockerfile.cuda \
--build-arg BASE_IMAGE=vllm/vllm-openai:v0.30.0 \
--build-arg BASE_IMAGE=vllm/vllm-openai:v0.31.0 \
-t vllm-omni-cuda .
```
docs/getting_started/installation/gpu/rocm.inc.md
+10 -8
View File
@@ -10,8 +10,9 @@
For ROCm, vLLM-Omni currently recommends the setup steps through Docker Images.
vLLM-Omni depends on the matching major/minor release of vLLM. The vLLM-Omni
0.30.x release line uses vLLM 0.30.x.
vLLM-Omni depends on the matching major/minor release of vLLM. This source
checkout targets vLLM 0.31.0. The pre-built Omni wheel instructions below install
the published 0.30.0 release with its matching vLLM dependency.
The Dockerfile's `BASE_IMAGE` pin applies only to Docker builds. The
`vllm-omni` package does not install vLLM as a dependency, so non-Docker source
@@ -49,13 +50,14 @@ uv pip install onnxruntime-rocm
#### Installation of vLLM
If you do not need to modify source code of vLLM, you can directly install the stable 0.30.0 release version of the library
If you do not need to modify source code of vLLM, install the final 0.31.0
release wheel for ROCm 7.2.3:
```bash
uv pip install vllm==0.30.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723
uv pip install vllm==0.31.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.31.0/rocm723
```
The pre-built 0.30.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
The pre-built 0.31.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
#### Installation of vLLM-Omni
@@ -75,7 +77,7 @@ If you want to check, modify or debug with source code of vLLM, install the libr
```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.30.0
git checkout v0.31.0
python3 -m pip install -r requirements/rocm.txt
python3 setup.py develop
```
@@ -87,7 +89,7 @@ python3 setup.py develop
#### Build docker image
The source build defaults to the published upstream base image
`vllm/vllm-openai-rocm:v0.30.0`, aligned with the vLLM release used by CI.
`vllm/vllm-openai-rocm:v0.31.0`, aligned with this source checkout.
Older or custom bases must provide the vLLM APIs checked by the Dockerfile's
image-build canary. This upstream base is distinct from the prebuilt
`vllm/vllm-omni-rocm` images discussed below; their published-tag availability
@@ -102,7 +104,7 @@ To select the upstream ROCm base explicitly, pass `BASE_IMAGE`:
```bash
DOCKER_BUILDKIT=1 docker build \
-f docker/Dockerfile.rocm \
--build-arg BASE_IMAGE=vllm/vllm-openai-rocm:v0.30.0 \
--build-arg BASE_IMAGE=vllm/vllm-openai-rocm:v0.31.0 \
-t vllm-omni-rocm .
```
docs/getting_started/installation/gpu/xpu.inc.md
+7 -3
View File
@@ -20,12 +20,16 @@ vLLM-Omni currently recommends using the Docker image setup steps below.
#### Build docker image
```bash
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.xpu -t vllm-omni-xpu --shm-size=4g .
export VLLM_VERSION=v0.31.0
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.xpu -t vllm-omni-xpu --shm-size=4g \
--build-arg VLLM_VERSION="$VLLM_VERSION" .
```
This layers vLLM-Omni on top of the published `vllm/vllm-openai-xpu:<VLLM_VERSION>`
base image (v0.30.0 by default), which Docker pulls automatically. To target a different vLLM release,
pass `--build-arg VLLM_VERSION=<tag>`. If that tag has not been published yet, build
base image, which Docker pulls automatically. The Dockerfile still defaults to
v0.30.0; the command above explicitly selects v0.31.0 to match this source
checkout. To target a different vLLM release, pass
`--build-arg VLLM_VERSION=<tag>`. If that tag has not been published yet, build
the base from upstream's own Dockerfile first and point the build at it:
```bash