diff --git a/docs/getting_started/installation/README.md b/docs/getting_started/installation/README.md index 5abd9728b..20a737dca 100644 --- a/docs/getting_started/installation/README.md +++ b/docs/getting_started/installation/README.md @@ -4,4 +4,5 @@ vLLM-Omni supports the following hardware platforms: - [GPU](gpu.md) - [NVIDIA CUDA](gpu.md) + - [AMD ROCm](gpu.md) - [NPU](npu.md) diff --git a/docs/getting_started/installation/gpu.md b/docs/getting_started/installation/gpu.md index 42f716f30..5956ed102 100644 --- a/docs/getting_started/installation/gpu.md +++ b/docs/getting_started/installation/gpu.md @@ -14,6 +14,10 @@ vLLM-Omni is a Python library that supports the following GPU variants. The libr --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:requirements" +=== "AMD ROCm" + + --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:requirements" + ## Set up using Python ### Create a new Python environment @@ -26,6 +30,11 @@ vLLM-Omni is a Python library that supports the following GPU variants. The libr --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:pre-built-wheels" + +=== "AMD ROCm" + + --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:pre-built-wheels" + [](){ #build-from-source } ### Build wheel from source @@ -34,8 +43,22 @@ vLLM-Omni is a Python library that supports the following GPU variants. The libr --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:build-wheel-from-source" +=== "AMD ROCm" + + --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:build-wheel-from-source" + ## Set up using Docker +### Build wheel from source + +=== "NVIDIA CUDA" + + --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:build-wheel-from-source-in-docker" + +=== "AMD ROCm" + + --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:build-wheel-from-source-in-docker" + ### Pre-built images === "NVIDIA CUDA" diff --git a/docs/getting_started/installation/gpu/cuda.inc.md b/docs/getting_started/installation/gpu/cuda.inc.md index ac519d20e..52d365d9c 100644 --- a/docs/getting_started/installation/gpu/cuda.inc.md +++ b/docs/getting_started/installation/gpu/cuda.inc.md @@ -74,6 +74,9 @@ uv pip install --no-build-isolation --editable . # --8<-- [end:build-wheel-from-source] +# --8<-- [start:build-wheel-from-source-in-docker] + +# --8<-- [end:build-wheel-from-source-in-docker] # --8<-- [start:pre-built-images] diff --git a/docs/getting_started/installation/gpu/rocm.inc.md b/docs/getting_started/installation/gpu/rocm.inc.md new file mode 100644 index 000000000..3b970267a --- /dev/null +++ b/docs/getting_started/installation/gpu/rocm.inc.md @@ -0,0 +1,115 @@ +# --8<-- [start:requirements] + +- GPU: Validated on gfx942 (It should be supported on the AMD GPUs that are supported by vLLM.) + +# --8<-- [end:requirements] +# --8<-- [start:set-up-using-python] + +vLLM-Omni current recommends the steps in under setup through Docker Images. + +# --8<-- [start:pre-built-wheels] + +# --8<-- [end:pre-built-wheels] + +# --8<-- [start:build-wheel-from-source] + +# --8<-- [end:build-wheel-from-source] + +# --8<-- [start:build-wheel-from-source-in-docker] + +#### Get the correct docker environment + +The v0.11.0 releases is on Oct 3, 2025, so we will pull a AMD nightly docker image close to the release period. `rocm/vllm-dev:nightly_main_20251005` is picked as it has a correct `transformers` version that supports qwen3_omni. + +`docker pull rocm/vllm-dev:nightly_main_20251005` + +#### Launch the docker environment + +In this docker environment, we have all the dependencies for vLLM installed, like `flash-attn` and `aiter` + +```bash +#!/bin/bash +docker run -it \ + --privileged \ + --network=host \ + --group-add=video \ + --ipc=host \ + --cap-add=SYS_PTRACE \ + --security-opt seccomp=unconfined \ + --device /dev/kfd \ + --device /dev/dri \ + --name vllmomni \ + rocm/vllm-dev:nightly_main_20251005 \ + bash +``` + +#### Update docker dependencies + +```bash +sudo apt update +sudo apt install ffmpeg -y +``` + +#### Installation of vLLM +Install the v0.11.0 release of vLLM as vLLM-Omni is currently depend on v0.11.0. + +```bash +# uninstall existing vllm which is based on the main branch, not on the releases branch +python3 -m pip uninstall -y vllm +git clone https://github.com/vllm-project/vllm.git vllm0110 +cd vllm0110/ + +# checkout the version tag v0.11.0 +git checkout -b v0.11.0 v0.11.0 + +# you should see 0.11.0 +python -c "import setuptools_scm; print(setuptools_scm.get_version())" + +# PYTORCH_ROCM_ARCH= python3 setup.py develop +PYTORCH_ROCM_ARCH=gfx942 python3 setup.py develop +``` + +!!! note + vLLM release wheels based on the branch with prefix `releases/`, not from the tag as vLLM may cherry pick bugfixes after cutting a branch. + + +#### Installation of vLLM-Omni + +There are two approaches to installing `vllm-omni` + + +1. Install from PyPI + +```bash +pip install vllm-omni + +# you might want to also download the source code of the same version to run the examples +``` + +2. Install from source + +```bash +# clone and checkout to the release version. +git clone https://github.com/vllm-project/vllm-omni.git +cd vllm-omni +git checkout v0.11.0rc1 +pip install -e . +``` + +#### Setup Environment Variables + +After installing `vllm` and `vllm-omni` please setup the environment variable as well before running any of the examples and command. + +```bash +export MIOPEN_FIND_MODE=FAST +export VLLM_ROCM_USE_AITER=1 +export VLLM_ROCM_USE_AITER_MHA=1 +export VLLM_ROCM_USE_AITER_LINEAR=0 +export VLLM_ROCM_USE_AITER_RMSNORM=0 +``` + +# --8<-- [end:build-wheel-from-source-in-docker] + +# --8<-- [start:pre-built-images] + +# --8<-- [end:pre-built-images] diff --git a/docs/models/supported_models.md b/docs/models/supported_models.md index c8a0e622c..391cab0b1 100644 --- a/docs/models/supported_models.md +++ b/docs/models/supported_models.md @@ -6,7 +6,7 @@ vLLM-Omni supports unified multimodal comprehension and generation models across If vLLM-Omni natively supports a model, its implementation can be found in and . -## List of Supported Models for Nvidia GPU +## List of Supported Models for Nvidia GPU / AMD GPU