[CI][ROCm] Align AMD image with vLLM 0.29 (#7395)

Signed-off-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>

docker/Dockerfile.rocm

@@ -1,16 +1,18 @@
-ARG BASE_IMAGE=vllm/vllm-openai-rocm:v0.28.0
+# Keep the ROCm runtime aligned with the vLLM release targeted by
+# docker/Dockerfile.ci. A stale base image can build successfully while failing
+# at runtime when vLLM-Omni imports APIs added by the newer vLLM release.
+ARG BASE_IMAGE=vllm/vllm-openai-rocm:v0.29.0
 FROM ${BASE_IMAGE} AS base
 
-# Declare a variable to know if we want to use the nightly build or the stable build.
 # NOTE: REMINDER to vLLM-Omni rebase maintainer
-# Remember to set `USE_NIGHTLY_BUILD` to 0, when switching back to stable vLLM docker image
+# BASE_IMAGE pins the default prebuilt runtime when USE_NIGHTLY_BUILD=0.
-# 1. If vLLM-omni maintainer is forced to use custom commits
+# VLLM_VERSION_OR_COMMIT_HASH is consumed only by the opt-in source rebuild when
-#    during rebasing, they can change the two variables
+# USE_NIGHTLY_BUILD=1. Keep both aligned with docker/Dockerfile.ci's
-# 2. Whenever vLLM upstream has released a stable version,
+# VLLM_BASE_TAG. This is enforced by tests/buildkite/test_rocm_dockerfile.py. An
-#    we should swap over to use stable release ASAP.
+# intentional custom source pin must update that regression in the same change,
-#    We should avoid relying on custom commits.
+# and should return to the stable release as soon as one is available.
 ARG USE_NIGHTLY_BUILD=0
-ARG VLLM_VERSION_OR_COMMIT_HASH=v0.28.0
+ARG VLLM_VERSION_OR_COMMIT_HASH=v0.29.0
 ARG ARG_PYTORCH_ROCM_ARCH
 ENV PYTORCH_ROCM_ARCH=${ARG_PYTORCH_ROCM_ARCH:-${PYTORCH_ROCM_ARCH}}
 
@@ -38,6 +40,10 @@ RUN if [ "${USE_NIGHTLY_BUILD}" = "1" ]; then \
         rm -rf vllm/.git; \
     fi
 
+# Fail during image construction instead of letting every AMD test fail at
+# runtime when the base image and vLLM-Omni target incompatible vLLM APIs.
+RUN python3 -c "import vllm; from vllm.v1.kv_cache_interface import compute_layout_strides; from vllm.v1.kv_cache_layout import KVCacheLayout; print(f'Validated vLLM {vllm.__version__}')"
+
 # Build TorchCodec after any optional nightly vLLM reinstall so it targets the
 # final torch/ROCm ABI. Keep this layer before COPY . so source changes do not
 # invalidate the expensive native build. FFmpeg development headers are

docs/getting_started/installation/gpu/rocm.inc.md

@@ -1,11 +1,23 @@
+<!-- markdownlint-disable MD001 MD025 -->
+
 # --8<-- [start:requirements]
 
 - GPU: Validated on gfx942 (It should be supported on the AMD GPUs that are supported by vLLM.)
 
 # --8<-- [end:requirements]
+
 # --8<-- [start:set-up-using-python]
 
-vLLM-Omni current recommends the steps in under setup through Docker Images.
+For ROCm, vLLM-Omni currently recommends the setup steps through Docker Images.
+
+vLLM-Omni depends on the matching major/minor release of vLLM. The 0.29
+development line uses vLLM 0.29.x. Published 0.28.0 wheels and images use vLLM
+0.28.x.
+
+The Dockerfile's `BASE_IMAGE` pin applies only to Docker builds. The
+`vllm-omni` package does not install vLLM as a dependency, so non-Docker source
+installs must install the matching ROCm vLLM release explicitly before
+installing vLLM-Omni, as shown below.
 
 # --8<-- [start:pre-built-wheels]
 
@@ -14,6 +26,7 @@ vLLM-Omni current recommends the steps in under setup through Docker Images.
 These pre-built wheel instructions install the published vLLM-Omni 0.28.0 release. For the 0.29 development line, use the source-install instructions below.
 
 vLLM-Omni is built based on vLLM. Please install it with command below.
+
 ```bash
 uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
 ```
@@ -36,6 +49,7 @@ uv pip install onnxruntime-rocm
 # --8<-- [start:build-wheel-from-source]
 
 #### Installation of vLLM
+
 If you do not need to modify source code of vLLM, you can directly install the stable 0.29.0 release version of the library
 
 ```bash
@@ -45,7 +59,9 @@ uv pip install vllm==0.29.0+rocm723 --extra-index-url https://wheels.vllm.ai/roc
 The pre-built 0.29.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
 
 #### Installation of vLLM-Omni
+
 Since vllm-omni is rapidly evolving, it's recommended to install it from source
+
 ```bash
 git clone https://github.com/vllm-project/vllm-omni.git
 cd vllm-omni
@@ -71,15 +87,31 @@ python3 setup.py develop
 
 #### Build docker image
 
+The source build defaults to the published upstream base image
+`vllm/vllm-openai-rocm:v0.29.0`, aligned with the vLLM release used by CI.
+Older or custom bases must provide the vLLM APIs checked by the Dockerfile's
+image-build canary. This upstream base is distinct from the prebuilt
+`vllm/vllm-omni-rocm` images discussed below; their published-tag availability
+is tracked in [#7405](https://github.com/vllm-project/vllm-omni/issues/7405).
+
 ```bash
 DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm-omni-rocm .
 ```
 
+To select the upstream ROCm base explicitly, pass `BASE_IMAGE`:
+
+```bash
+DOCKER_BUILDKIT=1 docker build \
+  -f docker/Dockerfile.rocm \
+  --build-arg BASE_IMAGE=vllm/vllm-openai-rocm:v0.29.0 \
+  -t vllm-omni-rocm .
+```
+
 #### Launch the docker image
 
 ##### Launch with OpenAI API Server
 
-```
+```bash
 docker run --rm \
 --group-add=video \
 --ipc=host \
@@ -97,7 +129,7 @@ vllm-omni-rocm \
 
 ##### Launch with interactive session for development
 
-```
+```bash
 docker run --rm -it \
 --network=host \
 --group-add=video \
@@ -119,7 +151,9 @@ vllm-omni-rocm
 vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as [vllm/vllm-omni-rocm](https://hub.docker.com/r/vllm/vllm-omni-rocm/tags). The version of vLLM-Omni indicates which release of vLLM it is based on.
 
 #### Launch vLLM-Omni Server
+
 Here's an example deployment command that has been verified on 2 x MI300's:
+
 ```bash
 docker run --rm \
   --group-add=video \
@@ -136,8 +170,10 @@ docker run --rm \
   --model Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
 ```
 
-#### Launch an interactive terminal with prebuilt docker image.
+#### Launch an interactive terminal with prebuilt docker image
+
 If you want to run in dev environment you can launch the docker image as follows:
+
 ```bash
 docker run --rm -it \
   --network=host \

tests/buildkite/test_rocm_dockerfile.py

@@ -0,0 +1,77 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+from pathlib import Path
+
+import pytest
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+AMD_TEMPLATE = REPO_ROOT / ".buildkite/amd/test-template-amd-omni.j2"
+CI_DOCKERFILE = REPO_ROOT / "docker/Dockerfile.ci"
+ROCM_DOCKERFILE = REPO_ROOT / "docker/Dockerfile.rocm"
+
+
+def _docker_arg(path: Path, name: str) -> str:
+    prefix = f"ARG {name}="
+    matches = [
+        line.removeprefix(prefix) for line in path.read_text(encoding="utf-8").splitlines() if line.startswith(prefix)
+    ]
+    assert len(matches) == 1, f"expected one {name} declaration in {path}, found {len(matches)}"
+    return matches[0]
+
+
+def _line_index(lines: list[str], prefix: str) -> int:
+    matches = [index for index, line in enumerate(lines) if line.startswith(prefix)]
+    assert len(matches) == 1, f"expected one line starting with {prefix!r}, found {len(matches)}"
+    return matches[0]
+
+
+def test_rocm_base_tracks_ci_vllm_release() -> None:
+    ci_release = _docker_arg(CI_DOCKERFILE, "VLLM_BASE_TAG")
+    rocm_base = _docker_arg(ROCM_DOCKERFILE, "BASE_IMAGE")
+
+    image_ref, separator, image_tag = rocm_base.rpartition(":")
+    assert separator, f"expected a tagged ROCm base image, got {rocm_base}"
+    assert image_ref.rsplit("/", 1)[-1] == "vllm-openai-rocm", image_ref
+    assert image_tag == ci_release
+
+
+def test_rocm_defaults_to_prebuilt_base_image() -> None:
+    assert _docker_arg(ROCM_DOCKERFILE, "USE_NIGHTLY_BUILD") == "0"
+
+
+def test_amd_build_uses_rocm_dockerfile_defaults() -> None:
+    template = AMD_TEMPLATE.read_text(encoding="utf-8")
+    build_commands = [line.strip() for line in template.splitlines() if '"docker build ' in line]
+
+    assert len(build_commands) == 1, f"expected one AMD image build command, found {len(build_commands)}"
+    build_command = build_commands[0]
+    assert "-f docker/Dockerfile.rocm" in build_command
+    for arg_name in ("BASE_IMAGE", "USE_NIGHTLY_BUILD"):
+        assert f"--build-arg {arg_name}" not in build_command
+        assert f"--build-arg={arg_name}" not in build_command
+
+
+def test_rocm_source_ref_tracks_ci_vllm_release() -> None:
+    ci_release = _docker_arg(CI_DOCKERFILE, "VLLM_BASE_TAG")
+    rocm_source_ref = _docker_arg(ROCM_DOCKERFILE, "VLLM_VERSION_OR_COMMIT_HASH")
+
+    assert rocm_source_ref == ci_release
+
+
+def test_rocm_dockerfile_contains_vllm_api_canary() -> None:
+    dockerfile = ROCM_DOCKERFILE.read_text(encoding="utf-8")
+    lines = dockerfile.splitlines()
+    nightly_start = _line_index(lines, 'RUN if [ "${USE_NIGHTLY_BUILD}" = "1" ]; then')
+    canary = (
+        'RUN python3 -c "import vllm; '
+        "from vllm.v1.kv_cache_interface import compute_layout_strides; "
+        "from vllm.v1.kv_cache_layout import KVCacheLayout;"
+    )
+    canary_index = _line_index(lines, canary)
+
+    assert any(line.strip() == "fi" for line in lines[nightly_start + 1 : canary_index]), (
+        "vLLM API canary must follow the optional nightly source reinstall block"
+    )