From c0b88a87ba6037895747c87c9367ccb21f79fe9b Mon Sep 17 00:00:00 2001 From: Alicia <115451386+congw729@users.noreply.github.com> Date: Thu, 4 Dec 2025 16:04:18 +0800 Subject: [PATCH] [Doc] Add installation part for pre built docker. (#141) Signed-off-by: WANG Cong <115451386+congw729@users.noreply.github.com> Signed-off-by: Alicia Wang <115451386+congw729@users.noreply.github.com> Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com> Signed-off-by: Roger Wang Co-authored-by: Roger Wang --- docs/getting_started/installation/gpu.md | 5 ++++- .../installation/gpu/cuda.inc.md | 21 +++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/docs/getting_started/installation/gpu.md b/docs/getting_started/installation/gpu.md index 4926f93f7..42f716f30 100644 --- a/docs/getting_started/installation/gpu.md +++ b/docs/getting_started/installation/gpu.md @@ -37,4 +37,7 @@ vLLM-Omni is a Python library that supports the following GPU variants. The libr ## Set up using Docker ### Pre-built images -To be released... Stay tuned! + +=== "NVIDIA CUDA" + + --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:pre-built-images" diff --git a/docs/getting_started/installation/gpu/cuda.inc.md b/docs/getting_started/installation/gpu/cuda.inc.md index 6eb5293aa..ac519d20e 100644 --- a/docs/getting_started/installation/gpu/cuda.inc.md +++ b/docs/getting_started/installation/gpu/cuda.inc.md @@ -73,3 +73,24 @@ uv pip install --no-build-isolation --editable . # --8<-- [end:build-wheel-from-source] + + +# --8<-- [start:pre-built-images] + +vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as [vllm/vllm-omni](https://hub.docker.com/r/vllm/vllm-omni/tags). The version of vLLM-Omni indicates which release of vLLM it is based on. + +Here's an example deployment command that has been verified on 2 x H100's: +```bash +docker run --runtime nvidia --gpus 2 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --env "HF_TOKEN=$HF_TOKEN" \ + -p 8091:8091 \ + --ipc=host \ + vllm/vllm-omni:v0.11.0rc1 \ + --model Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8091 +``` + +!!! tip + You can use this docker image to serve models the same way you would with in vLLM! To do so, make sure you overwrite the default entrypoint (`vllm serve --omni`) which works only for models supported in the vLLM-Omni project. + +# --8<-- [end:pre-built-images]