vllm-omni/docs/getting_started/installation/gpu.md

GPU

vLLM-Omni is a Python library that supports the following GPU variants. The library itself mainly contains python implementations for framework and models.

Requirements

• OS: Linux
• Python: 3.12

!!! note vLLM-Omni is currently not natively supported on Windows.

=== "NVIDIA CUDA"

--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:requirements"

Set up using Python

Create a new Python environment

--8<-- "docs/getting_started/installation/python_env_setup.inc.md"

Pre-built wheels

=== "NVIDIA CUDA"

--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:pre-built-wheels"

{ #build-from-source }

Build wheel from source

=== "NVIDIA CUDA"

--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:build-wheel-from-source"

Set up using Docker

Pre-built images

vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as vllm/vllm-omni. The version of vLLM-Omni indicates which release of vLLM it is based on.

!!! tip You can use this docker image to serve models the same way you would do in vLLM! To do so, make sure you overwrite the default entrypoint (vllm serve --omni) which works only for models supported in the vLLM-Omni project.