[Doc] Update README and docs for v0.30.0 (#8175)
Signed-off-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com> Co-authored-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com>
This commit is contained in:
1 parent
6715c57b2d
commit
7ab582ae25
9 files changed
+25
-25
No files matched your search
@@ -5,7 +5,7 @@
|
||||
# --8<-- [end:requirements]
|
||||
# --8<-- [start:set-up-using-python]
|
||||
|
||||
vLLM-Omni depends on the matching major/minor release of vLLM. The 0.30 development line uses vLLM 0.30.x. Published 0.28.0 wheels and images use vLLM 0.28.x.
|
||||
vLLM-Omni depends on the matching major/minor release of vLLM. The vLLM-Omni 0.30.x release line uses vLLM 0.30.x.
|
||||
|
||||
!!! note
|
||||
PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
|
||||
@@ -18,22 +18,22 @@ Therefore, it is recommended to install vLLM and vLLM-Omni with a **fresh new**
|
||||
|
||||
#### Installation of vLLM
|
||||
|
||||
These pre-built wheel instructions install the published vLLM-Omni 0.28.0 release. For the 0.30 development line, use the source-install instructions below.
|
||||
These pre-built wheel instructions install the published vLLM-Omni 0.30.0 release.
|
||||
|
||||
vLLM-Omni is built based on vLLM. Please install it with command below.
|
||||
```bash
|
||||
uv pip install vllm==0.28.0 --torch-backend=auto
|
||||
uv pip install vllm==0.30.0 --torch-backend=auto
|
||||
```
|
||||
|
||||
#### Installation of vLLM-Omni
|
||||
|
||||
```bash
|
||||
uv pip install vllm-omni==0.28.0
|
||||
uv pip install vllm-omni==0.30.0
|
||||
```
|
||||
|
||||
To run Gradio demos, also install the optional extras:
|
||||
```bash
|
||||
uv pip install 'vllm-omni[demo]==0.28.0'
|
||||
uv pip install 'vllm-omni[demo]==0.30.0'
|
||||
```
|
||||
|
||||
# --8<-- [end:pre-built-wheels]
|
||||
@@ -111,7 +111,7 @@ docker run --runtime nvidia --gpus 2 \
|
||||
--env "HF_TOKEN=$HF_TOKEN" \
|
||||
-p 8091:8091 \
|
||||
--ipc=host \
|
||||
vllm/vllm-omni:v0.28.0 \
|
||||
vllm/vllm-omni:v0.30.0 \
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
|
||||
|
||||
@@ -10,9 +10,8 @@
|
||||
|
||||
For ROCm, vLLM-Omni currently recommends the setup steps through Docker Images.
|
||||
|
||||
vLLM-Omni depends on the matching major/minor release of vLLM. The 0.30
|
||||
development line uses vLLM 0.30.x. Published 0.28.0 wheels and images use vLLM
|
||||
0.28.x.
|
||||
vLLM-Omni depends on the matching major/minor release of vLLM. The vLLM-Omni
|
||||
0.30.x release line uses vLLM 0.30.x.
|
||||
|
||||
The Dockerfile's `BASE_IMAGE` pin applies only to Docker builds. The
|
||||
`vllm-omni` package does not install vLLM as a dependency, so non-Docker source
|
||||
@@ -23,12 +22,12 @@ installing vLLM-Omni, as shown below.
|
||||
|
||||
#### Installation of vLLM
|
||||
|
||||
These pre-built wheel instructions install the published vLLM-Omni 0.28.0 release. For the 0.30 development line, use the source-install instructions below.
|
||||
These pre-built wheel instructions install the published vLLM-Omni 0.30.0 release.
|
||||
|
||||
vLLM-Omni is built based on vLLM. Please install it with command below.
|
||||
|
||||
```bash
|
||||
uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
|
||||
uv pip install vllm==0.30.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723
|
||||
```
|
||||
|
||||
#### Installation of vLLM-Omni
|
||||
@@ -37,7 +36,7 @@ uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/roc
|
||||
# we need to add --no-build-isolation as the torch
|
||||
# is not obtained from pypi, we have to install using the
|
||||
# torch installed in our environment
|
||||
uv pip install vllm-omni==0.28.0
|
||||
uv pip install vllm-omni==0.30.0
|
||||
|
||||
# Optional if want to run Qwen3 TTS
|
||||
uv pip uninstall onnxruntime # should be removed before we can install onnxruntime-rocm
|
||||
@@ -150,6 +149,8 @@ vllm-omni-rocm
|
||||
|
||||
vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as [vllm/vllm-omni-rocm](https://hub.docker.com/r/vllm/vllm-omni-rocm/tags). The version of vLLM-Omni indicates which release of vLLM it is based on.
|
||||
|
||||
The prebuilt ROCm image is published separately from the release pipeline: `v0.28.0` is the latest tag currently available on Docker Hub, and newer-tag availability is tracked in [#7405](https://github.com/vllm-project/vllm-omni/issues/7405).
|
||||
|
||||
#### Launch vLLM-Omni Server
|
||||
|
||||
Here's an example deployment command that has been verified on 2 x MI300's:
|
||||
|
||||
Reference in new issue
Block a user