tzhouamandClaude Fable 5
b3f5c97dec
Mirror of the docs update that shipped with the v0.24.0 rebase (#4709): - quickstart + CUDA/ROCm install guides: vllm==0.24.0 -> 0.25.0 pins, git checkout tag, precompiled-wheel URL (asset verified on the v0.25.0 GitHub release), and docker image tags - ROCm wheel index suffix changed upstream: rocm722 -> rocm723 (wheels.vllm.ai/rocm/0.25.0/ serves only rocm723; ROCm 7.2.2 -> 7.2.3) - gpu.md: add 0.25.0 to the pre-built wheels list NPU docs still reference v0.24.0 Ascend images intentionally — those images are published separately by the NPU team. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
hide
| hide | ||
|---|---|---|
|
Welcome to vLLM-Omni
Easy, fast, and cheap omni-modality model serving for everyone
<script async defer src="https://buttons.github.io/buttons.js"></script> Star Watch Fork
About
vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:
- Omni-modality: Text, image, video, and audio data processing
- Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
- Heterogeneous outputs: from traditional text generation to multimodal outputs
vLLM-Omni is fast with:
- State-of-the-art AR support by leveraging efficient KV cache management from vLLM
- Pipelined stage execution overlapping for high throughput performance
- Fully disaggregation based on OmniConnector and dynamic resource allocation across stages
vLLM-Omni is flexible and easy to use with:
- Heterogeneous pipeline abstraction to manage complex model workflows
- Seamless integration with popular Hugging Face models
- Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
- Omni-modality models (e.g. Qwen2.5-Omni, Qwen3-Omni)
- Multi-modality generation models (e.g. Qwen-Image)
For more information, checkout the following: