+Easy, fast, and cheap omni-modality model serving for everyone
+
-vLLM-omni is designed to extend vLLM capabilities to support multi-modality model inference and serving, particularly focusing on non-autoregressive structures and non-textual outputs.
+
-## 🎯 Overview
+---
-Traditional vLLM systems are limited to text-based, autoregressive generation. vLLM-omni addresses this limitation by enabling support for:
+*Latest News* 🔥
-- **Multi-modal Models**: Text, image, video, audio, and sensor data processing
-- **Non-autoregressive Architectures**: Diffusion Transformers (DiT) and other parallel generation models
-- **Heterogeneous Outputs**: Beyond traditional text generation to multimodal outputs
+- [2025/11] vLLM community officially released [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) in order to support omni-modality models serving.
-## 🏗️ Architecture
+---
-vLLM-omni is built on a modular architecture that extends vLLM's core functionality:
+## About
+[vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni extends its support for omni-modality model inference and serving:
-## 🚀 Key Features
+- **Omni-modality**: Text, image, video, and audio data processing
+- **Non-autoregressive Architectures**: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
+- **Heterogeneous outputs**: from traditional text generation to multimodal outputs
-### Multi-Engine Support
+
+
+
+
+
-- **Autoregressive Engine**: Traditional text generation with enhanced KV-caching
-- **Diffusion Engine**: Support for DiT models and iterative generation
-- **Hybrid Engine**: Combined AR+DiT processing pipelines
+vLLM-omni is fast with:
-### Modality Processing
+- Seamless AR support for efficient KV Cache management with vLLM
+- Pipelined stage execution overlapping
+- Fully disaggregation based on omniConnector and dynamic resource allocation across stages
-- **Text**: Advanced tokenization and embedding generation
-- **Image**: Vision encoder integration (CLIP, etc.)
-- **Audio**: Speech processing and audio embedding
+vLLM-omni is flexible and easy to use with:
-## 📋 Supported Models
+- Seamless integration with popular Hugging Face models
+- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
+- Tensor, pipeline, data and expert parallelism support for distributed inference
+- Streaming outputs
+- OpenAI-compatible API server
-### AR + Diffusion Transformer (DiT) Models
-- Qwen-omni (Thinker-Talker-Codec structure)
-- HunyunaImage 3.0 (Ongoing)
-- Qwen-Image (Ongoing)
+vLLM seamlessly supports most popular open-source models on HuggingFace, including:
+
+- Multi-modality generation models (e.g. Qwen-image)
+- Omni-modality models (e.g. Qwen-omni)
+
+## Getting Started
+
+Install vLLM with `pip` or [from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source):
+
+```bash
+pip install vllm
+```
## 🛠️ Installation
@@ -84,12 +109,6 @@ uv pip install -r requirements/build.txt
uv pip install --no-build-isolation --editable .
```
-### Verification for successful installation of vLLM
-Just run the command below. If no error, it demonstrates that the installation is successful.
-```bash
-python -c "import vllm._C"
-```
-
### Installation of vLLM-omni
Install additional requirements for vllm-omni
```bash
@@ -98,10 +117,21 @@ cd vllm_omni
uv pip install -e .
```
+
## Run examples (Qwen2.5-omni)
Please check the folder of [examples](examples)
-## Further details
+## Contributing
-For detailed architecture design, see [vllm_omni_design.md](docs/architecture/vllm_omni_design.md) and [high_level_arch_design.md](docs/architecture/high_level_arch_design.md).
+We welcome and value any contributions and collaborations.
+Please check out [Contributing to vLLM-omni](https://docs.vllm.ai/en/latest/contributing/index.html) for how to get involved.
+
+## Weekly Meeting
+
+- vLLM Omni Weekly Meeting: https://tinyurl.com/vllm-omni-meeting
+- Wednesday, 11:30 - 12:30 (UTC+8, [Convert to your timezone](https://dateful.com/convert/gmt8?t=15))
+
+## License
+
+Apache License 2.0, as found in the [LICENSE](./LICENSE) file.
diff --git a/docs/source/architecture/omni-modality model architecture.png b/docs/source/architecture/omni-modality model architecture.png
new file mode 100644
index 000000000..eb281184b
Binary files /dev/null and b/docs/source/architecture/omni-modality model architecture.png differ
diff --git a/docs/source/logos/vllm-omni-logo-text-dark.png b/docs/source/logos/vllm-omni-logo-text-dark.png
new file mode 100644
index 000000000..45611c319
Binary files /dev/null and b/docs/source/logos/vllm-omni-logo-text-dark.png differ
diff --git a/docs/source/logos/vllm-omni-logo-text-light.png b/docs/source/logos/vllm-omni-logo-text-light.png
new file mode 100644
index 000000000..0db3a87e6
Binary files /dev/null and b/docs/source/logos/vllm-omni-logo-text-light.png differ
diff --git a/scripts/README.md b/scripts/README.md
deleted file mode 100644
index ec097a9ed..000000000
--- a/scripts/README.md
+++ /dev/null
@@ -1,108 +0,0 @@
-# vLLM-omni Testing Scripts
-
-This directory contains automated testing scripts for vLLM-omni serving functionality.
-
-## Scripts Overview
-
-### 1. `test_serving.sh` - Comprehensive Test Suite
-Full-featured testing script that performs complete validation of the vLLM-omni serving functionality.
-
-**Features:**
-- Model existence validation
-- Environment setup verification
-- Import functionality testing
-- Server startup and health checks
-- Text generation testing
-- Performance benchmarking
-- API client integration testing
-- Comprehensive logging and error handling
-
-**Usage:**
-```bash
-# Use default model and port
-./scripts/test_serving.sh
-
-# Use specific model and port
-./scripts/test_serving.sh ./models/Qwen3-0.6B 8000
-
-# Use HuggingFace model
-./scripts/test_serving.sh Qwen/Qwen3-0.6B 8001
-
-# Show help
-./scripts/test_serving.sh --help
-```
-
-**Test Coverage:**
-- ✅ Model loading and server startup
-- ✅ Health and info endpoints
-- ✅ Text generation functionality
-- ✅ Performance metrics
-- ✅ API client integration
-- ✅ All imports working correctly
-
-### 2. `quick_test.sh` - Fast Validation
-Lightweight script for quick validation after making changes.
-
-**Features:**
-- Fast import testing
-- Basic server startup
-- Health endpoint validation
-- Simple text generation test
-- Retry mechanism for reliability
-
-**Usage:**
-```bash
-# Use default port (8000)
-./scripts/quick_test.sh
-
-# Use specific port
-./scripts/quick_test.sh 8001
-```
-
-**Test Coverage:**
-- ✅ Import functionality
-- ✅ Server startup
-- ✅ Health endpoint
-- ✅ Basic text generation
-
-## Prerequisites
-
-1. **Conda Environment**: Ensure `vllm_omni` environment is activated
-2. **Model Available**: Qwen3-0.6B model should be available in `./models/Qwen3-0.6B/`
-3. **Dependencies**: All vLLM-omni dependencies should be installed
-
-## Environment Setup
-
-```bash
-# Activate conda environment
-conda activate vllm_omni
-
-# Verify installation
-python -c "import vllm_omni; print('vLLM-omni ready')"
-```
-
-## Model Setup
-
-The scripts expect the Qwen3-0.6B model to be available. You can:
-
-1. **Use local model**: Place model in `./models/Qwen3-0.6B/`
-2. **Use HuggingFace model**: Pass `Qwen/Qwen3-0.6B` as model path
-3. **Download model**: Use the `download_models.py` script
-
-```bash
-# Download model using the provided script
-python scripts/download_models.py
-```
-
-## Output and Logging
-
-### Test Results
-- **Success**: Green `[SUCCESS]` messages
-- **Info**: Blue `[INFO]` messages
-- **Warnings**: Yellow `[WARNING]` messages
-- **Errors**: Red `[ERROR]` messages
-
-### Log Files
-- `server.log`: Server startup and runtime logs
-- `api_client_test.log`: API client test results
-- `simple_usage_test.log`: Simple usage test results
diff --git a/scripts/test_serving.sh b/scripts/test_serving.sh
deleted file mode 100755
index a9f1803cc..000000000
--- a/scripts/test_serving.sh
+++ /dev/null
@@ -1,363 +0,0 @@
-#!/bin/bash
-
-# vLLM-omni Serving Functionality Test Script
-# This script tests the complete serving functionality of vLLM-omni
-# Usage: ./scripts/test_serving.sh [model_path] [port]
-
-set -e # Exit on any error
-
-# Configuration
-MODEL_PATH=${1:-"./models/Qwen3-0.6B"}
-PORT=${2:-8000}
-HOST="localhost"
-TIMEOUT=30
-SERVER_STARTUP_WAIT=15
-
-# Colors for output
-RED='\033[0;31m'
-GREEN='\033[0;32m'
-YELLOW='\033[1;33m'
-BLUE='\033[0;34m'
-NC='\033[0m' # No Color
-
-# Logging functions
-log_info() {
- echo -e "${BLUE}[INFO]${NC} $1"
-}
-
-log_success() {
- echo -e "${GREEN}[SUCCESS]${NC} $1"
-}
-
-log_warning() {
- echo -e "${YELLOW}[WARNING]${NC} $1"
-}
-
-log_error() {
- echo -e "${RED}[ERROR]${NC} $1"
-}
-
-# Cleanup function
-cleanup() {
- log_info "Cleaning up..."
- if [ ! -z "$SERVER_PID" ]; then
- log_info "Stopping server (PID: $SERVER_PID)..."
- kill $SERVER_PID 2>/dev/null || true
- wait $SERVER_PID 2>/dev/null || true
- fi
- # Kill any remaining vllm processes
- pkill -f "vllm serve" 2>/dev/null || true
- log_info "Cleanup completed"
-}
-
-# Set up trap for cleanup on exit
-trap cleanup EXIT
-
-# Check if model exists
-check_model() {
- log_info "Checking if model exists at: $MODEL_PATH"
- if [ ! -d "$MODEL_PATH" ]; then
- log_error "Model directory not found: $MODEL_PATH"
- log_info "Available models:"
- ls -la models/ 2>/dev/null || log_warning "No models directory found"
- exit 1
- fi
- log_success "Model found: $MODEL_PATH"
-}
-
-# Check if conda environment is activated
-check_environment() {
- log_info "Checking conda environment..."
- if [[ "$CONDA_DEFAULT_ENV" != "vllm_omni" ]]; then
- log_warning "vllm_omni conda environment not activated"
- log_info "Attempting to activate vllm_omni environment..."
- source $(conda info --base)/etc/profile.d/conda.sh
- conda activate vllm_omni
- if [[ "$CONDA_DEFAULT_ENV" != "vllm_omni" ]]; then
- log_error "Failed to activate vllm_omni environment"
- exit 1
- fi
- fi
- log_success "Environment check passed: $CONDA_DEFAULT_ENV"
-}
-
-# Test import functionality
-test_imports() {
- log_info "Testing imports..."
- python -c "
-import vllm_omni
-from vllm_omni.entrypoints.omni_llm import OmniLLM, AsyncOmniLLM
-print('✅ All imports successful')
-" || {
- log_error "Import test failed"
- exit 1
- }
- log_success "Import test passed"
-}
-
-# Start the server
-start_server() {
- log_info "Starting vLLM-omni server on port $PORT..."
- log_info "Command: vllm serve $MODEL_PATH --omni --port $PORT"
-
- # Start server in background
- vllm serve "$MODEL_PATH" --omni --port "$PORT" > server.log 2>&1 &
- SERVER_PID=$!
-
- log_info "Server started with PID: $SERVER_PID"
- log_info "Waiting $SERVER_STARTUP_WAIT seconds for server to initialize..."
- sleep $SERVER_STARTUP_WAIT
-
- # Check if server is still running
- if ! kill -0 $SERVER_PID 2>/dev/null; then
- log_error "Server failed to start"
- log_info "Server logs:"
- cat server.log
- exit 1
- fi
-
- log_success "Server appears to be running"
-}
-
-# Test health endpoint
-test_health() {
- log_info "Testing health endpoint..."
- for attempt in {1..10}; do
- local response=$(curl -s -w "%{http_code}" http://$HOST:$PORT/health 2>/dev/null || echo "000")
- local http_code="${response: -3}"
- local body="${response%???}"
-
- if [ "$http_code" = "200" ]; then
- log_success "Health check passed: $body"
- return 0
- fi
-
- log_warning "Health check attempt $attempt failed (HTTP $http_code). Retrying in 3s..."
- sleep 3
- done
-
- log_error "Health check failed after multiple attempts"
- return 1
-}
-
-# Test info endpoint
-test_info() {
- log_info "Testing info endpoint..."
- local response=$(curl -s -w "%{http_code}" http://$HOST:$PORT/info 2>/dev/null || echo "000")
- local http_code="${response: -3}"
- local body="${response%???}"
-
- if [ "$http_code" = "200" ]; then
- log_success "Info endpoint working"
- echo "$body" | python -m json.tool 2>/dev/null || log_warning "Info response not valid JSON"
- else
- log_error "Info endpoint failed (HTTP $http_code): $body"
- return 1
- fi
-}
-
-# Test text generation
-test_generation() {
- log_info "Testing text generation..."
-
- # Test 1: Simple generation
- local response=$(curl -s -X POST http://$HOST:$PORT/generate \
- -H "Content-Type: application/json" \
- -d '{"prompts": ["Test the server functionality"], "max_tokens": 20, "temperature": 0.7}' \
- 2>/dev/null || echo "{}")
-
- if echo "$response" | python -c "import sys, json; data=json.load(sys.stdin); exit(0 if 'outputs' in data and len(data['outputs']) > 0 else 1)" 2>/dev/null; then
- log_success "Text generation test passed"
- echo "$response" | python -c "import sys, json; data=json.load(sys.stdin); print('Generated text:', data['outputs'][0]['text'][:100] + '...' if len(data['outputs'][0]['text']) > 100 else data['outputs'][0]['text'])" 2>/dev/null
- else
- log_error "Text generation test failed"
- echo "Response: $response"
- return 1
- fi
-}
-
-# Test API client example
-test_api_client() {
- log_info "Testing API client example..."
- if [ -f "examples/basic/api_client.py" ]; then
- python examples/basic/api_client.py > api_client_test.log 2>&1 || {
- log_error "API client test failed"
- log_info "API client logs:"
- cat api_client_test.log
- return 1
- }
- log_success "API client test passed"
- else
- log_warning "API client example not found, skipping"
- fi
-}
-
-# Test AR → DiT pipeline example
-test_ar_dit_pipeline() {
- log_info "Testing AR → DiT diffusers pipeline example..."
-
- if ! python - 2>/dev/null <<'PY'
-import importlib
-for pkg in ("torch", "diffusers"):
- if importlib.util.find_spec(pkg) is None:
- raise ImportError(pkg)
-PY
- then
- log_warning "Required packages for diffusers pipeline not found, skipping AR → DiT test"
- return 0
- fi
-
- if [ ! -f "examples/omni/ar_dit_diffusers.py" ]; then
- log_warning "AR → DiT example not found, skipping"
- return 0
- fi
-
- local output_path="logs/ar_dit_pipeline.png"
- local log_path="logs/ar_dit_pipeline.log"
- rm -f "$output_path"
-
- local cmd=(python examples/omni/ar_dit_diffusers.py
- --prompt "Smoke test prompt of a lighthouse"
- --seed 0
- --output "$output_path")
-
- local env_prefix=()
- if [ -n "$AR_MODEL" ]; then
- env_prefix+=("AR_MODEL=$AR_MODEL")
- fi
- if [ -n "$DIT_MODEL" ]; then
- env_prefix+=("DIT_MODEL=$DIT_MODEL")
- fi
-
- if ! env "${env_prefix[@]}" "${cmd[@]}" >"$log_path" 2>&1; then
- log_error "AR → DiT pipeline test failed"
- log_info "AR → DiT logs:"
- cat "$log_path"
- return 1
- fi
-
- if [ ! -f "$output_path" ]; then
- log_error "AR → DiT pipeline did not produce an image"
- log_info "AR → DiT logs:"
- cat "$log_path"
- return 1
- fi
-
- log_success "AR → DiT pipeline test passed (output: $output_path)"
-}
-
-# Test simple usage example
-test_simple_usage() {
- log_info "Testing simple usage example..."
- if [ -f "examples/basic/simple_usage.py" ]; then
- # Run with timeout to avoid hanging
- timeout 60 python examples/basic/simple_usage.py > simple_usage_test.log 2>&1 || {
- local exit_code=$?
- if [ $exit_code -eq 124 ]; then
- log_warning "Simple usage test timed out (60s), but this might be normal for model loading"
- else
- log_error "Simple usage test failed (exit code: $exit_code)"
- log_info "Simple usage logs:"
- cat simple_usage_test.log
- return 1
- fi
- }
- log_success "Simple usage test passed"
- else
- log_warning "Simple usage example not found, skipping"
- fi
-}
-
-# Performance test
-test_performance() {
- log_info "Running performance test..."
-
- local start_time=$(date +%s)
- local response=$(curl -s -X POST http://$HOST:$PORT/generate \
- -H "Content-Type: application/json" \
- -d '{"prompts": ["Performance test prompt"], "max_tokens": 50, "temperature": 0.7}' \
- 2>/dev/null)
- local end_time=$(date +%s)
- local duration=$((end_time - start_time))
-
- if echo "$response" | python -c "import sys, json; data=json.load(sys.stdin); exit(0 if 'outputs' in data else 1)" 2>/dev/null; then
- log_success "Performance test passed (${duration}s response time)"
- else
- log_error "Performance test failed"
- return 1
- fi
-}
-
-# Main test function
-run_tests() {
- log_info "Starting vLLM-omni serving functionality tests..."
- log_info "Model: $MODEL_PATH"
- log_info "Port: $PORT"
- log_info "Host: $HOST"
- echo "=========================================="
-
- # Run all tests
- check_model
- check_environment
- test_imports
- start_server
-
- # Wait a bit more for server to be fully ready
- sleep 5
-
- test_health
- test_info
- test_generation
- test_performance
- test_api_client
- test_ar_dit_pipeline
-
- # Note: Simple usage test is commented out as it starts its own server
- # test_simple_usage
-
- log_success "All tests completed successfully!"
- echo "=========================================="
- log_info "Test Summary:"
- log_info "✅ Model loading and server startup"
- log_info "✅ Health and info endpoints"
- log_info "✅ Text generation functionality"
- log_info "✅ Performance metrics"
- log_info "✅ API client integration"
- log_info "✅ AR → DiT diffusers pipeline"
- log_info "✅ All imports working correctly"
-}
-
-# Show usage
-show_usage() {
- echo "Usage: $0 [model_path] [port]"
- echo ""
- echo "Arguments:"
- echo " model_path Path to the model directory (default: ./models/Qwen3-0.6B)"
- echo " port Port to run the server on (default: 8000)"
- echo ""
- echo "Examples:"
- echo " $0 # Use default model and port"
- echo " $0 ./models/Qwen3-0.6B 8001 # Use specific model and port"
- echo " $0 Qwen/Qwen3-0.6B 8000 # Use HuggingFace model"
-}
-
-# Main execution
-main() {
- # Check for help flag
- if [[ "$1" == "-h" || "$1" == "--help" ]]; then
- show_usage
- exit 0
- fi
-
- # Change to script directory
- cd "$(dirname "$0")/.."
-
- # Create logs directory if it doesn't exist
- mkdir -p logs
-
- # Run tests
- run_tests
-}
-
-# Run main function with all arguments
-main "$@"
+ 
+