anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Doc] Fix readme typos (#108)

Browse Source 
Signed-off-by: hsliu <liuhongsheng4@huawei.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
This commit is contained in:
Hongsheng LiuandRoger Wang authored and GitHub committed 2025-11-30 21:32:04 +08:00
1 parent bfd3cad5ee
commit 0c245dd1f9
9 files changed
+10 -491
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+2 -4
View File
@@ -9,7 +9,7 @@ Easy, fast, and cheap omni-modality model serving for everyone
</h3>
<p align="center">
| <a href="https://vllm-omni.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href="https://tinyurl.com/vllm-omni-meeting"><b>Weekly Meeting</b></a> |
| <a href="https://vllm-omni.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href="https://tinyurl.com/vllm-omni-meeting"><b>Weekly Meeting</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> |
</p>
---
@@ -30,7 +30,7 @@ Easy, fast, and cheap omni-modality model serving for everyone
<p align="center">
<picture>
<img alt="vllm-omni" src="./docs/source/architecture/omni-modality model architecture.png" width=55%>
<img alt="vllm-omni" src="./docs/source/architecture/omni-modality-model-architecture.png" width=55%>
</picture>
</p>
@@ -57,8 +57,6 @@ vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, in
Visit our [documentation](https://vllm-omni.readthedocs.io/en/latest/) to learn more.
Visit our documentation to learn more.
- [Installation](https://vllm-omni.readthedocs.io/en/latest/getting_started/installation/)
- [Quickstart](https://vllm-omni.readthedocs.io/en/latest/getting_started/quickstart/)
- [List of Supported Models](https://vllm-omni.readthedocs.io/en/latest/models/supported_models/)
docs/design/api_design_doc.md
-143
View File
@@ -1,143 +0,0 @@
# API Design Documentation
This document contains comprehensive API design documentation for all core modules in vLLM-Omni. These templates provide a standardized structure for designing and implementing the core, engine, executor, and worker modules.
## 📋 Module API
### Core Module API
**Core module** provides fundamental scheduling, caching, and resource management functionality.
**Key Components:**
- Request scheduling and prioritization
- DiT cache management for diffusion models
- Resource allocation and coordination
- Inter-module communication
### Engine Module API
**Engine module** handles model loading, inference execution, and output processing.
**Key Components:**
- Model loading and initialization
- Inference execution for AR and diffusion models
- Input preprocessing for various modalities
- Output postprocessing and formatting
### Executor Module API
**Executor module** coordinates and manages request execution across different engines and workers.
**Key Components:**
- Request routing and load balancing
- Execution pipeline coordination
- Worker management and task distribution
- Error handling and recovery
### Worker Module API
**Worker module** provides the actual execution environment for model inference.
**Key Components:**
- Model execution and inference
- GPU resource management
- Request batching and processing
- Performance optimization
## 🔧 Implementation Guidelines
### Code Standards
- Use type hints for all methods
- Include comprehensive docstrings
- Follow PEP 8 style guidelines
- Use async/await for I/O operations
### Error Handling
- Define specific exception types
- Provide meaningful error messages
- Include error recovery strategies
- Log errors appropriately
### Configuration
- Use dataclasses for configuration
- Provide sensible defaults
- Include validation methods
- Support environment variables
### Testing
- Write unit tests for all public methods
- Include integration tests
- Test error conditions
- Validate configuration options
## 🚀 Getting Started
### For Developers
1. **Choose a Module**
- Pick the module you want to work on
- Read the corresponding API documentation
- Understand the responsibilities and interfaces
2. **Set Up Development Environment**
```bash
# Clone the repository
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
# Install dependencies
pip install -e ".[dev]"
# Set up pre-commit hooks
pre-commit install
```
3. **Start Implementation**
- Create the module directory
- Implement base classes first
- Add concrete implementations
- Write tests as you go
### For Contributors
1. **Review API Documentation**
- Read through all module APIs
- Understand the overall architecture
- Identify areas for improvement
2. **Propose Changes**
- Create issues for API changes
- Discuss in pull requests
- Update documentation accordingly
3. **Submit Contributions**
- Follow the coding standards
- Include tests for new features
- Update documentation
- Submit pull requests
## 🤝 Contributing
We welcome contributions to improve the API documentation and implementation. Please:
1. **Follow the Template Structure**
- Use the [api template](https://github.com/vllm-project/vllm-omni/docs/contributing/design_documents/api_design_template.md) as a guide
- Maintain consistency across modules
- Include all required sections
2. **Provide Working Examples**
- Test all code examples
- Include both basic and advanced usage
- Show integration patterns
3. **Keep Documentation Updated**
- Update docs when APIs change
- Version control documentation changes
- Maintain backward compatibility notes
## 📝 Notes
- All API documentation should be kept up-to-date with implementation
- Code examples should be tested and working
- Configuration options should be validated
- Error handling should be comprehensive
- Performance considerations should be documented
For questions or suggestions about the API documentation, please open an issue or start a discussion in the repository.
docs/design/api_design_template.md
-336
View File
@@ -1,336 +0,0 @@
# API Design Template for vLLM-Omni Modules
This template provides a standardized structure for designing APIs for core, engine, executor, and worker modules in vLLM-Omni. Use this template to ensure consistency and completeness across all modules.
## Template Structure
### 1. Module Overview
- **Purpose**: What this module does
- **Responsibilities**: Key responsibilities of the module
- **Dependencies**: What other modules this depends on
- **Integration Points**: How it integrates with other modules
### 2. Core Classes/Interfaces
- **Base Classes**: Abstract base classes and interfaces
- **Implementation Classes**: Concrete implementations
- **Data Structures**: Key data structures and models
### 3. Public API Methods
- **Initialization**: Constructor and setup methods
- **Core Operations**: Main functionality methods
- **Configuration**: Configuration and parameter methods
- **Lifecycle Management**: Start, stop, cleanup methods
- **Monitoring**: Status, metrics, and debugging methods
### 4. Configuration
- **Configuration Classes**: Dataclasses or config objects
- **Required Parameters**: Must-have configuration
- **Optional Parameters**: Optional configuration with defaults
- **Validation**: Parameter validation rules
### 5. Error Handling
- **Custom Exceptions**: Module-specific exceptions
- **Error Codes**: Standardized error codes
- **Recovery Strategies**: How to handle and recover from errors
### 6. Examples
- **Basic Usage**: Simple usage examples
- **Advanced Usage**: Complex scenarios
- **Integration Examples**: How to use with other modules
---
## Example: Core Module Template
### 1. Module Overview
**Purpose**: The core module provides fundamental scheduling and caching functionality for vLLM-Omni.
**Responsibilities**:
- Request scheduling and prioritization
- DiT cache management
- Resource allocation and coordination
- Inter-module communication
**Dependencies**:
- `vllm_omni.request` - Request handling
- `vllm_omni.config` - Configuration management
- `vllm_omni.utils` - Utility functions
**Integration Points**:
- Receives requests from entrypoints
- Coordinates with engine modules
- Manages worker allocation
- Provides status to monitoring systems
### 2. Core Classes/Interfaces
```python
from abc import ABC, abstractmethod
from typing import List, Optional, Dict, Any
from dataclasses import dataclass
from enum import Enum
class DiTCacheManager:
"""Manages DiT cache for diffusion models."""
def __init__(self, config: DiTCacheConfig):
self.config = config
self.cache = {}
self.cache_stats = {}
async def get_cache(self, cache_key: str) -> Optional[Any]:
"""Retrieve cached data."""
pass
async def set_cache(self, cache_key: str, data: Any) -> None:
"""Store data in cache."""
pass
async def invalidate_cache(self, cache_key: str) -> None:
"""Invalidate cached data."""
pass
```
### 3. Public API Methods
#### Initialization
```python
class OmniScheduler():
def __init__(self, config: SchedulerConfig):
"""Initialize the core scheduler."""
self.config = config
self.scheduler = self._create_scheduler()
self.cache_manager = DiTCacheManager(config.dit_cache_config)
self._running = False
def _create_scheduler(self) -> BaseScheduler:
"""Factory method to create appropriate scheduler."""
if self.config.scheduler_type == SchedulerType.FIFO:
return FIFOScheduler(self.config)
elif self.config.scheduler_type == SchedulerType.PRIORITY:
return PriorityScheduler(self.config)
# ... other scheduler types
```
#### Core Operations
```python
async def schedule(self, request: OmniRequest) -> bool:
"""
Schedule a request for processing.
Args:
request: The request to schedule
Returns:
bool: True if successfully scheduled, False otherwise
Raises:
SchedulerError: If scheduling fails
QueueFullError: If queue is at capacity
"""
pass
```
#### Configuration
```python
def update_config(self, new_config: SchedulerConfig) -> None:
"""Update scheduler configuration."""
pass
def get_config(self) -> SchedulerConfig:
"""Get current configuration."""
pass
```
#### Lifecycle Management
```python
async def start(self) -> None:
"""Start the scheduler."""
self._running = True
# Start background tasks
async def stop(self) -> None:
"""Stop the scheduler gracefully."""
self._running = False
# Cleanup resources
async def shutdown(self) -> None:
"""Force shutdown the scheduler."""
# Immediate cleanup
```
#### Monitoring
```python
def get_status(self) -> Dict[str, Any]:
"""Get current scheduler status."""
return {
"running": self._running,
"queue_size": self.scheduler.queue_size(),
"processed_requests": self.scheduler.processed_count(),
"cache_hit_rate": self.cache_manager.get_hit_rate()
}
def get_metrics(self) -> Dict[str, float]:
"""Get performance metrics."""
pass
```
### 4. Configuration
```python
@dataclass
class OmniConfig:
"""Configuration for the core module."""
# Scheduler configuration
scheduler: SchedulerConfig = field(default_factory=SchedulerConfig)
# Cache configuration
dit_cache: DiTCacheConfig = field(default_factory=DiTCacheConfig)
# Resource limits
max_memory_gb: float = 16.0
max_gpu_utilization: float = 0.8
# Timeouts
request_timeout: int = 300
worker_timeout: int = 60
def validate(self) -> None:
"""Validate configuration parameters."""
if self.max_memory_gb <= 0:
raise ValueError("max_memory_gb must be positive")
if not 0 < self.max_gpu_utilization <= 1:
raise ValueError("max_gpu_utilization must be between 0 and 1")
```
### 5. Error Handling
```python
class CoreModuleError(Exception):
"""Base exception for core module errors."""
pass
class SchedulerError(CoreModuleError):
"""Scheduler-related errors."""
pass
class QueueFullError(SchedulerError):
"""Queue is at capacity."""
pass
class CacheError(CoreModuleError):
"""Cache-related errors."""
pass
class ResourceError(CoreModuleError):
"""Resource allocation errors."""
pass
```
### 6. Examples
#### Basic Usage
```python
from vllm_omni.core import CoreScheduler, CoreModuleConfig
from vllm_omni.request import create_text_request
# Create configuration
config = CoreModuleConfig(
scheduler=SchedulerConfig(scheduler_type=SchedulerType.FIFO),
max_memory_gb=8.0
)
# Initialize scheduler
scheduler = CoreScheduler(config)
await scheduler.start()
# Create and schedule a request
request = create_text_request(
request_id="req_001",
prompt="Hello, world!",
sampling_params=sampling_params
)
success = await scheduler.schedule_request(request)
if success:
result = await scheduler.process_request(request)
print(f"Result: {result}")
```
#### Advanced Usage
```python
# Custom scheduler with priority handling
config = CoreModuleConfig(
scheduler=SchedulerConfig(
scheduler_type=SchedulerType.PRIORITY,
priority_weights={"high": 2.0, "normal": 1.0, "low": 0.5}
)
)
scheduler = CoreScheduler(config)
await scheduler.start()
# Monitor scheduler status
status = scheduler.get_status()
print(f"Queue size: {status['queue_size']}")
print(f"Cache hit rate: {status['cache_hit_rate']:.2%}")
```
---
## Module-Specific Guidelines
### Core Module
- Focus on scheduling, caching, and resource management
- Provide clear interfaces for other modules
- Handle concurrency and thread safety
- Implement comprehensive monitoring
### Engine Module
- Handle model loading and inference
- Support both AR and diffusion models
- Provide unified interface for different model types
- Implement efficient memory management
### Executor Module
- Coordinate between different engines
- Handle request routing and load balancing
- Manage execution pipelines
- Provide error recovery mechanisms
### Worker Module
- Handle actual model execution
- Manage GPU resources
- Implement batching strategies
- Provide performance optimization
---
## Checklist for API Design
- [ ] **Clear Purpose**: Module purpose is well-defined
- [ ] **Complete Interface**: All public methods are documented
- [ ] **Error Handling**: Comprehensive error handling strategy
- [ ] **Configuration**: Flexible configuration system
- [ ] **Examples**: Basic and advanced usage examples
- [ ] **Type Hints**: All methods have proper type hints
- [ ] **Documentation**: Comprehensive docstrings
- [ ] **Testing**: Unit tests for all public methods
- [ ] **Integration**: Clear integration points with other modules
- [ ] **Performance**: Performance considerations documented
---
## Submission Guidelines
1. Create a new file: `docs/api/[module_name]_api.md`
2. Follow the template structure exactly
3. Include all required sections
4. Provide working code examples
5. Ensure all methods have proper docstrings
6. Include error handling strategies
7. Submit for review before implementation
This template ensures consistency and completeness across all vLLM-Omni modules.
docs/source/architecture/omni-modality model architecture.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 51 KiB

docs/source/architecture/omni-modality-model-architecture.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

examples/offline_inference/qwen2_5_omni/README.md
+2 -2
View File
@@ -1,10 +1,10 @@
# Offline Example of vLLM-Omni for Qwen2.5-omni
# Offline Example of vLLM-Omni for Qwen2.5-Omni
## 🛠️ Installation
Please refer to [README.md](../../../README.md)
## Run examples (Qwen2.5-omni)
## Run examples (Qwen2.5-Omni)
### Multiple Prompts
Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). To get the prompt, you can:
```bash
examples/offline_inference/qwen3_omni/README.md
+3 -3
View File
@@ -1,12 +1,12 @@
# Offline Example of vLLM-Omni for Qwen3-omni
# Offline Example of vLLM-Omni for Qwen3-Omni
## 🛠️ Installation
Please refer to [README.md](../../../README.md)
## Run examples (Qwen3-omni)
## Run examples (Qwen3-Omni)
### Multiple Prompts
Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). For processing dataset please refer to [Qwen2.5-omni README.md](../qwen2_5_omni/README.md)
Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). For processing dataset please refer to [Qwen2.5-Omni README.md](../qwen2_5_omni/README.md)
Get into the example folder
```bash
cd examples/offline_inference/qwen3_omni
examples/online_serving/qwen2_5_omni/README.md
+2 -2
View File
@@ -1,10 +1,10 @@
# Online serving Example of vLLM-Omni for Qwen2.5-omni
# Online serving Example of vLLM-Omni for Qwen2.5-Omni
## 🛠️ Installation
Please refer to [README.md](../../README.md)
## Run examples (Qwen2.5-omni)
## Run examples (Qwen2.5-Omni)
Launch the server
```bash
examples/online_serving/qwen3_omni/README.md
+1 -1
View File
@@ -69,7 +69,7 @@ python gradio_demo.py \
```
- `--model`: Model name
- `--use-api-server`: If set, connect to an existing vLLM HTTP API server instead of running AsyncOmniLLM locally.
- `--use-api-server`: If set, connect to an existing vLLM HTTP API server instead of running AsyncOmni locally.
- `--api-base`: Base URL for vllm serve (only used when `use-api-server` is set, default: http://localhost:8091/v1)
- `--ip`: Host/IP for Gradio server (default: 127.0.0.1)
- `--port`: Port for Gradio server (default: 7861)