diff --git a/README.md b/README.md index 6201c4b44..4303aae2d 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Easy, fast, and cheap omni-modality model serving for everyone

-| Documentation | Weekly Meeting | +| Documentation | Weekly Meeting | User Forum | Developer Slack |

--- @@ -30,7 +30,7 @@ Easy, fast, and cheap omni-modality model serving for everyone

- vllm-omni + vllm-omni

@@ -57,8 +57,6 @@ vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, in Visit our [documentation](https://vllm-omni.readthedocs.io/en/latest/) to learn more. -Visit our documentation to learn more. - - [Installation](https://vllm-omni.readthedocs.io/en/latest/getting_started/installation/) - [Quickstart](https://vllm-omni.readthedocs.io/en/latest/getting_started/quickstart/) - [List of Supported Models](https://vllm-omni.readthedocs.io/en/latest/models/supported_models/) diff --git a/docs/design/api_design_doc.md b/docs/design/api_design_doc.md deleted file mode 100644 index b29c44ab5..000000000 --- a/docs/design/api_design_doc.md +++ /dev/null @@ -1,143 +0,0 @@ -# API Design Documentation - -This document contains comprehensive API design documentation for all core modules in vLLM-Omni. These templates provide a standardized structure for designing and implementing the core, engine, executor, and worker modules. - -## 📋 Module API - -### Core Module API -**Core module** provides fundamental scheduling, caching, and resource management functionality. - -**Key Components:** -- Request scheduling and prioritization -- DiT cache management for diffusion models -- Resource allocation and coordination -- Inter-module communication - -### Engine Module API -**Engine module** handles model loading, inference execution, and output processing. - -**Key Components:** -- Model loading and initialization -- Inference execution for AR and diffusion models -- Input preprocessing for various modalities -- Output postprocessing and formatting - -### Executor Module API -**Executor module** coordinates and manages request execution across different engines and workers. - -**Key Components:** -- Request routing and load balancing -- Execution pipeline coordination -- Worker management and task distribution -- Error handling and recovery - -### Worker Module API -**Worker module** provides the actual execution environment for model inference. - -**Key Components:** -- Model execution and inference -- GPU resource management -- Request batching and processing -- Performance optimization - - -## 🔧 Implementation Guidelines - -### Code Standards -- Use type hints for all methods -- Include comprehensive docstrings -- Follow PEP 8 style guidelines -- Use async/await for I/O operations - -### Error Handling -- Define specific exception types -- Provide meaningful error messages -- Include error recovery strategies -- Log errors appropriately - -### Configuration -- Use dataclasses for configuration -- Provide sensible defaults -- Include validation methods -- Support environment variables - -### Testing -- Write unit tests for all public methods -- Include integration tests -- Test error conditions -- Validate configuration options - -## 🚀 Getting Started - -### For Developers - -1. **Choose a Module** - - Pick the module you want to work on - - Read the corresponding API documentation - - Understand the responsibilities and interfaces - -2. **Set Up Development Environment** - ```bash - # Clone the repository - git clone https://github.com/vllm-project/vllm-omni.git - cd vllm-omni - - # Install dependencies - pip install -e ".[dev]" - - # Set up pre-commit hooks - pre-commit install - ``` - -3. **Start Implementation** - - Create the module directory - - Implement base classes first - - Add concrete implementations - - Write tests as you go - -### For Contributors - -1. **Review API Documentation** - - Read through all module APIs - - Understand the overall architecture - - Identify areas for improvement - -2. **Propose Changes** - - Create issues for API changes - - Discuss in pull requests - - Update documentation accordingly - -3. **Submit Contributions** - - Follow the coding standards - - Include tests for new features - - Update documentation - - Submit pull requests - -## 🤝 Contributing - -We welcome contributions to improve the API documentation and implementation. Please: - -1. **Follow the Template Structure** - - Use the [api template](https://github.com/vllm-project/vllm-omni/docs/contributing/design_documents/api_design_template.md) as a guide - - Maintain consistency across modules - - Include all required sections - -2. **Provide Working Examples** - - Test all code examples - - Include both basic and advanced usage - - Show integration patterns - -3. **Keep Documentation Updated** - - Update docs when APIs change - - Version control documentation changes - - Maintain backward compatibility notes - -## 📝 Notes - -- All API documentation should be kept up-to-date with implementation -- Code examples should be tested and working -- Configuration options should be validated -- Error handling should be comprehensive -- Performance considerations should be documented - -For questions or suggestions about the API documentation, please open an issue or start a discussion in the repository. diff --git a/docs/design/api_design_template.md b/docs/design/api_design_template.md deleted file mode 100644 index 2fe2b233f..000000000 --- a/docs/design/api_design_template.md +++ /dev/null @@ -1,336 +0,0 @@ -# API Design Template for vLLM-Omni Modules - -This template provides a standardized structure for designing APIs for core, engine, executor, and worker modules in vLLM-Omni. Use this template to ensure consistency and completeness across all modules. - -## Template Structure - -### 1. Module Overview -- **Purpose**: What this module does -- **Responsibilities**: Key responsibilities of the module -- **Dependencies**: What other modules this depends on -- **Integration Points**: How it integrates with other modules - -### 2. Core Classes/Interfaces -- **Base Classes**: Abstract base classes and interfaces -- **Implementation Classes**: Concrete implementations -- **Data Structures**: Key data structures and models - -### 3. Public API Methods -- **Initialization**: Constructor and setup methods -- **Core Operations**: Main functionality methods -- **Configuration**: Configuration and parameter methods -- **Lifecycle Management**: Start, stop, cleanup methods -- **Monitoring**: Status, metrics, and debugging methods - -### 4. Configuration -- **Configuration Classes**: Dataclasses or config objects -- **Required Parameters**: Must-have configuration -- **Optional Parameters**: Optional configuration with defaults -- **Validation**: Parameter validation rules - -### 5. Error Handling -- **Custom Exceptions**: Module-specific exceptions -- **Error Codes**: Standardized error codes -- **Recovery Strategies**: How to handle and recover from errors - -### 6. Examples -- **Basic Usage**: Simple usage examples -- **Advanced Usage**: Complex scenarios -- **Integration Examples**: How to use with other modules - ---- - -## Example: Core Module Template - -### 1. Module Overview - -**Purpose**: The core module provides fundamental scheduling and caching functionality for vLLM-Omni. - -**Responsibilities**: -- Request scheduling and prioritization -- DiT cache management -- Resource allocation and coordination -- Inter-module communication - -**Dependencies**: -- `vllm_omni.request` - Request handling -- `vllm_omni.config` - Configuration management -- `vllm_omni.utils` - Utility functions - -**Integration Points**: -- Receives requests from entrypoints -- Coordinates with engine modules -- Manages worker allocation -- Provides status to monitoring systems - -### 2. Core Classes/Interfaces - -```python -from abc import ABC, abstractmethod -from typing import List, Optional, Dict, Any -from dataclasses import dataclass -from enum import Enum - -class DiTCacheManager: - """Manages DiT cache for diffusion models.""" - - def __init__(self, config: DiTCacheConfig): - self.config = config - self.cache = {} - self.cache_stats = {} - - async def get_cache(self, cache_key: str) -> Optional[Any]: - """Retrieve cached data.""" - pass - - async def set_cache(self, cache_key: str, data: Any) -> None: - """Store data in cache.""" - pass - - async def invalidate_cache(self, cache_key: str) -> None: - """Invalidate cached data.""" - pass -``` - -### 3. Public API Methods - -#### Initialization -```python -class OmniScheduler(): - def __init__(self, config: SchedulerConfig): - """Initialize the core scheduler.""" - self.config = config - self.scheduler = self._create_scheduler() - self.cache_manager = DiTCacheManager(config.dit_cache_config) - self._running = False - - def _create_scheduler(self) -> BaseScheduler: - """Factory method to create appropriate scheduler.""" - if self.config.scheduler_type == SchedulerType.FIFO: - return FIFOScheduler(self.config) - elif self.config.scheduler_type == SchedulerType.PRIORITY: - return PriorityScheduler(self.config) - # ... other scheduler types -``` - -#### Core Operations -```python - async def schedule(self, request: OmniRequest) -> bool: - """ - Schedule a request for processing. - - Args: - request: The request to schedule - - Returns: - bool: True if successfully scheduled, False otherwise - - Raises: - SchedulerError: If scheduling fails - QueueFullError: If queue is at capacity - """ - pass -``` - -#### Configuration -```python - def update_config(self, new_config: SchedulerConfig) -> None: - """Update scheduler configuration.""" - pass - - def get_config(self) -> SchedulerConfig: - """Get current configuration.""" - pass -``` - -#### Lifecycle Management -```python - async def start(self) -> None: - """Start the scheduler.""" - self._running = True - # Start background tasks - - async def stop(self) -> None: - """Stop the scheduler gracefully.""" - self._running = False - # Cleanup resources - - async def shutdown(self) -> None: - """Force shutdown the scheduler.""" - # Immediate cleanup -``` - -#### Monitoring -```python - def get_status(self) -> Dict[str, Any]: - """Get current scheduler status.""" - return { - "running": self._running, - "queue_size": self.scheduler.queue_size(), - "processed_requests": self.scheduler.processed_count(), - "cache_hit_rate": self.cache_manager.get_hit_rate() - } - - def get_metrics(self) -> Dict[str, float]: - """Get performance metrics.""" - pass -``` - -### 4. Configuration - -```python -@dataclass -class OmniConfig: - """Configuration for the core module.""" - - # Scheduler configuration - scheduler: SchedulerConfig = field(default_factory=SchedulerConfig) - - # Cache configuration - dit_cache: DiTCacheConfig = field(default_factory=DiTCacheConfig) - - # Resource limits - max_memory_gb: float = 16.0 - max_gpu_utilization: float = 0.8 - - # Timeouts - request_timeout: int = 300 - worker_timeout: int = 60 - - def validate(self) -> None: - """Validate configuration parameters.""" - if self.max_memory_gb <= 0: - raise ValueError("max_memory_gb must be positive") - if not 0 < self.max_gpu_utilization <= 1: - raise ValueError("max_gpu_utilization must be between 0 and 1") -``` - -### 5. Error Handling - -```python -class CoreModuleError(Exception): - """Base exception for core module errors.""" - pass - -class SchedulerError(CoreModuleError): - """Scheduler-related errors.""" - pass - -class QueueFullError(SchedulerError): - """Queue is at capacity.""" - pass - -class CacheError(CoreModuleError): - """Cache-related errors.""" - pass - -class ResourceError(CoreModuleError): - """Resource allocation errors.""" - pass -``` - -### 6. Examples - -#### Basic Usage -```python -from vllm_omni.core import CoreScheduler, CoreModuleConfig -from vllm_omni.request import create_text_request - -# Create configuration -config = CoreModuleConfig( - scheduler=SchedulerConfig(scheduler_type=SchedulerType.FIFO), - max_memory_gb=8.0 -) - -# Initialize scheduler -scheduler = CoreScheduler(config) -await scheduler.start() - -# Create and schedule a request -request = create_text_request( - request_id="req_001", - prompt="Hello, world!", - sampling_params=sampling_params -) - -success = await scheduler.schedule_request(request) -if success: - result = await scheduler.process_request(request) - print(f"Result: {result}") -``` - -#### Advanced Usage -```python -# Custom scheduler with priority handling -config = CoreModuleConfig( - scheduler=SchedulerConfig( - scheduler_type=SchedulerType.PRIORITY, - priority_weights={"high": 2.0, "normal": 1.0, "low": 0.5} - ) -) - -scheduler = CoreScheduler(config) -await scheduler.start() - -# Monitor scheduler status -status = scheduler.get_status() -print(f"Queue size: {status['queue_size']}") -print(f"Cache hit rate: {status['cache_hit_rate']:.2%}") -``` - ---- - -## Module-Specific Guidelines - -### Core Module -- Focus on scheduling, caching, and resource management -- Provide clear interfaces for other modules -- Handle concurrency and thread safety -- Implement comprehensive monitoring - -### Engine Module -- Handle model loading and inference -- Support both AR and diffusion models -- Provide unified interface for different model types -- Implement efficient memory management - -### Executor Module -- Coordinate between different engines -- Handle request routing and load balancing -- Manage execution pipelines -- Provide error recovery mechanisms - -### Worker Module -- Handle actual model execution -- Manage GPU resources -- Implement batching strategies -- Provide performance optimization - ---- - -## Checklist for API Design - -- [ ] **Clear Purpose**: Module purpose is well-defined -- [ ] **Complete Interface**: All public methods are documented -- [ ] **Error Handling**: Comprehensive error handling strategy -- [ ] **Configuration**: Flexible configuration system -- [ ] **Examples**: Basic and advanced usage examples -- [ ] **Type Hints**: All methods have proper type hints -- [ ] **Documentation**: Comprehensive docstrings -- [ ] **Testing**: Unit tests for all public methods -- [ ] **Integration**: Clear integration points with other modules -- [ ] **Performance**: Performance considerations documented - ---- - -## Submission Guidelines - -1. Create a new file: `docs/api/[module_name]_api.md` -2. Follow the template structure exactly -3. Include all required sections -4. Provide working code examples -5. Ensure all methods have proper docstrings -6. Include error handling strategies -7. Submit for review before implementation - -This template ensures consistency and completeness across all vLLM-Omni modules. diff --git a/docs/source/architecture/omni-modality model architecture.png b/docs/source/architecture/omni-modality model architecture.png deleted file mode 100644 index eb281184b..000000000 Binary files a/docs/source/architecture/omni-modality model architecture.png and /dev/null differ diff --git a/docs/source/architecture/omni-modality-model-architecture.png b/docs/source/architecture/omni-modality-model-architecture.png new file mode 100644 index 000000000..53978b323 Binary files /dev/null and b/docs/source/architecture/omni-modality-model-architecture.png differ diff --git a/examples/offline_inference/qwen2_5_omni/README.md b/examples/offline_inference/qwen2_5_omni/README.md index 4c5292599..d17cdecd7 100644 --- a/examples/offline_inference/qwen2_5_omni/README.md +++ b/examples/offline_inference/qwen2_5_omni/README.md @@ -1,10 +1,10 @@ -# Offline Example of vLLM-Omni for Qwen2.5-omni +# Offline Example of vLLM-Omni for Qwen2.5-Omni ## 🛠️ Installation Please refer to [README.md](../../../README.md) -## Run examples (Qwen2.5-omni) +## Run examples (Qwen2.5-Omni) ### Multiple Prompts Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). To get the prompt, you can: ```bash diff --git a/examples/offline_inference/qwen3_omni/README.md b/examples/offline_inference/qwen3_omni/README.md index 22c765163..49d3b7049 100644 --- a/examples/offline_inference/qwen3_omni/README.md +++ b/examples/offline_inference/qwen3_omni/README.md @@ -1,12 +1,12 @@ -# Offline Example of vLLM-Omni for Qwen3-omni +# Offline Example of vLLM-Omni for Qwen3-Omni ## 🛠️ Installation Please refer to [README.md](../../../README.md) -## Run examples (Qwen3-omni) +## Run examples (Qwen3-Omni) ### Multiple Prompts -Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). For processing dataset please refer to [Qwen2.5-omni README.md](../qwen2_5_omni/README.md) +Download dataset from [seed_tts](https://drive.google.com/file/d/1GlSjVfSHkW3-leKKBlfrjuuTGqQ_xaLP/edit). For processing dataset please refer to [Qwen2.5-Omni README.md](../qwen2_5_omni/README.md) Get into the example folder ```bash cd examples/offline_inference/qwen3_omni diff --git a/examples/online_serving/qwen2_5_omni/README.md b/examples/online_serving/qwen2_5_omni/README.md index da9570270..a351e3b53 100644 --- a/examples/online_serving/qwen2_5_omni/README.md +++ b/examples/online_serving/qwen2_5_omni/README.md @@ -1,10 +1,10 @@ -# Online serving Example of vLLM-Omni for Qwen2.5-omni +# Online serving Example of vLLM-Omni for Qwen2.5-Omni ## 🛠️ Installation Please refer to [README.md](../../README.md) -## Run examples (Qwen2.5-omni) +## Run examples (Qwen2.5-Omni) Launch the server ```bash diff --git a/examples/online_serving/qwen3_omni/README.md b/examples/online_serving/qwen3_omni/README.md index 9ef40e1fe..ec80cedc0 100644 --- a/examples/online_serving/qwen3_omni/README.md +++ b/examples/online_serving/qwen3_omni/README.md @@ -69,7 +69,7 @@ python gradio_demo.py \ ``` - `--model`: Model name -- `--use-api-server`: If set, connect to an existing vLLM HTTP API server instead of running AsyncOmniLLM locally. +- `--use-api-server`: If set, connect to an existing vLLM HTTP API server instead of running AsyncOmni locally. - `--api-base`: Base URL for vllm serve (only used when `use-api-server` is set, default: http://localhost:8091/v1) - `--ip`: Host/IP for Gradio server (default: 127.0.0.1) - `--port`: Port for Gradio server (default: 7861)