[Docs] Setup Documentation System and Re-organize Dependencies (#49)

Signed-off-by: SamitHuang <285365963@qq.com>
This commit is contained in:
Samit authored and GitHub committed 2025-11-15 21:45:43 +08:00
1 parent a089dd4c4e
commit 8cafb4294d
37 files changed
+1217 -165

No files matched your search

-336
View File
@@ -1,336 +0,0 @@
# API Design Template for vLLM-omni Modules
This template provides a standardized structure for designing APIs for core, engine, executor, and worker modules in vLLM-omni. Use this template to ensure consistency and completeness across all modules.
## Template Structure
### 1. Module Overview
- **Purpose**: What this module does
- **Responsibilities**: Key responsibilities of the module
- **Dependencies**: What other modules this depends on
- **Integration Points**: How it integrates with other modules
### 2. Core Classes/Interfaces
- **Base Classes**: Abstract base classes and interfaces
- **Implementation Classes**: Concrete implementations
- **Data Structures**: Key data structures and models
### 3. Public API Methods
- **Initialization**: Constructor and setup methods
- **Core Operations**: Main functionality methods
- **Configuration**: Configuration and parameter methods
- **Lifecycle Management**: Start, stop, cleanup methods
- **Monitoring**: Status, metrics, and debugging methods
### 4. Configuration
- **Configuration Classes**: Dataclasses or config objects
- **Required Parameters**: Must-have configuration
- **Optional Parameters**: Optional configuration with defaults
- **Validation**: Parameter validation rules
### 5. Error Handling
- **Custom Exceptions**: Module-specific exceptions
- **Error Codes**: Standardized error codes
- **Recovery Strategies**: How to handle and recover from errors
### 6. Examples
- **Basic Usage**: Simple usage examples
- **Advanced Usage**: Complex scenarios
- **Integration Examples**: How to use with other modules
---
## Example: Core Module Template
### 1. Module Overview
**Purpose**: The core module provides fundamental scheduling and caching functionality for vLLM-omni.
**Responsibilities**:
- Request scheduling and prioritization
- DiT cache management
- Resource allocation and coordination
- Inter-module communication
**Dependencies**:
- `vllm_omni.request` - Request handling
- `vllm_omni.config` - Configuration management
- `vllm_omni.utils` - Utility functions
**Integration Points**:
- Receives requests from entrypoints
- Coordinates with engine modules
- Manages worker allocation
- Provides status to monitoring systems
### 2. Core Classes/Interfaces
```python
from abc import ABC, abstractmethod
from typing import List, Optional, Dict, Any
from dataclasses import dataclass
from enum import Enum
class DiTCacheManager:
"""Manages DiT cache for diffusion models."""
def __init__(self, config: DiTCacheConfig):
self.config = config
self.cache = {}
self.cache_stats = {}
async def get_cache(self, cache_key: str) -> Optional[Any]:
"""Retrieve cached data."""
pass
async def set_cache(self, cache_key: str, data: Any) -> None:
"""Store data in cache."""
pass
async def invalidate_cache(self, cache_key: str) -> None:
"""Invalidate cached data."""
pass
```
### 3. Public API Methods
#### Initialization
```python
class OmniScheduler():
def __init__(self, config: SchedulerConfig):
"""Initialize the core scheduler."""
self.config = config
self.scheduler = self._create_scheduler()
self.cache_manager = DiTCacheManager(config.dit_cache_config)
self._running = False
def _create_scheduler(self) -> BaseScheduler:
"""Factory method to create appropriate scheduler."""
if self.config.scheduler_type == SchedulerType.FIFO:
return FIFOScheduler(self.config)
elif self.config.scheduler_type == SchedulerType.PRIORITY:
return PriorityScheduler(self.config)
# ... other scheduler types
```
#### Core Operations
```python
async def schedule(self, request: OmniRequest) -> bool:
"""
Schedule a request for processing.
Args:
request: The request to schedule
Returns:
bool: True if successfully scheduled, False otherwise
Raises:
SchedulerError: If scheduling fails
QueueFullError: If queue is at capacity
"""
pass
```
#### Configuration
```python
def update_config(self, new_config: SchedulerConfig) -> None:
"""Update scheduler configuration."""
pass
def get_config(self) -> SchedulerConfig:
"""Get current configuration."""
pass
```
#### Lifecycle Management
```python
async def start(self) -> None:
"""Start the scheduler."""
self._running = True
# Start background tasks
async def stop(self) -> None:
"""Stop the scheduler gracefully."""
self._running = False
# Cleanup resources
async def shutdown(self) -> None:
"""Force shutdown the scheduler."""
# Immediate cleanup
```
#### Monitoring
```python
def get_status(self) -> Dict[str, Any]:
"""Get current scheduler status."""
return {
"running": self._running,
"queue_size": self.scheduler.queue_size(),
"processed_requests": self.scheduler.processed_count(),
"cache_hit_rate": self.cache_manager.get_hit_rate()
}
def get_metrics(self) -> Dict[str, float]:
"""Get performance metrics."""
pass
```
### 4. Configuration
```python
@dataclass
class OmniConfig:
"""Configuration for the core module."""
# Scheduler configuration
scheduler: SchedulerConfig = field(default_factory=SchedulerConfig)
# Cache configuration
dit_cache: DiTCacheConfig = field(default_factory=DiTCacheConfig)
# Resource limits
max_memory_gb: float = 16.0
max_gpu_utilization: float = 0.8
# Timeouts
request_timeout: int = 300
worker_timeout: int = 60
def validate(self) -> None:
"""Validate configuration parameters."""
if self.max_memory_gb <= 0:
raise ValueError("max_memory_gb must be positive")
if not 0 < self.max_gpu_utilization <= 1:
raise ValueError("max_gpu_utilization must be between 0 and 1")
```
### 5. Error Handling
```python
class CoreModuleError(Exception):
"""Base exception for core module errors."""
pass
class SchedulerError(CoreModuleError):
"""Scheduler-related errors."""
pass
class QueueFullError(SchedulerError):
"""Queue is at capacity."""
pass
class CacheError(CoreModuleError):
"""Cache-related errors."""
pass
class ResourceError(CoreModuleError):
"""Resource allocation errors."""
pass
```
### 6. Examples
#### Basic Usage
```python
from vllm_omni.core import CoreScheduler, CoreModuleConfig
from vllm_omni.request import create_text_request
# Create configuration
config = CoreModuleConfig(
scheduler=SchedulerConfig(scheduler_type=SchedulerType.FIFO),
max_memory_gb=8.0
)
# Initialize scheduler
scheduler = CoreScheduler(config)
await scheduler.start()
# Create and schedule a request
request = create_text_request(
request_id="req_001",
prompt="Hello, world!",
sampling_params=sampling_params
)
success = await scheduler.schedule_request(request)
if success:
result = await scheduler.process_request(request)
print(f"Result: {result}")
```
#### Advanced Usage
```python
# Custom scheduler with priority handling
config = CoreModuleConfig(
scheduler=SchedulerConfig(
scheduler_type=SchedulerType.PRIORITY,
priority_weights={"high": 2.0, "normal": 1.0, "low": 0.5}
)
)
scheduler = CoreScheduler(config)
await scheduler.start()
# Monitor scheduler status
status = scheduler.get_status()
print(f"Queue size: {status['queue_size']}")
print(f"Cache hit rate: {status['cache_hit_rate']:.2%}")
```
---
## Module-Specific Guidelines
### Core Module
- Focus on scheduling, caching, and resource management
- Provide clear interfaces for other modules
- Handle concurrency and thread safety
- Implement comprehensive monitoring
### Engine Module
- Handle model loading and inference
- Support both AR and diffusion models
- Provide unified interface for different model types
- Implement efficient memory management
### Executor Module
- Coordinate between different engines
- Handle request routing and load balancing
- Manage execution pipelines
- Provide error recovery mechanisms
### Worker Module
- Handle actual model execution
- Manage GPU resources
- Implement batching strategies
- Provide performance optimization
---
## Checklist for API Design
- [ ] **Clear Purpose**: Module purpose is well-defined
- [ ] **Complete Interface**: All public methods are documented
- [ ] **Error Handling**: Comprehensive error handling strategy
- [ ] **Configuration**: Flexible configuration system
- [ ] **Examples**: Basic and advanced usage examples
- [ ] **Type Hints**: All methods have proper type hints
- [ ] **Documentation**: Comprehensive docstrings
- [ ] **Testing**: Unit tests for all public methods
- [ ] **Integration**: Clear integration points with other modules
- [ ] **Performance**: Performance considerations documented
---
## Submission Guidelines
1. Create a new file: `docs/api/[module_name]_api.md`
2. Follow the template structure exactly
3. Include all required sections
4. Provide working code examples
5. Ensure all methods have proper docstrings
6. Include error handling strategies
7. Submit for review before implementation
This template ensures consistency and completeness across all vLLM-omni modules.
-235
View File
@@ -1,235 +0,0 @@
# vLLM-omni API Documentation
This directory contains comprehensive API design documentation for all core modules in vLLM-omni. These templates provide a standardized structure for designing and implementing the core, engine, executor, and worker modules.
## 📋 API Design Templates
### 1. [API Design Template](API_DESIGN_TEMPLATE.md)
**Master template** for creating API documentation for any module in vLLM-omni. Use this as a starting point for new modules.
### 2. [Core Module API](core_module_api.md)
**Core module** provides fundamental scheduling, caching, and resource management functionality.
**Key Components:**
- Request scheduling and prioritization
- DiT cache management for diffusion models
- Resource allocation and coordination
- Inter-module communication
### 3. [Engine Module API](engine_module_api.md)
**Engine module** handles model loading, inference execution, and output processing.
**Key Components:**
- Model loading and initialization
- Inference execution for AR and diffusion models
- Input preprocessing for various modalities
- Output postprocessing and formatting
### 4. [Executor Module API](executor_module_api.md)
**Executor module** coordinates and manages request execution across different engines and workers.
**Key Components:**
- Request routing and load balancing
- Execution pipeline coordination
- Worker management and task distribution
- Error handling and recovery
### 5. [Worker Module API](worker_module_api.md)
**Worker module** provides the actual execution environment for model inference.
**Key Components:**
- Model execution and inference
- GPU resource management
- Request batching and processing
- Performance optimization
## 🎯 How to Use These Templates
### For New Module Development
1. **Start with the Master Template**
- Copy `API_DESIGN_TEMPLATE.md`
- Rename to `[module_name]_api.md`
- Follow the structure exactly
2. **Fill in Module-Specific Details**
- Update the module overview
- Define core classes and interfaces
- Specify public API methods
- Add configuration options
- Define error handling strategy
- Provide usage examples
3. **Review and Validate**
- Ensure all sections are complete
- Verify code examples work
- Check for consistency with other modules
- Validate configuration options
### For Existing Module Updates
1. **Update API Documentation**
- Modify existing API files
- Add new methods and classes
- Update examples and configuration
- Maintain backward compatibility notes
2. **Version Control**
- Track changes in git
- Use clear commit messages
- Tag major API changes
## 📚 Template Structure
Each API template follows this standardized structure:
### 1. Module Overview
- **Purpose**: What the module does
- **Responsibilities**: Key responsibilities
- **Dependencies**: Required modules
- **Integration Points**: How it connects with other modules
### 2. Core Classes/Interfaces
- **Base Classes**: Abstract base classes
- **Implementation Classes**: Concrete implementations
- **Data Structures**: Key data models
### 3. Public API Methods
- **Initialization**: Constructor and setup
- **Core Operations**: Main functionality
- **Configuration**: Settings management
- **Lifecycle Management**: Start/stop/cleanup
- **Monitoring**: Status and metrics
### 4. Configuration
- **Configuration Classes**: Dataclasses for settings
- **Required Parameters**: Must-have settings
- **Optional Parameters**: Optional settings with defaults
- **Validation**: Parameter validation rules
### 5. Error Handling
- **Custom Exceptions**: Module-specific errors
- **Error Codes**: Standardized error codes
- **Recovery Strategies**: Error recovery approaches
### 6. Examples
- **Basic Usage**: Simple usage examples
- **Advanced Usage**: Complex scenarios
- **Integration Examples**: Multi-module usage
## 🔧 Implementation Guidelines
### Code Standards
- Use type hints for all methods
- Include comprehensive docstrings
- Follow PEP 8 style guidelines
- Use async/await for I/O operations
### Error Handling
- Define specific exception types
- Provide meaningful error messages
- Include error recovery strategies
- Log errors appropriately
### Configuration
- Use dataclasses for configuration
- Provide sensible defaults
- Include validation methods
- Support environment variables
### Testing
- Write unit tests for all public methods
- Include integration tests
- Test error conditions
- Validate configuration options
## 🚀 Getting Started
### For Developers
1. **Choose a Module**
- Pick the module you want to work on
- Read the corresponding API documentation
- Understand the responsibilities and interfaces
2. **Set Up Development Environment**
```bash
# Clone the repository
git clone https://github.com/hsliuustc0106/vllm-omni.git
cd vllm-omni
# Install dependencies
pip install -r requirements-dev.txt
# Set up pre-commit hooks
pre-commit install
```
3. **Start Implementation**
- Create the module directory
- Implement base classes first
- Add concrete implementations
- Write tests as you go
### For Contributors
1. **Review API Documentation**
- Read through all module APIs
- Understand the overall architecture
- Identify areas for improvement
2. **Propose Changes**
- Create issues for API changes
- Discuss in pull requests
- Update documentation accordingly
3. **Submit Contributions**
- Follow the coding standards
- Include tests for new features
- Update documentation
- Submit pull requests
## 📖 Additional Resources
### Architecture Overview
- [vLLM-omni Architecture Design](../architecture/vLLM-omni%20arch%20design%20doc.md)
- [Component Categorization](../../ARCHITECTURE_CATEGORIZATION.md)
### Development Guides
- [Development Setup](../../README.md#development)
- [Testing Guidelines](../../tests/README.md)
- [Contributing Guidelines](../../CONTRIBUTING.md)
### Examples
- [Basic Examples](../../examples/basic/)
- [Advanced Examples](../../examples/advanced/)
- [Multimodal Examples](../../examples/multimodal/)
## 🤝 Contributing
We welcome contributions to improve the API documentation and implementation. Please:
1. **Follow the Template Structure**
- Use the master template as a guide
- Maintain consistency across modules
- Include all required sections
2. **Provide Working Examples**
- Test all code examples
- Include both basic and advanced usage
- Show integration patterns
3. **Keep Documentation Updated**
- Update docs when APIs change
- Version control documentation changes
- Maintain backward compatibility notes
## 📝 Notes
- All API documentation should be kept up-to-date with implementation
- Code examples should be tested and working
- Configuration options should be validated
- Error handling should be comprehensive
- Performance considerations should be documented
For questions or suggestions about the API documentation, please open an issue or start a discussion in the repository.
+69
View File
@@ -0,0 +1,69 @@
# Summary
## Entry Points
Main entry points for vLLM-omni inference and serving.
- [vllm_omni.entrypoints.omni_llm.OmniLLM][]
- [vllm_omni.entrypoints.omni_llm.OmniStageLLM][]
- [vllm_omni.entrypoints.omni_stage.OmniStage][]
- [vllm_omni.entrypoints.cli.serve.OmniServeCommand][]
## Inputs
Input data structures for multi-modal inputs.
- [vllm_omni.inputs.data.OmniTokenInputs][]
- [vllm_omni.inputs.data.OmniTokensPrompt][]
- [vllm_omni.inputs.data.OmniEmbedsPrompt][]
- [vllm_omni.inputs.parse.parse_singleton_prompt_omni][]
- [vllm_omni.inputs.preprocess.OmniInputPreprocessor][]
## Engine
Engine classes for offline and online inference.
- [vllm_omni.engine.arg_utils.OmniEngineArgs][]
- [vllm_omni.engine.processor.OmniProcessor][]
- [vllm_omni.engine.output_processor.OmniRequestState][]
- [vllm_omni.engine.output_processor.MultimodalOutputProcessor][]
- [vllm_omni.engine.PromptEmbedsPayload][]
- [vllm_omni.engine.AdditionalInformationPayload][]
- [vllm_omni.engine.OmniEngineCoreRequest][]
## Core
Core scheduling and caching components.
- [vllm_omni.core.sched.scheduler.OmniScheduler][]
- [vllm_omni.core.sched.diffusion_scheduler.DiffusionScheduler][]
- [vllm_omni.core.sched.output.OmniNewRequestData][]
- [vllm_omni.core.dit_cache_manager.DiTCacheManager][]
## Model Executor
Model execution components.
- [vllm_omni.model_executor.models.qwen2_5_omni.Qwen2_5OmniForConditionalGeneration][]
- [vllm_omni.model_executor.models.qwen2_5_omni_talker.Qwen2_5OmniTalkerForConditionalGeneration][]
- [vllm_omni.model_executor.models.qwen2_5_omni_thinker.Qwen2_5OmniThinkerForConditionalGeneration][]
- [vllm_omni.model_executor.models.qwen2_5_omni_token2wav.Qwen2_5OmniToken2WavForConditionalGenerationVLLM][]
- [vllm_omni.model_executor.layers.mrope.MRotaryEmbedding][]
- [vllm_omni.model_executor.models.registry.OmniModelRegistry][]
## Configuration
Configuration classes.
- [vllm_omni.config.OmniModelConfig][]
## Workers
Worker classes and model runners for distributed inference.
- [vllm_omni.worker.gpu_ar_worker.GPUARWorker][]
- [vllm_omni.worker.gpu_diffusion_worker.GPUDiffusionWorker][]
- [vllm_omni.worker.gpu_model_runner.OmniGPUModelRunner][]
- [vllm_omni.worker.gpu_ar_model_runner.GPUARModelRunner][]
- [vllm_omni.worker.gpu_diffusion_model_runner.GPUDiffusionModelRunner][]