scheduler api design
This commit is contained in:
hsliu committed
1 parent
8dbe6db1be
commit
0c5caf6ab1
5 files changed
+1253
No files matched your search
@@ -0,0 +1,235 @@
|
||||
# vLLM-omni API Documentation
|
||||
|
||||
This directory contains comprehensive API design documentation for all core modules in vLLM-omni. These templates provide a standardized structure for designing and implementing the core, engine, executor, and worker modules.
|
||||
|
||||
## 📋 API Design Templates
|
||||
|
||||
### 1. [API Design Template](API_DESIGN_TEMPLATE.md)
|
||||
**Master template** for creating API documentation for any module in vLLM-omni. Use this as a starting point for new modules.
|
||||
|
||||
### 2. [Core Module API](core_module_api.md)
|
||||
**Core module** provides fundamental scheduling, caching, and resource management functionality.
|
||||
|
||||
**Key Components:**
|
||||
- Request scheduling and prioritization
|
||||
- DiT cache management for diffusion models
|
||||
- Resource allocation and coordination
|
||||
- Inter-module communication
|
||||
|
||||
### 3. [Engine Module API](engine_module_api.md)
|
||||
**Engine module** handles model loading, inference execution, and output processing.
|
||||
|
||||
**Key Components:**
|
||||
- Model loading and initialization
|
||||
- Inference execution for AR and diffusion models
|
||||
- Input preprocessing for various modalities
|
||||
- Output postprocessing and formatting
|
||||
|
||||
### 4. [Executor Module API](executor_module_api.md)
|
||||
**Executor module** coordinates and manages request execution across different engines and workers.
|
||||
|
||||
**Key Components:**
|
||||
- Request routing and load balancing
|
||||
- Execution pipeline coordination
|
||||
- Worker management and task distribution
|
||||
- Error handling and recovery
|
||||
|
||||
### 5. [Worker Module API](worker_module_api.md)
|
||||
**Worker module** provides the actual execution environment for model inference.
|
||||
|
||||
**Key Components:**
|
||||
- Model execution and inference
|
||||
- GPU resource management
|
||||
- Request batching and processing
|
||||
- Performance optimization
|
||||
|
||||
## 🎯 How to Use These Templates
|
||||
|
||||
### For New Module Development
|
||||
|
||||
1. **Start with the Master Template**
|
||||
- Copy `API_DESIGN_TEMPLATE.md`
|
||||
- Rename to `[module_name]_api.md`
|
||||
- Follow the structure exactly
|
||||
|
||||
2. **Fill in Module-Specific Details**
|
||||
- Update the module overview
|
||||
- Define core classes and interfaces
|
||||
- Specify public API methods
|
||||
- Add configuration options
|
||||
- Define error handling strategy
|
||||
- Provide usage examples
|
||||
|
||||
3. **Review and Validate**
|
||||
- Ensure all sections are complete
|
||||
- Verify code examples work
|
||||
- Check for consistency with other modules
|
||||
- Validate configuration options
|
||||
|
||||
### For Existing Module Updates
|
||||
|
||||
1. **Update API Documentation**
|
||||
- Modify existing API files
|
||||
- Add new methods and classes
|
||||
- Update examples and configuration
|
||||
- Maintain backward compatibility notes
|
||||
|
||||
2. **Version Control**
|
||||
- Track changes in git
|
||||
- Use clear commit messages
|
||||
- Tag major API changes
|
||||
|
||||
## 📚 Template Structure
|
||||
|
||||
Each API template follows this standardized structure:
|
||||
|
||||
### 1. Module Overview
|
||||
- **Purpose**: What the module does
|
||||
- **Responsibilities**: Key responsibilities
|
||||
- **Dependencies**: Required modules
|
||||
- **Integration Points**: How it connects with other modules
|
||||
|
||||
### 2. Core Classes/Interfaces
|
||||
- **Base Classes**: Abstract base classes
|
||||
- **Implementation Classes**: Concrete implementations
|
||||
- **Data Structures**: Key data models
|
||||
|
||||
### 3. Public API Methods
|
||||
- **Initialization**: Constructor and setup
|
||||
- **Core Operations**: Main functionality
|
||||
- **Configuration**: Settings management
|
||||
- **Lifecycle Management**: Start/stop/cleanup
|
||||
- **Monitoring**: Status and metrics
|
||||
|
||||
### 4. Configuration
|
||||
- **Configuration Classes**: Dataclasses for settings
|
||||
- **Required Parameters**: Must-have settings
|
||||
- **Optional Parameters**: Optional settings with defaults
|
||||
- **Validation**: Parameter validation rules
|
||||
|
||||
### 5. Error Handling
|
||||
- **Custom Exceptions**: Module-specific errors
|
||||
- **Error Codes**: Standardized error codes
|
||||
- **Recovery Strategies**: Error recovery approaches
|
||||
|
||||
### 6. Examples
|
||||
- **Basic Usage**: Simple usage examples
|
||||
- **Advanced Usage**: Complex scenarios
|
||||
- **Integration Examples**: Multi-module usage
|
||||
|
||||
## 🔧 Implementation Guidelines
|
||||
|
||||
### Code Standards
|
||||
- Use type hints for all methods
|
||||
- Include comprehensive docstrings
|
||||
- Follow PEP 8 style guidelines
|
||||
- Use async/await for I/O operations
|
||||
|
||||
### Error Handling
|
||||
- Define specific exception types
|
||||
- Provide meaningful error messages
|
||||
- Include error recovery strategies
|
||||
- Log errors appropriately
|
||||
|
||||
### Configuration
|
||||
- Use dataclasses for configuration
|
||||
- Provide sensible defaults
|
||||
- Include validation methods
|
||||
- Support environment variables
|
||||
|
||||
### Testing
|
||||
- Write unit tests for all public methods
|
||||
- Include integration tests
|
||||
- Test error conditions
|
||||
- Validate configuration options
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### For Developers
|
||||
|
||||
1. **Choose a Module**
|
||||
- Pick the module you want to work on
|
||||
- Read the corresponding API documentation
|
||||
- Understand the responsibilities and interfaces
|
||||
|
||||
2. **Set Up Development Environment**
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/hsliuustc0106/vllm-omni.git
|
||||
cd vllm-omni
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements-dev.txt
|
||||
|
||||
# Set up pre-commit hooks
|
||||
pre-commit install
|
||||
```
|
||||
|
||||
3. **Start Implementation**
|
||||
- Create the module directory
|
||||
- Implement base classes first
|
||||
- Add concrete implementations
|
||||
- Write tests as you go
|
||||
|
||||
### For Contributors
|
||||
|
||||
1. **Review API Documentation**
|
||||
- Read through all module APIs
|
||||
- Understand the overall architecture
|
||||
- Identify areas for improvement
|
||||
|
||||
2. **Propose Changes**
|
||||
- Create issues for API changes
|
||||
- Discuss in pull requests
|
||||
- Update documentation accordingly
|
||||
|
||||
3. **Submit Contributions**
|
||||
- Follow the coding standards
|
||||
- Include tests for new features
|
||||
- Update documentation
|
||||
- Submit pull requests
|
||||
|
||||
## 📖 Additional Resources
|
||||
|
||||
### Architecture Overview
|
||||
- [vLLM-omni Architecture Design](../architecture/vLLM-omni%20arch%20design%20doc.md)
|
||||
- [Component Categorization](../../ARCHITECTURE_CATEGORIZATION.md)
|
||||
|
||||
### Development Guides
|
||||
- [Development Setup](../../README.md#development)
|
||||
- [Testing Guidelines](../../tests/README.md)
|
||||
- [Contributing Guidelines](../../CONTRIBUTING.md)
|
||||
|
||||
### Examples
|
||||
- [Basic Examples](../../examples/basic/)
|
||||
- [Advanced Examples](../../examples/advanced/)
|
||||
- [Multimodal Examples](../../examples/multimodal/)
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
We welcome contributions to improve the API documentation and implementation. Please:
|
||||
|
||||
1. **Follow the Template Structure**
|
||||
- Use the master template as a guide
|
||||
- Maintain consistency across modules
|
||||
- Include all required sections
|
||||
|
||||
2. **Provide Working Examples**
|
||||
- Test all code examples
|
||||
- Include both basic and advanced usage
|
||||
- Show integration patterns
|
||||
|
||||
3. **Keep Documentation Updated**
|
||||
- Update docs when APIs change
|
||||
- Version control documentation changes
|
||||
- Maintain backward compatibility notes
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- All API documentation should be kept up-to-date with implementation
|
||||
- Code examples should be tested and working
|
||||
- Configuration options should be validated
|
||||
- Error handling should be comprehensive
|
||||
- Performance considerations should be documented
|
||||
|
||||
For questions or suggestions about the API documentation, please open an issue or start a discussion in the repository.
|
||||
Reference in new issue
Block a user