del unnessearcy files
This commit is contained in:
1 parent
f8a6606176
commit
c330e5e8f3
10 files changed
+10
-1196
No files matched your search
-179
@@ -1,179 +0,0 @@
|
||||
# vLLM-omni Product Requirements Document (PRD)
|
||||
|
||||
## 1. Product Overview
|
||||
|
||||
### 1.1 Product Name
|
||||
vLLM-omni: Multi-modality models inference and serving with non-autoregressive structures
|
||||
|
||||
### 1.2 Product Vision
|
||||
Extend vLLM beyond traditional text-based, autoregressive generation to support multi-modality models with non-autoregressive structures and non-textual outputs while maintaining vLLM's proven architecture and performance.
|
||||
|
||||
### 1.3 Target Users
|
||||
- AI researchers working with multimodal models
|
||||
- ML engineers building production inference systems
|
||||
- Developers integrating DiT (Diffusion Transformer) models
|
||||
- Organizations requiring efficient multimodal model serving
|
||||
|
||||
## 2. Core Requirements
|
||||
|
||||
### 2.1 Functional Requirements
|
||||
|
||||
#### 2.1.1 Multi-Stage Processing
|
||||
- **REQ-001**: Support stage-based model processing where each stage can use different engine types (AR/DiT)
|
||||
- **REQ-002**: Enable sequential processing through multiple stages with data flow between stages
|
||||
- **REQ-003**: Support both autoregressive (AR) and diffusion (DiT) model stages
|
||||
|
||||
#### 2.1.2 vLLM Compatibility
|
||||
- **REQ-004**: Maintain full compatibility with vLLM V1 architecture (AsyncLLM and EngineCore patterns)
|
||||
- **REQ-005**: Support existing vLLM CLI commands with `--omni` flag extension
|
||||
- **REQ-006**: Reuse vLLM's multiprocess worker architecture for scalability
|
||||
|
||||
#### 2.1.3 Multimodal Support
|
||||
- **REQ-007**: Support text, image, and latent space inputs/outputs
|
||||
- **REQ-008**: Enable image-to-image, text-to-image, and text-to-text generation
|
||||
- **REQ-009**: Support hidden state passing between AR and DiT stages
|
||||
|
||||
#### 2.1.4 CLI and API
|
||||
- **REQ-010**: Provide CLI command: `vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8000`
|
||||
- **REQ-011**: Support both online (AsyncOmniLLM) and offline (OmniLLM) inference modes
|
||||
- **REQ-012**: Maintain vLLM's existing API compatibility
|
||||
|
||||
### 2.2 Non-Functional Requirements
|
||||
|
||||
#### 2.2.1 Performance
|
||||
- **REQ-013**: Maintain vLLM's inference performance for AR stages
|
||||
- **REQ-014**: Optimize DiT stage processing with caching mechanisms
|
||||
- **REQ-015**: Support distributed inference across multiple GPUs
|
||||
|
||||
#### 2.2.2 Scalability
|
||||
- **REQ-016**: Support horizontal scaling through vLLM's worker process pattern
|
||||
- **REQ-017**: Enable efficient memory management for large multimodal models
|
||||
- **REQ-018**: Support batch processing for multiple requests
|
||||
|
||||
#### 2.2.3 Extensibility
|
||||
- **REQ-019**: Easy integration of new modalities and model architectures
|
||||
- **REQ-020**: Pluggable scheduler and executor components
|
||||
- **REQ-021**: Support for future non-autoregressive model types
|
||||
|
||||
## 3. Technical Architecture
|
||||
|
||||
### 3.1 Core Components
|
||||
|
||||
#### 3.1.1 Entry Points
|
||||
- **OmniServeCommand**: CLI wrapper that intercepts vLLM commands with `--omni` flag
|
||||
- **OmniLLM**: Offline inference class supporting multi-stage processing
|
||||
- **AsyncOmniLLM**: Online inference class with asynchronous processing
|
||||
|
||||
#### 3.1.2 Stage Management
|
||||
- **OmniStageConfig**: Configuration for each processing stage
|
||||
- **Stage Engine List**: Multiple AsyncLLM instances for each stage
|
||||
- **Stage I/O Management**: Data flow between stages
|
||||
|
||||
#### 3.1.3 Engine Components
|
||||
- **EngineCore**: Reused from vLLM (no changes needed)
|
||||
- **OmniDiffusionScheduler**: New scheduler for DiT models
|
||||
- **DiTCacheManager**: Caching system for DiT optimization
|
||||
- **MultiprocExecutor**: Reused from vLLM for DiT without diffusers
|
||||
- **DiffusersPipelineExecutor**: New executor for diffusers integration
|
||||
|
||||
#### 3.1.4 Model Runners
|
||||
- **OmniDiffusionModelRunner**: Handles DiT model execution
|
||||
- **OmniARModelRunner**: Handles AR model execution with hidden state output
|
||||
- **ModelRunnerOutput**: Extended to support multimodal outputs via pooler_output
|
||||
|
||||
#### 3.1.5 Output Processing
|
||||
- **MultimodalOutputProcessor**: Handles final multimodal output processing
|
||||
- **RequestState**: Extended to support pooling outputs
|
||||
- **Output Handlers**: Type-specific output processing
|
||||
|
||||
### 3.2 Data Flow
|
||||
```
|
||||
API Server → OmniLLM/AsyncOmniLLM → LLMEngine/AsyncLLM → Engine Core
|
||||
→ Scheduler (AR/DiT) → Executor (AR/DiT) → Worker (AR/DiT)
|
||||
→ ModelRunner (AR/DiT) → RequestState → OutputProcessor → Final Output
|
||||
```
|
||||
|
||||
## 4. Implementation Phases
|
||||
|
||||
### Phase 1: Foundation (Weeks 1-2)
|
||||
- Package structure and dependencies
|
||||
- Basic OmniLLM and AsyncOmniLLM classes
|
||||
- Stage configuration system
|
||||
- CLI integration
|
||||
|
||||
### Phase 2: Core Components (Weeks 3-4)
|
||||
- DiT scheduler implementation
|
||||
- Model runners for AR and DiT
|
||||
- Basic output processing
|
||||
|
||||
### Phase 3: Advanced Features (Weeks 5-6)
|
||||
- Caching system implementation
|
||||
- Multimodal output processing
|
||||
- Request state management
|
||||
|
||||
### Phase 4: Integration & Testing (Weeks 7-8)
|
||||
- End-to-end integration
|
||||
- Comprehensive testing
|
||||
- Performance optimization
|
||||
- Documentation
|
||||
|
||||
## 5. Success Criteria
|
||||
|
||||
### 5.1 Functional Success
|
||||
- [ ] Successfully run `vllm serve model --omni` command
|
||||
- [ ] Process multi-stage AR→DiT pipelines
|
||||
- [ ] Generate multimodal outputs (text + image)
|
||||
- [ ] Maintain vLLM API compatibility
|
||||
|
||||
### 5.2 Performance Success
|
||||
- [ ] AR stage performance within 5% of native vLLM
|
||||
- [ ] DiT stage processing with reasonable latency
|
||||
- [ ] Memory usage comparable to vLLM for equivalent models
|
||||
|
||||
### 5.3 Quality Success
|
||||
- [ ] 90%+ test coverage
|
||||
- [ ] All integration tests passing
|
||||
- [ ] Documentation complete and accurate
|
||||
|
||||
## 6. Risk Assessment
|
||||
|
||||
### 6.1 Technical Risks
|
||||
- **High**: vLLM API changes breaking compatibility
|
||||
- **Medium**: DiT model integration complexity
|
||||
- **Low**: Performance overhead from multi-stage processing
|
||||
|
||||
### 6.2 Mitigation Strategies
|
||||
- Regular vLLM compatibility testing
|
||||
- Incremental DiT integration with fallback options
|
||||
- Performance benchmarking at each stage
|
||||
|
||||
## 7. Dependencies
|
||||
|
||||
### 7.1 External Dependencies
|
||||
- vLLM >= 0.10.2
|
||||
- PyTorch >= 2.7
|
||||
- Transformers >= 4.30.0
|
||||
- FastAPI, Uvicorn for API serving
|
||||
- Ray for distributed computing
|
||||
|
||||
### 7.2 Optional Dependencies
|
||||
- xDiT for DiT acceleration
|
||||
- Cache-DiT for advanced caching
|
||||
- Diffusers for pipeline-based DiT models
|
||||
|
||||
## 8. Future Roadmap
|
||||
|
||||
### 8.1 Short-term (3 months)
|
||||
- Additional DiT model support
|
||||
- Performance optimizations
|
||||
- Enhanced caching strategies
|
||||
|
||||
### 8.2 Medium-term (6 months)
|
||||
- Support for video generation models
|
||||
- Advanced scheduling strategies
|
||||
- Multi-GPU DiT optimization
|
||||
|
||||
### 8.3 Long-term (12 months)
|
||||
- Custom model architecture support
|
||||
- Advanced multimodal fusion
|
||||
- Production deployment tools
|
||||
@@ -1,5 +0,0 @@
|
||||
# vLLM-omni Documentation
|
||||
|
||||
Welcome to the vLLM-omni documentation! This documentation provides comprehensive information about using and developing with vLLM-omni.
|
||||
|
||||
# TODO (add examples)
|
||||
@@ -1,157 +0,0 @@
|
||||
# vLLM-omni Software Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
vLLM-omni extends vLLM with multi-stage, multimodal capabilities. It supports both autoregressive (AR) stages and diffusion (DiT) stages, enabling pipelines such as text→image generation.
|
||||
|
||||
## Architecture Principles
|
||||
|
||||
1. **vLLM V1 Compatibility** – Reuse LLMEngine/AsyncLLM and executor abstractions.
|
||||
2. **Stage-based Processing** – Each stage encapsulates its own engine and configuration.
|
||||
3. **Executor/Worker Separation** – Executors remain thin; model-specific logic lives in workers and runners.
|
||||
4. **Extensibility** – New modalities can be added by implementing dedicated stage configs, workers, and output processors.
|
||||
|
||||
## Key Data Flow
|
||||
|
||||
```
|
||||
Request → OmniLLM/AsyncOmniLLM → StageManager → LLMEngine/Executor → Worker → ModelRunner → OutputProcessor → Response
|
||||
```
|
||||
|
||||
For diffusion stages, the executor hands off to `DiffusionGPUWorker`, which wraps `DiffusionModelRunner` (responsible for diffusers pipeline loading and inference).
|
||||
|
||||
## Core Components (Ordered by Data Flow)
|
||||
|
||||
### 1. Installation
|
||||
```bash
|
||||
pip install vllm
|
||||
pip install vllm-omni
|
||||
```
|
||||
|
||||
### 2. Online inference launch (Entry Point)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-0.6B --omni --port 8000
|
||||
```
|
||||
|
||||
### 3. Omni Serve Command (CLI)
|
||||
```python
|
||||
# vllm_omni/entrypoints/cli/main.py
|
||||
if "--omni" in sys.argv:
|
||||
omni_args = [arg for arg in sys.argv[1:] if arg not in {"--omni", "serve"}]
|
||||
OmniServeCommand().run(omni_args)
|
||||
else:
|
||||
from vllm.entrypoints.cli.main import main as vllm_main
|
||||
vllm_main()
|
||||
```
|
||||
|
||||
### 4. Offline Inference (OmniLLM)
|
||||
```python
|
||||
# vllm_omni/entrypoints/omni_llm.py
|
||||
class OmniLLM(LLM):
|
||||
def __init__(..., stage_configs):
|
||||
super().__init__(model=stage_configs[0].model_path if stage_configs else "test-model")
|
||||
self.stage_manager = StageManager(stage_configs)
|
||||
self.output_processor = MultimodalOutputProcessor()
|
||||
self.stage_manager.initialize_engines()
|
||||
|
||||
def generate(...):
|
||||
current_output = None
|
||||
for idx, (config, args) in enumerate(zip(self.stage_configs, stage_args_list)):
|
||||
engine = self.stage_manager.get_engine(idx)
|
||||
processed = self._process_stage_inputs(config, args, current_output)
|
||||
stage_output = self._execute_stage(engine, processed, stage_config=config)
|
||||
current_output = stage_output
|
||||
config.stage_output = stage_output
|
||||
return self.output_processor.process_output(current_output)
|
||||
```
|
||||
|
||||
### 5. Stage Manager
|
||||
```python
|
||||
# vllm_omni/core/stage_manager.py
|
||||
class StageManager:
|
||||
def initialize_engines(self):
|
||||
if self._initialized:
|
||||
return
|
||||
for config in self.stage_configs:
|
||||
if config.engine_type == "AR":
|
||||
engine = self._create_ar_engine(config)
|
||||
elif config.engine_type == "DiT":
|
||||
engine = self._create_dit_engine(config)
|
||||
self.engine_list.append(engine)
|
||||
self._initialized = True
|
||||
```
|
||||
|
||||
### 6. Diffusion Execution (DiT Stage)
|
||||
```python
|
||||
# vllm_omni/worker/diffusion_model_runner.py
|
||||
class DiffusionModelRunner:
|
||||
def __init__(self, model_path, pipeline_name=None, device=None, dtype=None):
|
||||
self.device, self.torch_dtype = self._resolve_device_and_dtype(device, dtype)
|
||||
self._pipeline = self._load_pipeline(model_path, pipeline_name)
|
||||
|
||||
def generate(...):
|
||||
output = self._pipeline(...)
|
||||
return DiffusionRunnerOutput(prompt=prompt, images=getattr(output, "images", []) or [])
|
||||
```
|
||||
|
||||
```python
|
||||
# vllm_omni/worker/gpu_diffusion_worker.py
|
||||
class DiffusionGPUWorker:
|
||||
def __init__(self, model_path, ...):
|
||||
self.model_runner = DiffusionModelRunner(model_path, ...)
|
||||
|
||||
def generate(...):
|
||||
return self.model_runner.generate(...)
|
||||
```
|
||||
|
||||
```python
|
||||
# vllm_omni/executor/diffusers_executor.py
|
||||
class DiffusersPipelineExecutor:
|
||||
def __init__(self, model_path, ...):
|
||||
self.worker = DiffusionGPUWorker(model_path, ...)
|
||||
|
||||
def generate(...):
|
||||
return self.worker.generate(...)
|
||||
```
|
||||
|
||||
This mirrors the standard executor→worker→runner layering in vLLM.
|
||||
|
||||
### 7. Stage Configuration
|
||||
```python
|
||||
# vllm_omni/config/stage_config.py
|
||||
@dataclass
|
||||
class OmniStageConfig:
|
||||
stage_id: int
|
||||
engine_type: Literal["AR", "DiT"]
|
||||
model_path: str
|
||||
input_modalities: List[str]
|
||||
output_modalities: List[str]
|
||||
vllm_config: Optional[VllmConfig] = None
|
||||
executor_class: Optional[type[Executor]] = None
|
||||
dit_config: Optional[DiTConfig] = None
|
||||
```
|
||||
|
||||
- AR stages default to `UniProcExecutor`.
|
||||
- DiT stages set `executor_class` to `DiffusersPipelineExecutor` when `use_diffusers=True`.
|
||||
|
||||
### 8. Output Processing
|
||||
```python
|
||||
# vllm_omni/engine/output_processor.py
|
||||
class MultimodalOutputProcessor(OutputProcessor):
|
||||
def process_output(self, engine_core_output):
|
||||
output_type = self._detect_output_type(engine_core_output)
|
||||
handler = self.output_handlers.get(output_type, self._process_pooling_output)
|
||||
return handler(engine_core_output)
|
||||
```
|
||||
|
||||
### 9. CLI Package Configuration
|
||||
```toml
|
||||
# pyproject.toml
|
||||
[project.scripts]
|
||||
vllm = "vllm_omni.entrypoints.cli.main:main"
|
||||
vllm-omni = "vllm_omni.entrypoints.cli.main:main"
|
||||
```
|
||||
|
||||
## Future Work
|
||||
- Scheduler adaptations for diffusion batching.
|
||||
- Advanced output processors for audio/video.
|
||||
- Multi-worker execution (ray / multiprocess) for heavy DiT stages.
|
||||
@@ -300,7 +300,7 @@ omni = "vllm_omni.plugin:OmniPlugin"
|
||||
|
||||
### 7.1 Package Installation
|
||||
```bash
|
||||
pip install vllm>=0.2.0
|
||||
pip install vllm>=0.10.2
|
||||
pip install vllm-omni
|
||||
```
|
||||
|
||||
|
||||
@@ -1,758 +0,0 @@
|
||||
# vLLM-omni Test Design Document
|
||||
|
||||
## 1. Testing Strategy Overview
|
||||
|
||||
### 1.1 Testing Pyramid
|
||||
```
|
||||
E2E Tests (10%)
|
||||
/ \
|
||||
Integration Tests (30%)
|
||||
/ \
|
||||
Unit Tests (60%)
|
||||
```
|
||||
|
||||
### 1.2 Test Categories
|
||||
- **Unit Tests**: Individual component testing with mocked dependencies
|
||||
- **Integration Tests**: Component interaction testing with real vLLM integration
|
||||
- **End-to-End Tests**: Full pipeline testing with real models
|
||||
- **Performance Tests**: Benchmarking and profiling
|
||||
- **Compatibility Tests**: vLLM version compatibility validation
|
||||
|
||||
## 2. Test Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── __init__.py
|
||||
├── conftest.py # Shared fixtures and configuration
|
||||
├── unit/ # Unit tests (60%)
|
||||
│ ├── __init__.py
|
||||
│ ├── test_config/ # Configuration testing
|
||||
│ │ ├── test_stage_config.py
|
||||
│ │ ├── test_dit_config.py
|
||||
│ │ └── test_cache_config.py
|
||||
│ ├── test_core/ # Core component testing
|
||||
│ │ ├── test_omni_llm.py
|
||||
│ │ ├── test_async_omni_llm.py
|
||||
│ │ ├── test_stage_manager.py
|
||||
│ │ └── test_dit_cache_manager.py
|
||||
│ ├── test_scheduler/ # Scheduler testing
|
||||
│ │ ├── test_diffusion_scheduler.py
|
||||
│ │ └── test_scheduler_interface.py
|
||||
│ ├── test_executor/ # Executor testing
|
||||
│ │ ├── test_base_executor.py
|
||||
│ │ └── test_diffusers_executor.py
|
||||
│ ├── test_model_executor/ # Model runner testing
|
||||
│ │ ├── test_ar_model_runner.py
|
||||
│ │ └── test_dit_model_runner.py
|
||||
│ ├── test_engine/ # Engine testing
|
||||
│ │ ├── test_output_processor.py
|
||||
│ │ └── test_multimodal_processor.py
|
||||
│ ├── test_worker/ # Worker testing
|
||||
│ │ └── test_omni_worker.py
|
||||
│ └── test_utils/ # Utility testing
|
||||
│ ├── test_multimodal.py
|
||||
│ └── test_vae.py
|
||||
├── integration/ # Integration tests (30%)
|
||||
│ ├── __init__.py
|
||||
│ ├── test_vllm_integration.py
|
||||
│ ├── test_stage_processing.py
|
||||
│ ├── test_cli_integration.py
|
||||
│ └── test_api_compatibility.py
|
||||
├── e2e/ # End-to-end tests (10%)
|
||||
│ ├── __init__.py
|
||||
│ ├── test_full_pipeline.py
|
||||
│ ├── test_multimodal_generation.py
|
||||
│ └── test_performance.py
|
||||
├── benchmarks/ # Performance benchmarks
|
||||
│ ├── __init__.py
|
||||
│ ├── test_memory_usage.py
|
||||
│ ├── test_latency.py
|
||||
│ └── test_throughput.py
|
||||
└── fixtures/ # Test data and fixtures
|
||||
├── __init__.py
|
||||
├── sample_models/
|
||||
├── test_images/
|
||||
└── test_configs/
|
||||
```
|
||||
|
||||
## 3. Unit Test Specifications
|
||||
|
||||
### 3.1 Configuration Tests
|
||||
|
||||
#### test_stage_config.py
|
||||
```python
|
||||
import pytest
|
||||
from vllm_omni.config.stage_config import OmniStageConfig, DiTConfig, DiTCacheConfig
|
||||
|
||||
class TestOmniStageConfig:
|
||||
def test_ar_stage_config_creation(self):
|
||||
"""Test AR stage configuration creation"""
|
||||
config = OmniStageConfig(
|
||||
stage_id=0,
|
||||
engine_type="AR",
|
||||
model_path="test-model",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["text"]
|
||||
)
|
||||
assert config.engine_type == "AR"
|
||||
assert config.stage_id == 0
|
||||
|
||||
def test_dit_stage_config_creation(self):
|
||||
"""Test DiT stage configuration creation"""
|
||||
dit_config = DiTConfig(
|
||||
num_inference_steps=50,
|
||||
guidance_scale=7.5,
|
||||
use_diffusers=False,
|
||||
)
|
||||
config = OmniStageConfig(
|
||||
stage_id=1,
|
||||
engine_type="DiT",
|
||||
model_path="test-dit-model",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["image"],
|
||||
dit_config=dit_config
|
||||
)
|
||||
assert config.engine_type == "DiT"
|
||||
assert config.dit_config is not None
|
||||
|
||||
def test_invalid_engine_type(self):
|
||||
"""Test validation of engine type"""
|
||||
with pytest.raises(ValueError):
|
||||
OmniStageConfig(
|
||||
stage_id=0,
|
||||
engine_type="INVALID",
|
||||
model_path="test-model",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["text"]
|
||||
)
|
||||
```
|
||||
|
||||
#### test_dit_config.py
|
||||
```python
|
||||
import pytest
|
||||
from vllm.config import ModelConfig
|
||||
from vllm_omni.config import DiTConfig
|
||||
|
||||
class TestDiTConfig:
|
||||
def test_dit_config_defaults(self):
|
||||
"""Test DiT configuration with defaults"""
|
||||
config = DiTConfig(num_inference_steps=50)
|
||||
assert config.model_config is None
|
||||
assert config.use_diffusers is False
|
||||
assert config.diffusers_pipeline is None
|
||||
|
||||
def test_diffusers_config(self):
|
||||
"""Test DiT configuration with diffusers"""
|
||||
config = DiTConfig(
|
||||
model_config=ModelConfig(model="stabilityai/stable-diffusion-2-1"),
|
||||
num_inference_steps=50,
|
||||
use_diffusers=True,
|
||||
diffusers_pipeline="stable-diffusion",
|
||||
)
|
||||
assert config.use_diffusers is True
|
||||
assert config.diffusers_pipeline == "stable-diffusion"
|
||||
```
|
||||
|
||||
### 3.2 Core Component Tests
|
||||
|
||||
#### test_omni_llm.py
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from vllm_omni.core.omni_llm import OmniLLM
|
||||
from vllm_omni.config.stage_config import OmniStageConfig
|
||||
|
||||
class TestOmniLLM:
|
||||
@pytest.fixture
|
||||
def mock_stage_configs(self):
|
||||
"""Create mock stage configurations"""
|
||||
return [
|
||||
OmniStageConfig(
|
||||
stage_id=0,
|
||||
engine_type="AR",
|
||||
model_path="test-ar-model",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["text"]
|
||||
),
|
||||
OmniStageConfig(
|
||||
stage_id=1,
|
||||
engine_type="DiT",
|
||||
model_path="test-dit-model",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["image"]
|
||||
)
|
||||
]
|
||||
|
||||
@patch('vllm_omni.core.omni_llm.LLMEngine')
|
||||
def test_omni_llm_initialization(self, mock_llm_engine, mock_stage_configs):
|
||||
"""Test OmniLLM initialization"""
|
||||
omni_llm = OmniLLM(mock_stage_configs)
|
||||
assert len(omni_llm.engine_list) == 2
|
||||
assert omni_llm.stage_configs == mock_stage_configs
|
||||
|
||||
@patch('vllm_omni.core.omni_llm.LLMEngine')
|
||||
def test_stage_engine_creation(self, mock_llm_engine, mock_stage_configs):
|
||||
"""Test stage engine creation"""
|
||||
omni_llm = OmniLLM(mock_stage_configs)
|
||||
# Verify engines are created for each stage
|
||||
assert mock_llm_engine.from_vllm_config.call_count == 2
|
||||
|
||||
def test_process_stage_inputs_ar(self, mock_stage_configs):
|
||||
"""Test AR stage input processing"""
|
||||
omni_llm = OmniLLM(mock_stage_configs)
|
||||
ar_config = mock_stage_configs[0]
|
||||
stage_args = {"prompt": "test prompt"}
|
||||
|
||||
result = omni_llm._process_stage_inputs(ar_config, stage_args, None)
|
||||
assert "prompt" in result
|
||||
|
||||
def test_process_stage_inputs_dit(self, mock_stage_configs):
|
||||
"""Test DiT stage input processing"""
|
||||
omni_llm = OmniLLM(mock_stage_configs)
|
||||
dit_config = mock_stage_configs[1]
|
||||
stage_args = {"image": "test_image.jpg"}
|
||||
|
||||
with patch.object(omni_llm, 'vae') as mock_vae:
|
||||
mock_vae.encode.return_value = "encoded_image"
|
||||
result = omni_llm._process_stage_inputs(dit_config, stage_args, None)
|
||||
assert "encoded_image" in result
|
||||
```
|
||||
|
||||
#### test_async_omni_llm.py
|
||||
```python
|
||||
import pytest
|
||||
import asyncio
|
||||
from unittest.mock import Mock, patch, AsyncMock
|
||||
from vllm_omni.core.omni_llm import AsyncOmniLLM
|
||||
|
||||
class TestAsyncOmniLLM:
|
||||
@pytest.fixture
|
||||
def mock_stage_configs(self):
|
||||
"""Create mock stage configurations"""
|
||||
return [
|
||||
OmniStageConfig(
|
||||
stage_id=0,
|
||||
engine_type="AR",
|
||||
model_path="test-ar-model",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["text"]
|
||||
)
|
||||
]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_generation(self, mock_stage_configs):
|
||||
"""Test async generation"""
|
||||
with patch('vllm_omni.core.omni_llm.AsyncLLM') as mock_async_llm:
|
||||
async_omni_llm = AsyncOmniLLM(mock_stage_configs)
|
||||
|
||||
# Mock the generate_async method
|
||||
mock_async_llm.return_value.generate_async = AsyncMock(return_value=[])
|
||||
|
||||
result = await async_omni_llm.generate_async([{"prompt": "test"}])
|
||||
assert result == []
|
||||
```
|
||||
|
||||
### 3.3 Scheduler Tests
|
||||
|
||||
#### test_diffusion_scheduler.py
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from vllm_omni.core.sched.diffusion_scheduler import OmniDiffusionScheduler
|
||||
|
||||
class TestOmniDiffusionScheduler:
|
||||
@pytest.fixture
|
||||
def mock_configs(self):
|
||||
"""Create mock configurations"""
|
||||
return {
|
||||
'vllm_config': Mock(),
|
||||
'kv_cache_config': Mock(),
|
||||
'dit_cache_config': Mock(),
|
||||
'structured_output_manager': Mock(),
|
||||
'mm_registry': Mock(),
|
||||
'include_finished_set': False,
|
||||
'log_stats': False
|
||||
}
|
||||
|
||||
def test_scheduler_initialization(self, mock_configs):
|
||||
"""Test scheduler initialization"""
|
||||
scheduler = OmniDiffusionScheduler(**mock_configs)
|
||||
assert scheduler.dit_cache_manager is not None
|
||||
|
||||
def test_schedule_method(self, mock_configs):
|
||||
"""Test scheduling method"""
|
||||
scheduler = OmniDiffusionScheduler(**mock_configs)
|
||||
|
||||
with patch.object(scheduler, 'dit_cache_manager') as mock_cache:
|
||||
mock_cache.allocate_cache.return_value = Mock()
|
||||
result = scheduler.schedule()
|
||||
# Verify scheduling logic
|
||||
assert result is not None
|
||||
```
|
||||
|
||||
### 3.4 Model Runner Tests
|
||||
|
||||
#### test_ar_model_runner.py
|
||||
```python
|
||||
import pytest
|
||||
import torch
|
||||
from unittest.mock import Mock, patch
|
||||
from vllm_omni.model_executor.ar_model_runner import OmniARModelRunner
|
||||
|
||||
class TestOmniARModelRunner:
|
||||
@pytest.fixture
|
||||
def mock_runner(self):
|
||||
"""Create mock model runner"""
|
||||
with patch('vllm_omni.model_executor.ar_model_runner.GPUModelRunner.__init__'):
|
||||
runner = OmniARModelRunner(Mock(), Mock(), Mock())
|
||||
return runner
|
||||
|
||||
def test_execute_model_ar(self, mock_runner):
|
||||
"""Test AR model execution"""
|
||||
mock_scheduler_output = Mock()
|
||||
mock_scheduler_output.req_ids = ["req1"]
|
||||
mock_scheduler_output.req_id_to_index = {"req1": 0}
|
||||
|
||||
with patch.object(mock_runner, 'model') as mock_model:
|
||||
mock_model.forward.return_value = (torch.randn(1, 10, 768), None)
|
||||
|
||||
result = mock_runner.execute_model(mock_scheduler_output)
|
||||
|
||||
assert result.req_ids == ["req1"]
|
||||
assert len(result.pooler_output) > 0 # Hidden states
|
||||
```
|
||||
|
||||
#### test_dit_model_runner.py
|
||||
```python
|
||||
import pytest
|
||||
import torch
|
||||
from unittest.mock import Mock, patch
|
||||
from vllm_omni.model_executor.dit_model_runner import OmniDiffusionModelRunner
|
||||
|
||||
class TestOmniDiffusionModelRunner:
|
||||
@pytest.fixture
|
||||
def mock_runner(self):
|
||||
"""Create mock DiT model runner"""
|
||||
with patch('vllm_omni.model_executor.dit_model_runner.GPUModelRunner.__init__'):
|
||||
runner = OmniDiffusionModelRunner(Mock(), Mock(), Mock())
|
||||
return runner
|
||||
|
||||
def test_execute_model_dit(self, mock_runner):
|
||||
"""Test DiT model execution"""
|
||||
mock_scheduler_output = Mock()
|
||||
mock_scheduler_output.req_ids = ["req1"]
|
||||
mock_scheduler_output.req_id_to_index = {"req1": 0}
|
||||
|
||||
with patch.object(mock_runner, 'model') as mock_model:
|
||||
mock_model.forward.return_value = torch.randn(1, 3, 512, 512)
|
||||
|
||||
result = mock_runner.execute_model(mock_scheduler_output)
|
||||
|
||||
assert result.req_ids == ["req1"]
|
||||
assert len(result.pooler_output) > 0 # DiT output tensors
|
||||
```
|
||||
|
||||
## 4. Integration Tests
|
||||
|
||||
### 4.1 vLLM Integration Tests
|
||||
|
||||
#### test_vllm_integration.py
|
||||
```python
|
||||
import pytest
|
||||
from vllm_omni.core.omni_llm import OmniLLM
|
||||
from vllm_omni.config.stage_config import OmniStageConfig
|
||||
|
||||
class TestVLLMIntegration:
|
||||
@pytest.mark.integration
|
||||
def test_vllm_engine_creation(self):
|
||||
"""Test creation of vLLM engines"""
|
||||
stage_configs = [
|
||||
OmniStageConfig(
|
||||
stage_id=0,
|
||||
engine_type="AR",
|
||||
model_path="Qwen/Qwen3-0.6B", # Small test model
|
||||
input_modalities=["text"],
|
||||
output_modalities=["text"]
|
||||
)
|
||||
]
|
||||
|
||||
omni_llm = OmniLLM(stage_configs)
|
||||
assert len(omni_llm.engine_list) == 1
|
||||
assert omni_llm.engine_list[0] is not None
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_stage_processing_integration(self):
|
||||
"""Test integration between stages"""
|
||||
# Test with real vLLM components
|
||||
pass
|
||||
```
|
||||
|
||||
### 4.2 CLI Integration Tests
|
||||
|
||||
#### test_cli_integration.py
|
||||
```python
|
||||
import pytest
|
||||
import subprocess
|
||||
import sys
|
||||
from vllm_omni.entrypoints.cli.main import main
|
||||
|
||||
class TestCLIIntegration:
|
||||
def test_omni_flag_detection(self):
|
||||
"""Test --omni flag detection"""
|
||||
original_argv = sys.argv
|
||||
try:
|
||||
sys.argv = ["vllm", "serve", "test-model", "--omni"]
|
||||
# Test that omni command is triggered
|
||||
with patch('vllm_omni.entrypoints.omni.OmniServeCommand') as mock_serve:
|
||||
main()
|
||||
mock_serve.assert_called_once()
|
||||
finally:
|
||||
sys.argv = original_argv
|
||||
|
||||
def test_forward_to_vllm(self):
|
||||
"""Test forwarding to vLLM when --omni not present"""
|
||||
original_argv = sys.argv
|
||||
try:
|
||||
sys.argv = ["vllm", "serve", "test-model"]
|
||||
with patch('vllm.entrypoints.cli.main.main') as mock_vllm_main:
|
||||
main()
|
||||
mock_vllm_main.assert_called_once()
|
||||
finally:
|
||||
sys.argv = original_argv
|
||||
```
|
||||
|
||||
## 5. End-to-End Tests
|
||||
|
||||
### 5.1 Full Pipeline Tests
|
||||
|
||||
#### test_full_pipeline.py
|
||||
```python
|
||||
import pytest
|
||||
from vllm.config import ModelConfig
|
||||
from vllm_omni.core.omni_llm import OmniLLM
|
||||
from vllm_omni.config.stage_config import OmniStageConfig, DiTConfig
|
||||
|
||||
class TestFullPipeline:
|
||||
@pytest.mark.e2e
|
||||
@pytest.mark.slow
|
||||
def test_ar_to_dit_pipeline(self):
|
||||
"""Test complete AR to DiT pipeline"""
|
||||
stage_configs = [
|
||||
OmniStageConfig(
|
||||
stage_id=0,
|
||||
engine_type="AR",
|
||||
model_path="Qwen/Qwen3-0.6B",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["text"]
|
||||
),
|
||||
OmniStageConfig(
|
||||
stage_id=1,
|
||||
engine_type="DiT",
|
||||
model_path="stabilityai/stable-diffusion-2-1",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["image"],
|
||||
dit_config=DiTConfig(
|
||||
model_config=ModelConfig(model="stabilityai/stable-diffusion-2-1"),
|
||||
num_inference_steps=10,
|
||||
guidance_scale=7.5,
|
||||
use_diffusers=False,
|
||||
)
|
||||
)
|
||||
]
|
||||
|
||||
omni_llm = OmniLLM(stage_configs)
|
||||
|
||||
stage_args = [
|
||||
{"prompt": "A beautiful landscape"},
|
||||
{"prompt": "A beautiful landscape"}
|
||||
]
|
||||
|
||||
result = omni_llm.generate(stage_args)
|
||||
assert result is not None
|
||||
assert len(result) > 0
|
||||
```
|
||||
|
||||
### 5.2 Multimodal Generation Tests
|
||||
|
||||
#### test_multimodal_generation.py
|
||||
```python
|
||||
import pytest
|
||||
from vllm_omni.core.omni_llm import OmniLLM
|
||||
|
||||
class TestMultimodalGeneration:
|
||||
@pytest.mark.e2e
|
||||
def test_text_to_image_generation(self):
|
||||
"""Test text to image generation"""
|
||||
# Test implementation
|
||||
pass
|
||||
|
||||
@pytest.mark.e2e
|
||||
def test_image_to_text_generation(self):
|
||||
"""Test image to text generation"""
|
||||
# Test implementation
|
||||
pass
|
||||
|
||||
@pytest.mark.e2e
|
||||
def test_text_and_image_generation(self):
|
||||
"""Test combined text and image generation"""
|
||||
# Test implementation
|
||||
pass
|
||||
```
|
||||
|
||||
## 6. Performance Tests
|
||||
|
||||
### 6.1 Memory Usage Tests
|
||||
|
||||
#### test_memory_usage.py
|
||||
```python
|
||||
import pytest
|
||||
import psutil
|
||||
import torch
|
||||
from vllm_omni.core.omni_llm import OmniLLM
|
||||
|
||||
class TestMemoryUsage:
|
||||
@pytest.mark.benchmark
|
||||
def test_memory_usage_ar_stage(self):
|
||||
"""Test memory usage for AR stage"""
|
||||
process = psutil.Process()
|
||||
initial_memory = process.memory_info().rss
|
||||
|
||||
# Run AR stage
|
||||
# ... test implementation
|
||||
|
||||
final_memory = process.memory_info().rss
|
||||
memory_increase = final_memory - initial_memory
|
||||
|
||||
# Assert memory usage is reasonable
|
||||
assert memory_increase < 1024 * 1024 * 1024 # Less than 1GB
|
||||
|
||||
@pytest.mark.benchmark
|
||||
def test_memory_usage_dit_stage(self):
|
||||
"""Test memory usage for DiT stage"""
|
||||
# Test implementation
|
||||
pass
|
||||
|
||||
@pytest.mark.benchmark
|
||||
def test_cache_memory_management(self):
|
||||
"""Test DiT cache memory management"""
|
||||
# Test implementation
|
||||
pass
|
||||
```
|
||||
|
||||
### 6.2 Latency Tests
|
||||
|
||||
#### test_latency.py
|
||||
```python
|
||||
import pytest
|
||||
import time
|
||||
from vllm_omni.core.omni_llm import OmniLLM
|
||||
|
||||
class TestLatency:
|
||||
@pytest.mark.benchmark
|
||||
def test_ar_stage_latency(self):
|
||||
"""Test AR stage latency"""
|
||||
start_time = time.time()
|
||||
|
||||
# Run AR stage
|
||||
# ... test implementation
|
||||
|
||||
end_time = time.time()
|
||||
latency = end_time - start_time
|
||||
|
||||
# Assert latency is reasonable
|
||||
assert latency < 5.0 # Less than 5 seconds
|
||||
|
||||
@pytest.mark.benchmark
|
||||
def test_dit_stage_latency(self):
|
||||
"""Test DiT stage latency"""
|
||||
# Test implementation
|
||||
pass
|
||||
|
||||
@pytest.mark.benchmark
|
||||
def test_end_to_end_latency(self):
|
||||
"""Test end-to-end pipeline latency"""
|
||||
# Test implementation
|
||||
pass
|
||||
```
|
||||
|
||||
## 7. Test Configuration
|
||||
|
||||
### 7.1 pytest.ini
|
||||
```ini
|
||||
[tool:pytest]
|
||||
testpaths = tests
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
addopts =
|
||||
--strict-markers
|
||||
--strict-config
|
||||
--cov=vllm_omni
|
||||
--cov-report=term-missing
|
||||
--cov-report=html
|
||||
--cov-report=xml
|
||||
--tb=short
|
||||
markers =
|
||||
unit: Unit tests
|
||||
integration: Integration tests
|
||||
e2e: End-to-end tests
|
||||
benchmark: Performance benchmark tests
|
||||
slow: Slow running tests
|
||||
```
|
||||
|
||||
### 7.2 conftest.py
|
||||
```python
|
||||
import pytest
|
||||
import torch
|
||||
from unittest.mock import Mock
|
||||
from vllm_omni.config.stage_config import OmniStageConfig, DiTConfig
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def device():
|
||||
"""Get available device for testing"""
|
||||
return "cuda" if torch.cuda.is_available() else "cpu"
|
||||
|
||||
@pytest.fixture
|
||||
def sample_stage_configs():
|
||||
"""Sample stage configurations for testing"""
|
||||
return [
|
||||
OmniStageConfig(
|
||||
stage_id=0,
|
||||
engine_type="AR",
|
||||
model_path="test-ar-model",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["text"]
|
||||
),
|
||||
OmniStageConfig(
|
||||
stage_id=1,
|
||||
engine_type="DiT",
|
||||
model_path="test-dit-model",
|
||||
input_modalities=["text"],
|
||||
output_modalities=["image"],
|
||||
dit_config=DiTConfig(
|
||||
model_config=ModelConfig(model="test-dit-model"),
|
||||
num_inference_steps=10,
|
||||
guidance_scale=7.5,
|
||||
)
|
||||
)
|
||||
]
|
||||
|
||||
@pytest.fixture
|
||||
def mock_vllm_config():
|
||||
"""Mock vLLM configuration"""
|
||||
config = Mock()
|
||||
config.model = "test-model"
|
||||
config.tensor_parallel_size = 1
|
||||
config.pipeline_parallel_size = 1
|
||||
return config
|
||||
```
|
||||
|
||||
## 8. Test Execution
|
||||
|
||||
### 8.1 Running Tests
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run specific test categories
|
||||
pytest -m unit
|
||||
pytest -m integration
|
||||
pytest -m e2e
|
||||
pytest -m benchmark
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=vllm_omni --cov-report=html
|
||||
|
||||
# Run specific test file
|
||||
pytest tests/unit/test_omni_llm.py
|
||||
|
||||
# Run with verbose output
|
||||
pytest -v
|
||||
|
||||
# Run in parallel
|
||||
pytest -n auto
|
||||
```
|
||||
|
||||
### 8.2 Continuous Integration
|
||||
```yaml
|
||||
# .github/workflows/test.yml
|
||||
name: Tests
|
||||
on: [push, pull_request]
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: [3.8, 3.9, 3.10, 3.11]
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install -e ".[dev]"
|
||||
- name: Run tests
|
||||
run: |
|
||||
pytest --cov=vllm_omni --cov-report=xml
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v3
|
||||
```
|
||||
|
||||
## 9. Test Data Management
|
||||
|
||||
### 9.1 Fixtures Directory
|
||||
```
|
||||
tests/fixtures/
|
||||
├── sample_models/
|
||||
│ ├── ar_model/
|
||||
│ └── dit_model/
|
||||
├── test_images/
|
||||
│ ├── landscape.jpg
|
||||
│ └── portrait.png
|
||||
├── test_configs/
|
||||
│ ├── ar_stage_config.json
|
||||
│ └── dit_stage_config.json
|
||||
└── expected_outputs/
|
||||
├── ar_output.json
|
||||
└── dit_output.json
|
||||
```
|
||||
|
||||
### 9.2 Test Data Loading
|
||||
```python
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
def load_test_config(config_name: str) -> dict:
|
||||
"""Load test configuration from fixtures"""
|
||||
config_path = Path(__file__).parent / "fixtures" / "test_configs" / f"{config_name}.json"
|
||||
with open(config_path) as f:
|
||||
return json.load(f)
|
||||
|
||||
def load_test_image(image_name: str) -> str:
|
||||
"""Load test image path from fixtures"""
|
||||
return str(Path(__file__).parent / "fixtures" / "test_images" / image_name)
|
||||
```
|
||||
|
||||
## 10. Test Quality Metrics
|
||||
|
||||
### 10.1 Coverage Targets
|
||||
- **Overall Coverage**: > 90%
|
||||
- **Unit Tests**: > 95%
|
||||
- **Integration Tests**: > 80%
|
||||
- **Critical Paths**: 100%
|
||||
|
||||
### 10.2 Performance Targets
|
||||
- **Unit Test Execution**: < 30 seconds
|
||||
- **Integration Test Execution**: < 5 minutes
|
||||
- **E2E Test Execution**: < 15 minutes
|
||||
- **Memory Usage**: < 8GB for full test suite
|
||||
|
||||
### 10.3 Quality Gates
|
||||
- All unit tests must pass
|
||||
- Integration tests must pass
|
||||
- Coverage threshold must be met
|
||||
- No critical security vulnerabilities
|
||||
- Performance regression tests must pass
|
||||
@@ -1,24 +0,0 @@
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class DiTCacheTensor:
|
||||
"""
|
||||
A class for specifying how the workers should initialize the DiT cache.
|
||||
"""
|
||||
size: int # the size of the cache tensor in bytes
|
||||
|
||||
|
||||
@dataclass
|
||||
class DiTCacheConfig:
|
||||
"""
|
||||
The DiT cache configuration of a model.
|
||||
"""
|
||||
"""How should model runner initialize the KV cache tensors for each layer"""
|
||||
dit_cache_tensors: list[DiTCacheTensor]
|
||||
"""
|
||||
The DiT cache groups of the model.
|
||||
For models with only one type of DiT, there is only one group that
|
||||
contains all layers.
|
||||
"""
|
||||
kv_cache_groups: list[DiTCacheTensor]
|
||||
@@ -9,11 +9,12 @@ from vllm.v1.engine.llm_engine import LLMEngine
|
||||
from vllm.outputs import RequestOutput, LoRARequest
|
||||
|
||||
from ..config import OmniStageConfig
|
||||
from ..core.stage_manager import StageManager
|
||||
from .stage_manager import StageManager
|
||||
from ..engine.output_processor import MultimodalOutputProcessor
|
||||
from ..engine.diffusion_engine import DiffusersPipelineEngine
|
||||
|
||||
|
||||
|
||||
class OmniLLM(LLM):
|
||||
"""Extended LLM supporting multiple engines and stage-based processing."""
|
||||
|
||||
@@ -482,12 +483,12 @@ class AsyncOmniLLM(LLM):
|
||||
if stage_config and stage_config.engine_type == "DiT":
|
||||
dit = stage_config.dit_config
|
||||
if dit and getattr(dit, "use_diffusers", False):
|
||||
if not hasattr(self, "_dit_executors"):
|
||||
self._dit_executors = {}
|
||||
exec_inst = self._dit_executors.get(stage_config.stage_id)
|
||||
if not hasattr(self, "_dit_engines"):
|
||||
self._dit_engines = {}
|
||||
exec_inst = self._dit_engines.get(stage_config.stage_id)
|
||||
if exec_inst is None:
|
||||
from vllm_omni.executor.diffusers_executor import (
|
||||
DiffusersPipelineExecutor,
|
||||
from vllm_omni.engine.diffusion_engine import (
|
||||
DiffusersPipelineEngine,
|
||||
)
|
||||
|
||||
pipeline_name = getattr(dit, "diffusers_pipeline", None)
|
||||
@@ -506,13 +507,13 @@ class AsyncOmniLLM(LLM):
|
||||
else:
|
||||
dtype = getattr(model_cfg, "dtype", None)
|
||||
|
||||
exec_inst = DiffusersPipelineExecutor(
|
||||
exec_inst = DiffusersPipelineEngine(
|
||||
model_path=stage_config.model_path,
|
||||
pipeline_name=pipeline_name,
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
)
|
||||
self._dit_executors[stage_config.stage_id] = exec_inst
|
||||
self._dit_engines[stage_config.stage_id] = exec_inst
|
||||
|
||||
return exec_inst.generate(
|
||||
prompt=processed_input.get("prompt", ""),
|
||||
|
||||
File renamed without changes.
Whitespace-only changes.
@@ -1,64 +0,0 @@
|
||||
"""
|
||||
Diffusers-backed executor for DiT stages.
|
||||
|
||||
Provides a minimal wrapper around HuggingFace diffusers pipelines to support
|
||||
text-to-image generation as the DiT stage backend.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Optional
|
||||
|
||||
from vllm_omni.worker.gpu_diffusion_worker import (
|
||||
DiffusionGPUWorker,
|
||||
DiffusionRunnerOutput,
|
||||
)
|
||||
|
||||
|
||||
class DiffusersPipelineExecutor:
|
||||
"""Thin executor that delegates to a diffusion worker."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_path: str,
|
||||
device: Optional[str] = None,
|
||||
dtype: Optional[str] = None,
|
||||
pipeline_name: Optional[str] = None,
|
||||
) -> None:
|
||||
try:
|
||||
import torch # noqa: F401
|
||||
import diffusers # noqa: F401
|
||||
except Exception as e: # pragma: no cover
|
||||
raise ImportError(
|
||||
"diffusers and torch are required for DiT diffusers backend."
|
||||
) from e
|
||||
|
||||
self.worker = DiffusionGPUWorker(
|
||||
model_path,
|
||||
pipeline_name=pipeline_name,
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
)
|
||||
|
||||
def generate(
|
||||
self,
|
||||
prompt: str,
|
||||
*,
|
||||
height: int = 512,
|
||||
width: int = 512,
|
||||
num_inference_steps: int = 30,
|
||||
guidance_scale: float = 5.0,
|
||||
negative_prompt: Optional[str] = None,
|
||||
seed: Optional[int] = None,
|
||||
image: Optional[Any] = None,
|
||||
) -> DiffusionRunnerOutput:
|
||||
return self.worker.generate(
|
||||
prompt=prompt,
|
||||
height=height,
|
||||
width=width,
|
||||
num_inference_steps=num_inference_steps,
|
||||
guidance_scale=guidance_scale,
|
||||
negative_prompt=negative_prompt,
|
||||
seed=seed,
|
||||
image=image,
|
||||
)
|
||||
Reference in new issue
Block a user