diff --git a/docs/PRD.md b/docs/PRD.md deleted file mode 100644 index 7b7850419..000000000 --- a/docs/PRD.md +++ /dev/null @@ -1,179 +0,0 @@ -# vLLM-omni Product Requirements Document (PRD) - -## 1. Product Overview - -### 1.1 Product Name -vLLM-omni: Multi-modality models inference and serving with non-autoregressive structures - -### 1.2 Product Vision -Extend vLLM beyond traditional text-based, autoregressive generation to support multi-modality models with non-autoregressive structures and non-textual outputs while maintaining vLLM's proven architecture and performance. - -### 1.3 Target Users -- AI researchers working with multimodal models -- ML engineers building production inference systems -- Developers integrating DiT (Diffusion Transformer) models -- Organizations requiring efficient multimodal model serving - -## 2. Core Requirements - -### 2.1 Functional Requirements - -#### 2.1.1 Multi-Stage Processing -- **REQ-001**: Support stage-based model processing where each stage can use different engine types (AR/DiT) -- **REQ-002**: Enable sequential processing through multiple stages with data flow between stages -- **REQ-003**: Support both autoregressive (AR) and diffusion (DiT) model stages - -#### 2.1.2 vLLM Compatibility -- **REQ-004**: Maintain full compatibility with vLLM V1 architecture (AsyncLLM and EngineCore patterns) -- **REQ-005**: Support existing vLLM CLI commands with `--omni` flag extension -- **REQ-006**: Reuse vLLM's multiprocess worker architecture for scalability - -#### 2.1.3 Multimodal Support -- **REQ-007**: Support text, image, and latent space inputs/outputs -- **REQ-008**: Enable image-to-image, text-to-image, and text-to-text generation -- **REQ-009**: Support hidden state passing between AR and DiT stages - -#### 2.1.4 CLI and API -- **REQ-010**: Provide CLI command: `vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8000` -- **REQ-011**: Support both online (AsyncOmniLLM) and offline (OmniLLM) inference modes -- **REQ-012**: Maintain vLLM's existing API compatibility - -### 2.2 Non-Functional Requirements - -#### 2.2.1 Performance -- **REQ-013**: Maintain vLLM's inference performance for AR stages -- **REQ-014**: Optimize DiT stage processing with caching mechanisms -- **REQ-015**: Support distributed inference across multiple GPUs - -#### 2.2.2 Scalability -- **REQ-016**: Support horizontal scaling through vLLM's worker process pattern -- **REQ-017**: Enable efficient memory management for large multimodal models -- **REQ-018**: Support batch processing for multiple requests - -#### 2.2.3 Extensibility -- **REQ-019**: Easy integration of new modalities and model architectures -- **REQ-020**: Pluggable scheduler and executor components -- **REQ-021**: Support for future non-autoregressive model types - -## 3. Technical Architecture - -### 3.1 Core Components - -#### 3.1.1 Entry Points -- **OmniServeCommand**: CLI wrapper that intercepts vLLM commands with `--omni` flag -- **OmniLLM**: Offline inference class supporting multi-stage processing -- **AsyncOmniLLM**: Online inference class with asynchronous processing - -#### 3.1.2 Stage Management -- **OmniStageConfig**: Configuration for each processing stage -- **Stage Engine List**: Multiple AsyncLLM instances for each stage -- **Stage I/O Management**: Data flow between stages - -#### 3.1.3 Engine Components -- **EngineCore**: Reused from vLLM (no changes needed) -- **OmniDiffusionScheduler**: New scheduler for DiT models -- **DiTCacheManager**: Caching system for DiT optimization -- **MultiprocExecutor**: Reused from vLLM for DiT without diffusers -- **DiffusersPipelineExecutor**: New executor for diffusers integration - -#### 3.1.4 Model Runners -- **OmniDiffusionModelRunner**: Handles DiT model execution -- **OmniARModelRunner**: Handles AR model execution with hidden state output -- **ModelRunnerOutput**: Extended to support multimodal outputs via pooler_output - -#### 3.1.5 Output Processing -- **MultimodalOutputProcessor**: Handles final multimodal output processing -- **RequestState**: Extended to support pooling outputs -- **Output Handlers**: Type-specific output processing - -### 3.2 Data Flow -``` -API Server → OmniLLM/AsyncOmniLLM → LLMEngine/AsyncLLM → Engine Core -→ Scheduler (AR/DiT) → Executor (AR/DiT) → Worker (AR/DiT) -→ ModelRunner (AR/DiT) → RequestState → OutputProcessor → Final Output -``` - -## 4. Implementation Phases - -### Phase 1: Foundation (Weeks 1-2) -- Package structure and dependencies -- Basic OmniLLM and AsyncOmniLLM classes -- Stage configuration system -- CLI integration - -### Phase 2: Core Components (Weeks 3-4) -- DiT scheduler implementation -- Model runners for AR and DiT -- Basic output processing - -### Phase 3: Advanced Features (Weeks 5-6) -- Caching system implementation -- Multimodal output processing -- Request state management - -### Phase 4: Integration & Testing (Weeks 7-8) -- End-to-end integration -- Comprehensive testing -- Performance optimization -- Documentation - -## 5. Success Criteria - -### 5.1 Functional Success -- [ ] Successfully run `vllm serve model --omni` command -- [ ] Process multi-stage AR→DiT pipelines -- [ ] Generate multimodal outputs (text + image) -- [ ] Maintain vLLM API compatibility - -### 5.2 Performance Success -- [ ] AR stage performance within 5% of native vLLM -- [ ] DiT stage processing with reasonable latency -- [ ] Memory usage comparable to vLLM for equivalent models - -### 5.3 Quality Success -- [ ] 90%+ test coverage -- [ ] All integration tests passing -- [ ] Documentation complete and accurate - -## 6. Risk Assessment - -### 6.1 Technical Risks -- **High**: vLLM API changes breaking compatibility -- **Medium**: DiT model integration complexity -- **Low**: Performance overhead from multi-stage processing - -### 6.2 Mitigation Strategies -- Regular vLLM compatibility testing -- Incremental DiT integration with fallback options -- Performance benchmarking at each stage - -## 7. Dependencies - -### 7.1 External Dependencies -- vLLM >= 0.10.2 -- PyTorch >= 2.7 -- Transformers >= 4.30.0 -- FastAPI, Uvicorn for API serving -- Ray for distributed computing - -### 7.2 Optional Dependencies -- xDiT for DiT acceleration -- Cache-DiT for advanced caching -- Diffusers for pipeline-based DiT models - -## 8. Future Roadmap - -### 8.1 Short-term (3 months) -- Additional DiT model support -- Performance optimizations -- Enhanced caching strategies - -### 8.2 Medium-term (6 months) -- Support for video generation models -- Advanced scheduling strategies -- Multi-GPU DiT optimization - -### 8.3 Long-term (12 months) -- Custom model architecture support -- Advanced multimodal fusion -- Production deployment tools diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index d52c6c099..000000000 --- a/docs/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# vLLM-omni Documentation - -Welcome to the vLLM-omni documentation! This documentation provides comprehensive information about using and developing with vLLM-omni. - -# TODO (add examples) \ No newline at end of file diff --git a/docs/architecture/detailed arch design.md b/docs/architecture/detailed arch design.md deleted file mode 100644 index 20c759ba1..000000000 --- a/docs/architecture/detailed arch design.md +++ /dev/null @@ -1,157 +0,0 @@ -# vLLM-omni Software Design Document - -## Overview - -vLLM-omni extends vLLM with multi-stage, multimodal capabilities. It supports both autoregressive (AR) stages and diffusion (DiT) stages, enabling pipelines such as text→image generation. - -## Architecture Principles - -1. **vLLM V1 Compatibility** – Reuse LLMEngine/AsyncLLM and executor abstractions. -2. **Stage-based Processing** – Each stage encapsulates its own engine and configuration. -3. **Executor/Worker Separation** – Executors remain thin; model-specific logic lives in workers and runners. -4. **Extensibility** – New modalities can be added by implementing dedicated stage configs, workers, and output processors. - -## Key Data Flow - -``` -Request → OmniLLM/AsyncOmniLLM → StageManager → LLMEngine/Executor → Worker → ModelRunner → OutputProcessor → Response -``` - -For diffusion stages, the executor hands off to `DiffusionGPUWorker`, which wraps `DiffusionModelRunner` (responsible for diffusers pipeline loading and inference). - -## Core Components (Ordered by Data Flow) - -### 1. Installation -```bash -pip install vllm -pip install vllm-omni -``` - -### 2. Online inference launch (Entry Point) -```bash -vllm serve Qwen/Qwen3-0.6B --omni --port 8000 -``` - -### 3. Omni Serve Command (CLI) -```python -# vllm_omni/entrypoints/cli/main.py -if "--omni" in sys.argv: - omni_args = [arg for arg in sys.argv[1:] if arg not in {"--omni", "serve"}] - OmniServeCommand().run(omni_args) -else: - from vllm.entrypoints.cli.main import main as vllm_main - vllm_main() -``` - -### 4. Offline Inference (OmniLLM) -```python -# vllm_omni/entrypoints/omni_llm.py -class OmniLLM(LLM): - def __init__(..., stage_configs): - super().__init__(model=stage_configs[0].model_path if stage_configs else "test-model") - self.stage_manager = StageManager(stage_configs) - self.output_processor = MultimodalOutputProcessor() - self.stage_manager.initialize_engines() - - def generate(...): - current_output = None - for idx, (config, args) in enumerate(zip(self.stage_configs, stage_args_list)): - engine = self.stage_manager.get_engine(idx) - processed = self._process_stage_inputs(config, args, current_output) - stage_output = self._execute_stage(engine, processed, stage_config=config) - current_output = stage_output - config.stage_output = stage_output - return self.output_processor.process_output(current_output) -``` - -### 5. Stage Manager -```python -# vllm_omni/core/stage_manager.py -class StageManager: - def initialize_engines(self): - if self._initialized: - return - for config in self.stage_configs: - if config.engine_type == "AR": - engine = self._create_ar_engine(config) - elif config.engine_type == "DiT": - engine = self._create_dit_engine(config) - self.engine_list.append(engine) - self._initialized = True -``` - -### 6. Diffusion Execution (DiT Stage) -```python -# vllm_omni/worker/diffusion_model_runner.py -class DiffusionModelRunner: - def __init__(self, model_path, pipeline_name=None, device=None, dtype=None): - self.device, self.torch_dtype = self._resolve_device_and_dtype(device, dtype) - self._pipeline = self._load_pipeline(model_path, pipeline_name) - - def generate(...): - output = self._pipeline(...) - return DiffusionRunnerOutput(prompt=prompt, images=getattr(output, "images", []) or []) -``` - -```python -# vllm_omni/worker/gpu_diffusion_worker.py -class DiffusionGPUWorker: - def __init__(self, model_path, ...): - self.model_runner = DiffusionModelRunner(model_path, ...) - - def generate(...): - return self.model_runner.generate(...) -``` - -```python -# vllm_omni/executor/diffusers_executor.py -class DiffusersPipelineExecutor: - def __init__(self, model_path, ...): - self.worker = DiffusionGPUWorker(model_path, ...) - - def generate(...): - return self.worker.generate(...) -``` - -This mirrors the standard executor→worker→runner layering in vLLM. - -### 7. Stage Configuration -```python -# vllm_omni/config/stage_config.py -@dataclass -class OmniStageConfig: - stage_id: int - engine_type: Literal["AR", "DiT"] - model_path: str - input_modalities: List[str] - output_modalities: List[str] - vllm_config: Optional[VllmConfig] = None - executor_class: Optional[type[Executor]] = None - dit_config: Optional[DiTConfig] = None -``` - -- AR stages default to `UniProcExecutor`. -- DiT stages set `executor_class` to `DiffusersPipelineExecutor` when `use_diffusers=True`. - -### 8. Output Processing -```python -# vllm_omni/engine/output_processor.py -class MultimodalOutputProcessor(OutputProcessor): - def process_output(self, engine_core_output): - output_type = self._detect_output_type(engine_core_output) - handler = self.output_handlers.get(output_type, self._process_pooling_output) - return handler(engine_core_output) -``` - -### 9. CLI Package Configuration -```toml -# pyproject.toml -[project.scripts] -vllm = "vllm_omni.entrypoints.cli.main:main" -vllm-omni = "vllm_omni.entrypoints.cli.main:main" -``` - -## Future Work -- Scheduler adaptations for diffusion batching. -- Advanced output processors for audio/video. -- Multi-worker execution (ray / multiprocess) for heavy DiT stages. diff --git a/docs/architecture/implementation_architecture.md b/docs/architecture/implementation_architecture.md index 9a1c4a82d..a83d96d1e 100644 --- a/docs/architecture/implementation_architecture.md +++ b/docs/architecture/implementation_architecture.md @@ -300,7 +300,7 @@ omni = "vllm_omni.plugin:OmniPlugin" ### 7.1 Package Installation ```bash -pip install vllm>=0.2.0 +pip install vllm>=0.10.2 pip install vllm-omni ``` diff --git a/docs/testing/test_design.md b/docs/testing/test_design.md deleted file mode 100644 index fdc69db79..000000000 --- a/docs/testing/test_design.md +++ /dev/null @@ -1,758 +0,0 @@ -# vLLM-omni Test Design Document - -## 1. Testing Strategy Overview - -### 1.1 Testing Pyramid -``` - E2E Tests (10%) - / \ - Integration Tests (30%) - / \ - Unit Tests (60%) -``` - -### 1.2 Test Categories -- **Unit Tests**: Individual component testing with mocked dependencies -- **Integration Tests**: Component interaction testing with real vLLM integration -- **End-to-End Tests**: Full pipeline testing with real models -- **Performance Tests**: Benchmarking and profiling -- **Compatibility Tests**: vLLM version compatibility validation - -## 2. Test Structure - -``` -tests/ -├── __init__.py -├── conftest.py # Shared fixtures and configuration -├── unit/ # Unit tests (60%) -│ ├── __init__.py -│ ├── test_config/ # Configuration testing -│ │ ├── test_stage_config.py -│ │ ├── test_dit_config.py -│ │ └── test_cache_config.py -│ ├── test_core/ # Core component testing -│ │ ├── test_omni_llm.py -│ │ ├── test_async_omni_llm.py -│ │ ├── test_stage_manager.py -│ │ └── test_dit_cache_manager.py -│ ├── test_scheduler/ # Scheduler testing -│ │ ├── test_diffusion_scheduler.py -│ │ └── test_scheduler_interface.py -│ ├── test_executor/ # Executor testing -│ │ ├── test_base_executor.py -│ │ └── test_diffusers_executor.py -│ ├── test_model_executor/ # Model runner testing -│ │ ├── test_ar_model_runner.py -│ │ └── test_dit_model_runner.py -│ ├── test_engine/ # Engine testing -│ │ ├── test_output_processor.py -│ │ └── test_multimodal_processor.py -│ ├── test_worker/ # Worker testing -│ │ └── test_omni_worker.py -│ └── test_utils/ # Utility testing -│ ├── test_multimodal.py -│ └── test_vae.py -├── integration/ # Integration tests (30%) -│ ├── __init__.py -│ ├── test_vllm_integration.py -│ ├── test_stage_processing.py -│ ├── test_cli_integration.py -│ └── test_api_compatibility.py -├── e2e/ # End-to-end tests (10%) -│ ├── __init__.py -│ ├── test_full_pipeline.py -│ ├── test_multimodal_generation.py -│ └── test_performance.py -├── benchmarks/ # Performance benchmarks -│ ├── __init__.py -│ ├── test_memory_usage.py -│ ├── test_latency.py -│ └── test_throughput.py -└── fixtures/ # Test data and fixtures - ├── __init__.py - ├── sample_models/ - ├── test_images/ - └── test_configs/ -``` - -## 3. Unit Test Specifications - -### 3.1 Configuration Tests - -#### test_stage_config.py -```python -import pytest -from vllm_omni.config.stage_config import OmniStageConfig, DiTConfig, DiTCacheConfig - -class TestOmniStageConfig: - def test_ar_stage_config_creation(self): - """Test AR stage configuration creation""" - config = OmniStageConfig( - stage_id=0, - engine_type="AR", - model_path="test-model", - input_modalities=["text"], - output_modalities=["text"] - ) - assert config.engine_type == "AR" - assert config.stage_id == 0 - - def test_dit_stage_config_creation(self): - """Test DiT stage configuration creation""" - dit_config = DiTConfig( - num_inference_steps=50, - guidance_scale=7.5, - use_diffusers=False, - ) - config = OmniStageConfig( - stage_id=1, - engine_type="DiT", - model_path="test-dit-model", - input_modalities=["text"], - output_modalities=["image"], - dit_config=dit_config - ) - assert config.engine_type == "DiT" - assert config.dit_config is not None - - def test_invalid_engine_type(self): - """Test validation of engine type""" - with pytest.raises(ValueError): - OmniStageConfig( - stage_id=0, - engine_type="INVALID", - model_path="test-model", - input_modalities=["text"], - output_modalities=["text"] - ) -``` - -#### test_dit_config.py -```python -import pytest -from vllm.config import ModelConfig -from vllm_omni.config import DiTConfig - -class TestDiTConfig: - def test_dit_config_defaults(self): - """Test DiT configuration with defaults""" - config = DiTConfig(num_inference_steps=50) - assert config.model_config is None - assert config.use_diffusers is False - assert config.diffusers_pipeline is None - - def test_diffusers_config(self): - """Test DiT configuration with diffusers""" - config = DiTConfig( - model_config=ModelConfig(model="stabilityai/stable-diffusion-2-1"), - num_inference_steps=50, - use_diffusers=True, - diffusers_pipeline="stable-diffusion", - ) - assert config.use_diffusers is True - assert config.diffusers_pipeline == "stable-diffusion" -``` - -### 3.2 Core Component Tests - -#### test_omni_llm.py -```python -import pytest -from unittest.mock import Mock, patch -from vllm_omni.core.omni_llm import OmniLLM -from vllm_omni.config.stage_config import OmniStageConfig - -class TestOmniLLM: - @pytest.fixture - def mock_stage_configs(self): - """Create mock stage configurations""" - return [ - OmniStageConfig( - stage_id=0, - engine_type="AR", - model_path="test-ar-model", - input_modalities=["text"], - output_modalities=["text"] - ), - OmniStageConfig( - stage_id=1, - engine_type="DiT", - model_path="test-dit-model", - input_modalities=["text"], - output_modalities=["image"] - ) - ] - - @patch('vllm_omni.core.omni_llm.LLMEngine') - def test_omni_llm_initialization(self, mock_llm_engine, mock_stage_configs): - """Test OmniLLM initialization""" - omni_llm = OmniLLM(mock_stage_configs) - assert len(omni_llm.engine_list) == 2 - assert omni_llm.stage_configs == mock_stage_configs - - @patch('vllm_omni.core.omni_llm.LLMEngine') - def test_stage_engine_creation(self, mock_llm_engine, mock_stage_configs): - """Test stage engine creation""" - omni_llm = OmniLLM(mock_stage_configs) - # Verify engines are created for each stage - assert mock_llm_engine.from_vllm_config.call_count == 2 - - def test_process_stage_inputs_ar(self, mock_stage_configs): - """Test AR stage input processing""" - omni_llm = OmniLLM(mock_stage_configs) - ar_config = mock_stage_configs[0] - stage_args = {"prompt": "test prompt"} - - result = omni_llm._process_stage_inputs(ar_config, stage_args, None) - assert "prompt" in result - - def test_process_stage_inputs_dit(self, mock_stage_configs): - """Test DiT stage input processing""" - omni_llm = OmniLLM(mock_stage_configs) - dit_config = mock_stage_configs[1] - stage_args = {"image": "test_image.jpg"} - - with patch.object(omni_llm, 'vae') as mock_vae: - mock_vae.encode.return_value = "encoded_image" - result = omni_llm._process_stage_inputs(dit_config, stage_args, None) - assert "encoded_image" in result -``` - -#### test_async_omni_llm.py -```python -import pytest -import asyncio -from unittest.mock import Mock, patch, AsyncMock -from vllm_omni.core.omni_llm import AsyncOmniLLM - -class TestAsyncOmniLLM: - @pytest.fixture - def mock_stage_configs(self): - """Create mock stage configurations""" - return [ - OmniStageConfig( - stage_id=0, - engine_type="AR", - model_path="test-ar-model", - input_modalities=["text"], - output_modalities=["text"] - ) - ] - - @pytest.mark.asyncio - async def test_async_generation(self, mock_stage_configs): - """Test async generation""" - with patch('vllm_omni.core.omni_llm.AsyncLLM') as mock_async_llm: - async_omni_llm = AsyncOmniLLM(mock_stage_configs) - - # Mock the generate_async method - mock_async_llm.return_value.generate_async = AsyncMock(return_value=[]) - - result = await async_omni_llm.generate_async([{"prompt": "test"}]) - assert result == [] -``` - -### 3.3 Scheduler Tests - -#### test_diffusion_scheduler.py -```python -import pytest -from unittest.mock import Mock, patch -from vllm_omni.core.sched.diffusion_scheduler import OmniDiffusionScheduler - -class TestOmniDiffusionScheduler: - @pytest.fixture - def mock_configs(self): - """Create mock configurations""" - return { - 'vllm_config': Mock(), - 'kv_cache_config': Mock(), - 'dit_cache_config': Mock(), - 'structured_output_manager': Mock(), - 'mm_registry': Mock(), - 'include_finished_set': False, - 'log_stats': False - } - - def test_scheduler_initialization(self, mock_configs): - """Test scheduler initialization""" - scheduler = OmniDiffusionScheduler(**mock_configs) - assert scheduler.dit_cache_manager is not None - - def test_schedule_method(self, mock_configs): - """Test scheduling method""" - scheduler = OmniDiffusionScheduler(**mock_configs) - - with patch.object(scheduler, 'dit_cache_manager') as mock_cache: - mock_cache.allocate_cache.return_value = Mock() - result = scheduler.schedule() - # Verify scheduling logic - assert result is not None -``` - -### 3.4 Model Runner Tests - -#### test_ar_model_runner.py -```python -import pytest -import torch -from unittest.mock import Mock, patch -from vllm_omni.model_executor.ar_model_runner import OmniARModelRunner - -class TestOmniARModelRunner: - @pytest.fixture - def mock_runner(self): - """Create mock model runner""" - with patch('vllm_omni.model_executor.ar_model_runner.GPUModelRunner.__init__'): - runner = OmniARModelRunner(Mock(), Mock(), Mock()) - return runner - - def test_execute_model_ar(self, mock_runner): - """Test AR model execution""" - mock_scheduler_output = Mock() - mock_scheduler_output.req_ids = ["req1"] - mock_scheduler_output.req_id_to_index = {"req1": 0} - - with patch.object(mock_runner, 'model') as mock_model: - mock_model.forward.return_value = (torch.randn(1, 10, 768), None) - - result = mock_runner.execute_model(mock_scheduler_output) - - assert result.req_ids == ["req1"] - assert len(result.pooler_output) > 0 # Hidden states -``` - -#### test_dit_model_runner.py -```python -import pytest -import torch -from unittest.mock import Mock, patch -from vllm_omni.model_executor.dit_model_runner import OmniDiffusionModelRunner - -class TestOmniDiffusionModelRunner: - @pytest.fixture - def mock_runner(self): - """Create mock DiT model runner""" - with patch('vllm_omni.model_executor.dit_model_runner.GPUModelRunner.__init__'): - runner = OmniDiffusionModelRunner(Mock(), Mock(), Mock()) - return runner - - def test_execute_model_dit(self, mock_runner): - """Test DiT model execution""" - mock_scheduler_output = Mock() - mock_scheduler_output.req_ids = ["req1"] - mock_scheduler_output.req_id_to_index = {"req1": 0} - - with patch.object(mock_runner, 'model') as mock_model: - mock_model.forward.return_value = torch.randn(1, 3, 512, 512) - - result = mock_runner.execute_model(mock_scheduler_output) - - assert result.req_ids == ["req1"] - assert len(result.pooler_output) > 0 # DiT output tensors -``` - -## 4. Integration Tests - -### 4.1 vLLM Integration Tests - -#### test_vllm_integration.py -```python -import pytest -from vllm_omni.core.omni_llm import OmniLLM -from vllm_omni.config.stage_config import OmniStageConfig - -class TestVLLMIntegration: - @pytest.mark.integration - def test_vllm_engine_creation(self): - """Test creation of vLLM engines""" - stage_configs = [ - OmniStageConfig( - stage_id=0, - engine_type="AR", - model_path="Qwen/Qwen3-0.6B", # Small test model - input_modalities=["text"], - output_modalities=["text"] - ) - ] - - omni_llm = OmniLLM(stage_configs) - assert len(omni_llm.engine_list) == 1 - assert omni_llm.engine_list[0] is not None - - @pytest.mark.integration - def test_stage_processing_integration(self): - """Test integration between stages""" - # Test with real vLLM components - pass -``` - -### 4.2 CLI Integration Tests - -#### test_cli_integration.py -```python -import pytest -import subprocess -import sys -from vllm_omni.entrypoints.cli.main import main - -class TestCLIIntegration: - def test_omni_flag_detection(self): - """Test --omni flag detection""" - original_argv = sys.argv - try: - sys.argv = ["vllm", "serve", "test-model", "--omni"] - # Test that omni command is triggered - with patch('vllm_omni.entrypoints.omni.OmniServeCommand') as mock_serve: - main() - mock_serve.assert_called_once() - finally: - sys.argv = original_argv - - def test_forward_to_vllm(self): - """Test forwarding to vLLM when --omni not present""" - original_argv = sys.argv - try: - sys.argv = ["vllm", "serve", "test-model"] - with patch('vllm.entrypoints.cli.main.main') as mock_vllm_main: - main() - mock_vllm_main.assert_called_once() - finally: - sys.argv = original_argv -``` - -## 5. End-to-End Tests - -### 5.1 Full Pipeline Tests - -#### test_full_pipeline.py -```python -import pytest -from vllm.config import ModelConfig -from vllm_omni.core.omni_llm import OmniLLM -from vllm_omni.config.stage_config import OmniStageConfig, DiTConfig - -class TestFullPipeline: - @pytest.mark.e2e - @pytest.mark.slow - def test_ar_to_dit_pipeline(self): - """Test complete AR to DiT pipeline""" - stage_configs = [ - OmniStageConfig( - stage_id=0, - engine_type="AR", - model_path="Qwen/Qwen3-0.6B", - input_modalities=["text"], - output_modalities=["text"] - ), - OmniStageConfig( - stage_id=1, - engine_type="DiT", - model_path="stabilityai/stable-diffusion-2-1", - input_modalities=["text"], - output_modalities=["image"], - dit_config=DiTConfig( - model_config=ModelConfig(model="stabilityai/stable-diffusion-2-1"), - num_inference_steps=10, - guidance_scale=7.5, - use_diffusers=False, - ) - ) - ] - - omni_llm = OmniLLM(stage_configs) - - stage_args = [ - {"prompt": "A beautiful landscape"}, - {"prompt": "A beautiful landscape"} - ] - - result = omni_llm.generate(stage_args) - assert result is not None - assert len(result) > 0 -``` - -### 5.2 Multimodal Generation Tests - -#### test_multimodal_generation.py -```python -import pytest -from vllm_omni.core.omni_llm import OmniLLM - -class TestMultimodalGeneration: - @pytest.mark.e2e - def test_text_to_image_generation(self): - """Test text to image generation""" - # Test implementation - pass - - @pytest.mark.e2e - def test_image_to_text_generation(self): - """Test image to text generation""" - # Test implementation - pass - - @pytest.mark.e2e - def test_text_and_image_generation(self): - """Test combined text and image generation""" - # Test implementation - pass -``` - -## 6. Performance Tests - -### 6.1 Memory Usage Tests - -#### test_memory_usage.py -```python -import pytest -import psutil -import torch -from vllm_omni.core.omni_llm import OmniLLM - -class TestMemoryUsage: - @pytest.mark.benchmark - def test_memory_usage_ar_stage(self): - """Test memory usage for AR stage""" - process = psutil.Process() - initial_memory = process.memory_info().rss - - # Run AR stage - # ... test implementation - - final_memory = process.memory_info().rss - memory_increase = final_memory - initial_memory - - # Assert memory usage is reasonable - assert memory_increase < 1024 * 1024 * 1024 # Less than 1GB - - @pytest.mark.benchmark - def test_memory_usage_dit_stage(self): - """Test memory usage for DiT stage""" - # Test implementation - pass - - @pytest.mark.benchmark - def test_cache_memory_management(self): - """Test DiT cache memory management""" - # Test implementation - pass -``` - -### 6.2 Latency Tests - -#### test_latency.py -```python -import pytest -import time -from vllm_omni.core.omni_llm import OmniLLM - -class TestLatency: - @pytest.mark.benchmark - def test_ar_stage_latency(self): - """Test AR stage latency""" - start_time = time.time() - - # Run AR stage - # ... test implementation - - end_time = time.time() - latency = end_time - start_time - - # Assert latency is reasonable - assert latency < 5.0 # Less than 5 seconds - - @pytest.mark.benchmark - def test_dit_stage_latency(self): - """Test DiT stage latency""" - # Test implementation - pass - - @pytest.mark.benchmark - def test_end_to_end_latency(self): - """Test end-to-end pipeline latency""" - # Test implementation - pass -``` - -## 7. Test Configuration - -### 7.1 pytest.ini -```ini -[tool:pytest] -testpaths = tests -python_files = test_*.py -python_classes = Test* -python_functions = test_* -addopts = - --strict-markers - --strict-config - --cov=vllm_omni - --cov-report=term-missing - --cov-report=html - --cov-report=xml - --tb=short -markers = - unit: Unit tests - integration: Integration tests - e2e: End-to-end tests - benchmark: Performance benchmark tests - slow: Slow running tests -``` - -### 7.2 conftest.py -```python -import pytest -import torch -from unittest.mock import Mock -from vllm_omni.config.stage_config import OmniStageConfig, DiTConfig - -@pytest.fixture(scope="session") -def device(): - """Get available device for testing""" - return "cuda" if torch.cuda.is_available() else "cpu" - -@pytest.fixture -def sample_stage_configs(): - """Sample stage configurations for testing""" - return [ - OmniStageConfig( - stage_id=0, - engine_type="AR", - model_path="test-ar-model", - input_modalities=["text"], - output_modalities=["text"] - ), - OmniStageConfig( - stage_id=1, - engine_type="DiT", - model_path="test-dit-model", - input_modalities=["text"], - output_modalities=["image"], - dit_config=DiTConfig( - model_config=ModelConfig(model="test-dit-model"), - num_inference_steps=10, - guidance_scale=7.5, - ) - ) - ] - -@pytest.fixture -def mock_vllm_config(): - """Mock vLLM configuration""" - config = Mock() - config.model = "test-model" - config.tensor_parallel_size = 1 - config.pipeline_parallel_size = 1 - return config -``` - -## 8. Test Execution - -### 8.1 Running Tests -```bash -# Run all tests -pytest - -# Run specific test categories -pytest -m unit -pytest -m integration -pytest -m e2e -pytest -m benchmark - -# Run with coverage -pytest --cov=vllm_omni --cov-report=html - -# Run specific test file -pytest tests/unit/test_omni_llm.py - -# Run with verbose output -pytest -v - -# Run in parallel -pytest -n auto -``` - -### 8.2 Continuous Integration -```yaml -# .github/workflows/test.yml -name: Tests -on: [push, pull_request] -jobs: - test: - runs-on: ubuntu-latest - strategy: - matrix: - python-version: [3.8, 3.9, 3.10, 3.11] - steps: - - uses: actions/checkout@v3 - - name: Set up Python - uses: actions/setup-python@v4 - with: - python-version: ${{ matrix.python-version }} - - name: Install dependencies - run: | - pip install -e ".[dev]" - - name: Run tests - run: | - pytest --cov=vllm_omni --cov-report=xml - - name: Upload coverage - uses: codecov/codecov-action@v3 -``` - -## 9. Test Data Management - -### 9.1 Fixtures Directory -``` -tests/fixtures/ -├── sample_models/ -│ ├── ar_model/ -│ └── dit_model/ -├── test_images/ -│ ├── landscape.jpg -│ └── portrait.png -├── test_configs/ -│ ├── ar_stage_config.json -│ └── dit_stage_config.json -└── expected_outputs/ - ├── ar_output.json - └── dit_output.json -``` - -### 9.2 Test Data Loading -```python -import json -import os -from pathlib import Path - -def load_test_config(config_name: str) -> dict: - """Load test configuration from fixtures""" - config_path = Path(__file__).parent / "fixtures" / "test_configs" / f"{config_name}.json" - with open(config_path) as f: - return json.load(f) - -def load_test_image(image_name: str) -> str: - """Load test image path from fixtures""" - return str(Path(__file__).parent / "fixtures" / "test_images" / image_name) -``` - -## 10. Test Quality Metrics - -### 10.1 Coverage Targets -- **Overall Coverage**: > 90% -- **Unit Tests**: > 95% -- **Integration Tests**: > 80% -- **Critical Paths**: 100% - -### 10.2 Performance Targets -- **Unit Test Execution**: < 30 seconds -- **Integration Test Execution**: < 5 minutes -- **E2E Test Execution**: < 15 minutes -- **Memory Usage**: < 8GB for full test suite - -### 10.3 Quality Gates -- All unit tests must pass -- Integration tests must pass -- Coverage threshold must be met -- No critical security vulnerabilities -- Performance regression tests must pass diff --git a/vllm_omni/dit_cache_interface.py b/vllm_omni/dit_cache_interface.py deleted file mode 100644 index 260acac5f..000000000 --- a/vllm_omni/dit_cache_interface.py +++ /dev/null @@ -1,24 +0,0 @@ -from dataclasses import dataclass - - -@dataclass -class DiTCacheTensor: - """ - A class for specifying how the workers should initialize the DiT cache. - """ - size: int # the size of the cache tensor in bytes - - -@dataclass -class DiTCacheConfig: - """ - The DiT cache configuration of a model. - """ - """How should model runner initialize the KV cache tensors for each layer""" - dit_cache_tensors: list[DiTCacheTensor] - """ - The DiT cache groups of the model. - For models with only one type of DiT, there is only one group that - contains all layers. - """ - kv_cache_groups: list[DiTCacheTensor] \ No newline at end of file diff --git a/vllm_omni/entrypoints/omni_llm.py b/vllm_omni/entrypoints/omni_llm.py index c37bff5e8..8223c6f20 100644 --- a/vllm_omni/entrypoints/omni_llm.py +++ b/vllm_omni/entrypoints/omni_llm.py @@ -9,11 +9,12 @@ from vllm.v1.engine.llm_engine import LLMEngine from vllm.outputs import RequestOutput, LoRARequest from ..config import OmniStageConfig -from ..core.stage_manager import StageManager +from .stage_manager import StageManager from ..engine.output_processor import MultimodalOutputProcessor from ..engine.diffusion_engine import DiffusersPipelineEngine + class OmniLLM(LLM): """Extended LLM supporting multiple engines and stage-based processing.""" @@ -482,12 +483,12 @@ class AsyncOmniLLM(LLM): if stage_config and stage_config.engine_type == "DiT": dit = stage_config.dit_config if dit and getattr(dit, "use_diffusers", False): - if not hasattr(self, "_dit_executors"): - self._dit_executors = {} - exec_inst = self._dit_executors.get(stage_config.stage_id) + if not hasattr(self, "_dit_engines"): + self._dit_engines = {} + exec_inst = self._dit_engines.get(stage_config.stage_id) if exec_inst is None: - from vllm_omni.executor.diffusers_executor import ( - DiffusersPipelineExecutor, + from vllm_omni.engine.diffusion_engine import ( + DiffusersPipelineEngine, ) pipeline_name = getattr(dit, "diffusers_pipeline", None) @@ -506,13 +507,13 @@ class AsyncOmniLLM(LLM): else: dtype = getattr(model_cfg, "dtype", None) - exec_inst = DiffusersPipelineExecutor( + exec_inst = DiffusersPipelineEngine( model_path=stage_config.model_path, pipeline_name=pipeline_name, device=device, dtype=dtype, ) - self._dit_executors[stage_config.stage_id] = exec_inst + self._dit_engines[stage_config.stage_id] = exec_inst return exec_inst.generate( prompt=processed_input.get("prompt", ""), diff --git a/vllm_omni/core/stage_manager.py b/vllm_omni/entrypoints/stage_manager.py similarity index 100% rename from vllm_omni/core/stage_manager.py rename to vllm_omni/entrypoints/stage_manager.py diff --git a/vllm_omni/executor/__init__.py b/vllm_omni/executor/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/vllm_omni/executor/diffusers_executor.py b/vllm_omni/executor/diffusers_executor.py deleted file mode 100644 index 8de5d7b9e..000000000 --- a/vllm_omni/executor/diffusers_executor.py +++ /dev/null @@ -1,64 +0,0 @@ -""" -Diffusers-backed executor for DiT stages. - -Provides a minimal wrapper around HuggingFace diffusers pipelines to support -text-to-image generation as the DiT stage backend. -""" - -from __future__ import annotations - -from typing import Any, Optional - -from vllm_omni.worker.gpu_diffusion_worker import ( - DiffusionGPUWorker, - DiffusionRunnerOutput, -) - - -class DiffusersPipelineExecutor: - """Thin executor that delegates to a diffusion worker.""" - - def __init__( - self, - model_path: str, - device: Optional[str] = None, - dtype: Optional[str] = None, - pipeline_name: Optional[str] = None, - ) -> None: - try: - import torch # noqa: F401 - import diffusers # noqa: F401 - except Exception as e: # pragma: no cover - raise ImportError( - "diffusers and torch are required for DiT diffusers backend." - ) from e - - self.worker = DiffusionGPUWorker( - model_path, - pipeline_name=pipeline_name, - device=device, - dtype=dtype, - ) - - def generate( - self, - prompt: str, - *, - height: int = 512, - width: int = 512, - num_inference_steps: int = 30, - guidance_scale: float = 5.0, - negative_prompt: Optional[str] = None, - seed: Optional[int] = None, - image: Optional[Any] = None, - ) -> DiffusionRunnerOutput: - return self.worker.generate( - prompt=prompt, - height=height, - width=width, - num_inference_steps=num_inference_steps, - guidance_scale=guidance_scale, - negative_prompt=negative_prompt, - seed=seed, - image=image, - )