del unnessearcy files

This commit is contained in:
hsliu committed 2025-09-29 17:49:32 +08:00
1 parent f8a6606176
commit c330e5e8f3
10 files changed
+10 -1196

No files matched your search

-179
View File
@@ -1,179 +0,0 @@
# vLLM-omni Product Requirements Document (PRD)
## 1. Product Overview
### 1.1 Product Name
vLLM-omni: Multi-modality models inference and serving with non-autoregressive structures
### 1.2 Product Vision
Extend vLLM beyond traditional text-based, autoregressive generation to support multi-modality models with non-autoregressive structures and non-textual outputs while maintaining vLLM's proven architecture and performance.
### 1.3 Target Users
- AI researchers working with multimodal models
- ML engineers building production inference systems
- Developers integrating DiT (Diffusion Transformer) models
- Organizations requiring efficient multimodal model serving
## 2. Core Requirements
### 2.1 Functional Requirements
#### 2.1.1 Multi-Stage Processing
- **REQ-001**: Support stage-based model processing where each stage can use different engine types (AR/DiT)
- **REQ-002**: Enable sequential processing through multiple stages with data flow between stages
- **REQ-003**: Support both autoregressive (AR) and diffusion (DiT) model stages
#### 2.1.2 vLLM Compatibility
- **REQ-004**: Maintain full compatibility with vLLM V1 architecture (AsyncLLM and EngineCore patterns)
- **REQ-005**: Support existing vLLM CLI commands with `--omni` flag extension
- **REQ-006**: Reuse vLLM's multiprocess worker architecture for scalability
#### 2.1.3 Multimodal Support
- **REQ-007**: Support text, image, and latent space inputs/outputs
- **REQ-008**: Enable image-to-image, text-to-image, and text-to-text generation
- **REQ-009**: Support hidden state passing between AR and DiT stages
#### 2.1.4 CLI and API
- **REQ-010**: Provide CLI command: `vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8000`
- **REQ-011**: Support both online (AsyncOmniLLM) and offline (OmniLLM) inference modes
- **REQ-012**: Maintain vLLM's existing API compatibility
### 2.2 Non-Functional Requirements
#### 2.2.1 Performance
- **REQ-013**: Maintain vLLM's inference performance for AR stages
- **REQ-014**: Optimize DiT stage processing with caching mechanisms
- **REQ-015**: Support distributed inference across multiple GPUs
#### 2.2.2 Scalability
- **REQ-016**: Support horizontal scaling through vLLM's worker process pattern
- **REQ-017**: Enable efficient memory management for large multimodal models
- **REQ-018**: Support batch processing for multiple requests
#### 2.2.3 Extensibility
- **REQ-019**: Easy integration of new modalities and model architectures
- **REQ-020**: Pluggable scheduler and executor components
- **REQ-021**: Support for future non-autoregressive model types
## 3. Technical Architecture
### 3.1 Core Components
#### 3.1.1 Entry Points
- **OmniServeCommand**: CLI wrapper that intercepts vLLM commands with `--omni` flag
- **OmniLLM**: Offline inference class supporting multi-stage processing
- **AsyncOmniLLM**: Online inference class with asynchronous processing
#### 3.1.2 Stage Management
- **OmniStageConfig**: Configuration for each processing stage
- **Stage Engine List**: Multiple AsyncLLM instances for each stage
- **Stage I/O Management**: Data flow between stages
#### 3.1.3 Engine Components
- **EngineCore**: Reused from vLLM (no changes needed)
- **OmniDiffusionScheduler**: New scheduler for DiT models
- **DiTCacheManager**: Caching system for DiT optimization
- **MultiprocExecutor**: Reused from vLLM for DiT without diffusers
- **DiffusersPipelineExecutor**: New executor for diffusers integration
#### 3.1.4 Model Runners
- **OmniDiffusionModelRunner**: Handles DiT model execution
- **OmniARModelRunner**: Handles AR model execution with hidden state output
- **ModelRunnerOutput**: Extended to support multimodal outputs via pooler_output
#### 3.1.5 Output Processing
- **MultimodalOutputProcessor**: Handles final multimodal output processing
- **RequestState**: Extended to support pooling outputs
- **Output Handlers**: Type-specific output processing
### 3.2 Data Flow
```
API Server → OmniLLM/AsyncOmniLLM → LLMEngine/AsyncLLM → Engine Core
→ Scheduler (AR/DiT) → Executor (AR/DiT) → Worker (AR/DiT)
→ ModelRunner (AR/DiT) → RequestState → OutputProcessor → Final Output
```
## 4. Implementation Phases
### Phase 1: Foundation (Weeks 1-2)
- Package structure and dependencies
- Basic OmniLLM and AsyncOmniLLM classes
- Stage configuration system
- CLI integration
### Phase 2: Core Components (Weeks 3-4)
- DiT scheduler implementation
- Model runners for AR and DiT
- Basic output processing
### Phase 3: Advanced Features (Weeks 5-6)
- Caching system implementation
- Multimodal output processing
- Request state management
### Phase 4: Integration & Testing (Weeks 7-8)
- End-to-end integration
- Comprehensive testing
- Performance optimization
- Documentation
## 5. Success Criteria
### 5.1 Functional Success
- [ ] Successfully run `vllm serve model --omni` command
- [ ] Process multi-stage AR→DiT pipelines
- [ ] Generate multimodal outputs (text + image)
- [ ] Maintain vLLM API compatibility
### 5.2 Performance Success
- [ ] AR stage performance within 5% of native vLLM
- [ ] DiT stage processing with reasonable latency
- [ ] Memory usage comparable to vLLM for equivalent models
### 5.3 Quality Success
- [ ] 90%+ test coverage
- [ ] All integration tests passing
- [ ] Documentation complete and accurate
## 6. Risk Assessment
### 6.1 Technical Risks
- **High**: vLLM API changes breaking compatibility
- **Medium**: DiT model integration complexity
- **Low**: Performance overhead from multi-stage processing
### 6.2 Mitigation Strategies
- Regular vLLM compatibility testing
- Incremental DiT integration with fallback options
- Performance benchmarking at each stage
## 7. Dependencies
### 7.1 External Dependencies
- vLLM >= 0.10.2
- PyTorch >= 2.7
- Transformers >= 4.30.0
- FastAPI, Uvicorn for API serving
- Ray for distributed computing
### 7.2 Optional Dependencies
- xDiT for DiT acceleration
- Cache-DiT for advanced caching
- Diffusers for pipeline-based DiT models
## 8. Future Roadmap
### 8.1 Short-term (3 months)
- Additional DiT model support
- Performance optimizations
- Enhanced caching strategies
### 8.2 Medium-term (6 months)
- Support for video generation models
- Advanced scheduling strategies
- Multi-GPU DiT optimization
### 8.3 Long-term (12 months)
- Custom model architecture support
- Advanced multimodal fusion
- Production deployment tools
-5
View File
@@ -1,5 +0,0 @@
# vLLM-omni Documentation
Welcome to the vLLM-omni documentation! This documentation provides comprehensive information about using and developing with vLLM-omni.
# TODO (add examples)
-157
View File
@@ -1,157 +0,0 @@
# vLLM-omni Software Design Document
## Overview
vLLM-omni extends vLLM with multi-stage, multimodal capabilities. It supports both autoregressive (AR) stages and diffusion (DiT) stages, enabling pipelines such as text→image generation.
## Architecture Principles
1. **vLLM V1 Compatibility** – Reuse LLMEngine/AsyncLLM and executor abstractions.
2. **Stage-based Processing** – Each stage encapsulates its own engine and configuration.
3. **Executor/Worker Separation** – Executors remain thin; model-specific logic lives in workers and runners.
4. **Extensibility** – New modalities can be added by implementing dedicated stage configs, workers, and output processors.
## Key Data Flow
```
Request → OmniLLM/AsyncOmniLLM → StageManager → LLMEngine/Executor → Worker → ModelRunner → OutputProcessor → Response
```
For diffusion stages, the executor hands off to `DiffusionGPUWorker`, which wraps `DiffusionModelRunner` (responsible for diffusers pipeline loading and inference).
## Core Components (Ordered by Data Flow)
### 1. Installation
```bash
pip install vllm
pip install vllm-omni
```
### 2. Online inference launch (Entry Point)
```bash
vllm serve Qwen/Qwen3-0.6B --omni --port 8000
```
### 3. Omni Serve Command (CLI)
```python
# vllm_omni/entrypoints/cli/main.py
if "--omni" in sys.argv:
omni_args = [arg for arg in sys.argv[1:] if arg not in {"--omni", "serve"}]
OmniServeCommand().run(omni_args)
else:
from vllm.entrypoints.cli.main import main as vllm_main
vllm_main()
```
### 4. Offline Inference (OmniLLM)
```python
# vllm_omni/entrypoints/omni_llm.py
class OmniLLM(LLM):
def __init__(..., stage_configs):
super().__init__(model=stage_configs[0].model_path if stage_configs else "test-model")
self.stage_manager = StageManager(stage_configs)
self.output_processor = MultimodalOutputProcessor()
self.stage_manager.initialize_engines()
def generate(...):
current_output = None
for idx, (config, args) in enumerate(zip(self.stage_configs, stage_args_list)):
engine = self.stage_manager.get_engine(idx)
processed = self._process_stage_inputs(config, args, current_output)
stage_output = self._execute_stage(engine, processed, stage_config=config)
current_output = stage_output
config.stage_output = stage_output
return self.output_processor.process_output(current_output)
```
### 5. Stage Manager
```python
# vllm_omni/core/stage_manager.py
class StageManager:
def initialize_engines(self):
if self._initialized:
return
for config in self.stage_configs:
if config.engine_type == "AR":
engine = self._create_ar_engine(config)
elif config.engine_type == "DiT":
engine = self._create_dit_engine(config)
self.engine_list.append(engine)
self._initialized = True
```
### 6. Diffusion Execution (DiT Stage)
```python
# vllm_omni/worker/diffusion_model_runner.py
class DiffusionModelRunner:
def __init__(self, model_path, pipeline_name=None, device=None, dtype=None):
self.device, self.torch_dtype = self._resolve_device_and_dtype(device, dtype)
self._pipeline = self._load_pipeline(model_path, pipeline_name)
def generate(...):
output = self._pipeline(...)
return DiffusionRunnerOutput(prompt=prompt, images=getattr(output, "images", []) or [])
```
```python
# vllm_omni/worker/gpu_diffusion_worker.py
class DiffusionGPUWorker:
def __init__(self, model_path, ...):
self.model_runner = DiffusionModelRunner(model_path, ...)
def generate(...):
return self.model_runner.generate(...)
```
```python
# vllm_omni/executor/diffusers_executor.py
class DiffusersPipelineExecutor:
def __init__(self, model_path, ...):
self.worker = DiffusionGPUWorker(model_path, ...)
def generate(...):
return self.worker.generate(...)
```
This mirrors the standard executor→worker→runner layering in vLLM.
### 7. Stage Configuration
```python
# vllm_omni/config/stage_config.py
@dataclass
class OmniStageConfig:
stage_id: int
engine_type: Literal["AR", "DiT"]
model_path: str
input_modalities: List[str]
output_modalities: List[str]
vllm_config: Optional[VllmConfig] = None
executor_class: Optional[type[Executor]] = None
dit_config: Optional[DiTConfig] = None
```
- AR stages default to `UniProcExecutor`.
- DiT stages set `executor_class` to `DiffusersPipelineExecutor` when `use_diffusers=True`.
### 8. Output Processing
```python
# vllm_omni/engine/output_processor.py
class MultimodalOutputProcessor(OutputProcessor):
def process_output(self, engine_core_output):
output_type = self._detect_output_type(engine_core_output)
handler = self.output_handlers.get(output_type, self._process_pooling_output)
return handler(engine_core_output)
```
### 9. CLI Package Configuration
```toml
# pyproject.toml
[project.scripts]
vllm = "vllm_omni.entrypoints.cli.main:main"
vllm-omni = "vllm_omni.entrypoints.cli.main:main"
```
## Future Work
- Scheduler adaptations for diffusion batching.
- Advanced output processors for audio/video.
- Multi-worker execution (ray / multiprocess) for heavy DiT stages.
@@ -300,7 +300,7 @@ omni = "vllm_omni.plugin:OmniPlugin"
### 7.1 Package Installation
```bash
pip install vllm>=0.2.0
pip install vllm>=0.10.2
pip install vllm-omni
```
-758
View File
@@ -1,758 +0,0 @@
# vLLM-omni Test Design Document
## 1. Testing Strategy Overview
### 1.1 Testing Pyramid
```
E2E Tests (10%)
/ \
Integration Tests (30%)
/ \
Unit Tests (60%)
```
### 1.2 Test Categories
- **Unit Tests**: Individual component testing with mocked dependencies
- **Integration Tests**: Component interaction testing with real vLLM integration
- **End-to-End Tests**: Full pipeline testing with real models
- **Performance Tests**: Benchmarking and profiling
- **Compatibility Tests**: vLLM version compatibility validation
## 2. Test Structure
```
tests/
├── __init__.py
├── conftest.py # Shared fixtures and configuration
├── unit/ # Unit tests (60%)
│ ├── __init__.py
│ ├── test_config/ # Configuration testing
│ │ ├── test_stage_config.py
│ │ ├── test_dit_config.py
│ │ └── test_cache_config.py
│ ├── test_core/ # Core component testing
│ │ ├── test_omni_llm.py
│ │ ├── test_async_omni_llm.py
│ │ ├── test_stage_manager.py
│ │ └── test_dit_cache_manager.py
│ ├── test_scheduler/ # Scheduler testing
│ │ ├── test_diffusion_scheduler.py
│ │ └── test_scheduler_interface.py
│ ├── test_executor/ # Executor testing
│ │ ├── test_base_executor.py
│ │ └── test_diffusers_executor.py
│ ├── test_model_executor/ # Model runner testing
│ │ ├── test_ar_model_runner.py
│ │ └── test_dit_model_runner.py
│ ├── test_engine/ # Engine testing
│ │ ├── test_output_processor.py
│ │ └── test_multimodal_processor.py
│ ├── test_worker/ # Worker testing
│ │ └── test_omni_worker.py
│ └── test_utils/ # Utility testing
│ ├── test_multimodal.py
│ └── test_vae.py
├── integration/ # Integration tests (30%)
│ ├── __init__.py
│ ├── test_vllm_integration.py
│ ├── test_stage_processing.py
│ ├── test_cli_integration.py
│ └── test_api_compatibility.py
├── e2e/ # End-to-end tests (10%)
│ ├── __init__.py
│ ├── test_full_pipeline.py
│ ├── test_multimodal_generation.py
│ └── test_performance.py
├── benchmarks/ # Performance benchmarks
│ ├── __init__.py
│ ├── test_memory_usage.py
│ ├── test_latency.py
│ └── test_throughput.py
└── fixtures/ # Test data and fixtures
├── __init__.py
├── sample_models/
├── test_images/
└── test_configs/
```
## 3. Unit Test Specifications
### 3.1 Configuration Tests
#### test_stage_config.py
```python
import pytest
from vllm_omni.config.stage_config import OmniStageConfig, DiTConfig, DiTCacheConfig
class TestOmniStageConfig:
def test_ar_stage_config_creation(self):
"""Test AR stage configuration creation"""
config = OmniStageConfig(
stage_id=0,
engine_type="AR",
model_path="test-model",
input_modalities=["text"],
output_modalities=["text"]
)
assert config.engine_type == "AR"
assert config.stage_id == 0
def test_dit_stage_config_creation(self):
"""Test DiT stage configuration creation"""
dit_config = DiTConfig(
num_inference_steps=50,
guidance_scale=7.5,
use_diffusers=False,
)
config = OmniStageConfig(
stage_id=1,
engine_type="DiT",
model_path="test-dit-model",
input_modalities=["text"],
output_modalities=["image"],
dit_config=dit_config
)
assert config.engine_type == "DiT"
assert config.dit_config is not None
def test_invalid_engine_type(self):
"""Test validation of engine type"""
with pytest.raises(ValueError):
OmniStageConfig(
stage_id=0,
engine_type="INVALID",
model_path="test-model",
input_modalities=["text"],
output_modalities=["text"]
)
```
#### test_dit_config.py
```python
import pytest
from vllm.config import ModelConfig
from vllm_omni.config import DiTConfig
class TestDiTConfig:
def test_dit_config_defaults(self):
"""Test DiT configuration with defaults"""
config = DiTConfig(num_inference_steps=50)
assert config.model_config is None
assert config.use_diffusers is False
assert config.diffusers_pipeline is None
def test_diffusers_config(self):
"""Test DiT configuration with diffusers"""
config = DiTConfig(
model_config=ModelConfig(model="stabilityai/stable-diffusion-2-1"),
num_inference_steps=50,
use_diffusers=True,
diffusers_pipeline="stable-diffusion",
)
assert config.use_diffusers is True
assert config.diffusers_pipeline == "stable-diffusion"
```
### 3.2 Core Component Tests
#### test_omni_llm.py
```python
import pytest
from unittest.mock import Mock, patch
from vllm_omni.core.omni_llm import OmniLLM
from vllm_omni.config.stage_config import OmniStageConfig
class TestOmniLLM:
@pytest.fixture
def mock_stage_configs(self):
"""Create mock stage configurations"""
return [
OmniStageConfig(
stage_id=0,
engine_type="AR",
model_path="test-ar-model",
input_modalities=["text"],
output_modalities=["text"]
),
OmniStageConfig(
stage_id=1,
engine_type="DiT",
model_path="test-dit-model",
input_modalities=["text"],
output_modalities=["image"]
)
]
@patch('vllm_omni.core.omni_llm.LLMEngine')
def test_omni_llm_initialization(self, mock_llm_engine, mock_stage_configs):
"""Test OmniLLM initialization"""
omni_llm = OmniLLM(mock_stage_configs)
assert len(omni_llm.engine_list) == 2
assert omni_llm.stage_configs == mock_stage_configs
@patch('vllm_omni.core.omni_llm.LLMEngine')
def test_stage_engine_creation(self, mock_llm_engine, mock_stage_configs):
"""Test stage engine creation"""
omni_llm = OmniLLM(mock_stage_configs)
# Verify engines are created for each stage
assert mock_llm_engine.from_vllm_config.call_count == 2
def test_process_stage_inputs_ar(self, mock_stage_configs):
"""Test AR stage input processing"""
omni_llm = OmniLLM(mock_stage_configs)
ar_config = mock_stage_configs[0]
stage_args = {"prompt": "test prompt"}
result = omni_llm._process_stage_inputs(ar_config, stage_args, None)
assert "prompt" in result
def test_process_stage_inputs_dit(self, mock_stage_configs):
"""Test DiT stage input processing"""
omni_llm = OmniLLM(mock_stage_configs)
dit_config = mock_stage_configs[1]
stage_args = {"image": "test_image.jpg"}
with patch.object(omni_llm, 'vae') as mock_vae:
mock_vae.encode.return_value = "encoded_image"
result = omni_llm._process_stage_inputs(dit_config, stage_args, None)
assert "encoded_image" in result
```
#### test_async_omni_llm.py
```python
import pytest
import asyncio
from unittest.mock import Mock, patch, AsyncMock
from vllm_omni.core.omni_llm import AsyncOmniLLM
class TestAsyncOmniLLM:
@pytest.fixture
def mock_stage_configs(self):
"""Create mock stage configurations"""
return [
OmniStageConfig(
stage_id=0,
engine_type="AR",
model_path="test-ar-model",
input_modalities=["text"],
output_modalities=["text"]
)
]
@pytest.mark.asyncio
async def test_async_generation(self, mock_stage_configs):
"""Test async generation"""
with patch('vllm_omni.core.omni_llm.AsyncLLM') as mock_async_llm:
async_omni_llm = AsyncOmniLLM(mock_stage_configs)
# Mock the generate_async method
mock_async_llm.return_value.generate_async = AsyncMock(return_value=[])
result = await async_omni_llm.generate_async([{"prompt": "test"}])
assert result == []
```
### 3.3 Scheduler Tests
#### test_diffusion_scheduler.py
```python
import pytest
from unittest.mock import Mock, patch
from vllm_omni.core.sched.diffusion_scheduler import OmniDiffusionScheduler
class TestOmniDiffusionScheduler:
@pytest.fixture
def mock_configs(self):
"""Create mock configurations"""
return {
'vllm_config': Mock(),
'kv_cache_config': Mock(),
'dit_cache_config': Mock(),
'structured_output_manager': Mock(),
'mm_registry': Mock(),
'include_finished_set': False,
'log_stats': False
}
def test_scheduler_initialization(self, mock_configs):
"""Test scheduler initialization"""
scheduler = OmniDiffusionScheduler(**mock_configs)
assert scheduler.dit_cache_manager is not None
def test_schedule_method(self, mock_configs):
"""Test scheduling method"""
scheduler = OmniDiffusionScheduler(**mock_configs)
with patch.object(scheduler, 'dit_cache_manager') as mock_cache:
mock_cache.allocate_cache.return_value = Mock()
result = scheduler.schedule()
# Verify scheduling logic
assert result is not None
```
### 3.4 Model Runner Tests
#### test_ar_model_runner.py
```python
import pytest
import torch
from unittest.mock import Mock, patch
from vllm_omni.model_executor.ar_model_runner import OmniARModelRunner
class TestOmniARModelRunner:
@pytest.fixture
def mock_runner(self):
"""Create mock model runner"""
with patch('vllm_omni.model_executor.ar_model_runner.GPUModelRunner.__init__'):
runner = OmniARModelRunner(Mock(), Mock(), Mock())
return runner
def test_execute_model_ar(self, mock_runner):
"""Test AR model execution"""
mock_scheduler_output = Mock()
mock_scheduler_output.req_ids = ["req1"]
mock_scheduler_output.req_id_to_index = {"req1": 0}
with patch.object(mock_runner, 'model') as mock_model:
mock_model.forward.return_value = (torch.randn(1, 10, 768), None)
result = mock_runner.execute_model(mock_scheduler_output)
assert result.req_ids == ["req1"]
assert len(result.pooler_output) > 0 # Hidden states
```
#### test_dit_model_runner.py
```python
import pytest
import torch
from unittest.mock import Mock, patch
from vllm_omni.model_executor.dit_model_runner import OmniDiffusionModelRunner
class TestOmniDiffusionModelRunner:
@pytest.fixture
def mock_runner(self):
"""Create mock DiT model runner"""
with patch('vllm_omni.model_executor.dit_model_runner.GPUModelRunner.__init__'):
runner = OmniDiffusionModelRunner(Mock(), Mock(), Mock())
return runner
def test_execute_model_dit(self, mock_runner):
"""Test DiT model execution"""
mock_scheduler_output = Mock()
mock_scheduler_output.req_ids = ["req1"]
mock_scheduler_output.req_id_to_index = {"req1": 0}
with patch.object(mock_runner, 'model') as mock_model:
mock_model.forward.return_value = torch.randn(1, 3, 512, 512)
result = mock_runner.execute_model(mock_scheduler_output)
assert result.req_ids == ["req1"]
assert len(result.pooler_output) > 0 # DiT output tensors
```
## 4. Integration Tests
### 4.1 vLLM Integration Tests
#### test_vllm_integration.py
```python
import pytest
from vllm_omni.core.omni_llm import OmniLLM
from vllm_omni.config.stage_config import OmniStageConfig
class TestVLLMIntegration:
@pytest.mark.integration
def test_vllm_engine_creation(self):
"""Test creation of vLLM engines"""
stage_configs = [
OmniStageConfig(
stage_id=0,
engine_type="AR",
model_path="Qwen/Qwen3-0.6B", # Small test model
input_modalities=["text"],
output_modalities=["text"]
)
]
omni_llm = OmniLLM(stage_configs)
assert len(omni_llm.engine_list) == 1
assert omni_llm.engine_list[0] is not None
@pytest.mark.integration
def test_stage_processing_integration(self):
"""Test integration between stages"""
# Test with real vLLM components
pass
```
### 4.2 CLI Integration Tests
#### test_cli_integration.py
```python
import pytest
import subprocess
import sys
from vllm_omni.entrypoints.cli.main import main
class TestCLIIntegration:
def test_omni_flag_detection(self):
"""Test --omni flag detection"""
original_argv = sys.argv
try:
sys.argv = ["vllm", "serve", "test-model", "--omni"]
# Test that omni command is triggered
with patch('vllm_omni.entrypoints.omni.OmniServeCommand') as mock_serve:
main()
mock_serve.assert_called_once()
finally:
sys.argv = original_argv
def test_forward_to_vllm(self):
"""Test forwarding to vLLM when --omni not present"""
original_argv = sys.argv
try:
sys.argv = ["vllm", "serve", "test-model"]
with patch('vllm.entrypoints.cli.main.main') as mock_vllm_main:
main()
mock_vllm_main.assert_called_once()
finally:
sys.argv = original_argv
```
## 5. End-to-End Tests
### 5.1 Full Pipeline Tests
#### test_full_pipeline.py
```python
import pytest
from vllm.config import ModelConfig
from vllm_omni.core.omni_llm import OmniLLM
from vllm_omni.config.stage_config import OmniStageConfig, DiTConfig
class TestFullPipeline:
@pytest.mark.e2e
@pytest.mark.slow
def test_ar_to_dit_pipeline(self):
"""Test complete AR to DiT pipeline"""
stage_configs = [
OmniStageConfig(
stage_id=0,
engine_type="AR",
model_path="Qwen/Qwen3-0.6B",
input_modalities=["text"],
output_modalities=["text"]
),
OmniStageConfig(
stage_id=1,
engine_type="DiT",
model_path="stabilityai/stable-diffusion-2-1",
input_modalities=["text"],
output_modalities=["image"],
dit_config=DiTConfig(
model_config=ModelConfig(model="stabilityai/stable-diffusion-2-1"),
num_inference_steps=10,
guidance_scale=7.5,
use_diffusers=False,
)
)
]
omni_llm = OmniLLM(stage_configs)
stage_args = [
{"prompt": "A beautiful landscape"},
{"prompt": "A beautiful landscape"}
]
result = omni_llm.generate(stage_args)
assert result is not None
assert len(result) > 0
```
### 5.2 Multimodal Generation Tests
#### test_multimodal_generation.py
```python
import pytest
from vllm_omni.core.omni_llm import OmniLLM
class TestMultimodalGeneration:
@pytest.mark.e2e
def test_text_to_image_generation(self):
"""Test text to image generation"""
# Test implementation
pass
@pytest.mark.e2e
def test_image_to_text_generation(self):
"""Test image to text generation"""
# Test implementation
pass
@pytest.mark.e2e
def test_text_and_image_generation(self):
"""Test combined text and image generation"""
# Test implementation
pass
```
## 6. Performance Tests
### 6.1 Memory Usage Tests
#### test_memory_usage.py
```python
import pytest
import psutil
import torch
from vllm_omni.core.omni_llm import OmniLLM
class TestMemoryUsage:
@pytest.mark.benchmark
def test_memory_usage_ar_stage(self):
"""Test memory usage for AR stage"""
process = psutil.Process()
initial_memory = process.memory_info().rss
# Run AR stage
# ... test implementation
final_memory = process.memory_info().rss
memory_increase = final_memory - initial_memory
# Assert memory usage is reasonable
assert memory_increase < 1024 * 1024 * 1024 # Less than 1GB
@pytest.mark.benchmark
def test_memory_usage_dit_stage(self):
"""Test memory usage for DiT stage"""
# Test implementation
pass
@pytest.mark.benchmark
def test_cache_memory_management(self):
"""Test DiT cache memory management"""
# Test implementation
pass
```
### 6.2 Latency Tests
#### test_latency.py
```python
import pytest
import time
from vllm_omni.core.omni_llm import OmniLLM
class TestLatency:
@pytest.mark.benchmark
def test_ar_stage_latency(self):
"""Test AR stage latency"""
start_time = time.time()
# Run AR stage
# ... test implementation
end_time = time.time()
latency = end_time - start_time
# Assert latency is reasonable
assert latency < 5.0 # Less than 5 seconds
@pytest.mark.benchmark
def test_dit_stage_latency(self):
"""Test DiT stage latency"""
# Test implementation
pass
@pytest.mark.benchmark
def test_end_to_end_latency(self):
"""Test end-to-end pipeline latency"""
# Test implementation
pass
```
## 7. Test Configuration
### 7.1 pytest.ini
```ini
[tool:pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts =
--strict-markers
--strict-config
--cov=vllm_omni
--cov-report=term-missing
--cov-report=html
--cov-report=xml
--tb=short
markers =
unit: Unit tests
integration: Integration tests
e2e: End-to-end tests
benchmark: Performance benchmark tests
slow: Slow running tests
```
### 7.2 conftest.py
```python
import pytest
import torch
from unittest.mock import Mock
from vllm_omni.config.stage_config import OmniStageConfig, DiTConfig
@pytest.fixture(scope="session")
def device():
"""Get available device for testing"""
return "cuda" if torch.cuda.is_available() else "cpu"
@pytest.fixture
def sample_stage_configs():
"""Sample stage configurations for testing"""
return [
OmniStageConfig(
stage_id=0,
engine_type="AR",
model_path="test-ar-model",
input_modalities=["text"],
output_modalities=["text"]
),
OmniStageConfig(
stage_id=1,
engine_type="DiT",
model_path="test-dit-model",
input_modalities=["text"],
output_modalities=["image"],
dit_config=DiTConfig(
model_config=ModelConfig(model="test-dit-model"),
num_inference_steps=10,
guidance_scale=7.5,
)
)
]
@pytest.fixture
def mock_vllm_config():
"""Mock vLLM configuration"""
config = Mock()
config.model = "test-model"
config.tensor_parallel_size = 1
config.pipeline_parallel_size = 1
return config
```
## 8. Test Execution
### 8.1 Running Tests
```bash
# Run all tests
pytest
# Run specific test categories
pytest -m unit
pytest -m integration
pytest -m e2e
pytest -m benchmark
# Run with coverage
pytest --cov=vllm_omni --cov-report=html
# Run specific test file
pytest tests/unit/test_omni_llm.py
# Run with verbose output
pytest -v
# Run in parallel
pytest -n auto
```
### 8.2 Continuous Integration
```yaml
# .github/workflows/test.yml
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8, 3.9, 3.10, 3.11]
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
pip install -e ".[dev]"
- name: Run tests
run: |
pytest --cov=vllm_omni --cov-report=xml
- name: Upload coverage
uses: codecov/codecov-action@v3
```
## 9. Test Data Management
### 9.1 Fixtures Directory
```
tests/fixtures/
├── sample_models/
│ ├── ar_model/
│ └── dit_model/
├── test_images/
│ ├── landscape.jpg
│ └── portrait.png
├── test_configs/
│ ├── ar_stage_config.json
│ └── dit_stage_config.json
└── expected_outputs/
├── ar_output.json
└── dit_output.json
```
### 9.2 Test Data Loading
```python
import json
import os
from pathlib import Path
def load_test_config(config_name: str) -> dict:
"""Load test configuration from fixtures"""
config_path = Path(__file__).parent / "fixtures" / "test_configs" / f"{config_name}.json"
with open(config_path) as f:
return json.load(f)
def load_test_image(image_name: str) -> str:
"""Load test image path from fixtures"""
return str(Path(__file__).parent / "fixtures" / "test_images" / image_name)
```
## 10. Test Quality Metrics
### 10.1 Coverage Targets
- **Overall Coverage**: > 90%
- **Unit Tests**: > 95%
- **Integration Tests**: > 80%
- **Critical Paths**: 100%
### 10.2 Performance Targets
- **Unit Test Execution**: < 30 seconds
- **Integration Test Execution**: < 5 minutes
- **E2E Test Execution**: < 15 minutes
- **Memory Usage**: < 8GB for full test suite
### 10.3 Quality Gates
- All unit tests must pass
- Integration tests must pass
- Coverage threshold must be met
- No critical security vulnerabilities
- Performance regression tests must pass
-24
View File
@@ -1,24 +0,0 @@
from dataclasses import dataclass
@dataclass
class DiTCacheTensor:
"""
A class for specifying how the workers should initialize the DiT cache.
"""
size: int # the size of the cache tensor in bytes
@dataclass
class DiTCacheConfig:
"""
The DiT cache configuration of a model.
"""
"""How should model runner initialize the KV cache tensors for each layer"""
dit_cache_tensors: list[DiTCacheTensor]
"""
The DiT cache groups of the model.
For models with only one type of DiT, there is only one group that
contains all layers.
"""
kv_cache_groups: list[DiTCacheTensor]
+9 -8
View File
@@ -9,11 +9,12 @@ from vllm.v1.engine.llm_engine import LLMEngine
from vllm.outputs import RequestOutput, LoRARequest
from ..config import OmniStageConfig
from ..core.stage_manager import StageManager
from .stage_manager import StageManager
from ..engine.output_processor import MultimodalOutputProcessor
from ..engine.diffusion_engine import DiffusersPipelineEngine
class OmniLLM(LLM):
"""Extended LLM supporting multiple engines and stage-based processing."""
@@ -482,12 +483,12 @@ class AsyncOmniLLM(LLM):
if stage_config and stage_config.engine_type == "DiT":
dit = stage_config.dit_config
if dit and getattr(dit, "use_diffusers", False):
if not hasattr(self, "_dit_executors"):
self._dit_executors = {}
exec_inst = self._dit_executors.get(stage_config.stage_id)
if not hasattr(self, "_dit_engines"):
self._dit_engines = {}
exec_inst = self._dit_engines.get(stage_config.stage_id)
if exec_inst is None:
from vllm_omni.executor.diffusers_executor import (
DiffusersPipelineExecutor,
from vllm_omni.engine.diffusion_engine import (
DiffusersPipelineEngine,
)
pipeline_name = getattr(dit, "diffusers_pipeline", None)
@@ -506,13 +507,13 @@ class AsyncOmniLLM(LLM):
else:
dtype = getattr(model_cfg, "dtype", None)
exec_inst = DiffusersPipelineExecutor(
exec_inst = DiffusersPipelineEngine(
model_path=stage_config.model_path,
pipeline_name=pipeline_name,
device=device,
dtype=dtype,
)
self._dit_executors[stage_config.stage_id] = exec_inst
self._dit_engines[stage_config.stage_id] = exec_inst
return exec_inst.generate(
prompt=processed_input.get("prompt", ""),
File renamed without changes.
View File
Whitespace-only changes.
-64
View File
@@ -1,64 +0,0 @@
"""
Diffusers-backed executor for DiT stages.
Provides a minimal wrapper around HuggingFace diffusers pipelines to support
text-to-image generation as the DiT stage backend.
"""
from __future__ import annotations
from typing import Any, Optional
from vllm_omni.worker.gpu_diffusion_worker import (
DiffusionGPUWorker,
DiffusionRunnerOutput,
)
class DiffusersPipelineExecutor:
"""Thin executor that delegates to a diffusion worker."""
def __init__(
self,
model_path: str,
device: Optional[str] = None,
dtype: Optional[str] = None,
pipeline_name: Optional[str] = None,
) -> None:
try:
import torch # noqa: F401
import diffusers # noqa: F401
except Exception as e: # pragma: no cover
raise ImportError(
"diffusers and torch are required for DiT diffusers backend."
) from e
self.worker = DiffusionGPUWorker(
model_path,
pipeline_name=pipeline_name,
device=device,
dtype=dtype,
)
def generate(
self,
prompt: str,
*,
height: int = 512,
width: int = 512,
num_inference_steps: int = 30,
guidance_scale: float = 5.0,
negative_prompt: Optional[str] = None,
seed: Optional[int] = None,
image: Optional[Any] = None,
) -> DiffusionRunnerOutput:
return self.worker.generate(
prompt=prompt,
height=height,
width=width,
num_inference_steps=num_inference_steps,
guidance_scale=guidance_scale,
negative_prompt=negative_prompt,
seed=seed,
image=image,
)