- Eliminated redundant numpy import statements to streamline the code.
- Changed the return type of extract_multimodal_outputs from dict to tuple for better clarity on output structure.
- Added logging for exceptions during tensor movement to CPU in OmniRequestState and MultimodalOutputProcessor.
- Improved robustness by ensuring the output pipeline continues without crashing on errors.
- Updated comments for clarity on error handling behavior.
- Deleted the `has_kv_transfer_group` import as it was not utilized.
- Cleaned up commented-out code related to model methods, focusing on the forward method.
- Added a TODO comment to implement the diffuse method for additional model support.
Squashed commits:\n- 4e2d812 Add Qwen2.5-Omni model components\n- 351f833 load the thinker embedding from the weights, not rely on local .pt files\n- 2c32533 Refactor Qwen2.5-Omni model files\n- 9163a60 remove unused codes\n- 192bdf8 remove
- Renamed `DiffusionModelRunner` to `GPUDiffusionModelRunner` for better clarity in class purpose.
- Updated `DiffusionGPUWorker` to utilize the renamed `GPUDiffusionModelRunner`.
- Simplified output handling in `GPUDiffusionModelRunner` by removing unnecessary checks, enhancing code readability.
- Simplified the assignment of sampled_token_ids by removing the conditional check for "model_stage".
- Ensured consistent handling of sampled_token_ids for improved clarity and maintainability.
- Updated assertions and tensor management to correctly handle multimodal outputs in the `DiffusionModelRunner` class.
- Ensured that the output tensors are processed based on the type of multimodal outputs received, improving robustness and clarity in the code.
- Removed unnecessary imports of numpy and torch within the request scheduling logic.
- Simplified the handling of auxiliary hidden states by initializing it to None when not present.
- Optimized CPU tensor conversion for text hidden states to improve performance.
- Introduced OmniGPUModelRunner for enhanced GPU model execution with state management.
- Added OmniModelInputForGPU and its builder to support additional information in model inputs.
- Implemented methods for updating request states and handling multimodal inputs.
- Enhanced input preparation for CUDA graph execution and LoRA integration.
- Introduced `DiffusionModelRunner` class to handle diffusion processes, extending `OmniGPUModelRunner`.
- Updated `DiffusionGPUWorker` to initialize device settings and manage memory more effectively.
- Enhanced model execution with support for multimodal inputs and improved tensor management.
- Removed legacy code and streamlined the generation process for better performance.
- Introduced ARModelRunner for autoregressive GPU model execution, enabling per-request hidden state outputs and token sampling.
- Added ARGPUWorker to initialize GPU settings and manage distributed environment for the AR model runner.
- Enhanced memory management and error handling for GPU resources during initialization.
- Introduced OmniRequestState to manage multimodal request states.
- Enhanced MultimodalOutputProcessor to handle various output types including images, text, and latents.
- Implemented methods for accumulating multimodal tensors and processing outputs.
- Updated output handling to ensure compatibility with vLLM's base processor while allowing custom modality handlers.
- Add test_serving.sh: Full-featured testing suite with comprehensive validation
- Add quick_test.sh: Fast validation script for quick testing after changes
- Add scripts/README.md: Complete documentation for testing scripts
- Include health checks, text generation, performance testing, and API integration
- Add retry mechanisms and proper error handling
- Support for different models and ports
- Comprehensive logging and colored output
- Ready for CI/CD integration
Usage:
- Quick test: ./scripts/quick_test.sh [port]
- Full test: ./scripts/test_serving.sh [model_path] [port]
- Move vllm_omni/core/omni_llm.py to vllm_omni/entrypoints/omni_llm.py
- Update all import statements across the codebase to reflect new location
- Fix relative imports within the moved file
- Maintain functionality while improving code organization
- All imports and functionality tested and working correctly
This change better reflects that OmniLLM and AsyncOmniLLM are the main entry points
for vLLM-omni functionality, rather than core implementation details.
Hongsheng Liu