- Eliminated redundant numpy import statements to streamline the code.
- Changed the return type of extract_multimodal_outputs from dict to tuple for better clarity on output structure.
- Deleted the `has_kv_transfer_group` import as it was not utilized.
- Cleaned up commented-out code related to model methods, focusing on the forward method.
- Added a TODO comment to implement the diffuse method for additional model support.
- Renamed `DiffusionModelRunner` to `GPUDiffusionModelRunner` for better clarity in class purpose.
- Updated `DiffusionGPUWorker` to utilize the renamed `GPUDiffusionModelRunner`.
- Simplified output handling in `GPUDiffusionModelRunner` by removing unnecessary checks, enhancing code readability.
- Simplified the assignment of sampled_token_ids by removing the conditional check for "model_stage".
- Ensured consistent handling of sampled_token_ids for improved clarity and maintainability.
- Updated assertions and tensor management to correctly handle multimodal outputs in the `DiffusionModelRunner` class.
- Ensured that the output tensors are processed based on the type of multimodal outputs received, improving robustness and clarity in the code.
- Removed unnecessary imports of numpy and torch within the request scheduling logic.
- Simplified the handling of auxiliary hidden states by initializing it to None when not present.
- Optimized CPU tensor conversion for text hidden states to improve performance.
- Introduced OmniGPUModelRunner for enhanced GPU model execution with state management.
- Added OmniModelInputForGPU and its builder to support additional information in model inputs.
- Implemented methods for updating request states and handling multimodal inputs.
- Enhanced input preparation for CUDA graph execution and LoRA integration.
- Introduced `DiffusionModelRunner` class to handle diffusion processes, extending `OmniGPUModelRunner`.
- Updated `DiffusionGPUWorker` to initialize device settings and manage memory more effectively.
- Enhanced model execution with support for multimodal inputs and improved tensor management.
- Removed legacy code and streamlined the generation process for better performance.
- Introduced ARModelRunner for autoregressive GPU model execution, enabling per-request hidden state outputs and token sampling.
- Added ARGPUWorker to initialize GPU settings and manage distributed environment for the AR model runner.
- Enhanced memory management and error handling for GPU resources during initialization.
- Add complete directory structure for multimodal and non-autoregressive processing
- Include AsyncLLM, OmniEngine, and DiffusionEngine modules
- Add processors, schedulers, executors, and output processors
- Add API interfaces for Higress, Gradio, and ComfyUI
- Add configuration management and CLI interface
- Add comprehensive test structure and documentation
- Add pyproject.toml with dependencies and project configuration
- Add examples and documentation structure
Gao Han