- Removed unnecessary imports of numpy and torch within the request scheduling logic.
- Simplified the handling of auxiliary hidden states by initializing it to None when not present.
- Optimized CPU tensor conversion for text hidden states to improve performance.
- Introduced ARModelRunner for autoregressive GPU model execution, enabling per-request hidden state outputs and token sampling.
- Added ARGPUWorker to initialize GPU settings and manage distributed environment for the AR model runner.
- Enhanced memory management and error handling for GPU resources during initialization.
- Add complete directory structure for multimodal and non-autoregressive processing
- Include AsyncLLM, OmniEngine, and DiffusionEngine modules
- Add processors, schedulers, executors, and output processors
- Add API interfaces for Higress, Gradio, and ComfyUI
- Add configuration management and CLI interface
- Add comprehensive test structure and documentation
- Add pyproject.toml with dependencies and project configuration
- Add examples and documentation structure
tzhouam