- Updated Dockerfile to use a new vllm wheel version.
- Modified test_put_get_inline to clarify that it always uses SHM and updated metadata assertions.
- Added handling for speculative configuration in GPU model runners to delay clearing connector metadata.
- Refactored payload decoding in OmniGPUModelRunner for better clarity and maintainability.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Update Dockerfile for vllm wheel version and enhance input processing in Omni models. Added decoding for byte messages in async requests, included new metrics in benchmarks, and improved pause state handling in the scheduler. Refined input and output processing methods for better multimodal support and error handling.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Translated comments from Chinese to English for better accessibility.
- Improved clarity of comments related to request handling and scheduling logic.
- Ensured comments align with the code functionality and maintain a uniform style.
- Introduced `DiffusionScheduler` to handle scheduling for diffusion models, allowing immediate request completion.
- Added `OmniNewRequestData` to encapsulate additional request data, including prompt embeddings and extra information.
- Updated `OmniScheduler` to ensure compatibility with new request data structures.
- Created `output.py` to define new data classes for enriched request handling.
- Enhanced `__init__.py` in the engine module to support new payload structures for prompt embeddings and additional information.
- Move vllm_omni/core/omni_llm.py to vllm_omni/entrypoints/omni_llm.py
- Update all import statements across the codebase to reflect new location
- Fix relative imports within the moved file
- Maintain functionality while improving code organization
- All imports and functionality tested and working correctly
This change better reflects that OmniLLM and AsyncOmniLLM are the main entry points
for vLLM-omni functionality, rather than core implementation details.
- Improve API server with better error handling and response formatting
- Enhance CLI with additional options for DiT stages and configuration
- Add comprehensive examples in examples/basic/ including:
- API client with health checks and text generation
- Docker setup and usage examples
- Simple usage patterns for different scenarios
- Add utility scripts for model downloading and Docker setup
- Update documentation with implementation details and testing guidelines
- Fix configuration validation issues in OmniLLM
- Improve stage configuration handling for AR and DiT stages
- Add proper error handling and fallback mechanisms
Tested with Qwen3-0.6B model:
- Server starts successfully on port 8000
- Health and info endpoints working correctly
- Text generation with various parameters functioning
- API client examples working as expected
- CLI help and configuration options working properly
- Add comprehensive PRD, architecture design, and test design documents
- Implement core modules: OmniLLM, AsyncOmniLLM, stage configurations
- Add DiT scheduler and cache manager for diffusion models
- Implement CLI integration with --omni flag support
- Add API server and plugin system for vLLM integration
- Create comprehensive test suite with fixtures
- Update dependencies to vLLM 0.10.2 and PyTorch 2.8.0
- Add conda environment setup and package installation
- Implement stage-based processing architecture
- Add multimodal output processing capabilities
This commit establishes the foundation for multi-modality models
inference and serving with non-autoregressive structures.
- Add complete directory structure for multimodal and non-autoregressive processing
- Include AsyncLLM, OmniEngine, and DiffusionEngine modules
- Add processors, schedulers, executors, and output processors
- Add API interfaces for Higress, Gradio, and ComfyUI
- Add configuration management and CLI interface
- Add comprehensive test structure and documentation
- Add pyproject.toml with dependencies and project configuration
- Add examples and documentation structure