- Updated the vLLM wheel to commit 24650715105a26b41c88f4164777b5e1a0f3200b in Dockerfile.ci.
- Modified the `calculate_metrics` function to handle `tokenizer` as optional (can be None) in metrics.py.
- Adjusted the `benchmark` function to accept `tokenizer` as optional in patch.py.
- Enhanced condition checks in OmniARScheduler to ensure valid token generation in omni_ar_scheduler.py.
- Updated input processing to include supported tasks in input_processor.py.
- Cleaned up unnecessary code in output_processor.py.
- Added support for SSL ciphers in the API server configuration in api_server.py.
- Enabled speculative decoding in GPU model runners for improved performance in gpu_generation_model_runner.py and gpu_model_runner.py.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated Dockerfile to use a new vllm wheel version.
- Modified test_put_get_inline to clarify that it always uses SHM and updated metadata assertions.
- Added handling for speculative configuration in GPU model runners to delay clearing connector metadata.
- Refactored payload decoding in OmniGPUModelRunner for better clarity and maintainability.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Update Dockerfile for vllm wheel version and enhance input processing in Omni models. Added decoding for byte messages in async requests, included new metrics in benchmarks, and improved pause state handling in the scheduler. Refined input and output processing methods for better multimodal support and error handling.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Translated comments from Chinese to English for better accessibility.
- Improved clarity of comments related to request handling and scheduling logic.
- Ensured comments align with the code functionality and maintain a uniform style.
- Introduced `DiffusionScheduler` to handle scheduling for diffusion models, allowing immediate request completion.
- Added `OmniNewRequestData` to encapsulate additional request data, including prompt embeddings and extra information.
- Updated `OmniScheduler` to ensure compatibility with new request data structures.
- Created `output.py` to define new data classes for enriched request handling.
- Enhanced `__init__.py` in the engine module to support new payload structures for prompt embeddings and additional information.
- Move vllm_omni/core/omni_llm.py to vllm_omni/entrypoints/omni_llm.py
- Update all import statements across the codebase to reflect new location
- Fix relative imports within the moved file
- Maintain functionality while improving code organization
- All imports and functionality tested and working correctly
This change better reflects that OmniLLM and AsyncOmniLLM are the main entry points
for vLLM-omni functionality, rather than core implementation details.
- Improve API server with better error handling and response formatting
- Enhance CLI with additional options for DiT stages and configuration
- Add comprehensive examples in examples/basic/ including:
- API client with health checks and text generation
- Docker setup and usage examples
- Simple usage patterns for different scenarios
- Add utility scripts for model downloading and Docker setup
- Update documentation with implementation details and testing guidelines
- Fix configuration validation issues in OmniLLM
- Improve stage configuration handling for AR and DiT stages
- Add proper error handling and fallback mechanisms
Tested with Qwen3-0.6B model:
- Server starts successfully on port 8000
- Health and info endpoints working correctly
- Text generation with various parameters functioning
- API client examples working as expected
- CLI help and configuration options working properly
- Add comprehensive PRD, architecture design, and test design documents
- Implement core modules: OmniLLM, AsyncOmniLLM, stage configurations
- Add DiT scheduler and cache manager for diffusion models
- Implement CLI integration with --omni flag support
- Add API server and plugin system for vLLM integration
- Create comprehensive test suite with fixtures
- Update dependencies to vLLM 0.10.2 and PyTorch 2.8.0
- Add conda environment setup and package installation
- Implement stage-based processing architecture
- Add multimodal output processing capabilities
This commit establishes the foundation for multi-modality models
inference and serving with non-autoregressive structures.
tzhouam