- Deleted the `has_kv_transfer_group` import as it was not utilized.
- Cleaned up commented-out code related to model methods, focusing on the forward method.
- Added a TODO comment to implement the diffuse method for additional model support.
Squashed commits:\n- 4e2d812 Add Qwen2.5-Omni model components\n- 351f833 load the thinker embedding from the weights, not rely on local .pt files\n- 2c32533 Refactor Qwen2.5-Omni model files\n- 9163a60 remove unused codes\n- 192bdf8 remove
- Renamed `DiffusionModelRunner` to `GPUDiffusionModelRunner` for better clarity in class purpose.
- Updated `DiffusionGPUWorker` to utilize the renamed `GPUDiffusionModelRunner`.
- Simplified output handling in `GPUDiffusionModelRunner` by removing unnecessary checks, enhancing code readability.
- Updated assertions and tensor management to correctly handle multimodal outputs in the `DiffusionModelRunner` class.
- Ensured that the output tensors are processed based on the type of multimodal outputs received, improving robustness and clarity in the code.
- Removed unnecessary imports of numpy and torch within the request scheduling logic.
- Simplified the handling of auxiliary hidden states by initializing it to None when not present.
- Optimized CPU tensor conversion for text hidden states to improve performance.
- Introduced `DiffusionModelRunner` class to handle diffusion processes, extending `OmniGPUModelRunner`.
- Updated `DiffusionGPUWorker` to initialize device settings and manage memory more effectively.
- Enhanced model execution with support for multimodal inputs and improved tensor management.
- Removed legacy code and streamlined the generation process for better performance.
- Introduced ARModelRunner for autoregressive GPU model execution, enabling per-request hidden state outputs and token sampling.
- Added ARGPUWorker to initialize GPU settings and manage distributed environment for the AR model runner.
- Enhanced memory management and error handling for GPU resources during initialization.
- Add test_serving.sh: Full-featured testing suite with comprehensive validation
- Add quick_test.sh: Fast validation script for quick testing after changes
- Add scripts/README.md: Complete documentation for testing scripts
- Include health checks, text generation, performance testing, and API integration
- Add retry mechanisms and proper error handling
- Support for different models and ports
- Comprehensive logging and colored output
- Ready for CI/CD integration
Usage:
- Quick test: ./scripts/quick_test.sh [port]
- Full test: ./scripts/test_serving.sh [model_path] [port]
- Move vllm_omni/core/omni_llm.py to vllm_omni/entrypoints/omni_llm.py
- Update all import statements across the codebase to reflect new location
- Fix relative imports within the moved file
- Maintain functionality while improving code organization
- All imports and functionality tested and working correctly
This change better reflects that OmniLLM and AsyncOmniLLM are the main entry points
for vLLM-omni functionality, rather than core implementation details.
- Improve API server with better error handling and response formatting
- Enhance CLI with additional options for DiT stages and configuration
- Add comprehensive examples in examples/basic/ including:
- API client with health checks and text generation
- Docker setup and usage examples
- Simple usage patterns for different scenarios
- Add utility scripts for model downloading and Docker setup
- Update documentation with implementation details and testing guidelines
- Fix configuration validation issues in OmniLLM
- Improve stage configuration handling for AR and DiT stages
- Add proper error handling and fallback mechanisms
Tested with Qwen3-0.6B model:
- Server starts successfully on port 8000
- Health and info endpoints working correctly
- Text generation with various parameters functioning
- API client examples working as expected
- CLI help and configuration options working properly
- Add comprehensive PRD, architecture design, and test design documents
- Implement core modules: OmniLLM, AsyncOmniLLM, stage configurations
- Add DiT scheduler and cache manager for diffusion models
- Implement CLI integration with --omni flag support
- Add API server and plugin system for vLLM integration
- Create comprehensive test suite with fixtures
- Update dependencies to vLLM 0.10.2 and PyTorch 2.8.0
- Add conda environment setup and package installation
- Implement stage-based processing architecture
- Add multimodal output processing capabilities
This commit establishes the foundation for multi-modality models
inference and serving with non-autoregressive structures.
Hongsheng Liu