- Updated OmniARScheduler to wrap output in OmniSchedulerOutput, including transfer metadata for finished requests needing KV cache transfer.
- Enhanced OmniGenerationScheduler to handle routed experts and finish reasons more effectively, ensuring proper output structure.
- Introduced performance statistics handling in both schedulers for improved monitoring.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Introduced `DiffusionScheduler` to handle scheduling for diffusion models, allowing immediate request completion.
- Added `OmniNewRequestData` to encapsulate additional request data, including prompt embeddings and extra information.
- Updated `OmniScheduler` to ensure compatibility with new request data structures.
- Created `output.py` to define new data classes for enriched request handling.
- Enhanced `__init__.py` in the engine module to support new payload structures for prompt embeddings and additional information.
tzhouam