- Introduced OmniMRotaryEmbedding to extend MRotaryEmbedding with methods for computing input positions across various multimodal scenarios, including image, video, and audio inputs.
- Updated patch.py to replace the original MRotaryEmbedding with OmniMRotaryEmbedding in the module patching logic.
- Created new files for the OmniMRotaryEmbedding implementation and its associated methods.
Signed-off-by: [Your Name] <[Your Email]>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Changed the multimodal_output field to a private attribute _multimodal_output for better encapsulation.
- Added a property method to access multimodal_output, allowing for a clearer interface and improved handling of request outputs.
Signed-off-by: [Your Name] <[Your Email]>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated the audio tensor extraction logic to access the correct output structure in Qwen2.5 Omni, Qwen3 Omni, and Qwen3 TTS implementations.
- Ensured consistency in handling audio outputs for improved functionality.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Eliminated the _process_embeds method and related imports from preprocess.py, streamlining the input preprocessing workflow.
- Updated the main processing logic to focus solely on token inputs, enhancing clarity and maintainability.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Reintroduced the load_chat_template import in api_server.py for better template handling.
- Updated the logic in serving_chat.py to streamline the resolution of chat template content formats, enhancing clarity and functionality.
- Adjusted the application of chat templates to ensure proper handling of tools and chat templates.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Changed the import path for multimodal processing to improve module organization.
- Removed the unused multimodal configuration parameter from the Qwen3OmniMoeThinker initialization, streamlining the class definition.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Removed the unused import of supports_mm_encoder_only from interfaces.
- Updated the dummy run check to utilize the multimodal configuration directly, enhancing clarity and future extensibility.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated import statements in chat_utils.py to replace the internal content format type with the public one for better clarity.
- Refactored api_server.py to streamline the chat template loading process by replacing the previous method with a direct function call.
- Enhanced serving_chat.py by reintroducing safe_apply_chat_template functions for better template handling.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Modified the pip install command for vllm to use the correct URL format for direct installation from the specified commit hash, ensuring a more reliable installation process.
- Removed the fallback option for pre-release installation to streamline the installation command.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated the import path for AnthropicServingMessages in api_server.py to enhance clarity and maintain consistency with the new module structure.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Changed the pip install command for vllm to download the wheel file directly instead of using an extra index URL, ensuring a more reliable installation process.
- Added a fallback option to install the pre-release version of vllm if the direct installation fails.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Refactored import statements in api_server.py to enhance module organization, including changes to the routing and server setup.
- Updated the method for removing routes to ensure compatibility with the new app structure.
- Adjusted the handling of chat completion requests to improve error responses and streamline processing.
- Modified serving modules to align with the new import paths and enhance code clarity.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Modified the pip install command for vllm to include --upgrade, --force-reinstall flags, and an extra index URL, ensuring the latest version is installed from the specified source during testing.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Modified the pip install command for vllm to include --upgrade and --force-reinstall flags, ensuring the latest version is installed during the CI process.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Added a warning in GPUARWorker and GPUGenerationWorker to indicate that the v2 model runner does not include omni hooks, forcing the use of the v1 model runner.
- Updated import statements in gpu_ar_model_runner.py and gpu_generation_model_runner.py to include necessary functions from parallel_state.
- Removed unused imports from both model runner files to streamline the code.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Deleted the MRotaryEmbedding class and its associated methods from mrope.py to streamline the codebase.
- Removed the get_llm_pos_ids_for_vision function from vision.py as it is no longer needed.
- Updated import statements in qwen2_5_omni.py to reflect the removal of the vision module.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated OmniInputProcessor to streamline input processing, including enhanced handling of prompt embeddings and additional information.
- Introduced OmniTextPrompt and OmniEmbedsInputs to support new input structures with optional metadata.
- Modified input preprocessing methods to accommodate new prompt types and ensure compatibility with multimodal inputs.
- Improved error handling and type checking for data parallel configurations.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Changed type hint for stage_connector_spec from 'any' to 'Any' for improved type checking and consistency.
- Ensured compatibility with typing standards in the arg_utils module.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Updated OmniARScheduler to wrap output in OmniSchedulerOutput, including transfer metadata for finished requests needing KV cache transfer.
- Enhanced OmniGenerationScheduler to handle routed experts and finish reasons more effectively, ensuring proper output structure.
- Introduced performance statistics handling in both schedulers for improved monitoring.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Introduced mm_encoder_only parameter to OmniModelConfig.
- Added a deprecation warning for the `--convert mm_encoder_only` option, advising users to switch to `--mm-encoder-only`.
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
- Modified the constructor to accept torch.Tensor as an option for prompt_embeds.
- Introduced a new static method _maybe_decode_prompt_embeds to handle the conversion from PromptEmbedsPayload to torch.Tensor.
- Preserved the original prompt_embeds payload for reference.
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
tzhouam