Add OmniMRotaryEmbedding for multimodal position computation
- Introduced OmniMRotaryEmbedding to extend MRotaryEmbedding with methods for computing input positions across various multimodal scenarios, including image, video, and audio inputs. - Updated patch.py to replace the original MRotaryEmbedding with OmniMRotaryEmbedding in the module patching logic. - Created new files for the OmniMRotaryEmbedding implementation and its associated methods. Signed-off-by: [Your Name] <[Your Email]> Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
This commit is contained in:
1 parent
3208afd82b
commit
7aea8e8d91
17 files changed
+744
-57
No files matched your search
@@ -105,6 +105,9 @@ class OmniEmbedsPrompt(EmbedsPrompt):
|
||||
# Must ensure that all additional prompt types are inherited from vLLM prompt types
|
||||
# Because TypedDict doesn't support isinstance and are dict. Cannot distinguish them in runtime.
|
||||
# Inheritance ensure that there are only additional fields but not removing fields--safe to route to LLM.generate()
|
||||
OmniSingletonPrompt: TypeAlias = str | OmniTextPrompt | OmniTokensPrompt | OmniEmbedsPrompt
|
||||
"""Omni singleton prompt type extending vLLM's SingletonPrompt with additional fields."""
|
||||
|
||||
OmniPromptType: TypeAlias = PromptType | OmniTextPrompt | OmniTokensPrompt | OmniEmbedsPrompt
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user