Add OmniMRotaryEmbedding for multimodal position computation
- Introduced OmniMRotaryEmbedding to extend MRotaryEmbedding with methods for computing input positions across various multimodal scenarios, including image, video, and audio inputs. - Updated patch.py to replace the original MRotaryEmbedding with OmniMRotaryEmbedding in the module patching logic. - Created new files for the OmniMRotaryEmbedding implementation and its associated methods. Signed-off-by: [Your Name] <[Your Email]> Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
This commit is contained in:
1 parent
3208afd82b
commit
7aea8e8d91
17 files changed
+744
-57
No files matched your search
@@ -609,7 +609,8 @@ class Qwen3OmniMoeConditionalGenerationMixin(Qwen2_5OmniConditionalGenerationMix
|
||||
feature_lens=audio_feature_lengths,
|
||||
aftercnn_lens=audio_output_lengths,
|
||||
)
|
||||
audio_features = audio_outputs.last_hidden_state
|
||||
# OMNI: audio_tower.forward() returns hidden_states tensor directly
|
||||
audio_features = audio_outputs
|
||||
return audio_features.split(audio_output_lengths.tolist())
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user