[Bugfix] Qwen3-Omni thinker: create deepstack buffers on the real device (#8461)
Signed-off-by: John Nunez <johnnynunez@users.noreply.github.com> Co-authored-by: John Nunez <johnnynunez@users.noreply.github.com> Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
This commit is contained in:
2 files changed
+96
No files matched your search
@@ -1143,10 +1143,17 @@ class Qwen3OmniMoeThinkerForConditionalGeneration(
|
||||
|
||||
# register buffer for deepstack
|
||||
if self.use_deepstack:
|
||||
# Create the buffers on the target device explicitly. They are
|
||||
# built inside the `_mark_tower_model` block, which runs under
|
||||
# vLLM's meta-device `no_init_weights` context when the image and
|
||||
# video limits are both 0 (audio-only serving), so a bare
|
||||
# torch.zeros(...) would stay on meta and the first profile_run
|
||||
# fails with "Tensor on device meta is not on the expected device".
|
||||
self.deepstack_input_embeds = [
|
||||
torch.zeros(
|
||||
vllm_config.scheduler_config.max_num_batched_tokens,
|
||||
thinker_config.text_config.hidden_size,
|
||||
device=vllm_config.device_config.device,
|
||||
)
|
||||
for _ in range(self.deepstack_num_level)
|
||||
]
|
||||
|
||||
Reference in new issue
Block a user