[Bugfix] Qwen3-Omni thinker: create deepstack buffers on the real device (#8461)

Signed-off-by: John Nunez <johnnynunez@users.noreply.github.com>
Co-authored-by: John Nunez <johnnynunez@users.noreply.github.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
This commit is contained in:
authored and GitHub committed 2026-10-05 21:00:04 +08:00
1 parent 091b256674
commit 5351cd6645
2 files changed
+96

No files matched your search

@@ -1143,10 +1143,17 @@ class Qwen3OmniMoeThinkerForConditionalGeneration(
# register buffer for deepstack
if self.use_deepstack:
# Create the buffers on the target device explicitly. They are
# built inside the `_mark_tower_model` block, which runs under
# vLLM's meta-device `no_init_weights` context when the image and
# video limits are both 0 (audio-only serving), so a bare
# torch.zeros(...) would stay on meta and the first profile_run
# fails with "Tensor on device meta is not on the expected device".
self.deepstack_input_embeds = [
torch.zeros(
vllm_config.scheduler_config.max_num_batched_tokens,
thinker_config.text_config.hidden_size,
device=vllm_config.device_config.device,
)
for _ in range(self.deepstack_num_level)
]