tzhouam
034d579d77
Update vLLM precompiled wheel version and refactor various components for improved functionality and error handling. Changes include modifying the get_name method in DiffusionQuantizationConfig to use class-level attributes, enhancing metadata handling in StageEngineCoreClient, and adjusting the AsyncOmni class constructor for better parameter management. Additionally, improved error handling in omni_snapshot_download and optimized initial chunk size computation in talker2code2wav_async_chunk for dynamic load adaptation.
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-03-19 05:14:34 +00:00
Taichang Zhou
dc6f2fb81f
Enhance cu_seqlens computation in Qwen3Omni_VisionTransformer by switching to tensor operations for improved performance and compatibility. Added error handling to fallback on a vectorized implementation if necessary. This update aligns with upstream changes and optimizes the handling of sequence lengths during attention execution.
2026-03-18 23:27:55 +08:00
Taichang Zhou
7a68bd37b7
Refactor cu_seqlens handling in Qwen3Omni_VisionTransformer to align with upstream vLLM changes. The sequence length helper now retains cu_seqlens in numpy format and converts it for attention execution, improving compatibility and performance.
2026-03-18 23:23:41 +08:00
Taichang Zhou
8a31e7865d
Increase timeout for Diffusion Model Test and Diffusion Images API LoRA E2E in Buildkite configurations to 30 minutes. Also, introduce a new timeout constant for diffusion initialization in relevant test files to enhance stability during testing. Update OmniServer initialization to utilize the new timeout settings.
2026-03-17 23:43:46 +08:00
tzhouam
722ad6bbae
rebase: align vllm-omni with vLLM 0a0a1a198be8
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-03-17 03:26:42 +00:00
tzhouam
a26e568dfc
Merge remote-tracking branch 'origin/main' into dev/vllm-align
2026-03-16 07:30:43 +00:00
Canlin Guo
9718a9e369
[NPU] Upgrade to v0.17.0 ( #1890 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
2026-03-16 14:42:59 +08:00
tzhouam
3a508d7b7a
rebase: align vllm-omni with vLLM e9163b536e72
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-03-16 02:32:39 +00:00
tzhouam
e17fb0e97e
Refactor OpenAI serving components and enhance multimodal handling
...
- Updated import for OpenAIServingEmbedding in api_server.py.
- Enhanced omni_init_app_state to initialize renderer for engine_client.
- Added handle_oov_mm_token parameter to multiple model classes for better multimodal token handling.
- Improved comments for clarity in various model files.
- Adjusted GPUARModelRunner to ensure proper handling of late interaction runner attributes.
Signed-off-by: [Your Name] <[Your Email]>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-03-13 01:55:17 +00:00
tzhouam
11b6c16007
rebase: align vllm-omni with vLLM 84e436ed1c94
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-03-11 19:24:44 +00:00
tzhouam
08a2673c66
rebase: align vllm-omni with vLLM 84e436ed1c94
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-03-11 07:29:52 +00:00
Junhong Liu
2ee4a07adb
[Refactor][Perf] Qwen3-omni: code predictor with re-prefill + SDPA and eliminate decode hot-path CPU round-trips ( #1758 )
...
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com >
2026-03-11 07:52:21 +08:00
89b527e533
Dev/rebase v0170 ( #1639 )
...
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk >
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Signed-off-by: princepride <wangzhipeng628@gmail.com >
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk >
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-03-09 17:29:38 +08:00
Ziming Huang
070ea0ddc3
[Optimize][Qwen3-Omni] Reduce inter-packet latency in async chunk ( #1656 )
...
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com >
2026-03-05 22:26:30 +08:00
Junhong Liu
742d845b8e
Make chunk_size and left_context_size configurable via YAML for async chunking ( #1423 )
...
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com >
Signed-off-by: Junhong Liu <ljh_lbj@163.com >
2026-03-03 22:06:52 +08:00
dsinghvi and Zhou Taichang
68764cc4f4
[Refactor]: Phase1 for rebasing_additional_info ( #1394 )
...
Signed-off-by: dsinghvi <divyanshsinghvi@gmail.com >
Signed-off-by: Divyansh Singhvi <divyanshsinghvi@gmail.com >
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk >
Co-authored-by: Zhou Taichang <tzhouam@connect.ust.hk >
2026-03-02 10:19:41 +08:00
Zhou Taichang
fe0c02ee78
[Debug] Merge vllm pull 35368 ( #1534 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-02-28 10:39:25 +08:00
e1337c89c7
[BugFix] Restore talker's config ( #1524 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Canlin Guo <961750412@qq.com >
2026-02-27 06:44:11 +08:00
amy-why-3459
66c3e04263
[Performance]Qwen3-Omni performance optimization ( #1378 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-02-26 21:21:36 +08:00
Canlin Guo
22c8cde328
[NPU] Upgrade to v0.16.0 ( #1375 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
2026-02-26 14:26:50 +08:00
Zhou Taichang
8e950daf6a
[Debug] Multi-Request for Qwen 3 Omni use_audio_in_video ( #1433 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-02-26 09:26:25 +08:00
Canlin Guo and Hongsheng Liu
af11b02081
[Minor][Refactor] Pass seq_token_counts explicitly ( #1425 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-02-24 14:10:00 +08:00
75770c9309
[Rebase] Rebase to vllm v0.16.0 ( #1357 )
...
ComfyUI Integration -- Publish to Comfy registry / Publish Custom Node to registry (push) Failing after 44s
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk >
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Signed-off-by: princepride <wangzhipeng628@gmail.com >
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: ZJY0516 <zhu.jiangyun@foxmail.com >
Co-authored-by: youkaichao <youkaichao@gmail.com >
2026-02-13 18:05:07 +08:00
0b56fba7b3
[Pref] Support Qwen3 Omni code2wav batch infernce with async chunk ( #1246 )
...
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com >
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Signed-off-by: Ziming Huang <1520787127@qq.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2026-02-12 11:50:10 +08:00
rein yang
963b64e7ae
[Bugfix] fix precision issues of qwen3-omni when enable async_chunk without system prompt ( #1288 )
...
Signed-off-by: Rein Yang <ruiruyang2@gmail.com >
2026-02-11 10:03:41 +08:00
Canlin Guo
285d71b26a
[Refactor] Reuse upstream Qwen3MoeSparseMoeBlock ( #1202 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
2026-02-06 01:06:50 +00:00
Zhou Taichang
f1960d8dfd
[Feature] Enable use_audio_in_video for Qwen 3 Omni Online ( #1198 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-02-04 20:48:25 +08:00
amy-why-3459
5304bc1e16
[Refactor] Refactor async chunk and fix the shape mismatch issue ( #1151 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-02-04 00:00:56 +08:00
Zhou Taichang and 汪志鹏
70ef548448
Dev/rebase v0.15.0 ( #1159 )
...
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk >
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Signed-off-by: princepride <wangzhipeng628@gmail.com >
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com >
2026-02-03 12:24:45 +08:00
f6cfc0d1e3
[BugFix] Fix Qwen3 Omni talker mtp torch.compile startup error ( #1104 )
...
Signed-off-by: ram16g <anlianfengjie@163.com >
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com >
Co-authored-by: ram16g <anlianfengjie@163.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-01-30 23:25:49 +08:00
Ziming Huang and Hongsheng Liu
a909d36415
[Perf] Qwen3 Omni talker mtp optimization ( #1005 )
...
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com >
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-01-29 21:23:19 +08:00
amy-why-3459 and Rein Yang
dd518973de
[BugFix] Fix the accuracy issue of multimodal input. ( #1020 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
Co-authored-by: Rein Yang <ruiruyang2@gmail.com >
2026-01-29 19:30:11 +08:00
Canlin Guo
c9392dff01
[Hardware] Support platforms and plugin system ( #774 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
2026-01-28 22:12:52 +08:00
amy-why-3459
d2b3a24af3
[BugFix] Modify the method of obtaining external_request_id ( #961 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-01-27 20:56:02 +08:00
Canlin Guo
cadd77262d
[NPU] Upgrade to v0.14.0 ( #820 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
2026-01-27 16:45:32 +08:00
52bf1e2f40
[Feature]Support async computation and communication across stages by chunks ( #727 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
Signed-off-by: Rein Yang <ruiruyang2@gmail.com >
Co-authored-by: Rein Yang <ruiruyang2@gmail.com >
Co-authored-by: CHEN <116010019@link.cuhk.edu.cn >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-01-26 11:21:24 +08:00
+13
776c3a794e
Dev/rebase 0.14.0 ( #847 )
...
Signed-off-by: JaredforReal <w13431838023@gmail.com >
Signed-off-by: root <root@hk01dgx039.cm.cluster >
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk >
Signed-off-by: TangPeng <85704592@qq.com >
Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com >
Signed-off-by: mxuax <mxuax@connect.ust.hk >
Signed-off-by: Sihyeon Jang <sihyeon.jang@navercorp.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: iwzbi <wzbi@zju.edu.cn >
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com >
Signed-off-by: Yuhan Liu <30294295+liuyuhanalex@users.noreply.github.com >
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com >
Signed-off-by: samithuang <285365963@qq.com >
Signed-off-by: linyueqian <linyueqian@outlook.com >
Signed-off-by: David Chen <530634352@qq.com >
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com >
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com >
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Signed-off-by: wangyu31577 <wangyu31577@hundsun.com >
Signed-off-by: princepride <wangzhipeng628@gmail.com >
Signed-off-by: John Liu BUAA <liukecheng97@gmail.com >
Signed-off-by: Dinesh G <G.Dinesh@ibm.com >
Signed-off-by: gDINESH13 <dinesh13g@gmail.com >
Signed-off-by: root <root@hk01dgx028.cm.cluster >
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk >
Co-authored-by: JaredforReal <w13431838023@gmail.com >
Co-authored-by: root <root@hk01dgx039.cm.cluster >
Co-authored-by: JustQJ <37905360+JustQJ@users.noreply.github.com >
Co-authored-by: XU Mingshi <91017482+mxuax@users.noreply.github.com >
Co-authored-by: Sihyeon Jang <uneedsihyeon@gmail.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
Co-authored-by: catcat <108673086+iwzbi@users.noreply.github.com >
Co-authored-by: Ziming Huang <hzm414167@alibaba-inc.com >
Co-authored-by: Yuhan Liu <30294295+liuyuhanalex@users.noreply.github.com >
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com >
Co-authored-by: Samit <285365963@qq.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com >
Co-authored-by: Peiqi Yin <60515999+yinpeiqi@users.noreply.github.com >
Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com >
Co-authored-by: wangyu31577 <wangyu31577@hundsun.com >
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com >
Co-authored-by: John Liu BUAA <liukecheng97@gmail.com >
Co-authored-by: D!NE$H <67671800+gDINESH13@users.noreply.github.com >
Co-authored-by: root <root@hk01dgx028.cm.cluster >
2026-01-21 00:25:25 +08:00
Ziming Huang
1444e1fc45
[Feature] Support Qwen3 Omni talker mtp batch inference ( #722 )
...
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com >
2026-01-14 19:07:31 +08:00
Zhou Taichang and root
2e13bddf87
[Debug] Clean code in Qwen 3 Omni and add warning for talker temperature. ( #688 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Co-authored-by: root <root@hk01dgx028.cm.cluster >
2026-01-12 23:13:45 +13:00
Junhong Liu
d0836d8459
[Bugfix] Fix multi-audio input shape alignment for Qwen3-Omni Thinker ( #697 )
...
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com >
2026-01-12 13:16:29 +08:00
Canlin Guo
dde96c2468
[Perf][Qwen3-Omni] Fuse QKV linear and gate_up proj ( #734 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
2026-01-11 11:06:24 +08:00
Canlin Guo and Isotr0py
9c2f7467a0
[Perf] Use vLLM's SharedFusedMoE in Qwen3-Omni ( #560 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-01-10 04:30:06 +00:00
Ziming Huang
978e3ad7d4
[Feature] Support Qwen3 Omni talker cudagraph ( #669 )
...
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com >
2026-01-09 11:45:58 +08:00
Ziming Huang
cddcc400d3
[BugFix] Fix stage engine outputs mismatch bug in online batching ( #691 )
...
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com >
2026-01-09 04:01:57 +13:00
Gao Han
9c7de5373a
[Feature]Basic version of supporting streaming output ( #367 )
...
Signed-off-by: Gaohan123 <hgaoaf@connect.ust.hk >
2026-01-01 03:02:04 +13:00
Ziming Huang and Hongsheng Liu
1ae0c64fe0
Support Qwen3 Omni thinker cuda graph ( #523 )
...
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2025-12-30 23:46:13 +13:00
Zhou Taichang
59ae174bba
[Debug] Debug qwen3 mix modality output empty string ( #431 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2025-12-30 15:32:15 +08:00
Canlin Guo and Hongsheng Liu
52d15c4c04
[NPU][Model] Support Qwen3-Omni on NPU ( #484 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2025-12-27 00:05:55 +13:00
Peiqi Yin
e9e6b3711c
[Core] customize process between stages, remove the need of "request_ids" in model forward, rebase to 0.12.0 ( #458 )
...
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com >
2025-12-24 13:26:33 +00:00
8939eb835d
[Rebase] Rebase to vllm 0.12.0 ( #335 )
...
Signed-off-by: Rein Yang <ruiruyang2@gmail.com >
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Rein Yang <ruiruyang2@gmail.com >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2025-12-19 16:42:02 +08:00