anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
873 Commits 235 Branches 31 Tags
3e539fcb6a017fa9d8ac86f4b8d6eb852ff0dde4
Commit Graph
68 Commits
Author SHA1 Message Date
tzhouam 26744bcb35 Merge origin/main into dev/vllm-align 
Resolved conflicts:
- vllm_omni/inputs/preprocess.py: took origin/main's elif mm_processor_kwargs branch, fixed undefined mm_uuids reference
- vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_code_predictor_vllm.py: took origin/main's NPU support

Made-with: Cursor

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-03-02 02:10:32 +00:00
WeiQing ChenIsotr0pyIsotr0pyHongsheng Liu
 		0b884ebc6c [Feature]: Native GGUF Quantization Support for DiT (#1285) 
Signed-off-by: David Chen <530634352@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-02-27 11:39:14 +08:00
tzhouam a252918008 [Update] Upgrade vLLM wheel version and enhance metrics handling 
- Updated the vLLM wheel to commit 24650715105a26b41c88f4164777b5e1a0f3200b in Dockerfile.ci.
- Modified the `calculate_metrics` function to handle `tokenizer` as optional (can be None) in metrics.py.
- Adjusted the `benchmark` function to accept `tokenizer` as optional in patch.py.
- Enhanced condition checks in OmniARScheduler to ensure valid token generation in omni_ar_scheduler.py.
- Updated input processing to include supported tasks in input_processor.py.
- Cleaned up unnecessary code in output_processor.py.
- Added support for SSL ciphers in the API server configuration in api_server.py.
- Enabled speculative decoding in GPU model runners for improved performance in gpu_generation_model_runner.py and gpu_model_runner.py.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-25 10:02:23 +00:00
tzhouam 861698f88f Update Dockerfile for vllm wheel version and enhance SHM connector tests 
- Updated Dockerfile to use a new vllm wheel version.
- Modified test_put_get_inline to clarify that it always uses SHM and updated metadata assertions.
- Added handling for speculative configuration in GPU model runners to delay clearing connector metadata.
- Refactored payload decoding in OmniGPUModelRunner for better clarity and maintainability.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-24 05:45:48 +00:00
tzhouam bf69cd94bb Rebase 2026/02/23 
Update Dockerfile for vllm wheel version and enhance input processing in Omni models. Added decoding for byte messages in async requests, included new metrics in benchmarks, and improved pause state handling in the scheduler. Refined input and output processing methods for better multimodal support and error handling.

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-23 03:36:01 +00:00
tzhouam 9869ea007e Update vLLM wheel version in Dockerfile and enhance input processing for omni models. Added functionality to reinject omni-specific fields into processed prompts and improved handling of multimodal audio extraction in serving_chat.py. Adjusted SharedMemoryConnector logic for payload size checks and refined input preprocessing methods for better compatibility with multimodal data. 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-20 08:34:53 +00:00
tzhouam b867bacd24 rebase 2025/02/16 
Qwen3 Omni offline/online passed

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-16 03:21:21 +00:00
Zhou Taichang汪志鹏Hongsheng LiuIsotr0pyZJY0516youkaichao
 		75770c9309 [Rebase] Rebase to vllm v0.16.0 (#1357)
ComfyUI Integration -- Publish to Comfy registry / Publish Custom Node to registry (push) Failing after 44s
Details 
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: ZJY0516 <zhu.jiangyun@foxmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2026-02-13 18:05:07 +08:00
Taichang Zhou 85a7b558ab rebase to v0.16.0 
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
 2026-02-12 21:30:54 +08:00
Taichang Zhou 7458ba44b7 merge main 
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
 2026-02-12 21:21:03 +08:00
Taichang ZhouandCursor e2ced80d59 Merge origin/main into dev/rebase-v0.16.0. 
Resolve arg_utils merge conflict and align CI base image tag to v0.16.0 for the vLLM rebase track.

Co-authored-by: Cursor <cursoragent@cursor.com>

Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
 2026-02-12 21:18:03 +08:00
ChenWenjingBhanu068Gao Han
 		ab1acb27ff [Feature]async scheduling to overlap chunk IO and compute (#951) 
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
Co-authored-by: Bhanu068 <voutharoja.bhanu06@gmail.com>
Co-authored-by: Gao Han <gaohan19@huawei.com>
 2026-02-12 16:05:02 +08:00
tzhouam 0099eebb3f Refactor OmniEngineArgs and AsyncOmniEngineArgs to improve compatibility with partial payloads by using getattr for various attributes. Update GPUGenerationModelRunner and OmniGPUModelRunner to replace the activate_lora parameter with num_active_loras for better clarity and backward compatibility. This change enhances the handling of LoRA activation during model runs. 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-12 03:33:50 +00:00
tzhouam 3663e803d1 rebase 2026/02/11 
vllm: 4a1550d22d7058e129d0e1257e726b3bf4a77025

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-11 02:43:49 +00:00
tzhouam 279b6b5050 Merge remote-tracking branch 'origin/main' into dev/vllm-align 2026-02-11 01:44:54 +00:00
dongbo910220andhsliuustc0106 43e8747c00 [Feat]: support VAE patch parallelism (#756) 
Signed-off-by: dongbo910220 <1275604947@qq.com>
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-02-10 17:36:10 +08:00
tzhouam 58d09ed656 reabse 2026/02/10 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-10 03:36:09 +00:00
tzhouam 7e4a604d4d reabse 2025/2/9 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-09 03:07:43 +00:00
tzhouam 24b04f58a0 rebase 2026/2/5 2026-02-05 10:52:28 +00:00
Zhou Taichangand汪志鹏 70ef548448 Dev/rebase v0.15.0 (#1159) 
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
 2026-02-03 12:24:45 +08:00
tzhouam 6568a78a26 Remove unused tokenizer argument from OmniInputProcessor class docstring 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-02-02 03:12:43 +00:00
tzhouam 4b895459e4 Merge branch 'main' into dev/vllm-align 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-01-30 02:53:03 +00:00
amy-why-3459andRein Yang dd518973de [BugFix] Fix the accuracy issue of multimodal input. (#1020) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: Rein Yang <ruiruyang2@gmail.com>
 2026-01-29 19:30:11 +08:00
Canlin Guo c9392dff01 [Hardware] Support platforms and plugin system (#774) 
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
 2026-01-28 22:12:52 +08:00
tzhouam 52d0122fb5 [Enhancement] Refactor OmniInputProcessor and related classes for improved prompt handling 
- Updated OmniInputProcessor to streamline input processing, including enhanced handling of prompt embeddings and additional information.
- Introduced OmniTextPrompt and OmniEmbedsInputs to support new input structures with optional metadata.
- Modified input preprocessing methods to accommodate new prompt types and ensure compatibility with multimodal inputs.
- Improved error handling and type checking for data parallel configurations.

Signed-off-by: [Your Name] <your.email@example.com>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-01-28 15:42:05 +08:00
tzhouam b26a9735e6 [Enhancement] Update type hint for stage_connector_spec in OmniEngineArgs and AsyncOmniEngineArgs 
- Changed type hint for stage_connector_spec from 'any' to 'Any' for improved type checking and consistency.
- Ensured compatibility with typing standards in the arg_utils module.

Signed-off-by: [Your Name] <your.email@example.com>

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-01-28 15:41:20 +08:00
Andy Zhoudongbo910220Samit
 		5037af1033 [Feature] Diffusion LoRA Adapter Support (PEFT compatible) for vLLM alignment (#758) 
Signed-off-by: AndyZhou952 <jzhoubc@connect.ust.hk>
Signed-off-by: Andy Zhou <jzhoubc@connect.ust.hk>
Signed-off-by: dongbo910220 <1275604947@qq.com>
Signed-off-by: Andy Zhou <46011930+AndyZhou952@users.noreply.github.com>
Signed-off-by: Samit <285365963@qq.com>
Co-authored-by: dongbo910220 <1275604947@qq.com>
Co-authored-by: Samit <285365963@qq.com>
 2026-01-27 10:22:40 +08:00
amy-why-3459Rein YangCHENHongsheng Liu
 		52bf1e2f40 [Feature]Support async computation and communication across stages by chunks (#727) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: Rein Yang <ruiruyang2@gmail.com>
Co-authored-by: Rein Yang <ruiruyang2@gmail.com>
Co-authored-by: CHEN <116010019@link.cuhk.edu.cn>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-01-26 11:21:24 +08:00
汪志鹏andwzliu 7f821be57b Support Bagel Model (#726) 
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Co-authored-by: wzliu <wzliu@connect.hku.hk>
 2026-01-22 23:09:55 +13:00
Gao Han a54c323952 [Model] Support Qwen3-TTS model series (#895) 
Signed-off-by: Gaohan123 <hgaoaf@connect.ust.hk>
 2026-01-22 16:57:37 +08:00
+13
Zhou TaichangJaredforRealrootJustQJXU MingshiSihyeon JangJiangyun ZhucatcatZiming HuangYuhan LiuAliciaSamitYueqian LinWeiQing ChenPeiqi YinDidan DengHongsheng Liuwangyuwangyu31577汪志鹏John Liu BUAAD!NE$Hroot
 		776c3a794e Dev/rebase 0.14.0 (#847) 
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: root <root@hk01dgx039.cm.cluster>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
Signed-off-by: TangPeng <85704592@qq.com>
Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: Sihyeon Jang <sihyeon.jang@navercorp.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: iwzbi <wzbi@zju.edu.cn>
Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com>
Signed-off-by: Yuhan Liu <30294295+liuyuhanalex@users.noreply.github.com>
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
Signed-off-by: samithuang <285365963@qq.com>
Signed-off-by: linyueqian <linyueqian@outlook.com>
Signed-off-by: David Chen <530634352@qq.com>
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Signed-off-by: John Liu BUAA <liukecheng97@gmail.com>
Signed-off-by: Dinesh G <G.Dinesh@ibm.com>
Signed-off-by: gDINESH13 <dinesh13g@gmail.com>
Signed-off-by: root <root@hk01dgx028.cm.cluster>
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk>
Co-authored-by: JaredforReal <w13431838023@gmail.com>
Co-authored-by: root <root@hk01dgx039.cm.cluster>
Co-authored-by: JustQJ <37905360+JustQJ@users.noreply.github.com>
Co-authored-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
Co-authored-by: Sihyeon Jang <uneedsihyeon@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
Co-authored-by: catcat <108673086+iwzbi@users.noreply.github.com>
Co-authored-by: Ziming Huang <hzm414167@alibaba-inc.com>
Co-authored-by: Yuhan Liu <30294295+liuyuhanalex@users.noreply.github.com>
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>
Co-authored-by: Samit <285365963@qq.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Co-authored-by: Peiqi Yin <60515999+yinpeiqi@users.noreply.github.com>
Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Co-authored-by: wangyu31577 <wangyu31577@hundsun.com>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: John Liu BUAA <liukecheng97@gmail.com>
Co-authored-by: D!NE$H <67671800+gDINESH13@users.noreply.github.com>
Co-authored-by: root <root@hk01dgx028.cm.cluster>
 2026-01-21 00:25:25 +08:00
Chenguang ZhengyinpeiqiHongsheng Liu
 		595e7c0032 [Core] Supports stage abstraction in the diffusion model (#391) 
Signed-off-by: Chenguang ZHENG <645327136@qq.com>
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>
Co-authored-by: yinpeiqi <yinpeiqi809@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2025-12-27 01:59:33 +13:00
wuhangandHongsheng Liu b3d212e985 [Perf] Optimise tensor concat perf in output processor (#467) 
Signed-off-by: wuhang <wuhang6@huawei.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2025-12-25 15:54:41 +00:00
dsinghvi b06a5148f6 bug resolution for omni models registration not present in vllm (#397) 
Signed-off-by: Divyansh Singhvi <divyanshsinghvi@gmail.com>
 2025-12-22 15:50:26 +08:00
Zhou TaichangRein YangIsotr0py
 		8939eb835d [Rebase] Rebase to vllm 0.12.0 (#335) 
Signed-off-by: Rein Yang <ruiruyang2@gmail.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Rein Yang <ruiruyang2@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-12-19 16:42:02 +08:00
Cyrus Leung c091a03d82 [Chore] Bump Python version and apply formatting changes for v0.12 release (#126) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-16 15:19:40 +08:00
Baoyuan Qi cfe7ad47c1 Fix: Safe handling for multimodal_config to avoid 'NoneType' object h… (#227) 
Signed-off-by: Baoyuan Qi <qibaoyuan@126.com>
 2025-12-07 11:40:17 +08:00
Canlin GuoIsotr0pyGao Han
 		f75c21bcd0 [E2E] Add Qwen2.5-Omni model test with OmniRunner (#168) 
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Gao Han <gaohan19@huawei.com>
 2025-12-06 06:20:26 +00:00
R2-YandRein 0a4e5a8710 [Feature] add support for Qwen3-omni (#55) 
Signed-off-by: Rein Yang <ruiruyang2@gmail.com>
Co-authored-by: Rein <rein@Reins-MacBook-Air.local>
 2025-11-30 04:44:01 +08:00
Roger WangandGao Han a5800cd2c2 [Doc] Misc documentation polishing (#98) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: Gao Han <gaohan19@huawei.com>
Co-authored-by: Gao Han <gaohan19@huawei.com>
 2025-11-29 20:42:39 +08:00
Gao Han db81ee9f69 [Docs] optimize and supplement docs system (#86) 
Signed-off-by: Gaohan123 <gaohan19@huawei.com>
 2025-11-28 11:03:52 +08:00
Jiangyun Zhu 5707fc78d5 [CI] fix ci and format existing code (#71) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
 2025-11-18 20:54:26 +08:00
Gao Han c5a89302f1 [Feature] Support online inference (#64) 
Signed-off-by: Gaohan123 <gaohan19@huawei.com>
 2025-11-17 14:44:10 +08:00
Gaohan123 6e1478222e Dependency refractored to vLLM v0.11.0 
Signed-off-by: Gaohan123 <gaohan19@huawei.com>
 2025-11-04 22:06:41 +08:00
Gaohan123 6cf73c4e80 Remove unused files and fix some bugs 2025-10-25 18:02:11 +08:00
tzhouam 3b1324ea39 Merge branch 'main' into end2end_fixup 2025-10-24 17:41:03 +08:00
tzhouam 3265e857eb Refactor OmniLLM and MultimodalOutputProcessor initialization 
- Removed AsyncOmniLLM from the main entry points as it is not supported yet.
- Updated the constructor of MultimodalOutputProcessor to set a default value for engine_core_output_type.
- Adjusted output type handling in various methods to utilize the new default engine_core_output_type for consistency.
 2025-10-24 17:15:34 +08:00
Gaohan123 171f3c1ea7 fix conflicts 2025-10-24 16:49:19 +08:00
Gaohan123 a04e945166 add precommit 2025-10-24 16:20:58 +08:00
Zhou Taichang 582639c77a Merge branch 'main' into feat/hidden-state-input-request-input-processor-patch 
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk>
 2025-10-24 11:52:45 +08:00