anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
2,706 Commits 235 Branches 31 Tags
c9bb65902e4e575abde837fd7941bc9415ebb32b
Commit Graph
72 Commits
Author SHA1 Message Date
hsliuustc0106 c9bb65902e [Model] Learn the retention budget directly from observed peaks 
Drop the staged-bytes seed per review direction: the budget is now
min(observed_peak * 1.25, 25% of device capacity), nothing else. A cold
start pays one release per observation step while the peak climbs (two on
the validated MiniMax workload), then boundaries retain steadily; the 5%
free floor and capacity cap are unchanged. The staged-byte ledger remains
only to detect full detachment, which now resets the observed peak so one
workload cannot permanently influence another. Trade-off accepted per
review: lower reserved memory and a simpler policy over a flush-free
first request.

Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-08-26 13:43:00 +08:00
hsliuustc0106 27f9726a6a [Model] Derive the retention budget from observed boundary peaks 
The staged-bytes seed left the transient half of the working set to a
guessed multiplier, and that half scales with the running workload rather
than the model topology. Track the largest cached-but-unallocated value
observed at any component boundary instead: the budget is the larger of
the seed and the observed peak times a 1.25 margin, still capped at a
quarter of device capacity. A workload that outgrows the seed pays one
release while the observation lands, then retains steadily. No warmup
flag is needed and no user-chosen fraction exists.

Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-08-26 12:52:19 +08:00
hsliuustc0106 c5295f3ffe [Model] Cap derived cache budget and account for stager detach 
Address review on #6629:

- Cap the derived retention budget at a quarter of device capacity so a
  large staged total cannot monopolize a small device; the multiplier
  itself stays pending an instrumented target run.
- Log every release decision (reason, cached bytes, effective budget and
  cap, staged total, free/total memory) and every staging OOM retry with
  its staged byte total, so a single target run validates the headroom
  multiplier directly.
- Make stager detach symmetric: set_cache_retention now shrinks the
  previous cache's budget before growing the new one, so cache -> None ->
  same cache counts the staged bytes exactly once (regression test added).

Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-08-26 11:12:09 +08:00
hsliuustc0106 ba5943dddb [Model] Derive MiniMax-H3 DLO component cache budget from staged bytes 
Replace BoundedAllocatorCache's device-capacity fraction bound (25% of
total memory) with a budget derived from the model topology: each attached
PinnedModuleStager reports its unique staged storage bytes once, and the
retention ceiling is that total times a fixed 2.0 headroom multiplier
covering allocator fragmentation and transient storages freed at component
boundaries. The 5% physically-free floor is unchanged.

The budget therefore scales with the staged components instead of device
capacity, so no user-chosen fraction is needed and behavior is identical
across differently sized devices with the same model. Re-attaching the
same cache does not double count; a second stager grows the budget by its
own bytes.

Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-08-26 09:49:55 +08:00
Hongsheng Liuandhsliuustc0106 c608910d20 [Diffusion] Register HWR mmap for direct DLO H2D (#6591) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
 2026-08-25 23:30:11 +08:00
Hongsheng Liu 5a8b81ebeb [Model] Optimize MiniMax-H3 DLO component lifecycle (#6526) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-08-25 23:13:41 +08:00
Hongsheng Liuandhsliuustc0106 8360346a66 [Diffusion] Integrate no-AllGather DLO with Host Weight Runtime (#6486) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
 2026-08-25 08:03:53 +08:00
bjf-frzandHongsheng Liu 072bfc02dd [FEAT]Add FastVideo VSA backend for Wan2.2 (#4820) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-24 14:48:31 +08:00
ivanustoandHongsheng Liu f09bdcbc7d [Bugfix] Make the async-output wait bound configurable and default it higher (#6255) 
Signed-off-by: ivanusto <ivanusto@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-22 22:10:30 +08:00
Hongsheng Liu 493fc64397 [Diffusion] Add final-layout BF16 Host Weight Runtime artifacts (#6445) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-22 21:37:20 +08:00
Hongsheng Liu 15e38ac173 [Core] Add explicit post-load Host Weight Runtime publication (#6427) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-21 14:26:49 +08:00
Hongsheng Liu 73b623f2f7 [Core] Add Host Weight Runtime foundation (#6419) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-21 08:50:59 +08:00
Yukim1 6f55a06fab [Docs] Remove stale stage config references (#6270) 
Signed-off-by: zwhzzz0821 <2831474076@qq.com>
 2026-08-20 07:09:12 +08:00
Hongsheng Liu 284e05c88b [Core] Support online FP8 with DLO AllGather (#6279) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-18 23:48:07 +08:00
Hongsheng Liu 9d2bb23ff6 [Diffusion] Add loader-owned host-weight plans for DLO (TP=1) (#6213) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-16 11:02:38 +08:00
Jack Chenzhangbin.327Hongsheng Liu
 		3875f8ded4 [Core] Add realtime AR-Diffusion tick sessions for LingBot World 2.0 (#5491) 
Signed-off-by: zhangbin.327 <zhangbin.327@bytedance.com>
Signed-off-by: Jack47 <scsvip@gmail.com>
Co-authored-by: zhangbin.327 <zhangbin.327@bytedance.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-16 09:11:20 +08:00
SYLARandHongsheng Liu d1e230c95b [Diffusion][Quantization] Enable MiniMax-H3 global FP8 with DLO (#5910) 
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-15 23:01:26 +08:00
Samit 136cb27b05 [Bugfix] Fix lost async diffusion outputs in request-level batching (#6023) 
Signed-off-by: samithuang <285365963@qq.com>
 2026-08-15 18:27:51 +08:00
Hongsheng Liu 1260ceaae6 [Docs] Split diffusion attention and CPU offload guides (#6075) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-15 14:41:09 +08:00
Hongsheng Liu 62bf8d630d docs: document distributed layerwise offload compatibility (#5839) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-06 23:21:43 +08:00
bjf-frz 9f978923f2 [docs] consolidate diffusion execution modes (#5599) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
 2026-08-05 22:41:18 +08:00
Chenguang Zheng d5d47e9be1 [Doc] Document async Omni output materialization (#5610) 
Signed-off-by: Chenguang ZHENG <645327136@qq.com>
 2026-08-04 14:50:38 +08:00
zzhanddengyunyang dba05c3517 Enable Diffusion Image models asynchronous output (#4981) 
Signed-off-by: zzh <943967662@qq.com>
Co-authored-by: dengyunyang <584797741@qq.com>
 2026-07-29 17:21:55 +00:00
wangyu 7c17111ff5 [Refactor][CI] Reorganize test tree, and move slow e2e to weekly (#4951) 
Signed-off-by: wangyu <410167048@qq.com>
 2026-07-29 23:53:03 +08:00
SYLARandlishunyang12 7762d664d8 [Attention] Add trtllm diffusion attention backend with Skip-Softmax (#5283) 
Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
 2026-07-27 22:48:11 +08:00
Yang_wdrs 8927ec057c [Refactor][1/N]diffusion/cache: cache-dit (#5226) 
Signed-off-by: yangjianjuan <66548561+yangjianjuan@users.noreply.github.com>
 2026-07-24 22:42:45 +08:00
Diya Peng acf941da4d [Diffusion] Add VAE patch parallel support for FLUX.2-dev (#5292) 
Signed-off-by: dpeng123 <diyapeng123@gmail.com>
 2026-07-24 07:00:57 +00:00
bjf-frzandHongsheng Liu be667a8067 [Refact] Refactor diffusion output streaming (#5166) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-07-23 10:06:08 +08:00
Zeyu Huang | 黃澤宇Copilot Autofix powered by AICursor
 		e39f4dddfb [Refactor]: clean up diffusion scheduler interfaces (#5216) 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Signed-off-by: Zeyu Huang | 黃澤宇 <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-22 14:05:49 +08:00
Bingyu (Spencer) Liu dcf8b07d39 [Core] Cleanup Omni connector runtime and transfer backends (#5096) 
Signed-off-by: spencerr221 <liubingyu62@gmail.com>
 2026-07-22 09:44:15 +08:00
Zeyu Huang | 黃澤宇andHongsheng Liu 1b0de21c55 [Chore]: refactor out unused/redundant params in diffusion pipelines (#1235) 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-07-02 16:32:05 +02:00
JiangJie ZhangandSamit c45ac74d43 [Core][Frontend] Support request-level batching for diffusion pipelines (#4079) 
Signed-off-by: jader <yjader@foxmail.com>
Co-authored-by: Samit <285365963@qq.com>
 2026-06-29 18:09:19 +02:00
Rahul SteigerandHongsheng Liu 327e9dcaf2 [Feature] Spatially-sharded (SP) decode for the Wan VAE (#4620) 
Signed-off-by: Rahul Steiger <rsteiger@nvidia.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-27 09:01:25 +08:00
inkcherryzejwangjunkang1991Zejian WangZejian WangCursor
 		e1b0b83238 add moriio transfer engine (#1742) 
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Signed-off-by: zejwang  <zejwang@amd.com>
Signed-off-by: junkang1991 <junkangchow@gmail.com>
Signed-off-by: Zejian Wang <zejianwang@sjtu.edu.cn>
Co-authored-by: zejwang <zejwang@amd.com>
Co-authored-by: junkang1991 <junkangchow@gmail.com>
Co-authored-by: Zejian Wang <zejianwang@sjtu.edu.cn>
Co-authored-by: Zejian Wang <Zejian.Wang@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-10 08:37:33 +08:00
Samit 083b5e363f [Diffusion] Support LoRA in step-wise execution (#3639) 
Signed-off-by: samithuang <285365963@qq.com>
 2026-05-20 22:26:18 +08:00
yangsonglin13 1c530d984d Yuanrong TransferEngine Connector for NPU (#3180) 
Signed-off-by: yangsonglin13 <yangsonglin566@gmail.com>
 2026-05-20 14:51:33 +08:00
Rustam Khadipash 89f8819525 [Feature] Add support for Pipeline Parallel and integrate it into Wan 2.2 (#2322) 
Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>
 2026-05-19 09:42:17 +08:00
Ting FUasukaqaq-sjaderHongsheng LiuGao Han
 		e49fbd8a2d [Feat] DiffusionEngine Support async batch infer (#2729) 
Signed-off-by: Semmer <semmer@live.cn>
Signed-off-by: jader <yjader@foxmail.com>
Signed-off-by: asukaqaq-s <1311722138@qq.com>
Co-authored-by: asukaqaq-s <1311722138@qq.com>
Co-authored-by: jader <yjader@foxmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-05-06 09:47:13 +12:00
rein yang d435fe0707 [Docs] update documents (#2921) 
Signed-off-by: Rein Yang <ruiruyang2@gmail.com>
 2026-04-20 12:55:04 +08:00
Alex Brooks f346f2fdc6 Add Teacache Support for LongCat Image (#1487) 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
 2026-04-18 07:52:35 +08:00
Alex Brooks f1e3f03726 [feature] Hidden State Prefix Caching (#2164) 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
 2026-04-15 17:01:45 +00:00
TaffyOfficial 1ad726f495 [Fix] HunyuanImage-3.0: unify naming hunyuan_image_3 → hunyuan_image3 (#2712) 2026-04-14 21:53:47 -07:00
amy-why-3459 2a1d5060ab [skip ci][doc]Update async_chunk design diagram (#2420) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-04-13 20:43:40 +08:00
zhou zhuoxin 16041ab550 [Refactor] Extend CFG Parallel to support 3 or 4 branch dispatch across M GPUs (#2423) 2026-04-12 22:06:49 +08:00
Jinhengahengljhnatureofnature
 		c3c736d488 [Feature] Port Bagel RDMA flow to latest main (#2000) 
Signed-off-by: Jinheng Li <ahengljh@gmail.com>
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: ahengljh <ahengljh@users.noreply.github.com>
Co-authored-by: natureofnature <wzliu@connect.hku.hk>
 2026-04-08 14:55:35 +08:00
skfandCanlin Guo 6433847249 [skipCI][Docs] Add expert_parallel.md (#2471) 
Signed-off-by: skf1999 <13234016272@163.com>
Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
 2026-04-08 02:35:14 +00:00
Canlin GuoandHongsheng Liu e771842781 [Feature] Support vae tiling parallel encode (#2368) 
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-04-06 14:47:45 +00:00
Yangshen DengandClaude Opus 4.6 837679dbaa [Diffusion] Refactor CFG parallel for extensibility and performance (#2063) 
Signed-off-by: Yangshen Deng <yangshen.d@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-30 16:48:58 +00:00
Didan Deng 911be0d426 [skip ci][Doc] Refine the Diffusion Features User Guide (#1928) 
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
 2026-03-30 14:28:04 +08:00
dongbo910220andDidan Deng c2888a8a36 [Docs] refine UAA documentation (#2148) 
Signed-off-by: dongbo910220 <1275604947@qq.com>
Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
 2026-03-25 15:46:15 +08:00