anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
2,199 Commits 235 Branches 31 Tags
f18bb69b7ddac5b2fe73e28c38c7fbe26a8da9cb
Commit Graph
214 Commits
Author SHA1 Message Date
tzhouamandClaude Opus 4.8 c898701d53 fix(engine): dedup ZMQ ports across stages to stop OmniMasterServer bind races 
Each (stage_id, replica_id) route allocated its 3 handshake/input/output
ports via a separate get_open_ports_list(count=3) call. That helper only
guarantees uniqueness *within* a single call, so across routes the OS could
hand back a just-closed ephemeral port. Two stages then drew the same port
and the second engine subprocess to bind() it died with
"zmq.error.ZMQError: Address already in use", surfacing as
"Orchestrator initialization failed" / "Stage N replica M exited before API
server became ready". This flaked multi-stage startup (e.g. Qwen3-Omni
thinker/talker/code2wav) — one CI shard passed while a parallel shard failed
on the same commit (build 2630, dev/vllm-align).

OmniMasterServer now tracks every port it hands out in self._allocated_ports
(seeded with the registration port and the coordinator ROUTER port) and a new
_alloc_unique_ports() redraws any collision with a bounded retry budget,
raising loudly on port exhaustion instead of spinning. Adds _port_from_zmq_address()
helper and two CPU-only regression tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-11 14:21:17 +08:00
tzhouam b73036f538 rebase: align vllm-omni with vLLM 740f379faefd 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-07-11 14:21:15 +08:00
Kushanamkushanamkahmadian
 		ff6f0da271 [Core] Composable parallel: Phase 1 strategy overlay (TP/DP/stage-replica) (RFC #4084) (#4281) 
Signed-off-by: Kushan A. <kahmadian@nvidia.com>
Signed-off-by: kushanam <kushanam@users.noreply.github.com>
Signed-off-by: Kushan A <kahmadian@nvidia.com>
Signed-off-by: kahmadian <kahmadian@users.noreply.github.com>
Co-authored-by: kushanam <kushanam@users.noreply.github.com>
Co-authored-by: kahmadian <kahmadian@users.noreply.github.com>
 2026-07-10 21:29:24 -06:00
amy-why-3459 9a44e7e019 [Refactor][Phase 1]Remove redundant functions and logs (#4986) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-07-10 15:13:35 +08:00
Alex Brooks b7d08fe5f6 [Core / Bugfix] Add Mechanism for Endpoint Rejection (#4762) 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
 2026-07-08 09:00:47 -06:00
Yuanheng Zhao 06329c6450 [Bugfix] Fix request batching regression on Ming-flash-omni ImageGen (#4837) 
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com>
 2026-07-03 03:04:17 +00:00
ZacheryAU b0f6b9de96 [Bugfix] clean per-stage audio RTF and align audio metrics (#4794) 2026-07-02 02:55:44 +00:00
Zhou Taichang a560ed184d [Rebase] Rebase to vllm v0.24.0 (#4709) 2026-06-30 19:41:06 +08:00
JiangJie ZhangandSamit c45ac74d43 [Core][Frontend] Support request-level batching for diffusion pipelines (#4079) 
Signed-off-by: jader <yjader@foxmail.com>
Co-authored-by: Samit <285365963@qq.com>
 2026-06-29 18:09:19 +02:00
Peiqi YinChenguang ZHENGCursorHongsheng LiuGao Han
 		504698dec8 [Core] Performance fix for orchestrator bottleneck: separates inter-stage from client outputs (#4527) 
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>
Co-authored-by: Chenguang ZHENG <645327136@qq.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-06-29 17:43:36 +02:00
Rahul SteigerandHongsheng Liu 327e9dcaf2 [Feature] Spatially-sharded (SP) decode for the Wan VAE (#4620) 
Signed-off-by: Rahul Steiger <rsteiger@nvidia.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-27 09:01:25 +08:00
LHXuuuYuanheng ZhaoYuanheng ZhaoHongsheng Liu
 		0b04f2a0d6 [Model] Add Ming-omni-tts MoE 16.8B-A3B + CFM CUDAGraph (#4341) 
Signed-off-by: LHXuuu <xulianhao.xlh@antgroup.com>
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com>
Co-authored-by: Yuanheng Zhao <54058983+yuanheng-zhao@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-25 22:11:35 +08:00
Canlin Guo 2e6c83c89c [Model] Support MOSS-TTS-local-v1.5 (#4664) 
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
 2026-06-25 15:43:03 +08:00
MrDongslsMr.DongYueqian LinYueqian Lin
 		1951dca786 [Model] add soulx-singer support (#3862) 
Signed-off-by: Mr.Dong <1042542469@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Mr.Dong <1042542469@qq.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
 2026-06-24 05:36:45 +00:00
Zeyu Huang | 黃澤宇Copilot Autofix powered by AICursor
 		3901ad8165 [feat] Streaming diffusion video generation output (#3737) 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Signed-off-by: Zeyu Huang | 黃澤宇 <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-23 09:57:53 +00:00
QianCyrusandHongsheng Liu b84c99c122 [Bugfix] Fix make_request_output TypeError on multimodal generation-stage outputs (#4579) 
Signed-off-by: QianCyrus <happyloser233@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-23 14:42:30 +08:00
BeatSeat 044240d8eb [Model] Add IndexTTS2 text-to-speech support (#3838) 
Signed-off-by: BeatSeat <wendavid552@gmail.com>
Signed-off-by: David <david@podcast.ai>
Signed-off-by: wendavid552 <wendavid552@gmail.com>
Signed-off-by: WenDavid <davidzhangcf@foxmail.com>
 2026-06-22 19:24:01 +00:00
rein yang afbcc51672 [Model] Aura support - Non async chunk path (#4257) 
Signed-off-by: Rein Yang <ruiruyang2@gmail.com>
Signed-off-by: R2-Y <ruiruyang2@gmail.com>
 2026-06-21 22:08:26 +08:00
amy-why-3459 f05e03aea8 [Perf][DreamZero]dreamzero supports CUDA graph, torch.compile, and DiT Caching. (#4213) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-06-19 14:24:37 +08:00
汪志鹏CursorHongsheng LiuClaude Sonnet 4.6
 		f75c437a60 [Refactor]Base class for dit pipelines with unified parameter declaration[1/N] (#4225) 
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-18 17:14:18 +08:00
Peiqi YinHongsheng LiuChenguang ZHENG
 		64a5bb92de [Refactor] Omni Stage Runtime and Distributed Replica Control Plane (#3855) 
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Chenguang ZHENG <645327136@qq.com>
 2026-06-17 14:13:20 +08:00
Zheng WengangandHongsheng Liu ba4932c364 [Bugfix] Fix multi-replica stage identity and clean up redundant deploy config (#4410) 
Signed-off-by: ZhengWG <zwg0606@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-16 14:08:11 +08:00
Hongsheng Liu aff0af9754 [Refactor] Extract diffusion output formatting boundary (#4407) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-06-15 22:50:50 +08:00
Zeng ChuangandHongsheng Liu 10404e4c5b [Perf] Restore parallel stage initialization for AR+DiT pipelines (#3641) 
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-15 09:37:11 +08:00
Zhou Taichang 7b837944a1 [Rebase] Rebase to vllm 0.23.0 (#4286) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-06-13 16:43:02 +00:00
Megha AgarwalCopilotHongsheng Liumeghaagr13Boao ShiTaichang ZhouCursor
 		8f869c8ea5 [Refactor] Output Processor Phase 2: separate multimodal output channel (#1601) (#2744) 
Signed-off-by: Megha Agarwal <agarwalmegha1308@gmail.com>
Signed-off-by: meghaagr13 <divyanshsinghvi@gmail.com>
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: meghaagr13 <divyanshsinghvi@gmail.com>
Co-authored-by: Boao Shi <aoibosh@connect.hku.hk>
Co-authored-by: Taichang Zhou <tzhouam@connect.ust.hk>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-12 15:19:14 +08:00
MaciejBalaNVandHongsheng Liu 03cc888485 [refactor] Refactor guardrail error handling - add 400 error code (#4297) 
Signed-off-by: Maciej Bala <mbala@nvidia.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-11 21:09:24 +00:00
dengyunyangandHongsheng Liu 14879d6619 [Bugfix][HunyuanImage] fix accuracy in stream mode (#4265) 
Signed-off-by: dengyunyang <584797741@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-11 10:39:00 +08:00
Mike QiuMike_QiuClaude Opus 4.6Yueqian LinHongsheng Liu
 		57ea047acf Step audio R1 reasoning parser (#2846) 
Signed-off-by: Mike_Qiu <qiudayu.qdy@antgroup.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Signed-off-by: Mike Qiu <qdy220091330@gmail.com>
Co-authored-by: Mike_Qiu <qiudayu.qdy@antgroup.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-10 21:03:19 +00:00
akshatvishuandHongsheng Liu 2fa4c7a4a9 [Model] Add Ming-omni-tts dense 0.5B pipeline (#2906) 
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-10 20:11:04 +08:00
inkcherryzejwangjunkang1991Zejian WangZejian WangCursor
 		e1b0b83238 add moriio transfer engine (#1742) 
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Signed-off-by: zejwang  <zejwang@amd.com>
Signed-off-by: junkang1991 <junkangchow@gmail.com>
Signed-off-by: Zejian Wang <zejianwang@sjtu.edu.cn>
Co-authored-by: zejwang <zejwang@amd.com>
Co-authored-by: junkang1991 <junkangchow@gmail.com>
Co-authored-by: Zejian Wang <zejianwang@sjtu.edu.cn>
Co-authored-by: Zejian Wang <Zejian.Wang@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-10 08:37:33 +08:00
Juan Pablo ZuluagaJuanPZuluagaGao Han
 		57227dc7f3 [Perf][Bugfix] qwen3-tts hot path: prefix-cache OOM guards + talker/orchestrator micro-opts (#3689) 
Signed-off-by: JuanPZuluaga <juanz9312@gmal.com>
Co-authored-by: JuanPZuluaga <juanz9312@gmal.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-06-09 10:36:57 -07:00
ChenWenjing e25239c2ec [bugfix] fix realtime ci timeout error (#4187) 
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
 2026-06-06 00:05:07 +08:00
ZacheryAU bfc7cedcf1 Benchmark data statistics for each stage of omni models (#3628) 
Signed-off-by: ZacheryAU <70869627+ZacheryAU@users.noreply.github.com>
 2026-06-04 21:43:35 +08:00
amy-why-3459 8949fdc440 [BugFix] Fix the issue of vllm failing to start. (#4105) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-06-03 15:44:46 +00:00
rein yangandHongsheng Liu 3a7c7f1422 [Bugfix] fix qwen3-omni performance regression (#3575) 
Signed-off-by: rein yang <ruiruyang2@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-02 23:02:07 +08:00
Alex Brooks 12e75c6618 [Core] Integrate TrackingArgumentParser (#3369) 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
 2026-06-02 21:56:24 +08:00
MaciejBalaNVSYLARlishunyang12
 		bc794e625f Add Cosmos3 model support (#3454) 
Signed-off-by: Maciej Bala <mbala@nvidia.com>
Signed-off-by: MaciejBalaNV <mbala@nvidia.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
Co-authored-by: lishunyang12 <lishunyang12@163.com>
 2026-06-02 04:04:31 +08:00
tc-mbGKangarooGKangarooHongsheng Liu
 		2b7249fbb3 [Model]Support MiniCPM-o 4.5 (#3642) 
Signed-off-by: tc-mb <tianchi_cai@icloud.com>
Co-authored-by: GKangaroo <1095103651@qq.com>
Co-authored-by: GKangaroo <gqx24@mails.tsinghua.edu.cn>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-01 22:12:24 +08:00
Yueqian LinandHongsheng Liu c067485030 [Perf][TTS] Bounded-K active-stream window for Stage 1 (RFC #3535) (#3592) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-01 21:59:10 +08:00
LHXuuuandvraiti 41a795b5d6 [Metrics] Add audio SLOs + cross-stage transfer families + per-(stage, replica) wrap for upstream vllm:* (#3576) 
Signed-off-by: vraiti <vraiti@redhat.com>
Signed-off-by: LHXuuu <xulianhao.xlh@antgroup.com>
Co-authored-by: vraiti <vraiti@redhat.com>
 2026-06-01 20:20:48 +08:00
+1
Zhou TaichangClaude Opus 4.7skfZheng WengangHongsheng LiuWeiQing ChenSYLARYueqian Linnainiu258JasonYueqian Lin
 		f92d84fcf9 [Rebase] Rebase to vllm releases/v0.22.0 (#3891) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: ZhengWG <zwg0606@gmail.com>
Signed-off-by: david6666666 <530634352@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Signed-off-by: nainiu258 <cperfect02@163.com>
Signed-off-by: nainiu258 <101917677+nainiu258@users.noreply.github.com>
Signed-off-by: JasonJ2021 <jasonj@zju.edu.cn>
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: skf <54565339+skf-1999@users.noreply.github.com>
Co-authored-by: Zheng Wengang <zwg0606@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: nainiu258 <101917677+nainiu258@users.noreply.github.com>
Co-authored-by: Jason <72191212+JasonJ2021@users.noreply.github.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-06-01 11:08:12 +08:00
Sy03Dan250124Yueqian LinYueqian Lin
 		d4c139508f [Feature] Add precomputed custom voices and Qwen3-TTS ref-context cache (#3492) 
Signed-off-by: Sy03 <1370724210@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Dan250124 <416947747@qq.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-05-31 17:51:24 +00:00
ChenWenjing 5dfdf584dd [Feat]audio streaming input for async chunk (#3614) 
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
 2026-05-31 17:39:25 +08:00
Lancer汪志鹏Gao Han
 		b690a10e3a [Feat] Support MagCache (#1287) 
Signed-off-by: Lancer <maruixiang6688@gmail.com>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-05-29 22:26:23 +08:00
knlnguyen1802SamitHongsheng Liu
 		c69ed4037d [feat] Support prompt embedding caching for diffusion model (#2962) 
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Co-authored-by: Samit <285365963@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-05-29 15:36:03 +08:00
LancerandAlicia 5c3c81920b [Bugfix] Ensure stage and diffusion subprocesses exit when parent dies unexpectedly (#3751) 
Signed-off-by: Lancer <maruixiang6688@gmail.com>
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>
 2026-05-28 08:22:04 +08:00
Hacoandzzhuoxin1508 37eebff674 [BugFix] Fix diffusion parallel_config YAML override and add deploy config field allowlist (#3483) 
Signed-off-by: xiaohajiayou <923390377@qq.com>
Co-authored-by: zzhuoxin1508 <zzhuoxin1508@users.noreply.github.com>
 2026-05-28 01:00:32 +08:00
Zheng WengangClaude Opus 4.6Yuanheng Zhao
 		a698269e47 [Model] Add Ming-flash-omni-2.0 Image Generation (Diffusion) Stage (#2875) 
Signed-off-by: ZhengWG <zwg0606@gmail.com>
Signed-off-by: Zheng Wengang <zwg0606@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Yuanheng Zhao <54058983+yuanheng-zhao@users.noreply.github.com>
 2026-05-27 23:21:20 +08:00
lyj-jjjCursorHongsheng Liu
 		283873411a glm-image: fix(npu)per-stage runtime env for HCCL ports + GLM-Image NPU stage config (#3235) 
Signed-off-by: lyj-jjj <liuyingjun5@huawei.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-05-27 12:18:47 +12:00