anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
2,942 Commits 235 Branches 31 Tags
c4e61133fbb5587e0cf8972ce32ffac4e37fa739
Commit Graph
111 Commits
Author SHA1 Message Date
tzhouamandClaude Fable 5.1 66319ed159 [Edge][WP8] Sample the chunk hop put->get latency before the shm get removes the lock file 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 15:25:43 +08:00
tzhouamandClaude Fable 5.1 37c0ea9a6b [Edge][WP8] Per-step overhead attribution, edge scheduling helpers, CPU thread sweep 
- vllm_omni/edge/step_stats.py: process-local counters enabled by
  VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core
  step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped
  per process and merged into an attribution table (share of the 80 ms frame).
  Hooks: stage_engine_core_proc (instrument after construction, dump at exit),
  chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime),
  orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats.
- vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and
  the no-preemption KV admission rule (exact bytes/token from the talker KV
  geometry); adapt.derive_overrides now sizes max_num_seqs with it and
  auto_profile passes the geometry read from the model config.
- Engine re-reads parallel_stage_init after apply_deploy_profile so the edge
  profile's orchestrator default takes effect (was read before the profile).
- benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor)
  with per-step attribution.
- Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py,
  tests/benchmarks/test_cpu_thread_sweep.py, profile default test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 14:55:55 +08:00
tzhouamandClaude Fable 5.1 74acd66307 [Edge][WP7] Add CudaIpcConnector (CUDA IPC for device tensor leaves over shm control plane) 
Wraps SharedMemoryConnector: CUDA tensor leaves are exported with
torch.multiprocessing reduce_tensor + an inter-process CUDA event, rebuilt and
cloned by the consumer, released by the producer after an ack file appears.
CPU leaves and metadata keep the shm path. Registered as "CudaIpcConnector";
deploy/qwen3_tts_cuda_ipc.yaml selects it for the talker->code2wav hop.
Adoption in the edge profile is measurement-gated (hop latency bench).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 14:41:33 +08:00
NATUREandamy-why-3459 ff6e906a25 [Model] Preserve MiniCPM-o Talker full attention until capacity (#6799) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-09-06 22:25:35 +08:00
Inesh Reddy ChappidiandGao Han eb11446b7f [Bugfix][Core] Do not advance the segment watermark past an unmaintained request counter (#6834) 
Signed-off-by: Inesh Reddy Chappidi <ineshreddy249@gmail.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-08-31 15:27:13 +00:00
Sy03Nathan PriceEric
 		21dd4b33ac [Bugfix][MiniCPM-o] Stabilize native duplex streaming and Stage-1 handoffs (#6529) 
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: Nathan Price <nathan@abridge.com>
Co-authored-by: Eric(Qun) <186020953+0xtoward@users.noreply.github.com>
 2026-08-29 13:05:24 -07:00
NATURE 87736a3cc5 [Bugfix][MiniCPM-o] Bound full-duplex Talker context with sliding recompute (#6626) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
 2026-08-30 00:49:42 +08:00
Wallbreazzz 4ad455e61e [Feature][TTS] adaptive chunk ramp (Phase 2 buffer-feedback controller) (#6001) 
Signed-off-by: Wallbreazzz <110282866+Wallbreazzz@users.noreply.github.com>
 2026-08-28 02:11:05 -07:00
Sy03andsy03 432b2c6ca3 [Model] Add native full-duplex Nemotron VoiceChat serving (#6089) 
Signed-off-by: Sysy <1370724210@qq.com>
Signed-off-by: sy03 <137072421@qq.com>
Co-authored-by: sy03 <137072421@qq.com>
 2026-08-26 13:19:20 -07:00
NATURE 702a409664 [Feature][MiniCPM-o] Support configurable concurrent duplex sessions (#6021) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
 2026-08-25 02:27:30 +00:00
NATUREandamy-why-3459 4f7c08c0eb [Bugfix][MiniCPM-o] Fix async-chunk snapshot replacement and prompt cleanup (#6406) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-21 16:37:21 +08:00
NATURERuirui Yang | ReinHongsheng Liu
 		b57dc70ec4 [Bugfix] Reclaim resumable async-chunk requests on finish (#6360) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: Ruirui Yang | Rein <73573651+R2-Y@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-20 15:59:39 +08:00
Yueqian LinandHongsheng Liu 611c083587 [BugFix][TTS] R1.1-R1.4: turn silent async-chunk failures into visible ones (#6033) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-17 10:51:17 +08:00
Shi Boao Bowie汪志鹏ruirui yangamy-why-3459Gao HanHongsheng Liu
 		fcf1085395 [Refactor][OutputProcessor 2/8]: OmniRequestOutput should inherit RequestOutput and no nested wrap-up (#5146) 
Signed-off-by: Boao Shi <aoibosh@connect.hku.hk>
Signed-off-by: bowieshi <aoibosh@connect.hku.hk>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Signed-off-by: R2-Y <ruiruyang2@gmail.com>
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: Shi Boao Bowie <aoibosh@connect.hku.hk>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: ruirui(rein) yang <73573651+R2-Y@users.noreply.github.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-12 22:58:23 +08:00
Zhou TaichangandClaude Fable 5 2c4e9a945b [BugFix] Make pure-TP KV receive fallback all-or-nothing across ranks (#5636) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-08-01 15:43:48 +00:00
SYLARandlishunyang12 6f98c807d2 [Bugfix] Restore BAGEL CFG KV transfers (#5620) 
Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
 2026-08-01 15:06:05 +00:00
akshatvishu cd5daaea6a [BugFix][Metrics] Fix per-stage time and token stats in metrics API (#4974) 
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
 2026-08-01 03:54:17 +00:00
Shenglei FuandClaude 3b0718f22e [Bugfix] Fix active_stream_window silently stalling audio generation (#5349) (#5373) 
Signed-off-by: Shenglei Fu <sfu@confluent.io>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-30 14:56:18 +00:00
Sy03andHongsheng Liu 0feb92bdfa [Bugfix][Minicpm] Support full-duplex three-stage serving (#5380) 
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-07-29 01:58:34 +08:00
Canlin Guo 61324e65d7 [Refactor] Make MOSS-TTS-Local stream session dynamic batch (#5235) 
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>
 2026-07-28 21:38:24 +08:00
Yueqian Lin 8f115164c2 [Bugfix][Connectors] Flush processor tail on terminal async-chunk sends (#5414) 
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-07-27 16:04:08 +00:00
Zhou TaichangClaude Fable 5tzhouam
 		cbab1d4d8f [Rebase] Rebase to vllm 0.26.0 (#5443) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: tzhouam <tzhouam@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: tzhouam <tzhouam@gmail.com>
 2026-07-27 23:35:49 +08:00
Wallbreazzz e28e389a76 feat(qwen3-tts): add codec_chunk_ramp for gradual chunk size warmup (#5152) 
Signed-off-by: Wallbreazzz <110282866+Wallbreazzz@users.noreply.github.com>
 2026-07-27 04:15:43 +00:00
rein yangandSy03 ad3a162eb8 [perf] Minicpm o4.5 optimization (#5228) 
Signed-off-by: R2-Y <ruiruyang2@gmail.com>
Co-authored-by: Sy03 <1370724210@qq.com>
 2026-07-24 17:20:39 +08:00
Sy03lishunyang12Yueqian LinStormMelodyGao HanHongsheng Liu
 		05b794f29b [Full Duplex] Feat: Support Full-Duplex realtime runtime & add MiniCPM-o 4.5 demo (#3907) 
Signed-off-by: Sy03 <1370724210@qq.com>
Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: StormMelody <22371184@zju.edu.cn>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-07-23 16:45:18 +08:00
Zeyu Huang | 黃澤宇Copilot Autofix powered by AICursor
 		e39f4dddfb [Refactor]: clean up diffusion scheduler interfaces (#5216) 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Signed-off-by: Zeyu Huang | 黃澤宇 <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-22 14:05:49 +08:00
Bingyu (Spencer) Liu dcf8b07d39 [Core] Cleanup Omni connector runtime and transfer backends (#5096) 
Signed-off-by: spencerr221 <liubingyu62@gmail.com>
 2026-07-22 09:44:15 +08:00
Honghan Zhuzhuhh97Claude Opus 4.7Gao HanHongsheng Liu
 		f84f0e72fd [Bug Fix] Hunyuan image cfg bugfix (#4752) 
Signed-off-by: zhuhh97 <zhuhonghan@huawei.com>
Signed-off-by: dph97 <nenbaying@163.com>
Co-authored-by: zhuhh97 <z00806815@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-07-15 13:08:15 +08:00
Zhou Taichang a560ed184d [Rebase] Rebase to vllm v0.24.0 (#4709) 2026-06-30 19:41:06 +08:00
Honghan ZhuClaude Opus 4.7zhuhh97
 		5db89b41c4 [Core] hunyuan-image prefetch kv (#4448) 
Signed-off-by: z00806815 <zhuhonghan@huawei.com>
Signed-off-by: zhuhh97 <zhuhonghan@huawei.com>
Signed-off-by: dph97 <nenbaying@163.com>
Signed-off-by: zhh <zhh@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: zhuhh97 <zhuhh97@users.noreply.github.com>
 2026-06-25 21:31:49 +08:00
yangyonggit ccb5c53ea5 [Cleanup] Remove dead/stale modules with no repo references (fixes #4009) (#4044) 
Signed-off-by: leo.yang <leo.yang.engineer@gmail.com>
 2026-06-22 09:23:04 +08:00
Peiqi YinHongsheng LiuChenguang ZHENG
 		64a5bb92de [Refactor] Omni Stage Runtime and Distributed Replica Control Plane (#3855) 
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Chenguang ZHENG <645327136@qq.com>
 2026-06-17 14:13:20 +08:00
TJian 5cce797c03 [Bugfix] [CI] [ROCm] [CUDA] Fix voxtral test (#4380) 
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2026-06-16 16:39:23 +08:00
Zhou Taichang 7b837944a1 [Rebase] Rebase to vllm 0.23.0 (#4286) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-06-13 16:43:02 +00:00
Megha AgarwalCopilotHongsheng Liumeghaagr13Boao ShiTaichang ZhouCursor
 		8f869c8ea5 [Refactor] Output Processor Phase 2: separate multimodal output channel (#1601) (#2744) 
Signed-off-by: Megha Agarwal <agarwalmegha1308@gmail.com>
Signed-off-by: meghaagr13 <divyanshsinghvi@gmail.com>
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: meghaagr13 <divyanshsinghvi@gmail.com>
Co-authored-by: Boao Shi <aoibosh@connect.hku.hk>
Co-authored-by: Taichang Zhou <tzhouam@connect.ust.hk>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-12 15:19:14 +08:00
Sy03Yueqian LinHongsheng Liu
 		e27fd46ab9 [Perf] Optimize Higgs Audio v3 serving (#4204) 
Signed-off-by: Sy03 <1370724210@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-11 13:29:31 -07:00
longguoandnpuichigo 73df8326d7 [Bugfix] purge chunk-transfer zombies on every schedule tick to keep engine-core alive on aborts (fixes #3736) (#3774) 
Signed-off-by: abinggo <107740309+abinggo@users.noreply.github.com>
Co-authored-by: npuichigo <11533479+npuichigo@users.noreply.github.com>
 2026-06-10 09:00:23 +08:00
inkcherryzejwangjunkang1991Zejian WangZejian WangCursor
 		e1b0b83238 add moriio transfer engine (#1742) 
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Signed-off-by: zejwang  <zejwang@amd.com>
Signed-off-by: junkang1991 <junkangchow@gmail.com>
Signed-off-by: Zejian Wang <zejianwang@sjtu.edu.cn>
Co-authored-by: zejwang <zejwang@amd.com>
Co-authored-by: junkang1991 <junkangchow@gmail.com>
Co-authored-by: Zejian Wang <zejianwang@sjtu.edu.cn>
Co-authored-by: Zejian Wang <Zejian.Wang@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-10 08:37:33 +08:00
ChenWenjing 3842970a82 [bugfix] fix streaming input issue after rebase 0.22.0 (#4085) 
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
 2026-06-03 16:07:41 +08:00
Yueqian LinandHongsheng Liu c067485030 [Perf][TTS] Bounded-K active-stream window for Stage 1 (RFC #3535) (#3592) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-01 21:59:10 +08:00
+1
Zhou TaichangClaude Opus 4.7skfZheng WengangHongsheng LiuWeiQing ChenSYLARYueqian Linnainiu258JasonYueqian Lin
 		f92d84fcf9 [Rebase] Rebase to vllm releases/v0.22.0 (#3891) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: ZhengWG <zwg0606@gmail.com>
Signed-off-by: david6666666 <530634352@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Signed-off-by: nainiu258 <cperfect02@163.com>
Signed-off-by: nainiu258 <101917677+nainiu258@users.noreply.github.com>
Signed-off-by: JasonJ2021 <jasonj@zju.edu.cn>
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: skf <54565339+skf-1999@users.noreply.github.com>
Co-authored-by: Zheng Wengang <zwg0606@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: nainiu258 <101917677+nainiu258@users.noreply.github.com>
Co-authored-by: Jason <72191212+JasonJ2021@users.noreply.github.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-06-01 11:08:12 +08:00
ChenWenjing 5dfdf584dd [Feat]audio streaming input for async chunk (#3614) 
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
 2026-05-31 17:39:25 +08:00
JiangJie Zhang 6fa9dbbe42 [Diffusion] Unify diffusion request identity on request_id (#3744) 
Signed-off-by: jader <yjader@foxmail.com>
 2026-05-28 14:56:56 +08:00
LancerandAlicia 5c3c81920b [Bugfix] Ensure stage and diffusion subprocesses exit when parent dies unexpectedly (#3751) 
Signed-off-by: Lancer <maruixiang6688@gmail.com>
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>
 2026-05-28 08:22:04 +08:00
Sy03andYueqian Lin d37912b071 [TTS][Perf] Optimize Fish Speech S2 Pro high-concurrency serving (#3773) 
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-05-27 13:25:49 -05:00
yangsonglin13 1c530d984d Yuanrong TransferEngine Connector for NPU (#3180) 
Signed-off-by: yangsonglin13 <yangsonglin566@gmail.com>
 2026-05-20 14:51:33 +08:00
chickeytonandherotai214 e277feacaf [large-scale-serving] Integrate OmniCoordinator into stage engine pipeline (#3569) 
Signed-off-by: chickeyton <ngton2014@gmail.com>
Signed-off-by: herotai214 <herotai214@gmail.com>
Co-authored-by: herotai214 <herotai214@gmail.com>
 2026-05-19 12:08:20 +08:00
amy-why-3459 e7ee5de09f [BugFix] Fix the issue of thinker requests being preempted, causing shape mismatch. (#3147) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-05-15 17:07:42 +08:00
NATUREandHongsheng Liu 82a0b3a467 [2/5] [core]refactor communication layer: PR 2 of 5 Qwen3 Omni non async (#2677) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-05-15 14:14:08 +08:00
dengyunyangand汪志鹏 94254e015f [BugFix] fix shm connector (#3583) 
Signed-off-by: dengyunyang <584797741@qq.com>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
 2026-05-14 20:07:12 +08:00