anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
2,900 Commits 235 Branches 31 Tags
073d31b9cd2cde8f50f69d4b7bdd60fcddfe6ded
Commit Graph
77 Commits
Author SHA1 Message Date
tzhouamandClaude Fable 5.1 073d31b9cd [Edge][WP2] Add initialization timeline and cold-start profiler 
* vllm_omni/engine/init_timeline.py: InitTimeline phase collector; engine
  phases (stages_init, g3_device_lock, g2_spawn/ready) and worker phases
  (proc_entry, engine_core_init, init_device, load_model/load_weights,
  talker_mtp_init, attn_prewarm, cudagraph_capture, talker_mtp_capture,
  code2wav_cudagraph, predictor_setup_compile/warmup/cudagraph) recorded via
  worker_phase/timed_phase; workers append JSONL under
  VLLM_OMNI_INIT_TIMELINE_DIR (inherited at spawn) and the engine merges them.
  Enabled by VLLM_OMNI_INIT_TIMELINE=<json> or log_stats; table logged at INFO,
  partial table on orchestrator startup timeout, suggested timeouts logged.
* omni_base.py: named deploy profiles default to stage_init_timeout=900 /
  init_timeout=1500 unless given explicitly.
* gpu_model_runner._init_talker_mtp tolerates max_cudagraph_capture_size=None;
  gpu_ar_model_runner.shutdown skips CUDA graph teardown on CPU.
* benchmarks/tts/init_profile.py: runs stream_latency_bench --init-only per
  configuration (default, cold_cache, eager_stage1, eager_both,
  parallel_stage_init, edge, ...) and tabulates phase durations.
* stream_latency_bench: --parallel-stage-init / --deploy-profile flags; the
  resource poller no longer shadows Thread._stop and skips GPU sampling on CPU.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 11:01:39 +08:00
tzhouamandClaude Fable 5.1 f804bbd679 [Edge][WP0] Record Qwen3-TTS streaming baselines on 1x L20X 
Default deploy config (1 -> 25 frame schedule), 12 prompts x 3 repeats,
warm-up excluded, one GPU via `gpu run`:
  1.7B CustomVoice: TTFA p50 32.0 ms, playback-start p50 116 ms (stall at
    TTFA 85 ms), RTF_total 0.086, chunk gap p50 163 ms, init 271 s (cold caches)
  0.6B CustomVoice: TTFA p50 27.8 ms, playback-start p50 95 ms (stall 67 ms),
    RTF_total 0.077, chunk gap p50 146 ms, init 152 s
Per-request chunk arrivals, memory and the full summaries are in the JSON
(force-added: *.json is gitignored repo-wide).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 10:49:37 +08:00
tzhouamandClaude Fable 5.1 b70207f57f [Edge][WP0] Add offline streaming-latency benchmark for chunked TTS 
Adds benchmarks/tts/stream_latency_bench.py (AsyncOmni driver reusing the
Qwen3-TTS example's prompt builder; --model override, warm-up/repeat,
--init-only, per-chunk arrivals, peak RSS and per-process GPU memory) and
benchmarks/tts/stream_latency_metrics.py (TTFA, underrun-free playback start,
stall at TTFA, RTF streamed/total, chunk gaps, percentiles) with CPU unit
tests. Results go under benchmarks/tts/edge_results/stream_latency/.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 10:24:53 +08:00
Weiming LiaoandGao Han 1cd92553cf [CI][Bugfix] Use the matching checkpoint for Qwen3-TTS VoiceDesign tests (#7054) 
Signed-off-by: Weiming Liao <liaowm5@gmail.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-09-06 04:27:21 +00:00
0z5aand0z5a 4927fd5412 [Core] Optimize diffusion LoRA expand accumulation (#6268) 
Signed-off-by: 0z5a <0z5a@users.noreply.github.com>
Signed-off-by: 0z5a <dinadegttyarenko6@gmail.com>
Co-authored-by: 0z5a <0z5a@users.noreply.github.com>
 2026-09-03 10:37:35 +08:00
DanaerLee ca6b4e2112 [Perf][Bugfix][OmniVoice] Restore float16 serving, and fuse the generator hot loop (#6317) 
Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
 2026-09-02 19:54:55 +00:00
Allen WuTRAE CLISy03TRAE CLI
 		f90e992dc1 [Model] Optimize CosyVoice3 TensorRT stream handoff (#5673) 
Signed-off-by: Allen Wu <allenwu2795@gmail.com>
Co-authored-by: TRAE CLI <noreply@bytedance.com>
Co-authored-by: Sy03 <1370724210@qq.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
 2026-08-25 22:51:46 -07:00
Yukim1 6f55a06fab [Docs] Remove stale stage config references (#6270) 
Signed-off-by: zwhzzz0821 <2831474076@qq.com>
 2026-08-20 07:09:12 +08:00
Hongsheng Liu 8b8d3f56db [Misc] Consolidate LingBot text-to-video runner (#6076) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-13 08:42:37 +08:00
Shi Boao Bowie汪志鹏ruirui yangamy-why-3459Gao HanHongsheng Liu
 		fcf1085395 [Refactor][OutputProcessor 2/8]: OmniRequestOutput should inherit RequestOutput and no nested wrap-up (#5146) 
Signed-off-by: Boao Shi <aoibosh@connect.hku.hk>
Signed-off-by: bowieshi <aoibosh@connect.hku.hk>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Signed-off-by: R2-Y <ruiruyang2@gmail.com>
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: Shi Boao Bowie <aoibosh@connect.hku.hk>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: ruirui(rein) yang <73573651+R2-Y@users.noreply.github.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-12 22:58:23 +08:00
Sy03andYueqian Lin 58007c0084 [New Model] Support IndexTTS 2.5 (#5957) 
Signed-off-by: Sysy <1370724210@qq.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-08-11 23:00:35 +00:00
WeiQing Chen a874b8e09b [Perf][CI] Add MiniMax-H3 4xH100 diffusion perf config (#5836) 
Signed-off-by: David Chen <530634352@qq.com>
 2026-08-07 00:12:07 +08:00
wtz2333andhsliuustc0106 5a6a93dc5f [diffusion][model] Add LingBot Video dense and MoE support (#5035) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Signed-off-by: wtz2333 <2955110911@qq.com>
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-07-21 17:27:29 +08:00
bjf-frz b14ff23041 [PERF][CI]Add Cosmos3 diffusion perf config (#5010) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
 2026-07-14 10:25:17 +00:00
faceless void 11f6a0b72e Make speech streaming default to SSE (#4679) 
Signed-off-by: syd520zy <529477025@qq.com>
 2026-06-30 02:17:24 +00:00
Canlin Guo 2e6c83c89c [Model] Support MOSS-TTS-local-v1.5 (#4664) 
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
 2026-06-25 15:43:03 +08:00
bjf-frz 2447740308 [Perf]perf: skip unused Cosmos3 I2V conditioning latents (#4614) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
 2026-06-22 09:51:47 +00:00
NATUREandAlicia 83e3b129f0 [Bugfix] Fix missing memory metrics info for bagel (ar->dit diffusion) (#4477) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>
 2026-06-17 14:06:14 +08:00
汪志鹏Yufeng ZhuZhengyuan SuClaude Opus 4.6Cursor
 		46e0b71423 [Enhancement] Introduce High-Performance MoT (Mixture-of-Tokens) Kernels: Triton Implementation (#3960) 
Signed-off-by: Yufeng Zhu <yzhu802@gatech.edu>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Co-authored-by: Yufeng Zhu <yzhu802@gatech.edu>
Co-authored-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-16 03:08:02 +00:00
Sy03Yueqian LinHongsheng Liu
 		e27fd46ab9 [Perf] Optimize Higgs Audio v3 serving (#4204) 
Signed-off-by: Sy03 <1370724210@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-11 13:29:31 -07:00
Jianbao Huang 6cefe91370 [Perf] Enable fused RMSNorm for HunyuanImage3 (#3959) 2026-06-05 07:41:56 +00:00
Zhang JianandCanlin Guo b550709bba [TTS][Model] support MOSS-TTS series (#3420) 
Signed-off-by: Zhang <jianmusings@gmail.com>
Signed-off-by: Zhang Jian <jianmusings@gmail.com>
Signed-off-by: jian <jianmusings@gmail.com>
Signed-off-by: Canlin Guo <961750412@qq.com>
Co-authored-by: Canlin Guo <961750412@qq.com>
 2026-06-02 15:46:01 +08:00
+1
Zhou TaichangClaude Opus 4.7skfZheng WengangHongsheng LiuWeiQing ChenSYLARYueqian Linnainiu258JasonYueqian Lin
 		f92d84fcf9 [Rebase] Rebase to vllm releases/v0.22.0 (#3891) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: ZhengWG <zwg0606@gmail.com>
Signed-off-by: david6666666 <530634352@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Signed-off-by: nainiu258 <cperfect02@163.com>
Signed-off-by: nainiu258 <101917677+nainiu258@users.noreply.github.com>
Signed-off-by: JasonJ2021 <jasonj@zju.edu.cn>
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: skf <54565339+skf-1999@users.noreply.github.com>
Co-authored-by: Zheng Wengang <zwg0606@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: nainiu258 <101917677+nainiu258@users.noreply.github.com>
Co-authored-by: Jason <72191212+JasonJ2021@users.noreply.github.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-06-01 11:08:12 +08:00
ChenWenjingandGao Han 834abdaf8b [ci][nightly] Voxcpm2 performance benchmark (#3864) 
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-05-29 09:44:29 +08:00
WeiQing ChenlishunyangGao Han
 		126fb4b4a9 [Blackwell][1/N] Add SageAttention3 diffusion backend on blackwell(GB200/B200/RTX5090/PRO6000/DGX Spark available) (#3015) 
Signed-off-by: lishunyang <lishunyang12@163.com>
Signed-off-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Co-authored-by: lishunyang <lishunyang12@163.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-05-28 14:08:45 +00:00
SYLARfrauttauteffasuZhang Jian
 		373d3c7476 [Blackwell] Add CUDNN_ATTN and FLASHINFER_ATTN backends for diffusion (auto-route) (#3079) 
Signed-off-by: lishunyang <lishunyang12@163.com>
Signed-off-by: Zhang <jianmusings@gmail.com>
Signed-off-by: Zhang Jian <jianmusings@gmail.com>
Co-authored-by: frauttauteffasu <247219357+frauttauteffasu@users.noreply.github.com>
Co-authored-by: Zhang Jian <jianmusings@gmail.com>
 2026-05-28 11:09:25 +08:00
Dnoob a3802d1e5c [Doc][Benchmark] Rewrite benchmarks/README.md as repo-wide index (#1939) 
Signed-off-by: root <Dnoob>
Signed-off-by: Dnoob <dxpouo@gmail.com>
 2026-05-26 16:13:31 -10:00
dengyunyang 36eb8afba5 [HunyuanImage][End2End Performance CI] Add hunyuan end2end test (#3849) 
Signed-off-by: dengyunyang <584797741@qq.com>
 2026-05-26 16:45:05 +08:00
Yuanheng Zhao 2bc9cb7f13 [Feat] Support online quantization (fp8/int8) for LTX-2 (#3700) 
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com>
Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com>
 2026-05-21 19:52:21 +08:00
NumberWan 474710dda9 [Benchmark] Route i2i/ti2i to POST /v1/images/edits in diffusion_benchmark_serving (#3693) 
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
 2026-05-21 17:18:02 +08:00
Yueqian Lin fc8486cf20 [Doc][TTS] CosyVoice3 online docs + residual TTS yaml cleanup + remove VoxCPM v1 (#3748) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-05-20 15:24:06 +08:00
Yueqian LinandHongsheng Liu f99e8b2d3c [Bench] Add audio-streaming continuity metric for TTS (#3618) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-05-20 05:08:56 +00:00
bjf-frzandbjfwhite 1fa734419e [Refactor]Rename diffusion benchmark backend to endpoint (#3137) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
Signed-off-by: bjfwhite <baijingfan1@huawei.com>
Co-authored-by: bjfwhite <baijingfan1@huawei.com>
 2026-05-19 11:30:33 +08:00
Lancer dbc589dbca [Bugfix] fix diffusion quantization benchmarking for Omni outputs (#3653) 
Signed-off-by: Lancer <maruixiang6688@gmail.com>
 2026-05-18 17:23:40 +08:00
Yueqian Lin c4f859bf56 [CI] Harden Qwen3-TTS perf nightly: enable Base voice_clone, add c=64/128, 2-GPU split (#3491) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-05-14 14:26:59 +08:00
wangyuandHuang, Zeyu 6f2ad7b403 [Test] Unify L2/L3 test layout, Buildkite steps, and test helpers (#2556) 
Signed-off-by: wangyu <410167048@qq.com>
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Co-authored-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
 2026-05-08 14:12:23 +08:00
Chenguang Zheng eeb7e698c9 [Clean] Remove multi-replica Bagel CI and related docs/configs (#3407) 
Signed-off-by: Chenguang ZHENG <645327136@qq.com>
 2026-05-08 09:28:52 +08:00
Nick Cao e6466cf0e6 [Refactor] Replace and ban a few torch.cuda functions in favor of torch.accelerator replacements. (#3365) 
Signed-off-by: Nick Cao <ncao@redhat.com>
 2026-05-07 23:23:59 +08:00
Zheng WengangPeiqi YinyinpeyinpeiqiHongsheng LiuGao HanChenguang Zheng
 		1e5f288a91 [FEAT] support multi-stage deployment (#2396) 
Signed-off-by: ZhengWG <zwg0606@gmail.com>
Signed-off-by: Zheng Wengang <zwg0606@gmail.com>
Signed-off-by: Peiqi Yin <60515999+yinpeiqi@users.noreply.github.com>
Signed-off-by: yinpe <11810305@mail.sustech.edu.cn>
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>
Co-authored-by: Peiqi Yin <60515999+yinpeiqi@users.noreply.github.com>
Co-authored-by: yinpe <11810305@mail.sustech.edu.cn>
Co-authored-by: yinpeiqi <yinpeiqi809@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Chenguang Zheng <645327136@qq.com>
 2026-05-06 19:33:18 +08:00
Ting FUasukaqaq-sjaderHongsheng LiuGao Han
 		e49fbd8a2d [Feat] DiffusionEngine Support async batch infer (#2729) 
Signed-off-by: Semmer <semmer@live.cn>
Signed-off-by: jader <yjader@foxmail.com>
Signed-off-by: asukaqaq-s <1311722138@qq.com>
Co-authored-by: asukaqaq-s <1311722138@qq.com>
Co-authored-by: jader <yjader@foxmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-05-06 09:47:13 +12:00
Juan Pablo ZuluagaandJuanPZuluaga b64ab05d09 [TTS][SpeakerCacheManager] A global speaker cache manager for Voice Cloning (#2630) 
Signed-off-by: JuanPZuluaga <juanz9312@gmal.com>
Co-authored-by: JuanPZuluaga <juanz9312@gmal.com>
 2026-05-05 10:53:45 -04:00
Jared WenandHongsheng Liu 6e35bfbbae Benchmark and Bugfix for GLM-Image (#3024) 
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-04-25 20:49:19 +08:00
Yueqian Lin 39e3fd3d12 [Benchmark] Universal TTS benchmark: Qwen3-TTS + VoxCPM2 with 3 task types (voice-clone/default/design) (#2835) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-04-24 23:35:33 +08:00
amy-why-3459 b8bd6fbbdb [Refactor] Remove redundant benchmarks from Qwen3-Omni (#3108) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-04-24 21:22:03 +08:00
bjf-frz fb4204ebf6 [CI][Perf] Add Wan22 i2v perf nightly ci (#3063) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
 2026-04-24 16:58:45 +08:00
NumberWan ead87aa521 [BugFix] Surface diffusion metrics in chat completions; sanitize Bagel img2img mm kwargs (#2932) 
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
 2026-04-22 16:57:01 +00:00
wangyu e18cb89d5a [CI] Update test markers and configurations to use 'full_model' for L4 nightly tests (#2641) 
Signed-off-by: wangyu <410167048@qq.com>
 2026-04-22 15:14:20 +08:00
SYLARreidliu41xiaohajiayouAlex BrooksHongsheng Liu
 		a3903810fe [Config Refactor][2/N] Pipeline + Deploy Config Schema (#2383) 
Signed-off-by: lishunyang <lishunyang12@163.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: xiaohajiayou <75477391+xiaohajiayou@users.noreply.github.com>
Co-authored-by: Alex Brooks <albrooks@redhat.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-04-19 14:36:00 +12:00
Nick CaoandClaude f2edb81a59 [Cleanup] Remove dead runtime.defaults config parameters (#2343) 
Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-04-17 23:36:41 +08:00
IsleOfDawnlightCeleste-jqlyj-jjjYueqian Lin
 		4bf4c63147 Add voxcpm model support. (#2467) 
Signed-off-by: Celeste-jq <591998922@qq.com>
Signed-off-by: lyj-jjj <liuyingjun5@huawei.com>
Signed-off-by: IsleOfDawnlight <stellamou@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Celeste-jq <591998922@qq.com>
Co-authored-by: lyj-jjj <liuyingjun5@huawei.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
 2026-04-15 15:04:58 +08:00