anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
2,900 Commits 235 Branches 31 Tags
073d31b9cd2cde8f50f69d4b7bdd60fcddfe6ded
Commit Graph
2900 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
tzhouamandClaude Fable 5.1 073d31b9cd [Edge][WP2] Add initialization timeline and cold-start profiler 
* vllm_omni/engine/init_timeline.py: InitTimeline phase collector; engine
  phases (stages_init, g3_device_lock, g2_spawn/ready) and worker phases
  (proc_entry, engine_core_init, init_device, load_model/load_weights,
  talker_mtp_init, attn_prewarm, cudagraph_capture, talker_mtp_capture,
  code2wav_cudagraph, predictor_setup_compile/warmup/cudagraph) recorded via
  worker_phase/timed_phase; workers append JSONL under
  VLLM_OMNI_INIT_TIMELINE_DIR (inherited at spawn) and the engine merges them.
  Enabled by VLLM_OMNI_INIT_TIMELINE=<json> or log_stats; table logged at INFO,
  partial table on orchestrator startup timeout, suggested timeouts logged.
* omni_base.py: named deploy profiles default to stage_init_timeout=900 /
  init_timeout=1500 unless given explicitly.
* gpu_model_runner._init_talker_mtp tolerates max_cudagraph_capture_size=None;
  gpu_ar_model_runner.shutdown skips CUDA graph teardown on CPU.
* benchmarks/tts/init_profile.py: runs stream_latency_bench --init-only per
  configuration (default, cold_cache, eager_stage1, eager_both,
  parallel_stage_init, edge, ...) and tabulates phase durations.
* stream_latency_bench: --parallel-stage-init / --deploy-profile flags; the
  resource poller no longer shadows Thread._stop and skips GPU sampling on CPU.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 11:01:39 +08:00
tzhouamandClaude Fable 5.1 f804bbd679 [Edge][WP0] Record Qwen3-TTS streaming baselines on 1x L20X 
Default deploy config (1 -> 25 frame schedule), 12 prompts x 3 repeats,
warm-up excluded, one GPU via `gpu run`:
  1.7B CustomVoice: TTFA p50 32.0 ms, playback-start p50 116 ms (stall at
    TTFA 85 ms), RTF_total 0.086, chunk gap p50 163 ms, init 271 s (cold caches)
  0.6B CustomVoice: TTFA p50 27.8 ms, playback-start p50 95 ms (stall 67 ms),
    RTF_total 0.077, chunk gap p50 146 ms, init 152 s
Per-request chunk arrivals, memory and the full summaries are in the JSON
(force-added: *.json is gitignored repo-wide).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 10:49:37 +08:00
tzhouamandClaude Fable 5.1 b22dcc59f7 [Edge][WP1] Add edge deploy profile and deploy_profile selector 
* vllm_omni/deploy/edge/qwen3_tts.yaml: single-device Qwen3-TTS profile with a
  ramped codec chunk schedule (2->4->8->16->25 frames) that removes the ~80 ms
  playback stall of the default 1->25 schedule, decode batch 1, max_num_seqs 4.
* deploy_profile="edge" (Omni/AsyncOmni kwarg, --deploy-profile CLI, headless
  serve) resolves vllm_omni/deploy/<profile>/<model_type>.yaml; mutually
  exclusive with deploy_config; explicit profiles never fall back silently.
* tests/helpers/stage_config.py: modify_stage_config anchors a relative
  base_config before writing the temp YAML so overlay-derived configs resolve.
* async_omni_engine.py also carries the init-timeline hooks used by WP2.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 10:27:33 +08:00
tzhouamandClaude Fable 5.1 b70207f57f [Edge][WP0] Add offline streaming-latency benchmark for chunked TTS 
Adds benchmarks/tts/stream_latency_bench.py (AsyncOmni driver reusing the
Qwen3-TTS example's prompt builder; --model override, warm-up/repeat,
--init-only, per-chunk arrivals, peak RSS and per-process GPU memory) and
benchmarks/tts/stream_latency_metrics.py (TTFA, underrun-free playback start,
stall at TTFA, RTF streamed/total, chunk gaps, percentiles) with CPU unit
tests. Results go under benchmarks/tts/edge_results/stream_latency/.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 10:24:53 +08:00
Lei KeandKrystalRay 7be014bce6 [Bugfix][NPU] Fix MiniMax-H3 INT8 quantization dispatch (#6876) 
Signed-off-by: KrystalRay <keeleiray@gmail.com>
Co-authored-by: KrystalRay <keeleiray@gmail.com>
 2026-09-07 14:50:44 +00:00
汪志鹏 9b6a308914 [Perf][Diffusion] Keep LoRA weights resident across activation cycles (#7195) 
Signed-off-by: princepride <wangzhipeng628@gmail.com>
 2026-09-07 14:27:24 +00:00
WeiQing Chen 30d6a0b4e6 [Bugfix][MiniMax-H3] Stabilize keyframe VAE encoding (#7191) 
Signed-off-by: david6666666 <530634352@qq.com>
Signed-off-by: David Chen <530634352@qq.com>
 2026-09-07 21:52:54 +08:00
Zeyu Huang | 黃澤宇andAlicia 4b96c57730 [skip ci][Doc] Module design doc for diffusion runtime (#6440) 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>
 2026-09-07 21:52:24 +08:00
Mu GuanLin 3204a0b2ad [Model][Perf] Optimize LTX Ulysses sequence parallelism (#7079) 
Signed-off-by: mglyn <1203789601@qq.com>
 2026-09-07 10:48:26 +00:00
Rahul SteigerandRahul Steiger f080f41e03 [Bugfix][Cosmos3] Auto-pad non-divisible Ulysses GEN sequences (#6918) 
Signed-off-by: Rahul Steiger <rsteiger@nvl72d188-T17.cm.cluster>
Co-authored-by: Rahul Steiger <rsteiger@nvl72d188-T17.cm.cluster>
 2026-09-07 08:58:28 +00:00
FredHuangNV 85b0bad2d3 [Frontend][Model] Add Cosmos3 control uploads to video API (#7027) 
Signed-off-by: Fred Huang <frhuang@nvidia.com>
 2026-09-07 01:38:56 -07:00
Rahul SteigerRahul SteigerRahul Steiger
 		677265b64b [Bugfix][Diffusion] Make Cosmos3 transfer image arrays writable (#6915) 
Signed-off-by: Rahul Steiger <rsteiger@nvl72d179-T17.cm.cluster>
Co-authored-by: Rahul Steiger <rsteiger@nvl72d179-T17.cm.cluster>
Co-authored-by: Rahul Steiger <rsteiger@aws-cmh-slurm-1-vscode-04.cm.cluster>
 2026-09-07 01:06:16 -07:00
Shenglei FuandClaude 91dfafcaab [Bugfix][Diffusion] Include guidance_scale_2_provided in the request-batch key (#7078) 
Signed-off-by: Shenglei Fu <sfu@confluent.io>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-07 15:58:19 +08:00
summerandAlicia 4b81f692d1 [Bugfix][Diffusion] Canonicalize cache_backend=None to "none" (#7041) 
Signed-off-by: summer <128961079+zhang-keliang@users.noreply.github.com>
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>
 2026-09-07 15:56:51 +08:00
WeiQing Chen b78c31eda6 [Doc] Update vLLM-Omni WeChat QR code (#7178) 
Signed-off-by: David Chen <530634352@qq.com>
 2026-09-07 06:28:30 +00:00
Lucas Ruan 34aa1d2afa [Doc] Add A100 recipe for Wan2.2-TI2V-5B (#7189) 
Signed-off-by: lucasruan1618 <huy40025071@gmail.com>
 2026-09-07 14:09:32 +08:00
BruceLoveDecimal 5cd925631c [Feature][LingBot World]stepwise execution (#6844) 
Signed-off-by: BruceLoveDecimal <liuqihao970610@gmail.com>
Signed-off-by: liuqihao <liuqihao970610@gmail.com>
 2026-09-07 13:38:23 +08:00
NumberWanandAlicia 7f32c326c6 [BugFix][Qwen-Image-Edit] restore txt_seq_lens RoPE width after #3588 regression (#5586) 
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>
 2026-09-07 12:10:47 +08:00
linzhenpl07andClaude Sonnet 5 38abc0ba28 [AR-Diffusion] Allocate K and V separately so the compiled path stops cloning the pool (#6463) 
Signed-off-by: linzhenpl07 <linzhenpl07@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
 2026-09-07 11:40:20 +08:00
Zeyu Huang | 黃澤宇 2df95de421 [Core][Diffusion] Refactor prompt updates into a modality-ready interaction framework (#6294) 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
 2026-09-07 11:40:04 +08:00
汪志鹏 eeee257913 [Bugfix][Diffusion] Restore the TRTLLM attention default for the MiniMax-H3 modular alias (#7162) 
Signed-off-by: princepride <wangzhipeng628@gmail.com>
 2026-09-07 07:21:21 +08:00
Xiaobaixia 615f97b9c7 [Feature] Define the runner preprocess phase contract and regression coverage (#7136) 
Signed-off-by: xiaobaixia222 <289982752@qq.com>
 2026-09-06 17:20:04 +00:00
NATUREandamy-why-3459 ff6e906a25 [Model] Preserve MiniCPM-o Talker full attention until capacity (#6799) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-09-06 22:25:35 +08:00
Hongsheng Liu 715b8b8746 [Model] MiniMax-H3: validate versioned text-conditioning handoff (#6720) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-06 21:45:10 +08:00
SYLARandhsliuustc0106 14d2d00536 [Bugfix] Fix Hunyuan Image-3 expert mapping contract (#7021) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-06 21:43:37 +08:00
汪志鹏andClaude Opus 5 af74a5a155 [Feature][MiniMax-H3] Support the full LightX2V Turbo LoRA matrix (#7062) 
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-09-06 20:23:36 +08:00
XenorynandClaude 039808e0d9 [Perf][Diffusion] Skip the Ulysses length all-gather when shards are padded equal (#5717) 
Signed-off-by: Xenoryn <timberh2000@outlook.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-06 16:23:30 +08:00
Mengjie ZhaoandCursor e8526cc552 [CI/Build] Stabilize Qwen3-TTS speaker-embedding tests against miss-EOS flake (#6924) 
Signed-off-by: Mengjie Zhao <zmj0129@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-09-06 07:44:23 +00:00
DMAN 06541d1f52 [Model] FP8 Support For Boogu-Image (#6925) 
Signed-off-by: Dmaner <2663515256@qq.com>
 2026-09-06 04:48:38 +00:00
Weiming LiaoandGao Han 1cd92553cf [CI][Bugfix] Use the matching checkpoint for Qwen3-TTS VoiceDesign tests (#7054) 
Signed-off-by: Weiming Liao <liaowm5@gmail.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-09-06 04:27:21 +00:00
prettygirlisnotme 32370968d0 [Model] Add Boogu-Image Turbo native DMD path (#6699) 
Signed-off-by: Dragonborn <1078757180@qq.com>
 2026-09-06 08:44:15 +08:00
Max Freedom Pollard 946b843e61 [Doc] Fix markdownlint findings in the serving API reference (#7111) 
Signed-off-by: Max Freedom Pollard <272618364+MaxFreedomPollard@users.noreply.github.com>
 2026-09-06 08:40:10 +08:00
Zeyu Huang | 黃澤宇asukaqaq-sHongsheng Liu
 		f26f9c3a7f [Diffusion] Implement paged AR↔DiT KV connector ("v1", reusing vllm's native mooncake) (#6310) 
Signed-off-by: asukaqaq-s <1311722138@qq.com>
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: asukaqaq-s <1311722138@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-09-06 05:41:54 +08:00
SYLAR ea3cf99894 [diffusion][feature] Add component-selective offload policies (#5929) 
Signed-off-by: lishunyang12 <lishunyang12@163.com>
 2026-09-06 05:40:09 +08:00
Samit 44d3ae100a [Bugfix][Diffusion] Fuse distilled LoRA weights before HSDP sharding (#6948) 
Signed-off-by: samithuang <285365963@qq.com>
Signed-off-by: Samit <285365963@qq.com>
 2026-09-06 01:29:59 +08:00
汪志鹏andClaude Opus 5 ae70479619 [Bugfix] Drop the removed diffusion_batch_size kwarg from the stage-init test (#7101) 
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-09-05 17:05:33 +00:00
Zhou TaichangandClaude Fable 5 142151fef5 Parallel stage initialization (admission + SH/EX device locks) (Worker refactor 1/N) (#5224) 
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-09-05 22:22:41 +08:00
chickeytonClaude Fable 5Gao Han
 		5d7fdf9350 [Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental (#6196) 
Signed-off-by: chickeyton <ngton2014@gmail.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-09-05 08:20:21 +00:00
Hongsheng Liu d4096e01fe [Model] Add native MAGI-2 Preview diffusion support (#5918) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-05 15:32:30 +08:00
vOv 13c617a140 [Model] Add sequence parallelism to SANA-Video 2B (#5940) 
Signed-off-by: cr-gao <gaochenrui@sjtu.edu.cn>
 2026-09-05 15:22:53 +08:00
boatman 66bbebd9d3 [Bugfix] Fix fallback loading for IndexTTS external models (#6137) 
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
 2026-09-04 19:58:33 -07:00
BruceLoveDecimalandClaude Opus 5 1e74807c35 [Feature][Moss-TTS] Reference Audio Preprocessing (#4982) 
Signed-off-by: BruceLoveDecimal <39156883+BruceLoveDecimal@users.noreply.github.com>
Signed-off-by: liuqihao <liuqihao970610@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-09-04 16:36:26 -07:00
akshatvishuandandyluo7 ae57e406fd [Perf][CI] Use Whisper on GPU when memory permits (#5675) 
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
 2026-09-04 16:35:48 -07:00
LOGO127 b6b992ac7b [Bugfix] Fix Higgs Audio v3 voice-clone token validation (#7065) 
Signed-off-by: luozijian <luozijian0924@gmail.com>
 2026-09-04 16:06:47 -07:00
Inesh Reddy ChappidiandGao Han 66d938100b [Bugfix] Give exact-shape codec models an unpadded input_ids view (#6775) 
Signed-off-by: Inesh Reddy <ineshreddy249@gmail.com>
Signed-off-by: IneshReddy249 <ineshreddy249@gmail.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-09-04 15:28:49 -07:00
boatman 348fd0fc07 [Feat][dots.tts] Add Online Serving Support (#6235) 
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
 2026-09-04 14:44:10 -07:00
TriggerandGao Han ea14b04e9f [Bugfix][BAGEL] Restore original denoise step count (#7049) 
Signed-off-by: Trigger <Meng.Bo.Wang@outlook.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-09-04 17:05:05 +00:00
Weiming Liao 0f9ee6afd8 [CI][Bugfix] Normalize spoken units in text comparison (#6947) 
Signed-off-by: Weiming Liao <liaowm5@gmail.com>
 2026-09-04 15:18:01 +00:00
SYLARandhsliuustc0106 22cefe04b4 [Bugfix] Update Hub kernel dependency floor (#7020) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-04 22:15:55 +08:00
SYLARandhsliuustc0106 7493cbd3c2 [Bugfix] Fix Qwen3-Omni MoE backend selection (#7019) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-04 21:16:58 +08:00