tzhouam
723c550209
Integrate portable edge graph backends with Omni runtime
...
Extract versioned stage contracts, model shared CPU/iGPU/NPU memory constraints, and add explicit graph routes, reservations, output acknowledgements, and worker retirement. Fix native Windows shared-buffer lifetime and host paths. Document qualified hardware paths and remaining rollout gates.
2026-09-22 13:12:33 +08:00
tzhouam and Claude Fable 5.1
33de5d8629
Merge vLLM-Omni v0.29.0rc1 into edge-infer/omni-0.29.0
...
The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of
rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while
our 0.28-based tree needs seven modules v0.29.0 no longer has
(vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args,
openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant,
models.hunyuan_v1).
Ten files conflicted. Upstream's structure is the base everywhere; the
fork's features are re-attached to it:
- deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config
pops deploy_config (async_omni_engine, cli/serve); the deploy-profile
selector and the resolve_model_config_path chain it needs stay in
entrypoints/utils.py; stage_overrides takes rc1's dict type.
- config_factory keeps the lazy diffusion output-type import; the diffusion
registry import in rc1's new config/resolver.py is made lazy for the same
WP6 reason (TTS path must not load diffusion).
- requirements: our core/media split kept; rc1's transformers>=5.13 floor
and kernels==0.16.1 applied in core.txt / media.txt.
- TTS: rc1's split_granularity and bounded seed plus our output_mode=
codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation;
our private _conditioning_cache_salt is superseded by rc1's public
tts_adapters/base.py::conditioning_cache_salt.
- stage_runtime: rc1's single ExitStack G2 path (lock released before READY)
replaces our two-path block, with the four worker_phase names the init
profiler consumes still emitted; _cpu_placed / host admission kept.
Not yet run against vLLM 0.29.0: the tests listed in the W6 record.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-18 09:57:20 +08:00
tzhouam and Claude Opus 5
a51461c131
[Edge][CPU] Admit CPU stages against host memory so parallel init can run
...
`parallel_stage_init` was structurally impossible on a CPU deployment.
`resolve_stage_physical_devices` returns None for `devices: "cpu"`, and the
admission resolver fails closed on None -- correctly, for a GPU replica whose
devices cannot be accounted -- so every CPU pipeline raised
StageAdmissionError and fell back to sequential stage startup. The edge
profile sets parallel_stage_init: true, so that profile was broken on CPU
rather than merely slower.
Measured on Qwen3-TTS 0.6B (Xeon 8480C, 24 threads, init only):
sequential 64.57 s
parallel 30.54 s 2.1x
The fix is not to skip admission. The per-device ledger is vacuous on CPU --
there is no device to overcommit -- but the resource concurrent CPU init
contends for is host memory, and two stages allocating KV caches at once can
exhaust it exactly as two GPU stages exhaust a card. So there is now a host
ledger with the same discipline: sum the declared KV budgets, compare against
available RAM with a margin, refuse to start if it does not fit, and fail
closed on a stage that declares no budget at all. The arithmetic is pure
(`evaluate_host`) so it is testable without a device, matching the existing
design.
Which ledger applies is decided by where the stages are *placed*, not by what
platform the process reports: a first version keyed on
`current_platform.is_cpu()` and broke five existing GPU-path tests, because a
CPU-only test host still has to admit GPU-shaped plans correctly. It now
routes on the replicas' own `devices`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-12 21:16:11 +08:00
Zhou Taichang and Claude Opus 5
aff7d64948
[Rebase] Rebase to vLLM 0.29.0 ( #7230 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-10 12:08:28 +08:00
Gao Han and amy-why-3459
6e63ea875e
[Bugfix] Allow LLM replicas on different GPUs to initialize concurrently ( #7292 )
...
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-09-10 09:54:20 +08:00
kunkun
2f1845fa4e
fix: reserve Qwen3-Omni NVFP4 backend fix ( #7200 )
...
Signed-off-by: kunkunblueberry <1833921874@qq.com >
2026-09-09 20:35:16 +08:00
Zheng Wengang and amy-why-3459
7c1b1e4189
[Bugfix][Engine] Group overlapping device stages into one sequential init component ( #7328 )
...
Signed-off-by: ZhengWG <zwg0606@gmail.com >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-09-09 20:25:41 +08:00
tzhouam and Claude Fable 5.1
073d31b9cd
[Edge][WP2] Add initialization timeline and cold-start profiler
...
* vllm_omni/engine/init_timeline.py: InitTimeline phase collector; engine
phases (stages_init, g3_device_lock, g2_spawn/ready) and worker phases
(proc_entry, engine_core_init, init_device, load_model/load_weights,
talker_mtp_init, attn_prewarm, cudagraph_capture, talker_mtp_capture,
code2wav_cudagraph, predictor_setup_compile/warmup/cudagraph) recorded via
worker_phase/timed_phase; workers append JSONL under
VLLM_OMNI_INIT_TIMELINE_DIR (inherited at spawn) and the engine merges them.
Enabled by VLLM_OMNI_INIT_TIMELINE=<json> or log_stats; table logged at INFO,
partial table on orchestrator startup timeout, suggested timeouts logged.
* omni_base.py: named deploy profiles default to stage_init_timeout=900 /
init_timeout=1500 unless given explicitly.
* gpu_model_runner._init_talker_mtp tolerates max_cudagraph_capture_size=None;
gpu_ar_model_runner.shutdown skips CUDA graph teardown on CPU.
* benchmarks/tts/init_profile.py: runs stream_latency_bench --init-only per
configuration (default, cold_cache, eager_stage1, eager_both,
parallel_stage_init, edge, ...) and tabulates phase durations.
* stream_latency_bench: --parallel-stage-init / --deploy-profile flags; the
resource poller no longer shadows Thread._stop and skips GPU sampling on CPU.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 11:01:39 +08:00
fdaf4e31bd
Add vLLM-Omni library info to Hugging Face Hub requests ( #5381 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-08 10:08:38 -06:00
Shuolei Wang
5378b77be4
[Model][JoyAI-VL-Interaction] Add native multi-stage pipeline ( #5352 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
2026-09-08 05:07:08 -07:00
LHXuuu
c9c25b33c3
[Frontend] Add server-side VAD for turn-based omni models ( #6618 )
...
Signed-off-by: LHXuuu <xulianhao.xlh@antgroup.com >
2026-09-08 03:17:00 -07:00
andyluo7 and wangyu
b58ff5cb8b
[CI/Build] Wait for duplex reaper recovery in retry test ( #7225 )
...
Signed-off-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com >
2026-09-08 17:10:51 +08:00
Chenguang Zheng
087ac46318
[Refactor 1/N] Centralize Omni config resolution ( #5140 )
...
Signed-off-by: Chenguang ZHENG <645327136@qq.com >
2026-09-08 10:30:26 +08:00
BruceLoveDecimal
5cd925631c
[Feature][LingBot World]stepwise execution ( #6844 )
...
Signed-off-by: BruceLoveDecimal <liuqihao970610@gmail.com >
Signed-off-by: liuqihao <liuqihao970610@gmail.com >
2026-09-07 13:38:23 +08:00
汪志鹏 and Claude Opus 5
ae70479619
[Bugfix] Drop the removed diffusion_batch_size kwarg from the stage-init test ( #7101 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-05 17:05:33 +00:00
Zhou Taichang and Claude Fable 5
142151fef5
Parallel stage initialization (admission + SH/EX device locks) (Worker refactor 1/N) ( #5224 )
...
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk >
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-09-05 22:22:41 +08:00
5d7fdf9350
[Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental ( #6196 )
...
Signed-off-by: chickeyton <ngton2014@gmail.com >
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-09-05 08:20:21 +00:00
SYLAR and hsliuustc0106
7493cbd3c2
[Bugfix] Fix Qwen3-Omni MoE backend selection ( #7019 )
...
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com >
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com >
2026-09-04 21:16:58 +08:00
Arifuzzaman Joy
3cddda9781
[Bugfix] Prevent build_engine_args_dict from mutating stage_config.engine_args ( #6783 )
...
Signed-off-by: Arifuzzamanjoy <s1710374103@ru.ac.bd >
2026-09-01 16:45:28 +00:00
Andy Zhou
2298359f67
[Bugfix][Diffusion] Preserve stage attention backend ( #6645 )
...
Signed-off-by: AndyZhou952 <jzhoubc@connect.ust.hk >
2026-09-01 20:18:44 +08:00
Yuxuan Rho and Hongsheng Liu
7992c2213e
[Config] Remove diffusion_batch_size and use max_num_seqs instead. ( #6484 )
...
Signed-off-by: LyxWxj <yuxuanlou621@foxmail.com >
Signed-off-by: Yuxuan Rho <104440472+LyxWxj@users.noreply.github.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-09-01 01:02:54 +00:00
TJian
ded8934626
[AMD] [CI] Fix tests for v0.28.0 ( #6830 )
...
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
2026-08-31 09:10:27 +08:00
Gao Han
f7f48929db
[Bugfix][Diffusion] Restore inline single-stage execution ( #6813 )
...
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk >
2026-08-30 21:40:47 +08:00
Yueqian Lin
a6b559cf54
[Bugfix][MiniMax-Music3] Resolve stage subdirs against a real snapshot ( #6640 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
2026-08-30 06:51:24 +00:00
21dd4b33ac
[Bugfix][MiniCPM-o] Stabilize native duplex streaming and Stage-1 handoffs ( #6529 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
Co-authored-by: Nathan Price <nathan@abridge.com >
Co-authored-by: Eric(Qun) <186020953+0xtoward@users.noreply.github.com >
2026-08-29 13:05:24 -07:00
Sy03
3adee400a0
[Bugfix][Qwen3-TTS] Handle codec generations that exhaust their token budget ( #6728 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
2026-08-29 08:59:05 -07:00
bb5bf8fd0a
[Bugfix] Fix colocate-async sleep admission race and deliver AR abort tokens ( #6367 )
...
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com >
Signed-off-by: Cursor <cursoragent@cursor.com >
Signed-off-by: Samit <285365963@qq.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Samit <285365963@qq.com >
2026-08-29 09:11:39 +00:00
d94061201d
[Bugfix][Engine] Make abort idempotent during shutdown ( #6327 )
...
Signed-off-by: Allen Wu <allenwu2795@gmail.com >
Co-authored-by: TRAE CLI <noreply@bytedance.com >
Co-authored-by: TRAE CLI <traecli@bytedance.com >
2026-08-28 19:51:25 -07:00
Sy03
009b992c5b
[Bugfix][MiniCPM-o] Isolate duplex handoff failures and barge-in ( #6170 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
2026-08-28 15:46:41 +08:00
b6a4baa659
[Feature] Support text encoder Disaggregation for H3 ( #5885 )
...
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com >
Signed-off-by: lishunyang12 <lishunyang12@163.com >
Signed-off-by: yuanwu <yuan.wu@intel.com >
Co-authored-by: lishunyang12 <lishunyang12@163.com >
Co-authored-by: yuanwu <yuan.wu@intel.com >
2026-08-28 14:27:05 +08:00
NATURE
54c1fc64b7
[Bugfix] Gate async-chunk segment resume by session mode ( #6680 )
...
Signed-off-by: natureofnature <wzliu@connect.hku.hk >
2026-08-27 18:52:09 +08:00
Zhou Taichang and Claude Opus 5
651b03225b
[Rebase] Rebase to vllm 0.28.0 ( #6606 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-27 06:12:55 +00:00
c2723979c0
[Bugfix] Validate per-stage device layout before spawning workers ( #5003 ) ( #5742 )
...
Signed-off-by: ChoHee15 <cc5281@126.com >
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-08-26 19:25:22 +00:00
Yueqian Lin and Yueqian Lin
8cfede4309
[Perf][Engine] Event-driven orchestration loop (opt-in) — S1 of #4855 ( #5221 )
...
Signed-off-by: Yueqian Lin <gao@yueqian.us >
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Co-authored-by: Yueqian Lin <gao@yueqian.us >
2026-08-26 22:15:25 +08:00
psv666 and amy-why-3459
5ec576ab4c
[Bugfix] Fix missing Realtime audio completion ( #6564 )
...
Signed-off-by: psv666 <2693925048@qq.com >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-26 18:27:25 +08:00
7ae15c74c8
text and image metric ( #6150 )
...
Signed-off-by: zhuhh97 <zhuhonghan@huawei.com >
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk >
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-08-26 02:54:45 +00:00
Alex Brooks
0eb47b76d0
[Bugfix] Handle DoS Overflow Cases ( #6598 )
...
Signed-off-by: Alex Brooks <albrooks@redhat.com >
2026-08-25 16:09:03 +00:00
NumberWan
bc8ea489ef
[BugFix] Qwen-image performance regressed - Avoid mapping diffusion_batch_size onto scheduler max_num_seqs ( #6525 )
...
Signed-off-by: NumberWan <wantszkin2003@gmail.com >
2026-08-25 03:36:11 -07:00
wy17003 and Hongsheng Liu
b454e34966
[Refactor][2/N Scheduler]Replace _FULL_PAYLOAD_INPUT_STAGES with a resolved stage transport capability. ( #6149 )
...
Signed-off-by: wy17003 <yaow17003@gmail.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-08-25 15:24:59 +08:00
f0808c9793
[Bugfix] Fix remote replica membership lifecycle races ( #5277 )
...
Signed-off-by: hbhflw2000 <417911774@qq.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
Co-authored-by: Samit <285365963@qq.com >
2026-08-25 10:34:59 +08:00
wangyu and amy-why-3459
4af0ddc1c2
[Test] Cleanup for helpers ( #6523 )
...
Signed-off-by: wangyu <410167048@qq.com >
Signed-off-by: [Your Name] <your.email@example.com >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-25 09:10:17 +08:00
Nguyen Kha Nhat Long and Cursor
d15b7ca3e1
[Feature]: Support pause / resume and sleep / wake for AR stages in AsyncOmni ( #6084 )
...
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-08-24 16:21:32 +08:00
akshatvishu
39c16d75a9
[tts][bugfix] Fix CosyVoice3 sampling and stage handoff ( #6424 )
...
Signed-off-by: akshatvishu <akshatnayak197@gmail.com >
2026-08-22 21:20:57 -07:00
Yash Jain
c6e38860e9
[Bugfix][Core] Fix replica device split when a stage omits devices ( #5445 )
...
Signed-off-by: Yash Jain <yashjainjain1704@gmail.com >
2026-08-21 00:44:03 +08:00
Inesh Reddy Chappidi
cd3867c373
[Model] Suppress silence codec tokens for the first N Qwen3-TTS decod… ( #5048 )
...
Signed-off-by: Inesh Reddy <ineshreddy249@gmail.com >
2026-08-20 09:22:37 -07:00
amy-why-3459 and Cursor
9c61d5b67b
[Bugfix][MiniCPM-o] Stop Talker mid-utterance truncation and load cod… ( #6346 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-08-20 11:50:07 +08:00
Wang fuyin and Hongsheng Liu
e1c5f59aab
[Config] Reuse vLLM configs in VllmOmniConfig ( #6050 )
...
Signed-off-by: Acerak01-fy <wfy2003324@163.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-08-18 21:27:58 +08:00
Yueqian Lin
b283215c28
[Model] Add MiniMax Music 3 text-to-music ( #6186 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
2026-08-18 04:33:13 +00:00
Yueqian Lin and Hongsheng Liu
611c083587
[BugFix][TTS] R1.1-R1.4: turn silent async-chunk failures into visible ones ( #6033 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-08-17 10:51:17 +08:00
hurukawa
2a315e1a2d
[Perf] Support request-level batching for Wan2.2 pipelines ( #5676 )
...
Signed-off-by: nagisa-kun <1434936049@qq.com >
Signed-off-by: nagisa <1434936049@qq.com >
2026-08-15 23:58:36 +08:00
tzhouam