tzhouam
723c550209
Integrate portable edge graph backends with Omni runtime
...
Extract versioned stage contracts, model shared CPU/iGPU/NPU memory constraints, and add explicit graph routes, reservations, output acknowledgements, and worker retirement. Fix native Windows shared-buffer lifetime and host paths. Document qualified hardware paths and remaining rollout gates.
2026-09-22 13:12:33 +08:00
tzhouam and Claude Fable 5.1
33de5d8629
Merge vLLM-Omni v0.29.0rc1 into edge-infer/omni-0.29.0
...
The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of
rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while
our 0.28-based tree needs seven modules v0.29.0 no longer has
(vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args,
openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant,
models.hunyuan_v1).
Ten files conflicted. Upstream's structure is the base everywhere; the
fork's features are re-attached to it:
- deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config
pops deploy_config (async_omni_engine, cli/serve); the deploy-profile
selector and the resolve_model_config_path chain it needs stay in
entrypoints/utils.py; stage_overrides takes rc1's dict type.
- config_factory keeps the lazy diffusion output-type import; the diffusion
registry import in rc1's new config/resolver.py is made lazy for the same
WP6 reason (TTS path must not load diffusion).
- requirements: our core/media split kept; rc1's transformers>=5.13 floor
and kernels==0.16.1 applied in core.txt / media.txt.
- TTS: rc1's split_granularity and bounded seed plus our output_mode=
codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation;
our private _conditioning_cache_salt is superseded by rc1's public
tts_adapters/base.py::conditioning_cache_salt.
- stage_runtime: rc1's single ExitStack G2 path (lock released before READY)
replaces our two-path block, with the four worker_phase names the init
profiler consumes still emitted; _cpu_placed / host admission kept.
Not yet run against vLLM 0.29.0: the tests listed in the W6 record.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-18 09:57:20 +08:00
tzhouam and Claude Fable 5.1
4e3e8a1ebc
[edge-infer] WIP snapshot: uncommitted M0/W1/W4/W5 edge work as of 2026-09-18
...
Verbatim copy of the edge/omni-edge working tree (tracked diffs + untracked
edge/local, tests/edge/local, docs/edge, pipeline files). Taken so that
vLLM-Omni v0.29.0rc1 can be merged on a branch without touching the
user's working tree. logs_edge/ and benchmark .npy outputs excluded.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-18 09:47:29 +08:00
Sy03
b02862beee
[Model] Optimize MOSS-TTS Local batched execution and streaming codec ( #7202 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
2026-09-09 13:02:12 -07:00
NATURE
5b002727b0
[Core] Split Omni connector model runner mixin ( #6903 )
...
Signed-off-by: natureofnature <wzliu@connect.hku.hk >
2026-09-09 13:49:43 +08:00
tzhouam and Claude Fable 5.1
35e43b9983
[Edge][WP3] Add CPU OmniPlatform (AR + generation stages on vLLM's CPU backend)
...
* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform)
selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed
(builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes
like vLLM does). NUMA-based memory queries, one logical device, no graph
wrapper (sentinel type for isinstance checks), inductor opt-in via
VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported.
* vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose
vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner /
CPUGenerationModelRunner compose the Omni runners with CPUModelRunner
(no graphs, no async omni output; sleep/wake return error ACKs).
* patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder
and prompt-embeds dtype follow the model dtype (float32/float16 on CPU).
* vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async
scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges).
* requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md.
Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads):
Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms,
RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline
committed under benchmarks/tts/edge_results/.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 11:02:14 +08:00
NancyFyong and NancyFyong
e74fe3c6e8
[Model] Add Audio8 TTS Preview 0.6B (DualAR, 44.1 kHz codec) ( #6157 )
...
Signed-off-by: NancyFyong <NancyFyong@users.noreply.github.com >
Co-authored-by: NancyFyong <NancyFyong@users.noreply.github.com >
2026-09-08 19:00:25 -07:00
Xiaobaixia
615f97b9c7
[Feature] Define the runner preprocess phase contract and regression coverage ( #7136 )
...
Signed-off-by: xiaobaixia222 <289982752@qq.com >
2026-09-06 17:20:04 +00:00
Zhou Taichang and Claude Fable 5
142151fef5
Parallel stage initialization (admission + SH/EX device locks) (Worker refactor 1/N) ( #5224 )
...
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk >
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-09-05 22:22:41 +08:00
5d7fdf9350
[Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental ( #6196 )
...
Signed-off-by: chickeyton <ngton2014@gmail.com >
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-09-05 08:20:21 +00:00
Inesh Reddy Chappidi and Gao Han
66d938100b
[Bugfix] Give exact-shape codec models an unpadded input_ids view ( #6775 )
...
Signed-off-by: Inesh Reddy <ineshreddy249@gmail.com >
Signed-off-by: IneshReddy249 <ineshreddy249@gmail.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-09-04 15:28:49 -07:00
akshatvishu
c41925689d
[BugFix][Higgs-Audio-V3] Fix transcript and concurrent sampling tests ( #6422 )
...
Signed-off-by: akshatvishu <akshatnayak197@gmail.com >
2026-08-30 08:52:15 +00:00
bb5bf8fd0a
[Bugfix] Fix colocate-async sleep admission race and deliver AR abort tokens ( #6367 )
...
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com >
Signed-off-by: Cursor <cursoragent@cursor.com >
Signed-off-by: Samit <285365963@qq.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Samit <285365963@qq.com >
2026-08-29 09:11:39 +00:00
amy-why-3459
da94c3ebe1
[Bugfix][MiniCPM-o] Bind omni-duplex video frames to the unit they close ( #6404 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-28 16:58:20 +08:00
Hongsheng Liu and hsliuustc0106
4245fd6955
[Fix] Restore MammothModa2 support ( #6694 )
...
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com >
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com >
2026-08-28 08:32:32 +08:00
Zhou Taichang and Claude Opus 5
651b03225b
[Rebase] Rebase to vllm 0.28.0 ( #6606 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-27 06:12:55 +00:00
Sy03 and sy03
432b2c6ca3
[Model] Add native full-duplex Nemotron VoiceChat serving ( #6089 )
...
Signed-off-by: Sysy <1370724210@qq.com >
Signed-off-by: sy03 <137072421@qq.com >
Co-authored-by: sy03 <137072421@qq.com >
2026-08-26 13:19:20 -07:00
wangyu
1e2a6446f7
[CI] Remove AudioX and MammothModa2 support ( #6353 )
...
Signed-off-by: wangyu <410167048@qq.com >
2026-08-23 04:22:05 +00:00
NATURE and amy-why-3459
4f7c08c0eb
[Bugfix][MiniCPM-o] Fix async-chunk snapshot replacement and prompt cleanup ( #6406 )
...
Signed-off-by: natureofnature <wzliu@connect.hku.hk >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-21 16:37:21 +08:00
Zhou Taichang and Claude Fable 5
10906e3a6d
[Refactor] Worker/ModelRunner runner correctness fixes (G2/N) ( #5452 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-20 16:47:50 +08:00
amy-why-3459 and Cursor
9c61d5b67b
[Bugfix][MiniCPM-o] Stop Talker mid-utterance truncation and load cod… ( #6346 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-08-20 11:50:07 +08:00
boatman
65b3d41b3a
[Migrate] Move sampling parameter overrides from legacy dispatches to TTS adapters ( #5272 )
...
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com >
2026-08-16 21:14:26 -07:00
Sy03 and Yueqian Lin
58007c0084
[New Model] Support IndexTTS 2.5 ( #5957 )
...
Signed-off-by: Sysy <1370724210@qq.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
2026-08-11 23:00:35 +00:00
Zhou Taichang and Claude Opus 5
2ce15ab7a0
[Rebase] Rebase to vllm 0.27.0 ( #5976 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-11 15:23:46 +00:00
e04210d690
[Hardware][MUSA] Enable Qwen3-Omni ModelOpt FP8 inference ( #5671 )
...
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-11 17:19:09 +08:00
Yueqian Lin and Sy0307
e788ef6e76
[Feature] PersonaPlex (Moshi-based full-duplex S2S): native vLLM port + duplex serving ( #4771 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Signed-off-by: Sy0307 <24841857+Sy0307@users.noreply.github.com >
Co-authored-by: Sy0307 <24841857+Sy0307@users.noreply.github.com >
2026-08-04 20:10:33 -07:00
Shenglei Fu and Claude
f24a6165ff
[Test] Prefix-cache passthrough test coverage prerequisite for W2 ( #4855 ) ( #5310 )
...
Signed-off-by: Shenglei Fu <sfu@confluent.io >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-04 20:03:57 -07:00
NATURE
1c4aca7f6e
[Bugfix][Hunyuan/Bagel]Avoid payload connector for KV-only senders ( #5744 )
...
Signed-off-by: natureofnature <wzliu@connect.hku.hk >
2026-08-04 14:40:27 +00:00
SYLAR and lishunyang12
6f98c807d2
[Bugfix] Restore BAGEL CFG KV transfers ( #5620 )
...
Signed-off-by: lishunyang12 <lishunyang03@gmail.com >
Co-authored-by: lishunyang12 <lishunyang03@gmail.com >
2026-08-01 15:06:05 +00:00
Sy03 and Hongsheng Liu
0feb92bdfa
[Bugfix][Minicpm] Support full-duplex three-stage serving ( #5380 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-07-29 01:58:34 +08:00
cbab1d4d8f
[Rebase] Rebase to vllm 0.26.0 ( #5443 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Signed-off-by: tzhouam <tzhouam@gmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: tzhouam <tzhouam@gmail.com >
2026-07-27 23:35:49 +08:00
Zhou Taichang and Claude Opus 4.8
776424f0de
[Refactor][WIP] Add missing test for modelrunner (G1/N) ( #5333 )
...
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-07-27 16:24:47 +08:00
boatman
76e4a78afc
[Bugfix][Higgs-Audio-V3] Propagate Sampling Parameters to Sampler ( #5406 )
...
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com >
2026-07-25 15:25:35 +00:00
rein yang and Sy03
ad3a162eb8
[perf] Minicpm o4.5 optimization ( #5228 )
...
Signed-off-by: R2-Y <ruiruyang2@gmail.com >
Co-authored-by: Sy03 <1370724210@qq.com >
2026-07-24 17:20:39 +08:00
05b794f29b
[Full Duplex] Feat: Support Full-Duplex realtime runtime & add MiniCPM-o 4.5 demo ( #3907 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
Signed-off-by: lishunyang12 <lishunyang03@gmail.com >
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Co-authored-by: lishunyang12 <lishunyang03@gmail.com >
Co-authored-by: Yueqian Lin <linyueqian@outlook.com >
Co-authored-by: StormMelody <22371184@zju.edu.cn >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-07-23 16:45:18 +08:00
vOv and Claude Opus 4.8
cc1939120f
[Bugfix] Skip non-positive scheduled token spans in omni stage runners ( #5269 )
...
Signed-off-by: cr-gao <gaochenrui@sjtu.edu.cn >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-07-22 16:01:45 +00:00
Bingyu (Spencer) Liu
dcf8b07d39
[Core] Cleanup Omni connector runtime and transfer backends ( #5096 )
...
Signed-off-by: spencerr221 <liubingyu62@gmail.com >
2026-07-22 09:44:15 +08:00
Yueqian Lin
da15a4cb1c
[Bugfix] Drop out-of-vocabulary stop ids from min-tokens masking (qwen3-tts min_tokens engine crash) ( #4971 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
2026-07-08 11:24:31 -07:00
amy-why-3459
9ceca4ec85
[Bugfix] Fix full-payload mm splitting for dual hidden/scheduled batch axes ( #4870 follow-up) ( #4910 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-07-06 15:08:05 +08:00
Yueqian Lin
d7b365df64
[Bugfix][Qwen3-TTS] Keep seeded residual MTP sampling batched ( #4889 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
2026-07-05 21:47:06 +08:00
Sy03 and Yueqian Lin
1b31f96240
[Bugfix] Fix Qwen3-TTS no-async-chunk payload splitting ( #4870 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Co-authored-by: Yueqian Lin <linyueqian@outlook.com >
2026-07-04 19:57:55 +00:00
Sy03 and Yueqian Lin
9eaca72382
[Perf][VoxCPM2] Add unified decode graph optimization ( #4606 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
2026-07-04 05:30:48 +00:00
Canlin Guo
a8b2bf11d2
[Feature] Support MOSS-TTS-Local-v1.5 Streaming ( #4804 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
2026-07-03 16:00:30 +08:00
Zhou Taichang
a560ed184d
[Rebase] Rebase to vllm v0.24.0 ( #4709 )
2026-06-30 19:41:06 +08:00
504698dec8
[Core] Performance fix for orchestrator bottleneck: separates inter-stage from client outputs ( #4527 )
...
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com >
Co-authored-by: Chenguang ZHENG <645327136@qq.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-06-29 17:43:36 +02:00
Chenguang Zheng and amy-why-3459
eaa16e1941
[Feature] Support Async Omni output materialization ( #4476 )
...
Signed-off-by: Chenguang ZHENG <645327136@qq.com >
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-06-22 16:07:07 +00:00
rein yang
afbcc51672
[Model] Aura support - Non async chunk path ( #4257 )
...
Signed-off-by: Rein Yang <ruiruyang2@gmail.com >
Signed-off-by: R2-Y <ruiruyang2@gmail.com >
2026-06-21 22:08:26 +08:00
8f869c8ea5
[Refactor] Output Processor Phase 2: separate multimodal output channel ( #1601 ) ( #2744 )
...
Signed-off-by: Megha Agarwal <agarwalmegha1308@gmail.com >
Signed-off-by: meghaagr13 <divyanshsinghvi@gmail.com >
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk >
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: meghaagr13 <divyanshsinghvi@gmail.com >
Co-authored-by: Boao Shi <aoibosh@connect.hku.hk >
Co-authored-by: Taichang Zhou <tzhouam@connect.ust.hk >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-06-12 15:19:14 +08:00
57227dc7f3
[Perf][Bugfix] qwen3-tts hot path: prefix-cache OOM guards + talker/orchestrator micro-opts ( #3689 )
...
Signed-off-by: JuanPZuluaga <juanz9312@gmal.com >
Co-authored-by: JuanPZuluaga <juanz9312@gmal.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-06-09 10:36:57 -07:00
c33ca60208
[Perf][VoxCPM2] Optimize VoxCPM2 high-concurrency decode throughput. ( #3882 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-06-03 23:08:32 +08:00
tzhouam