anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
3,473 Commits 235 Branches 31 Tags
ISSUE-5
Commit Graph
2043 Commits
Author SHA1 Message Date
admin 321e8bb47f fix(quant): load offline-quantized compressed-tensors checkpoints directly on CPU under offload 
The diffusion loader treated any quant config without the MXFP8/
is_checkpoint_quantized markers as online quantization: it loaded every
component on the accelerator before offloading, so serving the int8
weight-only compressed-tensors artifact OOM'd at model construction
(encoder 17 GiB + VAE + packed DiT skeleton > 24 GiB). compressed-tensors
is always a serialized format with no on-load quantization pass — detect
it (through component wrappers too) and load on the host, round-tripping
each quantized module through the accelerator only for the post-load
Marlin repack, which has no CPU path.
 2026-10-08 07:36:01 -04:00
admin e778d21f91 probe: int8 weight-only compressed-tensors DiT artifact kit (issue 5) 
quantize_dit_int8wo.py builds the offline W8A16 artifact (per-channel
symmetric int8, pack-quantized via compressed-tensors' own packer,
per-component config wrapper so encoder/VAE stay BF16, digest manifest);
serve.sh/probe.py/gate.py/psnr.py adapt the issue-2 kit for the
structure-gated capacity matrix (1-4 refs x seeds 2301-2304).
 2026-10-08 07:01:55 -04:00
admin 2f8363c5fc fix(quant): resolve hyphenated registry methods and disk-dict from_config in quant factory 
_normalize_method_name canonicalizes dashes to underscores for _OVERRIDES
lookup, but vLLM's QUANTIZATION_METHODS keeps historical spellings
('compressed-tensors'), so both --quantization compressed-tensors and
checkpoint auto-detection raised 'Unknown quantization method'. Resolve
the normalized name back to the registry spelling, and fall back to
from_config for registry configs whose __init__ does not take the
serialized schema (compressed-tensors config_groups/format/ignore).
 2026-10-08 07:00:05 -04:00
hurukawa 3e705fcdb1 [Model] Add JoyAI-Image-Edit pipeline support (#4112)
Build Wheel / build (3.12) (push) Failing after 36s
Details
Build Wheel / build (3.11) (push) Failing after 36s
Details
pre-commit / pre-commit (push) Failing after 44s
Details 
Signed-off-by: nagisa-kun <1434936049@qq.com>
Signed-off-by: nagisa <1434936049@qq.com>
Signed-off-by: hurukawa <61525444+nagisa-kunhah@users.noreply.github.com>
 2026-10-08 08:40:55 +08:00
Yueqian Lin 6a70bda0f6 [Bugfix][Qwen3-Omni] Default the talker voice to Ethan (#8568) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-10-07 17:07:53 -07:00
Yueqian Lin 1c71a2c03a [Bugfix] Report sample rate and channels for streamed speech PCM (#8567) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-10-07 17:07:49 -07:00
Alex Brooks 8484d85d1f [Bugfix] Fix Sender Address Propagation (#8599)
Build Wheel / build (3.12) (push) Failing after 1m16s
Details
Build Wheel / build (3.11) (push) Failing after 1m17s
Details
pre-commit / pre-commit (push) Failing after 59s
Details 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
 2026-10-07 15:04:01 -06:00
Nick CaoandCodex 744c202d86 [BugFix] Optimize turn-based /v1/realtime auto truncation (#8566) 
Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-07 15:57:09 -04:00
Alex Brooks f3743ebdb6 [Bugfix] Fix In Place Compile Overwrites for Qwen3Omni Embeddings with TP > 1 (#8532) 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
 2026-10-07 17:01:45 +00:00
amy-why-3459 3345fb7498 [CI/Build] Fix CPU groups in distributed VAE test mocks (#8591) 2026-10-07 10:24:15 -06:00
haic0andandyluo7 c8e69c08fb [CI/Build][ROCm] Stabilize R2-01 platform selection (#8561) 
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
 2026-10-07 08:03:03 -07:00
MaciejBalaNV e54e185593 [feature] Reduce memory usage with SP (#7728) 
Signed-off-by: Maciej Bala <mbala@nvidia.com>
 2026-10-07 22:36:32 +08:00
NumberWanIsotr0pyTianyu GuoCursorSamitHuangGao Han
 		3dc35694b3 [New model] Qwen-Image-2.1 support (rebase + review follow-ups) (#8099) 
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Signed-off-by: SamitHuang <285365963@qq.com>
Signed-off-by: Samit <285365963@qq.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: SamitHuang <285365963@qq.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-10-07 21:23:59 +08:00
QianCyrusandHongsheng Liu 8ef77be66e [Model] Avoid repeated attention and MLP work in Klein TP (#8553) 
Signed-off-by: QianCyrus <101633534+QianCyrus@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-10-07 21:21:05 +08:00
NATURE b21df3bcb6 [Frontend] Share regular and duplex chat initialization (#7646) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
 2026-10-07 10:49:34 +00:00
amy-why-3459 de253412b0 [Core] Defer generation payload host waits to output consumption (#8540) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-10-07 17:20:54 +08:00
amy-why-3459 9482bf1e8c [Model] MiniCPM-o 4.5: enable fused CFM body by default on CUDA (#8557) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-10-07 10:25:39 +08:00
DanaerLeezijianc2Yifan Tianamy-why-3459
 		712b90de9c [Perf][Qwen3-Omni] Capture the thinker image encoder with CUDA graphs (part of #7452) (#8215) 
Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
Co-authored-by: zijianc2 <157244773+zijianc2@users.noreply.github.com>
Co-authored-by: Yifan Tian <221685243+Sendoh-code@users.noreply.github.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-10-07 09:29:59 +08:00
Yueqian Lin ed6bc5ae0c [Perf][PersonaPlex] Batch live duplex appends and replay Mimi codec frames from CUDA graphs (#8356) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-10-07 00:05:08 +00:00
haic0Cursorandyluo7
 		e61cd4290f [CI/Build][ROCm] Move Pi0.5 CPU coverage to AMD nightly (#8559) 
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
 2026-10-06 11:05:14 -07:00
haic0Cursorakshatvishuandyluo7
 		a0d915254b [CI][ROCm] Add and stabilize HunyuanImage3 nightly coverage (#7934) 
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: akshatvishu <akshatnayak197@gmail.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
 2026-10-06 09:54:27 -07:00
chen hongweiWang ZhipengClaude Opus 5.5
 		bad88bca25 [Frontend] MiniMax-H3: Add latent-mask editing to the ComfyUI extension (#7575) 
Signed-off-by: chen hongwei <1792043268@qq.com>
Signed-off-by: Wang Zhipeng <wangzhipeng628@gmail.com>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Co-authored-by: Wang Zhipeng <wangzhipeng628@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-06 16:32:40 +00:00
Sy03 7be49aa0ce [Model] Default MOSS Local 1.5 to adaptive CUDA MRV2 with MPS (#8438) 
Signed-off-by: Sy03 <1370724210@qq.com>
 2026-10-06 08:20:46 -07:00
Dong1017andamy-why-3459 ee9ab9ea88 [Perf] Fuse AdaLayerNorm CUDA eager path with Triton (#8231) 
Signed-off-by: GUOGUO <xwdong1998@163.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-10-06 15:11:45 +00:00
BeatSeatandamy-why-3459 182b77da4c [Perf][MiniCPM-o] Stage 0 streaming audio/vision encoder CUDA graphs and candidate sampling (#8430) 
Signed-off-by: BeatSeat <wendavid552@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-10-06 07:53:03 -07:00
Nick CaoandCodex 65699fe94c [Core] Generalized turn-based /v1/realtime implementation (#8339) 
Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-06 22:07:31 +08:00
chen hongweiWang ZhipengClaude Opus 5.5
 		25fb257667 [Model] MiniMax-H3: accept raw video masks and resolve the latent grid server-side (#7947) 
Signed-off-by: chen hongwei <1792043268@qq.com>
Signed-off-by: Wang Zhipeng <wangzhipeng628@gmail.com>
Co-authored-by: Wang Zhipeng <wangzhipeng628@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-06 12:46:35 +00:00
Nikita OsterovleffffArtem Borisov
 		5f95115e70 Kandinsky6 (#8537) 
Signed-off-by: leffff <levnovitskiy@gmail.com>
Signed-off-by: Artem Borisov <divotionchannel@gmail.com>
Co-authored-by: leffff <levnovitskiy@gmail.com>
Co-authored-by: Artem Borisov <divotionchannel@gmail.com>
 2026-10-06 15:55:15 +08:00
haic0andCursor 1f63e97a40 [CI][ROCm] Add Cosmos3 nightly coverage (#7933) 
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-10-06 00:18:44 -07:00
QianCyrus c2cd13c009 [Core] Add batch parallel VAE decode (#8162) 
Signed-off-by: QianCyrus <101633534+QianCyrus@users.noreply.github.com>
 2026-10-06 13:51:35 +08:00
haic0andCursor 2bcd1e449a [CI][ROCm] Add LingBot nightly coverage (#7935) 
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-10-06 05:31:51 +00:00
QianCyrus 099f9b553d [Model] Optimize Wan VAE first-frame decoding with Conv2d (#8179) 
Signed-off-by: QianCyrus <101633534+QianCyrus@users.noreply.github.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-10-06 11:44:34 +08:00
haic0andCursor e4a8b2e858 [CI/Build][ROCm] Reuse AITER cache for Mammoth attention tests (#8527) 
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-10-05 18:35:01 -07:00
Tianyao Wuandamy-why-3459 cc444b8875 [Bugfix][MiniCPM-o] Take the duplex vision tile from the image-processor config (#8462) 
Signed-off-by: Tianyao Wu <rayroy31@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-10-06 09:04:21 +08:00
Linze Shi aad3090877 [Model][PersonaPlex] Vectorize prefill embedding construction (#7481) 
Signed-off-by: Linze-Shi <linzeshi0@gmail.com>
 2026-10-05 18:52:41 +00:00
LOGO127 ae430dddc0 [Model][PersonaPlex] Encode only consumed Mimi codebooks (#7399) 
Signed-off-by: luozijian <luozijian0924@gmail.com>
 2026-10-05 18:49:33 +00:00
BeatSeatandamy-why-3459 79562e97ad [Perf][MiniCPM-o] Code2Wav resident Euler slot pool, fused DiT body, and exact HiFT graphs (#8443) 
Signed-off-by: BeatSeat <wendavid552@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-10-05 18:36:28 +00:00
Wenbo Ji　嵇文博 539c533762 [Bugfix] Keep client error status in the single-GPU diffusion executor (#8474) 
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
 2026-10-05 13:09:34 -04:00
amy-why-3459 9146284c16 [Perf] Add CUDA graphs for MiniCPM-o 4.5 input encoders (#8332) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-10-05 15:01:07 +00:00
Yufeng He a0e626caaf [Bugfix] Validate sleep stage_ids before blocking admission (#8501) 
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
 2026-10-05 10:27:53 -04:00
Sy03 023716ecc4 [Bugfix] Do not assume a stream decoder in MRv2 eager-MTP preprocess (#8484) 
Signed-off-by: Sy03 <1370724210@qq.com>
 2026-10-05 09:29:38 -04:00
JohnnyJohn Nunezamy-why-3459
 		5351cd6645 [Bugfix] Qwen3-Omni thinker: create deepstack buffers on the real device (#8461) 
Signed-off-by: John Nunez <johnnynunez@users.noreply.github.com>
Co-authored-by: John Nunez <johnnynunez@users.noreply.github.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-10-05 21:00:04 +08:00
ztyan 091b256674 [Model][Perf] IndexTTS2: drop per-step hidden D2H in latent mode (#8478) 
Signed-off-by: ztyan <191593482+vivivi-111@users.noreply.github.com>
 2026-10-05 20:20:25 +08:00
Yueqian Lin 6dd0d1f931 [Bugfix] Update routed-expert tests for native auxiliary output (#8500) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-10-05 17:40:07 +08:00
JungleandCodex 67d21b5fff [Core] Add token_ids output type and enforce canonical names (#7843) 
Signed-off-by: Jungle430 <junglece430@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-05 16:49:03 +08:00
Zhou Taichang 0b0d2d7009 [Rebase] Align vLLM-Omni with vLLM v0.31.0 (#8459) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-10-05 14:11:01 +08:00
Scott Ye 1f47fa2f29 [Model][Core] Avoid unused CosyVoice3 payloads and redundant AR output work (#8473) 
Signed-off-by: Sorbett <sorbet111@163.com>
 2026-10-05 04:52:19 +00:00
汪志鹏 5ca23106fd [Model] Add SheetSage2 preprocessing and YuE2 score handoff (#8371) 
Signed-off-by: princepride <wangzhipeng628@gmail.com>
 2026-10-05 11:09:37 +08:00
MaciejBalaNVBartosz Stefaniak汪志鹏
 		bbd539d2b6 [feature] Enable multi-node inference through Ray backend (#7938) 
Signed-off-by: Maciej Bala <mbala@nvidia.com>
Co-authored-by: Bartosz Stefaniak <bstefaniak@nvidia.com>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
 2026-10-05 02:53:36 +00:00
Yueqian Lin c1e84ce946 [Bugfix] Wire output buffers into duplex model test harnesses (#8489) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-10-04 23:55:19 +00:00