tzhouam and Claude Opus 5
7869a8dc0d
Add cpu_int4, an asymmetric 4-bit weight path for vLLM's CPU backend
...
Single-stream CPU decode reads every weight to emit one token, so bits per
weight set the token rate. This registers a W4A16 group-wise method
out-of-tree through register_quantization_config -- vLLM itself is untouched
-- using PyTorch's _weight_int4pack_mm_for_cpu tinygemm kernel.
Two paths over one set of weights, because that kernel dequantizes once per
row of activations and so is linear in the batch (12.9x slower than bf16 at
512 rows). Single-row steps take tinygemm; wider batches unpack the weight to
bf16 once and use the ordinary oneDNN/AMX GEMM. Shipping only the first
measured pp512 = 557 tok/s against bf16's 4455; with both, 3589.
The batch threshold is 2 rather than the ~8 rows where the unpack path starts
winning on speed, because running tinygemm on a multi-row batch and then
vLLM's cpu_attention_with_kv_cache in the same forward dies with SIGILL
inside the attention kernel -- an AMX instruction with no valid tile config
loaded. The same prompt is fine in bf16 and fine once every multi-row GEMM
takes the unpack path.
Positioning, measured on a Xeon 8480C with 32 pinned cores and a warm cache
(analysis/spark_cpu_int4.md): vLLM's own GPTQ W4A8 AMX path is faster --
88.7 tok/s decode and 4355 prefill against 85.5 and 3589 here -- but its
config rejects sym=False at 4 bits, so it can only be fed a symmetric grid.
This one is asymmetric and costs +12.5% perplexity against that path's
+27.2%, and needs no AMX. llama.cpp Q4_K_M, for reference, is 98.0 tok/s
decode and 1356 prefill at +7.6% and 5.15 bits per weight.
Expanding a quantized checkpoint back to bf16 offline and scoring it
reproduces the quantized run to within 0.2% perplexity, so both GEMM paths,
the nibble packing and the quantized tied embedding are faithful.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com >
Claude-Session: https://claude.ai/code/session_01GXhbAfkSNRaeqZrus5Z2fx
2026-09-12 03:48:04 +08:00
tzhouam and Claude Opus 5
e4a565b3df
[Edge][Model] Spark-X2.5 (Spark2_5ForCausalLM) support
...
Spark-X2.5-1.7B is an Apache-2.0 on-device model built for agentic tool
use. vLLM 0.28 has no implementation for it, so it could not be served at
all; llama.cpp already ships one, which made it the only edge engine able
to run the model.
Three traits separate it from a Llama-style decoder:
* Hybrid attention -- layer_types interleaves three 512-token sliding
layers with one full-attention layer, which is what makes the 1M-token
context affordable on a phone.
* Per-layer-type RoPE -- sliding layers rotate all 256 head dims with
theta 1e4, full layers only the first quarter with theta 5e6.
* A head-wise attention output gate -- g_proj reads the same post-norm
input as QKV and scales each head's attention output by a sigmoid.
The checkpoint also stores QKV pre-fused as a single q_k_v_proj, so
load_weights splits it before vLLM's sharded QKVParallelLinear sees it.
Verified against the official remote-code implementation on transformers
4.57.1 (the version config.json targets; it does not import on the 5.x in
our venvs). In fp32 all three scenarios match exactly -- 32/32 greedy
tokens with a max top-20 logprob difference of 2.2e-05. In bf16 the short
and the 2465-token long-context prompts still match 32/32, which is what
exercises the sliding layers; the tool-calling prompt diverges at token 23
on a near-tie, and fp32 confirms that is rounding rather than a defect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com >
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-11 13:47:02 +08:00
tzhouam and Claude Fable 5.1
b22dcc59f7
[Edge][WP1] Add edge deploy profile and deploy_profile selector
...
* vllm_omni/deploy/edge/qwen3_tts.yaml: single-device Qwen3-TTS profile with a
ramped codec chunk schedule (2->4->8->16->25 frames) that removes the ~80 ms
playback stall of the default 1->25 schedule, decode batch 1, max_num_seqs 4.
* deploy_profile="edge" (Omni/AsyncOmni kwarg, --deploy-profile CLI, headless
serve) resolves vllm_omni/deploy/<profile>/<model_type>.yaml; mutually
exclusive with deploy_config; explicit profiles never fall back silently.
* tests/helpers/stage_config.py: modify_stage_config anchors a relative
base_config before writing the temp YAML so overlay-derived configs resolve.
* async_omni_engine.py also carries the init-timeline hooks used by WP2.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 10:27:33 +08:00
SYLAR
ea3cf99894
[diffusion][feature] Add component-selective offload policies ( #5929 )
...
Signed-off-by: lishunyang12 <lishunyang12@163.com >
2026-09-06 05:40:09 +08:00
Zhou Taichang and Claude Fable 5
142151fef5
Parallel stage initialization (admission + SH/EX device locks) (Worker refactor 1/N) ( #5224 )
...
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk >
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-09-05 22:22:41 +08:00
TJian
ded8934626
[AMD] [CI] Fix tests for v0.28.0 ( #6830 )
...
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
2026-08-31 09:10:27 +08:00
baonudesifeizhai
8be601ed7f
[Diffusion] Add native SymmMem Fast Ulysses transport ( #6340 )
...
Signed-off-by: roG0d <baonudesifeizhai@gmail.com >
2026-08-30 04:14:21 +00:00
b6a4baa659
[Feature] Support text encoder Disaggregation for H3 ( #5885 )
...
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com >
Signed-off-by: lishunyang12 <lishunyang12@163.com >
Signed-off-by: yuanwu <yuan.wu@intel.com >
Co-authored-by: lishunyang12 <lishunyang12@163.com >
Co-authored-by: yuanwu <yuan.wu@intel.com >
2026-08-28 14:27:05 +08:00
NATURE
54c1fc64b7
[Bugfix] Gate async-chunk segment resume by session mode ( #6680 )
...
Signed-off-by: natureofnature <wzliu@connect.hku.hk >
2026-08-27 18:52:09 +08:00
Hongsheng Liu and hsliuustc0106
c608910d20
[Diffusion] Register HWR mmap for direct DLO H2D ( #6591 )
...
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com >
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com >
2026-08-25 23:30:11 +08:00
wy17003 and Hongsheng Liu
b454e34966
[Refactor][2/N Scheduler]Replace _FULL_PAYLOAD_INPUT_STAGES with a resolved stage transport capability. ( #6149 )
...
Signed-off-by: wy17003 <yaow17003@gmail.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-08-25 15:24:59 +08:00
Hongsheng Liu and hsliuustc0106
8360346a66
[Diffusion] Integrate no-AllGather DLO with Host Weight Runtime ( #6486 )
...
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com >
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com >
2026-08-25 08:03:53 +08:00
bjf-frz and Hongsheng Liu
072bfc02dd
[FEAT]Add FastVideo VSA backend for Wan2.2 ( #4820 )
...
Signed-off-by: bjf-frz <frz123db@gmail.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-08-24 14:48:31 +08:00
Inesh Reddy Chappidi
cd3867c373
[Model] Suppress silence codec tokens for the first N Qwen3-TTS decod… ( #5048 )
...
Signed-off-by: Inesh Reddy <ineshreddy249@gmail.com >
2026-08-20 09:22:37 -07:00
Yueqian Lin
b283215c28
[Model] Add MiniMax Music 3 text-to-music ( #6186 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
2026-08-18 04:33:13 +00:00
Yukim1 and Hongsheng Liu
4b66f62e10
[Refactor] Remove internal stage_configs_path plumbing ( #5741 )
...
Signed-off-by: zwhzzz0821 <2831474076@qq.com >
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-08-13 21:29:03 +08:00
Yuekai Zhang and Claude Fable 5
5be7541d01
[Model] Support NVIDIA-NemotronLabs-VoiceChat-11B offline speech-to-speech ( #5842 )
...
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-12 16:25:48 +08:00
9b0df3909a
[Feature] timestamps for TTS via a forced-aligner pooling ( #4795 )
...
Signed-off-by: wjinxu <1299461899@qq.com >
Co-authored-by: xurui <xurui1@modelbest.onaliyun.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-12 05:33:34 +00:00
Sy03 and Yueqian Lin
58007c0084
[New Model] Support IndexTTS 2.5 ( #5957 )
...
Signed-off-by: Sysy <1370724210@qq.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
2026-08-11 23:00:35 +00:00
Zhou Taichang and Claude Opus 5
2ce15ab7a0
[Rebase] Rebase to vllm 0.27.0 ( #5976 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-11 15:23:46 +00:00
Jerry Song
89015d5cad
[Feature]: Adds distilled LoRA support for diffusion models ( #2783 )
...
Signed-off-by: Songrui625 <songrui625@gmail.com >
2026-08-11 14:23:43 +00:00
NumberWan
cfa6d66ed6
[BugFix][Nightly CI] Opt in FA deterministic for Qwen-Image accuracy ( #5887 )
...
Signed-off-by: NumberWan <wantszkin2003@gmail.com >
2026-08-07 17:15:22 +08:00
48298030d4
[Diffusion] Add Minimax-H3 modular pipeline support ( #5720 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: lishunyang12 <lishunyang12@163.com >
Signed-off-by: David Chen <530634352@qq.com >
Co-authored-by: lishunyang12 <lishunyang12@163.com >
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com >
Co-authored-by: David Chen <530634352@qq.com >
2026-08-06 11:43:23 +00:00
SYLAR and lishunyang12
1c2a81f6d8
feat(minimax-h3): enable 2x RTX 4090/5090 with DLO ( #5764 )
...
Signed-off-by: lishunyang12 <lishunyang03@gmail.com >
Signed-off-by: lishunyang12 <lishunyang12@163.com >
Co-authored-by: lishunyang12 <lishunyang03@gmail.com >
2026-08-06 09:02:29 +08:00
Yueqian Lin and Sy0307
e788ef6e76
[Feature] PersonaPlex (Moshi-based full-duplex S2S): native vLLM port + duplex serving ( #4771 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Signed-off-by: Sy0307 <24841857+Sy0307@users.noreply.github.com >
Co-authored-by: Sy0307 <24841857+Sy0307@users.noreply.github.com >
2026-08-04 20:10:33 -07:00
Isotr0py
24741961eb
[Model] Add MiniMax H3 diffusion support ( #5691 )
2026-08-03 14:10:40 +08:00
evanchueng and david6666666
744c65b730
[Feature] Distributed layerwise offload with DP multi-concurrency + mmap weight… ( #5397 )
...
Signed-off-by: evanchueng <evanchueng@outlook.com >
Signed-off-by: david6666666 <530634352@qq.com >
Co-authored-by: david6666666 <530634352@qq.com >
2026-07-30 13:35:38 +00:00
wangyu
7c17111ff5
[Refactor][CI] Reorganize test tree, and move slow e2e to weekly ( #4951 )
...
Signed-off-by: wangyu <410167048@qq.com >
2026-07-29 23:53:03 +08:00
Canlin Guo
61324e65d7
[Refactor] Make MOSS-TTS-Local stream session dynamic batch ( #5235 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com >
2026-07-28 21:38:24 +08:00
TaffyOfficial and TaffyOfficial
93786c9cd3
[perf] Add configurable diffusion compile granularity ( #4603 )
...
Signed-off-by: TaffyOfficial <2324465096@qq.com >
Co-authored-by: TaffyOfficial <2324465096@qq.com >
2026-07-24 15:22:06 +00:00
84bbd57e4d
[Feature] Add AllGather-KV sequence-parallel attention str… ( #4968 )
...
Signed-off-by: zhuhh97 <zhuhonghan@huawei.com >
Signed-off-by: dph97 <nenbaying@163.com >
Signed-off-by: zhuhh97 <zhh@unknown>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com >
Co-authored-by: dengyunyang <584797741@qq.com >
2026-07-24 14:45:26 +00:00
Bingyu (Spencer) Liu
dcf8b07d39
[Core] Cleanup Omni connector runtime and transfer backends ( #5096 )
...
Signed-off-by: spencerr221 <liubingyu62@gmail.com >
2026-07-22 09:44:15 +08:00
Shi Boao Bowie and bjf-frz
a432b8a325
[Refactor][OutputProcessor 1/8]: clean up multimodal payload accumulation and move payload logic into MultimodalPayload & directory refactor ( #4980 )
...
Signed-off-by: Boao Shi <aoibosh@connect.hku.hk >
Signed-off-by: bjf-frz <frz123db@gmail.com >
Co-authored-by: bjf-frz <frz123db@gmail.com >
2026-07-15 15:50:19 +08:00
ff6f0da271
[Core] Composable parallel: Phase 1 strategy overlay (TP/DP/stage-replica) (RFC #4084 ) ( #4281 )
...
Signed-off-by: Kushan A. <kahmadian@nvidia.com >
Signed-off-by: kushanam <kushanam@users.noreply.github.com >
Signed-off-by: Kushan A <kahmadian@nvidia.com >
Signed-off-by: kahmadian <kahmadian@users.noreply.github.com >
Co-authored-by: kushanam <kushanam@users.noreply.github.com >
Co-authored-by: kahmadian <kahmadian@users.noreply.github.com >
2026-07-10 21:29:24 -06:00
JiangJie Zhang and Samit
c45ac74d43
[Core][Frontend] Support request-level batching for diffusion pipelines ( #4079 )
...
Signed-off-by: jader <yjader@foxmail.com >
Co-authored-by: Samit <285365963@qq.com >
2026-06-29 18:09:19 +02:00
504698dec8
[Core] Performance fix for orchestrator bottleneck: separates inter-stage from client outputs ( #4527 )
...
Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com >
Co-authored-by: Chenguang ZHENG <645327136@qq.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-06-29 17:43:36 +02:00
Rahul Steiger and Hongsheng Liu
327e9dcaf2
[Feature] Spatially-sharded (SP) decode for the Wan VAE ( #4620 )
...
Signed-off-by: Rahul Steiger <rsteiger@nvidia.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-06-27 09:01:25 +08:00
0b04f2a0d6
[Model] Add Ming-omni-tts MoE 16.8B-A3B + CFM CUDAGraph ( #4341 )
...
Signed-off-by: LHXuuu <xulianhao.xlh@antgroup.com >
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com >
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com >
Co-authored-by: Yuanheng Zhao <54058983+yuanheng-zhao@users.noreply.github.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-06-25 22:11:35 +08:00
Canlin Guo
2e6c83c89c
[Model] Support MOSS-TTS-local-v1.5 ( #4664 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
2026-06-25 15:43:03 +08:00
BeatSeat
044240d8eb
[Model] Add IndexTTS2 text-to-speech support ( #3838 )
...
Signed-off-by: BeatSeat <wendavid552@gmail.com >
Signed-off-by: David <david@podcast.ai >
Signed-off-by: wendavid552 <wendavid552@gmail.com >
Signed-off-by: WenDavid <davidzhangcf@foxmail.com >
2026-06-22 19:24:01 +00:00
57ea047acf
Step audio R1 reasoning parser ( #2846 )
...
Signed-off-by: Mike_Qiu <qiudayu.qdy@antgroup.com >
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Signed-off-by: Mike Qiu <qdy220091330@gmail.com >
Co-authored-by: Mike_Qiu <qiudayu.qdy@antgroup.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: Yueqian Lin <linyueqian@outlook.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-06-10 21:03:19 +00:00
akshatvishu and Hongsheng Liu
2fa4c7a4a9
[Model] Add Ming-omni-tts dense 0.5B pipeline ( #2906 )
...
Signed-off-by: akshatvishu <akshatnayak197@gmail.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-06-10 20:11:04 +08:00
amy-why-3459
8949fdc440
[BugFix] Fix the issue of vllm failing to start. ( #4105 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-06-03 15:44:46 +00:00
Alex Brooks
12e75c6618
[Core] Integrate TrackingArgumentParser ( #3369 )
...
Signed-off-by: Alex Brooks <albrooks@redhat.com >
2026-06-02 21:56:24 +08:00
2b7249fbb3
[Model]Support MiniCPM-o 4.5 ( #3642 )
...
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
Co-authored-by: GKangaroo <1095103651@qq.com >
Co-authored-by: GKangaroo <gqx24@mails.tsinghua.edu.cn >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-06-01 22:12:24 +08:00
Yueqian Lin and Hongsheng Liu
c067485030
[Perf][TTS] Bounded-K active-stream window for Stage 1 (RFC #3535 ) ( #3592 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-06-01 21:59:10 +08:00
+1
f92d84fcf9
[Rebase] Rebase to vllm releases/v0.22.0 ( #3891 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Signed-off-by: ZhengWG <zwg0606@gmail.com >
Signed-off-by: david6666666 <530634352@qq.com >
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Signed-off-by: nainiu258 <cperfect02@163.com >
Signed-off-by: nainiu258 <101917677+nainiu258@users.noreply.github.com >
Signed-off-by: JasonJ2021 <jasonj@zju.edu.cn >
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk >
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com >
Co-authored-by: skf <54565339+skf-1999@users.noreply.github.com >
Co-authored-by: Zheng Wengang <zwg0606@gmail.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com >
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com >
Co-authored-by: Yueqian Lin <linyueqian@outlook.com >
Co-authored-by: nainiu258 <101917677+nainiu258@users.noreply.github.com >
Co-authored-by: Jason <72191212+JasonJ2021@users.noreply.github.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
2026-06-01 11:08:12 +08:00
BeatSeat and Yueqian Lin
28ce618f0e
[Model] Add GLM-TTS text-to-speech model support ( #3141 )
...
Signed-off-by: BeatSeat <wendavid552@gmail.com >
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
2026-05-24 05:42:26 +08:00
Yueqian Lin
fc8486cf20
[Doc][TTS] CosyVoice3 online docs + residual TTS yaml cleanup + remove VoxCPM v1 ( #3748 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
2026-05-20 15:24:06 +08:00
chickeyton and herotai214
e277feacaf
[large-scale-serving] Integrate OmniCoordinator into stage engine pipeline ( #3569 )
...
Signed-off-by: chickeyton <ngton2014@gmail.com >
Signed-off-by: herotai214 <herotai214@gmail.com >
Co-authored-by: herotai214 <herotai214@gmail.com >
2026-05-19 12:08:20 +08:00