anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
2,942 Commits 235 Branches 31 Tags
c4e61133fbb5587e0cf8972ce32ffac4e37fa739
Commit Graph
38 Commits
Author SHA1 Message Date
tzhouamandClaude Fable 5.1 6ce8de1b28 [Edge][CPU] KV-cached code predictor, int8-dynamic CPU fast path, model-side CPU sampler 
Measured on a Xeon 8480C (12+4 threads, 0.6B): the 15 residual-codebook sub-steps
re-ran the 5-layer predictor over the whole prefix each time (97 ms per frame of
the 134 ms talker step) and vLLM's stock sampler spent 9 ms per step in its
torch.compile'd random-sample kernel on CPU.

- code predictor: forward_cached (KV cache per layer, 2-token prefill then one
  token per sub-step) and a shape-static forward_static_step (tensor position +
  boolean mask) that torch.compile handles as one graph; incremental decoding
  is the default on CPU (VLLM_OMNI_PREDICTOR_INCREMENTAL), optional compile
  (VLLM_OMNI_PREDICTOR_COMPILE=step), optional thread cap.
- CPU fast path: dynamic int8 Linear layers (fp32 activations) for the predictor
  and its lm heads (VLLM_OMNI_PREDICTOR_INT8, default on; micro-benchmark 70 ->
  36 ms per frame for the 15 sub-steps); VLLM_OMNI_PREDICTOR_PARITY_CHECK=1
  records top-1 agreement / max logit diff against the bf16 model in situ.
- Qwen3-TTS talker: prefer_model_sampler on CPU with a sample() that mirrors
  vLLM's Sampler (logits processors + penalties, temperature, top-k/top-p,
  exponential-noise sampling with per-request generators) without the compiled
  kernel (VLLM_OMNI_CPU_FAST_SAMPLER, default on).
- step_stats: sampler phase timers and sample-metadata timers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-10 01:56:00 +08:00
GXINandHongsheng Liu 3ab582c993 [Bugfix][Qwen3-TTS] Reject task mismatches before EngineCore dispatch (#6113) 
Signed-off-by: XIN GAO <1037396230@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-09-02 11:16:55 -07:00
林鑫sheenlinyurainGao Han
 		37c11e6e4f [Perf][Qwen3-TTS] Fuse QKV and gate_up projections in code predictor (#5791) 
Signed-off-by: sheenlin <sheenlin@tencent.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: sheenlin <sheenlin@tencent.com>
Co-authored-by: yurain <yurain26@users.noreply.github.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-08-20 19:22:40 +00:00
Inesh Reddy Chappidi cd3867c373 [Model] Suppress silence codec tokens for the first N Qwen3-TTS decod… (#5048) 
Signed-off-by: Inesh Reddy <ineshreddy249@gmail.com>
 2026-08-20 09:22:37 -07:00
Sy03andYueqian Lin 58007c0084 [New Model] Support IndexTTS 2.5 (#5957) 
Signed-off-by: Sysy <1370724210@qq.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-08-11 23:00:35 +00:00
boatmanandYueqian Lin 4f9b65074d [Perf][Qwen3-TTS]Cached Incremental Decode (#5202) 
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-08-11 05:24:04 +00:00
Gao Han 78c144f3a8 [CI failed] Revert "[Perf][Qwen3-TTS] Fuse QKV and gate_up projections in code predictor" (#5777) 2026-08-04 15:21:38 +00:00
林鑫sheenlinyurain
 		6fdc4ca62d [Perf][Qwen3-TTS] Fuse QKV and gate_up projections in code predictor (#4958) 
Signed-off-by: sheenlin <sheenlin@tencent.com>
Co-authored-by: sheenlin <sheenlin@tencent.com>
Co-authored-by: yurain <yurain26@users.noreply.github.com>
 2026-08-04 11:59:32 +00:00
wangyu 7c17111ff5 [Refactor][CI] Reorganize test tree, and move slow e2e to weekly (#4951) 
Signed-off-by: wangyu <410167048@qq.com>
 2026-07-29 23:53:03 +08:00
Ma1oneZhangSy03Hongsheng Liu
 		af2f84582b perf(qwen3-tts): cache decoder masks and compile pre-transformer (#5107) 
Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Co-authored-by: Sy03 <1370724210@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-07-25 08:09:19 +00:00
zyz111222andamy-why-3459 2d58070f60 [Hardware][Ascend][Model] Optimize Qwen3-TTS on 310P (#4841) 
Signed-off-by: zouyizhou <zouyizhou@huawei.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-07-15 08:57:44 +08:00
林鑫andyurain 020792e6be [Perf][Qwen3-TTS] Sample code predictor outputs via Gumbel-max (#4261) 
Signed-off-by: 林鑫 <77673148+l-wave@users.noreply.github.com>
Co-authored-by: yurain <yurain26@users.noreply.github.com>
 2026-07-10 05:41:54 +00:00
Yueqian Lin d7b365df64 [Bugfix][Qwen3-TTS] Keep seeded residual MTP sampling batched (#4889) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-07-05 21:47:06 +08:00
Sy03Claude Sonnet 4.6Yueqian Lin
 		f11b32a42e [Bugfix] Qwen3-TTS talker handles multi-token replay on resume (#4559) 
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-07-02 04:46:05 +00:00
Zhou Taichang a560ed184d [Rebase] Rebase to vllm v0.24.0 (#4709) 2026-06-30 19:41:06 +08:00
Henry JYueqian LinClaude Opus 4.8
 		9c260b3774 Qwen3-TTS full-payload: emit empty-finished payload on degenerate take instead of dropping (fixes Stage-1 300s hang) (#4465) 
Signed-off-by: jjw <m@jjw.in>
Signed-off-by: henryj <349043+henryj@users.noreply.github.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-23 00:21:11 +00:00
Yueqian LinSy03Hongsheng Liu
 		cead615288 [Bugfix][Qwen3-Omni] Fix Code2Wav CUDA-graph output length surplus (#4466) (#4525) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Sy03 <1370724210@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-06-19 23:58:45 -07:00
Henry JjjwClaude Fable 5Yueqian Lin
 		1733831f78 [Bugfix] Fix cross-request codes.ref leak in Qwen3-TTS make_omni_output (#4373) 
Signed-off-by: jjw <m@jjw.in>
Co-authored-by: jjw <m@jjw.in>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-06-13 00:41:49 +00:00
Sy03Dan250124Yueqian LinYueqian Lin
 		d4c139508f [Feature] Add precomputed custom voices and Qwen3-TTS ref-context cache (#3492) 
Signed-off-by: Sy03 <1370724210@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Dan250124 <416947747@qq.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-05-31 17:51:24 +00:00
Viacheslav KlimkovandYueqian Lin 914e92cf80 [Refactor][Qwen3-TTS] Extract reusable prompt-embeds builder and make tts_pad_embed a persistent buffer (#3992) 
Signed-off-by: Viacheslav Klimkov <vklimkov@nvidia.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-05-30 16:11:00 +08:00
nperraud 9fdb3bcb89 [Bugfix][Model] Qwen3-TTS: don't collapse 2D ref_code list when estimating prompt length (#3940) 
Signed-off-by: nperraud <nathanael.perraudin@gmail.com>
 2026-05-29 09:18:25 -06:00
wuli666andwuli666 8935439428 [Bugfix] Fix Triton SnakeBeta kernel for bf16/fp16 inputs (#3472) 
Signed-off-by: wuli666 <djjpro975@gmail.com>
Co-authored-by: wuli666 <djjpro975@gmail.com>
 2026-05-28 17:10:36 +00:00
Nick CaoandClaude c20afebce4 [Refactor][Qwen3-TTS] Construct speech tokenizer encoder natively (#3360) 
Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-28 16:20:49 +08:00
Yueqian LinandRein Yang e996723bc9 [Bugfix] Fix Qwen3-TTS Stage 0 prefix-caching correctness (#3665) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Signed-off-by: Rein Yang <ruiruyang2@gmail.com>
Co-authored-by: Rein Yang <ruiruyang2@gmail.com>

the ci failure case is irrelavent to this PR and occurs rarely
 2026-05-25 23:09:38 +08:00
Sy03andamy-why-3459 ab59673f21 [Bugfix][Qwen3-Omni] Handle short Code2Wav chunk outputs (#3687) 
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-05-18 18:59:23 +00:00
Sy03Hongsheng LiuSYLAR
 		c99df1ebd9 [TTS][Perf] Optimize Qwen3-TTS high-concurrency serving (#3662) 
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
 2026-05-17 16:58:59 -04:00
+20
Zhou TaichangClaude Opus 4.7NumberWandsinghviDnoobYueqian LinHongsheng Liuknlnguyen1802Samitrein yangNick Caozhumingjue138Ricardo Noriegalyj-jjjCursorgcanlinwangyuweizhoublueweizhoubluedengyunyang汪志鹏WeiQing ChenJie LiuYueqian LinNATUREbjf-frzamy-why-3459Y. FisherZeyu Huang | 黃澤宇Copilot Autofix powered by AI
 		eb4e60ee64 [Rebase] Rebase to vllm v0.21.0 (#3530) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk>
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Signed-off-by: Divyansh Singhvi <divyanshsinghvi@gmail.com>
Signed-off-by: dsinghvi <divyanshsinghvi@gmail.com>
Signed-off-by: Dnoob <dxpouo@gmail.com>
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: rein yang <ruiruyang2@gmail.com>
Signed-off-by: Nick Cao <ncao@redhat.com>
Signed-off-by: zhumingjue <zhumingjue@huawei.com>
Signed-off-by: Ricardo Noriega De Soto <rnoriega@redhat.com>
Signed-off-by: lyj-jjj <liuyingjun5@huawei.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: wangyu <410167048@qq.com>
Signed-off-by: weizhoublue <weizhoublue@github.com>
Signed-off-by: weizhou.lan@daocloud.io <weizhou.lan@daocloud.io>
Signed-off-by: dengyunyang <584797741@qq.com>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
Signed-off-by: David Chen <530634352@qq.com>
Signed-off-by: Jie Liu <33612777+keeper-jie@users.noreply.github.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Signed-off-by: bjf-frz <frz123db@gmail.com>
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: KexiongYu <yukexiong1@huawei.com>
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Signed-off-by: Zeyu Huang | 黃澤宇 <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: NumberWan <wantszkin2003@gmail.com>
Co-authored-by: dsinghvi <divyanshsinghvi@gmail.com>
Co-authored-by: Dnoob <dxpouo@gmail.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Co-authored-by: Samit <285365963@qq.com>
Co-authored-by: rein yang <73573651+R2-Y@users.noreply.github.com>
Co-authored-by: Nick Cao <ncao@redhat.com>
Co-authored-by: zhumingjue138 <zhumingjue@huawei.com>
Co-authored-by: Ricardo Noriega <rnoriega@redhat.com>
Co-authored-by: lyj-jjj <liuyingjun5@huawei.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: gcanlin <canlinguosdu@gmail.com>
Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Co-authored-by: weizhoublue <45163302+weizhoublue@users.noreply.github.com>
Co-authored-by: weizhoublue <weizhoublue@github.com>
Co-authored-by: dengyunyang <584797741@qq.com>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Co-authored-by: Jie Liu <33612777+keeper-jie@users.noreply.github.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: NATURE <wzliu@connect.hku.hk>
Co-authored-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: Y. Fisher <yukexiong1@huawei.com>
Co-authored-by: Zeyu Huang | 黃澤宇 <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-05-16 18:18:43 +08:00
Sy03 2c73cf3522 [Perf] Fix Qwen3-TTS latency regression (#3485) 
Signed-off-by: Sy03 <1370724210@qq.com>
 2026-05-10 23:07:38 +00:00
Juan Pablo ZuluagaandYueqian Lin 19f8f42822 [Feat][Qwen3-Omni] Add CUDA graph support for Code2Wav decoder (#2376) 
Signed-off-by: JuanPZuluaga <juanz9312@gmail.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-05-07 06:16:03 +08:00
Sy03 2b85474c23 [Bugfix] Propagate seed to Qwen3-TTS Fast AR sampler (#3350) 
Signed-off-by: Sy03 <1370724210@qq.com>
 2026-05-05 14:57:55 -04:00
Nick CaoandClaude fde38ca54f [Refactor][Qwen3-TTS] Construct Code2Wav decoder natively (#2341) 
Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-05 15:26:30 +00:00
dsinghviZhou TaichangHongsheng Liu
 		2390350533 [Refactor] Standardize data entry key names to {type}.{qualifier} format (#1829) 
Signed-off-by: Divyansh Singhvi <divyanshsinghvi@gmail.com>
Signed-off-by: dsinghvi <divyanshsinghvi@gmail.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Co-authored-by: Zhou Taichang <tzhouam@connect.ust.hk>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-04-25 18:15:38 +08:00
Juan Pablo ZuluagaandHongsheng Liu 82f8c93343 [Feat][Qwen3-Omni] Shared code predictor module for Qwen3-TTS and Qwen3-Omni (#2375) 
Signed-off-by: JuanPZuluaga <juanz9312@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-04-15 15:14:57 +08:00
Yuanheng Zhao e0cdbe9a5d [Misc] Cleanup: use consistent pytest-mock in unit tests (#2698) 
Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com>
 2026-04-13 19:21:42 +08:00
Will.houwillamhouClaudeHappy
 		e23b2634d1 fix(qwen3_tts): align code predictor buffer dtype with model parameters (#2470) 
Signed-off-by: willamhou <willamhou@ceresman.com>
Co-authored-by: willamhou <willamhou@ceresman.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Happy <yesreply@happy.engineering>
 2026-04-05 20:54:15 +00:00
Sy03 025408f693 [Fix] [Qwen3-TTS] Qwen3-TTS streaming chunk-boundary artifacts (#2480) 
Signed-off-by: Sy03 <1370724210@qq.com>
 2026-04-05 14:40:38 -04:00
Juan Pablo ZuluagapabloJuanPZuluagaHongsheng LiuYueqian Lin
 		81a90d24ef [Feat][Qwen3TTS][Code2wav] triton SnakeBeta and Cuda Graph (#1797) 
Signed-off-by: pablo <pablo@agigo.ai>
Signed-off-by: JuanPZuluaga <juanz9312@gmal.com>
Co-authored-by: pablo <pablo@agigo.ai>
Co-authored-by: JuanPZuluaga <juanz9312@gmal.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-03-20 12:11:37 +08:00
Xu Luandxulusjb c9b31a4c1a feat(qwen3-tts): Add CUDA Graph support for speech tokenizer decoder (#1205) 
Signed-off-by: xulusjb <fdukeshik@gmail.com>
Co-authored-by: xulusjb <fdukeshik@gmail.com>
 2026-02-27 10:48:25 +08:00