anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
1,271 Commits 235 Branches 31 Tags
033e4041bebcf4c7d1e4363867ce6bc438fa2e1f
Commit Graph
815 Commits
Author SHA1 Message Date
Yueqian Lin 033e4041be fix(moss-tts-nano): remove unused voice variable (ruff F841) 
The upstream inference_stream() API has no voice/spk parameter;
voice preset selection is not yet wired into the call. Remove the
dead assignment to silence ruff F841.

Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-04-14 13:11:43 -04:00
Yueqian Lin 38aaaebeda feat(moss-tts-nano): enable progressive streaming via AR runner 
Switch from GPUGenerationWorker to AR worker with OmniARScheduler,
using the VoxCPM-style generator pattern for streaming:

- inference_stream() stored per-request in _stream_gens dict
- Each forward() call yields one audio chunk via next(generator)
- compute_logits() emits EOS only when last chunk is yielded
- AR scheduler loops until EOS, enabling progressive audio output

TTFP reduced from ~3.1s to ~0.11s (30x improvement).

Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-04-14 13:06:44 -04:00
Yueqian Lin fc81b8504a feat(moss-tts-nano): add online serving, Gradio demo, and fix /v1/audio/speech integration 
- Register moss_tts_nano as TTS model stage in serving_speech.py
  (model detection, validation, prompt building)
- Fix registry mod_relname: moss_tts_nano -> modeling_moss_tts_nano
- Fix stage config: is_comprehension=true (required for generate task)
- Fix default mode: voice_clone -> continuation (built-in presets)
- Add compute_logits stub for VllmModelForTextGeneration protocol
- Remove unused _sentinel nn.Parameter
- Add Gradio demo with AudioWorklet streaming player (48kHz stereo)
- Add online/offline serving docs and launch scripts

TODO: Single-stage generation models don't support true streaming
(progressive audio chunks). Current TTFP = full generation time.
Needs multi-step scheduling support in GPUGenerationModelRunner.

Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-04-14 13:06:44 -04:00
Yueqian Lin 571c8d4235 fix(moss-tts-nano): also patch torchaudio.save for soundfile fallback 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-04-14 13:06:12 -04:00
Yueqian Lin 0b8190d07f fix(moss-tts-nano): patch torchaudio.load to use soundfile when torchcodec unavailable 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-04-14 13:06:12 -04:00
Yueqian Lin da6b1b5741 feat(moss-tts-nano): add single-stage MOSS-TTS-Nano TTS support 
Integrates OpenMOSS-Team/MOSS-TTS-Nano (0.1B) as a single
GPUGenerationWorker stage.  Both the AR LM and MOSS-Audio-Tokenizer-Nano
codec run inside MossTTSNanoForGeneration.forward(), removing the need for
an inter-stage connector.

Key design choices:
- Weights loaded in load_weights() not __init__ (avoids pre-CUDA alloc)
- trust_remote_code delegates to upstream HF model classes
- codec path read from config.audio_tokenizer_pretrained_name_or_path
- inference_stream() collects progressive audio chunks for low latency
- 48 kHz stereo output; voice clone + continuation modes supported


Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-04-14 13:06:12 -04:00
Yiyang "Ian" LiuandSYLAR bc4a659f03 [Bugfix] Make mrope kwargs optional in HunyuanImage3 get_mrope_input_positions (#2654) 
Signed-off-by: Yiyang Liu <yiyangliu@microsoft.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
 2026-04-14 14:41:28 +00:00
amy-why-3459 f03ab38783 [BugFix] Fix NoneType' object has no attribute 'detach' (#2797) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-04-14 10:26:55 -04:00
Yiyang "Ian" LiuandCopilot 53a9cf49a6 fix: do not apply FP8 quant config to vision/audio encoders for pre-quantized checkpoints (#2702) 
Signed-off-by: Yiyang Liu <37043548+ianliuy@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-14 13:52:32 +00:00
Bvicii 4fb078a031 [Bugfix] Reject /v1/audio/speech for Qwen omni models (#2763) 
Signed-off-by: Bvicii <yizhanhuang2002@gmail.com>
 2026-04-14 12:53:06 +00:00
Alex Brooks cf1fcd5acf [CI/BugFix] Fix Flaky Test for Qwen Omni Perf (#2754) 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
 2026-04-14 20:49:57 +08:00
NATURE 6d01a8b506 [Omni Connector] Omni Transfer Engine Connector: Enable 1-receiver-to-N-senders to support Bagel TP/CFG parallel (#2731) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
 2026-04-14 16:06:37 +08:00
iancarrasco-b10 48c30bc399 [Qwen3-TTS] Remove hardcoded distributed_executor_backend to improve single-GPU performance (#2604) 
Signed-off-by: Ian Carrasco <ian.carrasco@baseten.co>
 2026-04-14 14:49:13 +08:00
Samit 644edac0b6 [Perf] Optimize MP4 encoding latency in video generation (#2735) 
Signed-off-by: samithuang <285365963@qq.com>
 2026-04-14 14:20:38 +08:00
amy-why-3459 a5b38b5d0d [BugFix] Remove stage_configs_path validation (#2741) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-04-14 17:32:00 +12:00
Chen-Yo Sun dd1389173b [Voxtral TTS] Fix Voxtral TTS input with text and ref_audio (#2750) 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
 2026-04-13 22:37:45 +00:00
Sy03Yueqian LinYueqian Lin
 		14f7910900 [Perf]: Speedup VoxCPM2 TTS performance and Support PagedAttention (#2690) 
Signed-off-by: Sy03 <1370724210@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-04-14 09:16:47 +12:00
Chen-Yo Sun c9e2e3e8d7 [Voxtral TTS] Correct decode steps param in Voxtral TTS (#2524) 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
 2026-04-13 17:53:35 +00:00
汪志鹏 2c67c30550 [Bagel]: Support think mode in single stage deployment of Bagel (#2650) 
Signed-off-by: princepride <wangzhipeng628@gmail.com>
 2026-04-13 11:15:49 +00:00
Chenguang Zheng 155583f49f [Bugfix] Release stage launch lock before handshake (#2717) 
Signed-off-by: Chenguang ZHENG <645327136@qq.com>
 2026-04-13 18:35:59 +08:00
JohnJanandwuzhongjian cd2761e15c [Feature]: support Flux.2-dev tea_cache (#1871) 
Co-authored-by: wuzhongjian <wuzhongjian@cmss.chinamobile.com>
 2026-04-13 17:51:48 +08:00
NATUREandHongsheng Liu 0d4e975e1b [core]refactor communication layer: PR1(Added Refactor Infra Only) (#1555) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-04-13 16:01:14 +08:00
amy-why-3459 2b70e89535 [Revert] Revert "[Log] Wire stat loggers into AsyncOmniEngine to match AsyncLL… (#2716) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-04-13 14:55:16 +08:00
Yueqian Lin d9e745ce2c [Fix] VoxCPM2: support raw audio for voice cloning via OpenAI API (#2720) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-04-13 06:30:56 +00:00
Jiaqian LiuandCanlin Guo 8097747a5d [Model] Adapt Wan2.2-I2V-A14B via LightX2V offline conversion path (#2134) 
Signed-off-by: Celeste-jq <591998922@qq.com>
Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
 2026-04-13 14:20:04 +08:00
Sy03 cb4d13a658 [Perf][Fish Speech] Enable CUDA Graph capture for Fast AR code predictor (#2520) 
Signed-off-by: Sy03 <1370724210@qq.com>
 2026-04-13 04:53:35 +00:00
NATURE e12250119b [Bugfix] Fix Bagel online mode for 1. Hang after several requests 2. Non-deterministic image quality regression. (#2458) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
 2026-04-13 03:36:20 +00:00
Zhengyuan Su (苏政渊)andClaude Opus 4.6 eb1a801b21 [Feat] Override single stage CLI args when stage_configs_path is set in OmniEngineArgs (#2684) 
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-13 11:08:52 +08:00
zhou zhuoxin 16041ab550 [Refactor] Extend CFG Parallel to support 3 or 4 branch dispatch across M GPUs (#2423) 2026-04-12 22:06:49 +08:00
Alex BrooksandSYLAR ef230ac720 [Bugfix] Fix Incompatible Multihook Integration (TeaCache <-> CPU Offload) (#2689) 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
 2026-04-12 21:44:06 +08:00
Canlin Guo 5d58abb11e [Log] Wire stat loggers into AsyncOmniEngine to match AsyncLLM (#2551) 
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
 2026-04-12 19:54:23 +08:00
ReidandHongsheng Liu 38dfe56fdf [Bugfix] Validate speaker in chat endpoint and fix case-insensitive lookup (#2407) 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-04-12 00:14:16 -04:00
Yuanheng ZhaoHongsheng LiuSYLAR
 		c20cac86ce [Bugfix] Restore user config/runtime stage init timeout (#2519) 
Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com>
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>
 2026-04-12 00:48:47 +08:00
Juan Pablo Zuluaga f7e8df9dbd [TTS][OmniVoice] Add voice cloning support for OmniVoice TTS (#2676) 
Signed-off-by: JuanPZuluaga <juanz9312@gmail.com>
 2026-04-11 12:14:03 -04:00
Nick Cao eccee21c6c [Perf] Use global CUDA graph pool for MiMo Audio (#2657) 
Signed-off-by: Nick Cao <ncao@redhat.com>
 2026-04-11 13:45:26 +00:00
TJian 25c0566393 [ROCm] [CI] [Bugfix] Resurface CI Signal, fix MHA AR selection, sync with cuda tests (#2340) 
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2026-04-11 19:54:22 +08:00
akshatvishu c9e8411d91 [feature] : add cache-dit for stable-audio-open-1.0 (#1341) 
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
 2026-04-11 17:27:05 +08:00
teithandYueqian Lin 001f2e32e2 [BUG FIX]: prevent EngineCore crash when Qwen TTS Base task is missing ref_text (#2203) 
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-04-11 02:46:38 +00:00
Yueqian LinandSYLAR a41174e083 [Model] VoxCPM2 native AR TTS support (#2658) 
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Co-authored-by: SYLAR <lishunyang12@users.noreply.github.com>
 2026-04-11 10:21:26 +08:00
Nick CaoandClaude 2bc183f6f0 [Refactor] Remove dependency on librosa (#2273) 
Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-04-11 02:11:31 +08:00
Yiyang "Ian" LiuYiyang LiuCopilot
 		687405c5f2 [Config] Remove invalid LLM-only engine_args from diffusion stage configs (#2622) 
Signed-off-by: Yiyang Liu <yiyangliu@microsoft.com>
Co-authored-by: Yiyang Liu <yiyangliu@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-11 01:36:29 +12:00
JinhengClaude Opus 4.5Canlin Guo
 		78bef62f82 [Profiler] Add Nsight Systems support for serving (#1098) 
Signed-off-by: Jinheng Li <ahengljh@gmail.com>
Signed-off-by: Canlin Guo <961750412@qq.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Canlin Guo <961750412@qq.com>
 2026-04-10 18:13:12 +08:00
bjf-frz fbb5dd5794 [Bugfix]: modify diffusion pipeline profiler result in videos (#2647) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
 2026-04-10 16:48:06 +08:00
fan2956 c2ae58bb84 [Bugfix] fix mindiesd laserattention unsupported error (#2673) 
Signed-off-by: fan2956 <zhoufan53@huawei.com>
 2026-04-10 16:18:27 +08:00
wuhangandDidan Deng cb91cbe61e [Entrypoint][Refactor]Stage CLI Refactor (#2020) 
Signed-off-by: wuhang <wuhang6@huawei.com>
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
 2026-04-10 14:40:37 +08:00
JohnJan f3f2dc590c [Feature]: support Flux.2-dev CFG-Parallel (#2010) 2026-04-10 13:55:21 +08:00
Yueqian Lin 94232436bb [Feat][FishSpeech] Cache DAC-encoded ref audio for voice cloning (#2609) 
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-04-10 10:34:03 +08:00
Haco 0c46ba57aa [Bugfix] restore legacy stage config precedence (#2663) 
Signed-off-by: xiaohajiayou <923390377@qq.com>
 2026-04-10 10:24:35 +08:00
Sy03 694be6f2e1 [Bugfix] Fix Fish Speech voice clone FileNotFoundError on multi-GPU (#2606) 
Signed-off-by: Sy03 <1370724210@qq.com>
 2026-04-09 20:34:34 +00:00
WeiQing Chen 85d63c47f9 [Bugfix] Fix Qwen-Image min-size normalization for tiny requests (#2637) 
Signed-off-by: David Chen <530634352@qq.com>
 2026-04-09 20:51:01 +08:00