 tzhouamandClaude Fable 5.1
|
49a839ecf9
|
[Edge][WP5] Add codec-token streaming mode with on-device decoding
* qwen3_tts_talker_only pipeline (single terminal LLM_AR stage,
final_output_type "latent") + vllm_omni/deploy/qwen3_tts_talker_only.yaml;
a terminal talker publishes its per-step codes.audio rows to the client
(omni_client_multimodal_output_keys).
* /v1/audio/speech/stream: session.config.output_mode="codec_tokens" streams
codec.start (JSON) + binary frames (<IHB seq/n_frames/flags + int16 rows,
vllm_omni/entrypoints/openai/codec_stream.py) + codec.done; new input.cancel
message aborts an in-flight generation for both codec and PCM sessions
(single reader task + session inbox replaces direct socket reads); optional
session.config.seed; audio.done gains "cancelled" only when it happened.
* serving_speech._generate_codec_chunks aligns the cumulative code tensor to
output token ids (drops prefill placeholder/EOS rows) and emits deltas.
* TTS adapters: supports_codec_stream / codec_stream_spec (Qwen3-TTS: 16
codebooks, 2048 ids, 12.5 Hz, 24 kHz, decoder qwen3_tts_tokenizer_v2_12hz).
* examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py
decodes locally with Qwen3TTSTokenizerV2Decoder (ramp schedule, 72-frame
context, cancel/reset), codec_parity_check.py and
benchmarks/tts/codec_stream_e2e.py compare against the two-stage server.
* docs/serving/codec_token_streaming.md; CPU unit tests for the wire format,
alignment, session config and handler (frames, unsupported adapter, cancel);
L3 e2e tests/e2e/online_serving/test_qwen3_tts_codec_stream.py.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
2026-09-09 11:10:13 +08:00 |
|
|
tzhouamandClaude Fable 5.1
|
35e43b9983
|
[Edge][WP3] Add CPU OmniPlatform (AR + generation stages on vLLM's CPU backend)
* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform)
selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed
(builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes
like vLLM does). NUMA-based memory queries, one logical device, no graph
wrapper (sentinel type for isinstance checks), inductor opt-in via
VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported.
* vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose
vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner /
CPUGenerationModelRunner compose the Omni runners with CPUModelRunner
(no graphs, no async omni output; sleep/wake return error ACKs).
* patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder
and prompt-embeds dtype follow the model dtype (float32/float16 on CPU).
* vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async
scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges).
* requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md.
Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads):
Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms,
RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline
committed under benchmarks/tts/edge_results/.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
2026-09-09 11:02:14 +08:00 |
|
|
tzhouamandClaude Fable 5.1
|
073d31b9cd
|
[Edge][WP2] Add initialization timeline and cold-start profiler
* vllm_omni/engine/init_timeline.py: InitTimeline phase collector; engine
phases (stages_init, g3_device_lock, g2_spawn/ready) and worker phases
(proc_entry, engine_core_init, init_device, load_model/load_weights,
talker_mtp_init, attn_prewarm, cudagraph_capture, talker_mtp_capture,
code2wav_cudagraph, predictor_setup_compile/warmup/cudagraph) recorded via
worker_phase/timed_phase; workers append JSONL under
VLLM_OMNI_INIT_TIMELINE_DIR (inherited at spawn) and the engine merges them.
Enabled by VLLM_OMNI_INIT_TIMELINE=<json> or log_stats; table logged at INFO,
partial table on orchestrator startup timeout, suggested timeouts logged.
* omni_base.py: named deploy profiles default to stage_init_timeout=900 /
init_timeout=1500 unless given explicitly.
* gpu_model_runner._init_talker_mtp tolerates max_cudagraph_capture_size=None;
gpu_ar_model_runner.shutdown skips CUDA graph teardown on CPU.
* benchmarks/tts/init_profile.py: runs stream_latency_bench --init-only per
configuration (default, cold_cache, eager_stage1, eager_both,
parallel_stage_init, edge, ...) and tabulates phase durations.
* stream_latency_bench: --parallel-stage-init / --deploy-profile flags; the
resource poller no longer shadows Thread._stop and skips GPU sampling on CPU.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
2026-09-09 11:01:39 +08:00 |
|
|
BruceLoveDecimal
|
5cd925631c
|
[Feature][LingBot World]stepwise execution (#6844)
Signed-off-by: BruceLoveDecimal <liuqihao970610@gmail.com>
Signed-off-by: liuqihao <liuqihao970610@gmail.com>
|
2026-09-07 13:38:23 +08:00 |
|
|
NATUREandamy-why-3459
|
ff6e906a25
|
[Model] Preserve MiniCPM-o Talker full attention until capacity (#6799)
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
|
2026-09-06 22:25:35 +08:00 |
|
|
Hongsheng Liu
|
715b8b8746
|
[Model] MiniMax-H3: validate versioned text-conditioning handoff (#6720)
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
|
2026-09-06 21:45:10 +08:00 |
|
|
SYLARandhsliuustc0106
|
14d2d00536
|
[Bugfix] Fix Hunyuan Image-3 expert mapping contract (#7021)
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
|
2026-09-06 21:43:37 +08:00 |
|
|
|
5d7fdf9350
|
[Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental (#6196)
Signed-off-by: chickeyton <ngton2014@gmail.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
|
2026-09-05 08:20:21 +00:00 |
|
|
boatman
|
66bbebd9d3
|
[Bugfix] Fix fallback loading for IndexTTS external models (#6137)
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
|
2026-09-04 19:58:33 -07:00 |
|
|
BruceLoveDecimalandClaude Opus 5
|
1e74807c35
|
[Feature][Moss-TTS] Reference Audio Preprocessing (#4982)
Signed-off-by: BruceLoveDecimal <39156883+BruceLoveDecimal@users.noreply.github.com>
Signed-off-by: liuqihao <liuqihao970610@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
2026-09-04 16:36:26 -07:00 |
|
|
LOGO127
|
b6b992ac7b
|
[Bugfix] Fix Higgs Audio v3 voice-clone token validation (#7065)
Signed-off-by: luozijian <luozijian0924@gmail.com>
|
2026-09-04 16:06:47 -07:00 |
|
|
Inesh Reddy ChappidiandGao Han
|
66d938100b
|
[Bugfix] Give exact-shape codec models an unpadded input_ids view (#6775)
Signed-off-by: Inesh Reddy <ineshreddy249@gmail.com>
Signed-off-by: IneshReddy249 <ineshreddy249@gmail.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
|
2026-09-04 15:28:49 -07:00 |
|
|
boatman
|
348fd0fc07
|
[Feat][dots.tts] Add Online Serving Support (#6235)
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
|
2026-09-04 14:44:10 -07:00 |
|
|
Yuekai ZhangandClaude Fable 5
|
4179b0cae1
|
[Perf][NemotronVoiceChat] Fast defaults for all three deploy profiles: 2.8x offline, realtime full-duplex audio (#6354)
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
2026-09-03 18:49:19 -07:00 |
|
|
Jim Ban
|
711daacabf
|
[Bugfix][TTS] Make CosyVoice3 TRT plan publication concurrency-safe (#6955)
Signed-off-by: BANANASJIM <bananasjim1@gmail.com>
|
2026-09-03 06:44:42 +00:00 |
|
|
Tai An
|
d1a5a7c178
|
[Bugfix][Qwen2.5-Omni] Fix two stale call sites in the speech helpers (#6886)
Signed-off-by: Anai-Guo <antai12232931@outlook.com>
|
2026-09-02 21:41:05 -07:00 |
|
|
Joshna-MedisettyandCursor
|
24c49cf392
|
[Bugfix][XPU] Add MoTRMSNorm.forward_xpu delegating to forward_native (#5568)
Signed-off-by: Joshna Medisetty <joshna.medisetty@intel.com>
Signed-off-by: Joshna-Medisetty <joshna.medisetty@intel.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
2026-09-02 23:06:17 -05:00 |
|
|
ZhanshenzhuiandSy03
|
48ae412a2f
|
[model]Add experimental Mage-VL full-duplex adapter and offline integration (#6537)
Signed-off-by: Zhanshenzhui <99891002+zyforsure@users.noreply.github.com>
Co-authored-by: Sy03 <1370724210@qq.com>
|
2026-09-02 20:25:32 -07:00 |
|
|
DanaerLee
|
ca6b4e2112
|
[Perf][Bugfix][OmniVoice] Restore float16 serving, and fuse the generator hot loop (#6317)
Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
|
2026-09-02 19:54:55 +00:00 |
|
|
Jim Ban
|
b9a88bf51e
|
[Bugfix][TTS] Isolate concurrent StepAudio2 TRT builds (#6957)
Signed-off-by: BANANASJIM <bananasjim1@gmail.com>
|
2026-09-02 19:53:02 +00:00 |
|
|
GXINandHongsheng Liu
|
3ab582c993
|
[Bugfix][Qwen3-TTS] Reject task mismatches before EngineCore dispatch (#6113)
Signed-off-by: XIN GAO <1037396230@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
|
2026-09-02 11:16:55 -07:00 |
|
|
jingchengtian
|
67ef0b64b9
|
[Hardware][Ascend] Use npu_rotary_mul fused kernel for MOSS-TTS codec RoPE (#6908)
Signed-off-by: jingchengtian <tjc1995@126.com>
|
2026-09-02 08:58:23 +00:00 |
|
|
Rakesh Kariya
|
b864374089
|
[Bugfix][MiMo-Audio] Restore make_empty_intermediate_tensors on the talker for vLLM 0.28 (#6803)
Signed-off-by: rk9595 <rakesh.kariya@somaiya.edu>
|
2026-08-31 15:26:35 +08:00 |
|
|
akshatvishu
|
f321cdbe16
|
[Bugfix] Use Seed-TTS reference audio with MiniCPM-o 4.5 (#6628)
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
|
2026-08-31 10:27:34 +08:00 |
|
|
akshatvishu
|
c41925689d
|
[BugFix][Higgs-Audio-V3] Fix transcript and concurrent sampling tests (#6422)
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
|
2026-08-30 08:52:15 +00:00 |
|
|
Yueqian Lin
|
a6b559cf54
|
[Bugfix][MiniMax-Music3] Resolve stage subdirs against a real snapshot (#6640)
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
|
2026-08-30 06:51:24 +00:00 |
|
|
NATURE
|
87736a3cc5
|
[Bugfix][MiniCPM-o] Bound full-duplex Talker context with sliding recompute (#6626)
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
|
2026-08-30 00:49:42 +08:00 |
|
|
psv666andGao Han
|
5c47fd0bc8
|
[Bugfix][Qwen3-Omni] Handle missing packed modules mapping (#6748)
Signed-off-by: psv666 <2693925048@qq.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
|
2026-08-29 15:24:49 +08:00 |
|
|
|
143ccc5a2f
|
Fix Qwen3-Omni audio encoder TP when heads are not divisible by TP size (#4322)
Signed-off-by: abrahamzewoudie <azewoudi@amd.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
|
2026-08-29 14:52:02 +08:00 |
|
|
boatman
|
bfe8967ca2
|
[Bugfix][StepAudio2] Fix async-chunk metadata mismatch and boundary glitches (#5917)
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
|
2026-08-28 21:45:24 -07:00 |
|
|
|
be335a86f9
|
fix(npu): extend bf16 autocast to NPU in MossAudioTokenizer decode (#6664)
Signed-off-by: Wallbreazzz <110282866+Wallbreazzz@users.noreply.github.com>
Signed-off-by: Canlin Guo <961750412@qq.com>
Co-authored-by: Canlin Guo <961750412@qq.com>
Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
|
2026-08-28 04:01:49 -07:00 |
|
|
Wallbreazzz
|
4ad455e61e
|
[Feature][TTS] adaptive chunk ramp (Phase 2 buffer-feedback controller) (#6001)
Signed-off-by: Wallbreazzz <110282866+Wallbreazzz@users.noreply.github.com>
|
2026-08-28 02:11:05 -07:00 |
|
|
Sy03
|
009b992c5b
|
[Bugfix][MiniCPM-o] Isolate duplex handoff failures and barge-in (#6170)
Signed-off-by: Sy03 <1370724210@qq.com>
|
2026-08-28 15:46:41 +08:00 |
|
|
|
b6a4baa659
|
[Feature] Support text encoder Disaggregation for H3 (#5885)
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: yuanwu <yuan.wu@intel.com>
Co-authored-by: lishunyang12 <lishunyang12@163.com>
Co-authored-by: yuanwu <yuan.wu@intel.com>
|
2026-08-28 14:27:05 +08:00 |
|
|
BruceLoveDecimalandamy-why-3459
|
ab8d45d3cc
|
[Bugfix][MiniCPM-o] Retire CFM DiT CUDA graphs a generation at a time (#6587)
Signed-off-by: BruceLoveDecimal <liuqihao970610@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
|
2026-08-28 14:07:57 +08:00 |
|
|
Hongsheng Liuandhsliuustc0106
|
4245fd6955
|
[Fix] Restore MammothModa2 support (#6694)
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
|
2026-08-28 08:32:32 +08:00 |
|
|
Zhou TaichangandClaude Opus 5
|
651b03225b
|
[Rebase] Rebase to vllm 0.28.0 (#6606)
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
2026-08-27 06:12:55 +00:00 |
|
|
Sy03andsy03
|
432b2c6ca3
|
[Model] Add native full-duplex Nemotron VoiceChat serving (#6089)
Signed-off-by: Sysy <1370724210@qq.com>
Signed-off-by: sy03 <137072421@qq.com>
Co-authored-by: sy03 <137072421@qq.com>
|
2026-08-26 13:19:20 -07:00 |
|
|
|
f90e992dc1
|
[Model] Optimize CosyVoice3 TensorRT stream handoff (#5673)
Signed-off-by: Allen Wu <allenwu2795@gmail.com>
Co-authored-by: TRAE CLI <noreply@bytedance.com>
Co-authored-by: Sy03 <1370724210@qq.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
|
2026-08-25 22:51:46 -07:00 |
|
|
wy17003andHongsheng Liu
|
b454e34966
|
[Refactor][2/N Scheduler]Replace _FULL_PAYLOAD_INPUT_STAGES with a resolved stage transport capability. (#6149)
Signed-off-by: wy17003 <yaow17003@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
|
2026-08-25 15:24:59 +08:00 |
|
|
NATURE
|
702a409664
|
[Feature][MiniCPM-o] Support configurable concurrent duplex sessions (#6021)
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
|
2026-08-25 02:27:30 +00:00 |
|
|
Pranav ThakurandZhou Taichang
|
53b60bd9bc
|
[Bugfix] Invalidate stale ref_audio cache on local file modification (#4873) (#5670)
Signed-off-by: pranavthakur0-0 <pranavthakurdew@gmail.com>
Co-authored-by: Zhou Taichang <tzhouam@connect.ust.hk>
|
2026-08-25 10:06:32 +08:00 |
|
|
bjf-frzandHongsheng Liu
|
072bfc02dd
|
[FEAT]Add FastVideo VSA backend for Wan2.2 (#4820)
Signed-off-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
|
2026-08-24 14:48:31 +08:00 |
|
|
SYLAR
|
1c917b2751
|
[Bugfix] Fix Qwen3-Omni AWQ quantization name mapping (#5687)
Signed-off-by: lishunyang12 <lishunyang12@163.com>
|
2026-08-23 15:13:36 +00:00 |
|
|
wangyu
|
1e2a6446f7
|
[CI] Remove AudioX and MammothModa2 support (#6353)
Signed-off-by: wangyu <410167048@qq.com>
|
2026-08-23 04:22:05 +00:00 |
|
|
akshatvishu
|
39c16d75a9
|
[tts][bugfix] Fix CosyVoice3 sampling and stage handoff (#6424)
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
|
2026-08-22 21:20:57 -07:00 |
|
|
wangyu
|
2e096788a6
|
[CI]Remove SoulX-Singer support (#6362)
Signed-off-by: wangyu <410167048@qq.com>
|
2026-08-23 07:16:32 +08:00 |
|
|
boatmanandHongsheng Liu
|
3781214675
|
[Refactor] Move model-specific capability metadata into TTS adapters (#6138)
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
|
2026-08-22 14:26:49 +08:00 |
|
|
amy-why-3459
|
42059f19d5
|
[Bugfix][MiniCPM-o] Cap offline Talker generation at remaining context (#6458)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
|
2026-08-22 12:42:49 +08:00 |
|
|
Nick Cao
|
cfbf1392c5
|
[Bugfix] Drop librosa reintroduced by Step-Audio2 (#6467)
Signed-off-by: Nick Cao <ncao@redhat.com>
|
2026-08-21 21:33:07 +00:00 |
|