tzhouam and Claude Opus 5
ca4dd105e4
[Edge][CPU] An explicit KV budget must beat the legacy env var
...
`CpuPlatform.check_and_update_config` ends with a block whose own comment
reads "Lagecy setting":
env_key = "VLLM_CPU_KVCACHE_SPACE"
if env_key in os.environ and os.environ[env_key] != "":
cache_config.kv_cache_memory_bytes = int(os.environ[env_key]) * GiB
It runs unconditionally, so `--kv-cache-memory-bytes` is silently discarded
whenever the env var is set -- which it has to be on CPU, or the engine tries
to claim most of the machine. Two consequences for edge serving: the budget
can only be expressed in whole GiB (parsed with `int()`, so the smallest
non-zero cache is 1 GiB), and vllm-omni's per-stage `kv_cache_memory_bytes`,
which is how the deploy YAMLs express per-stage budgets, has no effect at all
on CPU.
Verified by allocation rather than by memory, because allocation cannot drift:
with the env at 1 GiB, asking for 128 MiB produced the same 17 881 cache
tokens as asking for nothing, and with the env at 4 GiB it produced 71 527 --
the explicit value never applied. After restoring the obvious precedence
(explicit wins, env stays the default for callers that set nothing) the same
request yields 2 231 tokens, the 128 MiB it asked for.
The Qwen3-TTS CPU overlay now sizes stage 0 from the model -- 112 KiB/token,
so 2 seqs at 2048 tokens needs 560 MiB, not 4 GiB. Stage 1 is left alone on
purpose: Code2Wav is a generation stage whose cache is not a plain attention
KV, and under-sizing makes vLLM preempt by recompute, which on a single-stream
edge deployment is a stall.
What this saves in resident memory is deliberately *not* claimed. VmRSS on
this host moves 1.2-2.0 GB between identical runs, because it counts the
mmap'd checkpoint's file pages; an earlier 3.1 GB figure came from a
comparison that changed the model file as well as the budget and does not
stand. RssAnon is the metric to use and that measurement is still in flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-12 14:16:01 +08:00
tzhouam and Claude Fable 5.1
74acd66307
[Edge][WP7] Add CudaIpcConnector (CUDA IPC for device tensor leaves over shm control plane)
...
Wraps SharedMemoryConnector: CUDA tensor leaves are exported with
torch.multiprocessing reduce_tensor + an inter-process CUDA event, rebuilt and
cloned by the consumer, released by the producer after an ack file appears.
CPU leaves and metadata keep the shm path. Registered as "CudaIpcConnector";
deploy/qwen3_tts_cuda_ipc.yaml selects it for the talker->code2wav hop.
Adoption in the edge profile is measurement-gated (hop latency bench).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 14:41:33 +08:00
tzhouam and Claude Fable 5.1
602e39647e
[Edge][WP2] Finalize edge profile init knobs from init_profile; WP4 CPU emulation results
...
init_profile 20260909-112200 (1.7B, L20X, warm): default 152.0 s, eager_stage1
149.9 s, eager_both 110.9 s (first TTFA 15 s, no graphs), parallel_stage_init
94.6 s, edge 145.7 s. Edge profile now carries `orchestrator: parallel_stage_init:
true`, applied by the deploy_profile selector (explicit values win); graphs stay on.
hw_emulation 20260909-110229 (CPU venv, 0.6B, deploy_profile=auto):
x86_cpu_avx2 (4c, fp32) TTFA 2.65 s RTF 11.9; x86_cpu_avx512 (8c) TTFA 0.68 s
RTF 2.45; x86_cpu_amx (16c) TTFA 0.51 s RTF 1.72; arm64 logic-only same as avx2.
Emulation rows now record the auto-detected hardware class.
Adds tests/e2e/offline_inference/test_qwen3_tts_edge_profile.py (L3, one GPU).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 14:40:13 +08:00
tzhouam and Claude Fable 5.1
49a839ecf9
[Edge][WP5] Add codec-token streaming mode with on-device decoding
...
* qwen3_tts_talker_only pipeline (single terminal LLM_AR stage,
final_output_type "latent") + vllm_omni/deploy/qwen3_tts_talker_only.yaml;
a terminal talker publishes its per-step codes.audio rows to the client
(omni_client_multimodal_output_keys).
* /v1/audio/speech/stream: session.config.output_mode="codec_tokens" streams
codec.start (JSON) + binary frames (<IHB seq/n_frames/flags + int16 rows,
vllm_omni/entrypoints/openai/codec_stream.py) + codec.done; new input.cancel
message aborts an in-flight generation for both codec and PCM sessions
(single reader task + session inbox replaces direct socket reads); optional
session.config.seed; audio.done gains "cancelled" only when it happened.
* serving_speech._generate_codec_chunks aligns the cumulative code tensor to
output token ids (drops prefill placeholder/EOS rows) and emits deltas.
* TTS adapters: supports_codec_stream / codec_stream_spec (Qwen3-TTS: 16
codebooks, 2048 ids, 12.5 Hz, 24 kHz, decoder qwen3_tts_tokenizer_v2_12hz).
* examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py
decodes locally with Qwen3TTSTokenizerV2Decoder (ramp schedule, 72-frame
context, cancel/reset), codec_parity_check.py and
benchmarks/tts/codec_stream_e2e.py compare against the two-stage server.
* docs/serving/codec_token_streaming.md; CPU unit tests for the wire format,
alignment, session config and handler (frames, unsupported adapter, cancel);
L3 e2e tests/e2e/online_serving/test_qwen3_tts_codec_stream.py.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 11:10:13 +08:00
tzhouam and Claude Fable 5.1
35e43b9983
[Edge][WP3] Add CPU OmniPlatform (AR + generation stages on vLLM's CPU backend)
...
* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform)
selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed
(builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes
like vLLM does). NUMA-based memory queries, one logical device, no graph
wrapper (sentinel type for isinstance checks), inductor opt-in via
VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported.
* vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose
vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner /
CPUGenerationModelRunner compose the Omni runners with CPUModelRunner
(no graphs, no async omni output; sleep/wake return error ACKs).
* patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder
and prompt-embeds dtype follow the model dtype (float32/float16 on CPU).
* vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async
scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges).
* requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md.
Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads):
Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms,
RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline
committed under benchmarks/tts/edge_results/.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 11:02:14 +08:00
tzhouam and Claude Fable 5.1
55192de984
[Edge][WP4] Add hardware adaptation: probe, derived overrides, auto profile, calibration
...
* vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch
(ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores,
big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU
via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu /
npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests).
* vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from
available memory minus weights, max_num_seqs, per-stage disjoint CPU thread
ranges on the big cluster, eager/graph switches for small or unified memory,
adaptive-chunk parameters, timeouts from a load cost model.
* vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk
arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT.
* vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml ->
edge/hardware/<class>.yaml -> derived overrides -> cached calibration,
materialized as a temp deploy YAML (platform overlays are superseded).
* benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY,
taskset, profile overrides) driving stream_latency_bench with auto profiles.
* CPU-only tests for probe snapshots, derivation rules, calibration and auto.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 11:01:40 +08:00
tzhouam and Claude Fable 5.1
b22dcc59f7
[Edge][WP1] Add edge deploy profile and deploy_profile selector
...
* vllm_omni/deploy/edge/qwen3_tts.yaml: single-device Qwen3-TTS profile with a
ramped codec chunk schedule (2->4->8->16->25 frames) that removes the ~80 ms
playback stall of the default 1->25 schedule, decode batch 1, max_num_seqs 4.
* deploy_profile="edge" (Omni/AsyncOmni kwarg, --deploy-profile CLI, headless
serve) resolves vllm_omni/deploy/<profile>/<model_type>.yaml; mutually
exclusive with deploy_config; explicit profiles never fall back silently.
* tests/helpers/stage_config.py: modify_stage_config anchors a relative
base_config before writing the temp YAML so overlay-derived configs resolve.
* async_omni_engine.py also carries the init-timeline hooks used by WP2.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 10:27:33 +08:00
BruceLoveDecimal
5cd925631c
[Feature][LingBot World]stepwise execution ( #6844 )
...
Signed-off-by: BruceLoveDecimal <liuqihao970610@gmail.com >
Signed-off-by: liuqihao <liuqihao970610@gmail.com >
2026-09-07 13:38:23 +08:00
NATURE and amy-why-3459
ff6e906a25
[Model] Preserve MiniCPM-o Talker full attention until capacity ( #6799 )
...
Signed-off-by: natureofnature <wzliu@connect.hku.hk >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-09-06 22:25:35 +08:00
SYLAR and hsliuustc0106
7493cbd3c2
[Bugfix] Fix Qwen3-Omni MoE backend selection ( #7019 )
...
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com >
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com >
2026-09-04 21:16:58 +08:00
Yuekai Zhang and Claude Fable 5
4179b0cae1
[Perf][NemotronVoiceChat] Fast defaults for all three deploy profiles: 2.8x offline, realtime full-duplex audio ( #6354 )
...
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-09-03 18:49:19 -07:00
Yuxuan Rho and Hongsheng Liu
7992c2213e
[Config] Remove diffusion_batch_size and use max_num_seqs instead. ( #6484 )
...
Signed-off-by: LyxWxj <yuxuanlou621@foxmail.com >
Signed-off-by: Yuxuan Rho <104440472+LyxWxj@users.noreply.github.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-09-01 01:02:54 +00:00
akshatvishu
c41925689d
[BugFix][Higgs-Audio-V3] Fix transcript and concurrent sampling tests ( #6422 )
...
Signed-off-by: akshatvishu <akshatnayak197@gmail.com >
2026-08-30 08:52:15 +00:00
NATURE
87736a3cc5
[Bugfix][MiniCPM-o] Bound full-duplex Talker context with sliding recompute ( #6626 )
...
Signed-off-by: natureofnature <wzliu@connect.hku.hk >
2026-08-30 00:49:42 +08:00
Gao Han
676a35fa8b
Fix MiniCPM-o concurrent audio first-packet latency ( #6767 )
...
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk >
2026-08-29 23:27:47 +08:00
Wallbreazzz
4ad455e61e
[Feature][TTS] adaptive chunk ramp (Phase 2 buffer-feedback controller) ( #6001 )
...
Signed-off-by: Wallbreazzz <110282866+Wallbreazzz@users.noreply.github.com >
2026-08-28 02:11:05 -07:00
b6a4baa659
[Feature] Support text encoder Disaggregation for H3 ( #5885 )
...
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com >
Signed-off-by: lishunyang12 <lishunyang12@163.com >
Signed-off-by: yuanwu <yuan.wu@intel.com >
Co-authored-by: lishunyang12 <lishunyang12@163.com >
Co-authored-by: yuanwu <yuan.wu@intel.com >
2026-08-28 14:27:05 +08:00
Hongsheng Liu and hsliuustc0106
4245fd6955
[Fix] Restore MammothModa2 support ( #6694 )
...
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com >
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com >
2026-08-28 08:32:32 +08:00
Zheng Wengang and amy-why-3459
94b1546ce3
[Model] Qwen3-Omni: register thinker-only pipeline for Instruct serve ( #6284 )
...
Signed-off-by: ZhengWG <zwg0606@gmail.com >
Signed-off-by: Zheng Wengang <zwg0606@gmail.com >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-27 08:21:03 +00:00
Sy03 and sy03
432b2c6ca3
[Model] Add native full-duplex Nemotron VoiceChat serving ( #6089 )
...
Signed-off-by: Sysy <1370724210@qq.com >
Signed-off-by: sy03 <137072421@qq.com >
Co-authored-by: sy03 <137072421@qq.com >
2026-08-26 13:19:20 -07:00
Zhichao Zhang
816335cb46
[Doc] Add MiMo-Audio recipe for RTX 5090/5090D 32GB ( #6559 )
...
Signed-off-by: chaosansui <zzc15560846421@163.com >
2026-08-26 16:26:24 +08:00
amy-why-3459
5de6f7c37a
[Bugfix][MiniCPM-o] Serve native duplex from shipping YAMLs and fence… ( #6619 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-26 10:56:51 +08:00
NumberWan
bc8ea489ef
[BugFix] Qwen-image performance regressed - Avoid mapping diffusion_batch_size onto scheduler max_num_seqs ( #6525 )
...
Signed-off-by: NumberWan <wantszkin2003@gmail.com >
2026-08-25 03:36:11 -07:00
wangyu
1e2a6446f7
[CI] Remove AudioX and MammothModa2 support ( #6353 )
...
Signed-off-by: wangyu <410167048@qq.com >
2026-08-23 04:22:05 +00:00
wangyu
2e096788a6
[CI]Remove SoulX-Singer support ( #6362 )
...
Signed-off-by: wangyu <410167048@qq.com >
2026-08-23 07:16:32 +08:00
Zhou Taichang
3ad64c6a12
[Bugfix][Qwen3-Omni] Stabilize thinker MRoPE compilation ( #6449 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
2026-08-22 19:42:56 +00:00
amy-why-3459
42059f19d5
[Bugfix][MiniCPM-o] Cap offline Talker generation at remaining context ( #6458 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-22 12:42:49 +08:00
Inesh Reddy Chappidi
cd3867c373
[Model] Suppress silence codec tokens for the first N Qwen3-TTS decod… ( #5048 )
...
Signed-off-by: Inesh Reddy <ineshreddy249@gmail.com >
2026-08-20 09:22:37 -07:00
amy-why-3459 and Cursor
9c61d5b67b
[Bugfix][MiniCPM-o] Stop Talker mid-utterance truncation and load cod… ( #6346 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-08-20 11:50:07 +08:00
Yukim1
6f55a06fab
[Docs] Remove stale stage config references ( #6270 )
...
Signed-off-by: zwhzzz0821 <2831474076@qq.com >
2026-08-20 07:09:12 +08:00
Yueqian Lin
b283215c28
[Model] Add MiniMax Music 3 text-to-music ( #6186 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
2026-08-18 04:33:13 +00:00
Zhou248
5d09cf27a9
[MiniCPM-o][NPU] Enable MiniCPM-o Code2Wav NPUGraph replay ( #5604 )
...
Signed-off-by: Zhou248 <62542847+Zhou248@users.noreply.github.com >
2026-08-17 15:09:51 +08:00
Yichen Wang and Didan Deng
2a9f6aaa8e
Add π0 VLA model support ( #4222 )
...
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com >
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com >
Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com >
2026-08-17 14:29:17 +08:00
35cacf020c
[Model] Add Gepard-1.0 native-AR FSQ/NanoCodec TTS (offline inference) ( #5666 )
...
Signed-off-by: Mengjie Zhao <zmj0129@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
2026-08-14 22:06:05 -07:00
Lucas(Ruijie)
90a08c0eda
[Model][TTS][WIP] Add dots.tts (rednote-hilab) — continuous-AR 48kHz TTS ( #4765 )
...
Signed-off-by: Moore-Z <zhumoore9661996@gmail.com >
2026-08-13 17:58:33 +00:00
Sy03
dbc0dd6d85
[Bugfix][Higgs-Audio-V3] Disable XQA decode ( #6068 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
Signed-off-by: Sysy <1370724210@qq.com >
2026-08-13 10:00:50 -07:00
Canlin Guo
bbb709cfb5
[BugFix] Use MOSS-TTS-Local official sample params ( #6156 )
...
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com >
2026-08-13 08:56:26 +00:00
liwener and amy-why-3459
877beeaa49
[Perf][MiniCPM-O-4.5]Add CUDA Graph For CFM DiT estimator ( #6082 )
...
Signed-off-by: liwener <liwener.c@gmail.com >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-13 09:36:17 +08:00
Yuekai Zhang and Claude Fable 5
5be7541d01
[Model] Support NVIDIA-NemotronLabs-VoiceChat-11B offline speech-to-speech ( #5842 )
...
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-12 16:25:48 +08:00
9b0df3909a
[Feature] timestamps for TTS via a forced-aligner pooling ( #4795 )
...
Signed-off-by: wjinxu <1299461899@qq.com >
Co-authored-by: xurui <xurui1@modelbest.onaliyun.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-12 05:33:34 +00:00
Sy03 and Yueqian Lin
58007c0084
[New Model] Support IndexTTS 2.5 ( #5957 )
...
Signed-off-by: Sysy <1370724210@qq.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
2026-08-11 23:00:35 +00:00
e04210d690
[Hardware][MUSA] Enable Qwen3-Omni ModelOpt FP8 inference ( #5671 )
...
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-11 17:19:09 +08:00
322600bfb2
[CI][MiniCPM-o] Add MiniCPM-o 4.5 accuracy and performance coverage ( #5524 )
...
Signed-off-by: R2-Y <ruiruyang2@gmail.com >
Signed-off-by: psv666 <2693925048@qq.com >
Co-authored-by: psv666 <2693925048@qq.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com >
2026-08-11 14:53:49 +08:00
boatman and Yueqian Lin
4f9b65074d
[Perf][Qwen3-TTS]Cached Incremental Decode ( #5202 )
...
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com >
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com >
2026-08-11 05:24:04 +00:00
boatman
fc8946fcf5
[Perf][MiniCPM-O-4.5]Add CUDA Graph For HiFTGenerator ( #5869 )
...
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com >
2026-08-09 22:29:49 +08:00
Yueqian Lin and Sy0307
e788ef6e76
[Feature] PersonaPlex (Moshi-based full-duplex S2S): native vLLM port + duplex serving ( #4771 )
...
Signed-off-by: Yueqian Lin <linyueqian@outlook.com >
Signed-off-by: Sy0307 <24841857+Sy0307@users.noreply.github.com >
Co-authored-by: Sy0307 <24841857+Sy0307@users.noreply.github.com >
2026-08-04 20:10:33 -07:00
Yukim1
7a2007cc17
[Refactor] Remove legacy --stage-configs-path from the serve CLI ( #5647 )
...
Signed-off-by: zwhzzz0821 <2831474076@qq.com >
2026-08-03 23:41:52 +08:00
amy-why-3459 and Cursor
67aa6c55a4
[Bugfix][MiniCPM-o] Align Daily-Omni bench with MiniCPM interleaved A… ( #5625 )
...
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-07-31 22:41:00 -07:00
NATURE
0a03359224
[Bugfix][MiniCPM-o] Stabilize single-GPU audio generation ( #5637 )
...
Signed-off-by: natureofnature <wzliu@connect.hku.hk >
2026-08-01 03:01:06 +00:00
0a51a18570
[Bugfix] Leave MiniCPM-o vocoder memory headroom ( #5621 )
...
Signed-off-by: lishunyang12 <lishunyang03@gmail.com >
Signed-off-by: SYLAR <125541396+lishunyang12@users.noreply.github.com >
Co-authored-by: lishunyang12 <lishunyang03@gmail.com >
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
2026-07-31 03:35:13 +00:00
