tzhouam
723c550209
Integrate portable edge graph backends with Omni runtime
...
Extract versioned stage contracts, model shared CPU/iGPU/NPU memory constraints, and add explicit graph routes, reservations, output acknowledgements, and worker retirement. Fix native Windows shared-buffer lifetime and host paths. Document qualified hardware paths and remaining rollout gates.
2026-09-22 13:12:33 +08:00
tzhouam and Claude Fable 5.1
97d97b6503
[edge-infer W8] Omni serve CLI: uvloop optional, selector-loop asyncio.run on Windows
...
uvloop ships no Windows wheels, so 'import uvloop' at module scope in
entrypoints/cli/serve.py killed vllm-omni serve on Windows at import
(measured in the serving e2e after the fcntl fix). _run_forever() uses
uvloop where present and asyncio.run on WindowsSelectorEventLoopPolicy
otherwise, which zmq.asyncio needs. The remaining serve-import blocker
on the Windows venv was a missing dependency, torchaudio (vLLM's own
requirement), installed into the venvs rather than patched.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-18 19:09:22 +08:00
tzhouam and Claude Fable 5.1
33de5d8629
Merge vLLM-Omni v0.29.0rc1 into edge-infer/omni-0.29.0
...
The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of
rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while
our 0.28-based tree needs seven modules v0.29.0 no longer has
(vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args,
openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant,
models.hunyuan_v1).
Ten files conflicted. Upstream's structure is the base everywhere; the
fork's features are re-attached to it:
- deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config
pops deploy_config (async_omni_engine, cli/serve); the deploy-profile
selector and the resolve_model_config_path chain it needs stay in
entrypoints/utils.py; stage_overrides takes rc1's dict type.
- config_factory keeps the lazy diffusion output-type import; the diffusion
registry import in rc1's new config/resolver.py is made lazy for the same
WP6 reason (TTS path must not load diffusion).
- requirements: our core/media split kept; rc1's transformers>=5.13 floor
and kernels==0.16.1 applied in core.txt / media.txt.
- TTS: rc1's split_granularity and bounded seed plus our output_mode=
codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation;
our private _conditioning_cache_salt is superseded by rc1's public
tts_adapters/base.py::conditioning_cache_salt.
- stage_runtime: rc1's single ExitStack G2 path (lock released before READY)
replaces our two-path block, with the four worker_phase names the init
profiler consumes still emitted; _cpu_placed / host admission kept.
Not yet run against vLLM 0.29.0: the tests listed in the W6 record.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-18 09:57:20 +08:00
tzhouam and Claude Opus 5
a7e6c412ab
[Edge] A deploy YAML must mean the same thing whichever flag loaded it
...
`vllm_omni/deploy/edge/qwen3_tts.yaml` carries `orchestrator:
parallel_stage_init: true`, but `_apply_profile_orchestrator_defaults` ran
only on the `deploy_profile` path. Pointing `--deploy-config` at the same file
loaded its stages and silently dropped the block, so the edge deployment
started sequentially:
--deploy-profile edge 30.76 s
--deploy-config .../edge/qwen3_tts.yaml 63.76 s (same file)
with nothing in the log to say why. The block now applies however the config
was reached. Explicit arguments still win, and the allow-list stays narrow --
a test checks an unrecognised key cannot be injected through it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-12 21:29:10 +08:00
Zhou Taichang and Claude Opus 5
aff7d64948
[Rebase] Rebase to vLLM 0.29.0 ( #7230 )
...
Signed-off-by: tzhouam <tzhouam@connect.ust.hk >
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-10 12:08:28 +08:00
Sy03
b02862beee
[Model] Optimize MOSS-TTS Local batched execution and streaming codec ( #7202 )
...
Signed-off-by: Sy03 <1370724210@qq.com >
2026-09-09 13:02:12 -07:00
0e602509aa
[BugFix] Add field validators for /v1/audio/generate request ( #4741 )
...
Signed-off-by: Shaun Walsh <shaunwalsh24@gmail.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: Nick Cao <ncao@redhat.com >
2026-09-09 13:38:42 +00:00
tzhouam and Claude Fable 5.1
c893b638ad
[Edge][WP5] Accumulate per-step codec delta rows (DELTA mode delivers only the newest row)
...
Measured after the DELTA fix: 53 engine outputs for 52 rows, but the client
still received everything at once because each per-step payload is the newest
row only and the row alignment treated it as a cumulative snapshot (one yield
at the first row, one at the end). accumulate_codec_rows folds delta rows and
the final consolidated matrix into one growing matrix; unit test drives the
generator with per-step deltas and asserts one yield per step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 16:45:24 +08:00
tzhouam and Claude Fable 5.1
380e8022dd
[Edge][WP5] Normalize DELTA-mode codec snapshots (list of cumulative code matrices) before row alignment
...
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 16:31:56 +08:00
tzhouam and Claude Fable 5.1
b22dabe270
[Edge][WP5] Keep DELTA outputs for talker-only (codec-token) deployments
...
The speech path forced FINAL_ONLY for Qwen3-TTS deployments without async
chunks, so a talker-only server emitted one engine output per request and all
codec frames arrived at once (measured: 1 engine output per request, first
codes at output #1 ). A talker-only stage (final_output + latent) now keeps
DELTA outputs so codec frames stream per step.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 16:21:31 +08:00
tzhouam and Claude Fable 5.1
b8d1e44ebe
[Edge][WP5] Codec stream diagnostics: per-request output/codes counters, client saves received codes, offline parity diag
...
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 16:02:02 +08:00
tzhouam and Claude Fable 5.1
602e39647e
[Edge][WP2] Finalize edge profile init knobs from init_profile; WP4 CPU emulation results
...
init_profile 20260909-112200 (1.7B, L20X, warm): default 152.0 s, eager_stage1
149.9 s, eager_both 110.9 s (first TTFA 15 s, no graphs), parallel_stage_init
94.6 s, edge 145.7 s. Edge profile now carries `orchestrator: parallel_stage_init:
true`, applied by the deploy_profile selector (explicit values win); graphs stay on.
hw_emulation 20260909-110229 (CPU venv, 0.6B, deploy_profile=auto):
x86_cpu_avx2 (4c, fp32) TTFA 2.65 s RTF 11.9; x86_cpu_avx512 (8c) TTFA 0.68 s
RTF 2.45; x86_cpu_amx (16c) TTFA 0.51 s RTF 1.72; arm64 logic-only same as avx2.
Emulation rows now record the auto-detected hardware class.
Adds tests/e2e/offline_inference/test_qwen3_tts_edge_profile.py (L3, one GPU).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 14:40:13 +08:00
Allen Wu and TRAE CLI
f2e30f46b6
[Bugfix][TTS] Isolate shared Higgs v3 reference encode from request cancellation ( #7076 )
...
Signed-off-by: Allen Wu <allenwu2795@gmail.com >
Co-authored-by: TRAE CLI <traecli@bytedance.com >
2026-09-08 20:32:05 -07:00
Jim Ban
358b7d79c1
[Bugfix][Frontend] Enforce image pixel limits for video input references ( #6963 )
...
Signed-off-by: BANANASJIM <bananasjim1@gmail.com >
2026-09-08 20:31:54 -07:00
Ziming Wang
43b697541a
[Bugfix][Frontend] Clear the P0 multimodal cache through the renderer ( #7003 )
...
Signed-off-by: ZenAlexa <zimingwang945@gmail.com >
2026-09-08 20:31:42 -07:00
tzhouam and Claude Fable 5.1
49a839ecf9
[Edge][WP5] Add codec-token streaming mode with on-device decoding
...
* qwen3_tts_talker_only pipeline (single terminal LLM_AR stage,
final_output_type "latent") + vllm_omni/deploy/qwen3_tts_talker_only.yaml;
a terminal talker publishes its per-step codes.audio rows to the client
(omni_client_multimodal_output_keys).
* /v1/audio/speech/stream: session.config.output_mode="codec_tokens" streams
codec.start (JSON) + binary frames (<IHB seq/n_frames/flags + int16 rows,
vllm_omni/entrypoints/openai/codec_stream.py) + codec.done; new input.cancel
message aborts an in-flight generation for both codec and PCM sessions
(single reader task + session inbox replaces direct socket reads); optional
session.config.seed; audio.done gains "cancelled" only when it happened.
* serving_speech._generate_codec_chunks aligns the cumulative code tensor to
output token ids (drops prefill placeholder/EOS rows) and emits deltas.
* TTS adapters: supports_codec_stream / codec_stream_spec (Qwen3-TTS: 16
codebooks, 2048 ids, 12.5 Hz, 24 kHz, decoder qwen3_tts_tokenizer_v2_12hz).
* examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py
decodes locally with Qwen3TTSTokenizerV2Decoder (ramp schedule, 72-frame
context, cancel/reset), codec_parity_check.py and
benchmarks/tts/codec_stream_e2e.py compare against the two-stage server.
* docs/serving/codec_token_streaming.md; CPU unit tests for the wire format,
alignment, session config and handler (frames, unsupported adapter, cancel);
L3 e2e tests/e2e/online_serving/test_qwen3_tts_codec_stream.py.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 11:10:13 +08:00
tzhouam and Claude Fable 5.1
073d31b9cd
[Edge][WP2] Add initialization timeline and cold-start profiler
...
* vllm_omni/engine/init_timeline.py: InitTimeline phase collector; engine
phases (stages_init, g3_device_lock, g2_spawn/ready) and worker phases
(proc_entry, engine_core_init, init_device, load_model/load_weights,
talker_mtp_init, attn_prewarm, cudagraph_capture, talker_mtp_capture,
code2wav_cudagraph, predictor_setup_compile/warmup/cudagraph) recorded via
worker_phase/timed_phase; workers append JSONL under
VLLM_OMNI_INIT_TIMELINE_DIR (inherited at spawn) and the engine merges them.
Enabled by VLLM_OMNI_INIT_TIMELINE=<json> or log_stats; table logged at INFO,
partial table on orchestrator startup timeout, suggested timeouts logged.
* omni_base.py: named deploy profiles default to stage_init_timeout=900 /
init_timeout=1500 unless given explicitly.
* gpu_model_runner._init_talker_mtp tolerates max_cudagraph_capture_size=None;
gpu_ar_model_runner.shutdown skips CUDA graph teardown on CPU.
* benchmarks/tts/init_profile.py: runs stream_latency_bench --init-only per
configuration (default, cold_cache, eager_stage1, eager_both,
parallel_stage_init, edge, ...) and tabulates phase durations.
* stream_latency_bench: --parallel-stage-init / --deploy-profile flags; the
resource poller no longer shadows Thread._stop and skips GPU sampling on CPU.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 11:01:39 +08:00
tzhouam and Claude Fable 5.1
b22dcc59f7
[Edge][WP1] Add edge deploy profile and deploy_profile selector
...
* vllm_omni/deploy/edge/qwen3_tts.yaml: single-device Qwen3-TTS profile with a
ramped codec chunk schedule (2->4->8->16->25 frames) that removes the ~80 ms
playback stall of the default 1->25 schedule, decode batch 1, max_num_seqs 4.
* deploy_profile="edge" (Omni/AsyncOmni kwarg, --deploy-profile CLI, headless
serve) resolves vllm_omni/deploy/<profile>/<model_type>.yaml; mutually
exclusive with deploy_config; explicit profiles never fall back silently.
* tests/helpers/stage_config.py: modify_stage_config anchors a relative
base_config before writing the temp YAML so overlay-derived configs resolve.
* async_omni_engine.py also carries the init-timeline hooks used by WP2.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-09 10:27:33 +08:00
2bae550c04
[Frontend] Opt-in WebSocket TTS split_granularity and session seed ( #7046 )
...
Signed-off-by: rk9595 <rakesh.kariya@somaiya.edu >
Signed-off-by: Rakesh Kariya <rakesh.kariya@somaiya.edu >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-08 19:15:59 -07:00
zhengjia and Cursor
81c1c460b3
[Bugfix][Frontend] Accept the msgpack-numpy package's numpy markers on the OpenPI endpoint ( #6051 )
...
Signed-off-by: zjli2013 <leezhengjiang@126.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-08 19:15:48 -07:00
NancyFyong and NancyFyong
e74fe3c6e8
[Model] Add Audio8 TTS Preview 0.6B (DualAR, 44.1 kHz codec) ( #6157 )
...
Signed-off-by: NancyFyong <NancyFyong@users.noreply.github.com >
Co-authored-by: NancyFyong <NancyFyong@users.noreply.github.com >
2026-09-08 19:00:25 -07:00
boatman
cdf027874c
[Feat][OmniVoice]Support Varlen Attn, Request-Batch and Step-Execution ( #6408 )
...
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com >
2026-09-08 18:58:38 -07:00
fdaf4e31bd
Add vLLM-Omni library info to Hugging Face Hub requests ( #5381 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-08 10:08:38 -06:00
liangmenghuang and Claude Fable 5.1
7a80b79698
[Bugfix][Model] GR00T-N1.7: honor the per-request seed for flow-matching noise ( #7253 )
...
Signed-off-by: liangmengh <liangmengh@nvidia.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-08 15:53:20 +00:00
LHXuuu
c9c25b33c3
[Frontend] Add server-side VAD for turn-based omni models ( #6618 )
...
Signed-off-by: LHXuuu <xulianhao.xlh@antgroup.com >
2026-09-08 03:17:00 -07:00
tlysanhuo
7c2e30afd3
[Bugfix] Bound omni benchmark per-request timeout to 15 min by default ( #7130 )
...
Signed-off-by: tly <2200895168@qq.com >
2026-09-08 02:00:25 -07:00
Chenguang Zheng
f3fabe4145
[Bugfix] Keep --disable-log-stats out of stage config resolution ( #7237 )
...
Signed-off-by: Chenguang ZHENG <645327136@qq.com >
2026-09-08 16:19:07 +08:00
Chenguang Zheng
087ac46318
[Refactor 1/N] Centralize Omni config resolution ( #5140 )
...
Signed-off-by: Chenguang ZHENG <645327136@qq.com >
2026-09-08 10:30:26 +08:00
ZacheryAU
b1ec136bd7
[Frontend] Add omni benchmark support for image and video endpoints ( #4728 )
2026-09-08 09:28:21 +08:00
boatman
043f03d576
[Migrate]Move model-specific helpers out of serving_speech ( #6697 )
...
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com >
2026-09-08 08:59:45 +08:00
FredHuangNV
85b0bad2d3
[Frontend][Model] Add Cosmos3 control uploads to video API ( #7027 )
...
Signed-off-by: Fred Huang <frhuang@nvidia.com >
2026-09-07 01:38:56 -07:00
SYLAR
ea3cf99894
[diffusion][feature] Add component-selective offload policies ( #5929 )
...
Signed-off-by: lishunyang12 <lishunyang12@163.com >
2026-09-06 05:40:09 +08:00
5d7fdf9350
[Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental ( #6196 )
...
Signed-off-by: chickeyton <ngton2014@gmail.com >
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk >
2026-09-05 08:20:21 +00:00
Hongsheng Liu
d4096e01fe
[Model] Add native MAGI-2 Preview diffusion support ( #5918 )
...
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com >
2026-09-05 15:32:30 +08:00
BruceLoveDecimal and Claude Opus 5
1e74807c35
[Feature][Moss-TTS] Reference Audio Preprocessing ( #4982 )
...
Signed-off-by: BruceLoveDecimal <39156883+BruceLoveDecimal@users.noreply.github.com >
Signed-off-by: liuqihao <liuqihao970610@gmail.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-04 16:36:26 -07:00
LOGO127
b6b992ac7b
[Bugfix] Fix Higgs Audio v3 voice-clone token validation ( #7065 )
...
Signed-off-by: luozijian <luozijian0924@gmail.com >
2026-09-04 16:06:47 -07:00
boatman
348fd0fc07
[Feat][dots.tts] Add Online Serving Support ( #6235 )
...
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com >
2026-09-04 14:44:10 -07:00
Yuekai Zhang and Claude Fable 5
4179b0cae1
[Perf][NemotronVoiceChat] Fast defaults for all three deploy profiles: 2.8x offline, realtime full-duplex audio ( #6354 )
...
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-09-03 18:49:19 -07:00
Eugene Zhao
ec22c7ac06
[Bugfix] Restore Omni health route ownership ( #6723 )
...
Signed-off-by: qujing226 <qujing226@gmail.com >
Signed-off-by: Eugene Zhao <113281545+qujing226@users.noreply.github.com >
2026-09-03 00:54:36 -07:00
Chiquan Mo and MciG-ggg
2b47faf601
[Bugfix] Validate --stage-overrides shape in parse_stage_overrides (drop field allowlist) ( #6230 )
...
Signed-off-by: Chiquan Mo <mcigggg399026@gmail.com >
Signed-off-by: MciG-ggg <MciG-ggg@users.noreply.github.com >
Co-authored-by: MciG-ggg <MciG-ggg@users.noreply.github.com >
2026-09-02 23:19:08 -07:00
maithilijoshi20 and maithilijoshi20
63ed0fdef4
[Bugfix] Fix vLLM serve error response imports ( #6707 )
...
Signed-off-by: maithilijoshi20 <maithilijoshi2003@gmail.com >
Signed-off-by: maithilijoshi20 <97733343+maithilijoshi20@users.noreply.github.com >
Co-authored-by: maithilijoshi20 <maithilijoshi2003@gmail.com >
2026-09-02 15:48:01 -07:00
GXIN and Hongsheng Liu
3ab582c993
[Bugfix][Qwen3-TTS] Reject task mismatches before EngineCore dispatch ( #6113 )
...
Signed-off-by: XIN GAO <1037396230@qq.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-09-02 11:16:55 -07:00
AuFlow and Hongsheng Liu
49c6fc1540
[Frontend] Add configurable output sample rate to Speech API ( #6553 )
...
Signed-off-by: 郑倩倩 <2458983186@qq.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-09-01 17:25:09 +08:00
Yuxuan Rho and Hongsheng Liu
7992c2213e
[Config] Remove diffusion_batch_size and use max_num_seqs instead. ( #6484 )
...
Signed-off-by: LyxWxj <yuxuanlou621@foxmail.com >
Signed-off-by: Yuxuan Rho <104440472+LyxWxj@users.noreply.github.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-09-01 01:02:54 +00:00
XRay2016
8e26dce1f4
[Diffusion] Add Boogu Edit-Turbo support docs and tests ( #6701 )
...
Signed-off-by: xRay2016 <1150722393@qq.com >
2026-09-01 08:41:23 +08:00
汪志鹏 and Hongsheng Liu
86b85c078b
[Feature][MiniMax-H3] Fuse the FastH3 four-step adapter at load time ( #6714 )
...
Signed-off-by: princepride <wangzhipeng628@gmail.com >
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com >
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com >
2026-08-31 12:04:58 +08:00
akshatvishu
f321cdbe16
[Bugfix] Use Seed-TTS reference audio with MiniCPM-o 4.5 ( #6628 )
...
Signed-off-by: akshatvishu <akshatnayak197@gmail.com >
2026-08-31 10:27:34 +08:00
SYLAR
759aa4ffeb
[Diffusion][Performance] Optimize MiniMax-H3 video output transfer ( #6824 )
...
Signed-off-by: lishunyang12 <lishunyang12@163.com >
2026-08-31 07:07:34 +08:00
mokeke and mokashliu
1a9d598d2b
[Frontend] Encode interleaved video frames in parallel ( #6776 )
...
Signed-off-by: mokashliu <mokashliu@tencent.com >
Co-authored-by: mokashliu <mokashliu@tencent.com >
2026-08-31 07:04:27 +08:00
baonudesifeizhai
8be601ed7f
[Diffusion] Add native SymmMem Fast Ulysses transport ( #6340 )
...
Signed-off-by: roG0d <baonudesifeizhai@gmail.com >
2026-08-30 04:14:21 +00:00
tzhouam