anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
2,942 Commits 235 Branches 31 Tags
c4e61133fbb5587e0cf8972ce32ffac4e37fa739
Commit Graph
185 Commits
Author SHA1 Message Date
tzhouamandClaude Opus 5 c4e61133fb [Edge][CPU] Fused norms off by default, and register the switches this branch added 
Two corrections and the paperwork they imply.

The fused-RMSNorm registration defaulted to *on*, which contradicts the
measurement it was committed with: those kernels are 3.6x and 4.5x faster in
isolation but measured 72.7 tok/s against 74.0 end to end, because inductor
already fuses the native norm into the residual add and an opaque call breaks
that fusion. The implementations stay registered, so the provider remains
selectable through `kernel_config.ir_op_priority` on a machine where the trade
lands differently; only the default ranking is left alone now, and the test
asserts both states rather than just the opt-in one.

The four switches this branch added (VLLM_OMNI_AOT_CACHE_SHIM,
_CPU_FUSED_NORMS, _CPU_RELEASE_MEMORY, _SPARK_GATE_REDUCTION) were not in the
reviewed environment-variable inventory, so
`test_statically_resolvable_environment_accesses_are_classified` failed. They
are classified now, the reviewed count moves 28 -> 32 with a note saying why,
and each has a row in docs/configuration/environment_variables.md carrying the
measurement that justifies its default.

`VLLM_OMNI_CPU_FAST_SAMPLER` and `VLLM_OMNI_STEP_STATS_SYNC` remain
unclassified and that test still fails on them. They arrived in 6ce8de1b, in
files this branch does not touch, so they are left to whoever owns that work
rather than documented from a guess.

Under `-m "core_model and cpu"`: 2593 passed before these fixes with 9
failures, of which two were mine. The rest are CUDA-only tests in a CPU venv
("Torch not compiled with CUDA enabled") plus test_qnn_export.py, which
another session is actively editing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-12 14:55:24 +08:00
tzhouamandClaude Fable 5.1 536866b916 [Edge] Classify and document the edge env vars; fix the client decoder call 
- environment_variable_inventory: VLLM_OMNI_CPU_INDUCTOR / HW_PROFILE /
  INIT_TIMELINE / STEP_STATS_DIR (public), STAGE_ID / INIT_TIMELINE_DIR
  (internal); reference page rows added.
- edge_decoder_client.LocalDecoder: batched_chunked_decode takes a [B, Q, F]
  tensor, not a list (the WP5 e2e run failed with a TypeError); the decoder
  now also carries the ramp schedule attributes the Code2Wav stage sets.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 15:22:52 +08:00
tzhouamandClaude Fable 5.1 8e912db6c8 [Edge][WP6] Slim the AR/TTS import path and add an edge dependency set 
* diffusers is no longer imported by vllm_omni.diffusion.data (annotation only);
  vllm_omni.diffusion.io_support (diffusion registry + distributed state) is
  imported lazily by config_factory and async_omni_engine, so constructing a
  Qwen3-TTS pipeline config / AsyncOmni no longer loads diffusers,
  x_transformers or whisper (guarded by tests/config/test_import_guard.py).
* requirements/common.txt = core.txt (AR/speech runtime) + media.txt
  (diffusion/video/extra audio); VLLM_OMNI_EDGE_BUILD=1 makes setup.py use
  requirements/<device>-edge.txt (core only); `edge` extra adds websockets and
  sounddevice for on-device clients.

Measured on this host (CUDA venv): `import vllm` 7.0-7.9 s, `import vllm_omni`
adds ~20 s of which ~17 s is transformers' own import (transformers.utils.import_utils)
pulled in by vllm_omni.patch / vllm_omni.version; that floor is not
vLLM-Omni's and is unchanged. Guard test: no diffusion module loaded on the TTS path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 11:14:50 +08:00
tzhouamandClaude Fable 5.1 49a839ecf9 [Edge][WP5] Add codec-token streaming mode with on-device decoding 
* qwen3_tts_talker_only pipeline (single terminal LLM_AR stage,
  final_output_type "latent") + vllm_omni/deploy/qwen3_tts_talker_only.yaml;
  a terminal talker publishes its per-step codes.audio rows to the client
  (omni_client_multimodal_output_keys).
* /v1/audio/speech/stream: session.config.output_mode="codec_tokens" streams
  codec.start (JSON) + binary frames (<IHB seq/n_frames/flags + int16 rows,
  vllm_omni/entrypoints/openai/codec_stream.py) + codec.done; new input.cancel
  message aborts an in-flight generation for both codec and PCM sessions
  (single reader task + session inbox replaces direct socket reads); optional
  session.config.seed; audio.done gains "cancelled" only when it happened.
* serving_speech._generate_codec_chunks aligns the cumulative code tensor to
  output token ids (drops prefill placeholder/EOS rows) and emits deltas.
* TTS adapters: supports_codec_stream / codec_stream_spec (Qwen3-TTS: 16
  codebooks, 2048 ids, 12.5 Hz, 24 kHz, decoder qwen3_tts_tokenizer_v2_12hz).
* examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py
  decodes locally with Qwen3TTSTokenizerV2Decoder (ramp schedule, 72-frame
  context, cancel/reset), codec_parity_check.py and
  benchmarks/tts/codec_stream_e2e.py compare against the two-stage server.
* docs/serving/codec_token_streaming.md; CPU unit tests for the wire format,
  alignment, session config and handler (frames, unsupported adapter, cancel);
  L3 e2e tests/e2e/online_serving/test_qwen3_tts_codec_stream.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 11:10:13 +08:00
BruceLoveDecimal 5cd925631c [Feature][LingBot World]stepwise execution (#6844) 
Signed-off-by: BruceLoveDecimal <liuqihao970610@gmail.com>
Signed-off-by: liuqihao <liuqihao970610@gmail.com>
 2026-09-07 13:38:23 +08:00
Zeyu Huang | 黃澤宇asukaqaq-sHongsheng Liu
 		f26f9c3a7f [Diffusion] Implement paged AR↔DiT KV connector ("v1", reusing vllm's native mooncake) (#6310) 
Signed-off-by: asukaqaq-s <1311722138@qq.com>
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: asukaqaq-s <1311722138@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-09-06 05:41:54 +08:00
SYLAR ea3cf99894 [diffusion][feature] Add component-selective offload policies (#5929) 
Signed-off-by: lishunyang12 <lishunyang12@163.com>
 2026-09-06 05:40:09 +08:00
Zhou TaichangandClaude Fable 5 142151fef5 Parallel stage initialization (admission + SH/EX device locks) (Worker refactor 1/N) (#5224) 
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-09-05 22:22:41 +08:00
Hongsheng Liu d4096e01fe [Model] Add native MAGI-2 Preview diffusion support (#5918) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-05 15:32:30 +08:00
SYLARandhsliuustc0106 7493cbd3c2 [Bugfix] Fix Qwen3-Omni MoE backend selection (#7019) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-04 21:16:58 +08:00
Yuekai ZhangandClaude Fable 5 4179b0cae1 [Perf][NemotronVoiceChat] Fast defaults for all three deploy profiles: 2.8x offline, realtime full-duplex audio (#6354) 
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-09-03 18:49:19 -07:00
Trigger e43c787544 [Model] Add BAGEL step execution and continuous batching (#6359) 2026-09-04 01:36:11 +00:00
maithilijoshi20andmaithilijoshi20 63ed0fdef4 [Bugfix] Fix vLLM serve error response imports (#6707) 
Signed-off-by: maithilijoshi20 <maithilijoshi2003@gmail.com>
Signed-off-by: maithilijoshi20 <97733343+maithilijoshi20@users.noreply.github.com>
Co-authored-by: maithilijoshi20 <maithilijoshi2003@gmail.com>
 2026-09-02 15:48:01 -07:00
汪志鹏 c59b48b47e [Feature][MiniMax-H3] Complete VSA and Ulysses support (#6909) 
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
 2026-09-02 07:19:21 -04:00
DanaerLeeandHongsheng Liu 0288c3f56e [Model] Support SenseNova-U1.5-8B-MoT and its distilled 8-step LoRA (#6516) 
Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-09-01 22:53:53 +08:00
Andy Zhou 2298359f67 [Bugfix][Diffusion] Preserve stage attention backend (#6645) 
Signed-off-by: AndyZhou952 <jzhoubc@connect.ust.hk>
 2026-09-01 20:18:44 +08:00
Alex Brooks a65e89cbd4 [Core] Validate PipelineConfig in __post_init__ (#6829) 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
 2026-09-01 11:17:57 +08:00
m0g3r 8fe913e461 [Core] Flag pipelines with no terminal output stage in PipelineConfig.validate() (#6291) 
Signed-off-by: m0g3r <87276771+m0g3r@users.noreply.github.com>
 2026-08-30 21:33:06 +00:00
baonudesifeizhai 8be601ed7f [Diffusion] Add native SymmMem Fast Ulysses transport (#6340) 
Signed-off-by: roG0d <baonudesifeizhai@gmail.com>
 2026-08-30 04:14:21 +00:00
YancyandAsthenia 73893e73ea [Bugfix] Fix create_error_response import across vLLM versions (#6773) 
Signed-off-by: Asthenia <asthenia0412@gmail.com>
Co-authored-by: Asthenia <asthenia0412@gmail.com>
 2026-08-29 17:01:51 +00:00
Canlin Guolishunyang12yuanwu
 		b6a4baa659 [Feature] Support text encoder Disaggregation for H3 (#5885) 
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: yuanwu <yuan.wu@intel.com>
Co-authored-by: lishunyang12 <lishunyang12@163.com>
Co-authored-by: yuanwu <yuan.wu@intel.com>
 2026-08-28 14:27:05 +08:00
Hongsheng Liuandhsliuustc0106 4245fd6955 [Fix] Restore MammothModa2 support (#6694) 
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
 2026-08-28 08:32:32 +08:00
NATURE 54c1fc64b7 [Bugfix] Gate async-chunk segment resume by session mode (#6680) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
 2026-08-27 18:52:09 +08:00
Zheng Wengangandamy-why-3459 94b1546ce3 [Model] Qwen3-Omni: register thinker-only pipeline for Instruct serve (#6284) 
Signed-off-by: ZhengWG <zwg0606@gmail.com>
Signed-off-by: Zheng Wengang <zwg0606@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-27 08:21:03 +00:00
Zhou TaichangandClaude Opus 5 651b03225b [Rebase] Rebase to vllm 0.28.0 (#6606) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-27 06:12:55 +00:00
Yueqian LinandYueqian Lin 8cfede4309 [Perf][Engine] Event-driven orchestration loop (opt-in) — S1 of #4855 (#5221) 
Signed-off-by: Yueqian Lin <gao@yueqian.us>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Yueqian Lin <gao@yueqian.us>
 2026-08-26 22:15:25 +08:00
Gao Han 7439ed41c2 Revert "[Bugfix] Restore name-based model detection for HF cache snapshot paths" (#6642) 2026-08-26 15:02:47 +08:00
Yueqian Lin 7ba5e8a146 [Bugfix] Restore name-based model detection for HF cache snapshot paths (#6624) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-08-25 23:27:35 -07:00
Hongsheng Liu a708ae5b7a [Bugfix] Reconcile env-var inventory with post-#6217 main drift (#6631) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-08-26 06:13:28 +08:00
Hongsheng Liu 780e8d40aa [Doc] Add environment variables configuration reference (#6217) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-26 04:58:54 +08:00
Hongsheng Liuandhsliuustc0106 c608910d20 [Diffusion] Register HWR mmap for direct DLO H2D (#6591) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
 2026-08-25 23:30:11 +08:00
wy17003andHongsheng Liu b454e34966 [Refactor][2/N Scheduler]Replace _FULL_PAYLOAD_INPUT_STAGES with a resolved stage transport capability. (#6149) 
Signed-off-by: wy17003 <yaow17003@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-25 15:24:59 +08:00
Hongsheng Liuandhsliuustc0106 8360346a66 [Diffusion] Integrate no-AllGather DLO with Host Weight Runtime (#6486) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
 2026-08-25 08:03:53 +08:00
bjf-frzandHongsheng Liu 072bfc02dd [FEAT]Add FastVideo VSA backend for Wan2.2 (#4820) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-24 14:48:31 +08:00
wangyu 1e2a6446f7 [CI] Remove AudioX and MammothModa2 support (#6353) 
Signed-off-by: wangyu <410167048@qq.com>
 2026-08-23 04:22:05 +00:00
wangyu 2e096788a6 [CI]Remove SoulX-Singer support (#6362) 
Signed-off-by: wangyu <410167048@qq.com>
 2026-08-23 07:16:32 +08:00
maithilijoshi20andHongsheng Liu bd4f9acfd3 [Bugfix] Preserve pipeline sampling constraints (#6182) 
Signed-off-by: maithilijoshi20 <97733343+maithilijoshi20@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-22 14:57:09 +00:00
Rahul SteigerHongsheng LiuRahul Steiger
 		b39f599b09 [Core] Add UniProc diffusion executor for single-GPU (#6308) 
Signed-off-by: Rahul Steiger <rsteiger@nvidia.com>
Signed-off-by: Rahul Steiger <rsteiger@aws-cmh-slurm-1-vscode-04.cm.cluster>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Rahul Steiger <rsteiger@aws-cmh-slurm-1-vscode-04.cm.cluster>
 2026-08-22 21:58:21 +08:00
Wang fuyinandHongsheng Liu 72ee535fcd [Diffusion] Add native paged KV backend for diffusion workers (#6102) 
Signed-off-by: Acerak01-fy <wfy2003324@163.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-22 21:45:45 +08:00
Inesh Reddy Chappidi cd3867c373 [Model] Suppress silence codec tokens for the first N Qwen3-TTS decod… (#5048) 
Signed-off-by: Inesh Reddy <ineshreddy249@gmail.com>
 2026-08-20 09:22:37 -07:00
amy-why-3459andCursor 9c61d5b67b [Bugfix][MiniCPM-o] Stop Talker mid-utterance truncation and load cod… (#6346) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-08-20 11:50:07 +08:00
bjf-frzandHongsheng Liu 14bd8f8772 [REFACT]Refactor diffusion parallel state (#5531) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-19 11:23:10 +08:00
Hongsheng Liu c3de04e127 [Bugfix] Avoid eager pi0 runtime import in pipeline registry (#6322) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-18 23:42:03 +08:00
Wang fuyinandHongsheng Liu e1c5f59aab [Config] Reuse vLLM configs in VllmOmniConfig (#6050) 
Signed-off-by: Acerak01-fy <wfy2003324@163.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-18 21:27:58 +08:00
Yukim1andzwhzzz0821 0a1846aa45 [Diffusion] Add native KV cache initialization and Scheduler-managed block allocation (#6094) 
Signed-off-by: zwhzzz0821 <zwhzzz0821@users.noreply.github.com>
Co-authored-by: zwhzzz0821 <zwhzzz0821@users.noreply.github.com>
 2026-08-18 07:00:24 +00:00
Yueqian Lin b283215c28 [Model] Add MiniMax Music 3 text-to-music (#6186) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-08-18 04:33:13 +00:00
amy-why-3459 3c6648d8a5 [Bugfix][NPU] Break the pytest DiffusionOutput circular import (#6293) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-18 10:11:10 +08:00
Yichen WangandDidan Deng 2a9f6aaa8e Add π0 VLA model support (#4222) 
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
 2026-08-17 14:29:17 +08:00
Yukim1andHongsheng Liu baba7d1e2e [Refactor] Remove legacy stage_args YAML loader (#6200) 
Signed-off-by: zwhzzz0821 <2831474076@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-16 11:22:28 +08:00
Ma1oneZhang 6f36fd3b78 [Bugfix] Preserve object storage URIs during model resolution (#5036) 
Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
 2026-08-15 14:11:48 +08:00