anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
2,940 Commits 235 Branches 31 Tags
ca4dd105e4f4c334db6eb102a04ca833cf809299
Commit Graph
792 Commits
Author SHA1 Message Date
tzhouamandClaude Fable 5.1 536866b916 [Edge] Classify and document the edge env vars; fix the client decoder call 
- environment_variable_inventory: VLLM_OMNI_CPU_INDUCTOR / HW_PROFILE /
  INIT_TIMELINE / STEP_STATS_DIR (public), STAGE_ID / INIT_TIMELINE_DIR
  (internal); reference page rows added.
- edge_decoder_client.LocalDecoder: batched_chunked_decode takes a [B, Q, F]
  tensor, not a list (the WP5 e2e run failed with a TypeError); the decoder
  now also carries the ramp schedule attributes the Code2Wav stage sets.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 15:22:52 +08:00
tzhouamandClaude Fable 5.1 fa8b1a878e [Edge] Docs: chunk ramp / edge profile section and an edge recipe for Qwen3-TTS 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 15:03:25 +08:00
tzhouamandClaude Fable 5.1 49a839ecf9 [Edge][WP5] Add codec-token streaming mode with on-device decoding 
* qwen3_tts_talker_only pipeline (single terminal LLM_AR stage,
  final_output_type "latent") + vllm_omni/deploy/qwen3_tts_talker_only.yaml;
  a terminal talker publishes its per-step codes.audio rows to the client
  (omni_client_multimodal_output_keys).
* /v1/audio/speech/stream: session.config.output_mode="codec_tokens" streams
  codec.start (JSON) + binary frames (<IHB seq/n_frames/flags + int16 rows,
  vllm_omni/entrypoints/openai/codec_stream.py) + codec.done; new input.cancel
  message aborts an in-flight generation for both codec and PCM sessions
  (single reader task + session inbox replaces direct socket reads); optional
  session.config.seed; audio.done gains "cancelled" only when it happened.
* serving_speech._generate_codec_chunks aligns the cumulative code tensor to
  output token ids (drops prefill placeholder/EOS rows) and emits deltas.
* TTS adapters: supports_codec_stream / codec_stream_spec (Qwen3-TTS: 16
  codebooks, 2048 ids, 12.5 Hz, 24 kHz, decoder qwen3_tts_tokenizer_v2_12hz).
* examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py
  decodes locally with Qwen3TTSTokenizerV2Decoder (ramp schedule, 72-frame
  context, cancel/reset), codec_parity_check.py and
  benchmarks/tts/codec_stream_e2e.py compare against the two-stage server.
* docs/serving/codec_token_streaming.md; CPU unit tests for the wire format,
  alignment, session config and handler (frames, unsupported adapter, cancel);
  L3 e2e tests/e2e/online_serving/test_qwen3_tts_codec_stream.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 11:10:13 +08:00
tzhouamandClaude Fable 5.1 35e43b9983 [Edge][WP3] Add CPU OmniPlatform (AR + generation stages on vLLM's CPU backend) 
* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform)
  selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed
  (builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes
  like vLLM does). NUMA-based memory queries, one logical device, no graph
  wrapper (sentinel type for isinstance checks), inductor opt-in via
  VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported.
* vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose
  vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner /
  CPUGenerationModelRunner compose the Omni runners with CPUModelRunner
  (no graphs, no async omni output; sleep/wake return error ACKs).
* patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder
  and prompt-embeds dtype follow the model dtype (float32/float16 on CPU).
* vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async
  scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges).
* requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md.

Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads):
Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms,
RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline
committed under benchmarks/tts/edge_results/.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 11:02:14 +08:00
Zeyu Huang | 黃澤宇andAlicia 4b96c57730 [skip ci][Doc] Module design doc for diffusion runtime (#6440) 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>
 2026-09-07 21:52:24 +08:00
Mu GuanLin 3204a0b2ad [Model][Perf] Optimize LTX Ulysses sequence parallelism (#7079) 
Signed-off-by: mglyn <1203789601@qq.com>
 2026-09-07 10:48:26 +00:00
FredHuangNV 85b0bad2d3 [Frontend][Model] Add Cosmos3 control uploads to video API (#7027) 
Signed-off-by: Fred Huang <frhuang@nvidia.com>
 2026-09-07 01:38:56 -07:00
WeiQing Chen b78c31eda6 [Doc] Update vLLM-Omni WeChat QR code (#7178) 
Signed-off-by: David Chen <530634352@qq.com>
 2026-09-07 06:28:30 +00:00
BruceLoveDecimal 5cd925631c [Feature][LingBot World]stepwise execution (#6844) 
Signed-off-by: BruceLoveDecimal <liuqihao970610@gmail.com>
Signed-off-by: liuqihao <liuqihao970610@gmail.com>
 2026-09-07 13:38:23 +08:00
Zeyu Huang | 黃澤宇 2df95de421 [Core][Diffusion] Refactor prompt updates into a modality-ready interaction framework (#6294) 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
 2026-09-07 11:40:04 +08:00
Xiaobaixia 615f97b9c7 [Feature] Define the runner preprocess phase contract and regression coverage (#7136) 
Signed-off-by: xiaobaixia222 <289982752@qq.com>
 2026-09-06 17:20:04 +00:00
汪志鹏andClaude Opus 5 af74a5a155 [Feature][MiniMax-H3] Support the full LightX2V Turbo LoRA matrix (#7062) 
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-09-06 20:23:36 +08:00
DMAN 06541d1f52 [Model] FP8 Support For Boogu-Image (#6925) 
Signed-off-by: Dmaner <2663515256@qq.com>
 2026-09-06 04:48:38 +00:00
Max Freedom Pollard 946b843e61 [Doc] Fix markdownlint findings in the serving API reference (#7111) 
Signed-off-by: Max Freedom Pollard <272618364+MaxFreedomPollard@users.noreply.github.com>
 2026-09-06 08:40:10 +08:00
SYLAR ea3cf99894 [diffusion][feature] Add component-selective offload policies (#5929) 
Signed-off-by: lishunyang12 <lishunyang12@163.com>
 2026-09-06 05:40:09 +08:00
chickeytonClaude Fable 5Gao Han
 		5d7fdf9350 [Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental (#6196) 
Signed-off-by: chickeyton <ngton2014@gmail.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-09-05 08:20:21 +00:00
Hongsheng Liu d4096e01fe [Model] Add native MAGI-2 Preview diffusion support (#5918) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-05 15:32:30 +08:00
vOv 13c617a140 [Model] Add sequence parallelism to SANA-Video 2B (#5940) 
Signed-off-by: cr-gao <gaochenrui@sjtu.edu.cn>
 2026-09-05 15:22:53 +08:00
boatman 348fd0fc07 [Feat][dots.tts] Add Online Serving Support (#6235) 
Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>
 2026-09-04 14:44:10 -07:00
TriggerandGao Han ea14b04e9f [Bugfix][BAGEL] Restore original denoise step count (#7049) 
Signed-off-by: Trigger <Meng.Bo.Wang@outlook.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-09-04 17:05:05 +00:00
SYLARandhsliuustc0106 22cefe04b4 [Bugfix] Update Hub kernel dependency floor (#7020) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-04 22:15:55 +08:00
SYLARandhsliuustc0106 7493cbd3c2 [Bugfix] Fix Qwen3-Omni MoE backend selection (#7019) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Co-authored-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-04 21:16:58 +08:00
NATURERuirui Yang | Reinamy-why-3459
 		8d98cc0cb7 [Benchmark] Add local OmniInteract performance cases (#6817) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: Ruirui Yang | Rein <73573651+R2-Y@users.noreply.github.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-09-04 10:36:02 +08:00
Trigger e43c787544 [Model] Add BAGEL step execution and continuous batching (#6359) 2026-09-04 01:36:11 +00:00
hurukawa 8f284b343a [Feature][Diffusion]: CFG parallelism support for Boogu-Image (#6786) 
Signed-off-by: nagisa <1434936049@qq.com>
Signed-off-by: nagisa-kun <1434936049@qq.com>
 2026-09-03 10:37:21 +08:00
GXINandHongsheng Liu 3ab582c993 [Bugfix][Qwen3-TTS] Reject task mismatches before EngineCore dispatch (#6113) 
Signed-off-by: XIN GAO <1037396230@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-09-02 11:16:55 -07:00
wangyulishunyang12CursorBo Li
 		95a9311696 [CI] Select mirror_hardwares presets via MIRROR_HW (H100/B200) (#5543) 
Signed-off-by: wangyu <410167048@qq.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
Co-authored-by: lishunyang12 <lishunyang12@163.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Bo Li <22713281+bobboli@users.noreply.github.com>
 2026-09-02 15:39:49 +00:00
Bo Li 5d20f6b900 [Doc] Refine TRTLLM attention guidance (#6724) 
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
 2026-09-02 19:42:37 +08:00
汪志鹏 c59b48b47e [Feature][MiniMax-H3] Complete VSA and Ulysses support (#6909) 
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
 2026-09-02 07:19:21 -04:00
vOv 57c08d34c3 [Model] Add TP and CFG parallelism to SANA-Video 2B (#5861) 
Signed-off-by: cr-gao <gaochenrui@sjtu.edu.cn>
 2026-09-02 17:44:41 +08:00
wangyuandamy-why-3459 b6e72f3879 [CI] Migrate L4 jobs to l4-k8s and split by GPU count (#6890) 
Signed-off-by: wangyu <410167048@qq.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-09-02 17:09:59 +08:00
DanaerLeeandHongsheng Liu 0288c3f56e [Model] Support SenseNova-U1.5-8B-MoT and its distilled 8-step LoRA (#6516) 
Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-09-01 22:53:53 +08:00
Hongsheng Liuandhsliu_ustc 44c8d1c449 docs: add concise API server endpoint guide (#6114) 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com>
 2026-09-01 20:34:09 +08:00
Hongsheng Liu d1c8beeb18 [Doc] Update README and docs for v0.28.0 (#6858) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-09-01 19:50:26 +08:00
AuFlowandHongsheng Liu 49c6fc1540 [Frontend] Add configurable output sample rate to Speech API (#6553) 
Signed-off-by: 郑倩倩 <2458983186@qq.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-09-01 17:25:09 +08:00
XRay2016 8e26dce1f4 [Diffusion] Add Boogu Edit-Turbo support docs and tests (#6701) 
Signed-off-by: xRay2016 <1150722393@qq.com>
 2026-09-01 08:41:23 +08:00
WeiQing Chen 4cca28e9bd [Doc] Update vLLM-Omni WeChat QR code (#6832) 
Signed-off-by: David Chen <530634352@qq.com>
 2026-08-31 09:44:33 +08:00
wangyu e338f0ece3 [CI]Split GPU jobs by cards_* and reject hand-written SKU marks (#6650) 
Signed-off-by: wangyu <410167048@qq.com>
Signed-off-by: [Your Name] <your.email@example.com>
 2026-08-29 18:42:33 +08:00
HAAZZZEEEE c50b7de167 [NPU][Diffusion] MiniMax-H3 RainFusion end_step tail fallback (#6037) 
Signed-off-by: HAAZZZEEEE <wang-taicheng@qq.com>
 2026-08-29 18:36:05 +08:00
ooooooye 5a5bc55c25 [Feature][Diffusion] Allow online INT8 quantization with DLO AllGather (#6573) 
Signed-off-by: brandneway <gyuan4892@gmail.com>
 2026-08-29 09:50:30 +00:00
Nguyen Kha Nhat LongCursorSamit
 		bb5bf8fd0a [Bugfix] Fix colocate-async sleep admission race and deliver AR abort tokens (#6367) 
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Samit <285365963@qq.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Samit <285365963@qq.com>
 2026-08-29 09:11:39 +00:00
Zijun HuangandWeiQing Chen 08cdee8dc4 [Feature] Support MiniMax-H3 FlashGen native LoRA with the legacy manager (#6666) 
Signed-off-by: Zijun Huang <zijunhua@usc.edu>
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
 2026-08-29 06:52:19 +00:00
Chendi.Xue 034af1f657 [CI][XPU]switch to use vLLM base docker for XPU (#6727) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2026-08-29 09:03:00 +08:00
Hongsheng Liu 6ec5ab11bd [Bugfix] Make HWR filesystem lifecycle transitions fail-closed (#6692) 
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
 2026-08-28 09:30:17 +08:00
Hongsheng Liuandhsliuustc0106 4245fd6955 [Fix] Restore MammothModa2 support (#6694) 
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
 2026-08-28 08:32:32 +08:00
Zheng Wengangandamy-why-3459 94b1546ce3 [Model] Qwen3-Omni: register thinker-only pipeline for Instruct serve (#6284) 
Signed-off-by: ZhengWG <zwg0606@gmail.com>
Signed-off-by: Zheng Wengang <zwg0606@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-27 08:21:03 +00:00
Zhou TaichangandClaude Opus 5 651b03225b [Rebase] Rebase to vllm 0.28.0 (#6606) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-27 06:12:55 +00:00
Hongsheng Liuandhsliuustc0106 da4a08b65b [Diffusion] Add FLUX.2-klein BF16 HWR contract (#6651) 
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
 2026-08-27 12:51:15 +08:00
SYLARandHongsheng Liu 0e44fc7406 [diffusion][feature] Add offline SVDQuant W4A4 support (#6162) 
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-27 08:45:53 +08:00
ChoHeeClaudeGao Han
 		c2723979c0 [Bugfix] Validate per-stage device layout before spawning workers (#5003) (#5742) 
Signed-off-by: ChoHee15 <cc5281@126.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-08-26 19:25:22 +00:00