anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
2,942 Commits 235 Branches 31 Tags
c4e61133fbb5587e0cf8972ce32ffac4e37fa739
Commit Graph
45 Commits
Author SHA1 Message Date
tzhouamandClaude Fable 5.1 37c0ea9a6b [Edge][WP8] Per-step overhead attribution, edge scheduling helpers, CPU thread sweep 
- vllm_omni/edge/step_stats.py: process-local counters enabled by
  VLLM_OMNI_STEP_STATS_DIR (scheduler schedule/update_from_output, engine-core
  step, chunk hop enqueue->put / put->get / get, orchestrator dispatch), dumped
  per process and merged into an attribution table (share of the 80 ms frame).
  Hooks: stage_engine_core_proc (instrument after construction, dump at exit),
  chunk_transfer_adapter (put/get timing, put->get via the shm lock-file mtime),
  orchestrator dispatch loop. Bench: stream_latency_bench.py --step-stats.
- vllm_omni/edge/scheduling.py: playback slack, prefill-throttle decision and
  the no-preemption KV admission rule (exact bytes/token from the talker KV
  geometry); adapt.derive_overrides now sizes max_num_seqs with it and
  auto_profile passes the geometry read from the model config.
- Engine re-reads parallel_stage_init after apply_deploy_profile so the edge
  profile's orchestrator default takes effect (was read before the profile).
- benchmarks/tts/cpu_thread_sweep.py: CPU-only 8/16/32-thread sweep (+inductor)
  with per-step attribution.
- Tests: tests/core/test_edge_scheduling.py, tests/edge/test_step_stats.py,
  tests/benchmarks/test_cpu_thread_sweep.py, profile default test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 14:55:55 +08:00
tzhouamandClaude Fable 5.1 55192de984 [Edge][WP4] Add hardware adaptation: probe, derived overrides, auto profile, calibration 
* vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch
  (ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores,
  big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU
  via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu /
  npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests).
* vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from
  available memory minus weights, max_num_seqs, per-stage disjoint CPU thread
  ranges on the big cluster, eager/graph switches for small or unified memory,
  adaptive-chunk parameters, timeouts from a load cost model.
* vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk
  arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT.
* vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml ->
  edge/hardware/<class>.yaml -> derived overrides -> cached calibration,
  materialized as a temp deploy YAML (platform overlays are superseded).
* benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY,
  taskset, profile overrides) driving stream_latency_bench with auto profiles.
* CPU-only tests for probe snapshots, derivation rules, calibration and auto.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 11:01:40 +08:00
tzhouamandClaude Fable 5.1 073d31b9cd [Edge][WP2] Add initialization timeline and cold-start profiler 
* vllm_omni/engine/init_timeline.py: InitTimeline phase collector; engine
  phases (stages_init, g3_device_lock, g2_spawn/ready) and worker phases
  (proc_entry, engine_core_init, init_device, load_model/load_weights,
  talker_mtp_init, attn_prewarm, cudagraph_capture, talker_mtp_capture,
  code2wav_cudagraph, predictor_setup_compile/warmup/cudagraph) recorded via
  worker_phase/timed_phase; workers append JSONL under
  VLLM_OMNI_INIT_TIMELINE_DIR (inherited at spawn) and the engine merges them.
  Enabled by VLLM_OMNI_INIT_TIMELINE=<json> or log_stats; table logged at INFO,
  partial table on orchestrator startup timeout, suggested timeouts logged.
* omni_base.py: named deploy profiles default to stage_init_timeout=900 /
  init_timeout=1500 unless given explicitly.
* gpu_model_runner._init_talker_mtp tolerates max_cudagraph_capture_size=None;
  gpu_ar_model_runner.shutdown skips CUDA graph teardown on CPU.
* benchmarks/tts/init_profile.py: runs stream_latency_bench --init-only per
  configuration (default, cold_cache, eager_stage1, eager_both,
  parallel_stage_init, edge, ...) and tabulates phase durations.
* stream_latency_bench: --parallel-stage-init / --deploy-profile flags; the
  resource poller no longer shadows Thread._stop and skips GPU sampling on CPU.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 11:01:39 +08:00
tzhouamandClaude Fable 5.1 b70207f57f [Edge][WP0] Add offline streaming-latency benchmark for chunked TTS 
Adds benchmarks/tts/stream_latency_bench.py (AsyncOmni driver reusing the
Qwen3-TTS example's prompt builder; --model override, warm-up/repeat,
--init-only, per-chunk arrivals, peak RSS and per-process GPU memory) and
benchmarks/tts/stream_latency_metrics.py (TTFA, underrun-free playback start,
stall at TTFA, RTF streamed/total, chunk gaps, percentiles) with CPU unit
tests. Results go under benchmarks/tts/edge_results/stream_latency/.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-09 10:24:53 +08:00
chickeytonClaude Fable 5Gao Han
 		5d7fdf9350 [Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental (#6196) 
Signed-off-by: chickeyton <ngton2014@gmail.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-09-05 08:20:21 +00:00
NATURERuirui Yang | Reinamy-why-3459
 		8d98cc0cb7 [Benchmark] Add local OmniInteract performance cases (#6817) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Co-authored-by: Ruirui Yang | Rein <73573651+R2-Y@users.noreply.github.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-09-04 10:36:02 +08:00
akshatvishu f321cdbe16 [Bugfix] Use Seed-TTS reference audio with MiniCPM-o 4.5 (#6628) 
Signed-off-by: akshatvishu <akshatnayak197@gmail.com>
 2026-08-31 10:27:34 +08:00
mokekeandmokashliu 1a9d598d2b [Frontend] Encode interleaved video frames in parallel (#6776) 
Signed-off-by: mokashliu <mokashliu@tencent.com>
Co-authored-by: mokashliu <mokashliu@tencent.com>
 2026-08-31 07:04:27 +08:00
Gao Han 71748388f8 [Bugfix] Recover Omni benchmark TPOT from Stage 0 metrics (#6818) 
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
 2026-08-31 00:43:25 +08:00
Honghan Zhuandamy-why-3459 72a1ce488b [Bugfix] Avoid invalid TPOT metrics (#6696) 
Signed-off-by: zhuhh97 <zhuhonghan@huawei.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-29 11:56:37 +00:00
amy-why-3459 da94c3ebe1 [Bugfix][MiniCPM-o] Bind omni-duplex video frames to the unit they close (#6404) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-28 16:58:20 +08:00
Zhanshenzhuiandamy-why-3459 c6df39e9f7 [Benchmark][MiniCPM-o] Add Omni-DuplexEval support (#6634) 
Signed-off-by: Zhanshenzhui <99891002+zyforsure@users.noreply.github.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-27 16:31:50 +08:00
NATURE 497c537c6f [Benchmark] Add local OmniInteract realtime benchmark (#6522) 
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
 2026-08-25 15:32:58 +00:00
Zhang BowenandHongsheng Liu 4882a36c41 [Model][Frontend] MiniMax-H3: Add opt-in planar video response encoding (#6288) 
Signed-off-by: MosCloud <bwzhang1991@163.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-08-22 04:04:14 +00:00
psv666andamy-why-3459 199ef7779c [Bugfix][MiniCPM-o] Load Daily-Omni QA metadata from the hub cache (#6276) 
Signed-off-by: psv666 <2693925048@qq.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-18 10:53:55 +08:00
amy-why-3459 7cd826d28c [Bugfix][MiniCPM-o] Align Daily-Omni offline loading and duplex soft-… (#6095) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-12 18:36:31 +08:00
Sy03andYueqian Lin 58007c0084 [New Model] Support IndexTTS 2.5 (#5957) 
Signed-off-by: Sysy <1370724210@qq.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-08-11 23:00:35 +00:00
Ruirui Yang | Reinpsv666Cursoramy-why-3459
 		322600bfb2 [CI][MiniCPM-o] Add MiniCPM-o 4.5 accuracy and performance coverage (#5524) 
Signed-off-by: R2-Y <ruiruyang2@gmail.com>
Signed-off-by: psv666 <2693925048@qq.com>
Co-authored-by: psv666 <2693925048@qq.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-08-11 14:53:49 +08:00
amy-why-3459andCursor 67aa6c55a4 [Bugfix][MiniCPM-o] Align Daily-Omni bench with MiniCPM interleaved A… (#5625) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-31 22:41:00 -07:00
rein yangandSy03 ad3a162eb8 [perf] Minicpm o4.5 optimization (#5228) 
Signed-off-by: R2-Y <ruiruyang2@gmail.com>
Co-authored-by: Sy03 <1370724210@qq.com>
 2026-07-24 17:20:39 +08:00
ZacheryAU 4c23a3164b [Refactor] Split vllm_omni/entrypoints/cli/benchmark/serve.py (#5206) 2026-07-23 16:00:32 +00:00
amy-why-3459 d897852258 [BugFix] Fix the issue of incorrect generated tokens (#4461) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-06-17 02:31:48 +00:00
Zhou Taichang 7b837944a1 [Rebase] Rebase to vllm 0.23.0 (#4286) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
 2026-06-13 16:43:02 +00:00
Jianbao Huang 6cefe91370 [Perf] Enable fused RMSNorm for HunyuanImage3 (#3959) 2026-06-05 07:41:56 +00:00
Yueqian LinandHongsheng Liu f99e8b2d3c [Bench] Add audio-streaming continuity metric for TTS (#3618) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-05-20 05:08:56 +00:00
bjf-frzandbjfwhite 1fa734419e [Refactor]Rename diffusion benchmark backend to endpoint (#3137) 
Signed-off-by: bjf-frz <frz123db@gmail.com>
Signed-off-by: bjfwhite <baijingfan1@huawei.com>
Co-authored-by: bjfwhite <baijingfan1@huawei.com>
 2026-05-19 11:30:33 +08:00
Yueqian Lin 990566aef1 [Bugfix][TTS] Drop meaningless TTFT from speech-endpoint benchmarks (#3674) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
 2026-05-18 17:18:18 +08:00
Zhou TaichangandYuanheng Zhao 7f2b1e9386 [Rebase] Rebase to vllm 0.20.0 (#3232) 
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk>
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com>
Co-authored-by: Yuanheng Zhao <54058983+yuanheng-zhao@users.noreply.github.com>
 2026-04-30 09:56:59 +12:00
Yueqian LinandHongsheng Liu afa2b09684 feat: add MOSS-TTS-Nano single-stage TTS support (#2753) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-04-27 10:26:26 +08:00
Alex Brooks 52c9ce2dac [CI] Fix Seed TTS Simple Unit Tests (#3126) 
Signed-off-by: Alex Brooks <albrooks@redhat.com>
 2026-04-25 10:21:49 +08:00
Yueqian Lin 39e3fd3d12 [Benchmark] Universal TTS benchmark: Qwen3-TTS + VoxCPM2 with 3 task types (voice-clone/default/design) (#2835) 
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Signed-off-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
 2026-04-24 23:35:33 +08:00
amy-why-3459 3aa84c4742 [Tests]Add accuracy benchmark L4 test cases (#2843) 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
 2026-04-23 09:40:43 +08:00
NumberWan ead87aa521 [BugFix] Surface diffusion metrics in chat completions; sanitize Bagel img2img mm kwargs (#2932) 
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
 2026-04-22 16:57:01 +00:00
wangyu e18cb89d5a [CI] Update test markers and configurations to use 'full_model' for L4 nightly tests (#2641) 
Signed-off-by: wangyu <410167048@qq.com>
 2026-04-22 15:14:20 +08:00
Dnoob 7e7efdd1ab [Bugfix] Fix benchmark Total input tokens for multimodal requests (#2540) (#2549) 
Signed-off-by: Dnoob <dxpouo@gmail.com>
 2026-04-08 11:00:53 +00:00
WeiQing Chen c392ce21e9 [Accuracy Benchmark] feat: add accuracy benchmark integrations for t2i and i2i (#1917) 
Signed-off-by: David Chen <530634352@qq.com>
Signed-off-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
 2026-03-26 17:27:29 +08:00
wangyu fafa248665 [Test] Implement mock HTTP request handling in benchmark CLI tests (#2014) 
Signed-off-by: yenuo26 <410167048@qq.com>
Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
 2026-03-20 16:13:39 +08:00
TJian 0239057cba [CI] [ROCm] Setup test-ready.yml and test-merge.yml (#2017) 
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2026-03-19 14:26:31 +00:00
Daniel Huang c107f0b336 Buildkite hardware ci xpu test (#1340) 
Signed-off-by: Daniel Huang <daniel1.huang@intel.com>
 2026-03-14 11:24:05 +08:00
Zeyu Huang | 黃澤宇andHongsheng Liu a19098233b [Test] L4 complete diffusion feature test for Qwen-Image-Edit models (#1682) 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-03-11 18:10:06 +08:00
wangyu cd2234a1bf [CI] Modify some CI test cases to run on L4 environment to reduce H100 resource usage. (#1543) 
Signed-off-by: yenuo26 <410167048@qq.com>
Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
 2026-02-28 14:41:51 +08:00
Yuanheng Zhao ed08a216c1 [Misc] Migrate L1 tests to use pytest-mock (#1315) 
Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com>
Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com>
 2026-02-26 17:08:39 +08:00
John Liu BUAA f117a07271 [Bugfix] reused metrics to modify the API Server token statistics in Stream Response (#1301) 
Signed-off-by: John Liu BUAA <liukecheng97@gmail.com>
 2026-02-12 17:24:01 +08:00
AliciaandHongsheng Liu d3ea94309d [CI] Add pytest markers to current tests and update the doc. (#577) 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-02-09 16:49:45 +08:00
wangyuwangyu31577Hongsheng Liu
 		07f4c3fdd6 [Bugfix] fix benchmark audio timing error and add benchmark test (#1109) 
Signed-off-by: wangyu31577 <wangyu31577@hundsun.com>
Co-authored-by: wangyu31577 <wangyu31577@hundsun.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
 2026-01-31 10:15:26 +08:00