vllm-omni

0 Fork 0

Files

T

tzhouamandClaude Fable 5.1 55192de984 [Edge][WP4] Add hardware adaptation: probe, derived overrides, auto profile, calibration

* vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch
  (ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores,
  big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU
  via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu /
  npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests).
* vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from
  available memory minus weights, max_num_seqs, per-stage disjoint CPU thread
  ranges on the big cluster, eager/graph switches for small or unified memory,
  adaptive-chunk parameters, timeouts from a load cost model.
* vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk
  arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT.
* vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml ->
  edge/hardware/<class>.yaml -> derived overrides -> cached calibration,
  materialized as a temp deploy YAML (platform overlays are superseded).
* benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY,
  taskset, profile overrides) driving stream_latency_bench with auto profiles.
* CPU-only tests for probe snapshots, derivation rules, calibration and auto.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-09 11:01:40 +08:00

duplex

[Benchmark][MiniCPM-o] Add Omni-DuplexEval support (#6634 )

2026-08-27 16:31:50 +08:00

metrics

[Bugfix] Avoid invalid TPOT metrics (#6696 )

2026-08-29 11:56:37 +00:00

patch

[Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental (#6196 )

2026-09-05 08:20:21 +00:00

benchmark_video_response_encoding.py

[Frontend] Encode interleaved video frames in parallel (#6776 )