vllm-omni

2,942 Commits 235 Branches 31 Tags

2 Commits

Include renames

Author	SHA1	Message	Date
tzhouamandClaude Fable 5.1	602e39647e	[Edge][WP2] Finalize edge profile init knobs from init_profile; WP4 CPU emulation results init_profile 20260909-112200 (1.7B, L20X, warm): default 152.0 s, eager_stage1 149.9 s, eager_both 110.9 s (first TTFA 15 s, no graphs), parallel_stage_init 94.6 s, edge 145.7 s. Edge profile now carries `orchestrator: parallel_stage_init: true`, applied by the deploy_profile selector (explicit values win); graphs stay on. hw_emulation 20260909-110229 (CPU venv, 0.6B, deploy_profile=auto): x86_cpu_avx2 (4c, fp32) TTFA 2.65 s RTF 11.9; x86_cpu_avx512 (8c) TTFA 0.68 s RTF 2.45; x86_cpu_amx (16c) TTFA 0.51 s RTF 1.72; arm64 logic-only same as avx2. Emulation rows now record the auto-detected hardware class. Adds tests/e2e/offline_inference/test_qwen3_tts_edge_profile.py (L3, one GPU). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-09 14:40:13 +08:00
tzhouamandClaude Fable 5.1	55192de984	[Edge][WP4] Add hardware adaptation: probe, derived overrides, auto profile, calibration * vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch (ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores, big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu / npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests). * vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from available memory minus weights, max_num_seqs, per-stage disjoint CPU thread ranges on the big cluster, eager/graph switches for small or unified memory, adaptive-chunk parameters, timeouts from a load cost model. * vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT. * vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml -> edge/hardware/<class>.yaml -> derived overrides -> cached calibration, materialized as a temp deploy YAML (platform overlays are superseded). * benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY, taskset, profile overrides) driving stream_latency_bench with auto profiles. * CPU-only tests for probe snapshots, derivation rules, calibration and auto. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-09 11:01:40 +08:00

Author

SHA1

Message

Date

tzhouamandClaude Fable 5.1

602e39647e

[Edge][WP2] Finalize edge profile init knobs from init_profile; WP4 CPU emulation results

init_profile 20260909-112200 (1.7B, L20X, warm): default 152.0 s, eager_stage1
149.9 s, eager_both 110.9 s (first TTFA 15 s, no graphs), parallel_stage_init
94.6 s, edge 145.7 s. Edge profile now carries `orchestrator: parallel_stage_init:
true`, applied by the deploy_profile selector (explicit values win); graphs stay on.

hw_emulation 20260909-110229 (CPU venv, 0.6B, deploy_profile=auto):
x86_cpu_avx2 (4c, fp32) TTFA 2.65 s RTF 11.9; x86_cpu_avx512 (8c) TTFA 0.68 s
RTF 2.45; x86_cpu_amx (16c) TTFA 0.51 s RTF 1.72; arm64 logic-only same as avx2.
Emulation rows now record the auto-detected hardware class.

Adds tests/e2e/offline_inference/test_qwen3_tts_edge_profile.py (L3, one GPU).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-09 14:40:13 +08:00

tzhouamandClaude Fable 5.1

55192de984

[Edge][WP4] Add hardware adaptation: probe, derived overrides, auto profile, calibration

* vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch
  (ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores,
  big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU
  via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu /
  npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests).
* vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from
  available memory minus weights, max_num_seqs, per-stage disjoint CPU thread
  ranges on the big cluster, eager/graph switches for small or unified memory,
  adaptive-chunk parameters, timeouts from a load cost model.
* vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk
  arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT.
* vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml ->
  edge/hardware/<class>.yaml -> derived overrides -> cached calibration,
  materialized as a temp deploy YAML (platform overlays are superseded).
* benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY,
  taskset, profile overrides) driving stream_latency_bench with auto profiles.
* CPU-only tests for probe snapshots, derivation rules, calibration and auto.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-09 11:01:40 +08:00