tzhouamandClaude Fable 5.1
55192de984
* vllm_omni/edge/hardware_probe.py: HardwareProfile from /proc, /sys and torch (ISA flags incl. AVX-512/AMX and NEON/dotprod/i8mm/SVE, physical cores, big.LITTLE clusters, NUMA, RAM, CUDA discrete vs Tegra unified memory, NPU via override), hardware classes jetson / cuda_discrete / arm64_cpu / x86_cpu / npu_phone; VLLM_OMNI_HW_PROFILE=<json> overrides the probe (emulation, tests). * vllm_omni/edge/adapt.py: derive_overrides -> dtype, absolute KV budgets from available memory minus weights, max_num_seqs, per-stage disjoint CPU thread ranges on the big cluster, eager/graph switches for small or unified memory, adaptive-chunk parameters, timeouts from a load cost model. * vllm_omni/edge/calibrate.py: chunk-schedule calibration from measured chunk arrivals with a per-hardware-fingerprint cache under VLLM_CACHE_ROOT. * vllm_omni/edge/auto_profile.py + deploy_profile="auto": edge/<model>.yaml -> edge/hardware/<class>.yaml -> derived overrides -> cached calibration, materialized as a temp deploy YAML (platform overlays are superseded). * benchmarks/tts/hw_emulation.py: emulated classes (ATEN_CPU_CAPABILITY, taskset, profile overrides) driving stream_latency_bench with auto profiles. * CPU-only tests for probe snapshots, derivation rules, calibration and auto. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Hardware-class overlays for deploy_profile="auto"
When a deploy profile is auto, vLLM-Omni probes the host
(vllm_omni/edge/hardware_probe.py), maps it to one of the classes below, and
materializes a deploy YAML as: edge/<model_type>.yaml (or the model default)
→ overlay edge/hardware/<class>.yaml (this directory; same stages: /
connectors: shape as a deploy file, merged per stage id) → derived overrides
from vllm_omni/edge/adapt.py (dtype, KV budgets, thread binding, eager
switches, adaptive chunking, timeouts) → cached calibration
(vllm_omni/edge/calibrate.py) if one exists for this hardware fingerprint.
Class files are optional; add one to pin values that the derivation should not
touch for that class. Emulate a class on any host with
VLLM_OMNI_HW_PROFILE=<json> (see benchmarks/tts/hw_emulation.py).
| class | selected when |
|---|---|
jetson |
CUDA on unified memory (Tegra release file present) |
cuda_discrete |
CUDA device with its own memory |
arm64_cpu |
aarch64, no accelerator |
x86_cpu |
x86_64, no accelerator |
npu_phone |
accelerator: npu:<vendor> (only via profile override; vLLM-Omni does not run on the NPU itself) |