58 lines
2.7 KiB
Markdown
58 lines
2.7 KiB
Markdown
# Host weight dependency memory diagnostic
|
|
|
|
`safetensors_retention.py` isolates repeated CPU `get_tensor()` calls from
|
|
access to one cached tensor view. It imports neither vLLM nor HWR and uses a
|
|
synthetic float32 payload controlled by `--tensor-elements` (default: 64).
|
|
Tensor size can affect retention: a negative result for 16 elements does not
|
|
rule out growth for 64 elements. The selected size is recorded in the JSON
|
|
arguments and used for both file creation and the correctness check.
|
|
HWR itself calls `get_tensor()` when
|
|
acquiring a lease, then exposes cached views; this probe does not measure HWR
|
|
requests or repeated lease acquisition.
|
|
|
|
Run one mode per fresh process in the same Python environment and CPU affinity:
|
|
|
|
```bash
|
|
git rev-parse HEAD > revision.txt
|
|
git status --short > working-tree.txt
|
|
git diff > working-tree.patch
|
|
# Select an allowed CPU from your process affinity; 0 is only an example.
|
|
CUDA_VISIBLE_DEVICES='' taskset -c 0 timeout 120s python \
|
|
benchmarks/host_weight_runtime/safetensors_retention.py \
|
|
--mode get_tensor --iterations 10000 --sample-every 5000 > feasibility.json
|
|
```
|
|
|
|
For a size comparison, run 16 and 64 elements twice each in fresh processes,
|
|
with cached reuse as a control at both sizes:
|
|
|
|
```bash
|
|
for repetition in 1 2; do
|
|
for elements in 16 64; do
|
|
for mode in get_tensor reuse; do
|
|
CUDA_VISIBLE_DEVICES='' taskset -c 0 timeout 120s python \
|
|
benchmarks/host_weight_runtime/safetensors_retention.py \
|
|
--mode "$mode" --tensor-elements "$elements" \
|
|
> "$mode-$elements-$repetition.json"
|
|
done
|
|
done
|
|
done
|
|
```
|
|
|
|
The diagnostic sets Torch to one CPU thread, warms up before the baseline,
|
|
and reports preparation separately from timed loop work. Checkpoints run GC
|
|
and report Linux private/anonymous memory, RSS, and open descriptors. It checks
|
|
that transient tensor objects and the final cached view are released. Linux
|
|
`/proc/self/smaps_rollup` must be readable; invalid arguments and missing proc
|
|
support fail explicitly. Keep raw JSON and the repository snapshot with results.
|
|
|
|
Compare the final loop sample with the warmed baseline, then inspect the closed
|
|
sample. Private-memory growth after Python tensor objects disappear is evidence
|
|
of retention in the dependency/allocator path, not proof of a leak's root cause.
|
|
RSS can include file-backed pages; it is not a private-memory metric. Samples
|
|
also include small diagnostic bookkeeping allocations. Do not assert a portable
|
|
memory threshold or turn this synthetic loop into a per-request service estimate.
|
|
|
|
Do not drop shared caches, trim allocators, or add serving-time GC to make the
|
|
numbers look smaller. A dependency replacement or version change requires its
|
|
own reproducible comparison and correctness/lifecycle validation.
|