[Model] Cap derived cache budget and account for stager detach

Address review on #6629:

- Cap the derived retention budget at a quarter of device capacity so a
  large staged total cannot monopolize a small device; the multiplier
  itself stays pending an instrumented target run.
- Log every release decision (reason, cached bytes, effective budget and
  cap, staged total, free/total memory) and every staging OOM retry with
  its staged byte total, so a single target run validates the headroom
  multiplier directly.
- Make stager detach symmetric: set_cache_retention now shrinks the
  previous cache's budget before growing the new one, so cache -> None ->
  same cache counts the staged bytes exactly once (regression test added).

Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
hsliuustc0106 committed 2026-08-26 11:12:09 +08:00
1 parent ba5943dddb
commit c5295f3ffe
3 files changed
+97 -9

No files matched your search

@@ -180,9 +180,14 @@ After a component is offloaded, cached-but-unallocated memory is retained only
while it stays within a budget derived from the model topology — the staged
component byte total reported by the attached `PinnedModuleStager` instances
times a fixed headroom multiplier — and at least 5% of device capacity
remains physically free. Crossing either bound releases the allocator cache,
so the budget scales with the staged components rather than with device
capacity and needs no user-chosen fraction.
remains physically free. The derived budget is capped at a quarter of device
capacity so a large staged total cannot monopolize a small device, and a
detached stager returns its bytes to the budget. Crossing either bound
releases the allocator cache, so the budget scales with the staged components
rather than with device capacity and needs no user-chosen fraction. Each
release decision logs the staged byte total, the effective budget and cap,
cached-but-unallocated bytes, and free memory, so a target run can validate
the headroom multiplier directly.
Missing allocator telemetry also releases it conservatively. Component or
staging failure forces a release, and an out-of-memory allocation gets one
retry after release. This is a component-local policy rather than a global