[Model] Cap derived cache budget and account for stager detach
Address review on #6629: - Cap the derived retention budget at a quarter of device capacity so a large staged total cannot monopolize a small device; the multiplier itself stays pending an instrumented target run. - Log every release decision (reason, cached bytes, effective budget and cap, staged total, free/total memory) and every staging OOM retry with its staged byte total, so a single target run validates the headroom multiplier directly. - Make stager detach symmetric: set_cache_retention now shrinks the previous cache's budget before growing the new one, so cache -> None -> same cache counts the staged bytes exactly once (regression test added). Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
ba5943dddb
commit
c5295f3ffe
3 files changed
+97
-9
No files matched your search
@@ -180,9 +180,14 @@ After a component is offloaded, cached-but-unallocated memory is retained only
|
||||
while it stays within a budget derived from the model topology — the staged
|
||||
component byte total reported by the attached `PinnedModuleStager` instances
|
||||
times a fixed headroom multiplier — and at least 5% of device capacity
|
||||
remains physically free. Crossing either bound releases the allocator cache,
|
||||
so the budget scales with the staged components rather than with device
|
||||
capacity and needs no user-chosen fraction.
|
||||
remains physically free. The derived budget is capped at a quarter of device
|
||||
capacity so a large staged total cannot monopolize a small device, and a
|
||||
detached stager returns its bytes to the budget. Crossing either bound
|
||||
releases the allocator cache, so the budget scales with the staged components
|
||||
rather than with device capacity and needs no user-chosen fraction. Each
|
||||
release decision logs the staged byte total, the effective budget and cap,
|
||||
cached-but-unallocated bytes, and free memory, so a target run can validate
|
||||
the headroom multiplier directly.
|
||||
Missing allocator telemetry also releases it conservatively. Component or
|
||||
staging failure forces a release, and an out-of-memory allocation gets one
|
||||
retry after release. This is a component-local policy rather than a global
|
||||
|
||||
Reference in new issue
Block a user