Drop the staged-bytes seed per review direction: the budget is now
min(observed_peak * 1.25, 25% of device capacity), nothing else. A cold
start pays one release per observation step while the peak climbs (two on
the validated MiniMax workload), then boundaries retain steadily; the 5%
free floor and capacity cap are unchanged. The staged-byte ledger remains
only to detect full detachment, which now resets the observed peak so one
workload cannot permanently influence another. Trade-off accepted per
review: lower reserved memory and a simpler policy over a flush-free
first request.
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
The staged-bytes seed left the transient half of the working set to a
guessed multiplier, and that half scales with the running workload rather
than the model topology. Track the largest cached-but-unallocated value
observed at any component boundary instead: the budget is the larger of
the seed and the observed peak times a 1.25 margin, still capped at a
quarter of device capacity. A workload that outgrows the seed pays one
release while the observation lands, then retains steadily. No warmup
flag is needed and no user-chosen fraction exists.
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Address review on #6629:
- Cap the derived retention budget at a quarter of device capacity so a
large staged total cannot monopolize a small device; the multiplier
itself stays pending an instrumented target run.
- Log every release decision (reason, cached bytes, effective budget and
cap, staged total, free/total memory) and every staging OOM retry with
its staged byte total, so a single target run validates the headroom
multiplier directly.
- Make stager detach symmetric: set_cache_retention now shrinks the
previous cache's budget before growing the new one, so cache -> None ->
same cache counts the staged bytes exactly once (regression test added).
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Replace BoundedAllocatorCache's device-capacity fraction bound (25% of
total memory) with a budget derived from the model topology: each attached
PinnedModuleStager reports its unique staged storage bytes once, and the
retention ceiling is that total times a fixed 2.0 headroom multiplier
covering allocator fragmentation and transient storages freed at component
boundaries. The 5% physically-free floor is unchanged.
The budget therefore scales with the staged components instead of device
capacity, so no user-chosen fraction is needed and behavior is identical
across differently sized devices with the same model. Re-attaching the
same cache does not double count; a second stager grows the budget by its
own bytes.
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
hsliuustc0106