[Model] Derive MiniMax-H3 DLO component cache budget from staged bytes
Replace BoundedAllocatorCache's device-capacity fraction bound (25% of total memory) with a budget derived from the model topology: each attached PinnedModuleStager reports its unique staged storage bytes once, and the retention ceiling is that total times a fixed 2.0 headroom multiplier covering allocator fragmentation and transient storages freed at component boundaries. The 5% physically-free floor is unchanged. The budget therefore scales with the staged components instead of device capacity, so no user-chosen fraction is needed and behavior is identical across differently sized devices with the same model. Re-attaching the same cache does not double count; a second stager grows the budget by its own bytes. Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
780e8d40aa
commit
ba5943dddb
3 files changed
+72
-12
No files matched your search
@@ -177,8 +177,12 @@ component can be reused by the next. The DiT prefetch path and its two shared
|
||||
device buffers are unchanged.
|
||||
|
||||
After a component is offloaded, cached-but-unallocated memory is retained only
|
||||
while it is at most 25% of device capacity and at least 5% of device capacity
|
||||
remains physically free. Crossing either bound releases the allocator cache.
|
||||
while it stays within a budget derived from the model topology — the staged
|
||||
component byte total reported by the attached `PinnedModuleStager` instances
|
||||
times a fixed headroom multiplier — and at least 5% of device capacity
|
||||
remains physically free. Crossing either bound releases the allocator cache,
|
||||
so the budget scales with the staged components rather than with device
|
||||
capacity and needs no user-chosen fraction.
|
||||
Missing allocator telemetry also releases it conservatively. Component or
|
||||
staging failure forces a release, and an out-of-memory allocation gets one
|
||||
retry after release. This is a component-local policy rather than a global
|
||||
|
||||
Reference in new issue
Block a user