One request builder instead of two: when the generic dummy run opts out
(num_frames <= 0), _make_dummy_request itself falls back to the model's
declared dlo_dummy_run_recipe under distributed layerwise offload, and
_dummy_run keeps its original shape. The recipe request is text-only even
for image-capable models — the recipe names its own task, and t2va rejects
image conditions; caught by a boot smoke after folding (the standalone
builder had built the prompt fresh) and pinned by a regression test.
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Models whose request geometry opts out of the generic dummy run (H3: t2va
needs an explicit aspect ratio and a bounded duration) can declare a
dlo_dummy_run_recipe. Under distributed layerwise offload the engine's
startup warmup runs it, so the learned cache-retention budget is primed
before real traffic instead of on the first request.
H3 declares a maximum-duration t2va generation: duration is the only
workload variable at the fixed 768 short edge, so the learned peak
dominates every production request and no real request re-learns. Validated
on L20X: all cold-start releases happen during the startup warmup, zero
during requests; first request 15.91 s vs 16.7 s unwarmed, steady 14.9 s;
startup gains one max-length generation (~10-20 s). Non-DLO deployments
keep the opted-out generic warmup unchanged.
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Drop the staged-bytes seed per review direction: the budget is now
min(observed_peak * 1.25, 25% of device capacity), nothing else. A cold
start pays one release per observation step while the peak climbs (two on
the validated MiniMax workload), then boundaries retain steadily; the 5%
free floor and capacity cap are unchanged. The staged-byte ledger remains
only to detect full detachment, which now resets the observed peak so one
workload cannot permanently influence another. Trade-off accepted per
review: lower reserved memory and a simpler policy over a flush-free
first request.
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
The fixture reported device capacity 100, making the 25% cap 25 — below
the learned budget of 37 — so the post-adaptation checks released again
and the test failed. Raise the fixture capacity to 400 so the cap stays
above the learned budget and the test isolates peak adaptation; verified
against the real suite on the L20X validation host (15 passed).
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
The staged-bytes seed left the transient half of the working set to a
guessed multiplier, and that half scales with the running workload rather
than the model topology. Track the largest cached-but-unallocated value
observed at any component boundary instead: the budget is the larger of
the seed and the observed peak times a 1.25 margin, still capped at a
quarter of device capacity. A workload that outgrows the seed pays one
release while the observation lands, then retains steadily. No warmup
flag is needed and no user-chosen fraction exists.
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Address review on #6629:
- Cap the derived retention budget at a quarter of device capacity so a
large staged total cannot monopolize a small device; the multiplier
itself stays pending an instrumented target run.
- Log every release decision (reason, cached bytes, effective budget and
cap, staged total, free/total memory) and every staging OOM retry with
its staged byte total, so a single target run validates the headroom
multiplier directly.
- Make stager detach symmetric: set_cache_retention now shrinks the
previous cache's budget before growing the new one, so cache -> None ->
same cache counts the staged bytes exactly once (regression test added).
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
Replace BoundedAllocatorCache's device-capacity fraction bound (25% of
total memory) with a budget derived from the model topology: each attached
PinnedModuleStager reports its unique staged storage bytes once, and the
retention ceiling is that total times a fixed 2.0 headroom multiplier
covering allocator fragmentation and transient storages freed at component
boundaries. The 5% physically-free floor is unchanged.
The budget therefore scales with the staged components instead of device
capacity, so no user-chosen fraction is needed and behavior is identical
across differently sized devices with the same model. Re-attaching the
same cache does not double count; a second stager grows the budget by its
own bytes.
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
hsliuustc0106