[Model] Prime DLO cache retention with a startup warmup
Models whose request geometry opts out of the generic dummy run (H3: t2va needs an explicit aspect ratio and a bounded duration) can declare a dlo_dummy_run_recipe. Under distributed layerwise offload the engine's startup warmup runs it, so the learned cache-retention budget is primed before real traffic instead of on the first request. H3 declares a maximum-duration t2va generation: duration is the only workload variable at the fixed 768 short edge, so the learned peak dominates every production request and no real request re-learns. Validated on L20X: all cold-start releases happen during the startup warmup, zero during requests; first request 15.91 s vs 16.7 s unwarmed, steady 14.9 s; startup gains one max-length generation (~10-20 s). Non-DLO deployments keep the opted-out generic warmup unchanged. Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
c9bb65902e
commit
f4394f1d3d
6 files changed
+201
-3
No files matched your search
@@ -184,7 +184,10 @@ the weight-reuse payload plus the transient footprint whatever the
|
||||
resolution. It starts at zero: a cold start pays one release per observation
|
||||
step while the peak climbs, after which boundaries retain steadily, and a
|
||||
workload change whose cached footprint grows by more than the margin pays one
|
||||
further release while the new peak lands. The budget is capped at a quarter
|
||||
further release while the new peak lands. Models whose request geometry opts
|
||||
out of the generic dummy run can declare `dlo_dummy_run_recipe`; MiniMax-H3
|
||||
declares one minimal t2va generation, so under DLO the engine's startup
|
||||
warmup runs it and the cold-start flushes land before real traffic. The budget is capped at a quarter
|
||||
of device capacity so a spurious observation cannot monopolize a small
|
||||
device, and detaching the last attached stager resets the learned peak so one
|
||||
workload never permanently influences another. No user-chosen fraction, no
|
||||
|
||||
Reference in new issue
Block a user