[Model] Prime DLO cache retention with a startup warmup
Models whose request geometry opts out of the generic dummy run (H3: t2va needs an explicit aspect ratio and a bounded duration) can declare a dlo_dummy_run_recipe. Under distributed layerwise offload the engine's startup warmup runs it, so the learned cache-retention budget is primed before real traffic instead of on the first request. H3 declares a maximum-duration t2va generation: duration is the only workload variable at the fixed 768 short edge, so the learned peak dominates every production request and no real request re-learns. Validated on L20X: all cold-start releases happen during the startup warmup, zero during requests; first request 15.91 s vs 16.7 s unwarmed, steady 14.9 s; startup gains one max-length generation (~10-20 s). Non-DLO deployments keep the opted-out generic warmup unchanged. Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
c9bb65902e
commit
f4394f1d3d
6 files changed
+201
-3
No files matched your search
@@ -704,6 +704,20 @@ class MiniMaxH3Pipeline(
|
||||
"post_decode",
|
||||
]
|
||||
dummy_run_num_frames: ClassVar[int] = 0
|
||||
# Startup warmup used only when distributed layerwise offload is enabled:
|
||||
# the generic dummy run cannot satisfy t2va's required sampling arguments
|
||||
# (explicit aspect ratio, bounded duration). Duration is the only workload
|
||||
# variable at the fixed 768 short edge, so warming at the maximum makes the
|
||||
# learned cache-retention peak dominate every production request and moves
|
||||
# all cold-start flushes ahead of real traffic, at the cost of one
|
||||
# max-length generation in boot time.
|
||||
dlo_dummy_run_recipe: ClassVar[Mapping[str, Any] | None] = {
|
||||
"task": "t2va",
|
||||
"duration": MINIMAX_H3_MAX_OUTPUT_SECONDS,
|
||||
"aspect_ratio": "16:9",
|
||||
"num_inference_steps": 2,
|
||||
"guidance_scale": 1.0,
|
||||
}
|
||||
# Only distilled releases pin a schedule, so the default keeps the legacy
|
||||
# uniform path available to partially constructed pipelines.
|
||||
_base_schedule_by_partition: ClassVar[Mapping[str, DMD2SigmaSchedule | None]] = {}
|
||||
|
||||
Reference in new issue
Block a user