[Model] Prime DLO cache retention with a startup warmup
Models whose request geometry opts out of the generic dummy run (H3: t2va needs an explicit aspect ratio and a bounded duration) can declare a dlo_dummy_run_recipe. Under distributed layerwise offload the engine's startup warmup runs it, so the learned cache-retention budget is primed before real traffic instead of on the first request. H3 declares a maximum-duration t2va generation: duration is the only workload variable at the fixed 768 short edge, so the learned peak dominates every production request and no real request re-learns. Validated on L20X: all cold-start releases happen during the startup warmup, zero during requests; first request 15.91 s vs 16.7 s unwarmed, steady 14.9 s; startup gains one max-length generation (~10-20 s). Non-DLO deployments keep the opted-out generic warmup unchanged. Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
c9bb65902e
commit
f4394f1d3d
6 files changed
+201
-3
No files matched your search
@@ -67,6 +67,16 @@ omni = Omni(
|
||||
| `--host-weight-runtime-root PATH` | Writable node-local HWR store shared by workers in one storage domain; required for `preferred` and `required` | unset |
|
||||
| `--dlo-host-registration-limit-gib N` | Optional per-worker ceiling for registering an HWR mmap; zero adds no ceiling | `0` |
|
||||
|
||||
## Startup warmup
|
||||
|
||||
Under DLO, models that need model-specific sampling arguments for a warmup
|
||||
generation (currently MiniMax-H3: `t2va`, an explicit aspect ratio) run one
|
||||
at engine startup. It runs at the maximum output duration so the component
|
||||
allocator-cache retention policy learns a footprint that covers every
|
||||
production request: the cold-start flushes happen during startup and no real
|
||||
request pays them. The tradeoff is one maximum-length generation added to
|
||||
boot time; on deployments without DLO the behavior is unchanged.
|
||||
|
||||
## Host-weight loading
|
||||
|
||||
The diffusion loader chooses host storage before DLO is enabled. It first
|
||||
|
||||
Reference in new issue
Block a user