diff --git a/docs/design/feature/offloader/distributed_layerwise_offload.md b/docs/design/feature/offloader/distributed_layerwise_offload.md index c3d85f9ea..3d3f71dbf 100644 --- a/docs/design/feature/offloader/distributed_layerwise_offload.md +++ b/docs/design/feature/offloader/distributed_layerwise_offload.md @@ -392,6 +392,14 @@ one and streams complete blocks using H2D copies only. The host backing may be a loader-approved checkpoint mapping, a normalized runtime-cache mapping, or ordinary runtime tensors. +![No-AllGather DLO shared host weights and independent runtime orchestration](../../figures/dlo/dlo_no_allgather_shared_host.svg) + +The figure uses TP1 to make physical sharing explicit. With TP greater than +one, the cache keeps one immutable entry per TP coordinate and equivalent +workers in independent engines map the corresponding entry. An external +router or orchestrator chooses each engine's requests; DLO coordinates only +the worker-local mapping, hooks, transfer streams, events, and device slots. + For direct mmap, each process retains immutable safetensors views and uses two bounded pinned host staging slots. Processes on the same node that map the same files share physical checkpoint pages through the OS page cache. This removes diff --git a/docs/design/figures/dlo/dlo_no_allgather_shared_host.svg b/docs/design/figures/dlo/dlo_no_allgather_shared_host.svg new file mode 100644 index 000000000..2217106db --- /dev/null +++ b/docs/design/figures/dlo/dlo_no_allgather_shared_host.svg @@ -0,0 +1,174 @@ + + No-AllGather DLO shared host weights and independent runtime orchestration + Two independently scheduled diffusion engines on one node map the same immutable runtime-cache weight pages. Each worker owns its mmap view, optional host registration, DLO hooks, streams, events, and two GPU device slots. Complete blocks move independently from the shared host pages to each GPU without a cross-engine DLO AllGather. TP1 is shown; tensor parallel deployments repeat one shared cache entry per TP coordinate. + + + + + + + + + + + + + + + + No-AllGather DLO: one shared host payload, independent runtimes + TP1 shown; TP>1 repeats this mapping once per TP coordinate, shared by equivalent workers across engines + + + + External router / orchestrator + independent request placement · outside vLLM-Omni DLO + + + + ONE NODE · SHARED PHYSICAL HOST MEMORY, PROCESS-LOCAL RUNTIMES + + + SHARING SCOPE + TP1: one full cache entry + TP>1: one entry per TP coordinate + Equivalent engines reuse that entry; + different TP ranks never alias layouts. + + + NODE-LOCAL IMMUTABLE RUNTIME CACHE + + 1× physical runtime-weight payload + file-backed tensor pages shared by the OS page cache + Identity includes final layout + TP coordinate; engine and DP replica are excluded + + + WHAT STAYS PRIVATE + Per worker / process + mmap object and validation state + optional host registration + hooks, streams, events, GPU slots + + + + Independent engine A + own request schedule · own DLO backend state + + + STARTUP + ordinary loader + final transforms → publish / join + + + HOST VIEW + mmap of shared pages + optional process-local registration + + + RUNTIME CONTROL + DLO backend + block hook + select · prefetch · repoint · rotate + + + GPU A · TWO DEVICE SLOTS + + SLOT 0 + current + + SLOT 1 + prefetch + + + + Independent engine B + own request schedule · own DLO backend state + + + STARTUP + ordinary loader + final transforms → publish / join + + + HOST VIEW + mmap of shared pages + optional process-local registration + + + RUNTIME CONTROL + DLO backend + block hook + select · prefetch · repoint · rotate + + + GPU B · TWO DEVICE SLOTS + + SLOT 0 + current + + SLOT 1 + prefetch + + + + request A + + request B + + + + + same pages + + same pages + + + + tensor views + + full-block H2D + + + tensor views + + full-block H2D + + Ordinary TP / SP model collectives, when configured, remain inside each engine + + + + No cross-engine DLO AllGather + shared host pages do not couple request order, progress, or failure domains +