From b732ad2ceaef07ed4079d307c5e540a49e7fdcad Mon Sep 17 00:00:00 2001 From: Hongsheng Liu Date: Mon, 17 Aug 2026 00:43:06 +0800 Subject: [PATCH] [Diffusion] Document runtime cache hardware tradeoffs Signed-off-by: Hongsheng Liu --- .../distributed_layerwise_offload.md | 76 ++++++++++++++++--- .../distributed_layerwise_offload.md | 16 ++++ 2 files changed, 83 insertions(+), 9 deletions(-) diff --git a/docs/design/feature/offloader/distributed_layerwise_offload.md b/docs/design/feature/offloader/distributed_layerwise_offload.md index 3fb494478..0746eafc1 100644 --- a/docs/design/feature/offloader/distributed_layerwise_offload.md +++ b/docs/design/feature/offloader/distributed_layerwise_offload.md @@ -115,7 +115,11 @@ DLO maps this plan without moving the DiT to `meta` and without rerunning any post-load hook. It changes tensor storage while preserving `Parameter` objects, then runs the existing `gc.collect()`/`malloc_trim()` cleanup before installing streaming hooks. The source mappings remain open through DLO disable and feed -the existing bounded two-slot host staging path. +the existing bounded two-slot host staging path. Because file-backed pages are +not pinned, every streamed block is first packed from mmap storage into one of +those pinned slots and is then copied to the device. This removes model-sized +private pinned storage, but adds a recurrent host-to-host copy and can amplify +TP synchronization waits when several ranks compete for host bandwidth. This version is deliberately opt-in and steady-state-oriented. It requires roughly one local-disk model copy per runtime identity, has no eviction, and @@ -310,12 +314,64 @@ from 167.53 to 70.24 GiB for worker 0 and from 115.40 to 17.44 GiB for worker because it counts a shared physical page in every process that maps it; summed PSS is the appropriate node-memory comparison. -The highest-value missing coverage is the Phase B hardware matrix from -[issue #6231](https://github.com/vllm-project/vllm-omni/issues/6231): cold and -warm publication, inode sharing, PSS/`Private_Dirty`, output equivalence, and -independently scheduled `TP=2` engines on target hardware. The historical -B300 rows above predate the runtime cache and must not be treated as Phase B -memory validation. +### L20X runtime-cache smoke matrix + +Phase B was also exercised on four L20X GPUs in one NVLink domain with +MiniMax-H3 FL2VA, CUDNN attention, eager execution, 256x256 four-second +outputs, two denoising steps, `dlo_resident_layers=0`, text-encoder TP1, and +VAE TP1. This is a functional, memory, and profiler smoke test rather than a +production-quality performance benchmark. Host values below are node-wide +maxima during request waves from descendant-process `smaps_rollup`; they do +not represent the private startup peak. HBM is the engine-reported request +peak because sampled `nvidia-smi` data can miss short-lived allocations. + +One internal DP2xTP2 engine compared the three host-storage/transfer choices. +Each measured wave contained two requests. NVLink TX is the physical per-link +counter delta summed across the four GPUs; RX had the same value and is not +added again here. + +| Mode | Two-request wave | Throughput | Host PSS | `Private_Dirty` | Engine peak HBM | NVLink TX / wave | +|---|---:|---:|---:|---:|---:|---:| +| AllGather, ordinary loader | 3.41 s | 0.587 req/s | 210.70 GiB | 209.72 GiB | 12,498 MB | 73.37 GiB | +| no-AllGather, private loader weights | 6.96 s | 0.287 req/s | 312.56 GiB | 311.56 GiB | 11,882 MB | 19.78 GiB | +| no-AllGather, runtime cache (cold publish) | 18.94 s | 0.106 req/s | 178.71 GiB | 116.00 GiB | 11,882 MB | 19.78 GiB | +| no-AllGather, runtime cache (warm reuse) | 24.23 s | 0.083 req/s | 178.29 GiB | 115.58 GiB | 11,882 MB | 19.78 GiB | + +The no-AllGather cache preserved the no-AllGather NVLink traffic and HBM +footprint while reducing node PSS by about 134 GiB relative to private loader +weights. Cold and warm labels describe cache publication or reuse during +initialization; both request phases ran after initialization. A warm hit still +performed ordinary loading and full content hashing, so it was not a startup +fast path. All measured no-AllGather private/cache outputs were byte-identical +for matching seeds. + +The deployment shape targeted by the cache was tested separately as two +independent DP1xTP2 engines on the same node. Both engines mapped the same two +TP-coordinate entries, including the same file device/inode tuples; the cache +occupied 66,281,995,903 bytes. A start barrier kept all four measured requests +ahead of profiling. + +| Storage | Mean request latency, engines A / B | Combined throughput | Host PSS | `Private_Dirty` | Engine peak HBM | +|---|---:|---:|---:|---:|---:| +| Private loader weights | 3.50 / 3.60 s | 0.463 req/s | 366.20 GiB | 365.22 GiB | 11,882 MB | +| Shared runtime cache | 8.96 / 12.30 s | 0.142 req/s | 232.23 GiB | 169.52 GiB | 11,882 MB | + +The shared cache reduced request-time PSS by 133.97 GiB (36.6%) and +`Private_Dirty` by 195.70 GiB (53.6%), but reduced combined throughput by +69.3% in this host-contention smoke. Across one profiled request per engine, +aggregate GPU compute was unchanged (1.359 versus 1.357 seconds) and H2D +payload was exactly 264.24 GiB in both modes. Aggregate CPU `aten::copy_` time +rose from 3.75 to 32.49 seconds, while communication-kernel time rose from +3.56 to 5.80 seconds as TP ranks waited on skewed staging. These durations are +summed across ranks and can overlap; they identify the mechanism, not wall +time. Matching private/cache outputs were byte-identical. + +The result confirms that runtime-cache v1 is a host-capacity option, not a +latency-neutral replacement for private pinned weights. Broader SP, model, +platform, and production-quality validation remains tracked in +[issue #6231](https://github.com/vllm-project/vllm-omni/issues/6231). The +historical B300 rows above predate the runtime cache and must not be treated as +Phase B memory validation. ## Recommendations @@ -325,7 +381,9 @@ memory validation. not the goal. - Use **no-AllGather** when independent replica execution is required. TP1 direct-mmap deployments can share checkpoint pages per node. Enable the - runtime cache when independent replicas have repeated final layouts, - especially matching coordinates across multiple TP engines. + runtime cache only when independent replicas have repeated final layouts and + host capacity is more important than the added mmap-to-pinned staging cost. + Benchmark the target CPU/memory topology, especially for matching + coordinates across multiple TP engines. - Prefer **HSDP alone** for production HSDP deployments until the combined HSDP + DLO no-AllGather path has broader end-to-end coverage. diff --git a/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md b/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md index 9e4632a6b..d9297a431 100644 --- a/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md +++ b/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md @@ -110,6 +110,14 @@ independent `TP=2` engines normally create two entries: both TP0 workers map one entry and both TP1 workers map the other. Cache sharing does not create a process group or an inference-time collective. +At inference time, cached tensors remain pageable mmap views. DLO packs each +streamed block into one of two bounded pinned host slots before the asynchronous +H2D copy. The private ordinary-loader path instead creates model-sized pinned +storage once during initialization. The cache therefore exchanges recurrent +host-to-host copies and DDR bandwidth for lower steady-state PSS; competing TP +ranks or engines can turn staging skew into collective wait time. Treat this as +a host-capacity mode and benchmark it on the target CPU/memory topology. + The mmap plan skips only dedicated DiT weight sources. Other component sources, such as a text encoder loaded through the shared diffusion loader, continue to use their ordinary component loader. A checkpoint source that mixes DiT and @@ -204,5 +212,13 @@ loads private weights before remapping, and validation adds full content-hash passes. Skip-load-on-cache-hit and cache eviction remain follow-up work in [RFC #6195](https://github.com/vllm-project/vllm-omni/issues/6195). +It is also not latency-neutral. In the four-L20X MiniMax-H3 smoke documented in +the L20X runtime-cache matrix in the +[feature design](../../../design/feature/offloader/distributed_layerwise_offload.md#l20x-runtime-cache-smoke-matrix), +two independent TP2 engines reduced request-time host PSS by 36.6% with the +cache, but recurrent mmap-to-pinned staging reduced combined throughput by +69.3%. HBM and GPU compute were unchanged. Use the cache only when that +measured memory/latency tradeoff is appropriate for the deployment. + See the [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md) for an end-to-end example.