From 8f93a74951f64290e0010caf8454da00848204a5 Mon Sep 17 00:00:00 2001 From: Hongsheng Liu Date: Mon, 17 Aug 2026 15:03:03 +0800 Subject: [PATCH] [Docs] Remove stale DLO validation results Signed-off-by: Hongsheng Liu --- .../distributed_layerwise_offload.md | 196 ++---------------- .../distributed_layerwise_offload.md | 13 +- 2 files changed, 22 insertions(+), 187 deletions(-) diff --git a/docs/design/feature/offloader/distributed_layerwise_offload.md b/docs/design/feature/offloader/distributed_layerwise_offload.md index 290c021b0..573dfbcfc 100644 --- a/docs/design/feature/offloader/distributed_layerwise_offload.md +++ b/docs/design/feature/offloader/distributed_layerwise_offload.md @@ -56,9 +56,9 @@ ordinary-loader compatibility path: DLO can consume the resulting TP-local tensors. Direct checkpoint mmap still does not cover TP, but the Phase B cache can share each final TP coordinate among equivalent processes. -The compatibility matrix below describes the current implementation. The -unit-level guards are covered, but not every parallelism combination has a -full model-and-hardware end-to-end test. +The compatibility matrix below defines the current implementation boundary. +Hardware and performance qualification is tracked separately from this +long-lived design contract. ## Architecture @@ -446,8 +446,8 @@ This mode means: |---|---|---| | **DP** | Supported primary path. DLO shards host weights across the DP group and can run DP multi-concurrency. | Supported rank-local path. Compatible TP1 replicas can share checkpoint pages; the optional runtime cache shares equivalent final layouts while excluding DP rank from its identity. CUDA workers may register the complete final mapping for direct H2D. | | **SP** | Supported in the implementation. With DP=1, DLO uses the SP group for host-weight sharding; SP still shards sequence/activation work. | SP remains active without a DLO weight collective. Runtime-cache v1 excludes SP rank and includes conservative SP implementation/world-size guards. Registered transfer does not remove ordinary SP activation/attention collectives. | -| **TP > 1** | Outside the Phase A direct-mmap scope. The loader preserves TP-local layouts and DLO may apply DP/SP host sharding to ordinary runtime tensors. | The ordinary TP-aware loader produces rank-local tensors. Runtime-cache v1 creates one entry per TP coordinate, shareable by equivalent DP/SP processes. Registering those final mappings removes staging-driven TP skew but does not remove ordinary TP collectives. | -| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted by configuration. HSDP owns parameter sharding and its own gathers; DLO only stages rank-local parameters. End-to-end coverage is limited. | +| **TP > 1** | Outside the Phase A direct-mmap scope. The loader preserves TP-local layouts and DLO may apply DP/SP host sharding to ordinary runtime tensors. | The ordinary TP-aware loader produces rank-local tensors. Runtime-cache v1 creates one entry per TP coordinate, shareable by equivalent DP/SP processes. Registering those final mappings avoids the recurrent staging copy but does not remove ordinary TP collectives. | +| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted only with ordinary-loader rank-local tensors. HSDP owns parameter sharding and its own gathers; runtime-cache v1 rejects HSDP/DTensor layouts. | ### Combined dimensions @@ -487,183 +487,19 @@ configurations, HSDP/DTensor, expert parallelism, CFG parallelism, PP, non-contiguous tensors, and tied/shared storage. -## Validation coverage +## Validation status -Current source-level validation includes: - -- HSDP + DLO + AllGather rejection; -- HSDP + DLO without AllGather acceptance at configuration level; -- loader preflight fallback for TP, HSDP, online quantization, unknown custom - loaders, missing keys, and shape/dtype mismatches; -- ordinary-loader fallback for per-tensor online FP8 linears followed by DLO - sharding of finalized weights and scales; -- exact loader-to-backend plan transfer and ordinary-loader fallback; -- rank-local mmap source retention, bounded two-slot staging, and adapter - transforms without parameter-side flags; -- runtime-cache content identity, sharded atomic publication, concurrent - writer election, corruption rebuild, lock timeout, and fail-closed layout - rejection; -- loader ordering through final post-load mutation and DLO remapping without - rerunning post-load hooks; -- all-or-nothing page-aligned registration, budget rejection, partial-failure - rollback, direct-copy staging bypass, and explicit shutdown cleanup; -- resident-layer requests requiring no-AllGather; -- DP request-wave validation for denoising-step compatibility; -- sharding, double-buffer, AllGather-size, and heterogeneous-block regression - tests. - -### B300 parallel-topology smoke matrix - -A four-GPU B300 smoke test covered MiniMax-H3 FL2VA with the same prompt, seed, -CUDNN attention backend, 256x256 output, two denoising steps, and -`dlo_resident_layers=0`. The TP2 rows used DiT DP2xTP2 with the text encoder and -VAEs at TP1. They validate the ordinary-loader fallback only, not direct mmap -or shared-mmap host-memory savings. - -| Configuration | Result | Warm E2E | Peak device memory | Host PSS | -|---|---:|---:|---:|---:| -| DP4xTP1 AllGather | Passed, 4 concurrent requests | 2.87 s / 4 requests | 13.84 GiB | 211.99 GiB | -| DP4xTP1 no-AllGather | Passed, 1 request | 15.02 s | 13.23 GiB | 187.77 GiB | -| DP2xTP2 AllGather | Passed, 2 concurrent requests | 4.16 s / 2 requests | 12.50 GiB | 211.97 GiB | -| DP2xTP2 no-AllGather | Passed, 1 request | 3.51 s | 11.88 GiB | 314.01 GiB | - -Within each topology, the AllGather and no-AllGather video and audio outputs -were byte-identical. All four runs completed without an `ERROR` or traceback -and released their device allocations. For DP4xTP1, no-AllGather direct mmap -reduced total PSS by 24.22 GiB (11.4%) and `Private_Dirty` from 211.33 to -125.32 GiB (40.7%) relative to AllGather. For DP2xTP2, preflight selected the -ordinary loader as designed; no-AllGather PSS was 314.01 GiB, about 48% above -AllGather, because DP replicas did not share checkpoint-backed runtime -weights. This is a functional and memory smoke test, not a production-quality -performance or output-quality benchmark. - -### Host-memory measurement - -A two-worker MiniMax-H3 FL2VA measurement on one L20X node compared the -ordinary-loader fallback with direct mmap. Both runs used -DP=2, TP=1, no DLO AllGather, BF16 weights, two denoising steps, and a -256x256 four-second request. The ordinary-loader workers were sampled after -initialization. The mmap workers were sampled after one completed request, so -the checkpoint working set had been faulted into the page cache; this is the -more conservative point for mmap. - -The values below come from `/proc//smaps_rollup` and include the whole -worker, not only the DiT. The stable rank-to-rank difference comes from other -pipeline components, so each worker should be compared with the same worker in -the other storage mode. - -| Worker | Ordinary RSS | mmap RSS | Ordinary PSS | mmap PSS | PSS reduction | -|---|---:|---:|---:|---:|---:| -| DP worker 0 | 168.27 GiB | 132.76 GiB | 167.84 GiB | 101.43 GiB | 66.40 GiB | -| DP worker 1 | 116.19 GiB | 79.97 GiB | 115.73 GiB | 48.64 GiB | 67.09 GiB | -| **Two-worker total** | — | — | **283.56 GiB** | **150.08 GiB** | **133.48 GiB (47.1%)** | - -The direct-mmap workers each reported 62.45 GiB `Shared_Clean` but only -31.20 GiB `Pss_File`, which is the proportional charge expected when the same -resident checkpoint pages are mapped by two workers. `Private_Dirty` also fell -from 167.53 to 70.24 GiB for worker 0 and from 115.40 to 17.44 GiB for worker -1, a reduction of about 97–98 GiB per worker. RSS understates this benefit -because it counts a shared physical page in every process that maps it; summed -PSS is the appropriate node-memory comparison. - -### L20X runtime-cache smoke matrix - -Phase B was also exercised on four L20X GPUs in one NVLink domain with -MiniMax-H3 FL2VA, CUDNN attention, eager execution, 256x256 four-second -outputs, two denoising steps, `dlo_resident_layers=0`, text-encoder TP1, and -VAE TP1. This is a functional, memory, and profiler smoke test rather than a -production-quality performance benchmark. Host values below are node-wide -maxima during request waves from descendant-process `smaps_rollup`; they do -not represent the private startup peak. HBM is the engine-reported request -peak because sampled `nvidia-smi` data can miss short-lived allocations. - -One internal DP2xTP2 engine compared the three host-storage/transfer choices. -Each measured wave contained two requests. NVLink TX is the physical per-link -counter delta summed across the four GPUs; RX had the same value and is not -added again here. - -| Mode | Two-request wave | Throughput | Host PSS | `Private_Dirty` | Engine peak HBM | NVLink TX / wave | -|---|---:|---:|---:|---:|---:|---:| -| AllGather, ordinary loader | 3.41 s | 0.587 req/s | 210.70 GiB | 209.72 GiB | 12,498 MB | 73.37 GiB | -| no-AllGather, private loader weights | 6.96 s | 0.287 req/s | 312.56 GiB | 311.56 GiB | 11,882 MB | 19.78 GiB | -| no-AllGather, runtime cache (cold publish) | 18.94 s | 0.106 req/s | 178.71 GiB | 116.00 GiB | 11,882 MB | 19.78 GiB | -| no-AllGather, runtime cache (warm reuse) | 24.23 s | 0.083 req/s | 178.29 GiB | 115.58 GiB | 11,882 MB | 19.78 GiB | - -The no-AllGather cache preserved the no-AllGather NVLink traffic and HBM -footprint while reducing node PSS by about 134 GiB relative to private loader -weights. Cold and warm labels describe cache publication or reuse during -initialization; both request phases ran after initialization. A warm hit still -performed ordinary loading and full content hashing, so it was not a startup -fast path. All measured no-AllGather private/cache outputs were byte-identical -for matching seeds. - -The deployment shape targeted by the cache was tested separately as two -independent DP1xTP2 engines on the same node. Both engines mapped the same two -TP-coordinate entries, including the same file device/inode tuples; the cache -occupied 66,281,995,903 bytes. A start barrier kept all four measured requests -ahead of profiling. - -| Storage | Mean request latency, engines A / B | Combined throughput | Host PSS | `Private_Dirty` | Engine peak HBM | -|---|---:|---:|---:|---:|---:| -| Private loader weights | 3.50 / 3.60 s | 0.463 req/s | 366.20 GiB | 365.22 GiB | 11,882 MB | -| Shared runtime cache | 8.96 / 12.30 s | 0.142 req/s | 232.23 GiB | 169.52 GiB | 11,882 MB | - -The shared cache reduced request-time PSS by 133.97 GiB (36.6%) and -`Private_Dirty` by 195.70 GiB (53.6%), but reduced combined throughput by -69.3% in this host-contention smoke. Across one profiled request per engine, -aggregate GPU compute was unchanged (1.359 versus 1.357 seconds) and H2D -payload was exactly 264.24 GiB in both modes. Aggregate CPU `aten::copy_` time -rose from 3.75 to 32.49 seconds, while communication-kernel time rose from -3.56 to 5.80 seconds as TP ranks waited on skewed staging. These durations are -summed across ranks and can overlap; they identify the mechanism, not wall -time. Matching private/cache outputs were byte-identical. - -The result confirms that zero-budget runtime-cache staging is a host-capacity -option, not a latency-neutral replacement for private pinned weights. Broader -SP, model, platform, and production-quality validation remains tracked in -[issue #6231](https://github.com/vllm-project/vllm-omni/issues/6231). The -historical B300 rows above predate the runtime cache and must not be treated as -Phase B memory validation. - -### L20X registered-runtime-cache smoke - -The read-only registered path was then tested with the same MiniMax-H3 -workload. These are functional and mechanism-validation measurements, not a -production benchmark. The TP1 pair used one request per wave. Both modes moved -exactly 121.86 GiB H2D in the profiled request and produced byte-identical video -and audio for both measured seeds. - -| TP1 no-AllGather mode | Mean wave | Throughput | Request-time host PSS | `Private_Dirty` | Engine peak HBM | CPU `aten::copy_` | GPU compute | -|---|---:|---:|---:|---:|---:|---:|---:| -| Checkpoint mmap + bounded staging | 33.87 s | 0.0295 req/s | 133.65 GiB | 71.16 GiB | 13,226 MB | 21.91 s | 419.28 ms | -| Final runtime cache + registered direct H2D | 4.88 s | 0.2055 req/s | 129.41 GiB | 66.93 GiB | 13,226 MB | 1.64 s | 419.05 ms | - -Registration covered 61.73 GiB in 13 page-aligned ranges. Mean wave latency -fell by 85.6% and throughput increased 6.94x. GPU compute and H2D payload were -unchanged, identifying recurrent host transform/packing as the removed work. -The direct path issued 2,425 H2D events versus 1,945 for packed staging, but -that launch increase was much smaller than the eliminated CPU cost. - -Two independent TP2 engines also registered the same two TP-coordinate cache -entries across four workers. Compared with the zero-budget cache, mean wave -latency improved from 9.21/12.55 seconds to 3.80/3.86 seconds for engines A/B; -the private-loader reference was 3.61/3.72 seconds. Registered request-time -host PSS was 224.26 GiB versus 232.23 GiB staged and 366.20 GiB private. -Aggregate CPU `aten::copy_` time was 3.83 seconds registered, 32.49 seconds -staged, and 3.75 seconds private; aggregate GPU compute and the 264.24 GiB H2D -payload were effectively unchanged. Ordinary TP communication remained, but -the extra wait caused by staging-driven rank skew largely disappeared. - -All workers explicitly unregistered before releasing their mappings, the -system's acquired-minus-released pin counter returned to its pre-run value, -and every GPU allocation was released. Cold runtime-cache publication remained -expensive because it still performs ordinary loading and multiple full-content -passes; that startup problem is outside the registered transfer scope. +Historical test counts and hardware measurements are intentionally not kept +in this design document because they can drift as the implementation changes. +Revalidation must run against the final commit and publish commit-pinned +artifacts before making hardware or performance claims. Track that work in +[issue #6231](https://github.com/vllm-project/vllm-omni/issues/6231). ## Recommendations -- Use **DP + DLO AllGather** for the supported throughput and host-memory - scaling path. +- Use **DP + DLO AllGather** when ranks already execute the same block path in + lockstep and belong to the same fast P2P domain. It shards persistent host + weights across that group. - Use **SP + DLO AllGather** for long-sequence workloads when DP concurrency is not the goal. - Use **no-AllGather** when independent replica execution is required. TP1 @@ -674,5 +510,5 @@ passes; that startup problem is outside the registered transfer scope. registration budget large enough for the complete final layout when recurrent staging cost is unacceptable; leave it at zero when page-locking that layout is not operationally acceptable. -- Prefer **HSDP alone** for production HSDP deployments until the combined - HSDP + DLO no-AllGather path has broader end-to-end coverage. +- Do not combine HSDP with DLO AllGather or runtime-cache v1. HSDP with + no-AllGather remains an ordinary-loader rank-local path. diff --git a/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md b/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md index 60934e228..34829c8fc 100644 --- a/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md +++ b/docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md @@ -7,7 +7,7 @@ no-AllGather path can optionally share node-local runtime weights through mmap. See the [DLO component architecture](../../../design/feature/offloader/distributed_layerwise_offload.md#architecture) for ownership and lifecycle coordination, and the remaining feature design for -loader contracts, compatibility details, failure handling, and benchmarks. +loader contracts, compatibility details, and failure handling. ## Choose a mode @@ -92,12 +92,11 @@ SP model collectives still synchronize ranks within each engine. until their runtime layouts are validated. - Runtime-cache v1 rejects quantized, non-contiguous, aliased/tied, device-only, HSDP, expert-parallel, CFG-parallel, and PP layouts. -- HSDP plus DLO AllGather is unsupported. HSDP without AllGather has limited - end-to-end validation. -- The runtime cache lowers steady-state host PSS, not startup peak: each worker - still performs ordinary loading and full content validation. -- Skip-load-on-hit, cache eviction, cross-node sharing, and broader SP hardware - validation remain follow-up work in +- HSDP plus DLO AllGather is unsupported, and runtime-cache v1 rejects HSDP. +- The runtime cache changes steady-state host backing, not startup loading: + each worker still performs ordinary loading and full content validation. +- Skip-load-on-hit, cache eviction, and cross-node sharing remain follow-up + work in [RFC #6195](https://github.com/vllm-project/vllm-omni/issues/6195). See the [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)