anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Docs] Remove stale DLO validation results

Browse Source 
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu committed 2026-08-19 11:22:52 +08:00
1 parent 4acb3b1387
commit 8f93a74951
2 files changed
+22 -187
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/design/feature/offloader/distributed_layerwise_offload.md
+16 -180
View File
@@ -56,9 +56,9 @@ ordinary-loader compatibility path: DLO can consume the resulting TP-local
tensors. Direct checkpoint mmap still does not cover TP, but the Phase B cache
can share each final TP coordinate among equivalent processes.
The compatibility matrix below describes the current implementation. The
unit-level guards are covered, but not every parallelism combination has a
full model-and-hardware end-to-end test.
The compatibility matrix below defines the current implementation boundary.
Hardware and performance qualification is tracked separately from this
long-lived design contract.
## Architecture
@@ -446,8 +446,8 @@ This mode means:
|---|---|---|
| **DP** | Supported primary path. DLO shards host weights across the DP group and can run DP multi-concurrency. | Supported rank-local path. Compatible TP1 replicas can share checkpoint pages; the optional runtime cache shares equivalent final layouts while excluding DP rank from its identity. CUDA workers may register the complete final mapping for direct H2D. |
| **SP** | Supported in the implementation. With DP=1, DLO uses the SP group for host-weight sharding; SP still shards sequence/activation work. | SP remains active without a DLO weight collective. Runtime-cache v1 excludes SP rank and includes conservative SP implementation/world-size guards. Registered transfer does not remove ordinary SP activation/attention collectives. |
| **TP > 1** | Outside the Phase A direct-mmap scope. The loader preserves TP-local layouts and DLO may apply DP/SP host sharding to ordinary runtime tensors. | The ordinary TP-aware loader produces rank-local tensors. Runtime-cache v1 creates one entry per TP coordinate, shareable by equivalent DP/SP processes. Registering those final mappings removes staging-driven TP skew but does not remove ordinary TP collectives. |
| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted by configuration. HSDP owns parameter sharding and its own gathers; DLO only stages rank-local parameters. End-to-end coverage is limited. |
| **TP > 1** | Outside the Phase A direct-mmap scope. The loader preserves TP-local layouts and DLO may apply DP/SP host sharding to ordinary runtime tensors. | The ordinary TP-aware loader produces rank-local tensors. Runtime-cache v1 creates one entry per TP coordinate, shareable by equivalent DP/SP processes. Registering those final mappings avoids the recurrent staging copy but does not remove ordinary TP collectives. |
| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted only with ordinary-loader rank-local tensors. HSDP owns parameter sharding and its own gathers; runtime-cache v1 rejects HSDP/DTensor layouts. |
### Combined dimensions
@@ -487,183 +487,19 @@ configurations, HSDP/DTensor, expert parallelism, CFG parallelism, PP,
non-contiguous tensors, and
tied/shared storage.
## Validation coverage
## Validation status
Current source-level validation includes:
- HSDP + DLO + AllGather rejection;
- HSDP + DLO without AllGather acceptance at configuration level;
- loader preflight fallback for TP, HSDP, online quantization, unknown custom
loaders, missing keys, and shape/dtype mismatches;
- ordinary-loader fallback for per-tensor online FP8 linears followed by DLO
sharding of finalized weights and scales;
- exact loader-to-backend plan transfer and ordinary-loader fallback;
- rank-local mmap source retention, bounded two-slot staging, and adapter
transforms without parameter-side flags;
- runtime-cache content identity, sharded atomic publication, concurrent
writer election, corruption rebuild, lock timeout, and fail-closed layout
rejection;
- loader ordering through final post-load mutation and DLO remapping without
rerunning post-load hooks;
- all-or-nothing page-aligned registration, budget rejection, partial-failure
rollback, direct-copy staging bypass, and explicit shutdown cleanup;
- resident-layer requests requiring no-AllGather;
- DP request-wave validation for denoising-step compatibility;
- sharding, double-buffer, AllGather-size, and heterogeneous-block regression
tests.
### B300 parallel-topology smoke matrix
A four-GPU B300 smoke test covered MiniMax-H3 FL2VA with the same prompt, seed,
CUDNN attention backend, 256x256 output, two denoising steps, and
`dlo_resident_layers=0`. The TP2 rows used DiT DP2xTP2 with the text encoder and
VAEs at TP1. They validate the ordinary-loader fallback only, not direct mmap
or shared-mmap host-memory savings.
| Configuration | Result | Warm E2E | Peak device memory | Host PSS |
|---|---:|---:|---:|---:|
| DP4xTP1 AllGather | Passed, 4 concurrent requests | 2.87 s / 4 requests | 13.84 GiB | 211.99 GiB |
| DP4xTP1 no-AllGather | Passed, 1 request | 15.02 s | 13.23 GiB | 187.77 GiB |
| DP2xTP2 AllGather | Passed, 2 concurrent requests | 4.16 s / 2 requests | 12.50 GiB | 211.97 GiB |
| DP2xTP2 no-AllGather | Passed, 1 request | 3.51 s | 11.88 GiB | 314.01 GiB |
Within each topology, the AllGather and no-AllGather video and audio outputs
were byte-identical. All four runs completed without an `ERROR` or traceback
and released their device allocations. For DP4xTP1, no-AllGather direct mmap
reduced total PSS by 24.22 GiB (11.4%) and `Private_Dirty` from 211.33 to
125.32 GiB (40.7%) relative to AllGather. For DP2xTP2, preflight selected the
ordinary loader as designed; no-AllGather PSS was 314.01 GiB, about 48% above
AllGather, because DP replicas did not share checkpoint-backed runtime
weights. This is a functional and memory smoke test, not a production-quality
performance or output-quality benchmark.
### Host-memory measurement
A two-worker MiniMax-H3 FL2VA measurement on one L20X node compared the
ordinary-loader fallback with direct mmap. Both runs used
DP=2, TP=1, no DLO AllGather, BF16 weights, two denoising steps, and a
256x256 four-second request. The ordinary-loader workers were sampled after
initialization. The mmap workers were sampled after one completed request, so
the checkpoint working set had been faulted into the page cache; this is the
more conservative point for mmap.
The values below come from `/proc/<worker>/smaps_rollup` and include the whole
worker, not only the DiT. The stable rank-to-rank difference comes from other
pipeline components, so each worker should be compared with the same worker in
the other storage mode.
| Worker | Ordinary RSS | mmap RSS | Ordinary PSS | mmap PSS | PSS reduction |
|---|---:|---:|---:|---:|---:|
| DP worker 0 | 168.27 GiB | 132.76 GiB | 167.84 GiB | 101.43 GiB | 66.40 GiB |
| DP worker 1 | 116.19 GiB | 79.97 GiB | 115.73 GiB | 48.64 GiB | 67.09 GiB |
| **Two-worker total** | — | — | **283.56 GiB** | **150.08 GiB** | **133.48 GiB (47.1%)** |
The direct-mmap workers each reported 62.45 GiB `Shared_Clean` but only
31.20 GiB `Pss_File`, which is the proportional charge expected when the same
resident checkpoint pages are mapped by two workers. `Private_Dirty` also fell
from 167.53 to 70.24 GiB for worker 0 and from 115.40 to 17.44 GiB for worker
1, a reduction of about 97–98 GiB per worker. RSS understates this benefit
because it counts a shared physical page in every process that maps it; summed
PSS is the appropriate node-memory comparison.
### L20X runtime-cache smoke matrix
Phase B was also exercised on four L20X GPUs in one NVLink domain with
MiniMax-H3 FL2VA, CUDNN attention, eager execution, 256x256 four-second
outputs, two denoising steps, `dlo_resident_layers=0`, text-encoder TP1, and
VAE TP1. This is a functional, memory, and profiler smoke test rather than a
production-quality performance benchmark. Host values below are node-wide
maxima during request waves from descendant-process `smaps_rollup`; they do
not represent the private startup peak. HBM is the engine-reported request
peak because sampled `nvidia-smi` data can miss short-lived allocations.
One internal DP2xTP2 engine compared the three host-storage/transfer choices.
Each measured wave contained two requests. NVLink TX is the physical per-link
counter delta summed across the four GPUs; RX had the same value and is not
added again here.
| Mode | Two-request wave | Throughput | Host PSS | `Private_Dirty` | Engine peak HBM | NVLink TX / wave |
|---|---:|---:|---:|---:|---:|---:|
| AllGather, ordinary loader | 3.41 s | 0.587 req/s | 210.70 GiB | 209.72 GiB | 12,498 MB | 73.37 GiB |
| no-AllGather, private loader weights | 6.96 s | 0.287 req/s | 312.56 GiB | 311.56 GiB | 11,882 MB | 19.78 GiB |
| no-AllGather, runtime cache (cold publish) | 18.94 s | 0.106 req/s | 178.71 GiB | 116.00 GiB | 11,882 MB | 19.78 GiB |
| no-AllGather, runtime cache (warm reuse) | 24.23 s | 0.083 req/s | 178.29 GiB | 115.58 GiB | 11,882 MB | 19.78 GiB |
The no-AllGather cache preserved the no-AllGather NVLink traffic and HBM
footprint while reducing node PSS by about 134 GiB relative to private loader
weights. Cold and warm labels describe cache publication or reuse during
initialization; both request phases ran after initialization. A warm hit still
performed ordinary loading and full content hashing, so it was not a startup
fast path. All measured no-AllGather private/cache outputs were byte-identical
for matching seeds.
The deployment shape targeted by the cache was tested separately as two
independent DP1xTP2 engines on the same node. Both engines mapped the same two
TP-coordinate entries, including the same file device/inode tuples; the cache
occupied 66,281,995,903 bytes. A start barrier kept all four measured requests
ahead of profiling.
| Storage | Mean request latency, engines A / B | Combined throughput | Host PSS | `Private_Dirty` | Engine peak HBM |
|---|---:|---:|---:|---:|---:|
| Private loader weights | 3.50 / 3.60 s | 0.463 req/s | 366.20 GiB | 365.22 GiB | 11,882 MB |
| Shared runtime cache | 8.96 / 12.30 s | 0.142 req/s | 232.23 GiB | 169.52 GiB | 11,882 MB |
The shared cache reduced request-time PSS by 133.97 GiB (36.6%) and
`Private_Dirty` by 195.70 GiB (53.6%), but reduced combined throughput by
69.3% in this host-contention smoke. Across one profiled request per engine,
aggregate GPU compute was unchanged (1.359 versus 1.357 seconds) and H2D
payload was exactly 264.24 GiB in both modes. Aggregate CPU `aten::copy_` time
rose from 3.75 to 32.49 seconds, while communication-kernel time rose from
3.56 to 5.80 seconds as TP ranks waited on skewed staging. These durations are
summed across ranks and can overlap; they identify the mechanism, not wall
time. Matching private/cache outputs were byte-identical.
The result confirms that zero-budget runtime-cache staging is a host-capacity
option, not a latency-neutral replacement for private pinned weights. Broader
SP, model, platform, and production-quality validation remains tracked in
[issue #6231](https://github.com/vllm-project/vllm-omni/issues/6231). The
historical B300 rows above predate the runtime cache and must not be treated as
Phase B memory validation.
### L20X registered-runtime-cache smoke
The read-only registered path was then tested with the same MiniMax-H3
workload. These are functional and mechanism-validation measurements, not a
production benchmark. The TP1 pair used one request per wave. Both modes moved
exactly 121.86 GiB H2D in the profiled request and produced byte-identical video
and audio for both measured seeds.
| TP1 no-AllGather mode | Mean wave | Throughput | Request-time host PSS | `Private_Dirty` | Engine peak HBM | CPU `aten::copy_` | GPU compute |
|---|---:|---:|---:|---:|---:|---:|---:|
| Checkpoint mmap + bounded staging | 33.87 s | 0.0295 req/s | 133.65 GiB | 71.16 GiB | 13,226 MB | 21.91 s | 419.28 ms |
| Final runtime cache + registered direct H2D | 4.88 s | 0.2055 req/s | 129.41 GiB | 66.93 GiB | 13,226 MB | 1.64 s | 419.05 ms |
Registration covered 61.73 GiB in 13 page-aligned ranges. Mean wave latency
fell by 85.6% and throughput increased 6.94x. GPU compute and H2D payload were
unchanged, identifying recurrent host transform/packing as the removed work.
The direct path issued 2,425 H2D events versus 1,945 for packed staging, but
that launch increase was much smaller than the eliminated CPU cost.
Two independent TP2 engines also registered the same two TP-coordinate cache
entries across four workers. Compared with the zero-budget cache, mean wave
latency improved from 9.21/12.55 seconds to 3.80/3.86 seconds for engines A/B;
the private-loader reference was 3.61/3.72 seconds. Registered request-time
host PSS was 224.26 GiB versus 232.23 GiB staged and 366.20 GiB private.
Aggregate CPU `aten::copy_` time was 3.83 seconds registered, 32.49 seconds
staged, and 3.75 seconds private; aggregate GPU compute and the 264.24 GiB H2D
payload were effectively unchanged. Ordinary TP communication remained, but
the extra wait caused by staging-driven rank skew largely disappeared.
All workers explicitly unregistered before releasing their mappings, the
system's acquired-minus-released pin counter returned to its pre-run value,
and every GPU allocation was released. Cold runtime-cache publication remained
expensive because it still performs ordinary loading and multiple full-content
passes; that startup problem is outside the registered transfer scope.
Historical test counts and hardware measurements are intentionally not kept
in this design document because they can drift as the implementation changes.
Revalidation must run against the final commit and publish commit-pinned
artifacts before making hardware or performance claims. Track that work in
[issue #6231](https://github.com/vllm-project/vllm-omni/issues/6231).
## Recommendations
- Use **DP + DLO AllGather** for the supported throughput and host-memory
scaling path.
- Use **DP + DLO AllGather** when ranks already execute the same block path in
lockstep and belong to the same fast P2P domain. It shards persistent host
weights across that group.
- Use **SP + DLO AllGather** for long-sequence workloads when DP concurrency is
not the goal.
- Use **no-AllGather** when independent replica execution is required. TP1
@@ -674,5 +510,5 @@ passes; that startup problem is outside the registered transfer scope.
registration budget large enough for the complete final layout when
recurrent staging cost is unacceptable; leave it at zero when page-locking
that layout is not operationally acceptable.
- Prefer **HSDP alone** for production HSDP deployments until the combined
HSDP + DLO no-AllGather path has broader end-to-end coverage.
- Do not combine HSDP with DLO AllGather or runtime-cache v1. HSDP with
no-AllGather remains an ordinary-loader rank-local path.
docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md
+6 -7
View File
@@ -7,7 +7,7 @@ no-AllGather path can optionally share node-local runtime weights through mmap.
See the [DLO component architecture](../../../design/feature/offloader/distributed_layerwise_offload.md#architecture)
for ownership and lifecycle coordination, and the remaining feature design for
loader contracts, compatibility details, failure handling, and benchmarks.
loader contracts, compatibility details, and failure handling.
## Choose a mode
@@ -92,12 +92,11 @@ SP model collectives still synchronize ranks within each engine.
until their runtime layouts are validated.
- Runtime-cache v1 rejects quantized, non-contiguous, aliased/tied, device-only,
HSDP, expert-parallel, CFG-parallel, and PP layouts.
- HSDP plus DLO AllGather is unsupported. HSDP without AllGather has limited
end-to-end validation.
- The runtime cache lowers steady-state host PSS, not startup peak: each worker
still performs ordinary loading and full content validation.
- Skip-load-on-hit, cache eviction, cross-node sharing, and broader SP hardware
validation remain follow-up work in
- HSDP plus DLO AllGather is unsupported, and runtime-cache v1 rejects HSDP.
- The runtime cache changes steady-state host backing, not startup loading:
each worker still performs ordinary loading and full content validation.
- Skip-load-on-hit, cache eviction, and cross-node sharing remain follow-up
work in
[RFC #6195](https://github.com/vllm-project/vllm-omni/issues/6195).
See the [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)