[Diffusion] Register HWR mmap for direct DLO H2D (#6591)

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
Co-authored-by: hsliuustc0106 <hsliuustc0106@users.noreply.github.com>
This commit is contained in:
Hongsheng Liuandhsliuustc0106 authored and GitHub committed 2026-08-25 23:30:11 +08:00
1 parent 5a8b81ebeb
commit c608910d20
28 files changed
+1869 -153

No files matched your search

@@ -196,16 +196,15 @@ adopt it.
### Final-layout Host Weight Runtime consumer
> **Status:** PR2 implementation. The representation-independent identity,
> producer, and restoration contracts landed in
> [PR #6445](https://github.com/vllm-project/vllm-omni/pull/6445); this consumer
> wires the final-layout BF16 path into eligible no-AllGather DLO.
> **Status:** PR2 landed in
> [PR #6486](https://github.com/vllm-project/vllm-omni/pull/6486). PR3 adds
> registered direct H2D plus generic HWR publication/source-digest
> optimizations without changing AllGather behavior.
Final-layout Host Weight Runtime (HWR) backing is opt-in and currently applies
only to no-AllGather DLO. Enable it with
`--host-weight-runtime-mode preferred` (or `required`) and
`--host-weight-runtime-root <node-local-root>`. Registration of shared mappings
and direct asynchronous H2D remain a separate PR3 transport change.
`--host-weight-runtime-root <node-local-root>`.
The modes express operator fallback policy, not different artifact formats:
@@ -254,6 +253,44 @@ outcomes are authoritative:
Checkpoint mmap remains an unchanged control path whenever final-layout HWR is
not selected.
Local source identity and publication avoid repeated full-payload passes while
preserving the same semantic contract:
- Canonical files without a trusted immutable Hub blob identity are hashed in
parallel. Their content IDs are cached per storage domain under the HWR root
and reused only when the path, inode, size, timestamps, symlink target, and
cache-record checksum still match. Corrupt entries are rebuilt; cache-lock or
cache-I/O failure falls back to hashing the canonical file directly.
- Producers that emit payload bytes in canonical storage-key order may declare
`ordered=True`. The filesystem writer then computes file and tensor SHA256
values during the write and overlaps payload `fsync` with later producer
work. Producers without that guarantee retain unordered writes and use a
parallel readback checksum fallback.
- Manifest and `READY` publication still wait for every payload `fsync` and
checksum to complete. These optimizations do not weaken artifact validation
or change lease ownership.
#### Registered mmap transport
After DLO takes an HWR lease, the no-AllGather backend may register the lease's
complete immutable mapped ranges under the existing `pin_cpu_memory` policy.
Registration is transport state: it does not change artifact identity, the
store, tensor ownership, or H2D payload. On success, each tensor view copies
directly into the existing rotating HBM block buffers and the two private host
staging slots are not allocated.
`--dlo-host-registration-limit-gib` is an optional per-worker preflight ceiling
over page-aligned registered bytes. Zero adds no ceiling. A disabled pinned
memory policy, unsupported platform/capability, over-budget mapping, or fully
rolled-back registration error selects the existing two-slot staging path. A
partial registration that cannot be rolled back aborts startup because closing
the lease would unmap memory still owned by the platform.
Direct checkpoint mmap remains unchanged and continues to use staging. It may
require loader-owned per-block transforms, while the HWR artifact already
contains final runtime bytes. DLO AllGather never receives an HWR final-layout
lease and therefore never enters this registration path.
#### Pre-service transaction boundary
The startup transaction does not end at restore commit. A lease-backed model is
@@ -317,6 +354,7 @@ loader owns
-> backend takes once before asynchronous work
-> backend drains pending H2D work
-> backend releases hooks, staging, and model references
-> backend unregisters every mapped range
-> backend closes lease
```
@@ -326,13 +364,15 @@ it. Once the backend takes ownership, only backend abort or teardown may drain
work and close it.
The implementation keeps the final-layout lease through backend setup and
initial prefetch. The backend uses the existing two bounded rank-local staging
slots for the immutable HWR tensors, and its transactional `enable()` cleanup
removes partial hooks and closes the lease before reporting a setup failure.
Preferred mode then uses the runner's fresh canonical retry; required mode
propagates the failure.
initial prefetch. Successful registration copies directly from immutable HWR
views; otherwise the backend uses the existing two bounded rank-local staging
slots. Transactional cleanup drains device work, removes source references,
unregisters mappings, and only then closes the lease. An unregistration failure
retains both registration and lease for retry/process teardown. Preferred mode
then uses the runner's fresh canonical retry; required mode propagates the
failure.
#### PR2 promotion gates
#### PR2 and PR3 promotion gates
- Warm hit performs zero ordinary DiT materialization and zero producer calls.
- Shared warm finalization changes neither restored bytes nor backing pointers.
@@ -344,6 +384,10 @@ propagates the failure.
- Disabled mode and AllGather emit zero final-layout HWR interaction.
- The lease carrier rejects duplicate take and serialization.
- Backend setup or prefetch failure drains asynchronous work before lease close.
- Registration is all-or-nothing; rollback/unregistration failure never closes
a mapping still owned by the platform.
- Registered HWR transport bypasses host staging while preserving output and
H2D payload; unsupported registration retains the bounded staging path.
- Compatible checkpoint mmap remains an unchanged benchmark and control path.
- Prewarm uses one matching producer cohort per TP/SP topology and storage
domain.
@@ -394,11 +438,12 @@ reconstructs those tensors with their recorded layouts. Other online methods
must use `--dlo-no-use-allgather` or disable online quantization until their
runtime layouts are validated.
The Host Weight Runtime representation and publication contracts are merged;
see [RFC #6414](https://github.com/vllm-project/vllm-omni/issues/6414) and
[PR #6445](https://github.com/vllm-project/vllm-omni/pull/6445). The planned
no-AllGather consumer above is the PR2 implementation; registration and direct
asynchronous H2D remain a separate transport follow-up.
The Host Weight Runtime representation, publication, and no-AllGather consumer
contracts are merged; see
[RFC #6414](https://github.com/vllm-project/vllm-omni/issues/6414),
[PR #6445](https://github.com/vllm-project/vllm-omni/pull/6445), and
[PR #6486](https://github.com/vllm-project/vllm-omni/pull/6486). Registered
direct H2D is an optional transport layer over that merged lease contract.
## Validation coverage
@@ -411,8 +456,12 @@ Current source-level validation includes:
- ordinary-loader fallback for per-tensor online FP8 linears followed by DLO
sharding of finalized weights and scales;
- exact loader-to-backend plan transfer and ordinary-loader fallback;
- ordered publication hashing, overlapped durability, unordered parallel
checksum fallback, and node-local source-digest reuse/invalidation;
- rank-local mmap source retention, bounded two-slot staging, and adapter
transforms without parameter-side flags;
- read-only registration preflight, direct source-to-device copies, safe
fallback, partial rollback, retryable unregistration, and lease ordering;
- resident-layer requests requiring no-AllGather;
- DP request-wave validation for denoising-step compatibility;
- bounded component allocator-cache retention, conservative/forced release,
+14 -1
View File
@@ -16,7 +16,7 @@ validation_paths:
- tests/host_weight_runtime/**
upstream_refs:
- https://github.com/vllm-project/vllm-omni/issues/6414
last_reviewed: 2026-08-21
last_reviewed: 2026-08-25
---
# Host Weight Runtime
@@ -84,6 +84,13 @@ another. `WeightArtifactIdentity` includes:
- producer version, implementation fingerprint, manifest schema, and restorer
schema.
Loader integrations may persist content digests for canonical source shards in
the selected storage domain. Reuse requires the same path, inode, size,
timestamps, symlink target, and cache-record checksum. Corrupt records are
rebuilt, and coordination or cache-I/O failure falls back to hashing the
canonical source directly. This is an identity-computation optimization, not a
second source of truth or an artifact substitution rule.
The identity excludes transfer policy such as registered mmap versus private
pinned staging. Those paths move the same host representation and must not
produce duplicate artifacts.
@@ -237,6 +244,12 @@ Every publication computes full file and tensor hashes. Local lookup supports:
and
- `full_checksum`, which additionally reads and hashes every payload file.
An ordered producer may compute file and tensor digests while payload bytes are
written and overlap each closed payload's `fsync` with later producer work. An
unordered producer retains parallel readback hashing. In both cases manifest
and `READY` publication wait for every payload digest and `fsync`; the atomic
publication and validation contract is unchanged.
Filesystem verity is modeled but unsupported. Invalid artifacts receive an
external deny marker before their shared lookup lock is released; a build owner
then quarantines the immutable directory under the exclusive lock.
@@ -65,6 +65,7 @@ omni = Omni(
| `--dlo-resident-layers N` | Keep N leading main-DiT blocks on device; requires no-AllGather and model-declared resident paths | `0` |
| `--host-weight-runtime-mode {disabled,preferred,required}` | HWR policy: no interaction, populate on a miss, or require an exact hit | `disabled` |
| `--host-weight-runtime-root PATH` | Writable node-local HWR store shared by workers in one storage domain; required for `preferred` and `required` | unset |
| `--dlo-host-registration-limit-gib N` | Optional per-worker ceiling for registering an HWR mmap; zero adds no ceiling | `0` |
## Host-weight loading
@@ -178,12 +179,48 @@ returning to `required`.
On a cold start, the canonical loader remains authoritative and publishes a
validated final-layout artifact for later workers. A warm start restores the
DiT final tensors without ordinary DiT materialization, then DLO streams them
through the same two bounded host staging slots. The artifact identity includes
the TP rank/size and SP layout, so TP1, TP2 rank-local shards, and distinct SP
layouts do not alias one another. Publication failure remains separate from a
`preferred` serving startup; the next `required` startup provides the explicit
artifact-availability check.
DiT final tensors without ordinary DiT materialization. DLO then attempts to
register the immutable mapping for direct H2D; if registration is unavailable,
it streams through the same two bounded host staging slots. The artifact
identity includes the TP rank/size and SP layout, so TP1, TP2 rank-local shards,
and distinct SP layouts do not alias one another. Publication failure remains
separate from a `preferred` serving startup; the next `required` startup
provides the explicit artifact-availability check.
For local canonical checkpoints, the first eligible worker may hash source
shards to establish immutable identity. HWR caches those digests in the same
node-local storage domain and validates file metadata before reuse, so later
workers normally avoid repeating that read. Cold BF16 publication also hashes
ordered payloads as they are written and overlaps payload durability work with
later shards; this changes startup work only, not artifact contents or runtime
H2D behavior.
#### Registered direct H2D
Registration is attempted automatically only for an eligible warm HWR hit when
`pin_cpu_memory` is enabled. A successful path is:
```text
shared read-only HWR mmap -> existing rotating HBM block buffer -> GPU kernel
```
It removes the recurrent mmap-to-private-staging CPU copy and does not allocate
the two host staging slots. It does not reduce the H2D payload or make GPU
kernels access host memory directly.
CUDA requires read-only host-registration capability for the immutable HWR
mapping. Unsupported capability, a positive registration limit smaller than
the complete page-aligned mapping, or a safely rolled-back registration error
falls back to bounded staging. Programmatic configurations can set
`pin_cpu_memory=False` to disable registration explicitly. A successful
registration locks the complete mapped range in host memory for that worker's
lifetime, so use
`--dlo-host-registration-limit-gib` when an operator-enforced ceiling is
required.
Shutdown drains H2D work, releases hook/source references, unregisters every
range, and only then closes the HWR lease. AllGather and direct checkpoint mmap
retain their existing transfer paths.
When HWR is disabled, DLO is disabled, or AllGather is enabled, the loader does
not resolve HWR sources or construct its store.