[Core] Add Host Weight Runtime foundation (#6419)

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu authored and GitHub committed 2026-08-21 08:50:59 +08:00
1 parent 5730af0a95
commit 73b623f2f7
21 files changed
+6108 -6

No files matched your search

+2
View File
@@ -113,6 +113,7 @@ nav:
- Feature Design:
- Runtime and Stage Execution:
- Disaggregated Inference: design/feature/disaggregated_inference.md
- Host Weight Runtime: design/feature/host_weight_runtime.md
- Async Chunk: design/feature/async_chunk.md
- Async Diffusion Output: design/feature/async_diffusion_output.md
- Async Omni Output Materialization: design/feature/omni_async_output_materialization.md
@@ -169,6 +170,7 @@ nav:
- Offloader: design/module/diffusion/offloader.md
- Execution Platforms: design/module/execution_platforms.md
- Cache Management: design/module/cache_management.md
- Host Weight Runtime: design/module/host_weight_runtime.md
- Quantization: design/module/quantization.md
- Observability: design/module/observability.md
- Profiling: design/module/profiling.md
+302
View File
@@ -0,0 +1,302 @@
# Host Weight Runtime
The Host Weight Runtime is a loader-facing feature for reusing immutable,
runtime-ready model weights across workers on one host. It lets a model
integration replace repeated checkpoint loading and transformation with an
exact lookup of a previously published host representation, while keeping the
canonical model source authoritative.
This feature is shared infrastructure. Diffusion, autoregressive, speech, and
future model integrations may depend on the top-level
`vllm_omni.host_weight_runtime` package without depending on one another's
loader or execution stack. The internal component and filesystem contracts are
specified in the [Host Weight Runtime module design](../module/host_weight_runtime.md).
## Status
The first implementation provides contracts and a CPU local-filesystem store.
It does not enable cached loading for a model by itself. A consumer must still
provide an exact identity, a representation producer, and a model restorer.
V1 includes:
- exact runtime-weight identity and immutable manifests;
- coordinated local lookup and one-producer publication;
- descriptor-backed safetensors mmap leases;
- preferred and required resolution policy;
- validation, deny, quarantine, cleanup, and capacity controls; and
- typed reports for every terminal resolution outcome.
V1 does not include:
- a public CLI or default-on model integration;
- a generic BF16, FP8, quantized, or model-specific producer;
- CUDA registration, pinned staging, H2D scheduling, or GPU kernels;
- a remote artifact provider or cross-node coordination;
- post-load publication (V1 accepts only pre-load-safe producers);
- automatic eviction; or
- a change to DLO AllGather or no-AllGather behavior.
## Motivation and use cases
Model loading can create the same final host representation repeatedly. This
is especially expensive when loading performs checkpoint decoding, tensor
renaming, TP slicing, quantization, packing, or scale construction before GPU
transfer. Independent workers may then retain private CPU copies of identical
runtime weights.
The Host Weight Runtime is useful when:
- multiple same-node workers request semantically identical final weights;
- constructing those weights is materially more expensive than validating and
mapping a local artifact;
- the runtime layout can be identified exactly and reproduced deterministically;
- the workers can share immutable file-backed pages through the OS page cache;
and
- canonical loading remains available when policy permits fallback.
Typical consumers include same-node diffusion replicas, DLO host-weight
sources, and future transformed or quantized model loaders. Request routing and
replica orchestration remain outside this feature.
This feature is not zero-copy GPU execution. A lease exposes stable CPU tensor
views and mapped ranges. A separate transport still decides whether to use
registered mmap, private pinned staging, synchronous copies, or asynchronous
H2D transfer.
## Resolution behavior
The loader resolves the immutable canonical source and computes the exact
requested identity before asking the runtime for a representation.
```mermaid
flowchart TD
L["Loader resolves canonical source"] --> I["Compute exact WeightArtifactIdentity"]
I --> M{"Runtime mode"}
M -->|"disabled"| C["Canonical loader"]
M -->|"preferred or required"| A{"Validated local artifact?"}
A -->|"yes"| R["Return HostWeightLease"]
A -->|"no"| X["Remote exact artifact (future)"]
X --> P{"Registered producer available?"}
P -->|"yes"| B["Build, validate, and publish atomically"]
B --> R
P -->|"no or recoverable failure"| F{"Policy permits fallback?"}
F -->|"preferred"| C
F -->|"required"| E["Fail startup"]
P -->|"nonretryable failure"| E
```
The modes are:
| Mode | Observable behavior |
| --- | --- |
| `disabled` | Use canonical loading directly without probing storage, identity, credentials, or topology. |
| `preferred` | Return an exact lease when available; otherwise use canonical fallback only for a miss, invalid cache entry, or typed retryable failure. |
| `required` | Fail when an exact lease cannot be acquired. |
An unsupported capability, semantic identity collision, producer failure, or
publication failure is nonretryable and remains visible even in preferred
mode. Storage policy must never disguise a semantic or configuration error as
a cache miss.
## Loader and restoration sequence
The model integration owns the end-to-end loading transaction. The runtime
does not construct a model or invoke the canonical loader.
```mermaid
sequenceDiagram
participant L as Loader adapter
participant R as HostWeightRuntime
participant S as HostWeightStore
participant P as WeightProducer
participant X as WeightRestorer
participant T as GPU transport
L->>L: Resolve canonical revision and exact identity
L->>R: resolve(identity, producer)
R->>S: lookup(exact identity)
alt validated hit
S-->>R: HostWeightLease
else miss with allowed producer
R->>S: get_or_build(identity, producer)
S->>P: produce(store-scoped writer)
P-->>S: final-layout tensors and metadata
S-->>R: validated HostWeightLease
else policy permits canonical fallback
R-->>L: CANONICAL_FALLBACK
L->>L: Run canonical loader
else required or nonretryable failure
R-->>L: FAILED report
end
R-->>L: lease and terminal report
L->>X: plan_restore(model, lease)
X-->>L: validation-only restore plan
L->>X: commit() once
L->>T: consume restored CPU tensors
T-->>L: transfer teardown complete
L->>L: close lease
```
`plan_restore()` must not mutate the model or lease. `commit() -> None` is the
sole one-shot model mutation. If planning fails, canonical fallback may reuse
the untouched model. If commit begins and fails, the partially hydrated model
must be discarded and canonical fallback must construct a fresh model.
## Exact representation identity
The store performs exact lookup. It never silently converts one representation
into another or substitutes the first backing that responds.
Identity includes:
- immutable model revision and source fingerprint;
- model component and ownership boundary;
- representation name, dtype, and format metadata;
- final tensor layout and semantic parallel coordinates;
- static adaptation identity; and
- producer implementation, manifest, and restorer schema versions.
The requested representation is selected by the loader. The store chooses
where to obtain that exact representation: validated local artifact, future
remote materialization, or a registered producer.
### Parallelism
Parallel coordinates are included only when they change bytes, shape, layout,
or component ownership.
| Dimension | Identity and sharing rule |
| --- | --- |
| DP | Exclude DP rank for replicated weights so same-node replicas can acquire the same artifact. Include a coordinate only if DP changes weight ownership. |
| TP | Include TP size and rank when each rank owns different tensor slices or packed layouts. Each distinct TP shard is a separate exact artifact. |
| SP | Include SP size and backend when they change the weight layout. Exclude SP rank when all SP ranks consume identical bytes. |
| PP | Encode component ownership or stage-local layout when PP changes which weights belong to the consumer. |
| EP | Encode expert ownership and layout whenever ranks own different expert weights. |
The runtime does not infer these rules from process-group topology. The model
adapter is responsible for constructing a correct identity.
### Quantization and adaptations
Runtime BF16, FP8, packed quantized weights, and future formats are distinct
representations. Conversion is performed by an explicitly versioned producer,
never by cache coercion. A producer must publish the final layout consumed by
its matching restorer.
Dynamic LoRA overlays are not part of a reusable base-weight artifact. A
statically merged adapter is cacheable only as a separate identity containing
the adapter fingerprint and merge semantics.
## Host sharing and GPU transport
Every process receives its own virtual mappings and `HostWeightLease`, but
workers mapping the same local artifact can share physical file-backed pages
through the OS page cache. This avoids independent private tensor allocations;
it does not guarantee that process PSS reports exactly one model copy.
The ownership boundary is:
- the store owns immutable artifact files, publication, and lifecycle;
- the kernel owns page-cache residency and placement;
- the lease owns process-local mappings and the shared artifact lock; and
- transport owns page registration, page locking, private staging, H2D copies,
streams, device buffers, and lease release ordering.
DLO AllGather and no-AllGather remain transport and execution choices. A DLO
integration may consume a lease, but the Host Weight Runtime does not choose a
parallel collective or orchestrate DP requests. See the
[DLO feature design](offloader/distributed_layerwise_offload.md).
## Locality and NUMA policy
The V1 backend is node-local. It accepts an allowlist of known local disk
filesystems plus tmpfs/ramfs, records the detected kernel filesystem type, and
rejects known remote or unknown filesystems. NFS, CIFS, Lustre, Ceph, and other
cross-node mounts cannot silently satisfy the local backend because their
page-cache and advisory-lock behavior does not provide the required node-local
contract.
A NUMA-specific root creates a separate storage domain but does not by itself
place page-cache pages on that NUMA node. A topology-aware integration must
also pin workers, establish local first touch or prefault, perform transport
registration in the same domain, and collect residency evidence.
Tmpfs artifacts consume host memory and may consume swap. They must be
accounted as memory-backed storage rather than ordinary disk capacity.
## Concurrency and timeouts
One process per exact identity owns a build; other workers wait and then acquire
leases for the published artifact. Publication is invisible until all payloads
and metadata are validated, hashed, fsynced, and atomically renamed.
`coordination_timeout_seconds` bounds filesystem lock acquisition. It does not
cancel synchronous validation, a producer that has already started, or atomic
publication. A hung in-process producer therefore blocks its owning process and
must be handled by external process supervision. Enforceable producer
cancellation requires a future process-isolated producer contract.
Leases keep a shared artifact lock for their lifetime. A forked child may close
its inherited descriptors but cannot unlock the parent process's lease. Cleanup
uses a nonblocking exclusive lock and reports an active lease instead of
removing live mappings.
## Failure and fallback contract
| Condition | Preferred mode | Required mode |
| --- | --- | --- |
| Exact local hit | Return lease | Return lease |
| Local miss | Try later backing or canonical fallback | Try later backing or fail |
| Invalid or denied artifact | Quarantine/rebuild when possible; otherwise canonical fallback | Quarantine/rebuild or fail |
| Retryable lock, domain, or capacity failure | Canonical fallback | Fail |
| Unsupported producer or backend | Fail | Fail |
| Identity collision | Fail | Fail |
| Producer or publication failure | Fail | Fail |
| Restore planning failure | Canonical fallback may reuse untouched model | Fail |
| Restore commit failure | Discard model; fallback requires a fresh instance | Fail |
Cache failures should normally reduce performance rather than availability,
but only when the typed failure explicitly permits fallback. Canonical source,
authentication, configuration, and semantic errors remain visible.
## Consumer integration requirements
A consumer PR must:
1. Resolve an immutable canonical model source before cache lookup.
2. Construct the exact final representation identity, including relevant
parallel, quantization, and adaptation semantics.
3. Register a deterministic producer for that identity or support lookup-only
operation.
4. Provide a validation-only restorer with a one-shot commit.
5. Keep GPU transport outside the producer, store, and restorer contracts.
6. Handle preferred fallback and required failure without reusing a model after
a failed restore commit.
7. Retain the lease until every transport operation that may access mapped
memory has completed.
8. Emit and test the terminal resolution report.
Consumer validation must prove:
- output parity with canonical loading;
- exact warm-hit work avoidance;
- correct TP/SP/PP/EP ownership and layout identity;
- correct BF16, FP8, quantized, or adaptation semantics;
- shared-backing evidence when host-memory savings are claimed;
- clean unmapping, unregistration, and artifact-lock release; and
- startup, host-memory, and transport effects for the claimed deployment.
## Rollout
The rollout is intentionally staged:
1. Land the neutral contracts and local filesystem store.
2. Add one model-specific producer/restorer integration with parity evidence.
3. Connect eligible DLO or other transport paths without moving transport
ownership into the runtime.
4. Add additional final-layout and quantized producers independently.
5. Introduce remote materialization only through the same local lease contract.
The architectural decisions and deferred work are tracked in
[RFC #6414](https://github.com/vllm-project/vllm-omni/issues/6414).
+8 -6
View File
@@ -18,6 +18,7 @@ implementation contract; it is not, by itself, a general support claim.
### Runtime and stage execution
- [Disaggregated Inference](feature/disaggregated_inference.md)
- [Host Weight Runtime](feature/host_weight_runtime.md)
- [Async Chunk](feature/async_chunk.md)
- [Async Diffusion Output](feature/async_diffusion_output.md)
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
@@ -88,14 +89,15 @@ The design contracts separate selection mechanics from backend algorithms:
- [Model Integration](module/model_integration.md)
- [Autoregressive Runtime](module/ar_runtime.md)
- Diffusion
- [Overview](module/diffusion/index.md)
- [Runtime](module/diffusion/diffusion_runtime.md)
- [Model Integration](module/diffusion/diffusion_model_integration.md)
- [Continuous Batching](module/diffusion/continuous_batching.md)
- [Parallelism](module/diffusion/parallelism.md)
- [Offloader](module/diffusion/offloader.md)
- [Overview](module/diffusion/index.md)
- [Runtime](module/diffusion/diffusion_runtime.md)
- [Model Integration](module/diffusion/diffusion_model_integration.md)
- [Continuous Batching](module/diffusion/continuous_batching.md)
- [Parallelism](module/diffusion/parallelism.md)
- [Offloader](module/diffusion/offloader.md)
- [Execution Platforms](module/execution_platforms.md)
- [Cache Management](module/cache_management.md)
- [Host Weight Runtime](module/host_weight_runtime.md)
- [Quantization](module/quantization.md)
- [Observability](module/observability.md)
- [Profiling](module/profiling.md)
+264
View File
@@ -0,0 +1,264 @@
---
title: Host Weight Runtime
kind: module
status: draft
owners:
- "@hsliuustc0106"
primary_code_paths:
- vllm_omni/host_weight_runtime/**
related_code_paths:
- vllm_omni/diffusion/model_loader/**
- vllm_omni/diffusion/offloader/**
depends_on:
- model_integration.md
- quantization.md
validation_paths:
- tests/host_weight_runtime/**
upstream_refs:
- https://github.com/vllm-project/vllm-omni/issues/6414
last_reviewed: 2026-08-21
---
# Host Weight Runtime
The Host Weight Runtime is a loader-adjacent, transport-independent foundation
for reusing immutable, runtime-ready model weights. It separates the semantic
question—"are these exactly the weights this loader requested?"—from the
physical question—"where is the validated host representation stored?"
The runtime can save repeated checkpoint loading, transformation, and peak host
memory when several workers need the same final representation. It also creates
a second, versioned path for reconstructing a valid model, so a cache miss or
recoverable store failure normally falls back to the canonical loader.
This module does not enable host-weight caching for any model by itself. A model
integration must still provide an exact identity, a producer, and a restorer.
For feature behavior and consumer integration requirements, see the
[Host Weight Runtime feature design](../feature/host_weight_runtime.md).
## Component boundary
```mermaid
flowchart LR
LA["Model loader adapter"] -->|"exact identity + policy"| RT["HostWeightRuntime"]
RT -->|"lookup / get_or_build"| ST["HostWeightStore"]
PR["WeightProducer"] -->|"runtime-ready tensors"| ST
ST -->|"validated mmap views"| LE["HostWeightLease"]
LE --> RE["WeightRestorer"]
RE --> MO["Restored host model"]
MO --> TR["DLO or another GPU transport"]
CS["Canonical model source"] --> PR
LA -. "fallback" .-> CS
```
The contracts have deliberately narrow ownership:
- `HostWeightRuntime` applies resolution and fallback policy and emits one
terminal report. It does not own a model or call the canonical loader.
- `HostWeightStore` owns artifact lookup, coordinated construction,
publication, validation, quarantine, and lifecycle.
- `HostWeightLease` owns process-local tensor views, mapped-file resources, and
the shared artifact lock that stabilizes their lifetime.
- `WeightProducer` creates one declared final-layout representation. It writes
only through a store-scoped writer and cannot publish paths directly.
- `WeightRestorer` validates a lease-to-model plan without mutation. Its
one-shot `commit() -> None` is the sole model-mutating phase. Concrete
restorers remain with the model or quantization integration.
- GPU transport owns page registration, private staging, H2D scheduling, and
lease release. DLO must not parse manifests or manage store artifacts.
## Exact semantic identity
Storage lookup is exact; the store never coerces one representation into
another. `WeightArtifactIdentity` includes:
- immutable model revision and source-file fingerprint;
- component and ownership boundary;
- runtime representation, dtype, and representation metadata;
- final tensor layout and semantic parallel coordinates;
- static adaptation identity; and
- producer version, implementation fingerprint, manifest schema, and restorer
schema.
The identity excludes transfer policy such as registered mmap versus private
pinned staging. Those paths move the same host representation and must not
produce duplicate artifacts.
Parallel coordinates follow semantics rather than process topology:
- TP size and rank are included when they change tensor contents or shape.
- DP rank is excluded for replicated weights.
- SP size and backend are included when they affect layout. SP rank is excluded
only when all SP ranks consume an identical representation.
- PP, EP, or future parallel modes must be added by the model adapter whenever
they change component ownership or tensor layout.
Dynamic LoRA overlays are not cached as base weights. A statically merged
adapter is a distinct artifact identity with its own fingerprint.
## Deterministic resolution
The loader first resolves the canonical source and computes the exact requested
identity. The runtime then follows this order:
```text
validated local artifact
-> exact remote artifact materialized locally (future)
-> registered local producer
-> canonical loader fallback, if policy permits
```
An already published local artifact wins because it avoids network transfer and
is already in node-accessible storage. "Wins" means the highest-priority valid
representation, not whichever backing responds first.
Configuration modes are:
- `disabled`: return the canonical-direct outcome without probing a root,
credentials, topology, or identity;
- `preferred`: recoverable store failures may return canonical fallback; and
- `required`: fail when no exact lease can be acquired.
Failure-to-action policy is centralized in `HostWeightRuntime`. A clean miss or
invalid cache artifact may fall back in preferred mode. A typed failure may do
so only when it is marked retryable; unsupported capabilities, identity
collisions, producer failures, and publication failures remain visible as
startup failures.
`coordination_timeout_seconds` bounds acquisition of lookup and build locks; it
is not a hard wall-clock deadline for in-process validation, production, or
atomic publication. Once this V1 implementation becomes the producer, the
synchronous producer and publication run to completion. A hung producer blocks
its owning process and requires external process supervision; enforceable
producer cancellation requires a future process-isolated producer contract. A
waiter timing out never cancels another process's valid build.
Remote providers are represented by protocols but are explicitly unsupported
in the first implementation. V1 also invokes only `PRE_LOAD_SAFE` producers;
enabling post-load publication is rejected during configuration until a
separate post-load entry point and model-lifetime contract exist.
## Restoration transaction boundary
`WeightRestorer.plan_restore()` is validation-only: it must not modify the model
or lease. The returned plan may be committed exactly once, `commit()` returns
`None`, and commit is the sole phase allowed to mutate the model. Registration,
staging, and GPU transfer remain transport responsibilities rather than restore
side effects.
If planning fails, preferred-mode canonical fallback may reuse the untouched
model. If commit begins and then fails, the model is considered partially
hydrated and must be discarded; canonical fallback must construct a fresh model
instance instead of reusing it.
## Filesystem domain and artifact lifecycle
A storage domain describes physical locality and policy; it is not part of
semantic identity. The first implementation supports node- or NUMA-scoped
roots on verified local disk filesystems or tmpfs. The detected kernel
filesystem type is preserved in `domain.json` and inspection output. Known
remote filesystems and unknown filesystem types are rejected rather than being
silently treated as local disk, because cross-node page cache and advisory-lock
semantics do not satisfy this backend's contract. A NUMA root alone does not
guarantee page placement; CPU pinning, local first-touch/prefault, and residency
evidence belong to a later topology-aware integration.
```text
<domain>/
├── domain.json
├── domain-policy.json
├── locks/
│ ├── <identity>.build.lock
│ └── <identity>.artifact.lock
├── artifacts/<identity>/
│ ├── READY.json
│ ├── manifest.json
│ └── *.safetensors
├── tmp/
├── quarantine/
└── deny/
```
Lock files live outside replaceable artifact directories. Every cooperative
mutation follows one lock order: build lock, then exclusive artifact lock.
Lookup acquires a shared artifact lock, validates `READY` and the manifest,
opens payloads with non-following descriptor-based access, creates mmap-backed
views, and transfers that shared lock to the lease.
Lease acquisition rechecks the producer and restorer schemas against the exact
identity. Malformed safetensors payloads are reported as typed invalid
artifacts and denied rather than escaping the resolution policy as raw parser
exceptions.
Each `FileLock` records its creator PID. A forked child closes an inherited lock
descriptor without issuing `LOCK_UN`, because `flock` state belongs to the
shared open-file description and unlocking in the child would also release the
parent's active lease lock.
Publication follows this lifecycle:
```text
ABSENT -> invisible temporary build -> fully hashed and fsynced
-> atomic rename -> READY
READY -> denied -> quarantined -> explicit removal
```
`READY.json` is written last in the temporary directory. Readers therefore see
either a complete validated artifact or a miss, never an in-progress build.
Failed process-local builds are left invisible and a later build owner removes
their stale temporary directories. Explicit cleanup is nonblocking and reports
an active builder or lease instead of unlinking live mappings.
## Validation and capacity
Every publication computes full file and tensor hashes. Local lookup supports:
- `manifest_and_metadata`, which validates identity, immutable publication
metadata, file descriptors, sizes, tensor keys, shapes, dtypes, and strides;
and
- `full_checksum`, which additionally reads and hashes every payload file.
Filesystem verity is modeled but unsupported. Invalid artifacts receive an
external deny marker before their shared lookup lock is released; a build owner
then quarantines the immutable directory under the exclusive lock.
Lookup checks the marker both before validation and after constructing the
lease so a weaker overlapping lookup cannot return a lease after a stronger
validation has denied the artifact.
`inspect_domain()` and `inspect_artifact()` return typed, read-only snapshots.
Inspection validation does not create a deny marker. Reported lock activity is
point-in-time evidence only; kernel locks are not a persistent owner registry.
Capacity policy covers all store-owned bytes, including ready, temporary, and
quarantined data. The local writer preflights `max_artifact_bytes`,
`max_store_bytes`, and `min_free_bytes`, and preallocates payloads where the
filesystem supports it. `ENOSPC` remains a normal typed store failure because
concurrent preflight is inherently racy. Automatic eviction and strict
concurrent reservations are not implemented.
For tmpfs, artifact bytes are host-memory consumption and may consume swap;
they must not be reported operationally as ordinary disk capacity. The store
owns files and reports capacity. The kernel controls page-cache residency, and
transport controls registration or page locking.
## First implementation scope
The foundation implements a CPU local filesystem store and
`SINGLE_PROCESS` production for one exact artifact. It intentionally adds no:
- public CLI or DLO configuration;
- BF16, FP8, or model-specific producer/restorer;
- change to DLO AllGather or no-AllGather behavior;
- CUDA registration, pinned staging, or H2D scheduling;
- remote provider, group-cohort producer, artifact bundle, eviction, or
cross-node coordination.
Consumer PRs must prove ordinary-loader parity, warm-hit work avoidance,
parallel-layout correctness, clean teardown, shared-backing evidence, and
representation-specific startup and memory behavior. Generic foundation tests
remain CPU-only and make no production performance claim.
The complete rationale and staged rollout are tracked in
[RFC #6414](https://github.com/vllm-project/vllm-omni/issues/6414).