[Core] Add Host Weight Runtime foundation (#6419)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
5730af0a95
commit
73b623f2f7
21 files changed
+6108
-6
No files matched your search
@@ -113,6 +113,7 @@ nav:
|
||||
- Feature Design:
|
||||
- Runtime and Stage Execution:
|
||||
- Disaggregated Inference: design/feature/disaggregated_inference.md
|
||||
- Host Weight Runtime: design/feature/host_weight_runtime.md
|
||||
- Async Chunk: design/feature/async_chunk.md
|
||||
- Async Diffusion Output: design/feature/async_diffusion_output.md
|
||||
- Async Omni Output Materialization: design/feature/omni_async_output_materialization.md
|
||||
@@ -169,6 +170,7 @@ nav:
|
||||
- Offloader: design/module/diffusion/offloader.md
|
||||
- Execution Platforms: design/module/execution_platforms.md
|
||||
- Cache Management: design/module/cache_management.md
|
||||
- Host Weight Runtime: design/module/host_weight_runtime.md
|
||||
- Quantization: design/module/quantization.md
|
||||
- Observability: design/module/observability.md
|
||||
- Profiling: design/module/profiling.md
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
# Host Weight Runtime
|
||||
|
||||
The Host Weight Runtime is a loader-facing feature for reusing immutable,
|
||||
runtime-ready model weights across workers on one host. It lets a model
|
||||
integration replace repeated checkpoint loading and transformation with an
|
||||
exact lookup of a previously published host representation, while keeping the
|
||||
canonical model source authoritative.
|
||||
|
||||
This feature is shared infrastructure. Diffusion, autoregressive, speech, and
|
||||
future model integrations may depend on the top-level
|
||||
`vllm_omni.host_weight_runtime` package without depending on one another's
|
||||
loader or execution stack. The internal component and filesystem contracts are
|
||||
specified in the [Host Weight Runtime module design](../module/host_weight_runtime.md).
|
||||
|
||||
## Status
|
||||
|
||||
The first implementation provides contracts and a CPU local-filesystem store.
|
||||
It does not enable cached loading for a model by itself. A consumer must still
|
||||
provide an exact identity, a representation producer, and a model restorer.
|
||||
|
||||
V1 includes:
|
||||
|
||||
- exact runtime-weight identity and immutable manifests;
|
||||
- coordinated local lookup and one-producer publication;
|
||||
- descriptor-backed safetensors mmap leases;
|
||||
- preferred and required resolution policy;
|
||||
- validation, deny, quarantine, cleanup, and capacity controls; and
|
||||
- typed reports for every terminal resolution outcome.
|
||||
|
||||
V1 does not include:
|
||||
|
||||
- a public CLI or default-on model integration;
|
||||
- a generic BF16, FP8, quantized, or model-specific producer;
|
||||
- CUDA registration, pinned staging, H2D scheduling, or GPU kernels;
|
||||
- a remote artifact provider or cross-node coordination;
|
||||
- post-load publication (V1 accepts only pre-load-safe producers);
|
||||
- automatic eviction; or
|
||||
- a change to DLO AllGather or no-AllGather behavior.
|
||||
|
||||
## Motivation and use cases
|
||||
|
||||
Model loading can create the same final host representation repeatedly. This
|
||||
is especially expensive when loading performs checkpoint decoding, tensor
|
||||
renaming, TP slicing, quantization, packing, or scale construction before GPU
|
||||
transfer. Independent workers may then retain private CPU copies of identical
|
||||
runtime weights.
|
||||
|
||||
The Host Weight Runtime is useful when:
|
||||
|
||||
- multiple same-node workers request semantically identical final weights;
|
||||
- constructing those weights is materially more expensive than validating and
|
||||
mapping a local artifact;
|
||||
- the runtime layout can be identified exactly and reproduced deterministically;
|
||||
- the workers can share immutable file-backed pages through the OS page cache;
|
||||
and
|
||||
- canonical loading remains available when policy permits fallback.
|
||||
|
||||
Typical consumers include same-node diffusion replicas, DLO host-weight
|
||||
sources, and future transformed or quantized model loaders. Request routing and
|
||||
replica orchestration remain outside this feature.
|
||||
|
||||
This feature is not zero-copy GPU execution. A lease exposes stable CPU tensor
|
||||
views and mapped ranges. A separate transport still decides whether to use
|
||||
registered mmap, private pinned staging, synchronous copies, or asynchronous
|
||||
H2D transfer.
|
||||
|
||||
## Resolution behavior
|
||||
|
||||
The loader resolves the immutable canonical source and computes the exact
|
||||
requested identity before asking the runtime for a representation.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
L["Loader resolves canonical source"] --> I["Compute exact WeightArtifactIdentity"]
|
||||
I --> M{"Runtime mode"}
|
||||
M -->|"disabled"| C["Canonical loader"]
|
||||
M -->|"preferred or required"| A{"Validated local artifact?"}
|
||||
A -->|"yes"| R["Return HostWeightLease"]
|
||||
A -->|"no"| X["Remote exact artifact (future)"]
|
||||
X --> P{"Registered producer available?"}
|
||||
P -->|"yes"| B["Build, validate, and publish atomically"]
|
||||
B --> R
|
||||
P -->|"no or recoverable failure"| F{"Policy permits fallback?"}
|
||||
F -->|"preferred"| C
|
||||
F -->|"required"| E["Fail startup"]
|
||||
P -->|"nonretryable failure"| E
|
||||
```
|
||||
|
||||
The modes are:
|
||||
|
||||
| Mode | Observable behavior |
|
||||
| --- | --- |
|
||||
| `disabled` | Use canonical loading directly without probing storage, identity, credentials, or topology. |
|
||||
| `preferred` | Return an exact lease when available; otherwise use canonical fallback only for a miss, invalid cache entry, or typed retryable failure. |
|
||||
| `required` | Fail when an exact lease cannot be acquired. |
|
||||
|
||||
An unsupported capability, semantic identity collision, producer failure, or
|
||||
publication failure is nonretryable and remains visible even in preferred
|
||||
mode. Storage policy must never disguise a semantic or configuration error as
|
||||
a cache miss.
|
||||
|
||||
## Loader and restoration sequence
|
||||
|
||||
The model integration owns the end-to-end loading transaction. The runtime
|
||||
does not construct a model or invoke the canonical loader.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant L as Loader adapter
|
||||
participant R as HostWeightRuntime
|
||||
participant S as HostWeightStore
|
||||
participant P as WeightProducer
|
||||
participant X as WeightRestorer
|
||||
participant T as GPU transport
|
||||
|
||||
L->>L: Resolve canonical revision and exact identity
|
||||
L->>R: resolve(identity, producer)
|
||||
R->>S: lookup(exact identity)
|
||||
alt validated hit
|
||||
S-->>R: HostWeightLease
|
||||
else miss with allowed producer
|
||||
R->>S: get_or_build(identity, producer)
|
||||
S->>P: produce(store-scoped writer)
|
||||
P-->>S: final-layout tensors and metadata
|
||||
S-->>R: validated HostWeightLease
|
||||
else policy permits canonical fallback
|
||||
R-->>L: CANONICAL_FALLBACK
|
||||
L->>L: Run canonical loader
|
||||
else required or nonretryable failure
|
||||
R-->>L: FAILED report
|
||||
end
|
||||
R-->>L: lease and terminal report
|
||||
L->>X: plan_restore(model, lease)
|
||||
X-->>L: validation-only restore plan
|
||||
L->>X: commit() once
|
||||
L->>T: consume restored CPU tensors
|
||||
T-->>L: transfer teardown complete
|
||||
L->>L: close lease
|
||||
```
|
||||
|
||||
`plan_restore()` must not mutate the model or lease. `commit() -> None` is the
|
||||
sole one-shot model mutation. If planning fails, canonical fallback may reuse
|
||||
the untouched model. If commit begins and fails, the partially hydrated model
|
||||
must be discarded and canonical fallback must construct a fresh model.
|
||||
|
||||
## Exact representation identity
|
||||
|
||||
The store performs exact lookup. It never silently converts one representation
|
||||
into another or substitutes the first backing that responds.
|
||||
|
||||
Identity includes:
|
||||
|
||||
- immutable model revision and source fingerprint;
|
||||
- model component and ownership boundary;
|
||||
- representation name, dtype, and format metadata;
|
||||
- final tensor layout and semantic parallel coordinates;
|
||||
- static adaptation identity; and
|
||||
- producer implementation, manifest, and restorer schema versions.
|
||||
|
||||
The requested representation is selected by the loader. The store chooses
|
||||
where to obtain that exact representation: validated local artifact, future
|
||||
remote materialization, or a registered producer.
|
||||
|
||||
### Parallelism
|
||||
|
||||
Parallel coordinates are included only when they change bytes, shape, layout,
|
||||
or component ownership.
|
||||
|
||||
| Dimension | Identity and sharing rule |
|
||||
| --- | --- |
|
||||
| DP | Exclude DP rank for replicated weights so same-node replicas can acquire the same artifact. Include a coordinate only if DP changes weight ownership. |
|
||||
| TP | Include TP size and rank when each rank owns different tensor slices or packed layouts. Each distinct TP shard is a separate exact artifact. |
|
||||
| SP | Include SP size and backend when they change the weight layout. Exclude SP rank when all SP ranks consume identical bytes. |
|
||||
| PP | Encode component ownership or stage-local layout when PP changes which weights belong to the consumer. |
|
||||
| EP | Encode expert ownership and layout whenever ranks own different expert weights. |
|
||||
|
||||
The runtime does not infer these rules from process-group topology. The model
|
||||
adapter is responsible for constructing a correct identity.
|
||||
|
||||
### Quantization and adaptations
|
||||
|
||||
Runtime BF16, FP8, packed quantized weights, and future formats are distinct
|
||||
representations. Conversion is performed by an explicitly versioned producer,
|
||||
never by cache coercion. A producer must publish the final layout consumed by
|
||||
its matching restorer.
|
||||
|
||||
Dynamic LoRA overlays are not part of a reusable base-weight artifact. A
|
||||
statically merged adapter is cacheable only as a separate identity containing
|
||||
the adapter fingerprint and merge semantics.
|
||||
|
||||
## Host sharing and GPU transport
|
||||
|
||||
Every process receives its own virtual mappings and `HostWeightLease`, but
|
||||
workers mapping the same local artifact can share physical file-backed pages
|
||||
through the OS page cache. This avoids independent private tensor allocations;
|
||||
it does not guarantee that process PSS reports exactly one model copy.
|
||||
|
||||
The ownership boundary is:
|
||||
|
||||
- the store owns immutable artifact files, publication, and lifecycle;
|
||||
- the kernel owns page-cache residency and placement;
|
||||
- the lease owns process-local mappings and the shared artifact lock; and
|
||||
- transport owns page registration, page locking, private staging, H2D copies,
|
||||
streams, device buffers, and lease release ordering.
|
||||
|
||||
DLO AllGather and no-AllGather remain transport and execution choices. A DLO
|
||||
integration may consume a lease, but the Host Weight Runtime does not choose a
|
||||
parallel collective or orchestrate DP requests. See the
|
||||
[DLO feature design](offloader/distributed_layerwise_offload.md).
|
||||
|
||||
## Locality and NUMA policy
|
||||
|
||||
The V1 backend is node-local. It accepts an allowlist of known local disk
|
||||
filesystems plus tmpfs/ramfs, records the detected kernel filesystem type, and
|
||||
rejects known remote or unknown filesystems. NFS, CIFS, Lustre, Ceph, and other
|
||||
cross-node mounts cannot silently satisfy the local backend because their
|
||||
page-cache and advisory-lock behavior does not provide the required node-local
|
||||
contract.
|
||||
|
||||
A NUMA-specific root creates a separate storage domain but does not by itself
|
||||
place page-cache pages on that NUMA node. A topology-aware integration must
|
||||
also pin workers, establish local first touch or prefault, perform transport
|
||||
registration in the same domain, and collect residency evidence.
|
||||
|
||||
Tmpfs artifacts consume host memory and may consume swap. They must be
|
||||
accounted as memory-backed storage rather than ordinary disk capacity.
|
||||
|
||||
## Concurrency and timeouts
|
||||
|
||||
One process per exact identity owns a build; other workers wait and then acquire
|
||||
leases for the published artifact. Publication is invisible until all payloads
|
||||
and metadata are validated, hashed, fsynced, and atomically renamed.
|
||||
|
||||
`coordination_timeout_seconds` bounds filesystem lock acquisition. It does not
|
||||
cancel synchronous validation, a producer that has already started, or atomic
|
||||
publication. A hung in-process producer therefore blocks its owning process and
|
||||
must be handled by external process supervision. Enforceable producer
|
||||
cancellation requires a future process-isolated producer contract.
|
||||
|
||||
Leases keep a shared artifact lock for their lifetime. A forked child may close
|
||||
its inherited descriptors but cannot unlock the parent process's lease. Cleanup
|
||||
uses a nonblocking exclusive lock and reports an active lease instead of
|
||||
removing live mappings.
|
||||
|
||||
## Failure and fallback contract
|
||||
|
||||
| Condition | Preferred mode | Required mode |
|
||||
| --- | --- | --- |
|
||||
| Exact local hit | Return lease | Return lease |
|
||||
| Local miss | Try later backing or canonical fallback | Try later backing or fail |
|
||||
| Invalid or denied artifact | Quarantine/rebuild when possible; otherwise canonical fallback | Quarantine/rebuild or fail |
|
||||
| Retryable lock, domain, or capacity failure | Canonical fallback | Fail |
|
||||
| Unsupported producer or backend | Fail | Fail |
|
||||
| Identity collision | Fail | Fail |
|
||||
| Producer or publication failure | Fail | Fail |
|
||||
| Restore planning failure | Canonical fallback may reuse untouched model | Fail |
|
||||
| Restore commit failure | Discard model; fallback requires a fresh instance | Fail |
|
||||
|
||||
Cache failures should normally reduce performance rather than availability,
|
||||
but only when the typed failure explicitly permits fallback. Canonical source,
|
||||
authentication, configuration, and semantic errors remain visible.
|
||||
|
||||
## Consumer integration requirements
|
||||
|
||||
A consumer PR must:
|
||||
|
||||
1. Resolve an immutable canonical model source before cache lookup.
|
||||
2. Construct the exact final representation identity, including relevant
|
||||
parallel, quantization, and adaptation semantics.
|
||||
3. Register a deterministic producer for that identity or support lookup-only
|
||||
operation.
|
||||
4. Provide a validation-only restorer with a one-shot commit.
|
||||
5. Keep GPU transport outside the producer, store, and restorer contracts.
|
||||
6. Handle preferred fallback and required failure without reusing a model after
|
||||
a failed restore commit.
|
||||
7. Retain the lease until every transport operation that may access mapped
|
||||
memory has completed.
|
||||
8. Emit and test the terminal resolution report.
|
||||
|
||||
Consumer validation must prove:
|
||||
|
||||
- output parity with canonical loading;
|
||||
- exact warm-hit work avoidance;
|
||||
- correct TP/SP/PP/EP ownership and layout identity;
|
||||
- correct BF16, FP8, quantized, or adaptation semantics;
|
||||
- shared-backing evidence when host-memory savings are claimed;
|
||||
- clean unmapping, unregistration, and artifact-lock release; and
|
||||
- startup, host-memory, and transport effects for the claimed deployment.
|
||||
|
||||
## Rollout
|
||||
|
||||
The rollout is intentionally staged:
|
||||
|
||||
1. Land the neutral contracts and local filesystem store.
|
||||
2. Add one model-specific producer/restorer integration with parity evidence.
|
||||
3. Connect eligible DLO or other transport paths without moving transport
|
||||
ownership into the runtime.
|
||||
4. Add additional final-layout and quantized producers independently.
|
||||
5. Introduce remote materialization only through the same local lease contract.
|
||||
|
||||
The architectural decisions and deferred work are tracked in
|
||||
[RFC #6414](https://github.com/vllm-project/vllm-omni/issues/6414).
|
||||
@@ -18,6 +18,7 @@ implementation contract; it is not, by itself, a general support claim.
|
||||
### Runtime and stage execution
|
||||
|
||||
- [Disaggregated Inference](feature/disaggregated_inference.md)
|
||||
- [Host Weight Runtime](feature/host_weight_runtime.md)
|
||||
- [Async Chunk](feature/async_chunk.md)
|
||||
- [Async Diffusion Output](feature/async_diffusion_output.md)
|
||||
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
|
||||
@@ -88,14 +89,15 @@ The design contracts separate selection mechanics from backend algorithms:
|
||||
- [Model Integration](module/model_integration.md)
|
||||
- [Autoregressive Runtime](module/ar_runtime.md)
|
||||
- Diffusion
|
||||
- [Overview](module/diffusion/index.md)
|
||||
- [Runtime](module/diffusion/diffusion_runtime.md)
|
||||
- [Model Integration](module/diffusion/diffusion_model_integration.md)
|
||||
- [Continuous Batching](module/diffusion/continuous_batching.md)
|
||||
- [Parallelism](module/diffusion/parallelism.md)
|
||||
- [Offloader](module/diffusion/offloader.md)
|
||||
- [Overview](module/diffusion/index.md)
|
||||
- [Runtime](module/diffusion/diffusion_runtime.md)
|
||||
- [Model Integration](module/diffusion/diffusion_model_integration.md)
|
||||
- [Continuous Batching](module/diffusion/continuous_batching.md)
|
||||
- [Parallelism](module/diffusion/parallelism.md)
|
||||
- [Offloader](module/diffusion/offloader.md)
|
||||
- [Execution Platforms](module/execution_platforms.md)
|
||||
- [Cache Management](module/cache_management.md)
|
||||
- [Host Weight Runtime](module/host_weight_runtime.md)
|
||||
- [Quantization](module/quantization.md)
|
||||
- [Observability](module/observability.md)
|
||||
- [Profiling](module/profiling.md)
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
---
|
||||
title: Host Weight Runtime
|
||||
kind: module
|
||||
status: draft
|
||||
owners:
|
||||
- "@hsliuustc0106"
|
||||
primary_code_paths:
|
||||
- vllm_omni/host_weight_runtime/**
|
||||
related_code_paths:
|
||||
- vllm_omni/diffusion/model_loader/**
|
||||
- vllm_omni/diffusion/offloader/**
|
||||
depends_on:
|
||||
- model_integration.md
|
||||
- quantization.md
|
||||
validation_paths:
|
||||
- tests/host_weight_runtime/**
|
||||
upstream_refs:
|
||||
- https://github.com/vllm-project/vllm-omni/issues/6414
|
||||
last_reviewed: 2026-08-21
|
||||
---
|
||||
|
||||
# Host Weight Runtime
|
||||
|
||||
The Host Weight Runtime is a loader-adjacent, transport-independent foundation
|
||||
for reusing immutable, runtime-ready model weights. It separates the semantic
|
||||
question—"are these exactly the weights this loader requested?"—from the
|
||||
physical question—"where is the validated host representation stored?"
|
||||
|
||||
The runtime can save repeated checkpoint loading, transformation, and peak host
|
||||
memory when several workers need the same final representation. It also creates
|
||||
a second, versioned path for reconstructing a valid model, so a cache miss or
|
||||
recoverable store failure normally falls back to the canonical loader.
|
||||
|
||||
This module does not enable host-weight caching for any model by itself. A model
|
||||
integration must still provide an exact identity, a producer, and a restorer.
|
||||
For feature behavior and consumer integration requirements, see the
|
||||
[Host Weight Runtime feature design](../feature/host_weight_runtime.md).
|
||||
|
||||
## Component boundary
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
LA["Model loader adapter"] -->|"exact identity + policy"| RT["HostWeightRuntime"]
|
||||
RT -->|"lookup / get_or_build"| ST["HostWeightStore"]
|
||||
PR["WeightProducer"] -->|"runtime-ready tensors"| ST
|
||||
ST -->|"validated mmap views"| LE["HostWeightLease"]
|
||||
LE --> RE["WeightRestorer"]
|
||||
RE --> MO["Restored host model"]
|
||||
MO --> TR["DLO or another GPU transport"]
|
||||
|
||||
CS["Canonical model source"] --> PR
|
||||
LA -. "fallback" .-> CS
|
||||
```
|
||||
|
||||
The contracts have deliberately narrow ownership:
|
||||
|
||||
- `HostWeightRuntime` applies resolution and fallback policy and emits one
|
||||
terminal report. It does not own a model or call the canonical loader.
|
||||
- `HostWeightStore` owns artifact lookup, coordinated construction,
|
||||
publication, validation, quarantine, and lifecycle.
|
||||
- `HostWeightLease` owns process-local tensor views, mapped-file resources, and
|
||||
the shared artifact lock that stabilizes their lifetime.
|
||||
- `WeightProducer` creates one declared final-layout representation. It writes
|
||||
only through a store-scoped writer and cannot publish paths directly.
|
||||
- `WeightRestorer` validates a lease-to-model plan without mutation. Its
|
||||
one-shot `commit() -> None` is the sole model-mutating phase. Concrete
|
||||
restorers remain with the model or quantization integration.
|
||||
- GPU transport owns page registration, private staging, H2D scheduling, and
|
||||
lease release. DLO must not parse manifests or manage store artifacts.
|
||||
|
||||
## Exact semantic identity
|
||||
|
||||
Storage lookup is exact; the store never coerces one representation into
|
||||
another. `WeightArtifactIdentity` includes:
|
||||
|
||||
- immutable model revision and source-file fingerprint;
|
||||
- component and ownership boundary;
|
||||
- runtime representation, dtype, and representation metadata;
|
||||
- final tensor layout and semantic parallel coordinates;
|
||||
- static adaptation identity; and
|
||||
- producer version, implementation fingerprint, manifest schema, and restorer
|
||||
schema.
|
||||
|
||||
The identity excludes transfer policy such as registered mmap versus private
|
||||
pinned staging. Those paths move the same host representation and must not
|
||||
produce duplicate artifacts.
|
||||
|
||||
Parallel coordinates follow semantics rather than process topology:
|
||||
|
||||
- TP size and rank are included when they change tensor contents or shape.
|
||||
- DP rank is excluded for replicated weights.
|
||||
- SP size and backend are included when they affect layout. SP rank is excluded
|
||||
only when all SP ranks consume an identical representation.
|
||||
- PP, EP, or future parallel modes must be added by the model adapter whenever
|
||||
they change component ownership or tensor layout.
|
||||
|
||||
Dynamic LoRA overlays are not cached as base weights. A statically merged
|
||||
adapter is a distinct artifact identity with its own fingerprint.
|
||||
|
||||
## Deterministic resolution
|
||||
|
||||
The loader first resolves the canonical source and computes the exact requested
|
||||
identity. The runtime then follows this order:
|
||||
|
||||
```text
|
||||
validated local artifact
|
||||
-> exact remote artifact materialized locally (future)
|
||||
-> registered local producer
|
||||
-> canonical loader fallback, if policy permits
|
||||
```
|
||||
|
||||
An already published local artifact wins because it avoids network transfer and
|
||||
is already in node-accessible storage. "Wins" means the highest-priority valid
|
||||
representation, not whichever backing responds first.
|
||||
|
||||
Configuration modes are:
|
||||
|
||||
- `disabled`: return the canonical-direct outcome without probing a root,
|
||||
credentials, topology, or identity;
|
||||
- `preferred`: recoverable store failures may return canonical fallback; and
|
||||
- `required`: fail when no exact lease can be acquired.
|
||||
|
||||
Failure-to-action policy is centralized in `HostWeightRuntime`. A clean miss or
|
||||
invalid cache artifact may fall back in preferred mode. A typed failure may do
|
||||
so only when it is marked retryable; unsupported capabilities, identity
|
||||
collisions, producer failures, and publication failures remain visible as
|
||||
startup failures.
|
||||
|
||||
`coordination_timeout_seconds` bounds acquisition of lookup and build locks; it
|
||||
is not a hard wall-clock deadline for in-process validation, production, or
|
||||
atomic publication. Once this V1 implementation becomes the producer, the
|
||||
synchronous producer and publication run to completion. A hung producer blocks
|
||||
its owning process and requires external process supervision; enforceable
|
||||
producer cancellation requires a future process-isolated producer contract. A
|
||||
waiter timing out never cancels another process's valid build.
|
||||
|
||||
Remote providers are represented by protocols but are explicitly unsupported
|
||||
in the first implementation. V1 also invokes only `PRE_LOAD_SAFE` producers;
|
||||
enabling post-load publication is rejected during configuration until a
|
||||
separate post-load entry point and model-lifetime contract exist.
|
||||
|
||||
## Restoration transaction boundary
|
||||
|
||||
`WeightRestorer.plan_restore()` is validation-only: it must not modify the model
|
||||
or lease. The returned plan may be committed exactly once, `commit()` returns
|
||||
`None`, and commit is the sole phase allowed to mutate the model. Registration,
|
||||
staging, and GPU transfer remain transport responsibilities rather than restore
|
||||
side effects.
|
||||
|
||||
If planning fails, preferred-mode canonical fallback may reuse the untouched
|
||||
model. If commit begins and then fails, the model is considered partially
|
||||
hydrated and must be discarded; canonical fallback must construct a fresh model
|
||||
instance instead of reusing it.
|
||||
|
||||
## Filesystem domain and artifact lifecycle
|
||||
|
||||
A storage domain describes physical locality and policy; it is not part of
|
||||
semantic identity. The first implementation supports node- or NUMA-scoped
|
||||
roots on verified local disk filesystems or tmpfs. The detected kernel
|
||||
filesystem type is preserved in `domain.json` and inspection output. Known
|
||||
remote filesystems and unknown filesystem types are rejected rather than being
|
||||
silently treated as local disk, because cross-node page cache and advisory-lock
|
||||
semantics do not satisfy this backend's contract. A NUMA root alone does not
|
||||
guarantee page placement; CPU pinning, local first-touch/prefault, and residency
|
||||
evidence belong to a later topology-aware integration.
|
||||
|
||||
```text
|
||||
<domain>/
|
||||
├── domain.json
|
||||
├── domain-policy.json
|
||||
├── locks/
|
||||
│ ├── <identity>.build.lock
|
||||
│ └── <identity>.artifact.lock
|
||||
├── artifacts/<identity>/
|
||||
│ ├── READY.json
|
||||
│ ├── manifest.json
|
||||
│ └── *.safetensors
|
||||
├── tmp/
|
||||
├── quarantine/
|
||||
└── deny/
|
||||
```
|
||||
|
||||
Lock files live outside replaceable artifact directories. Every cooperative
|
||||
mutation follows one lock order: build lock, then exclusive artifact lock.
|
||||
Lookup acquires a shared artifact lock, validates `READY` and the manifest,
|
||||
opens payloads with non-following descriptor-based access, creates mmap-backed
|
||||
views, and transfers that shared lock to the lease.
|
||||
|
||||
Lease acquisition rechecks the producer and restorer schemas against the exact
|
||||
identity. Malformed safetensors payloads are reported as typed invalid
|
||||
artifacts and denied rather than escaping the resolution policy as raw parser
|
||||
exceptions.
|
||||
|
||||
Each `FileLock` records its creator PID. A forked child closes an inherited lock
|
||||
descriptor without issuing `LOCK_UN`, because `flock` state belongs to the
|
||||
shared open-file description and unlocking in the child would also release the
|
||||
parent's active lease lock.
|
||||
|
||||
Publication follows this lifecycle:
|
||||
|
||||
```text
|
||||
ABSENT -> invisible temporary build -> fully hashed and fsynced
|
||||
-> atomic rename -> READY
|
||||
|
||||
READY -> denied -> quarantined -> explicit removal
|
||||
```
|
||||
|
||||
`READY.json` is written last in the temporary directory. Readers therefore see
|
||||
either a complete validated artifact or a miss, never an in-progress build.
|
||||
Failed process-local builds are left invisible and a later build owner removes
|
||||
their stale temporary directories. Explicit cleanup is nonblocking and reports
|
||||
an active builder or lease instead of unlinking live mappings.
|
||||
|
||||
## Validation and capacity
|
||||
|
||||
Every publication computes full file and tensor hashes. Local lookup supports:
|
||||
|
||||
- `manifest_and_metadata`, which validates identity, immutable publication
|
||||
metadata, file descriptors, sizes, tensor keys, shapes, dtypes, and strides;
|
||||
and
|
||||
- `full_checksum`, which additionally reads and hashes every payload file.
|
||||
|
||||
Filesystem verity is modeled but unsupported. Invalid artifacts receive an
|
||||
external deny marker before their shared lookup lock is released; a build owner
|
||||
then quarantines the immutable directory under the exclusive lock.
|
||||
Lookup checks the marker both before validation and after constructing the
|
||||
lease so a weaker overlapping lookup cannot return a lease after a stronger
|
||||
validation has denied the artifact.
|
||||
|
||||
`inspect_domain()` and `inspect_artifact()` return typed, read-only snapshots.
|
||||
Inspection validation does not create a deny marker. Reported lock activity is
|
||||
point-in-time evidence only; kernel locks are not a persistent owner registry.
|
||||
|
||||
Capacity policy covers all store-owned bytes, including ready, temporary, and
|
||||
quarantined data. The local writer preflights `max_artifact_bytes`,
|
||||
`max_store_bytes`, and `min_free_bytes`, and preallocates payloads where the
|
||||
filesystem supports it. `ENOSPC` remains a normal typed store failure because
|
||||
concurrent preflight is inherently racy. Automatic eviction and strict
|
||||
concurrent reservations are not implemented.
|
||||
|
||||
For tmpfs, artifact bytes are host-memory consumption and may consume swap;
|
||||
they must not be reported operationally as ordinary disk capacity. The store
|
||||
owns files and reports capacity. The kernel controls page-cache residency, and
|
||||
transport controls registration or page locking.
|
||||
|
||||
## First implementation scope
|
||||
|
||||
The foundation implements a CPU local filesystem store and
|
||||
`SINGLE_PROCESS` production for one exact artifact. It intentionally adds no:
|
||||
|
||||
- public CLI or DLO configuration;
|
||||
- BF16, FP8, or model-specific producer/restorer;
|
||||
- change to DLO AllGather or no-AllGather behavior;
|
||||
- CUDA registration, pinned staging, or H2D scheduling;
|
||||
- remote provider, group-cohort producer, artifact bundle, eviction, or
|
||||
cross-node coordination.
|
||||
|
||||
Consumer PRs must prove ordinary-loader parity, warm-hit work avoidance,
|
||||
parallel-layout correctness, clean teardown, shared-backing evidence, and
|
||||
representation-specific startup and memory behavior. Generic foundation tests
|
||||
remain CPU-only and make no production performance claim.
|
||||
|
||||
The complete rationale and staged rollout are tracked in
|
||||
[RFC #6414](https://github.com/vllm-project/vllm-omni/issues/6414).
|
||||
Reference in new issue
Block a user