[Core] Reduce local stage payload overhead with shared host rings
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
1 parent
023716ecc4
commit
06202d8f2a
9 files changed
+1108
-12
No files matched your search
@@ -0,0 +1,79 @@
|
||||
# Local stage host transport
|
||||
|
||||
`SharedMemoryConnector` uses a persistent host ring for ordinary local stage
|
||||
payloads by default. Models keep the existing `put`/`get` and typed payload
|
||||
contracts; no model-specific sender, receiver or deployment change is required.
|
||||
This applies to both V1 and MRV2 paths using this connector, including CPU codec
|
||||
codes and CPU hidden states. Remote connectors use their existing transports.
|
||||
|
||||
Each producer process and destination stage lazily allocates a 4 MiB ring.
|
||||
Small CPU tensor trees use a msgpack header and raw tensor regions. Plain
|
||||
serialized payloads can use the same ring. Frames larger than a quarter of the
|
||||
ring, and frames arriving while it is full, use the existing per-key SHM path.
|
||||
The fallback preserves delivery without blocking a model step on ring capacity.
|
||||
Large or unsupported tensor trees do not allocate an unused contiguous snapshot.
|
||||
|
||||
## Ownership and delivery
|
||||
|
||||
The producer copies the payload before `put` returns. Readers hold the ring's
|
||||
file lock while copying into independently owned CPU storage, then mark the
|
||||
frame consumed. Multiple receiving processes may compete for a key; one reader
|
||||
claims it. Existing TP fanout remains responsible for distributing that owned
|
||||
payload to peer ranks. This transport does not provide multicast.
|
||||
|
||||
Small tensor leaves share one owned allocation. Large multi-leaf messages use
|
||||
separate allocations, so retaining small metadata does not retain a large
|
||||
hidden state. No view of ring storage escapes the receive boundary.
|
||||
|
||||
Cancellation retires exact keys or tracked numeric chunk suffixes. Replacing
|
||||
an unclaimed key retires its previous frame, including transitions between
|
||||
the ring and per-key SHM. Descriptors include monotonic frame positions; an old
|
||||
descriptor cannot read a newer publication after wraparound. Keys must have
|
||||
one active publisher, as assumed by the existing stage request routing.
|
||||
|
||||
Only producers register ring allocations with Python's shared-memory resource
|
||||
tracker. Readers map files without taking ownership. Producer process identity
|
||||
includes its start time, and available Linux pidfds detect producer exit, so
|
||||
old frames cannot be delivered after a producer restart. Normal close removes
|
||||
owned allocations and discovery markers. Resource tracking covers allocation
|
||||
cleanup when the owning application exits; an abnormal exit can leave small
|
||||
discovery markers, which contain no payload data.
|
||||
Live receivers release cached mappings after a producer exits and prune stale
|
||||
markers during discovery within their deployment scope.
|
||||
|
||||
## Configuration and observation
|
||||
|
||||
Connector extras are shared by the endpoints of an edge. For example:
|
||||
|
||||
```yaml
|
||||
connectors:
|
||||
shm:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
host_ring_bytes: 4194304
|
||||
```
|
||||
|
||||
Set `host_ring_bytes: 0` at both endpoints to use the per-key path, including
|
||||
controlled comparisons or deployments with older receivers. Nonzero capacities
|
||||
must be between 4 KiB and 64 MiB. All ring endpoints must support this protocol.
|
||||
The launcher supplies a common deployment scope; discovery scans only that
|
||||
scope's directory. Receive attempts do not wait for another process's file lock.
|
||||
|
||||
`health()` exposes `host_ring_puts`, `host_ring_gets` and `host_ring_fallbacks`
|
||||
alongside the existing counters. Ring setup failure logs once per edge and uses
|
||||
per-key SHM. CPU payload size and the cost of serialization determine the benefit;
|
||||
reduced transfer latency does not imply higher model throughput.
|
||||
|
||||
## Scope
|
||||
|
||||
This mechanism reduces host serialization copies and per-message segment work.
|
||||
It does not retain worker outputs on the GPU, allocate a CUDA arena, or replace
|
||||
GPU stream synchronization. It also does not impose a global byte budget on
|
||||
queued snapshots, downstream model caches or client audio delivery. The bounded
|
||||
ring with a compatibility fallback is not end-to-end streaming backpressure.
|
||||
|
||||
Regression tests are in
|
||||
`tests/distributed/omni_connectors/test_shm_ring.py`, collected by the existing
|
||||
`core_model and cpu` CI lane. They cover ownership, tensor types, typed payloads,
|
||||
pressure fallback, cancellation, overwrite, deadlines, competing processes and
|
||||
producer restart.
|
||||
Reference in new issue
Block a user