[Feature] Transfer AR-to-diffusion payloads over NIXL (#6264)

Signed-off-by: yuanwu <yuan.wu@intel.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Yuan WuandHongsheng Liu authored and GitHub committed 2026-09-23 18:26:53 +08:00
1 parent 8953d082c0
commit cb5f508212
60 files changed
+4397 -641

No files matched your search

@@ -1,5 +1,8 @@
# NixlConnector
For the reusable model-runner API, illustrated control flow, deadline semantics,
and failure/ownership contracts, see [Cross-Stage Payload Transport](stage_payload.md).
## When to Use
Multi-node or intra-node stage transfer over NIXL, which brokers RDMA/shared-memory
@@ -46,6 +49,15 @@ provides a matching torch-xpu, UCX with Level Zero support, and NIXL runtime.
## Configuration
MiniMax-H3's default disaggregated and Turbo deploy configs use shared memory
for same-host transfer. To opt into NIXL, select
`vllm_omni/deploy/minimax_h3_disaggregated_nixl.yaml` or
`vllm_omni/deploy/minimax_h3_disaggregated_turbo_nixl.yaml` with
`--deploy-config`. These overlays inherit the corresponding model and
parallel settings and switch both ends of the stage edge. They require NIXL
and a working UCX backend; initialization failures do not automatically fall
back to shared memory.
```yaml
connectors:
nixl_connector:
@@ -92,9 +104,44 @@ Parameters:
- `transfer_timeout_s`: how long a `get()` waits for its `READ` to complete
(default 300). NIXL 1.3 has no transfer cancellation API, so a timed-out transfer's
buffers and registrations remain owned by the connector until NIXL reports a
terminal state; `close()` waits for that state before releasing them.
terminal state. `close()` polls remaining transfers once and returns without
releasing active DMA resources. A serialized background closer automatically
finishes cleanup after completion; no second call is required.
`VLLM_OMNI_NIXL_XFER_TIMEOUT_S` overrides it.
### Diffusion worker integration
Diffusion initializes `OmniConnectorModelRunnerMixin` in synchronous mode,
borrowing its KV transfer manager's lazy connector. This mode creates no second
connector, starts no receive/save threads, and leaves KV key builders and
sharding callbacks unchanged. Connector lifetime remains with the manager.
The shared `recv_stage_payload` entry point and asynchronous `_poll_single_request`
use the same key and endpoint metadata builder. Diffusion passes its actual
incoming edge and external request ID; explicit transfer handles take precedence.
Sender endpoints are passed per request, without mutating KV sender state.
Diffusion-specific prompt merging, output handles and device placement live in
`diffusion/worker/stage_payload.py`.
Diffusion supplies ordered TP and SP groups to the shared payload fanout.
Only the rank leading both groups reads the connector. Each subsequent broadcast
uses its own group-local rank zero, including ranks that received the payload
from the preceding group. Tensor payloads use tensor-dictionary broadcasts rather
than serializing the full payload as a Python object. Receive misses still reach
the same collectives. Inline fallback is allowed only when all required payload
keys remain present; otherwise each rank raises an explicit payload error.
The synchronous path passes one monotonic deadline to `get_with_deadline()` for
discovery and DMA completion waits, rather than calling the ordinary `get()`
with its potentially much longer transfer timeout. Native calls, serialization,
and distributed collectives are not preemptible by this deadline.
`GroupCoordinator.broadcast_tensor_dict` preserves the group-local source for
`broadcast_object` while passing the mapped global source to PyTorch tensor
collectives. For subgroup `[1, 3]`, local source `0` maps to global source `1`;
passing `1` back to `broadcast_object` would incorrectly select global rank `3`.
The nonzero-subgroup regression covers both sender and receiver behavior.
### Source ownership and failure limits
Claims and expiry use the same lock. Completion identifies both a payload
@@ -105,12 +152,24 @@ transfers retain the complete destination bundle and source claim until terminal
Zero-element tensors retain their shape/dtype/skeleton positions but never create
DMA descriptors or registrations.
A lost metadata response, lost completion ACK, or abandoned consumer can retain a
claim indefinitely. NIXL 1.3 cannot prove remote cancellation, so neither TTL nor
After a lost metadata response, retries on the same consumer thread reuse the
claim ID for that endpoint, key and requested generation until a reply arrives.
This makes claim acquisition idempotent; subsequent independent reads still get
distinct claims. Callers retry on the same thread and call `abandon_get()` when
giving up. Synchronous stage payload receive does this automatically. Background
recovery reacquires the same claim solely to acknowledge it, without issuing a
READ. Lost completion ACKs are retried against the exact payload generation.
A permanently unreachable or dead consumer can still retain a claim indefinitely.
NIXL 1.3 cannot prove remote cancellation, so neither TTL nor
`cleanup()` frees those allocations. Producer `close()` rejects new work and
retains its agent, listener and claimed allocations; a later `close()` finishes
retains its agent, listener and claimed allocations; the background closer finishes
teardown after claims drain. Permanently abandoned claims remain until process
exit. This favors memory safety over bounded shutdown/memory usage. Only trusted
exit. A consumer with unfinished local transfers likewise retains a strong reference
to its connector, agent, tensors and registrations after `close()` returns. Its
background closer polls and releases resources only after a terminal state;
permanently stuck transfers retain resources until process exit. Closing connectors
reject new work and report unhealthy. This avoids an unbounded transfer-polling loop
in shutdown without freeing DMA-owned memory. Only trusted
peers may access this unauthenticated control plane. Both endpoints must use the
claim-aware protocol; rolling compatibility with older GET_META clients is not
provided. Legacy externally-owned metadata without a generation is accepted only
@@ -0,0 +1,280 @@
# Cross-Stage Payload Transport
Stage payload transport moves model outputs such as encoder conditioning between
workers without routing their tensor contents through the orchestrator. It is
separate from native diffusion KV transfer: a deployment can use both, either, or
neither. MiniMax-H3 is one consumer of this interface, not its implementation.
## Architecture
```mermaid
flowchart TB
Config[Topology and deploy configuration] --> Keys[Input and output payload keys]
Keys --> Producer[Producer model runner]
Keys --> Consumer[Consumer model runner]
Producer --> Adapter[DiffusionStagePayloadMixin]
Consumer --> Adapter
Adapter --> Runtime[Shared OmniConnector runtime]
Runtime --> Descriptor[Request key and endpoint resolution]
Runtime --> Fanout[TP then SP or HSDP shard then replicate fanout]
Runtime --> Manager[OmniKVTransferManager owns lazy connector]
Manager --> SHM[SharedMemoryConnector]
Manager --> NIXL[NixlConnector]
Producer -. small transfer handle .-> Orchestrator[Orchestrator]
Orchestrator -. prompt and handle .-> Consumer
Consumer --> NativeKV[Separate native diffusion KV path]
```
The diffusion adapter owns model-facing selection and validation. The shared
runtime owns request-key construction, endpoint metadata, retry policy, and
distributed fanout. Backends own storage, transfer progress, and allocation
lifetime. No pipeline should instantiate a NIXL agent or implement its own
metadata-query loop.
The synchronous runtime borrows the manager's connector. It does not create
background AR receive/save loops or replace the manager's KV key callbacks.
The asynchronous AR path continues using `get()` through its existing poller.
Both paths share the descriptor and fanout helpers; their scheduling policies
are deliberately different.
## Configuration and Identity
`stage_output_payload_keys` selects non-`None` entries in a producer's
`DiffusionOutput.custom_output`. `stage_input_payload_keys` declares the required
entries in the consumer's `prompt.additional_information`. Both normalize to
tuples in `OmniDiffusionConfig`, including lists loaded from YAML.
The model topology declares these keys; deploy configuration selects the
connector for the edge. MiniMax-H3's standard and Turbo disaggregated profiles
default to SharedMemory. Their `_nixl.yaml` overlays opt into NIXL with
`--deploy-config`; choosing NIXL does not imply fallback to SharedMemory.
Without an explicit handle, the key is:
```text
{external_req_id or request_id}_{from_stage}_{chunk_id}
```
The synchronous complete-payload path uses chunk zero. Source and destination
IDs come from the configured edge, not from an assumption that stages are
adjacent or that the producer is stage zero. An explicit handle can override
the key, source, destination, and backend metadata:
```python
{
"key": "request-42_2_0",
"from_stage": "2",
"to_stage": "5",
"size_bytes": 4096,
"metadata": {...},
"payload_keys": ["text_encoder_output"],
}
```
This descriptor is attached under `_stage_payload_transfer`. Backend metadata
is opaque to the pipeline. If it is absent, request-scoped sender information
can supply `source_host` and `source_port`. `payload_sender_info` takes
precedence over legacy `kv_sender_info`. A flat endpoint is accepted directly;
a stage-keyed map is resolved using the actual source stage, accepting string
and integer keys. A single unambiguous candidate remains supported. Multiple
unmatched candidates are not arbitrarily resolved to stage zero.
## API Contracts
| API | Owner | Contract |
| --- | --- | --- |
| `init_omni_connectors(..., synchronous=True)` | Shared runtime | Borrow an existing manager; no AR polling threads or KV callback mutation. |
| `_stage_payload_recv_spec(...)` | Shared transport | Resolve explicit handle or conventional key plus request-scoped endpoint. |
| `recv_stage_payload(..., retry_seconds=2.0)` | Shared transport | Leader retries with one monotonic deadline, retires discovery, then fans out success or absence. |
| `get_with_deadline(..., deadline=...)` | Connector | Bound backend-controlled waits by the supplied absolute monotonic deadline. Return `(payload, bytes)` or `None`. |
| `abandon_get(key)` | Connector | Retire unresolved discovery on the calling thread after the retry loop. Never free an active READ. Default is a no-op for stateless discovery. |
| `_maybe_recv_stage_payload(request)` | Diffusion adapter | Skip warmup, receive, merge declared keys on the target device, and reject missing required values. |
| `_maybe_send_stage_payload(requests, outputs)` | Diffusion adapter | Leader publishes selected complete outputs; all ranks attach the same handle and remove only transferred fields. |
| `OmniKVTransferManager.close()` | Manager | Drain prefetch, close only the connector already created, and disable later lazy initialization. |
`get_with_deadline()` is intentionally not a wrapper around an arbitrary
blocking `get()`. The base implementation raises `NotImplementedError`.
An unsupported backend must implement this capability before being used for
synchronous remote-only payloads. Existing asynchronous `get()` users are
unchanged.
The deadline covers retry sleeps, metadata socket waits, and NIXL completion
polling. It is **not a hard real-time bound** on Python execution, allocation,
serialization, individual native driver calls, device copies, or payload-group
collectives. A stuck native call cannot safely be preempted with a Python
future timeout. The implementation does not start a detached READ and discard
its ownership when the model-thread budget expires.
## Successful Transfer
The synchronous diffusion producer uses an explicit handle:
```mermaid
sequenceDiagram
participant P as Producer leader
participant B as Connector
participant O as Orchestrator
participant C as Consumer leader
participant R as Consumer peer ranks
P->>B: put(key, selected custom_output)
B-->>P: success, bytes, backend metadata
P->>P: Broadcast handle; remove transferred inline keys
P->>O: Output carrying small handle
O->>C: Prompt carrying handle and endpoint
C->>B: get_with_deadline(key, deadline)
B-->>C: Complete payload or absence
C->>B: abandon_get(key)
C->>R: Delivery status, then tensors if present
C->>C: Merge and validate required keys
R->>R: Merge and validate required keys
```
MiniMax-H3 instead uses the AR encoder's asynchronous sender. It does not
forward the diffusion producer's explicit handle:
```mermaid
sequenceDiagram
participant A as AR encoder leader
participant S as Background sender
participant B as Connector
participant O as Orchestrator
participant C as Diffusion consumer leader
participant R as Consumer peer ranks
A->>S: Queue encoder conditioning for conventional key
A->>O: Output without inline conditioning
O->>C: Request identity and request-scoped sender endpoint
par Asynchronous publication
S->>B: put(conventional key, conditioning), with retries
and Bounded receive
C->>B: get_with_deadline(conventional key, endpoint)
B-->>C: Payload or absence
end
C->>B: abandon_get(key)
C->>R: Delivery status, then tensors if present
C->>C: Merge and validate encoder_output
R->>R: Merge and validate encoder_output
```
Within a TP-by-SP payload grid, exactly one rank, the leader in both groups,
accesses the connector. The shared broadcaster traverses TP and then SP in
the same order on every rank: TP first populates the source SP row, then SP
propagates each TP column.
When initialized HSDP groups span the stage, the adapter instead uses the
HSDP shard and replicate dimensions. The shard broadcast populates the first
replica row; the replicate broadcast propagates every shard position to the
other rows. Both dimensions participate in leader election. This includes
standalone HSDP, whose TP/SP groups are singletons, as well as pure sharding
and pure replication. The two HSDP dimensions replace, rather than overlap,
the TP/SP fanout.
The single-consumer guarantee is scoped to these configured payload groups;
it is not a global election across arbitrary independent DP/CFG/PP groups or
stage replicas. Followers do not initialize or query a connector. Absence is
broadcast too, so peers do not wait for tensor broadcasts that the leader will
never issue.
Pipelines must gather sharded producer outputs before publication. Selecting
payload keys does not gather tensor shards automatically.
## Execution and Failure Policy
Full forward receives before pipeline execution and sends after output
preparation. Stepwise execution receives once when a new cached state is
created. That state retains the external request ID; final decoded output is
published exactly once. Intermediate denoising steps and non-final streaming
chunks are not published under the complete-payload key. Chunk-addressed
streaming transfer remains the asynchronous path's separate contract.
Dummy request IDs (`dummy_req_id` and its slash-prefixed variants) bypass both
directions. A pure warmup batch does not lazily initialize the connector,
publish buffers, wait for remote data, or enter payload fanout.
For the synchronous diffusion producer, failed `put()` calls preserve inline
fields; only successful publications replace those fields with a handle.
This guarantee does not apply to H3's AR encoder asynchronous sender. That
path omits inline conditioning before the background `put()` completes;
exhausting its send retries does not restore conditioning to the forwarded
output. It uses the conventional key and request-scoped sender endpoint,
not the diffusion producer's handle-based handoff.
If a receive fails, inline continuation is valid only when **all required
keys have non-`None` values**. Missing required values raise an explicit
stage-payload error on each rank after fanout. H3 with omitted inline
conditioning therefore fails explicitly if remote publication or delivery
fails; the original prompt is not equivalent conditioning. With no declared
input keys, an explicit handle's `payload_keys` defines the required set.
## NIXL Ownership and Recovery
```mermaid
stateDiagram-v2
[*] --> Published: put registers source
Published --> Released: unclaimed lease expires
Published --> Claimed: atomic metadata query
Claimed --> Reading: consumer submits READ
Claimed --> Recovering: reply lost and caller abandons
Recovering --> Recovering: query or ACK lost
Recovering --> Released: same claim recovered and ACKed
Reading --> Deferred: deadline expires while DMA may be active
Reading --> Released: terminal DMA and completion ACK
Deferred --> Deferred: active or unknown DMA state
Deferred --> Released: terminal state and completion ACK
```
Claims are scoped by payload generation and UUID. Query timeouts or lost
metadata replies retain the cached UUID for retries, since the producer may
already have granted ownership. An explicit `META_NOT_FOUND` reply clears
the cached claim; a subsequent query can use a new UUID. At the end of the
synchronous retry loop, `abandon_get()`
moves unresolved queries to background recovery. Recovery asks for the same
claim, obtains the generation, and acknowledges it **without submitting a
READ**. A lost completion reply is retried against that exact generation,
not by acquiring a fresh payload with the same key. Existing claims may be
recovered while a producer is closing; new claims may not.
If a READ was submitted and its state is active or unknown, timeout does not
release source or destination allocations. The deferred-transfer reaper keeps
tensors, registrations, descriptor lists, and transfer handles until NIXL
reports a terminal state. Completion ACK failure also stays in background
retry. Producer close can therefore remain deferred until ownership drains.
Recovery is eventual, not guaranteed under permanent network partition or
consumer process death. A producer cannot safely infer that an unreachable
consumer has stopped DMA. Conservatively retained ownership in that case is
an explicit safety limit, distinct from a live consumer silently abandoning
a recoverable discovery claim.
## Reusing the Interface
| Backend | Synchronous support | Backend responsibility |
| --- | --- | --- |
| SharedMemory | Implemented | Nonblocking read-lock acquisition; return absence on contention; shared runtime retries. Copy and deserialize after acquiring the lock. |
| NIXL | Implemented | Deadline-aware socket and completion waits, generation-scoped claims, deferred active-DMA cleanup, discovery recovery. |
| Other existing connectors | Opt-in required | Implement `get_with_deadline`; implement `abandon_get` only when discovery creates ownership. Do not inherit an unbounded blocking call. |
To add a model, declare input/output keys and use ordinary prompt and output
structures. To add a backend, implement the connector capability and ownership
tests, register it through the existing factory, and select it in deploy
configuration. Neither requires a backend-specific diffusion model runner.
Worker teardown disables offloading, closes the manager after draining
prefetch, then tears down native KV and distributed resources in `finally`
blocks. The synchronous mixin does not close the borrowed connector itself.
Repeated manager close is safe, including for an unused manager.
## Validation Map
| Test module | Main invariants |
| --- | --- |
| `tests/diffusion/test_diffusion_stage_payload.py` | Warmup bypass, strict missing-payload errors, valid inline fallback, source-stage endpoints, TP/SP fanout, standalone HSDP election and SHM fanout, handle publication. |
| `tests/diffusion/test_diffusion_model_runner.py` | New-request receive and final-only stepwise publication with cached external identity. |
| `tests/distributed/omni_connectors/test_nixl_connector.py` | Lost replies, same-claim recovery, deadlines, generation isolation, active DMA retention, deferred close and ACKs. |
| `tests/distributed/omni_connectors/test_shm_connector.py` | Key and metadata reads, lock contention, expired deadline, cleanup. |
| `tests/worker/test_omni_connector_mixin.py` | Borrowed connector ownership and shared synchronous/asynchronous helpers. |
| `tests/distributed/omni_connectors/test_kv_async_prefetch.py` | Prefetch ordering and idempotent manager close. |
| `tests/config/test_omni_config.py` | Tuple normalization across topology, deploy, stage CLI, and serialization. |
Mocked ownership tests do not establish native transport performance or
accelerator correctness. Native NIXL and model E2E runs are separate checks;
their results must name the tested revision and runtime.