vllm-omni/docs/design/feature/omni_connectors/nixl_connector.md

NixlConnector

When to Use

Multi-node or intra-node stage transfer over NIXL, which brokers RDMA/shared-memory transports through a single API. Useful when the deployment already standardises on NIXL for KV transfer, or when the target accelerator has no Mooncake/Mori backend -- NIXL reaches Intel XPU through UCX's Level Zero support.

Mechanism

Uses vLLM's NixlWrapper (vllm.distributed.nixl_utils) to register the producer's tensors and let the consumer pull them with a NIXL READ.

• Data Plane: NIXL agent-to-agent READ, GPU-to-GPU where the backend allows it.
• Control Plane: a ZMQ ROUTER socket atomically claims the source allocation before returning descriptors. Forwarded put() metadata includes the generation and endpoint; the consumer claims that generation before submitting any READ.

Transfer metadata uses schema version 1. Tensor descriptors are grouped by NIXL memory type (DRAM, VRAM, and so on), while each descriptor retains its global tensor index so mixed CPU/accelerator payloads can be reconstructed in order.

Payloads are not restricted to tensors. A single tensor or a list of tensors is transferred as-is; a nested structure has its tensor leaves extracted and shipped alongside a msgpack-encoded skeleton; anything else is msgpack-encoded into one uint8 tensor. Mapping and sequence structure is preserved. msgspec.Struct values, including OmniPayloadStruct, are normalised to mappings so their tensor leaves use the direct NIXL path rather than CPU object serialisation.

Each NIXL agent uses strict thread synchronization. A consumer performs one short metadata query per scheduler poll (10ms by default), which prevents an unavailable producer from blocking unrelated requests. The regular transfer-completion handshake keeps the configured transfer timeout.

Installation

CUDA hosts can install the published wheel:

pip install nixl

For Intel XPU, use docker/Dockerfile.xpu. Its upstream vLLM XPU base image provides a matching torch-xpu, UCX with Level Zero support, and NIXL runtime.

Configuration

connectors:
  nixl_connector:
    name: NixlConnector
    extra:
      host: "auto"
      zmq_port: 50061
      backends: ["UCX"]

stages:
  - stage_id: 0
    output_connectors:
      to_stage_1: nixl_connector

  - stage_id: 1
    input_connectors:
      from_stage_0: nixl_connector

Parameters:

• host: address the producer binds its handshake socket to ("auto" to detect).
• zmq_port: handshake port. If a producer omits it, an ephemeral port is opened for ownership claims and returned in put() metadata. The value in the deploy YAML is a base port. Purpose, replica, rank, and producer-stage offsets are added centrally, so colocated connectors and tensor-parallel workers receive distinct endpoints without one config entry per rank.
• sender_host / sender_zmq_port: consumer-side override naming the producer's handshake endpoint. Only needed when the consumer cannot learn it from metadata. In replicated deployments, the producer endpoint is carried with each request and selected at get() time; it is not mutable connector-wide state. Explicit sender ports are preserved, not recomputed using the receiver replica. Intermediate NIXL stages retain their incoming receiver configuration and a separate outgoing bind configuration, so one worker can receive and publish.
• backends: NIXL backends to register memory with. Defaults to ["UCX"].
• receive_device: forces where received tensors land. By default the consumer keeps the producer's device type but uses its own current device of that type.
• memory_type: overrides the NIXL memory type, otherwise DRAM for CPU tensors and VRAM for accelerator tensors.
• lease_seconds: how long a put() payload stays registered while waiting to be read (default 3600). The consumer reports completion, so this only bounds payloads nobody has claimed. An internal reaper releases only unclaimed registrations. VLLM_OMNI_NIXL_LEASE_S overrides it.
• transfer_timeout_s: how long a get() waits for its READ to complete (default 300). NIXL 1.3 has no transfer cancellation API, so a timed-out transfer's buffers and registrations remain owned by the connector until NIXL reports a terminal state; close() waits for that state before releasing them. VLLM_OMNI_NIXL_XFER_TIMEOUT_S overrides it.

Source ownership and failure limits

Claims and expiry use the same lock. Completion identifies both a payload generation and a unique READ claim; duplicate or stale ACKs cannot release a different reader or replacement payload. Replacing a claimed key is rejected. Terminal errors also release claims; timeout, unknown status, and active sibling transfers retain the complete destination bundle and source claim until terminal. Zero-element tensors retain their shape/dtype/skeleton positions but never create DMA descriptors or registrations.

A lost metadata response, lost completion ACK, or abandoned consumer can retain a claim indefinitely. NIXL 1.3 cannot prove remote cancellation, so neither TTL nor cleanup() frees those allocations. Producer close() rejects new work and retains its agent, listener and claimed allocations; a later close() finishes teardown after claims drain. Permanently abandoned claims remain until process exit. This favors memory safety over bounded shutdown/memory usage. Only trusted peers may access this unauthenticated control plane. Both endpoints must use the claim-aware protocol; rolling compatibility with older GET_META clients is not provided. Legacy externally-owned metadata without a generation is accepted only under the caller's lifetime guarantee, not managed as a leased allocation here.

Validation

The native CUDA smoke test requires NIXL and at least two visible CUDA devices:

UCX_RCACHE_MAX_UNRELEASED=1024 pytest -o addopts='' -q -s \
  tests/distributed/omni_connectors/test_nixl_connector_native.py

It starts independent producer and consumer processes, loads the real UCX backend, performs the ZMQ GET_META handshake and a native NIXL READ from GPU 0 to GPU 1, checks mixed CPU/GPU tensor leaves from an OmniPayloadStruct, sends XFER_DONE, and verifies that producer registrations are released. This validates same-host, cross-GPU operation. Inter-node UCX transport is supported by the design but was not validated by this test.

For more details, refer to the NIXL repository.