vllm-omni/docs/user_guide/diffusion/device_side_video_postprocess.md

Device-side video postprocessing

vLLM-Omni can convert decoded WAN2.2 video from normalized floating-point [B, C, T, H, W] tensors to contiguous uint8 [B, T, H, W, C] frames before the worker-to-engine device-to-host transfer. This reduces the transported video payload by four times when the original IPC path widens bfloat16 to float32.

The optimization is disabled by default. Enable it with a structured diffusion configuration:

from vllm_omni import Omni

engine = Omni(
    model="Wan-AI/Wan2.2-TI2V-5B-Diffusers",
    video_output_transport={"enable_device_postprocess": True},
)

For the server CLI, pass the same object as JSON:

vllm serve Wan-AI/Wan2.2-TI2V-5B-Diffusers --omni \
  --video-output-transport '{"enable_device_postprocess": true}'

For a deploy configuration, place it on the diffusion stage:

stages:
  - stage_id: 0
    video_output_transport:
      enable_device_postprocess: true

Runtime contract

WAN2.2 emits decoded video through the typed DiffusionOutput.media field. The model runner validates and splits the batch into request-local tensors before preparing each video for transport. The worker then performs the D2H copy and shared-memory IPC using the prepared representation. The engine uses the generic media finalizer instead of a model-specific postprocessor.

The runtime leaves the video in normalized floating-point form when:

• enable_device_postprocess is false;
• the requested output type is not np; or
• frame interpolation still requires the floating-point tensor.

These are policy fallbacks, not malformed-contract errors. If the device cannot allocate the temporary conversion buffer, the runner logs a warning and prepares the request-local float representation instead. Invalid tensor layout, encoding, value range, lifecycle state, and non-memory runtime failures still fail before IPC.

Precision

Device preparation converts to float32 before denormalization and quantization. For bfloat16 WAN output this is more precise than the historical host path, which denormalizes in bfloat16. Relative to that path, values may differ by at most one level out of 255. Float32 inputs remain byte-identical after quantization.

Scope

WAN2.2 is the reference migration for the first contract version. Other video pipelines continue using their existing output paths until they can declare the same typed media contract without bypassing model-specific float consumers such as safety checks or audio/video packaging.

See RFC #6541 for the contract, lifecycle, batching rules, and migration plan.

Experimental registered-SHM video transport

video_output_transport.enable_registered_shm is a separate, default-off option for CUDA request-mode workers. It copies eligible video tensors directly into CUDA-registered shared memory and returns borrowed CPU views, removing the intermediate pinned-to-SHM and SHM-to-consumer host copies. The D2H transfer still occurs.

video_output_transport.enable_borrowed_frames independently enables borrowed RGB AVFrames in the Videos API MP4 response encoder, removing the third host copy into PyAV frame buffers. It also defaults to false. Eligible uint8 RGB frames are wrapped with VideoFrame.from_numpy_buffer; other dtypes/layouts use the existing encoder path. Color conversion, compression and muxing still occur.

Enable it through the same CLI, Python or deploy-stage configuration:

--video-output-transport '{"enable_registered_shm": true, "enable_borrowed_frames": true}'

Registered SHM applies only to prepared typed video media (currently WAN2.2) and explicitly marked legacy video outputs (currently MiniMax-H3). H3 marks the video entry of its (video, audio) output; audio, trajectory data, images and unmarked legacy outputs keep their existing transport. Device-side postprocessing is independent and does not need to be enabled for registered SHM. Pre-encoded MP4 outputs bypass raw-frame transport and API-side encoding.

The existing SHM routing threshold is unchanged: the tensor or its backing storage must exceed 1,000,000 bytes. This is not a measured performance crossover. SHM is allocated and registered on every transfer, without pooling; small outputs can regress and registration latency can vary. Benchmark representative output sizes before opting in. CPU outputs, non-CUDA backends, and synchronous/step-mode transfers keep their existing path.

Borrowed views keep the unlinked mapping alive until their last user releases it. CUDA registration/copy failures fail the request; if CUDA cleanup cannot be confirmed, the worker retains the mapping for safety and must be restarted before further registered-SHM transfers.