vllm-omni/recipes/cosmos3/Cosmos3-Edge.md

Cosmos3-Edge

Compact Nemotron-based Cosmos3 checkpoint for T2I / T2V / I2V and action-conditioned (Physical-AI) generation.

Summary

• Vendor: NVIDIA
• Model: nvidia/Cosmos3-Edge
• Task: T2I, T2V, I2V, and action-conditioned generation (Physical-AI: policy / forward_dynamics / inverse_dynamics)
• Mode: Online (OpenAI-compatible image/video APIs) plus offline generation
• Maintainer: Community

When to use

The small Cosmos3 checkpoint, for constrained deployments. It runs on the shared Cosmos3OmniDiffusersPipeline but differs from Nano/Super in two ways:

• Transformer: Cosmos3EdgeVFMTransformer (Nemotron-based), auto-selected from backbone_type: cosmos3_edge_nemotron_dense — nothing to pass.
• Defaults differ. The offline examples auto-detect Edge (from edge in the model id / transformer class) and apply the Edge column below, so you can omit --height / --width / --guidance-scale / --flow-shift and still get native Edge output:

	Edge	Nano / Super
T2I	640×640	1024²
T2V / I2V	480×832	1280×720
Video guidance_scale	5.0	6.0
Video flow_shift	3.0	10.0

Important: these auto-defaults live in the example CLIs (text_to_video.py / image_to_video.py). Do not pass the Nano/Super numbers (720p / guidance_scale 6.0 / flow_shift 10.0) to Edge — they produce degenerate output.

For the larger checkpoints see Cosmos3-Nano and Cosmos3-Super.

Support notes

• Rejected (raise at request time): video-to-video, transfer V2V (edge / blur / depth / seg / wsm), sound (generate_sound=true).
• Acceleration: --quantization fp8 and attention-backend selection are supported. Cache-DiT and Sequence Parallel are inherited from the base Cosmos3 transformer (Edge overrides neither) and act on the shared GEN pathway — not separately benchmarked here. Tensor Parallel is untested on this checkpoint.

Usage

Guardrails are on by default (gated nvidia/Cosmos-1.0-Guardrail; pip install cosmos-guardrail + an HF_TOKEN). The examples pass "guardrails": false for a quick local run — you own license compliance.

Offline

# Text-to-image -> 640x640
python examples/offline_inference/text_to_image/text_to_image.py \
  --model nvidia/Cosmos3-Edge \
  --prompt "A photorealistic red sports car at golden hour, cinematic lighting." \
  --extra-body '{"guardrails": false}' --output t2i.png

# Text-to-video -> 480x832  (for I2V: use image_to_video.py + --image <ref>)
python examples/offline_inference/text_to_video/text_to_video.py \
  --model nvidia/Cosmos3-Edge \
  --prompt "A robot arm is cleaning a plate in the kitchen." \
  --num-frames 49 \
  --extra-body '{"max_sequence_length": 4096, "guardrails": false}' --output t2v.mp4

Online

vllm serve nvidia/Cosmos3-Edge --omni --host 0.0.0.0 --port 8000 --init-timeout 1800

Add --no-guardrails to skip the safety checker. Request formats match the Cosmos3-Nano recipe, minus the rejected modes above.

Action (Physical-AI)

Action-conditioned generation uses the same shared pipeline — pass extra_params={"action_mode": ...}:

• forward_dynamics — first frame/video plus an action trajectory → roll out the video (sync POST /v1/videos/sync).
• policy — first frame/video + a language instruction → predict the action trajectory (async POST /v1/videos; read the top-level action field).
• inverse_dynamics — a video → recover the action trajectory (async POST /v1/videos).

See the Cosmos3-Nano action section for request/response shapes. For a DROID-specialized policy model, use nvidia/Cosmos3-Edge-Policy-DROID.

Verification

python -c "from PIL import Image; im=Image.open('t2i.png'); print('image', im.size, im.mode)"
ffprobe -v error -select_streams v:0 -show_entries stream=codec_type,nb_frames,width,height -of csv=p=0 t2v.mp4

Expected: image (640, 640) RGB and video,49,832,480.

References

• Model card: https://huggingface.co/nvidia/Cosmos3-Edge
• Pipeline: pipeline_cosmos3.py
• Edge transformer: transformer_cosmos3_edge.py