From 2bb618903afc9dc29a1f29578dcac31caa047ad5 Mon Sep 17 00:00:00 2001 From: SYLAR <125541396+lishunyang12@users.noreply.github.com> Date: Sun, 26 Jul 2026 17:25:00 +0800 Subject: [PATCH] [Recipe] Add Cosmos3-Edge (#5313) Signed-off-by: lishunyang12 Co-authored-by: lishunyang12 --- recipes/README.md | 1 + recipes/cosmos3/Cosmos3-Edge.md | 98 +++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+) create mode 100644 recipes/cosmos3/Cosmos3-Edge.md diff --git a/recipes/README.md b/recipes/README.md index 7129918e1..13c2320be 100644 --- a/recipes/README.md +++ b/recipes/README.md @@ -52,6 +52,7 @@ recipes/ | [`mistralai/Voxtral-TTS.md`](./mistralai/Voxtral-TTS.md) | Online serving for TTS | 1x RTX 4090 24GB | | [`Robbyant/LingBot-Video.md`](./Robbyant/LingBot-Video.md) | Native dense and MoE text-to-video serving | Dense: 1x L20X; MoE: 1x L20X (~67.7 GiB peak) | | [`cosmos3/Cosmos3-Nano.md`](./cosmos3/Cosmos3-Nano.md) | Text-to-image, text-to-video, image-to-video, video-to-video generation, text to video with sound, action policy | 1x H200 141GB / B300 | +| [`cosmos3/Cosmos3-Edge.md`](./cosmos3/Cosmos3-Edge.md) | T2I / T2V / I2V + action (Physical-AI) generation on the Nemotron-based Edge transformer | 1x GPU (~8 GiB) | | [`cosmos3/Cosmos3-Super.md`](./cosmos3/Cosmos3-Super.md) | 64B T2I / T2V / I2V / V2V generation (+ optional audio) / Action policy | 8x H200/H100/A100 / 2x H200 / B300 | | [`OpenBMB/MiniCPM-o-4_5.md`](./OpenBMB/MiniCPM-o-4_5.md) | Online serving for omni multimodal chat (text / image / audio / video → text + 24 kHz speech) | 2x A100/H100 80GB / 3x mid-tier GPU / 8x RTX 4090 24GB | | [`OpenBMB/VoxCPM2.md`](./OpenBMB/VoxCPM2.md) | Online + offline TTS with native AR pipeline (48 kHz, 30+ languages) | 1x RTX 4090 24GB | diff --git a/recipes/cosmos3/Cosmos3-Edge.md b/recipes/cosmos3/Cosmos3-Edge.md new file mode 100644 index 000000000..dc783236e --- /dev/null +++ b/recipes/cosmos3/Cosmos3-Edge.md @@ -0,0 +1,98 @@ +# Cosmos3-Edge + +> Compact Nemotron-based Cosmos3 checkpoint for T2I / T2V / I2V and action-conditioned (Physical-AI) generation. + +## Summary + +- Vendor: NVIDIA +- Model: `nvidia/Cosmos3-Edge` +- Task: T2I, T2V, I2V, and action-conditioned generation (Physical-AI: `policy` / `forward_dynamics` / `inverse_dynamics`) +- Mode: Online (OpenAI-compatible image/video APIs) plus offline generation +- Maintainer: Community + +## When to use + +The small Cosmos3 checkpoint, for constrained deployments. It runs on the shared +`Cosmos3OmniDiffusersPipeline` but differs from Nano/Super in two ways: + +- **Transformer**: `Cosmos3EdgeVFMTransformer` (Nemotron-based), auto-selected from + `backbone_type: cosmos3_edge_nemotron_dense` — nothing to pass. +- **Defaults** differ; omit `--height` / `--width` to get them: + +| | Edge | Nano / Super | +|---|---|---| +| T2I | 640×640 | 1024² | +| T2V / I2V | 480×832 | 1280×720 | +| Video `guidance_scale` | 5.0 | 6.0 | +| Video `flow_shift` | 3.0 | 10.0 | + +For the larger checkpoints see [Cosmos3-Nano](./Cosmos3-Nano.md) and [Cosmos3-Super](./Cosmos3-Super.md). + +## Support notes + +- **Rejected** (raise at request time): video-to-video, transfer V2V + (`edge` / `blur` / `depth` / `seg` / `wsm`), sound (`generate_sound=true`). +- **Acceleration**: `--quantization fp8` and attention-backend selection are supported. Cache-DiT and + Sequence Parallel are inherited from the base Cosmos3 transformer (Edge overrides neither) and act on + the shared GEN pathway — not separately benchmarked here. Tensor Parallel is untested on this checkpoint. + +## Usage + +Guardrails are on by default (gated `nvidia/Cosmos-1.0-Guardrail`; `pip install cosmos-guardrail` + +an `HF_TOKEN`). The examples pass `"guardrails": false` for a quick local run — you own license +compliance. + +### Offline + +```bash +# Text-to-image -> 640x640 +python examples/offline_inference/text_to_image/text_to_image.py \ + --model nvidia/Cosmos3-Edge \ + --prompt "A photorealistic red sports car at golden hour, cinematic lighting." \ + --extra-body '{"guardrails": false}' --output t2i.png + +# Text-to-video -> 480x832 (for I2V: use image_to_video.py + --image ) +python examples/offline_inference/text_to_video/text_to_video.py \ + --model nvidia/Cosmos3-Edge \ + --prompt "A robot arm is cleaning a plate in the kitchen." \ + --num-frames 49 \ + --extra-body '{"max_sequence_length": 4096, "guardrails": false}' --output t2v.mp4 +``` + +### Online + +```bash +vllm serve nvidia/Cosmos3-Edge --omni --host 0.0.0.0 --port 8000 --init-timeout 1800 +``` + +Add `--no-guardrails` to skip the safety checker. Request formats match the +[Cosmos3-Nano recipe](./Cosmos3-Nano.md#verification), minus the rejected modes above. + +### Action (Physical-AI) + +Action-conditioned generation uses the same shared pipeline — pass +`extra_params={"action_mode": ...}`: + +- `forward_dynamics` — first frame/video **plus** an action trajectory → roll out the video + (sync `POST /v1/videos/sync`). +- `policy` — first frame/video + a language instruction → **predict** the action trajectory + (async `POST /v1/videos`; read the top-level `action` field). +- `inverse_dynamics` — a video → **recover** the action trajectory (async `POST /v1/videos`). + +See the [Cosmos3-Nano action section](./Cosmos3-Nano.md) for request/response shapes. For a +DROID-specialized policy model, use `nvidia/Cosmos3-Edge-Policy-DROID`. + +## Verification + +```bash +python -c "from PIL import Image; im=Image.open('t2i.png'); print('image', im.size, im.mode)" +ffprobe -v error -select_streams v:0 -show_entries stream=codec_type,nb_frames,width,height -of csv=p=0 t2v.mp4 +``` + +Expected: `image (640, 640) RGB` and `video,49,832,480`. + +## References + +- Model card: +- Pipeline: [`pipeline_cosmos3.py`](../../vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py) +- Edge transformer: [`transformer_cosmos3_edge.py`](../../vllm_omni/diffusion/models/cosmos3/transformer_cosmos3_edge.py)