SYLARandlishunyang12
04b408ffd2
Signed-off-by: lishunyang12 <lishunyang03@gmail.com> Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
105 lines
4.3 KiB
Markdown
105 lines
4.3 KiB
Markdown
# Cosmos3-Edge
|
||
|
||
> Compact Nemotron-based Cosmos3 checkpoint for T2I / T2V / I2V and action-conditioned (Physical-AI) generation.
|
||
|
||
## Summary
|
||
|
||
- Vendor: NVIDIA
|
||
- Model: `nvidia/Cosmos3-Edge`
|
||
- Task: T2I, T2V, I2V, and action-conditioned generation (Physical-AI: `policy` / `forward_dynamics` / `inverse_dynamics`)
|
||
- Mode: Online (OpenAI-compatible image/video APIs) plus offline generation
|
||
- Maintainer: Community
|
||
|
||
## When to use
|
||
|
||
The small Cosmos3 checkpoint, for constrained deployments. It runs on the shared
|
||
`Cosmos3OmniDiffusersPipeline` but differs from Nano/Super in two ways:
|
||
|
||
- **Transformer**: `Cosmos3EdgeVFMTransformer` (Nemotron-based), auto-selected from
|
||
`backbone_type: cosmos3_edge_nemotron_dense` — nothing to pass.
|
||
- **Defaults** differ. The offline examples auto-detect Edge (from `edge` in the model id /
|
||
transformer class) and apply the Edge column below, so you can omit `--height` / `--width` /
|
||
`--guidance-scale` / `--flow-shift` and still get native Edge output:
|
||
|
||
| | Edge | Nano / Super |
|
||
|---|---|---|
|
||
| T2I | 640×640 | 1024² |
|
||
| T2V / I2V | 480×832 | 1280×720 |
|
||
| Video `guidance_scale` | 5.0 | 6.0 |
|
||
| Video `flow_shift` | 3.0 | 10.0 |
|
||
|
||
> **Important:** these auto-defaults live in the example CLIs
|
||
> (`text_to_video.py` / `image_to_video.py`). Do **not** pass the Nano/Super numbers
|
||
> (720p / `guidance_scale` 6.0 / `flow_shift` 10.0) to Edge — they produce degenerate output.
|
||
|
||
For the larger checkpoints see [Cosmos3-Nano](./Cosmos3-Nano.md) and [Cosmos3-Super](./Cosmos3-Super.md).
|
||
|
||
## Support notes
|
||
|
||
- **Rejected** (raise at request time): video-to-video, transfer V2V
|
||
(`edge` / `blur` / `depth` / `seg` / `wsm`), sound (`generate_sound=true`).
|
||
- **Acceleration**: `--quantization fp8` and attention-backend selection are supported. Cache-DiT and
|
||
Sequence Parallel are inherited from the base Cosmos3 transformer (Edge overrides neither) and act on
|
||
the shared GEN pathway — not separately benchmarked here. Tensor Parallel is untested on this checkpoint.
|
||
|
||
## Usage
|
||
|
||
Guardrails are on by default (gated `nvidia/Cosmos-1.0-Guardrail`; `pip install cosmos-guardrail` +
|
||
an `HF_TOKEN`). The examples pass `"guardrails": false` for a quick local run — you own license
|
||
compliance.
|
||
|
||
### Offline
|
||
|
||
```bash
|
||
# Text-to-image -> 640x640
|
||
python examples/offline_inference/text_to_image/text_to_image.py \
|
||
--model nvidia/Cosmos3-Edge \
|
||
--prompt "A photorealistic red sports car at golden hour, cinematic lighting." \
|
||
--extra-body '{"guardrails": false}' --output t2i.png
|
||
|
||
# Text-to-video -> 480x832 (for I2V: use image_to_video.py + --image <ref>)
|
||
python examples/offline_inference/text_to_video/text_to_video.py \
|
||
--model nvidia/Cosmos3-Edge \
|
||
--prompt "A robot arm is cleaning a plate in the kitchen." \
|
||
--num-frames 49 \
|
||
--extra-body '{"max_sequence_length": 4096, "guardrails": false}' --output t2v.mp4
|
||
```
|
||
|
||
### Online
|
||
|
||
```bash
|
||
vllm serve nvidia/Cosmos3-Edge --omni --host 0.0.0.0 --port 8000 --init-timeout 1800
|
||
```
|
||
|
||
Add `--no-guardrails` to skip the safety checker. Request formats match the
|
||
[Cosmos3-Nano recipe](./Cosmos3-Nano.md#verification), minus the rejected modes above.
|
||
|
||
### Action (Physical-AI)
|
||
|
||
Action-conditioned generation uses the same shared pipeline — pass
|
||
`extra_params={"action_mode": ...}`:
|
||
|
||
- `forward_dynamics` — first frame/video **plus** an action trajectory → roll out the video
|
||
(sync `POST /v1/videos/sync`).
|
||
- `policy` — first frame/video + a language instruction → **predict** the action trajectory
|
||
(async `POST /v1/videos`; read the top-level `action` field).
|
||
- `inverse_dynamics` — a video → **recover** the action trajectory (async `POST /v1/videos`).
|
||
|
||
See the [Cosmos3-Nano action section](./Cosmos3-Nano.md) for request/response shapes. For a
|
||
DROID-specialized policy model, use `nvidia/Cosmos3-Edge-Policy-DROID`.
|
||
|
||
## Verification
|
||
|
||
```bash
|
||
python -c "from PIL import Image; im=Image.open('t2i.png'); print('image', im.size, im.mode)"
|
||
ffprobe -v error -select_streams v:0 -show_entries stream=codec_type,nb_frames,width,height -of csv=p=0 t2v.mp4
|
||
```
|
||
|
||
Expected: `image (640, 640) RGB` and `video,49,832,480`.
|
||
|
||
## References
|
||
|
||
- Model card: <https://huggingface.co/nvidia/Cosmos3-Edge>
|
||
- Pipeline: [`pipeline_cosmos3.py`](../../vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py)
|
||
- Edge transformer: [`transformer_cosmos3_edge.py`](../../vllm_omni/diffusion/models/cosmos3/transformer_cosmos3_edge.py)
|