vllm-omni/recipes/nvidia/Cosmos3-Nano.md

Cosmos3-Nano

Text-to-image, text-to-video, and image-to-video serving

Summary

• Vendor: NVIDIA
• Model: nvidia/Cosmos3-Nano
• Task: Text-to-image (T2I), text-to-video (T2V), and image-to-video (I2V) generation
• Mode: Online serving with the OpenAI-compatible image/video APIs, plus offline generation via the Omni API
• Maintainer: Community

When to use this recipe

Use this recipe to deploy nvidia/Cosmos3-Nano for image and video generation. A single pipeline class (Cosmos3OmniDiffusersPipeline) serves all three modes; the mode is selected per request:

• T2I — POST /v1/images/generations (or a prompt carrying modalities=["image"]).
• T2V — POST /v1/videos/sync with num_frames > 1 and no reference image.
• I2V — POST /v1/videos/sync with a reference image (input_reference file upload, or image_reference JSON).

References

• Model card (authoritative usage + example assets): https://huggingface.co/nvidia/Cosmos3-Nano
• Example inputs/outputs live in the repo's assets/ (example_t2v_prompt.json, example_i2v_prompt.json, example_i2v_input.jpg, negative_prompt.json).
• Prompt upsampling (recommended for quality): the model expects JSON-upsampled structured prompts; see NVIDIA's cosmos-framework prompt-upsampling docs.
• Pipeline: vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py
• Smoke tests (canonical request formats): tests/e2e/accuracy/test_cosmos3_similarity.py

Hardware Support

GPU

1x H200 141GB / B300 (Online serving)

Environment

• OS: Ubuntu 22.04+
• Python: 3.12+
• Driver / runtime: NVIDIA CUDA environment
• vLLM version: match the repository requirements from your current checkout
• vLLM-Omni version or commit: use the commit you are deploying from

Command

Safety guardrails are on by default (NVIDIA Open Model License). They load the gated nvidia/Cosmos-1.0-Guardrail model, so to keep them on you must:

1. pip install cosmos-guardrail
2. Accept the license at https://huggingface.co/nvidia/Cosmos-1.0-Guardrail
3. Export a token with access: export HF_TOKEN=hf_...

Then launch the recommended server:

vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --host 0.0.0.0 --port 8000 \
  --init-timeout 1800

To run without guardrails (you are responsible for license compliance), add --no-guardrails (no token/cosmos-guardrail needed). For extra GPUs use --ulysses-degree N (context parallel) or --tensor-parallel-size N; --enable-layerwise-offload reduces VRAM on smaller GPUs. The pipeline auto-resolves from model_index.json; pass --model-class-name Cosmos3OmniDiffusersPipeline to force it explicitly.

Verification

Best quality uses the JSON-upsampled prompts from assets/ (download with hf download nvidia/Cosmos3-Nano assets/ --local-dir Cosmos3-Nano). Minimal self-contained examples:

curl http://localhost:8000/v1/models

# Text-to-image -> /v1/images/generations  (1024x1024, 50 steps; base64 PNG)
curl -sS -X POST http://localhost:8000/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Cosmos3-Nano",
    "prompt": "A photorealistic red sports car on a city street at golden hour, cinematic lighting.",
    "negative_prompt": "blurry, distorted, low quality",
    "size": "1024x1024", "n": 1, "response_format": "b64_json",
    "num_inference_steps": 50, "guidance_scale": 7.0, "seed": 42
  }' | python -c "import sys,json,base64; open('cosmos3_t2i.png','wb').write(base64.b64decode(json.load(sys.stdin)['data'][0]['b64_json']))"

# Text-to-video -> /v1/videos/sync  (720p, 189 frames @ 24fps; official params)
curl -sS -X POST http://localhost:8000/v1/videos/sync \
  -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Nano" \
  -F "prompt=A robot arm is cleaning a plate in the kitchen" \
  -F "negative_prompt=blurry, distorted, low quality, jittery, deformed" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" \
  -F "num_inference_steps=35" -F "guidance_scale=6.0" \
  -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -F "seed=123" \
  -o cosmos3_t2v.mp4

# Image-to-video -> /v1/videos/sync with an uploaded reference image
curl -sS -X POST http://localhost:8000/v1/videos/sync \
  -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Nano" \
  -F "prompt=The scene comes to life with smooth, natural motion." \
  -F "negative_prompt=blurry, distorted, low quality" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" \
  -F "num_inference_steps=35" -F "guidance_scale=6.0" \
  -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -F "seed=1111" \
  -F "input_reference=@/path/to/reference.jpg;type=image/jpeg" \
  -o cosmos3_i2v.mp4

Notes

• Measured latency (1x B300, bf16, guardrails off):
  • T2I 1024² — 10 / 25 / 50 steps → ~0.4 / 0.7 / 1.3 s
  • T2V 1280×720 @ 35 steps — 25 / 49 / 93 / 189 frames → ~7 / 15 / 33 / ~93 s
  • I2V 1280×720, 189 frames @ 35 steps → ~99 s
  • Guardrails-on overhead: ~8% on T2I, negligible on video.
• Memory: transformer ~17 GiB (bf16); peak ~46 GiB for 720p video on 1 GPU; full repo (transformer + Wan VAE + Qwen3-VL vision encoder + audio tokenizer) ~33 GB on disk.
• Determinism: identical seed reproduces identical output on the same hardware; outputs are not bit-identical across different GPU types.
• Supported sizes (per model card): 256p / 480p / 720p at 16:9, 4:3, 1:1, 3:4, 9:16. Defaults: T2I 1024², 50 steps, guidance 7.0; T2V/I2V 1280×720, 189 frames, 35 steps, guidance 6.0, flow_shift=10.0.
• Key flags / params: --no-guardrails (server) or extra_params={"guardrails":false} (per request) toggles safety; use_resolution_template / use_duration_template are off by default and only needed when not using upsampled prompts that already encode resolution/duration.
• Known limitations:
  • Guardrails-on requires cosmos-guardrail and access to the gated nvidia/Cosmos-1.0-Guardrail repo (accept license + HF_TOKEN); otherwise the server fails at pipeline build with a gated-repo / safety-checker error.
  • A guardrail-blocked prompt currently returns HTTP 500 ("Guardrail blocked prompt").
  • Video + audio, and action (policy / forward- / inverse-dynamics) modalities are not part of this integration yet.

1x GPU (Offline generation)

Environment

• OS: Ubuntu 22.04+
• Python: 3.12+
• Driver / runtime: NVIDIA CUDA environment
• vLLM-Omni version or commit: use the commit you are deploying from

Command

# cosmos3_offline.py  —  run with:  python cosmos3_offline.py
import torch
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams


def main():
    omni = Omni(
        model="nvidia/Cosmos3-Nano",
        model_class_name="Cosmos3OmniDiffusersPipeline",
        trust_remote_code=True,
        enforce_eager=True,
        # Keep guardrails on by installing cosmos-guardrail + gated-repo access;
        # this disables them for a quick local run.
        model_config={"guardrails": False},
    )
    gen = torch.Generator(device="cpu").manual_seed(42)

    # Text-to-image (modalities=["image"]). For T2V use modalities=["video"]
    # plus num_frames/fps; for I2V add multi_modal_data={"image": <PIL.Image>}.
    outputs = omni.generate(
        {
            "prompt": "A photorealistic red sports car at golden hour, cinematic lighting.",
            "negative_prompt": "blurry, distorted, low quality",
            "modalities": ["image"],
        },
        OmniDiffusionSamplingParams(
            height=1024, width=1024, generator=gen,
            guidance_scale=7.0, num_inference_steps=50, num_outputs_per_prompt=1,
        ),
    )
    outputs[0].request_output.images[0].save("cosmos3_t2i.png")
    omni.close()


if __name__ == "__main__":
    main()

Verification

python cosmos3_offline.py
python -c "from PIL import Image; im=Image.open('cosmos3_t2i.png'); print('image', im.size, im.mode)"

Notes

• Same Cosmos3OmniDiffusersPipeline as online; mode is chosen by prompt["modalities"] (["image"] → T2I, ["video"] → T2V) plus num_frames/fps, and multi_modal_data={"image": ...} for I2V. For video, frames are returned in outputs[0].request_output.images as an (B, F, H, W, 3) array.
• The offline entry must be guarded by if __name__ == "__main__": — the engine spawns workers with the spawn start method.