# Cosmos3-Nano > Text-to-image, text-to-video, and image-to-video serving ## Summary - Vendor: NVIDIA - Model: `nvidia/Cosmos3-Nano` - Task: Text-to-image (T2I), text-to-video (T2V), and image-to-video (I2V) generation - Mode: Online serving with the OpenAI-compatible image/video APIs, plus offline generation via the `Omni` API - Maintainer: Community ## When to use this recipe Use this recipe to deploy `nvidia/Cosmos3-Nano` for image and video generation. A single pipeline class (`Cosmos3OmniDiffusersPipeline`) serves all three modes; the mode is selected per request: - **T2I** — `POST /v1/images/generations` (or a prompt carrying `modalities=["image"]`). - **T2V** — `POST /v1/videos/sync` with `num_frames > 1` and no reference image. - **I2V** — `POST /v1/videos/sync` with a reference image (`input_reference` file upload, or `image_reference` JSON). ## References - Model card (authoritative usage + example assets): - Example inputs/outputs live in the repo's `assets/` (`example_t2v_prompt.json`, `example_i2v_prompt.json`, `example_i2v_input.jpg`, `negative_prompt.json`). - Prompt upsampling (recommended for quality): the model expects JSON-upsampled structured prompts; see NVIDIA's `cosmos-framework` prompt-upsampling docs. - Pipeline: [`vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py`](../../vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py) - Smoke tests (canonical request formats): [`tests/e2e/accuracy/test_cosmos3_similarity.py`](../../tests/e2e/accuracy/test_cosmos3_similarity.py) ## Hardware Support ## GPU ### 1x H200 141GB / B300 (Online serving) #### Environment - OS: Ubuntu 22.04+ - Python: 3.12+ - Driver / runtime: NVIDIA CUDA environment - vLLM version: match the repository requirements from your current checkout - vLLM-Omni version or commit: use the commit you are deploying from #### Command Safety guardrails are **on by default** (NVIDIA Open Model License). They load the **gated** `nvidia/Cosmos-1.0-Guardrail` model, so to keep them on you must: 1. `pip install cosmos-guardrail` 2. Accept the license at 3. Export a token with access: `export HF_TOKEN=hf_...` Then launch the recommended server: ```bash vllm serve nvidia/Cosmos3-Nano \ --omni \ --host 0.0.0.0 --port 8000 \ --init-timeout 1800 ``` To run **without** guardrails (you are responsible for license compliance), add `--no-guardrails` (no token/`cosmos-guardrail` needed). For extra GPUs use `--ulysses-degree N` (context parallel) or `--tensor-parallel-size N`; `--enable-layerwise-offload` reduces VRAM on smaller GPUs. The pipeline auto-resolves from `model_index.json`; pass `--model-class-name Cosmos3OmniDiffusersPipeline` to force it explicitly. #### Verification Best quality uses the JSON-upsampled prompts from `assets/` (download with `hf download nvidia/Cosmos3-Nano assets/ --local-dir Cosmos3-Nano`). Minimal self-contained examples: ```bash curl http://localhost:8000/v1/models # Text-to-image -> /v1/images/generations (1024x1024, 50 steps; base64 PNG) curl -sS -X POST http://localhost:8000/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Cosmos3-Nano", "prompt": "A photorealistic red sports car on a city street at golden hour, cinematic lighting.", "negative_prompt": "blurry, distorted, low quality", "size": "1024x1024", "n": 1, "response_format": "b64_json", "num_inference_steps": 50, "guidance_scale": 7.0, "seed": 42 }' | python -c "import sys,json,base64; open('cosmos3_t2i.png','wb').write(base64.b64decode(json.load(sys.stdin)['data'][0]['b64_json']))" # Text-to-video -> /v1/videos/sync (720p, 189 frames @ 24fps; official params) curl -sS -X POST http://localhost:8000/v1/videos/sync \ -H "Accept: video/mp4" \ -F "model=nvidia/Cosmos3-Nano" \ -F "prompt=A robot arm is cleaning a plate in the kitchen" \ -F "negative_prompt=blurry, distorted, low quality, jittery, deformed" \ -F "size=1280x720" -F "num_frames=189" -F "fps=24" \ -F "num_inference_steps=35" -F "guidance_scale=6.0" \ -F "max_sequence_length=4096" -F "flow_shift=10.0" \ -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \ -F "seed=123" \ -o cosmos3_t2v.mp4 # Image-to-video -> /v1/videos/sync with an uploaded reference image curl -sS -X POST http://localhost:8000/v1/videos/sync \ -H "Accept: video/mp4" \ -F "model=nvidia/Cosmos3-Nano" \ -F "prompt=The scene comes to life with smooth, natural motion." \ -F "negative_prompt=blurry, distorted, low quality" \ -F "size=1280x720" -F "num_frames=189" -F "fps=24" \ -F "num_inference_steps=35" -F "guidance_scale=6.0" \ -F "max_sequence_length=4096" -F "flow_shift=10.0" \ -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \ -F "seed=1111" \ -F "input_reference=@/path/to/reference.jpg;type=image/jpeg" \ -o cosmos3_i2v.mp4 ``` #### Notes - **Measured latency (1x B300, bf16, guardrails off):** - T2I 1024² — 10 / 25 / 50 steps → ~0.4 / 0.7 / **1.3 s** - T2V 1280×720 @ 35 steps — 25 / 49 / 93 / **189** frames → ~7 / 15 / 33 / **~93 s** - I2V 1280×720, 189 frames @ 35 steps → ~**99 s** - Guardrails-on overhead: ~8% on T2I, negligible on video. - **Memory:** transformer ~17 GiB (bf16); peak ~46 GiB for 720p video on 1 GPU; full repo (transformer + Wan VAE + Qwen3-VL vision encoder + audio tokenizer) ~33 GB on disk. - **Determinism:** identical seed reproduces identical output on the same hardware; outputs are not bit-identical across different GPU types. - **Supported sizes (per model card):** 256p / 480p / 720p at 16:9, 4:3, 1:1, 3:4, 9:16. Defaults: T2I 1024², 50 steps, guidance 7.0; T2V/I2V 1280×720, 189 frames, 35 steps, guidance 6.0, `flow_shift=10.0`. - **Key flags / params:** `--no-guardrails` (server) or `extra_params={"guardrails":false}` (per request) toggles safety; `use_resolution_template` / `use_duration_template` are off by default and only needed when not using upsampled prompts that already encode resolution/duration. - **Known limitations:** - Guardrails-on requires `cosmos-guardrail` **and** access to the gated `nvidia/Cosmos-1.0-Guardrail` repo (accept license + `HF_TOKEN`); otherwise the server fails at pipeline build with a gated-repo / safety-checker error. - A guardrail-blocked prompt currently returns HTTP 500 (`"Guardrail blocked prompt"`). - Video + audio, and action (policy / forward- / inverse-dynamics) modalities are not part of this integration yet. ### 1x GPU (Offline generation) #### Environment - OS: Ubuntu 22.04+ - Python: 3.12+ - Driver / runtime: NVIDIA CUDA environment - vLLM-Omni version or commit: use the commit you are deploying from #### Command ```python # cosmos3_offline.py — run with: python cosmos3_offline.py import torch from vllm_omni.entrypoints.omni import Omni from vllm_omni.inputs.data import OmniDiffusionSamplingParams def main(): omni = Omni( model="nvidia/Cosmos3-Nano", model_class_name="Cosmos3OmniDiffusersPipeline", trust_remote_code=True, enforce_eager=True, # Keep guardrails on by installing cosmos-guardrail + gated-repo access; # this disables them for a quick local run. model_config={"guardrails": False}, ) gen = torch.Generator(device="cpu").manual_seed(42) # Text-to-image (modalities=["image"]). For T2V use modalities=["video"] # plus num_frames/fps; for I2V add multi_modal_data={"image": }. outputs = omni.generate( { "prompt": "A photorealistic red sports car at golden hour, cinematic lighting.", "negative_prompt": "blurry, distorted, low quality", "modalities": ["image"], }, OmniDiffusionSamplingParams( height=1024, width=1024, generator=gen, guidance_scale=7.0, num_inference_steps=50, num_outputs_per_prompt=1, ), ) outputs[0].request_output.images[0].save("cosmos3_t2i.png") omni.close() if __name__ == "__main__": main() ``` #### Verification ```bash python cosmos3_offline.py python -c "from PIL import Image; im=Image.open('cosmos3_t2i.png'); print('image', im.size, im.mode)" ``` #### Notes - Same `Cosmos3OmniDiffusersPipeline` as online; mode is chosen by `prompt["modalities"]` (`["image"]` → T2I, `["video"]` → T2V) plus `num_frames`/`fps`, and `multi_modal_data={"image": ...}` for I2V. For video, frames are returned in `outputs[0].request_output.images` as an `(B, F, H, W, 3)` array. - The offline entry must be guarded by `if __name__ == "__main__":` — the engine spawns workers with the `spawn` start method.