vllm-omni/recipes/cosmos3/Cosmos3-Super.md

Cosmos3-Super

Frontier 64B world model: text-to-image, text-to-video, image-to-video, video-to-video (+ optional audio)

Summary

• Vendor: NVIDIA
• Model: nvidia/Cosmos3-Super (64B; also Cosmos3-Super-Text2Image, Cosmos3-Super-Image2Video)
• Task: T2I, T2V, I2V, V2V generation, with optional synchronized audio (video + sound)
• Mode: Online serving with the OpenAI-compatible image/video APIs
• Maintainer: Community

When to use this recipe

Use this recipe to deploy the 64B nvidia/Cosmos3-Super for the highest-quality Cosmos3 generation. It shares the same Cosmos3OmniDiffusersPipeline and request formats as Cosmos3-Nano — only the checkpoint size and the recommended parallelism differ. Mode is selected per request (T2I → /v1/images/generations; T2V/I2V/V2V → /v1/videos/sync; add generate_sound=true for audio).

References

• Model card (authoritative usage + example assets): https://huggingface.co/nvidia/Cosmos3-Super
• Nano recipe (same APIs/params): Cosmos3-Nano.md
• Pipeline: vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py

Hardware Support

GPU

Requires the vllm-omni package (or the vllm/vllm-omni:cosmos3 container), which provides the vllm serve … --omni entrypoint used below.

8x H200/H100/A100 (recommended, per model card)

vllm serve nvidia/Cosmos3-Super \
  --omni \
  --host 0.0.0.0 --port 8000 \
  --cfg-parallel-size 2 \
  --ulysses-degree 4 \
  --use-hsdp --hsdp-shard-size 8 \
  --init-timeout 1800

2x H200 / B300 (minimum)

vllm serve nvidia/Cosmos3-Super \
  --omni \
  --host 0.0.0.0 --port 8000 \
  --cfg-parallel-size 2 \
  --use-hsdp --hsdp-shard-size 2 \
  --init-timeout 1800

Guardrails are on by default (gated nvidia/Cosmos-1.0-Guardrail — pip install cosmos-guardrail, accept the license, set HF_TOKEN); add --no-guardrails to disable, or pass vllm_omni/deploy/cosmos3_omni.yaml via --deploy-config on either multi-GPU command above (opt-in omni deploy overlay, pipeline: cosmos3_omni_deploy; without that pipeline: key --deploy-config cannot apply for Cosmos3). Same YAML works for Super / Nano (T2I and video); topology final_output_type matches the CLI default (video). --enable-layerwise-offload reduces VRAM on smaller GPUs; --vae-fast-path channels_last speeds up the Wan VAE video decode by switching the decoder convolutions to channels-last kernels (output no longer bit-identical to diffusers; the default lossless fast path is bit-exact, see Wan VAE Decoder Fast Path); --quantization fp8 (online, no calibration) cuts peak VRAM for 720p video generation from ~83 GB to ~55 GB per GPU (2-GPU) with BF16-level quality (T2V composition can shift at the same seed).

SeaCache is the recommended default choice when opting into diffusion caching, which remains opt-in. Add --cache-backend sea_cache to either command above; its defaults are tuned for Cosmos3.

Override individual defaults with --cache-config; for example:

--cache-config '{"sea_threshold":0.2,"sea_max_consecutive_cached":3}'

Lower sea_threshold values and smaller sea_max_consecutive_cached caps are more conservative. Higher values allow more cached steps and may improve speed, but can increase quality loss. Setting sea_max_consecutive_cached to 0 removes the streak cap.

Verification

Requests are identical to Nano (see Cosmos3-Nano.md for full T2I/T2V/I2V/V2V/T2VS curls); official params: size=1280x720, num_frames=189, fps=24, num_inference_steps=35, guidance_scale=6.0, flow_shift=10.0, max_sequence_length=4096.

curl http://localhost:8000/v1/models
# T2V (official prompt assets give best quality)
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=A robot arm is cleaning a plate in the kitchen" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -F "seed=17" -o cosmos3_super_t2v.mp4

# I2V — add an uploaded reference image
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=The scene comes to life with smooth, natural motion." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -F "seed=1111" -F "input_reference=@/path/to/reference.jpg;type=image/jpeg" \
  -o cosmos3_super_i2v.mp4

# V2V — add an uploaded reference video. condition_video_keep can be "first" or "last".
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=Continue the same scene with smooth natural motion." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"condition_frame_indexes_vision":[0,1],"condition_video_keep":"first"}' \
  -F "seed=2222" -F "input_reference=@/path/to/reference.mp4;type=video/mp4" \
  -o cosmos3_super_v2v.mp4

# T2V + sound — add generate_sound/sound_duration (output muxes AAC 48 kHz stereo)
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=A robot arm is cleaning a plate in the kitchen" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F "generate_sound=true" -F "sound_duration=7.875" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -F "seed=17" -o cosmos3_super_t2vs.mp4

Notes

• Measured (2x B300, bf16, guardrails off, diffusion cache off, official 2-GPU config above):
  • T2I 1024², 50 steps → ~6 s
  • T2V 1280×720, 189 frames, 35 steps → ~197 s
  • I2V 1280×720, 189 frames, 35 steps → ~200 s
  • T2V + sound (189 frames, 35 steps) → ~198 s, output muxes AAC 48 kHz stereo
  • (NVIDIA's reference: 8×H200 @ 50 steps ≈ 55 s/video; 2×H200 @ 35 steps ≈ 3 min/video.)
• Memory: ~61.5 GiB per GPU when sharded across 2 GPUs (HSDP shard 2); repo ~135 GB on disk.
• Same generation defaults, supported sizes, V2V reference-video controls (condition_frame_indexes_vision, condition_video_keep), and generate_sound/sound_duration semantics as Nano, including the action modality: forward_dynamics, policy, and inverse_dynamics — see the Cosmos3-Nano recipe for the request shapes. Use async /v1/videos when you need predicted/recovered action metadata under the top-level action field. Verified on the 64B Super under --cfg-parallel-size 2: async policy returns the predicted action ([16, 10]) and the rollout video reliably.
• Transfer controls: same semantics as Nano — extra_params may include edge, blur, depth, seg, or wsm hints; see the Cosmos3-Nano recipe for the request shapes and full option list. Every hint accepts a non-negative control_weight; weights are normalized across active controls and therefore only set their relative influence. A single positive weight always normalizes to 1.0; use control_guidance to change the absolute strength of a single control. With two or more active controls, the per-control attention passes run replicated on every sequence-parallel (Ulysses) rank, so Ulysses does not reduce per-rank memory or latency for multi-control transfer requests. Transfer always uses Cosmos3's transfer-specific system prompt, appends a control-adherence directive by default (emphasize_control_in_prompt: false disables it), and enables duration/FPS and resolution metadata on both CFG branches (use_duration_template / use_resolution_template to disable; negative_metadata_mode accepts same, inverse, or none, defaulting to same). Transfer does not add a negative prompt automatically; an optional reference prompt is provided in negative_prompt.json.

NPU

8× Ascend910 (A2, A3)

Environment

• OS: Linux
• Python: 3.10+
• Driver / runtime: Ascend NPU driver with CANN toolkit
• Recommended operator library: mindie-sd (Ascend high-performance fused operators — enables adalayernorm and other fused kernels automatically upon installation)
• vLLM version: Match the repository requirements for your checkout
• vLLM-Omni version or commit: Use the commit you are deploying from

A pre-built Docker image is available on Docker Hub and Quay.io. Ensure the image tag matches your vLLM-Omni checkout so that NPU-specific code is in sync with the container.

Prerequisites

Install the mindie-sd operator library to enable Ascend-optimized fused operators (adalayernorm, etc.):

git clone https://gitcode.com/Ascend/MindIE-SD.git && cd MindIE-SD

# Comment out the tik_ops build step (not needed for this use case)
sed -i 's|^\(\s*\)source ${current_script_dir}/build_tik_ops.sh|\1# source ${current_script_dir}/build_tik_ops.sh|' build/build_ops.sh

python setup.py bdist_wheel
cd dist
pip install mindiesd-*.whl

After installation, enable the Laser Attention kernel for significant long-sequence speedups:

export MINDIE_SD_FA_TYPE=ascend_laser_attention

Command

export MINDIE_SD_FA_TYPE=ascend_laser_attention

vllm serve nvidia/Cosmos3-Super \
  --omni \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 8 \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --no-guardrails \
  --init-timeout 1800

Verification

Same requests as the GPU section above — all modes (T2I, T2V, I2V, V2V, T2VS, I2VS) work identically on NPU. Quick reference with --no-guardrails:

curl http://localhost:8000/v1/models

# T2I (1024x1024, 50 steps)
curl -sS -X POST http://localhost:8000/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Cosmos3-Super",
    "prompt": "A photorealistic red sports car on a city street at golden hour, cinematic lighting.",
    "size": "1024x1024", "n": 1, "response_format": "b64_json",
    "num_inference_steps": 50, "guidance_scale": 7.0, "seed": 42
  }' | python3 -c "import sys,json,base64; open('cosmos3_super_t2i.png','wb').write(base64.b64decode(json.load(sys.stdin)['data'][0]['b64_json']))"

# T2V (1280×720, 189 frames, 35 steps — official params)
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=A robot arm is cleaning a plate in the kitchen" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":false}' \
  -F "seed=17" -o cosmos3_super_t2v.mp4

# I2V — add an uploaded reference image
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=The scene comes to life with smooth, natural motion." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":false}' \
  -F "seed=1111" -F "input_reference=@/path/to/reference.jpg;type=image/jpeg" \
  -o cosmos3_super_i2v.mp4

# V2V — add an uploaded reference video. condition_video_keep can be "first" or "last".
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=Continue the same scene with smooth natural motion." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"condition_frame_indexes_vision":[0,1],"condition_video_keep":"first","guardrails":false}' \
  -F "seed=2222" -F "input_reference=@/path/to/reference.mp4;type=video/mp4" \
  -o cosmos3_super_v2v.mp4

# T2V + sound — add generate_sound/sound_duration (output muxes AAC 48 kHz stereo)
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=A robot arm is cleaning a plate in the kitchen" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F "generate_sound=true" -F "sound_duration=7.875" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":false}' \
  -F "seed=17" -o cosmos3_super_t2vs.mp4

# I2V + sound — reference image with synchronized audio
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=The scene comes to life with smooth, natural motion and ambient sound." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F "generate_sound=true" -F "sound_duration=7.875" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":false}' \
  -F "seed=1111" -F "input_reference=@/path/to/reference.jpg;type=image/jpeg" \
  -o cosmos3_super_i2vs.mp4

Notes

• Parallelism: --tensor-parallel-size 8 matches the model's 8 KV heads (num_key_value_heads: 8, num_attention_heads: 64 → GQA). This uses 8 davinci devices (NPU 0–3, both cores each). The remaining NPU 4–7 stay idle.
• Model config: 64 layers × 5120 hidden size × 64 attention heads, ~120 GB transformer on disk (27 shards). Each NPU device loads ~15 GB of model weights.
• Peak HBM: ~22.7 GB per device at startup (bf16 weights). Additional memory is allocated per-generation for KV cache and activations — resolution and frame count drive the peak.
• Performance (verified on 8× Ascend910): T2I 256² / 2 steps / guidance 1.0 → ~1.5 s.
• Guardrails are disabled with --no-guardrails (guards are on by default). The gated nvidia/Cosmos-1.0-Guardrail model and cosmos-guardrail package are not shipped. Add guardrails: false in extra_params for per-request overrides when the server has guardrails enabled.
• Known limitations: FP8 quantization not yet validated on Ascend NPU. --enable-layerwise-offload is available but untested on NPU.