vllm-omni/recipes/cosmos3/Cosmos3-Super.md

# Cosmos3-Super

> Frontier 64B world model: text-to-image, text-to-video, image-to-video, video-to-video (+ optional audio)

## Summary

- Vendor: NVIDIA
- Model: `nvidia/Cosmos3-Super` (64B; also `Cosmos3-Super-Text2Image`, `Cosmos3-Super-Image2Video`)
- Task: T2I, T2V, I2V, V2V generation, with optional synchronized audio (video + sound)
- Mode: Online serving with the OpenAI-compatible image/video APIs
- Maintainer: Community

## When to use this recipe

Use this recipe to deploy the 64B `nvidia/Cosmos3-Super` for the highest-quality
Cosmos3 generation. It shares the same `Cosmos3OmniDiffusersPipeline` and request
formats as [Cosmos3-Nano](./Cosmos3-Nano.md) — only the checkpoint size and the
recommended parallelism differ. Mode is selected per request (T2I →
`/v1/images/generations`; T2V/I2V/V2V → `/v1/videos/sync`; add
`generate_sound=true` for audio).

## References

- Model card (authoritative usage + example assets): <https://huggingface.co/nvidia/Cosmos3-Super>
- Nano recipe (same APIs/params): [`Cosmos3-Nano.md`](./Cosmos3-Nano.md)
- Pipeline: [`vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py`](../../vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py)

## Hardware Support

## GPU

Requires the `vllm-omni` package (or the `vllm/vllm-omni:cosmos3` container),
which provides the `vllm serve … --omni` entrypoint used below.

### 8x H200/H100/A100 (recommended, per model card)

```bash
vllm serve nvidia/Cosmos3-Super \
  --omni \
  --host 0.0.0.0 --port 8000 \
  --cfg-parallel-size 2 \
  --ulysses-degree 4 \
  --use-hsdp --hsdp-shard-size 8 \
  --init-timeout 1800
```

### 2x H200 / B300 (minimum)

```bash
vllm serve nvidia/Cosmos3-Super \
  --omni \
  --host 0.0.0.0 --port 8000 \
  --cfg-parallel-size 2 \
  --use-hsdp --hsdp-shard-size 2 \
  --init-timeout 1800
```

Guardrails are on by default (gated `nvidia/Cosmos-1.0-Guardrail` — `pip install
cosmos-guardrail`, accept the license, set `HF_TOKEN`); add `--no-guardrails` to
disable, or pass
[`vllm_omni/deploy/cosmos3_omni.yaml`](../../vllm_omni/deploy/cosmos3_omni.yaml)
via `--deploy-config` on either multi-GPU command above (opt-in omni deploy
overlay, `pipeline: cosmos3_omni_deploy`; without that `pipeline:` key
`--deploy-config` cannot apply for Cosmos3). Same YAML works for Super / Nano
(T2I and video); topology `final_output_type` matches the CLI default (`video`).
`--enable-layerwise-offload` reduces VRAM on smaller GPUs;
`--vae-fast-path channels_last` speeds up the Wan VAE video decode by switching the
decoder convolutions to channels-last kernels (output no longer bit-identical to
diffusers; the default `lossless` fast path is bit-exact, see
[Wan VAE Decoder Fast Path](../../docs/user_guide/diffusion/vae_fast_path.md));
`--quantization fp8` (online, no calibration) cuts peak VRAM for 720p video
generation from ~83 GB to ~55 GB per GPU (2-GPU) with BF16-level quality (T2V
composition can shift at the same seed).

SeaCache is the recommended default choice when opting into diffusion caching,
which remains opt-in. Add `--cache-backend sea_cache` to either command above;
its defaults are tuned for Cosmos3.

Override individual defaults with `--cache-config`; for example:

```bash
--cache-config '{"sea_threshold":0.2,"sea_max_consecutive_cached":3}'
```

Lower `sea_threshold` values and smaller `sea_max_consecutive_cached` caps are
more conservative. Higher values allow more cached steps and may improve
speed, but can increase quality loss. Setting `sea_max_consecutive_cached` to
`0` removes the streak cap.

#### Verification

Requests are identical to Nano (see [`Cosmos3-Nano.md`](./Cosmos3-Nano.md) for full
T2I/T2V/I2V/V2V/T2VS curls); official params: `size=1280x720, num_frames=189,
fps=24, num_inference_steps=35, guidance_scale=6.0, flow_shift=10.0,
max_sequence_length=4096`.

```bash
curl http://localhost:8000/v1/models
# T2V (official prompt assets give best quality)
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=A robot arm is cleaning a plate in the kitchen" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -F "seed=17" -o cosmos3_super_t2v.mp4

# I2V — add an uploaded reference image
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=The scene comes to life with smooth, natural motion." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -F "seed=1111" -F "input_reference=@/path/to/reference.jpg;type=image/jpeg" \
  -o cosmos3_super_i2v.mp4

# V2V — add an uploaded reference video. condition_video_keep can be "first" or "last".
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=Continue the same scene with smooth natural motion." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"condition_frame_indexes_vision":[0,1],"condition_video_keep":"first"}' \
  -F "seed=2222" -F "input_reference=@/path/to/reference.mp4;type=video/mp4" \
  -o cosmos3_super_v2v.mp4

# T2V + sound — add generate_sound/sound_duration (output muxes AAC 48 kHz stereo)
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=A robot arm is cleaning a plate in the kitchen" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F "generate_sound=true" -F "sound_duration=7.875" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -F "seed=17" -o cosmos3_super_t2vs.mp4
```

#### Notes

- **Measured (2x B300, bf16, guardrails off, diffusion cache off, official 2-GPU config above):**
    - T2I 1024², 50 steps → **~6 s**
    - T2V 1280×720, 189 frames, 35 steps → **~197 s**
    - I2V 1280×720, 189 frames, 35 steps → **~200 s**
    - T2V + sound (189 frames, 35 steps) → **~198 s**, output muxes **AAC 48 kHz stereo**
    - (NVIDIA's reference: 8×H200 @ 50 steps ≈ 55 s/video; 2×H200 @ 35 steps ≈ 3 min/video.)
- **Memory:** ~61.5 GiB per GPU when sharded across 2 GPUs (HSDP shard 2); repo ~135 GB on disk.
- Same generation defaults, supported sizes, V2V reference-video controls
  (`condition_frame_indexes_vision`, `condition_video_keep`), and
  `generate_sound`/`sound_duration`
  semantics as Nano, including the **action** modality: `forward_dynamics`,
  `policy`, and `inverse_dynamics` — see the Cosmos3-Nano recipe for the request
  shapes. Use async `/v1/videos` when you need predicted/recovered action metadata
  under the top-level `action` field. Verified on the 64B Super under
  `--cfg-parallel-size 2`: async `policy` returns the predicted action (`[16, 10]`)
  and the rollout video reliably.
- **Transfer controls:** same semantics as Nano — `extra_params` may include
  `edge`, `blur`, `depth`, `seg`, or `wsm` hints; see the Cosmos3-Nano recipe
  for the request shapes and full option list. Every hint accepts a
  non-negative `control_weight`; weights are normalized across active controls
  and therefore only set their relative influence. A single positive weight
  always normalizes to `1.0`; use `control_guidance` to change the absolute
  strength of a single control. With two or more active controls, the
  per-control attention passes run replicated on every sequence-parallel
  (Ulysses) rank, so Ulysses does not reduce per-rank memory or latency for
  multi-control transfer requests. Transfer always uses Cosmos3's
  transfer-specific system prompt, appends a control-adherence directive by
  default (`emphasize_control_in_prompt: false` disables it), and enables
  duration/FPS and resolution metadata on both CFG branches
  (`use_duration_template` / `use_resolution_template` to disable;
  `negative_metadata_mode` accepts `same`, `inverse`, or `none`, defaulting to
  `same`). Transfer does not add a negative prompt automatically; an optional
  reference prompt is provided in [`negative_prompt.json`](negative_prompt.json).

## NPU

### 8× Ascend910 (A2, A3)

#### Environment

- OS: Linux
- Python: 3.10+
- Driver / runtime: Ascend NPU driver with CANN toolkit
- Recommended operator library: **mindie-sd** (Ascend high-performance fused
  operators — enables `adalayernorm` and other fused kernels automatically upon
  installation)
- vLLM version: Match the repository requirements for your checkout
- vLLM-Omni version or commit: Use the commit you are deploying from

A pre-built Docker image is available on
[Docker Hub](https://hub.docker.com/r/vllm/vllm-omni) and
[Quay.io](https://quay.io/ascend/vllm-omni). Ensure the image tag matches your
vLLM-Omni checkout so that NPU-specific code is in sync with the container.

#### Prerequisites

Install the **mindie-sd** operator library to enable Ascend-optimized fused
operators (`adalayernorm`, etc.):

```bash
git clone https://gitcode.com/Ascend/MindIE-SD.git && cd MindIE-SD

# Comment out the tik_ops build step (not needed for this use case)
sed -i 's|^\(\s*\)source ${current_script_dir}/build_tik_ops.sh|\1# source ${current_script_dir}/build_tik_ops.sh|' build/build_ops.sh

python setup.py bdist_wheel
cd dist
pip install mindiesd-*.whl
```

After installation, enable the Laser Attention kernel for significant
long-sequence speedups:

```bash
export MINDIE_SD_FA_TYPE=ascend_laser_attention
```

#### Command

```bash
export MINDIE_SD_FA_TYPE=ascend_laser_attention

vllm serve nvidia/Cosmos3-Super \
  --omni \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 8 \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --no-guardrails \
  --init-timeout 1800
```

#### Verification

Same requests as the GPU section above — all modes (T2I, T2V, I2V, V2V,
T2VS, I2VS) work identically on NPU. Quick reference with
`--no-guardrails`:

```bash
curl http://localhost:8000/v1/models

# T2I (1024x1024, 50 steps)
curl -sS -X POST http://localhost:8000/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Cosmos3-Super",
    "prompt": "A photorealistic red sports car on a city street at golden hour, cinematic lighting.",
    "size": "1024x1024", "n": 1, "response_format": "b64_json",
    "num_inference_steps": 50, "guidance_scale": 7.0, "seed": 42
  }' | python3 -c "import sys,json,base64; open('cosmos3_super_t2i.png','wb').write(base64.b64decode(json.load(sys.stdin)['data'][0]['b64_json']))"

# T2V (1280×720, 189 frames, 35 steps — official params)
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=A robot arm is cleaning a plate in the kitchen" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":false}' \
  -F "seed=17" -o cosmos3_super_t2v.mp4

# I2V — add an uploaded reference image
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=The scene comes to life with smooth, natural motion." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":false}' \
  -F "seed=1111" -F "input_reference=@/path/to/reference.jpg;type=image/jpeg" \
  -o cosmos3_super_i2v.mp4

# V2V — add an uploaded reference video. condition_video_keep can be "first" or "last".
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=Continue the same scene with smooth natural motion." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F 'extra_params={"condition_frame_indexes_vision":[0,1],"condition_video_keep":"first","guardrails":false}' \
  -F "seed=2222" -F "input_reference=@/path/to/reference.mp4;type=video/mp4" \
  -o cosmos3_super_v2v.mp4

# T2V + sound — add generate_sound/sound_duration (output muxes AAC 48 kHz stereo)
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=A robot arm is cleaning a plate in the kitchen" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F "generate_sound=true" -F "sound_duration=7.875" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":false}' \
  -F "seed=17" -o cosmos3_super_t2vs.mp4

# I2V + sound — reference image with synchronized audio
curl -sS -X POST http://localhost:8000/v1/videos/sync -H "Accept: video/mp4" \
  -F "model=nvidia/Cosmos3-Super" -F "prompt=The scene comes to life with smooth, natural motion and ambient sound." \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
  -F "guidance_scale=6.0" -F "max_sequence_length=4096" -F "flow_shift=10.0" \
  -F "generate_sound=true" -F "sound_duration=7.875" \
  -F 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":false}' \
  -F "seed=1111" -F "input_reference=@/path/to/reference.jpg;type=image/jpeg" \
  -o cosmos3_super_i2vs.mp4
```

#### Notes

- **Parallelism:** `--tensor-parallel-size 8` matches the model's 8 KV heads (`num_key_value_heads: 8`, `num_attention_heads: 64` → GQA). This uses 8 davinci devices (NPU 0–3, both cores each). The remaining NPU 4–7 stay idle.
- **Model config:** 64 layers × 5120 hidden size × 64 attention heads, ~120 GB transformer on disk (27 shards). Each NPU device loads ~15 GB of model weights.
- **Peak HBM:** ~22.7 GB per device at startup (bf16 weights). Additional memory is allocated per-generation for KV cache and activations — resolution and frame count drive the peak.
- **Performance (verified on 8× Ascend910):** T2I 256² / 2 steps / guidance 1.0 → ~1.5 s.
- **Guardrails** are disabled with `--no-guardrails` (guards are on by default). The gated `nvidia/Cosmos-1.0-Guardrail` model and `cosmos-guardrail` package are not shipped. Add `guardrails: false` in `extra_params` for per-request overrides when the server has guardrails enabled.
- **Known limitations:** FP8 quantization not yet validated on Ascend NPU. `--enable-layerwise-offload` is available but untested on NPU.