vllm-omni/recipes/NVIDIA/SANA-WM.md

SANA-WM

Camera-controllable first-frame image-to-video world model.

Summary

• Vendor: Efficient-Large-Model / NVlabs SANA
• Model: BBBBruce/SANA-WM_bidirectional-stage1-diffusers (standard diffusers layout, converted offline from the NVlabs release; Stage-1 transformer + VAE only)
• Task: First-frame image-to-video generation with camera control
• Mode: Online serving with the OpenAI-compatible video API
• Model weights: about 13 GB for the Stage-1 transformer (10 GB) and VAE (2.3 GB)
• Local disk: reserve about 40 GB for the Hugging Face cache and runtime artifacts
• Recommended GPU: 24 GB or larger CUDA GPU
• Maintainer: Community

When to use this recipe

Use this recipe when you want to serve SANA-WM through /v1/videos or /v1/videos/sync. The model takes a text prompt, a first-frame image, and either an action DSL string or explicit camera poses. vLLM-Omni serves the SANA-WM Stage-1 DiT, decoded through the SANA VAE. The optional LTX-2 refiner stage is not supported by this integration; it is a planned follow-up.

References

• Upstream model card: https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional
• Video API: docs/serving/videos_api.md

Hardware Support

GPU

1x NVIDIA RTX PRO 6000 Blackwell 96GB

Capacity

• Model storage: the Stage-1 transformer is about 10 GB and the VAE about 2.3 GB. The Gemma text encoder is a separate 4.9 GB repo. The engine prefetches the whole model repo at startup (allow_patterns=["*"]), which is why the Stage-1 weights live in their own repo — the two-stage one carries an additional 84 GB refiner/ that this path never loads.
• Text encoder: the pipeline tries google/gemma-2-2b-it first, then falls back to the ungated mirror Efficient-Large-Model/gemma-2-2b-it. The first repo is gated, so without an accepted licence and hf auth login you will see one failed load before the fallback succeeds — that is expected, not an error to chase. Set VLLM_OMNI_SANA_WM_STAGE1_TEXT_ENCODER to pin a specific repo or a local path and skip both.
• Disk sizing: provision about 40 GB of local disk or Hugging Face cache volume so the model, temporary downloads, and generated artifacts fit without cache eviction.
• GPU sizing: the default 1280x704, 161-frame, 60-step serving profile peaks at 22.6 GB of device memory and takes about 133 s to generate on one RTX PRO 6000 Blackwell. The peak lands in the VAE decode, which is why the pipeline forces VAE tiling on regardless of vae_use_tiling — without it the same request costs about 9 GB more, and a 321-frame one OOMs outright. On smaller GPUs, lower width, height, or num_frames before serving production requests.

Environment

• OS: Linux
• Python: 3.10+
• Driver / runtime: NVIDIA driver with CUDA runtime supported by your PyTorch build
• Recommended operator library: Triton, installed through the vLLM/vLLM-Omni Python environment
• vLLM version: Match the repository requirements for your checkout
• vLLM-Omni version or commit: Use the commit you are deploying from

Command

The repo ships the standard Diffusers layout (model_index.json + transformer/, vae/), and its model_index.json names SanaWmPipeline, so the pipeline class resolves on its own.

CUDA_VISIBLE_DEVICES=0 \
vllm serve BBBBruce/SANA-WM_bidirectional-stage1-diffusers \
  --omni \
  --host 0.0.0.0 \
  --port 8091

No deploy config: single-stage diffusion models are deliberately absent from OMNI_PIPELINES (vllm_omni/config/pipeline_registry.py), so stage resolution falls back to the default stage config and a YAML's stage settings — including default_sampling_params — would not be applied. The production generation settings therefore live in the model (num_inference_steps=60, guidance_scale=5.0); a request that omits a field gets them. The examples below still pass every field explicitly so the numbers are visible.

If you point this at the older two-stage repo (BBBBruce/SANA-WM_bidirectional-diffusers), startup fails with Model class SanaWmTwoStagesPipeline not found in diffusion model registry, because that repo's model_index.json names a class this build does not register. Add --model-class-name SanaWmPipeline to override it.

Verification

Use a short smoke request first:

curl -sS -X POST http://localhost:8091/v1/videos/sync \
  -H "Accept: video/mp4" \
  -F "prompt=A slow forward camera move through a quiet city street." \
  -F "negative_prompt=blurry, low quality, distorted, watermark" \
  -F "input_reference=@/path/to/first_frame.png;type=image/png" \
  -F "width=1280" \
  -F "height=704" \
  -F "num_frames=9" \
  -F "fps=16" \
  -F "num_inference_steps=2" \
  -F "guidance_scale=5.0" \
  -F "seed=42" \
  --form-string 'extra_params={"sana_wm":{"action":"w-8","translation_speed":0.055,"rotation_speed_deg":1.2,"intrinsics":{"fx":640,"fy":640,"cx":640,"cy":352}}}' \
  -o sana_wm_smoke.mp4

For a production-length request, note that the action durations must sum to num_frames - 1 — the rollout includes the identity start pose — and a mismatch is rejected rather than padded or truncated:

curl -sS -X POST http://localhost:8091/v1/videos/sync \
  -H "Accept: video/mp4" \
  -F "prompt=A slow forward camera move through a quiet city street." \
  -F "negative_prompt=blurry, low quality, distorted, watermark" \
  -F "input_reference=@/path/to/first_frame.png;type=image/png" \
  -F "width=1280" \
  -F "height=704" \
  -F "num_frames=161" \
  -F "fps=16" \
  -F "num_inference_steps=60" \
  -F "guidance_scale=5.0" \
  -F "seed=42" \
  --form-string 'extra_params={"sana_wm":{"action":"w-160","translation_speed":0.055,"rotation_speed_deg":1.2,"intrinsics":{"fx":640,"fy":640,"cx":640,"cy":352}}}' \
  -o sana_wm_output.mp4

Use POST /v1/videos instead when you want job storage and polling rather than inline MP4 bytes. It accepts the same form fields as /v1/videos/sync.

create_response=$(curl -sS -X POST http://localhost:8091/v1/videos \
  -H "Accept: application/json" \
  -F "prompt=A slow forward camera move through a quiet city street." \
  -F "negative_prompt=blurry, low quality, distorted, watermark" \
  -F "input_reference=@/path/to/first_frame.png;type=image/png" \
  -F "width=1280" \
  -F "height=704" \
  -F "num_frames=161" \
  -F "fps=16" \
  -F "num_inference_steps=60" \
  -F "guidance_scale=5.0" \
  -F "seed=42" \
  --form-string 'extra_params={"sana_wm":{"action":"w-160","translation_speed":0.055,"rotation_speed_deg":1.2,"intrinsics":{"fx":640,"fy":640,"cx":640,"cy":352}}}')

video_id=$(echo "$create_response" | jq -r '.id')
curl -sS "http://localhost:8091/v1/videos/${video_id}" | jq .
curl -L "http://localhost:8091/v1/videos/${video_id}/content" -o sana_wm_output.mp4

Notes

• Sequence parallelism is not supported. The bidirectional gated delta recurrence carries state across frames, so a rank cannot denoise a slice of the token sequence in isolation; supporting it needs a distributed scan or an all-gather before the GDN blocks.

• input_reference is required for the first frame. Use image_reference only when you need a JSON-safe image URL or data URL instead of a multipart upload.

• sana_wm must provide exactly one of action or camera.

• Action strings use comma-separated <keys>-<duration> segments. Supported keys are w, a, s, d for translation and i, j, k, l for pitch/yaw rotation. The durations must sum to num_frames - 1.

• Explicit camera control (alternative to action): pass "camera": {"poses": [...]} where poses is a list of num_frames camera-to-world 4x4 matrices (row-major, OpenCV +X right, +Y down, +Z forward convention), e.g. extra_params={"sana_wm":{"camera":{"poses":[[[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]], ...]},"intrinsics":{...}}}. Most callers should prefer action; explicit poses exist for callers that already have a per-frame trajectory.

• Explicit intrinsics are recommended and take the mapping form {"fx":640,"fy":640,"cx":640,"cy":352} (for 1280x704). This {fx,fy,cx,cy} mapping is the only accepted intrinsics form; omit intrinsics to derive them from the output resolution. All four values must be finite, and fx/fy must be positive — the ray map divides by them.

• The video API returns decoded MP4 bytes and has no output_type field, so raw Stage-1 latents are reachable only from the offline API (OmniDiffusionSamplingParams(output_type="latent")); see tests/e2e/offline_inference/test_sana_wm.py. Putting output_type in extra_params does not work: it lands in sampling_params.extra_args, while the pipeline reads the top-level field.