0 Fork 0

Files

T

Zhisen Zhang 0814d845dc [Refactor] Migrate VACE example to standard task examples + model_extras (#4648 )

Signed-off-by: ForestWisdom <zszhang01@gmail.com>

2026-06-24 13:54:36 +00:00

5.8 KiB

Raw Blame History

Wan2.1 VACE Unified Video Generation

Summary

Vendor: Wan-AI
Models: Wan2.1-VACE-1.3B-diffusers, Wan2.1-VACE-14B-diffusers
Tasks: T2V, I2V, V2LF, FLF2V, inpainting, and R2V
Mode: offline inference through the shared task examples

The shared examples construct VACE's pipeline-native conditioning data from the provided media inputs. They do not require a mode parameter or a model-specific example script.

References

Hardware Support

The 1.3B checkpoint was validated on an RTX 5090; the 81-frame T2V command below used approximately 20 GiB of peak reserved GPU memory with VAE tiling. The 14B checkpoint requires approximately 70 GiB of disk space and was validated on a single 48 GiB L40S with VAE tiling and layerwise offload. Start with the smoke configuration before attempting an 81-frame run.

Setup

The public examples use the Hugging Face model ID by default:

MODEL="${MODEL:-Wan-AI/Wan2.1-VACE-14B-diffusers}"
ASSET_DIR="${ASSET_DIR:-$HOME/datasets/vace}"
OUT="${OUT:-$PWD/vace-results/14b}"
mkdir -p "$ASSET_DIR" "$OUT"

For environments where Hugging Face downloads are unavailable, download the same checkpoint from ModelScope and point MODEL at the local directory:

MODEL="$HOME/models/Wan2.1-VACE-14B-diffusers"
modelscope download --model Wan-AI/Wan2.1-VACE-14B-diffusers --local_dir "$MODEL"

Download the public conditioning images and create a mask matching the output size. White pixels are regenerated and black pixels are preserved.

wget -O "$ASSET_DIR/astronaut.jpg" \
  https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg
wget -O "$ASSET_DIR/vace_first_frame.png" \
  https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_first_frame.png
wget -O "$ASSET_DIR/vace_last_frame.png" \
  https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_last_frame.png

python - "$ASSET_DIR/vace_center_mask.png" <<'PY'
import sys
from PIL import Image

mask = Image.new("L", (832, 480), 0)
mask.paste(255, (336, 0, 496, 480))
mask.save(sys.argv[1])
PY

ASTRONAUT="$ASSET_DIR/astronaut.jpg"
FIRST="$ASSET_DIR/vace_first_frame.png"
LAST="$ASSET_DIR/vace_last_frame.png"
MASK="$ASSET_DIR/vace_center_mask.png"

Commands

The commands below reproduce the original VACE examples with 81 frames / 30 steps. Most tasks use 480x832; FLF2V uses 512x512. For a smoke test, replace the frame and step counts with 5 frames / 2 steps.

# T2V
python examples/offline_inference/text_to_video/text_to_video.py \
  --model "$MODEL" \
  --prompt "A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves." \
  --seed 0 \
  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
  --output "$OUT/t2v.mp4"

# I2V
python examples/offline_inference/image_to_video/image_to_video.py \
  --model "$MODEL" --image "$ASTRONAUT" \
  --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \
  --seed 42 \
  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
  --output "$OUT/i2v.mp4"

# V2LF
python examples/offline_inference/image_to_video/image_to_video.py \
  --model "$MODEL" --last-image "$ASTRONAUT" \
  --prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \
  --seed 42 \
  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
  --output "$OUT/v2lf.mp4"

# FLF2V
python examples/offline_inference/image_to_video/image_to_video.py \
  --model "$MODEL" --image "$FIRST" --last-image "$LAST" \
  --prompt "CG animation style, a small blue bird takes off from a branch and lands on another branch" \
  --seed 42 --height 512 --width 512 --num-frames 81 --num-inference-steps 30 \
  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
  --output "$OUT/flf2v.mp4"

# Inpaint
python examples/offline_inference/image_to_video/image_to_video.py \
  --model "$MODEL" --image "$ASTRONAUT" --mask-image "$MASK" \
  --prompt "Shrek, the ogre, walks out of a building in a happy mood" \
  --seed 42 \
  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
  --output "$OUT/inpaint.mp4"

# R2V; repeat --reference-image to add references
python examples/offline_inference/image_to_video/image_to_video.py \
  --model "$MODEL" --reference-image "$ASTRONAUT" \
  --prompt "Camera slowly zooms out from the character walking in a garden" \
  --seed 42 \
  --height 480 --width 832 --num-frames 81 --num-inference-steps 30 \
  --guidance-scale 5.0 --flow-shift 5.0 --vae-use-tiling --enable-layerwise-offload \
  --output "$OUT/r2v.mp4"

Verification

for video in "$OUT"/*.mp4; do
  ffprobe -v error -select_streams v:0 \
    -show_entries stream=codec_name,width,height,nb_frames,r_frame_rate,duration \
    -of default=noprint_wrappers=1 "$video"
done

For the 1.3B checkpoint, --enable-layerwise-offload is generally unnecessary.

5.8 KiB Raw Blame History