[Refactor] Migrate VACE example to standard task examples + model_extras (#4648)

Signed-off-by: ForestWisdom <zszhang01@gmail.com>
This commit is contained in:
Zhisen Zhang authored and GitHub committed 2026-06-24 13:54:36 +00:00
1 parent bb973cb10c
commit 0814d845dc
13 files changed
+645 -398

No files matched your search

@@ -1,8 +1,7 @@
# Image-To-Video
This example demonstrates how to generate videos from images (image-to-video)
using Wan2.2, LTX2, HunyuanVideo-1.5, and Cosmos3 with vLLM-Omni's offline
inference API.
This shared example generates videos from images with VACE, Wan2.2, LTX2,
HunyuanVideo-1.5, Cosmos3, and other compatible pipelines.
## Local CLI Usage
@@ -68,7 +67,10 @@ python image_to_video.py \
Key arguments:
- `--model`: Model ID (I2V-A14B for MoE, TI2V-5B for unified T2V+I2V).
- `--image`: Path to input image (required).
- `--image`: Path to the first-frame or source image.
- `--last-image`: Optional last-frame condition for models such as VACE.
- `--mask-image`: Optional inpainting mask. White pixels are regenerated and black pixels are preserved.
- `--reference-image`: Optional reference image; repeat it to provide multiple references.
- `--extra-body`: JSON object of model-specific generation params, filtered against the model's declared `extra_body_params` (see [`vllm_omni/model_extras`](../../../vllm_omni/model_extras)). Used by Cosmos3.
- `--prompt`: Text description of desired motion/animation.
- `--height/--width`: Output resolution (auto-calculated from image if not set). Dimensions should be multiples of 16.
@@ -94,5 +96,98 @@ Key arguments:
> ℹ️ If you encounter OOM errors, try using `--vae-use-slicing` and `--vae-use-tiling` to reduce memory usage.
## Wan2.1 VACE Conditional Tasks
The shared script selects the VACE conditioning structure from the media inputs.
No explicit mode parameter is required: the script constructs the source video,
mask, or reference images consumed by the VACE pipeline.
Download the same Hugging Face assets used by the original VACE example:
```bash
wget -O astronaut.jpg https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg
wget -O vace_first_frame.png https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_first_frame.png
wget -O vace_last_frame.png https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_last_frame.png
```
### Image-to-Video (I2V)
```bash
python image_to_video.py \
--model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
--image astronaut.jpg \
--prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \
--seed 42 --height 480 --width 832 --num-frames 81 \
--num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
--vae-use-tiling --output vace_i2v_output.mp4
```
### Video-to-Last-Frame (V2LF)
```bash
python image_to_video.py \
--model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
--last-image astronaut.jpg \
--prompt "An astronaut emerging from a cracked, otherworldly egg on the surface of the moon" \
--seed 42 --height 480 --width 832 --num-frames 81 \
--num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
--vae-use-tiling --output vace_v2lf_output.mp4
```
### First-Last-Frame-to-Video (FLF2V)
```bash
python image_to_video.py \
--model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
--image vace_first_frame.png \
--last-image vace_last_frame.png \
--prompt "CG animation style, a small blue bird takes off from a branch and lands on another branch" \
--seed 42 --height 512 --width 512 --num-frames 81 \
--num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
--vae-use-tiling --output vace_flf2v_output.mp4
```
### Inpainting
Create a mask matching the original VACE example: a 160-pixel-wide white
vertical stripe marks the region to regenerate.
```bash
python - <<'PY'
from PIL import Image
mask = Image.new("L", (832, 480), 0)
mask.paste(255, (336, 0, 496, 480))
mask.save("vace_center_mask.png")
PY
```
```bash
python image_to_video.py \
--model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
--image astronaut.jpg \
--mask-image vace_center_mask.png \
--prompt "Shrek, the ogre, walks out of a building in a happy mood" \
--seed 42 --height 480 --width 832 --num-frames 81 \
--num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
--vae-use-tiling --output vace_inpaint_output.mp4
```
### Reference-to-Video (R2V)
Repeat `--reference-image` to provide more than one reference image.
```bash
python image_to_video.py \
--model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
--reference-image astronaut.jpg \
--prompt "Camera slowly zooms out from the character walking in a garden" \
--seed 42 --height 480 --width 832 --num-frames 81 \
--num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
--vae-use-tiling --output vace_r2v_output.mp4
```
The VACE T2V command is documented in the shared
[`text_to_video.py`](../text_to_video/text_to_video.md#wan21-vace-t2v) example.
For Wan2.2 LightX2V-converted local Diffusers directories and related LoRA
assets, see the [LoRA guide](../../../docs/user_guide/diffusion/lora.md#wan22-lightx2v-offline-assembly).
@@ -2,13 +2,14 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
Image-to-Video generation example using Wan2.2 I2V/TI2V models, LTX2, or HunyuanVideo-1.5.
Image-to-Video generation example using Wan2.2 I2V/TI2V models, LTX2, HunyuanVideo-1.5, or Wan2.1 VACE.
Supports:
- Wan2.2-I2V-A14B-Diffusers: MoE model with CLIP image encoder
- Wan2.2-TI2V-5B-Diffusers: Unified T2V+I2V model (dense 5B)
- LTX2 image-to-video pipeline
- HunyuanVideo-1.5 I2V: SigLIP + VAE dual image conditioning
- Wan2.1 VACE: first/last-frame, inpainting, and reference conditioning
Usage:
# Wan I2V-A14B (MoE)
@@ -53,7 +54,11 @@ from vllm_omni.diffusion.data import DiffusionParallelConfig
from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
from vllm_omni.model_extras import get_extra_body_params, get_model_class_name
from vllm_omni.model_extras import (
build_image_to_video_prompt,
get_extra_body_params,
get_model_class_name,
)
from vllm_omni.outputs import OmniRequestOutput
from vllm_omni.platforms import current_omni_platform
@@ -73,18 +78,28 @@ parse_profiler_config = functools.partial(parse_json_object, flag_name="--profil
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Generate a video from an image (Wan2.2, LTX2, HunyuanVideo-1.5).")
parser = argparse.ArgumentParser(
description="Generate a video from one or more images (Wan2.2, LTX2, HunyuanVideo-1.5, or Wan2.1 VACE)."
)
parser.add_argument(
"--model",
default="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
help="Diffusers I2V model ID or local path (Wan2.2 or HunyuanVideo-1.5).",
help="Diffusers I2V model ID or local path (Wan2.2, HunyuanVideo-1.5, or Wan2.1 VACE).",
)
parser.add_argument(
"--model-class-name",
default=None,
help="Override model class name (e.g., LTX2ImageToVideoPipeline).",
)
parser.add_argument("--image", required=True, help="Path to input image.")
parser.add_argument("--image", help="Path to the first-frame or source image.")
parser.add_argument("--last-image", help="Path to a last-frame condition (used by models such as VACE).")
parser.add_argument("--mask-image", help="Path to an inpainting mask (used by models such as VACE).")
parser.add_argument(
"--reference-image",
action="append",
default=None,
help="Path to a reference image. Repeat to provide multiple references.",
)
parser.add_argument("--prompt", default="", help="Text prompt describing the desired motion.")
parser.add_argument("--negative-prompt", default="", help="Negative prompt.")
parser.add_argument("--seed", type=int, default=42, help="Random seed.")
@@ -302,8 +317,15 @@ def main():
if model_class_name is None and is_ltx2:
model_class_name = "LTX2ImageToVideoPipeline"
# Load input image
image = PIL.Image.open(args.image).convert("RGB")
image = PIL.Image.open(args.image).convert("RGB") if args.image else None
last_image = PIL.Image.open(args.last_image).convert("RGB") if args.last_image else None
mask_image = PIL.Image.open(args.mask_image).convert("L") if args.mask_image else None
reference_images = (
[PIL.Image.open(path).convert("RGB") for path in args.reference_image] if args.reference_image else None
)
dimension_image = image or last_image or (reference_images[0] if reference_images else None)
if dimension_image is None:
raise ValueError("Provide --image, --last-image, or at least one --reference-image.")
# Per-model generation defaults, applied only when the matching flag is omitted.
# Cosmos3 would otherwise silently inherit the Wan2.2 defaults (wrong size/steps/shift).
@@ -333,12 +355,21 @@ def main():
height = args.height
width = args.width
if height is None or width is None:
calc_height, calc_width = calculate_dimensions(image, max_area=d_max_area, mod_value=d_mod)
calc_height, calc_width = calculate_dimensions(dimension_image, max_area=d_max_area, mod_value=d_mod)
height = height or calc_height
width = width or calc_width
# Resize image to target dimensions
image = image.resize((width, height), PIL.Image.Resampling.LANCZOS)
media_inputs: dict[str, Any] = {}
if image is not None:
media_inputs["image"] = image.resize((width, height), PIL.Image.Resampling.LANCZOS)
if last_image is not None:
media_inputs["last_image"] = last_image.resize((width, height), PIL.Image.Resampling.LANCZOS)
if mask_image is not None:
media_inputs["mask"] = mask_image.resize((width, height), PIL.Image.Resampling.NEAREST)
if reference_images is not None:
media_inputs["reference_images"] = [
reference.resize((width, height), PIL.Image.Resampling.LANCZOS) for reference in reference_images
]
# Configure cache based on backend type
cache_config = None
@@ -409,8 +440,8 @@ def main():
if args.extra_body and "guardrails" in args.extra_body:
omni_kwargs["model_config"] = {"guardrails": bool(args.extra_body["guardrails"])}
omni = Omni(**omni_kwargs)
resolved_model_class_name = get_model_class_name(omni)
declared_extra_body_params = get_extra_body_params(resolved_model_class_name)
model_class_name = get_model_class_name(omni) or model_class_name
declared_extra_body_params = get_extra_body_params(model_class_name)
if profiler_enabled:
print("[Profiler] Starting profiling...")
@@ -434,6 +465,15 @@ def main():
print(f" Video size: {args.width}x{args.height}")
print(f"{'=' * 60}\n")
prompt_dict = build_image_to_video_prompt(
model_class_name=model_class_name,
prompt=args.prompt,
negative_prompt=args.negative_prompt,
media_inputs=media_inputs,
height=height,
width=width,
num_frames=num_frames,
)
sampling_params = OmniDiffusionSamplingParams(
height=height,
width=width,
@@ -461,14 +501,7 @@ def main():
generation_start = time.perf_counter()
# omni.generate() returns Generator[OmniRequestOutput, None, None]
frames = omni.generate(
{
"prompt": args.prompt,
"negative_prompt": args.negative_prompt,
"multi_modal_data": {"image": image},
},
sampling_params,
)
frames = omni.generate(prompt_dict, sampling_params)
generation_end = time.perf_counter()
generation_time = generation_end - generation_start
@@ -6,6 +6,7 @@ A unified script for text-to-video generation. Supports multiple models with mod
| Model | Default Resolution | Default Frames | Default Steps | Guidance | VRAM (BF16) |
|---|---|---|---|---|---|
| `Wan-AI/Wan2.1-VACE-1.3B-diffusers` | 480x832 | 81 | 30 | 5.0 | ~20 GiB (RTX 5090, VAE tiling) |
| `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | 720x1280 | 81 | 40 | 4.0 | ~60 GiB |
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v` | 480x832 | 121 | 50 | 6.0 | 1×A100 80GB |
| `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | 720x1280 | 121 | 50 | 6.0 | FP8 + VAE tiling required |
@@ -31,6 +32,28 @@ python text_to_video.py \
--output t2v_out.mp4
```
### Wan2.1 VACE (T2V)
VACE text-to-video uses this shared entrypoint. Conditional VACE tasks use
the shared [`image_to_video.py`](../image_to_video/README.md#wan21-vace-conditional-tasks)
entrypoint, which constructs the pipeline-native conditioning data from the
provided media inputs. No explicit mode parameter is required.
```bash
python text_to_video.py \
--model Wan-AI/Wan2.1-VACE-1.3B-diffusers \
--prompt "A sleek, humanoid robot stands in a vast warehouse filled with neatly stacked cardboard boxes on industrial shelves." \
--seed 0 \
--height 480 \
--width 832 \
--num-frames 81 \
--num-inference-steps 30 \
--guidance-scale 5.0 \
--flow-shift 5.0 \
--vae-use-tiling \
--output vace_t2v_output.mp4
```
LTX2 example:
```bash
@@ -19,6 +19,15 @@ from vllm_omni.outputs import OmniRequestOutput
from vllm_omni.platforms import current_omni_platform
_MODEL_PRESETS = {
"vace": {
"height": 480,
"width": 832,
"num_frames": 81,
"num_inference_steps": 30,
"guidance_scale": 5.0,
"fps": 16,
"output": "vace_t2v_output.mp4",
},
"wan": {
"height": 720,
"width": 1280,
@@ -61,6 +70,8 @@ _MODEL_PRESETS = {
def _detect_preset(model: str) -> dict:
model_lower = model.lower()
if "vace" in model_lower:
return _MODEL_PRESETS["vace"]
if "cosmos" in model_lower:
return _MODEL_PRESETS["cosmos"]
if "hunyuan" in model_lower:
@@ -381,7 +392,7 @@ def main():
omni_kwargs["model_config"] = {"guardrails": bool(args.extra_body["guardrails"])}
omni = Omni(**omni_kwargs)
model_class_name = get_model_class_name(omni)
model_class_name = get_model_class_name(omni) or model_class_name
declared_extra_body_params = get_extra_body_params(model_class_name)
if profiler_enabled:
@@ -419,7 +430,6 @@ def main():
sampling_kwargs["guidance_scale_2"] = args.guidance_scale_high
sampling_params = OmniDiffusionSamplingParams(**sampling_kwargs)
# Route model-specific knobs through extra_body, filtered against the model's
# declared extra_body_params. Models without a declaration only forward explicit
# --extra-body JSON (preserving the generic flags' legacy behavior).
@@ -1,148 +0,0 @@
# VACE Video Generation
Generate videos from text prompts, images, or video conditions using vLLM-Omni's VACE diffusion pipeline.
- `vace_video_generation.py`: command-line script for multi-mode video generation with advanced options.
## Table of Contents
- [Overview](#overview)
- [Quick Start](#quick-start)
- [Key Arguments](#key-arguments)
- [More CLI Examples](#more-cli-examples)
## Overview
[VACE](https://github.com/ali-vilab/VACE) (Video All-in-one Creation Engine) supports multiple video tasks through a single unified model, including text-to-video, image-to-video, first-last-frame interpolation, inpainting, and reference image-guided generation.
### Supported Models
| Model | Architecture | Peak VRAM (GiB) * | Model Weights (GiB) | HuggingFace |
|-------|-------------|-------------------|---------------------|-------------|
| Wan2.1-VACE (1.3B) | Wan2.1 | TBD | ~10 | [Wan-AI/Wan2.1-VACE-1.3B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-1.3B-diffusers) |
| Wan2.1-VACE (14B) | Wan2.1 | TBD | ~38 | [Wan-AI/Wan2.1-VACE-14B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B-diffusers) |
!!! info
*Peak VRAM: based on basic single-card usage, 480x832 resolution, 81 frames, 30 inference steps, without any acceleration/optimization features.
Default model: `Wan-AI/Wan2.1-VACE-14B-diffusers`
## Quick Start
### Python API
Text-to-video generation:
```python
import torch
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
if __name__ == "__main__":
omni = Omni(model="Wan-AI/Wan2.1-VACE-1.3B-diffusers")
prompt = "A sleek robot stands in a vast warehouse filled with boxes"
outputs = omni.generate(
prompt,
OmniDiffusionSamplingParams(
height=480,
width=832,
num_frames=81,
num_inference_steps=30,
guidance_scale=5.0,
),
)
video = outputs[0].images
from diffusers.utils import export_to_video
export_to_video(list(video[0]), "t2v_output.mp4", fps=16)
omni.close()
```
### Local CLI Usage
```bash
python vace_video_generation.py \
--mode t2v \
--prompt "A sleek robot stands in a vast warehouse filled with boxes" \
--height 480 --width 832 --num-frames 81 \
--output t2v_output.mp4
```
## Key Arguments
| Argument | Type | Default | Description |
| -------- | ---- | ------- | ----------- |
| `--mode` | str | `"t2v"` | VACE task mode: `t2v`, `i2v`, `v2lf`, `flf2v`, `inpaint`, `r2v` |
| `--model` | str | `"Wan-AI/Wan2.1-VACE-14B-diffusers"` | Model ID or local path |
| `--prompt` | str | `"A cat walking in a garden"` | Text description of desired video |
| `--negative-prompt` | str | `""` | Negative prompt for classifier-free guidance |
| `--image` | str | `None` | Input image path (for I2V, R2V, FLF2V, inpaint modes) |
| `--last-image` | str | `None` | Last frame image path (for FLF2V mode) |
| `--height` | int | `480` | Output video height in pixels (should be a multiple of 16) |
| `--width` | int | `832` | Output video width in pixels (should be a multiple of 16) |
| `--num-frames` | int | `81` | Number of video frames to generate |
| `--num-inference-steps` | int | `30` | Number of denoising steps (more steps = higher quality, slower) |
| `--guidance-scale` | float | `5.0` | Classifier-free guidance scale |
| `--flow-shift` | float | `5.0` | Scheduler flow shift parameter |
| `--seed` | int | `42` | Random seed for deterministic sampling |
| `--fps` | int | `16` | Frames per second for the saved MP4 |
| `--output` | str | `"vace_output.mp4"` | Path to save the generated video |
| `--vae-use-tiling` | flag | on | Enable VAE tiling for memory optimization |
| `--cfg-parallel-size` | int | `1` | Set to `2` to enable CFG Parallel |
| `--ulysses-degree` | int | `1` | Ulysses sequence parallel degree for multi-GPU inference |
| `--ring-degree` | int | `1` | Ring sequence parallel degree for hybrid Ulysses + Ring inference |
| `--tensor-parallel-size` | int | — | Tensor parallel size |
| `--enforce-eager` | flag | off | Disable torch.compile |
> If you encounter OOM errors, try `--vae-use-tiling` or multi-GPU parallelism options (`--ulysses-degree`, `--cfg-parallel-size`).
## More CLI Examples
### Image-to-Video (I2V)
First frame is kept, remaining frames are generated:
```bash
python vace_video_generation.py \
--mode i2v \
--image astronaut.jpg \
--prompt "An astronaut emerging from a cracked egg on the moon" \
--height 480 --width 832 --num-frames 81 \
--output i2v_output.mp4
```
### First-Last-Frame Interpolation (FLF2V)
```bash
python vace_video_generation.py \
--mode flf2v \
--image first_frame.jpg --last-image last_frame.jpg \
--prompt "A bird takes off from a branch and lands on another" \
--height 512 --width 512 --num-frames 81 \
--output flf2v_output.mp4
```
### Inpainting
Center vertical stripe is masked and regenerated:
```bash
python vace_video_generation.py \
--mode inpaint \
--image scene.jpg \
--prompt "Shrek walks out of a building" \
--height 480 --width 832 --num-frames 81 \
--output inpaint_output.mp4
```
### Reference Image-guided (R2V)
```bash
python vace_video_generation.py \
--mode r2v \
--image reference.jpg \
--prompt "Camera slowly zooms out from the character" \
--height 480 --width 832 --num-frames 81 \
--output r2v_output.mp4
```
@@ -1,209 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""VACE video generation example.
VACE (Video All-in-one Creation Engine) supports multiple video tasks:
- T2V: Text-to-Video
- I2V: Image-to-Video (first frame conditioning)
- V2LF: Video-to-Last-Frame
- FLF2V: First-Last-Frame interpolation
- Inpainting: Masked region generation
- R2V: Reference image-guided generation
Usage examples:
# T2V (text-to-video)
python vace_video_generation.py --mode t2v --prompt "A robot in a warehouse"
# I2V (image-to-video, first frame kept)
python vace_video_generation.py --mode i2v --image input.jpg --prompt "..."
# FLF2V (first-last frame interpolation)
python vace_video_generation.py --mode flf2v --image first.jpg --last-image last.jpg
# R2V (reference image guided)
python vace_video_generation.py --mode r2v --image ref.jpg --prompt "..."
"""
import argparse
import time
from pathlib import Path
import numpy as np
import PIL.Image
import torch
from vllm_omni.diffusion.data import DiffusionParallelConfig
from vllm_omni.entrypoints.omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
from vllm_omni.platforms import current_omni_platform
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="VACE video generation.")
parser.add_argument(
"--model",
default="Wan-AI/Wan2.1-VACE-14B-diffusers",
help="VACE model ID or local path.",
)
parser.add_argument(
"--mode",
default="t2v",
choices=["t2v", "i2v", "v2lf", "flf2v", "inpaint", "r2v"],
help="Generation mode.",
)
parser.add_argument("--prompt", default="A cat walking in a garden", help="Text prompt.")
parser.add_argument("--negative-prompt", default="", help="Negative prompt.")
parser.add_argument("--image", type=str, default=None, help="Input image path (for I2V, R2V, FLF2V, inpaint).")
parser.add_argument("--last-image", type=str, default=None, help="Last frame image path (for FLF2V).")
parser.add_argument("--video-dir", type=str, default=None, help="Directory of video frames (for inpaint).")
parser.add_argument("--seed", type=int, default=42, help="Random seed.")
parser.add_argument("--guidance-scale", type=float, default=5.0, help="CFG guidance scale.")
parser.add_argument("--height", type=int, default=480, help="Video height.")
parser.add_argument("--width", type=int, default=832, help="Video width.")
parser.add_argument("--num-frames", type=int, default=81, help="Number of frames.")
parser.add_argument("--num-inference-steps", type=int, default=30, help="Sampling steps.")
parser.add_argument("--flow-shift", type=float, default=5.0, help="Scheduler flow_shift.")
parser.add_argument("--output", type=str, default="vace_output.mp4", help="Output video path.")
parser.add_argument("--fps", type=int, default=16, help="Output video FPS.")
parser.add_argument("--vae-use-tiling", action="store_true", default=True, help="Enable VAE tiling.")
parser.add_argument("--enforce-eager", action="store_true", help="Disable torch.compile.")
parser.add_argument("--ulysses-degree", type=int, default=1, help="Ulysses SP degree.")
parser.add_argument("--ring-degree", type=int, default=1, help="Ring attention degree.")
parser.add_argument("--cfg-parallel-size", type=int, default=1, choices=[1, 2], help="CFG parallel size.")
return parser.parse_args()
def build_prompts(args):
"""Build prompt dict with multi_modal_data based on mode."""
h, w, nf = args.height, args.width, args.num_frames
gray = PIL.Image.new("RGB", (w, h), (128, 128, 128))
mask_black = PIL.Image.new("L", (w, h), 0)
mask_white = PIL.Image.new("L", (w, h), 255)
prompt_data = {
"prompt": args.prompt,
"negative_prompt": args.negative_prompt,
}
if args.mode == "t2v":
return prompt_data
if args.mode == "r2v":
assert args.image, "--image required for R2V mode"
ref_img = PIL.Image.open(args.image).convert("RGB").resize((w, h))
prompt_data["multi_modal_data"] = {"reference_images": [ref_img]}
return prompt_data
if args.mode == "i2v":
assert args.image, "--image required for I2V mode"
img = PIL.Image.open(args.image).convert("RGB").resize((w, h))
prompt_data["multi_modal_data"] = {
"video": [img] + [gray] * (nf - 1),
"mask": [mask_black] + [mask_white] * (nf - 1),
}
return prompt_data
if args.mode == "v2lf":
assert args.image, "--image required for V2LF mode"
img = PIL.Image.open(args.image).convert("RGB").resize((w, h))
prompt_data["multi_modal_data"] = {
"video": [gray] * (nf - 1) + [img],
"mask": [mask_white] * (nf - 1) + [mask_black],
}
return prompt_data
if args.mode == "flf2v":
assert args.image and args.last_image, "--image and --last-image required for FLF2V"
first = PIL.Image.open(args.image).convert("RGB").resize((w, h))
last = PIL.Image.open(args.last_image).convert("RGB").resize((w, h))
prompt_data["multi_modal_data"] = {
"video": [first] + [gray] * (nf - 2) + [last],
"mask": [mask_black] + [mask_white] * (nf - 2) + [mask_black],
}
return prompt_data
if args.mode == "inpaint":
assert args.image, "--image required for inpaint mode"
img = PIL.Image.open(args.image).convert("RGB").resize((w, h))
d = 80
frames, masks = [], []
for _ in range(nf):
base = np.array(img).copy()
mask = PIL.Image.new("L", (w, h), 0)
stripe = PIL.Image.new("L", (2 * d, h), 255)
mask.paste(stripe, (w // 2 - d, 0))
base[np.array(mask) > 128] = 128
frames.append(PIL.Image.fromarray(base))
masks.append(mask)
prompt_data["multi_modal_data"] = {"video": frames, "mask": masks}
return prompt_data
raise ValueError(f"Unknown mode: {args.mode}")
def main():
args = parse_args()
generator = torch.Generator(device=current_omni_platform.device_type).manual_seed(args.seed)
parallel_config = DiffusionParallelConfig(
ulysses_degree=args.ulysses_degree,
ring_degree=args.ring_degree,
cfg_parallel_size=args.cfg_parallel_size,
)
omni = Omni(
model=args.model,
vae_use_tiling=args.vae_use_tiling,
flow_shift=args.flow_shift,
enforce_eager=args.enforce_eager,
parallel_config=parallel_config,
)
prompt_data = build_prompts(args)
print(f"\n{'=' * 60}")
print(f"VACE {args.mode.upper()} Generation")
print(f" Model: {args.model}")
print(f" Size: {args.width}x{args.height}, {args.num_frames} frames, {args.num_inference_steps} steps")
print(f"{'=' * 60}\n")
start = time.perf_counter()
outputs = omni.generate(
prompt_data,
OmniDiffusionSamplingParams(
height=args.height,
width=args.width,
num_frames=args.num_frames,
num_inference_steps=args.num_inference_steps,
guidance_scale=args.guidance_scale,
generator=generator,
),
)
elapsed = time.perf_counter() - start
video = outputs[0].images
if isinstance(video, list):
video = video[0]
if isinstance(video, torch.Tensor):
video = video.cpu().numpy()
if video.ndim == 5:
video = video[0]
print(f"Output shape: {video.shape}, Time: {elapsed:.1f}s")
output_path = Path(args.output)
output_path.parent.mkdir(parents=True, exist_ok=True)
from diffusers.utils import export_to_video
if np.issubdtype(video.dtype, np.integer):
video = video.astype(np.float32) / 255.0
export_to_video(list(video), str(output_path), fps=args.fps)
print(f"Saved to {output_path}")
omni.close()
if __name__ == "__main__":
main()